Commit Graph

873 Commits

Author SHA1 Message Date
HugoCasa
1741ce7823 fix ws killpill (#4551)
* websocket killpill

* oups

* nit

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-19 01:17:15 +02:00
HugoCasa
a4d84b837f websocket killpill (#4550) 2024-10-19 00:02:48 +02:00
Ruben Fiszel
0e45cca78d adapt websocket triggers to multi-tenant cloud 2024-10-18 00:52:23 +02:00
HugoCasa
b0fbcd8305 feat: websocket triggers (#4505)
* feat: websocket triggers

* fix

* frontend nits

* nits

* new trigger UI

* sqlx
2024-10-18 00:21:52 +02:00
Ruben Fiszel
2953154a25 fix: cache js static assets by default 2024-10-18 00:12:07 +02:00
pyranota
11623682c2 Deploy collision warning fix (#4544)
* feat: Multiplayer deploy collision warning

Show warning with diff viewer if 2+ users try to deploy flow/script/app at the same time

In other words, if user A and user B started editing same flow/script/app and user A deployed it,
user B will get warning if they try to deploy as well.

* Add warning for scripts

* Add `deployedBy` to Apps

* Format

* Fix advanced deployment on scripts

* Write comments and cleanup

* feat(frontend): unify all triggers UX and simplify flow settings (#4259)

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): clean up

* feat(frontend): improve UX

* feat(frontend): triggers

* feat(frontend): triggers

* feat(frontend): done

* feat(frontend): fix trigger when position when a preprocessor is presetn

* Glm/rework flow settings v2 (#4497)

* fat(frontend): simplify flow settings menu

* improve scroll

* changing mute toggle

* Add advanced settings badge

* Add nord theme colors

* Add bage for advanced options

* fix minor issue

* fix minor issue

* Add triggers menu to flow settings

* Add quick trigger access

* remove triggers in flow settings

* fix minor issue

* Move triggers settings to flow right panel

* polishing

* fix unset store

* remove save up to for triggers

* fix padding

* reset default tag color

* remove custom select component

* revert path change

* revert section modif

* Revert unused feature

---------

Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>

* Connect top bar cron to schedules settings

* Turn copilot into node

* fix copilot placement

* remove useless import

* fix center copilot

* fix binding

* remove copilot on top of preprocessor

* render copilot node on condition

* quickfix

* remove copilot node

* fix minor issues

* fix route count update

* fix schedule sync

* harmonize colors

* fix alignment and add edges

* recenter node summary

* fix schedules sync

* Add id title

* all

* all

* all

* iteration

* all

* all

* done

* fix

* more fixes

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>

* Update ScriptBuilder.svelte

* Remove `onMount` for flows

* Use version instead of last_updated_at in flows

* Use only versions for apps

* Fetch latest data in Diffs in Apps

* Optimize with (script/flow/app)GetLatestVersion

Create several new endpoints, that returns just latest version without rest of the history

* Sync Diffs with deployed

* Improve Diff's data

* Use `getFlowLatestVersion`

* fix failing endpoints for deployed collision warning

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-10-17 18:05:44 +02:00
pyranota
84441b15dc Multiplayer deploy collision warning (#4533)
* feat: Multiplayer deploy collision warning

Show warning with diff viewer if 2+ users try to deploy flow/script/app at the same time

In other words, if user A and user B started editing same flow/script/app and user A deployed it,
user B will get warning if they try to deploy as well.

* Add warning for scripts

* Add `deployedBy` to Apps

* Format

* Fix advanced deployment on scripts

* Write comments and cleanup

* feat(frontend): unify all triggers UX and simplify flow settings (#4259)

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): clean up

* feat(frontend): improve UX

* feat(frontend): triggers

* feat(frontend): triggers

* feat(frontend): done

* feat(frontend): fix trigger when position when a preprocessor is presetn

* Glm/rework flow settings v2 (#4497)

* fat(frontend): simplify flow settings menu

* improve scroll

* changing mute toggle

* Add advanced settings badge

* Add nord theme colors

* Add bage for advanced options

* fix minor issue

* fix minor issue

* Add triggers menu to flow settings

* Add quick trigger access

* remove triggers in flow settings

* fix minor issue

* Move triggers settings to flow right panel

* polishing

* fix unset store

* remove save up to for triggers

* fix padding

* reset default tag color

* remove custom select component

* revert path change

* revert section modif

* Revert unused feature

---------

Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>

* Connect top bar cron to schedules settings

* Turn copilot into node

* fix copilot placement

* remove useless import

* fix center copilot

* fix binding

* remove copilot on top of preprocessor

* render copilot node on condition

* quickfix

* remove copilot node

* fix minor issues

* fix route count update

* fix schedule sync

* harmonize colors

* fix alignment and add edges

* recenter node summary

* fix schedules sync

* Add id title

* all

* all

* all

* iteration

* all

* all

* done

* fix

* more fixes

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>

* Update ScriptBuilder.svelte

* Remove `onMount` for flows

* Use version instead of last_updated_at in flows

* Use only versions for apps

* Fetch latest data in Diffs in Apps

* Optimize with (script/flow/app)GetLatestVersion

Create several new endpoints, that returns just latest version without rest of the history

* Sync Diffs with deployed

* Improve Diff's data

* Use `getFlowLatestVersion`

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-10-17 17:14:48 +02:00
pyranota
c3ba832444 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel
3ec8720a65 fix(git-sync): propagate update of folders with git sync 2024-10-16 23:44:56 +02:00
Faton Ramadani
258445e523 feat(frontend): unify all triggers UX and simplify flow settings (#4259)
* feat(frontend): added list of triggers in the flow graph

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): clean up

* feat(frontend): improve UX

* feat(frontend): triggers

* feat(frontend): triggers

* feat(frontend): done

* feat(frontend): fix trigger when position when a preprocessor is presetn

* Glm/rework flow settings v2 (#4497)

* fat(frontend): simplify flow settings menu

* improve scroll

* changing mute toggle

* Add advanced settings badge

* Add nord theme colors

* Add bage for advanced options

* fix minor issue

* fix minor issue

* Add triggers menu to flow settings

* Add quick trigger access

* remove triggers in flow settings

* fix minor issue

* Move triggers settings to flow right panel

* polishing

* fix unset store

* remove save up to for triggers

* fix padding

* reset default tag color

* remove custom select component

* revert path change

* revert section modif

* Revert unused feature

---------

Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>

* Connect top bar cron to schedules settings

* Turn copilot into node

* fix copilot placement

* remove useless import

* fix center copilot

* fix binding

* remove copilot on top of preprocessor

* render copilot node on condition

* quickfix

* remove copilot node

* fix minor issues

* fix route count update

* fix schedule sync

* harmonize colors

* fix alignment and add edges

* recenter node summary

* fix schedules sync

* Add id title

* all

* all

* all

* iteration

* all

* all

* done

* fix

* more fixes

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-10-16 17:58:15 +02:00
wendrul
a51cbc9e08 fix: canceled jobs button reporting 0 jobs cancelled (#4534) 2024-10-16 16:15:25 +02:00
Ruben Fiszel
f0e48c7228 use push isolation level to isolated in more push cases 2024-10-12 17:00:54 +02:00
Ruben Fiszel
1218bd3831 use isolated transctions for workflow_as_code 2024-10-12 16:23:53 +02:00
Ruben Fiszel
d3d6d9d8b7 add trace_id to all request spans 2024-10-12 15:19:43 +02:00
Ruben Fiszel
7a60166a9c add more tracing on cloud to workflow_as_code 2024-10-12 14:11:21 +02:00
Ruben Fiszel
a4142ef4e2 support skip_update 2024-10-12 13:39:59 +02:00
Ruben Fiszel
8232b4cdb0 fix: improve workflow as code row-lock on db to handle more concurrency 2024-10-12 03:39:43 +02:00
HugoCasa
8841180b5f BYPASSRLS admin policy falllback (#4493)
* BYPASSRLS admin policy falllback

* use fixed list instead

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-08 20:08:29 +02:00
pyranota
1e7a37ac8b feat: Replace pip-compile with uv (#4460)
* Update `shell.nix`

- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout

* Replace `pip-compile` with `uv` (dirty + untested)

* Fix arguments passed to uv

Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url

Also uv does not support `--pip-args` and suggests to directly pass args to uv.

* Remove extra `dbg!`

* Replace 'pip-compile' with 'uv' in Dockerfile

* Add fallback option to `pip-compile` (Disabled)

* Add `uv` to `docker/DockerfileSlim*`

* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`

* Add option to fallback to pip-compile

Put `# no_uv` on top the file for specific python script

Or set `USE_PIP_COMPILE` variable to `true`

* Put back `pip-tools` into shell.nix

* Make sure lockfile resolves again if `#no_uv` used

Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

* Put `pip install pip-tools` in original spot

* Fix compilation error

* Fix EE compilation error

error[E0658]: attributes on expressions are experimental
   --> windmill-worker/src/python_executor.rs:144:5
    |
144 |     #[cfg(feature = "enterprise")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information

* Add `no_cache` annotation

Will force recalculation of lockfile
And block uv from using cached values

* Target uv cache to /tmp/windmill/cache

* Prohibit uv from managing python

* Add uv to DockerfileBackendTests

* Pin uv version to 0.4.18 in Dockerfiles

* Dont put `#no_uv` in requirements.in

Instead postfix hash for requirements.in with `-no_uv`

* Push Warning to logs if fallbacked to pip-compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 16:00:35 +02:00
Alexander Petric
6eb7f8ce81 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00
Ruben Fiszel
16692d5ff6 fix: fix new instance db setup 2024-10-01 16:03:58 +02:00
HugoCasa
62d88436b2 feat: flow step skipping (#4461) 2024-09-30 16:30:59 +02:00
Ruben Fiszel
f438d1adbe fix: skip one migration to avoid using md5 for azure support 2024-09-30 14:02:31 +02:00
HugoCasa
41ff40f777 send stats when renewing key if last >24h (#4430)
* feat: send stats when renewing key if last >24h

* nits

* fix: sqlx

* nit

* renewal reason

* stats reason

* update ee ref

* Update ee-repo-ref.txt

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-09-30 12:23:07 +02:00
Ruben Fiszel
da969da96f improve benchmarking tools (#4450) 2024-09-27 10:48:04 +02:00
Ruben Fiszel
048625404d add suspended flows to cli queues 2024-09-26 16:52:09 +02:00
Ruben Fiszel
5a191d83ca improve list queue api 2024-09-26 14:31:17 +02:00
Ruben Fiszel
bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
wendrul
08ae91e6dd feat: add static variable and resources support to ansible (#4435)
* Add static variable and resources to ansible

* Add wasm parser to cli

* Fix pulling of resources, rs and playbooks

* Remove unused import, add publish pkg-yaml

* Update package.json

* Update generate-metadata

* Fix null fields being serialized in script metadata

* npm install
2024-09-25 15:05:53 +02:00
HugoCasa
1feccd493e feat: add tag filtering to external JWT authentication (#4425)
* feat: tag filtering jwt ext auth

* move tags to scopes

* fix symlink

* update ee ref
2024-09-25 11:21:19 +02:00
pyranota
945d1a7e51 fix get_job_update returning error (#4427)
When get_progress specified to true, but there is no record of progress for given job
Endpoint will fail and return error.

Nothing cricial but should be fixed.
2024-09-24 10:24:26 +02:00
wendrul
a7ef3ed293 fix: various improvements for ansible (#4419)
* Improve error messages, fix dependency job issue

* Add nsjail support for ansible

* Remove invalid comment syntax

* Change nsjail config and ansible controller config to be ephemeral

* Move ansible cfg to later to overwrite possible files named ansible.cfg

* Fix ansible.cfg env var

* Fix tmp dir issue
2024-09-23 16:44:37 +02:00
Ruben Fiszel
b1862ab2bd critical channels build on non ee 2024-09-21 22:54:15 +02:00
Ruben Fiszel
efbcb1ffdd fix: migrate smtp instance settings to global settings (#4416)
* all

* all

* all

* fix

* update ee private

* all
2024-09-21 01:42:15 +02:00
HugoCasa
ef74468506 fix: allow no body in job requests (#4413) 2024-09-20 18:26:50 +02:00
HugoCasa
ae8f29a4cb feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
wendrul
b0cb1e0b27 feat: ansible playbook support (#4399)
* Ansible execution and parsing

* Working collections and pip dependencies

* Remove unused vars

* File resources logic for ansible

* Make frontend for ansible and resource file

* Format file

* Change naming of file resource and inventory fields

* Add autocomplete for file extension resource creation

* Add endpoint to list file formats for resource types

* Add CLI functionality to pull/push file resources

* Add beta tag to ansible

* Add Full image dockerfiles containing ansible

* Prepare sqlx

* Update cargo.lock

* Update ansible path

* Remove unused imports

* Add back import removed by rust-analyzer

* Improve ansible init code

* Prepare sqlx

* Change dockerfile to make the full windmill image

* Remove old dockerfile

* Improve autocomplete file resource extension select

* Remove comment

* Validate file extension

* Add icons to text file resources

* Remove editability of file resource types

* Remove unused import

* Missing space

* Add yaml parser
2024-09-19 00:21:15 +02:00
pyranota
d3e0b1c636 feat: Allow setProgress and getProgress from within the script (#4400)
* Allow setting progress explicitly from script body.

This feature exposes:
 * `getProgress`
 * `setProgress`
 * `incProgress`

API in TypeScript client (python is coming soon).

NOTE: Progress cannot be out of range 0..100 and cannot decrease.

With exposed APIs there is also UI changes, so progress can be shown for individual jobs as well.

For optimization reasons, jobs start to ask for progress only after N-seconds of execution.

* feat: Add `shell.nix`

If you dont have anything but nix, dont worry, run nix-shell in root, or activate with direnv and get all needed dependencies

NOTE: You will still need docker

* feat: Add `dev.nu` to typescript client

Little helper function, allowing developer to work on ts client easier.

To use:
  `./dev.nu watch`

Now add import of windmill in body of your script and `//nobundle` on top of the file

Edit ts client in your favourite editor and hit save. Script will do the rest.

* Cleanup files

* Fix: Failed to deserialize query string: missing field `get_progress`

* perf: Implement non-naive polling mechanism for getting job progress

* Add independant delay for getProgress

Problem in `TestJobLoader`:

There should be 2 delays:

    One until we find our first progress (every 5s)
    Once we found our first progress, we can do it every second

* nit: Use `query_scalar!` instead of `query_as`

* Fix: Sql error, no rows returned by a query that expected to return at least one row

* refactor: Remove global CSS for JobProgressBar

* Change UI for progress of flow subjobs

* Replace `Step 1` with `Running` in ProgressBar for individual jobs

* Remove `incProgress`

incProgress is not very usefull and error-prone

* perf: Set metric only for jobs that are actually using it

(https://github.com/windmill-labs/windmill/pull/4373#discussion_r1759843773)

* Offload registering progress from clients to server

* Add `jobId?` argument to typescript-client's `setProgress` and `getProgress`

Allows to set progress of other jobs and flows,
if jobId specified, than flow id will be inferred automatically.

Could be used by SDK.

* Add `Error::MetricNotFound` for better error handling

* Fix: Make `JobProgressBar` display in red when failed

* Add persistant progress bar

Now you can reload the page after job is done and progress will be still there

* Allow succeeded individual job's progress bar stick to 100%

* Add python support 

* nit: Remove usage of undefined variable in python-client

* Add `async` in ts client (for error handling)

* nit(frontend): Remove unused import

* Dont load JobProgressBar when it is not needed

* nit: npm check fix

* cargo sqlx prepare

* fix sqlx

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-18 01:16:19 +02:00
Ruben Fiszel
87bf6b8511 rm legacy hubScripts 2024-09-17 17:33:52 +02:00
Ruben Fiszel
f864709ab2 fix: cannot create duplicate apps and raw_apps 2024-09-16 21:15:29 +02:00
HugoCasa
d3e8731be2 feat: failed key renewal alert + renew on start if no recent renewal (#4387)
* feat: failed key renewal alert + renew on start if no recent renewal

* Update ee-repo-ref.txt
2024-09-16 16:54:35 +02:00
Ruben Fiszel
d533a017a7 fix: improve first time setup experience 2024-09-15 21:02:36 +02:00
HugoCasa
6ca628c811 fix: queue metrics graph (#4388) 2024-09-13 23:34:48 +02:00
HugoCasa
d3dc65b740 fix: tighten inputs for granular kinds (#4379)
* fix(backend): acls perms

* nit
2024-09-12 14:40:22 +02:00
Ruben Fiszel
7d29651d3e fix: graceful worker exits for same worker jobs (#4371)
* all

* fix compile
2024-09-11 18:32:08 +02:00
Ruben Fiszel
d55159367d fix: default success handler key can be viewed by anyone in the workspace 2024-09-11 12:16:10 +02:00
Ruben Fiszel
8423e5f8c2 put sed helpers to remove anscii codes in downloaded logs 2024-09-09 10:26:17 +02:00
Ruben Fiszel
5817efc16f fix(cli): guide users to migrate to node version of the CLI 2024-09-07 21:27:36 +02:00
HugoCasa
a009c8df23 feat: schedule success handler (#4346) 2024-09-06 19:03:09 +02:00
Ruben Fiszel
4c174cb359 feat: add load more to runs page if nb of jobs >= 1000 2024-09-04 22:45:35 +02:00