Commit Graph

125 Commits

Author SHA1 Message Date
Diego Imbert
7467833239 EE Refactor (#5844)
* app compiles with every ee substituted

* Replace all oss files content

* Revert "Replace all oss files content"

This reverts commit ea4017d59f.

* delete all ee

* hide all _ee files under private flag

* hide every oss stuff when private flag set

* pub use *

* gitignore and substitute script

* pub mod for ee needed for ee repo

* small mistakes

* remove oidc_oss impl

* ee ref (temp)

* ee ref

* fix --all-features selecting private in OSS CI

* ee repo ref

* allow unused
2025-06-02 22:12:33 +02:00
dieriba
b2c7bdff13 feat: raise error if end early in flow (#5653)
* update front

* handle err_msg display if enabled when stop condition is met

* nits

* Update backend/windmill-worker/src/worker_flow.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update openflow.openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update worker_flow.rs

* Update worker_flow.rs

* update flow test

* update condition

* remove println

* nits

* fix flow test

* nits and missing property on struct definition

* nits

* fix unsaved state

* update .sqlx

* cleanup sqlx

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-05-07 18:23:40 +02:00
Diego Imbert
1d47c4ad08 feat: list references upon renaming a script or a flow (#5487)
* Refactored flow_workspace_runnables to more generic workspace_runnable_dependencies

* list flows referencing an item upon renaming it

* Refactor with two exclusive columns to avoid breaking FK constraints

* Show apps depending on item upon renaming

* sqlx prepare

* list-disc instead of •

* on delete and on update cascade

* displayPathChangedWarning oneOf check instead noneOf

* combine migrations + add "on update cascade" to flow fk

* unique index on app dependencies to avoid duplicates

* create new workspace_runnable_dependencies instead of renaming old table

* Add "looking for references" loading msg

* Revert "create new workspace_runnable_dependencies instead of renaming old table"

This reverts commit 015c38ca8f.

* flow_workspace_runnables view for backwards compatibility

* Add warning for script imports on rename

* support import dependency tracking in deno

* number of using scripts / flows / apps tooltip

* forgot sqlx prepare

* delete app-related rows in down migration
2025-03-24 22:08:19 +01:00
HugoCasa
1e1bc56f33 feat: keep captures across drafts and deploys (#5482) 2025-03-14 15:17:48 +01:00
HugoCasa
fd3459568f feat: track workspace runnables used in flows (#5369)
* feat: track workspace runnables used in flows

* track script hash

* weird

* do it with lock

* Revert "feat: add support for | None and Optional in python (#5361)"

This reverts commit 69a2d85f25.

* Revert "Revert "feat: add support for | None and Optional in python (#5361)""

This reverts commit bb8f709894.

* update openapi

* delete old in lock_modules + don't track hub scripts
2025-02-27 10:01:12 +01:00
Ruben Fiszel
654c7be0ee fix missing lock error logs 2025-02-23 09:18:50 +01:00
Ruben Fiszel
3d6015e44a feat: make flow lock deployment error visible in UI 2025-02-22 15:02:16 +01:00
Lucas Abel
6ded2c598e backend: use v2 tables through views where possible (v2 phase 3) (#5119) 2025-02-06 12:43:24 +01:00
Ruben Fiszel
dd3fb0f955 improve error messages for internal err 2025-02-05 16:32:01 +01:00
HugoCasa
175379fbe9 feat: on behalf of (#5058)
* feat: on behalf of

* fix

* revert mistakke

* temp ee ref

* renaming + CLI

* sqlx

* nits + ee ref
2025-01-14 18:32:25 +01:00
HugoCasa
fcb6ced8df feat: captures (#4807)
* feat: captures

* flow UI and improvements

* fix: build

* fix sqlx

* Move Capture WIP

* Add capture to webhook and websocket

* Move connection status viewer in the head

* change trigger section label

* Add popover capture picker using melt ui

* Add shortcut to triggers capture from input form

* remove capture tab in input

* Allways show capture

* remove useless logs

* Add email capture

* Add kafka capture into triggers

* Add edit option in capture table

* use light header for arg input

* Add prefilled group id

* Change button label

* fix logic

* Open resource drawer if prototype has fields

* fix default completion

* Change name Prototype

* Dissociate Editor Mode for triggers

* Fix bug for script

* Fix apply args

* fix apply schema

* Add capture table to script

* fix apply args

* Add capture button for script

* Delete capture tab

* Set capture on when opening triggers capture

* fix connection indicator

* fix minor issues

* Add preprocessor logic

* Use slot in log Panel for captures

* handle capture refresh in script

* Delete capture tab from script editor

* reset kafka resource on toggle static

* fix minor issue

* Allow resource in kafka capture

* use simple capture button in flow

* Remove capture panel

* Polish route trigger editor

* Fix resource saving

* Remove excessive padding

* merge nits

* add workflow_dispatch to build

* add workflow_dispatch to build

* better capture UI

* fix sqlx

* fix build

* fix build

* build

* make initial_messages optional in line with db

* fix npm check

* better handle args for capture webhook and http

* improve migration + http capture fixes

* fix sqlx

* update ee ref

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-06 12:07:09 +01:00
Lucas Abel
4e52e54314 feat(backend): store flow inner modules into flow_node table (#4778) 2024-11-27 16:08:31 +01:00
Ruben Fiszel
fd7bf27f27 remove rsmq 2024-11-24 11:04:54 +01:00
Guilhem
cc1d3f2193 feat(frontend): improve and simplify scheduled poll flows (#4560) 2024-11-04 23:18:09 +01:00
Ruben Fiszel
2a136effe7 nit test serializer 2024-10-29 09:09:08 +01:00
pyranota
11623682c2 Deploy collision warning fix (#4544)
* feat: Multiplayer deploy collision warning

Show warning with diff viewer if 2+ users try to deploy flow/script/app at the same time

In other words, if user A and user B started editing same flow/script/app and user A deployed it,
user B will get warning if they try to deploy as well.

* Add warning for scripts

* Add `deployedBy` to Apps

* Format

* Fix advanced deployment on scripts

* Write comments and cleanup

* feat(frontend): unify all triggers UX and simplify flow settings (#4259)

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): clean up

* feat(frontend): improve UX

* feat(frontend): triggers

* feat(frontend): triggers

* feat(frontend): done

* feat(frontend): fix trigger when position when a preprocessor is presetn

* Glm/rework flow settings v2 (#4497)

* fat(frontend): simplify flow settings menu

* improve scroll

* changing mute toggle

* Add advanced settings badge

* Add nord theme colors

* Add bage for advanced options

* fix minor issue

* fix minor issue

* Add triggers menu to flow settings

* Add quick trigger access

* remove triggers in flow settings

* fix minor issue

* Move triggers settings to flow right panel

* polishing

* fix unset store

* remove save up to for triggers

* fix padding

* reset default tag color

* remove custom select component

* revert path change

* revert section modif

* Revert unused feature

---------

Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>

* Connect top bar cron to schedules settings

* Turn copilot into node

* fix copilot placement

* remove useless import

* fix center copilot

* fix binding

* remove copilot on top of preprocessor

* render copilot node on condition

* quickfix

* remove copilot node

* fix minor issues

* fix route count update

* fix schedule sync

* harmonize colors

* fix alignment and add edges

* recenter node summary

* fix schedules sync

* Add id title

* all

* all

* all

* iteration

* all

* all

* done

* fix

* more fixes

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>

* Update ScriptBuilder.svelte

* Remove `onMount` for flows

* Use version instead of last_updated_at in flows

* Use only versions for apps

* Fetch latest data in Diffs in Apps

* Optimize with (script/flow/app)GetLatestVersion

Create several new endpoints, that returns just latest version without rest of the history

* Sync Diffs with deployed

* Improve Diff's data

* Use `getFlowLatestVersion`

* fix failing endpoints for deployed collision warning

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-10-17 18:05:44 +02:00
pyranota
84441b15dc Multiplayer deploy collision warning (#4533)
* feat: Multiplayer deploy collision warning

Show warning with diff viewer if 2+ users try to deploy flow/script/app at the same time

In other words, if user A and user B started editing same flow/script/app and user A deployed it,
user B will get warning if they try to deploy as well.

* Add warning for scripts

* Add `deployedBy` to Apps

* Format

* Fix advanced deployment on scripts

* Write comments and cleanup

* feat(frontend): unify all triggers UX and simplify flow settings (#4259)

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): clean up

* feat(frontend): improve UX

* feat(frontend): triggers

* feat(frontend): triggers

* feat(frontend): done

* feat(frontend): fix trigger when position when a preprocessor is presetn

* Glm/rework flow settings v2 (#4497)

* fat(frontend): simplify flow settings menu

* improve scroll

* changing mute toggle

* Add advanced settings badge

* Add nord theme colors

* Add bage for advanced options

* fix minor issue

* fix minor issue

* Add triggers menu to flow settings

* Add quick trigger access

* remove triggers in flow settings

* fix minor issue

* Move triggers settings to flow right panel

* polishing

* fix unset store

* remove save up to for triggers

* fix padding

* reset default tag color

* remove custom select component

* revert path change

* revert section modif

* Revert unused feature

---------

Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>

* Connect top bar cron to schedules settings

* Turn copilot into node

* fix copilot placement

* remove useless import

* fix center copilot

* fix binding

* remove copilot on top of preprocessor

* render copilot node on condition

* quickfix

* remove copilot node

* fix minor issues

* fix route count update

* fix schedule sync

* harmonize colors

* fix alignment and add edges

* recenter node summary

* fix schedules sync

* Add id title

* all

* all

* all

* iteration

* all

* all

* done

* fix

* more fixes

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>

* Update ScriptBuilder.svelte

* Remove `onMount` for flows

* Use version instead of last_updated_at in flows

* Use only versions for apps

* Fetch latest data in Diffs in Apps

* Optimize with (script/flow/app)GetLatestVersion

Create several new endpoints, that returns just latest version without rest of the history

* Sync Diffs with deployed

* Improve Diff's data

* Use `getFlowLatestVersion`

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-10-17 17:14:48 +02:00
Faton Ramadani
258445e523 feat(frontend): unify all triggers UX and simplify flow settings (#4259)
* feat(frontend): added list of triggers in the flow graph

* feat(frontend): added list of triggers in the flow graph

* feat(frontend): clean up

* feat(frontend): improve UX

* feat(frontend): triggers

* feat(frontend): triggers

* feat(frontend): done

* feat(frontend): fix trigger when position when a preprocessor is presetn

* Glm/rework flow settings v2 (#4497)

* fat(frontend): simplify flow settings menu

* improve scroll

* changing mute toggle

* Add advanced settings badge

* Add nord theme colors

* Add bage for advanced options

* fix minor issue

* fix minor issue

* Add triggers menu to flow settings

* Add quick trigger access

* remove triggers in flow settings

* fix minor issue

* Move triggers settings to flow right panel

* polishing

* fix unset store

* remove save up to for triggers

* fix padding

* reset default tag color

* remove custom select component

* revert path change

* revert section modif

* Revert unused feature

---------

Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>

* Connect top bar cron to schedules settings

* Turn copilot into node

* fix copilot placement

* remove useless import

* fix center copilot

* fix binding

* remove copilot on top of preprocessor

* render copilot node on condition

* quickfix

* remove copilot node

* fix minor issues

* fix route count update

* fix schedule sync

* harmonize colors

* fix alignment and add edges

* recenter node summary

* fix schedules sync

* Add id title

* all

* all

* all

* iteration

* all

* all

* done

* fix

* more fixes

---------

Co-authored-by: Guilhem <guilhemlemouel@gmail.com>
Co-authored-by: Guilhem <guilhem@mbp-de-windmill.home>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-10-16 17:58:15 +02:00
HugoCasa
62d88436b2 feat: flow step skipping (#4461) 2024-09-30 16:30:59 +02:00
Ruben Fiszel
da969da96f improve benchmarking tools (#4450) 2024-09-27 10:48:04 +02:00
HugoCasa
ae8f29a4cb feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
HugoCasa
304de14674 fix: items with starred info (#4298) 2024-08-28 11:45:50 +02:00
HugoCasa
9efad477da optional starred info on get item + update name of user instance (#4295) 2024-08-27 21:58:51 +02:00
HugoCasa
6a99d999a2 feat: improve early stop (#4257) 2024-08-21 00:47:34 +02:00
HugoCasa
904b5b4ea1 feat: cache common hub scripts in image (#4249)
* feat: cache common hub scripts in image

* Delete backend/src/hubPaths.json

* fix: missing file

* fix: dockerfile

* fix: cache by path

* fix

* fix

* precreate cache folder
2024-08-16 17:30:44 +02:00
HugoCasa
4ef357db17 fix: update flow path (#4053) 2024-07-09 17:13:21 +02:00
HugoCasa
39bcc151dd fix(backend): correct version join of list search flows (#4022) 2024-07-04 15:13:16 +02:00
Ruben Fiszel
2812792326 fix: memory optimization for flow with big args (#4019) 2024-07-04 10:08:42 +02:00
HugoCasa
7f2289d4d5 feat: flow versioning (#4009)
* feat: flow versioning

* fix: sqlx

* fix: update schedule test for flow versioning

* fix: with_deployment_msg + UI nits

* fix: nit

* fix: improve down migration

* patch: keep latest flow version in flow table for backward compat

* fix: app deployments in list view

* chore: update ee ref

* fix: merge

* fix: tests
2024-07-03 15:53:45 +02:00
HugoCasa
4885ac9f4a fix: hide draft only items in pickers and from ops (#3977)
* fix: hide draft only scripts to ops and in pickers/flow builder

* fix: default to hiding draft only scripts

* fix: hide script drafts from ops

* fix: generalize to hide draft only/no main func by default

* fix: archived openapi nit
2024-06-26 12:21:45 +02:00
HugoCasa
6933f06f1c feat: add extra metadata to folders/resources/types + path prefix filtering for all + improve groups UI (#3936)
* feat: add extra metadata to folders/schedules/resources/types

* feat: add path prefix filter to api

* chore: updatee ref

* feat: improve groups UI

* fix: nit

* fix: nit
2024-06-22 01:29:23 +02:00
HugoCasa
93f58056a0 feat: replace ephemeral tokens by jwt (#3908)
* feat: replace ephemeral tokens by jwt

* fix: migration + sqlx

* fix: tests, agent, ee ref

* fix: sqlx

* fix: tests

* fix: jwt prefix change + fix tests

* fix: nit

* fix: handle agent mode
2024-06-14 13:07:35 +02:00
Ruben Fiszel
5cb2c9d1f7 feat(cli): add wmill flow generate-locks 2024-06-10 11:44:53 +02:00
Ruben Fiszel
6a332bc30f fix: make all error strings more verbose 2024-05-25 22:15:15 +02:00
Ruben Fiszel
e207c1d77d fix: improve concurrency re-scheduling at scale 2024-05-25 15:56:56 +02:00
HugoCasa
126459f972 feat: improve permissioned audit logs (#3799)
* fix: increase token label size in db

* Revert "Revert "feat:  improve premissioned audit logs (#3793)" (#3798)"

This reverts commit afae33f5f8.

* feat: add username to email function

* fix: nit

* fix: add docstring to global func
2024-05-23 17:23:50 +02:00
Ruben Fiszel
afae33f5f8 Revert "feat: improve premissioned audit logs (#3793)" (#3798)
This reverts commit 35f8eecf2d.
2024-05-23 11:12:51 +02:00
HugoCasa
35f8eecf2d feat: improve premissioned audit logs (#3793)
* feat:  improve premissioned audit logs

* chore: update ee ref

* fix: build

* fix: sqlx
2024-05-23 08:55:48 +02:00
Ruben Fiszel
35e735d787 further memory reduction 2024-05-20 11:38:11 +02:00
Ruben Fiszel
c43d04b95d feat: reduce memory usage (#3768)
* all

* all

* optimize deserialize

* all

* all

* fix issue

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile
2024-05-19 21:58:38 +02:00
wendrul
60b036c1ae Add concurrency limits observability (#3586)
* Add concurrency_key table and field to payload

Add a new table custom_concurrency_key for storing concurrency keys on a
per job basis. Add the field to the job payload to populate the table on
creation of a new job.

* Rename to custom_concurrency_key

Be more explicit about the nature of the option variable by calling it
a more appropriate name.

* Delete concurrency_keys on delete_expire_items

* Propagate db errors up

* Add ref to EE changes

* update sqlx

* Add concurrency_key to tests

* Revert renaming of FlowValue field

FlowValue field concurrency_key must be named concurrency_key for
serialization purposes.

* Remove debug print statement

* Get concurrency_key from Job

Whether it is a Completed job or a queued one, add a ways to get the
associated concurrency key

* Add endpoint to get concurrency_key

* Add endpoint and button to get job concurrency_key

Add a line with a link on the FlowMetadata that goes to the
concurrency_groups page. add endpoint to get the concurrency_key of a
job

* Merge concurrency_key tables into one

migration + change all related querrys

* Add concurrency plot to the runs page

* Prepare sqlx

* Add concurrency Key filter on UI

* Prepare sqlx

* Prepare sqlx

* Fix filter and order query

The limit makes more sense if we cut the older rows, so order by
started_at

* Factor interpolation to reuse on concurrency key

Factor the arg interpolation logic into a function and finish the
processing of concurrency key before insertion

* Remove old concurrency key processing logic

* Second transaction with userdb

To send all concurrency intervals but revealing only uuids of accessible
jobs, make a second transaction with the userdb.

* Remove old second endpoint

the intervals endpoint now also gets the concurrencyt key information
for all jobs

* Show external jobs

* Put filters into a dropdown

Create a ToggleButtonMore and put elements into a dropdown

* Add toggle between two graphs

* Add filter functionality to concurrency graph

* Improve concurrency graph front

* Fix concurrency groups page

* Prepare sqlx

* Add ref to ee

* Change migration to create new table instead

Instead of renaming the custom_concurrency_key_ended atble, we create a
new table and we will delete custom_concurrency_key_ended in the future
when it is no longer linked to any jobs

* Do small UI improvements

* Fix range fusion by not filtering past jobs

Instead of filtering jobs in the backend on a startedAfter value, limit
the query to 1000 and query all possible towards the past to get a good
context for the graph in most sane situations

* Improve frontend UI
2024-05-15 20:15:14 +02:00
HugoCasa
cb132671d3 feat: critical error side channel (#3625)
* feat: critical error side channel

* feat: trigger workspace error handler or critical channel on schedule error handling failure

* fix: build

* feat: finish critical error side channel

* feat: channels instead of just emails + move to EE

* fix: remove superfluous func

* fix: nits + ee ref

* fix: open source build

* fix: nits

* chore: update ee ref
2024-05-02 20:21:23 +02:00
Ruben Fiszel
59ecd1de29 box flowmodule to reduce stack allocation of flowvalue 2024-05-02 10:06:16 +02:00
Ruben Fiszel
a1d2c59fce feat: flow concurrency limits support custom concurrency key 2024-04-09 22:03:39 +02:00
Ruben Fiszel
9369716918 fix: improve CLI with visible_to_runner_only and priority 2024-04-08 12:22:48 +02:00
HugoCasa
9b6ed95ef2 feat: private hub (#3491)
* feat: private hub v0

* feat: add UI setting

* Update ee.rs

* Update ee.rs

* fix: remove ee symlink

* fix: add back CE ee.rs

* feat: disable s3 embedding loading when private hub

* feat: reload embeddings on hub url change

* fix: only reload embeddings db on server mode

* patch: set default hub url const

* fix: nit
2024-04-04 04:16:53 +02:00
Ruben Fiszel
e66d4b6708 continue on error option for steps 2024-04-02 10:32:47 +02:00
HugoCasa
0e0d964fc8 feat: add visible to runner only default value (#3472) 2024-03-26 19:07:21 +01:00
Ruben Fiszel
7714221f3e feat: large log disk and distributed storage compaction 2024-03-23 10:55:29 +01:00
Ruben Fiszel
16b3b659e9 chore: update deps 2024-03-21 13:37:17 +01:00