Commit Graph

56 Commits

Author SHA1 Message Date
wendrul
1f03eb96de feat: Better tracing for audit logs, including a graph to visualize them (#6078)
* Migrate audit log page to svelte 5

* Add email and span cols to audit table

* Add token_prefixs to audit logs (into AuditAuthorable trait)

* Add audit logs graph (wip)

* Add audit span on push and jwt

* Unify same job audit into the same audit span

* Improve the graph visually

* Fix typo

* functioning graph with svelte issue

* Fix leak

* feat: migrate AuditLogsTable from DataTable to VirtualList for performance

- Replace DataTable component with VirtualList for handling thousands of rows
- Migrate to Svelte 5 runes ($props, $bindable, $derived, $state)
- Implement flattenLogs() for virtual scrolling with grouped date headers
- Add sticky indices and dynamic height calculation
- Update parent component to use callback prop pattern instead of events
- Preserve all existing functionality: filtering, selection, pagination
- Follows RunsTable.svelte implementation pattern

Resolves performance issues when displaying large audit log datasets.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>

* Fix remaining virtual list issues

* WIP graph

* Fix chart styling

* Fix npm check

* Fix missing audit_span arguments

* Update sqlx

* use varchar 255 for email as in other tables

* Remove syntax inconsistency

* Match struct with ee crate

* Update ee-repo-ref.txt

* Update worker_flow.rs

* Remove redefinition of trait to prevent shadowing

* Re add trait on oss but only when no `private` flag

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-03 21:02:45 +00:00
Diego Imbert
7467833239 EE Refactor (#5844)
* app compiles with every ee substituted

* Replace all oss files content

* Revert "Replace all oss files content"

This reverts commit ea4017d59f.

* delete all ee

* hide all _ee files under private flag

* hide every oss stuff when private flag set

* pub use *

* gitignore and substitute script

* pub mod for ee needed for ee repo

* small mistakes

* remove oidc_oss impl

* ee ref (temp)

* ee ref

* fix --all-features selecting private in OSS CI

* ee repo ref

* allow unused
2025-06-02 22:12:33 +02:00
HugoCasa
d2787cfd78 feat: preprocessor refactor (#5629)
* feat: preprocessor refactor

* nit

* oneOf kind support + new capture format + fix build

* fix build

* fix build

* update preprocessor templates

* cache runnable version info

* update ee ref

* update hub scripts && ee ref
2025-05-09 09:50:01 +02:00
Ruben Fiszel
5765d4cf75 nit zombie job monitoring 2025-03-11 07:08:28 +01:00
Ruben Fiszel
4ae8cad4d4 fix: schedules do not accept 5 units cron syntax on update/create anymore 2025-02-20 17:59:32 +01:00
Lucas Abel
ecbec1fe50 fix(backend): improve schedule queries plan to leverage indices better for performance (#5273)
* backend: fix schedule queries plan

* rework

* fix skipped as non success

* update sqlx

* update

* all

* fix npm run check

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-02-12 16:34:03 +01:00
Lucas Abel
6ded2c598e backend: use v2 tables through views where possible (v2 phase 3) (#5119) 2025-02-06 12:43:24 +01:00
Ruben Fiszel
dd3fb0f955 improve error messages for internal err 2025-02-05 16:32:01 +01:00
HugoCasa
175379fbe9 feat: on behalf of (#5058)
* feat: on behalf of

* fix

* revert mistakke

* temp ee ref

* renaming + CLI

* sqlx

* nits + ee ref
2025-01-14 18:32:25 +01:00
Lucas Abel
3f0a818fa3 backend: do not force get_schedule_opt to be within a transaction (#5019) 2025-01-06 09:36:57 +01:00
Ruben Fiszel
fd7bf27f27 remove rsmq 2024-11-24 11:04:54 +01:00
Alexander Petric
54b419d8ab feat(schedule): support for extended cron syntax (#4754)
* init

* croner with fall back on scheduling failure

* adding packages

* fallback mechanism v1

* removing debug console log

* improving cargo organization

* queue cargo.toml

* use workspace cargo-tz

* test next occurrence when creating new croner schedule

* toggle label

* treat legacy edit requests where no cron_version is provided as v1

* fix edit schedule
2024-11-21 19:24:38 +01:00
HugoCasa
ae8f29a4cb feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
HugoCasa
e46c2ea513 fix: improve license key ui (#4220)
* fix: improve license key ui

* fix: stop schedules on key expiration

* update ee ref
2024-08-11 01:01:24 +02:00
Ruben Fiszel
e6e08108a6 fix: improve handling of schedules with retries and concurrency limits 2024-07-05 20:37:30 +02:00
Ruben Fiszel
2812792326 fix: memory optimization for flow with big args (#4019) 2024-07-04 10:08:42 +02:00
HugoCasa
ec673ac6b4 feat: schedule pausing (#3976)
* feat: schedule pausing

* fix: improve UI
2024-06-25 23:39:34 +02:00
HugoCasa
93f58056a0 feat: replace ephemeral tokens by jwt (#3908)
* feat: replace ephemeral tokens by jwt

* fix: migration + sqlx

* fix: tests, agent, ee ref

* fix: sqlx

* fix: tests

* fix: jwt prefix change + fix tests

* fix: nit

* fix: handle agent mode
2024-06-14 13:07:35 +02:00
Ruben Fiszel
c43d04b95d feat: reduce memory usage (#3768)
* all

* all

* optimize deserialize

* all

* all

* fix issue

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile

* fix compile
2024-05-19 21:58:38 +02:00
wendrul
60b036c1ae Add concurrency limits observability (#3586)
* Add concurrency_key table and field to payload

Add a new table custom_concurrency_key for storing concurrency keys on a
per job basis. Add the field to the job payload to populate the table on
creation of a new job.

* Rename to custom_concurrency_key

Be more explicit about the nature of the option variable by calling it
a more appropriate name.

* Delete concurrency_keys on delete_expire_items

* Propagate db errors up

* Add ref to EE changes

* update sqlx

* Add concurrency_key to tests

* Revert renaming of FlowValue field

FlowValue field concurrency_key must be named concurrency_key for
serialization purposes.

* Remove debug print statement

* Get concurrency_key from Job

Whether it is a Completed job or a queued one, add a ways to get the
associated concurrency key

* Add endpoint to get concurrency_key

* Add endpoint and button to get job concurrency_key

Add a line with a link on the FlowMetadata that goes to the
concurrency_groups page. add endpoint to get the concurrency_key of a
job

* Merge concurrency_key tables into one

migration + change all related querrys

* Add concurrency plot to the runs page

* Prepare sqlx

* Add concurrency Key filter on UI

* Prepare sqlx

* Prepare sqlx

* Fix filter and order query

The limit makes more sense if we cut the older rows, so order by
started_at

* Factor interpolation to reuse on concurrency key

Factor the arg interpolation logic into a function and finish the
processing of concurrency key before insertion

* Remove old concurrency key processing logic

* Second transaction with userdb

To send all concurrency intervals but revealing only uuids of accessible
jobs, make a second transaction with the userdb.

* Remove old second endpoint

the intervals endpoint now also gets the concurrencyt key information
for all jobs

* Show external jobs

* Put filters into a dropdown

Create a ToggleButtonMore and put elements into a dropdown

* Add toggle between two graphs

* Add filter functionality to concurrency graph

* Improve concurrency graph front

* Fix concurrency groups page

* Prepare sqlx

* Add ref to ee

* Change migration to create new table instead

Instead of renaming the custom_concurrency_key_ended atble, we create a
new table and we will delete custom_concurrency_key_ended in the future
when it is no longer linked to any jobs

* Do small UI improvements

* Fix range fusion by not filtering past jobs

Instead of filtering jobs in the backend on a startedAfter value, limit
the query to 1000 and query all possible towards the past to get a good
context for the graph in most sane situations

* Improve frontend UI
2024-05-15 20:15:14 +02:00
HugoCasa
cb132671d3 feat: critical error side channel (#3625)
* feat: critical error side channel

* feat: trigger workspace error handler or critical channel on schedule error handling failure

* fix: build

* feat: finish critical error side channel

* feat: channels instead of just emails + move to EE

* fix: remove superfluous func

* fix: nits + ee ref

* fix: open source build

* fix: nits

* chore: update ee ref
2024-05-02 20:21:23 +02:00
Ruben Fiszel
a6cae2915a handle schedule with empty tags 2024-02-21 23:08:53 +01:00
Ruben Fiszel
fcf99217c5 nit logs around schedules 2024-02-11 23:40:56 +01:00
Guillaume Bouvignies
2edba18038 feat: Tag override for scheduled scripts (#2998)
* feat: Tag override for scheduled scripts

* Refactor WorkerTagPicker to component

* Fix test
2024-01-13 00:06:54 +01:00
Guillaume Bouvignies
2870d4882d feat: Retry for scheduled scripts (#2893)
* feat: Retry for scheduled scripts

* rename scheduldedscriptwithretry to singlescriptflow

* Schedule options under tabs
2023-12-22 14:59:46 +01:00
Guillaume Bouvignies
7c71b65a99 feat: Add default timeout for instance and custom timeout for scripts (#2811)
* feat: Add default timeout for instance and custom timeout for scripts

* final

* sqlx prepare

* Logs a warning in the job logs

* Fix merge

* JOB_DEFAULT_TIMEOUT is an optional

* Fix JOB_DEFAULT_TIMEOUT monitor loading

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2023-12-13 07:08:07 +01:00
Ruben Fiszel
518a8396b3 feat: dedicated workers for flows (#2637)
* dw for flows

* dw for flows

* all

* sqlx

* sqlx

* dedicated workers

* more comments
2023-11-17 10:00:04 +01:00
Ruben Fiszel
f15d12c3ab fix compile 2023-11-07 09:42:28 +01:00
Guillaume Bouvignies
bd69bafe59 feat: Priority queue (#2476)
* BE v1 for priority queue

* Add FE prototype

* sqlx prepare

* Disply priority badge in job detail view

* sqlx prepare post merge

* Add priority number input field next to toggle

* increase dnd timeout for app events

* Update db migration

* Fix FE import

* Add badge to script/flow details page and fix EE-only mention in tooltip

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2023-10-23 14:22:59 +02:00
Ruben Fiszel
5ac99b817e feat: dedicated workers for native-throughput performance (EE only) (#2239)
* dedicated worker

* dedicated worker

* merge

* progress

* progress

* progress

* merge

* update

* sqlx

* fix imports
2023-09-06 10:32:53 +02:00
Ruben Fiszel
8d62fbdbdc feat: add caching to flows and scripts (#2193)
* cachettl

* all
2023-08-29 13:00:20 +02:00
Ruben Fiszel
6787c8f824 fix schedule edit issue 2023-08-22 19:30:55 +02:00
Ruben Fiszel
c77d38563d remove unecessary postgres transactions and improve performance 2023-08-22 00:34:44 +02:00
Ruben Fiszel
9970f6daeb feat: migrate state path to new schema 2023-08-11 18:54:47 +02:00
Ruben Fiszel
b0660619c8 feat: add step's custom timeout 2023-07-30 20:42:24 +02:00
Jakub Kołodziejczak
223b80768a chore(backend): upgrade sqlx to ^0.7 (#1865)
* DRAFT chore(backend): upgrade sqlx to ^0.7

related to:
* https://github.com/windmill-labs/windmill/pull/1858
* https://github.com/launchbadge/sqlx/issues/1163#issuecomment-1627685514

* (vol. 2) in 0.7, `Transaction` can no longer implement `Executor` directly

ref: afb6b1066e/examples/postgres/transaction/src/main.rs (L14-L17)

notice that I'm temporarly using my custom patch
16e4c9a8f3
it's related to https://github.com/launchbadge/sqlx/issues/2611

* post git rebase chores

* use upstream fix from 0.7.1

* fix compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2023-07-15 14:06:20 +02:00
Guillaume Bouvignies
0327b85f89 feat: Per script concurrency limit with time window (#1816)
* Add concurrency a per-script concurrency limit

* Add a per-script per time window concurrency limit

* Improve feature with a better estimation of the next schedule timestamp

* Add concurrency limits for inline scripts

* Fix tests

* run sqlx prepare

* Add requeue event to job logs
2023-07-11 23:35:26 +02:00
Ruben Fiszel
c93393d5c0 feat: schedule error handler (#1636)
* handleError

* revert

* all
2023-05-23 20:25:41 +02:00
Ruben Fiszel
9ea18b4593 feat(backend): add job_id as a query arg to force set the new job_id 2023-05-15 10:50:43 +02:00
Ruben Fiszel
2d62ced3a5 fix script in schedule should use latest hash 2023-04-27 18:03:12 +02:00
Ruben Fiszel
02ba82064f feat(backend): worker groups (#1452)
* tagged scripts

* tagged scripts

* fix

* fix

* fix

* all

* update sqlx
2023-04-23 16:57:23 +02:00
Ruben Fiszel
1ba0e24155 feat(backend): only run fully deployed scripts 2023-04-21 12:49:47 +02:00
Ruben Fiszel
be41e45130 split worker.rs into more files 2023-04-13 17:44:47 +02:00
Kai Jellinghaus
63102b036c feat(backend): Redis based queue (#1324)
* Merge?

* Fix V8 breaking change

* WIP

* WIP

* Cleanup

* Move to git reference

* Fix Merge conflict

* update

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2023-04-11 12:24:13 +02:00
Oliver Veal
0b444ce395 feat: improved cron/schedule editor (#1362)
* basic cron schedule editing ui

* schedules run in a user-specified timezone

* fix other uses of CronInput component

* use now() from database to schedule next job

* offset -> IANA timezone conversion on db migration

* sqlx ci

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2023-04-05 16:37:24 +02:00
Ruben Fiszel
0263c20a5b introduce root_job and leaf_jobs for efficient result_by_id 2023-03-03 12:44:44 +01:00
Ruben Fiszel
49aeed88fa fix permissioned_as from username 2023-01-17 12:49:24 +01:00
Ruben Fiszel
bb4ca8c99f playwright against uffizzi (#1109)
* all

* fix perms

* move build to depot

* login

* revert to docker build

* revert to docker build

* fix playwright
2023-01-16 18:01:45 +01:00
Ruben Fiszel
e19fcce575 fix schedule permissions 2023-01-16 16:42:42 +01:00
Ruben Fiszel
6e2890f0f1 require ownership to move items && automatically edit schedule paths 2022-12-18 18:45:32 +01:00