Compare commits

..

15 Commits

Author SHA1 Message Date
Ruben Fiszel
461306ab04 nits 2025-05-06 17:48:41 +02:00
Ruben Fiszel
062e904141 nits 2025-05-06 17:43:21 +02:00
Ruben Fiszel
b0ba8350f7 update 2025-05-06 17:22:08 +02:00
Ruben Fiszel
2adafcd265 orderByRefactor 2025-05-06 17:17:33 +02:00
Ruben Fiszel
9bdd301f52 fix: ansible in agent mode can use inventory.ini 2025-05-06 11:57:08 +02:00
Ruben Fiszel
afc9cf5940 update deps versions (#5703) 2025-05-05 16:19:39 +02:00
pyranota
4fd0561239 feat: critical alert if disk near full (#5549)
* feat: critical alert if disk near full

* update logic to cover edge-case

* update logic

* windows support

* Update README.md

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* make it periodically

* add extra safety

* respect killpills

* do not check every 3 secs

* move to monitor.rs

* rework

* alter readme

* extending functionality

* fix worker_mode

* fix compilation

* fix typo

* make use of AI suggestion

* update ee-repo-ref

* update ee ref

* logs are CE

* update ee repo ref

* remove systemstat from worker crate

* fix comp error + sync cargo.lock

* more comptime fixing

* Update ee-repo-ref.txt

* fix compilation error

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-05-05 08:54:48 +02:00
Ruben Fiszel
815c3b6d00 chore(main): release 1.486.1 (#5701)
* chore(main): release 1.486.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-05-04 18:24:52 +02:00
Ruben Fiszel
36da8aec08 fix: improve MultiSelectWrapper behavior 2025-05-04 18:18:58 +02:00
Ruben Fiszel
d2ad2cdf33 nit feature flag better backend tests 2025-05-04 17:58:06 +02:00
Eugene Chernyavsky
6b2ba3f261 google-genai and google.generativeai requirement from import (#5699) 2025-05-02 19:50:20 +02:00
Ruben Fiszel
d103f5a466 chore(main): release 1.486.0 (#5690)
* chore(main): release 1.486.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2025-05-02 00:58:16 +02:00
Ruben Fiszel
7eb9d7d46c fix: do not track relative deps for scripts with raw defined deps from CLI (#5696) 2025-05-02 00:37:28 +02:00
Alexander Petric
8ad730b290 extra logging for agent_token flake (#5693)
* extra logging for agent_token flake

* extra logging for agent_token flake
2025-05-01 19:46:52 +02:00
centdix
ec701a9ee7 feat: Integrate MCP with hub (#5685)
* draft

* working draft

* tool to list scripts

* working sse server

* add proxy + parse query params

* working mcp that runs script

* remove useless dependencies

* log context

* update crate

* mcp no proxy

* use custom sdk

* integrate in windmill api

* draft

* put rmcp sdk here

* do not use mcp as crate

* use extensions for get scripts

* remove unused crate

* list actual scripts

* fix

* give schema in list tools

* cleaning + take workspace id from context

* implement calling the script with tool

* cleaning + fix ctrl-c

* make post path a param

* cleaning

* better name for tools

* fix error with tool name

* cleaning

* draft cleaning

* more cleaning

* list script based on settings

* fix query

* add params in openapi

* use rmcp fork from git

* remove files

* cleaning + fix query

* remove settings and use favorite by default

* add flows in tools list

* handle running flows

* remove frontend for mcp settings

* handle resource args

* send list of resource in shema

* handle mcp url setting with token scope

* cleaning

* avoid calling list tools in call tool

* apply scope to flows

* cleaning

* cleaning

* cleaning

* cleaning

* format files

* fix typos

* remove log

* add back missing dispatch

* fix transform for resource-obj + put every resource in description

* transform obj to string

* cleaner code

* better frontend

* cleaner code

* cleaner logic

* add parentheses just in case

* add func to fetch hub scripts

* fix typos

* working fetch and run hub script

* also fetch flows from hub

* improvments

* merge create tool logic

* add integrations in description

* cleaning

* cleaning

* small fix

* get schema for flow

* filter tools fetch by token scope

* remove hub flows

* remove prints

* add hub script integration choice

* higher limits

* cleaning

* fix merge

* better naming for hub scripts

* no workspace for hub

* alow multiple app in one token

* plural

* fix

* cleaning

* add documentation

* fix bad code

* use id directly

* cleaner bindings

* fix disabled condition

* add cancel button + reset apps if not hub token

* reset mcp apps

* combine all/favorites + hub

* small fix
2025-05-01 19:31:56 +02:00
59 changed files with 2099 additions and 848 deletions

View File

@@ -1,5 +1,28 @@
# Changelog
## [1.486.1](https://github.com/windmill-labs/windmill/compare/v1.486.0...v1.486.1) (2025-05-04)
### Bug Fixes
* improve MultiSelectWrapper behavior ([36da8ae](https://github.com/windmill-labs/windmill/commit/36da8aec080742e13f23e1dee12b3954947f53dd))
## [1.486.0](https://github.com/windmill-labs/windmill/compare/v1.485.3...v1.486.0) (2025-05-01)
### Features
* add run now directly on schedule drawer and duplicate schedule option ([#5674](https://github.com/windmill-labs/windmill/issues/5674)) ([dfb947f](https://github.com/windmill-labs/windmill/commit/dfb947ff37c688f54a32de5aa3c5c3d142cb80f4))
* Database Manager ([#5586](https://github.com/windmill-labs/windmill/issues/5586)) ([41c15fc](https://github.com/windmill-labs/windmill/commit/41c15fc78aaf844c559d3d6c772e04ecce436e9d))
* Integrate MCP with hub ([#5685](https://github.com/windmill-labs/windmill/issues/5685)) ([ec701a9](https://github.com/windmill-labs/windmill/commit/ec701a9ee74c9d890b54234362392deca63a77c7))
### Bug Fixes
* Ai Chat: do not send tools if empty + respond even if tool fails ([#5692](https://github.com/windmill-labs/windmill/issues/5692)) ([9c55040](https://github.com/windmill-labs/windmill/commit/9c55040e47e76af8b7e2864b82fa30505545dcb5))
* do not track relative deps for scripts with raw defined deps from CLI ([#5696](https://github.com/windmill-labs/windmill/issues/5696)) ([7eb9d7d](https://github.com/windmill-labs/windmill/commit/7eb9d7d46cb48ae69a3fd3ff852a57abae450a3b))
* improve CLI file scanning performances ([0916978](https://github.com/windmill-labs/windmill/commit/09169784bd2d0ab7acf5f40dc86f36f1cae967b7))
## [1.485.3](https://github.com/windmill-labs/windmill/compare/v1.485.2...v1.485.3) (2025-04-29)

View File

@@ -363,6 +363,7 @@ you to have it being synced automatically everyday.
| DENO_AUTH_TOKENS | None | Custom DENO_AUTH_TOKENS to pass to worker to allow the use of private modules | Worker |
| DISABLE_RESPONSE_LOGS | false | Disable response logs | Server |
| CREATE_WORKSPACE_REQUIRE_SUPERADMIN | true | If true, only superadmins can create new workspaces | Server |
| MIN_FREE_DISK_SPACE_MB | 15000 | Minimum amount of free space on worker. Sends critical alert if worker has less free space. | Worker |
## Run a local dev setup

View File

@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE script SET ws_error_handler_muted = $3 WHERE workspace_id = $2 AND path = $1 AND created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Bool"
]
},
"nullable": []
},
"hash": "1182fe055306d7ea435d76b74d781e066915c8397e6bbc9e408ff3dda9fec27f"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 AND\n created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2))",
"query": "SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1)",
"describe": {
"columns": [
{
@@ -19,5 +19,5 @@
null
]
},
"hash": "ea2b88dc050aec038641ea37399d68d4385c5bdc721d1351609f27ca45e4dbdc"
"hash": "2a49e5b5486b650d96f3e9038cba8a5f2e75d3b12ee4718452e82c7318b1bcf4"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by created_at DESC",
"query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by s.created_at DESC",
"describe": {
"columns": [
{
@@ -25,5 +25,5 @@
true
]
},
"hash": "362419eb262c83d6a98a0200b116e831ada60399fe5f55a56d930cc69aff2675"
"hash": "726e956cfcd3ac7c07abeecdf92cf0996efe7fa7b671ac2b3b000ead0ea307de"
}

View File

@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE script \n SET ws_error_handler_muted = $3 \n WHERE ctid = (\n SELECT ctid FROM script\n WHERE path = $1 AND workspace_id = $2\n ORDER BY created_at DESC\n LIMIT 1\n )\n",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Bool"
]
},
"nullable": []
},
"hash": "848c8371eeb17ebd4b36a33f7d8a61eb8f07c54d291bb857ddd41a549cbc88dd"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt) VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)",
"query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
"describe": {
"columns": [],
"parameters": {
@@ -17,5 +17,5 @@
},
"nullable": []
},
"hash": "33c1793e55b1127d88d2509aadd0eb04e042463200f237b4c2cb176612fa16fe"
"hash": "92faee8714a45a403b623e04d789f7f99067a05e9dfe270223164db8a1df2e4b"
}

View File

@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT content FROM script WHERE path = $1 AND workspace_id = $2 AND\n created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND archived = false AND workspace_id = $2)",
"query": "SELECT content FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false ORDER BY created_at DESC LIMIT 1",
"describe": {
"columns": [
{
@@ -19,5 +19,5 @@
false
]
},
"hash": "a17260a1f1ee02e786690994d98c84ddf81e2eeb883f895c9cfc47e144d422cb"
"hash": "b5860f6a7672a368d740dcd367a8d5ab98fa93e0382a57a698564695db6c40ac"
}

View File

@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by s.created_at DESC LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "hash",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "deployment_msg",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
true
]
},
"hash": "cf2a6ad6471a40b6298775cda9300aeecdd75503bed59d80cd62091d1642d1ec"
}

1411
backend/Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.485.3"
version = "1.486.1"
authors.workspace = true
edition.workspace = true
@@ -32,7 +32,7 @@ members = [
]
[workspace.package]
version = "1.485.3"
version = "1.486.1"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -132,6 +132,8 @@ quote.workspace = true
memchr.workspace = true
v8 = { workspace = true, optional = true }
rustls.workspace = true
systemstat.workspace = true
size.workspace = true
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator = { optional = true, workspace = true }
@@ -192,7 +194,7 @@ serde_json = { version = "^1", features = ["preserve_order", "raw_value"] }
uuid = { version = "^1", features = ["serde", "v4"] }
thiserror = "^2"
anyhow = "^1"
chrono = { version = "=0.4.39", features = ["serde"] }
chrono = { version = "^0.4", features = ["serde"] }
chrono-tz = "^0.10.1"
tracing = "^0"
tracing-subscriber = { version = "^0", features = ["env-filter", "json"] }
@@ -315,9 +317,9 @@ nix = { version = "0.27.1", features = ["process", "signal"] }
tinyvector = { git = "https://github.com/windmill-labs/tinyvector", rev = "20823b94c20f2b9093f318badd24026cf54dcc85" }
hf-hub = "0.3.2"
tokenizers = "0.14.1"
candle-core = "0.3.0"
candle-transformers = "0.3.0"
candle-nn = "0.3.0"
candle-core = "0.9.1"
candle-transformers = "0.9.1"
candle-nn = "0.9.1"
tiberius = { version = "0.12.3", default-features = false, features = ["rustls", "tds73", "chrono", "sql-browser-tokio"]}
pin-project = "1"
indexmap = { version = "2.2.5", features = ["serde"]}
@@ -331,8 +333,8 @@ async-nats = "0.38.0"
nkeys = "0.4.4"
nu-parser = { version = "0.101.0", default-features = false }
datafusion = "39.0.0"
object_store = { version = "0.10.0", features = ["aws", "azure"] }
datafusion = "47.0.0"
object_store = { version = "0.12.0", features = ["aws", "azure"] }
openidconnect = { version = "4.0.0-rc.1" }
aws-config = "^1"
aws-sdk-sqs = "1.57.0"
@@ -355,9 +357,6 @@ bollard = "0.18.1"
tonic = { version = "=0.12.3", features = ["tls-native-roots"] }
byteorder = "1.5.0"
# todo remove
half = "=2.4.1"
tikv-jemallocator = { version = "0.5" }
tikv-jemalloc-sys = { version = "^0.5" }
tikv-jemalloc-ctl = { version = "^0.5" }
@@ -368,6 +367,8 @@ pin-project-lite = "^0"
tantivy = "0.22.0"
backon = "1.3.0"
systemstat = "0.2.4"
size = "0.5.0"
flume = { version = "0.11.1", features = ["async"] }

View File

@@ -1 +1 @@
96a2129f585a1bc6567ef08bd27aaa4aca70543f
868ccad87afb804fe22818ecd3d5a091199bcdbf

View File

@@ -0,0 +1 @@
DELETE FROM global_settings WHERE name = 'critical_alerts_on_db_oversize';

View File

@@ -0,0 +1,3 @@
INSERT INTO global_settings (name, value)
VALUES ('critical_alerts_on_db_oversize', '{}')
ON CONFLICT (name) DO NOTHING;

View File

@@ -0,0 +1 @@
-- Add down migration script here

View File

@@ -0,0 +1,3 @@
-- Add up migration script here
DROP INDEX IF EXISTS index_script_on_path_created_at;
CREATE INDEX IF NOT EXISTS index_script_on_path_created_at ON script (workspace_id, path, created_at DESC);

View File

@@ -142,6 +142,8 @@ pub static FULL_IMPORTS_MAP: PyMap = phf_map! {
"google.cloud.dns" => "google-cloud-dns",
"google.cloud.runtimeconfig" => "google-cloud-runtimeconfig",
"google.cloud.iot" => "google-cloud-iot",
"google.generativeai" => "google-generativeai",
"google.genai" => "google-genai",
// Azure
"azure.mgmt.hybridkubernetes" => "azure-mgmt-hybridkubernetes",
"azure.mgmt.sql" => "azure-mgmt-sql",

View File

@@ -8,11 +8,12 @@
use anyhow::Context;
use monitor::{
load_base_url, load_otel, reload_delete_logs_periodically_setting, reload_indexer_config,
load_base_url, load_otel, reload_critical_alerts_on_db_oversize,
reload_delete_logs_periodically_setting, reload_indexer_config,
reload_instance_python_version_setting, reload_maven_repos_setting,
reload_no_default_maven_setting, reload_nuget_config_setting,
reload_timeout_wait_result_setting, send_current_log_file_to_object_store,
send_logs_to_object_store,
send_logs_to_object_store, WORKERS_NAMES,
};
use rand::Rng;
use sqlx::postgres::PgListener;
@@ -33,15 +34,16 @@ use windmill_common::{
agent_workers::build_agent_http_client,
get_database_url,
global_settings::{
BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING,
CRITICAL_ERROR_CHANNELS_SETTING, CUSTOM_TAGS_SETTING, DEFAULT_TAGS_PER_WORKSPACE_SETTING,
DEFAULT_TAGS_WORKSPACES_SETTING, EMAIL_DOMAIN_SETTING, ENV_SETTINGS,
EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING, EXTRA_PIP_INDEX_URL_SETTING,
HUB_BASE_URL_SETTING, INDEXER_SETTING, INSTANCE_PYTHON_VERSION_SETTING,
JOB_DEFAULT_TIMEOUT_SECS_SETTING, JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING,
LICENSE_KEY_SETTING, MAVEN_REPOS_SETTING, MONITOR_LOGS_ON_OBJECT_STORE_SETTING,
NO_DEFAULT_MAVEN_SETTING, NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OAUTH_SETTING,
OTEL_SETTING, PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING,
CRITICAL_ALERT_MUTE_UI_SETTING, CRITICAL_ERROR_CHANNELS_SETTING, CUSTOM_TAGS_SETTING,
DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING, EMAIL_DOMAIN_SETTING,
ENV_SETTINGS, EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING,
EXTRA_PIP_INDEX_URL_SETTING, HUB_BASE_URL_SETTING, INDEXER_SETTING,
INSTANCE_PYTHON_VERSION_SETTING, JOB_DEFAULT_TIMEOUT_SECS_SETTING, JWT_SECRET_SETTING,
KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, MAVEN_REPOS_SETTING,
MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NO_DEFAULT_MAVEN_SETTING,
NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OAUTH_SETTING, OTEL_SETTING,
PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, SMTP_SETTING, TEAMS_SETTING,
TIMEOUT_WAIT_RESULT_SETTING,
@@ -917,6 +919,12 @@ Windmill Community Edition {GIT_VERSION}
tracing::error!(error = %e, "Could not reload critical error emails setting");
}
},
CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING => {
if let Err(e) = reload_critical_alerts_on_db_oversize(&db).await {
tracing::error!(error = %e, "Could not reload critical alerts on db oversize setting");
}
},
JWT_SECRET_SETTING => {
if let Err(e) = reload_jwt_secret_setting(&db).await {
tracing::error!(error = %e, "Could not reload jwt secret setting");
@@ -1214,10 +1222,12 @@ pub async fn run_workers(
"Starting {num_workers} workers and SLEEP_QUEUE={}ms",
*windmill_worker::SLEEP_QUEUE
);
for i in 1..(num_workers + 1) {
let wk_conf = &workers[i as usize - 1];
let conn1 = wk_conf.conn.clone();
let worker_name = wk_conf.worker_name.clone();
WORKERS_NAMES.write().await.push(worker_name.clone());
let ip = ip.clone();
let rx = killpill_rxs.pop().unwrap();
let tx = tx.clone();

View File

@@ -13,7 +13,7 @@ use std::{
use chrono::{NaiveDateTime, Utc};
use futures::{stream::FuturesUnordered, StreamExt};
use serde::de::DeserializeOwned;
use serde::{de::DeserializeOwned, Deserialize};
use sqlx::{Pool, Postgres};
use tokio::{
join,
@@ -30,6 +30,8 @@ use windmill_api::{
#[cfg(feature = "enterprise")]
use windmill_common::ee::{jobs_waiting_alerts, worker_groups_alerts};
#[cfg(feature = "enterprise")]
use windmill_common::ee::low_disk_alerts;
#[cfg(feature = "oauth2")]
use windmill_common::global_settings::OAUTH_SETTING;
@@ -40,13 +42,14 @@ use windmill_common::{
error,
flow_status::{FlowStatus, FlowStatusModule},
global_settings::{
BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING,
CRITICAL_ERROR_CHANNELS_SETTING, DEFAULT_TAGS_PER_WORKSPACE_SETTING,
DEFAULT_TAGS_WORKSPACES_SETTING, EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING,
EXTRA_PIP_INDEX_URL_SETTING, HUB_BASE_URL_SETTING, INSTANCE_PYTHON_VERSION_SETTING,
JOB_DEFAULT_TIMEOUT_SECS_SETTING, JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING,
LICENSE_KEY_SETTING, MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NPM_CONFIG_REGISTRY_SETTING,
NUGET_CONFIG_SETTING, OTEL_SETTING, PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING,
CRITICAL_ALERT_MUTE_UI_SETTING, CRITICAL_ERROR_CHANNELS_SETTING,
DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING,
EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING, EXTRA_PIP_INDEX_URL_SETTING,
HUB_BASE_URL_SETTING, INSTANCE_PYTHON_VERSION_SETTING, JOB_DEFAULT_TIMEOUT_SECS_SETTING,
JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING,
MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING,
OTEL_SETTING, PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
},
@@ -65,10 +68,10 @@ use windmill_common::{
DEFAULT_TAGS_WORKSPACES, INDEXER_CONFIG, SCRIPT_TOKEN_EXPIRY, SMTP_CONFIG, TMP_DIR,
WORKER_CONFIG, WORKER_GROUP,
},
KillpillSender, BASE_URL, CRITICAL_ALERT_MUTE_UI_ENABLED, CRITICAL_ERROR_CHANNELS, DB,
DEFAULT_HUB_BASE_URL, HUB_BASE_URL, JOB_RETENTION_SECS, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED,
SERVICE_LOG_RETENTION_SECS,
KillpillSender, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE, CRITICAL_ALERT_MUTE_UI_ENABLED,
CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL, HUB_BASE_URL, JOB_RETENTION_SECS,
METRICS_DEBUG_ENABLED, METRICS_ENABLED, MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED,
OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED, SERVICE_LOG_RETENTION_SECS,
};
use windmill_queue::{cancel_job, MiniPulledJob, SameWorkerPayload};
use windmill_worker::{
@@ -134,7 +137,7 @@ lazy_static::lazy_static! {
.and_then(|x| x.parse::<bool>().ok())
.unwrap_or(false);
pub static ref WORKERS_NAMES: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
static ref QUEUE_COUNT_TAGS: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
static ref DISABLE_CONCURRENCY_LIMIT: bool = std::env::var("DISABLE_CONCURRENCY_LIMIT").is_ok_and(|s| s == "true");
@@ -183,6 +186,12 @@ pub async fn initial_load(
if server_mode {
if let Some(db) = conn.as_sql() {
load_require_preexisting_user(db).await;
if let Err(e) = reload_critical_alerts_on_db_oversize(db).await {
tracing::error!(
"Error reloading critical alerts on db oversize setting: {:?}",
e
)
}
}
}
@@ -638,7 +647,9 @@ async fn send_log_file_to_object_store(
let (ok_lines, err_lines) = read_log_counters(ts_str);
if let Some(db) = conn.as_sql() {
if let Err(e) = sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt) VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)",
if let Err(e) = sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
hostname, mode.to_string(), worker_group.clone(), ts, highest_file, ok_lines as i64, err_lines as i64, *JSON_FMT)
.execute(db)
.await {
@@ -1404,6 +1415,23 @@ pub async fn monitor_db(
}
};
let low_disk_alerts_f = async {
#[cfg(feature = "enterprise")]
if let Some(db) = conn.as_sql() {
low_disk_alerts(
&db,
server_mode,
_worker_mode,
WORKERS_NAMES.read().await.clone(),
)
.await;
}
#[cfg(not(feature = "enterprise"))]
{
()
}
};
let apply_autoscaling_f = async {
#[cfg(feature = "enterprise")]
if server_mode && !initial_load {
@@ -1426,6 +1454,7 @@ pub async fn monitor_db(
verify_license_key_f,
worker_groups_alerts_f,
jobs_waiting_alerts_f,
low_disk_alerts_f,
apply_autoscaling_f,
update_min_worker_version_f,
);
@@ -2181,6 +2210,37 @@ pub async fn reload_critical_error_channels_setting(conn: &DB) -> error::Result<
Ok(())
}
pub async fn reload_critical_alerts_on_db_oversize(conn: &DB) -> error::Result<()> {
#[derive(Deserialize)]
struct DBOversize {
enabled: bool,
value: f32,
}
let db_oversize_value =
load_value_from_global_settings(conn, CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING).await?;
let db_oversize = if let Some(q) = db_oversize_value {
match serde_json::from_value::<DBOversize>(q.clone()) {
Ok(DBOversize { enabled: true, value }) => Some(value),
Ok(_) => None,
Err(q) => {
tracing::error!(
"Could not parse critical_alerts_on_db_oversize setting, found: {:#?}",
&q
);
None
}
}
} else {
None
};
let mut l = CRITICAL_ALERTS_ON_DB_OVERSIZE.write().await;
*l = db_oversize;
Ok(())
}
async fn generate_and_save_jwt_secret(db: &DB) -> error::Result<String> {
let secret = rd_string(32);
sqlx::query!(

View File

@@ -16,16 +16,20 @@ use tokio::sync::RwLock;
#[cfg(feature = "enterprise")]
use tokio::time::{timeout, Duration};
#[cfg(feature = "python")]
use windmill_api_client::types::{CreateFlowBody, RawScript};
#[cfg(feature = "enterprise")]
use windmill_api_client::types::{EditSchedule, NewSchedule, ScriptArgs};
use windmill_api_client::types::{NewScript, ScriptLang as NewScriptLanguage};
use serde::Serialize;
#[cfg(feature = "deno_core")]
use windmill_common::flows::InputTransform;
use windmill_common::worker::WORKER_CONFIG;
use windmill_common::{
flow_status::{FlowStatus, FlowStatusModule, RestartedFrom},
flows::{FlowModule, FlowModuleValue, FlowValue, InputTransform},
flows::{FlowModule, FlowModuleValue, FlowValue},
jobs::{JobKind, JobPayload, RawCode},
jwt::JWT_SECRET,
scripts::{ScriptHash, ScriptLang},
@@ -141,7 +145,7 @@ impl ApiServer {
format!("http://localhost:{}", addr.port()),
));
_port_rx.await.unwrap();
_port_rx.await.expect("failed to receive port");
// clear the cache between tests
windmill_common::cache::clear();
@@ -167,6 +171,7 @@ impl ApiServer {
// Ok(())
// }
#[cfg(feature = "python")]
fn get_module(cjob: &CompletedJob, id: &str) -> Option<FlowStatusModule> {
cjob.flow_status.clone().and_then(|fs| {
find_module_in_vec(
@@ -176,6 +181,7 @@ fn get_module(cjob: &CompletedJob, id: &str) -> Option<FlowStatusModule> {
})
}
#[cfg(feature = "python")]
fn find_module_in_vec(modules: Vec<FlowStatusModule>, id: &str) -> Option<FlowStatusModule> {
modules.into_iter().find(|s| s.id() == id)
}
@@ -284,6 +290,7 @@ mod suspend_resume {
.unwrap()
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -366,6 +373,7 @@ mod suspend_resume {
);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn cancel_from_job(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -391,6 +399,7 @@ mod suspend_resume {
);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn cancel_after_suspend(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -564,6 +573,7 @@ def main(last, port):
.unwrap()
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_pass(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -609,6 +619,7 @@ def main(last, port):
assert_eq!(json!([3, 5, 7, 9]), result);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_fail_step_zero(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -652,6 +663,7 @@ def main(last, port):
);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_fail_step_one(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -693,6 +705,7 @@ def main(last, port):
.contains("index out of range"));
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_with_failure_module(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -769,6 +782,7 @@ def main(error, port):
}
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_iteration(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -827,6 +841,7 @@ async fn test_iteration(db: Pool<Postgres>) {
.contains("2"));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_iteration_parallel(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1105,6 +1120,7 @@ trait StreamFind: futures::Stream + Unpin + Sized {
impl<T: futures::Stream + Unpin + Sized> StreamFind for T {}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_deno_flow(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1223,6 +1239,7 @@ async fn test_deno_flow(db: Pool<Postgres>) {
}
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_identity(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1260,6 +1277,7 @@ async fn test_identity(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!(42));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_deno_flow_same_worker(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1535,6 +1553,7 @@ async fn test_flow_result_by_id(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!([[42]]));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_stop_after_if(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1588,6 +1607,7 @@ async fn test_stop_after_if(db: Pool<Postgres>) {
assert_eq!(json!(-123), result);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_stop_after_if_nested(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1646,6 +1666,7 @@ async fn test_stop_after_if_nested(db: Pool<Postgres>) {
assert_eq!(json!([-123]), result);
}
#[cfg(all(feature = "deno_core", feature = "python"))]
#[sqlx::test(fixtures("base"))]
async fn test_python_flow(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1703,6 +1724,7 @@ async fn test_python_flow(db: Pool<Postgres>) {
}
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_python_flow_2(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -1777,6 +1799,7 @@ func main(derp string) (string, error) {
assert_eq!(result, serde_json::json!("hello world"));
}
#[cfg(feature = "rust")]
#[sqlx::test(fixtures("base"))]
async fn test_rust_job(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2032,6 +2055,7 @@ public class Main {
assert_eq!(job.json_result(), Some(json!("hello world")));
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_python_job(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2065,6 +2089,7 @@ def main():
assert_eq!(result, serde_json::json!("hello world"));
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_python_job_heavy_dep(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2101,6 +2126,7 @@ def main():
assert_eq!(result, serde_json::json!(3));
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_python_job_with_imports(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2204,6 +2230,7 @@ export async function main(a: Date) {
assert_eq!(result, serde_json::json!("object"));
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_python_job_datetime_and_bytes(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2239,6 +2266,7 @@ def main(a: datetime, b: bytes):
assert_eq!(result, serde_json::json!([true, true]));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_empty_loop_1(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2295,6 +2323,7 @@ async fn test_empty_loop_1(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!(0));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_invalid_first_step(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2375,6 +2404,7 @@ async fn test_empty_loop_2(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!([]));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_step_after_loop(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2498,6 +2528,7 @@ async fn test_branchone_simple(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!([1, 2]));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_branchone_with_cond(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2534,6 +2565,7 @@ async fn test_branchone_with_cond(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!([1, 3]));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_branchall_sequential(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2572,6 +2604,7 @@ async fn test_branchall_sequential(db: Pool<Postgres>) {
assert_eq!(result, serde_json::json!([[1, 2], [1, 3]]));
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_branchall_simple(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2699,6 +2732,7 @@ async fn test_branchall_skip_failure(db: Pool<Postgres>) {
);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_branchone_nested(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2818,6 +2852,7 @@ async fn test_branchall_nested(db: Pool<Postgres>) {
);
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_failure_module(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -2930,6 +2965,7 @@ async fn test_failure_module(db: Pool<Postgres>) {
assert_eq!(json!({ "l": [0, 1, 2] }), result);
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base"))]
async fn test_flow_lock_all(db: Pool<Postgres>) {
use futures::StreamExt;
@@ -3068,6 +3104,7 @@ async fn test_flow_lock_all(db: Pool<Postgres>) {
});
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base"))]
async fn test_complex_flow_restart(db: Pool<Postgres>) {
@@ -3752,6 +3789,7 @@ export async function main() {
run_preview_relative_imports(&db, content, ScriptLang::Bun).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "relative_bun"))]
async fn test_nested_imports_bun(db: Pool<Postgres>) {
let content = r#"
@@ -3800,6 +3838,7 @@ export async function main() {
run_preview_relative_imports(&db, content, ScriptLang::Deno).await;
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base", "relative_python"))]
async fn test_relative_imports_python(db: Pool<Postgres>) {
let content = r#"
@@ -3817,6 +3856,7 @@ def main():
run_preview_relative_imports(&db, content, ScriptLang::Python3).await;
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base", "relative_python"))]
async fn test_nested_imports_python(db: Pool<Postgres>) {
let content = r#"
@@ -3832,6 +3872,7 @@ def main():
run_preview_relative_imports(&db, content, ScriptLang::Python3).await;
}
#[cfg(feature = "python")]
async fn assert_lockfile(
db: &Pool<Postgres>,
script_content: String,
@@ -3924,6 +3965,8 @@ async fn assert_lockfile(
)
.await;
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base", "lockfile_python"))]
async fn test_requirements_python(db: Pool<Postgres>) {
let content = r#"
@@ -3949,6 +3992,8 @@ def main():
)
.await;
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base", "lockfile_python"))]
async fn test_extra_requirements_python(db: Pool<Postgres>) {
{
@@ -3976,6 +4021,8 @@ def main():
.await;
}
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base", "lockfile_python"))]
async fn test_extra_requirements_python2(db: Pool<Postgres>) {
let content = r#"
@@ -3998,6 +4045,7 @@ def main():
.await;
}
#[cfg(feature = "python")]
#[sqlx::test(fixtures("base", "lockfile_python"))]
async fn test_pins_python(db: Pool<Postgres>) {
let content = r#"
@@ -4281,6 +4329,7 @@ mod job_payload {
];
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_script_hash_payload(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -4441,6 +4490,7 @@ mod job_payload {
test_for_versions(VERSION_FLAGS.iter().cloned(), test).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_flow_node_payload(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -4625,6 +4675,7 @@ mod job_payload {
test_for_versions(VERSION_FLAGS.iter().cloned(), test).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_flow_payload(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -4667,6 +4718,7 @@ mod job_payload {
test_for_versions(VERSION_FLAGS.iter().cloned(), test).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_flow_payload_with_preprocessor(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -4734,6 +4786,7 @@ mod job_payload {
test_for_versions(VERSION_FLAGS.iter().cloned(), test).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_restarted_flow_payload(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -4786,6 +4839,7 @@ mod job_payload {
test_for_versions(VERSION_FLAGS.iter().cloned(), test).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_raw_flow_payload(db: Pool<Postgres>) {
initialize_tracing().await;
@@ -4832,6 +4886,7 @@ mod job_payload {
test_for_versions(VERSION_FLAGS.iter().cloned(), test).await;
}
#[cfg(feature = "deno_core")]
#[sqlx::test(fixtures("base", "hello"))]
async fn test_raw_flow_payload_with_restarted_from(db: Pool<Postgres>) {
initialize_tracing().await;

View File

@@ -15,7 +15,7 @@ stripe = []
agent_worker_server = []
enterprise_saml = ["dep:samael", "dep:libxml"]
benchmark = []
embedding = ["dep:tinyvector", "dep:hf-hub", "dep:tokenizers", "dep:candle-core", "dep:candle-transformers", "dep:candle-nn", "dep:half"]
embedding = ["dep:tinyvector", "dep:hf-hub", "dep:tokenizers", "dep:candle-core", "dep:candle-transformers", "dep:candle-nn"]
parquet = ["dep:datafusion", "dep:object_store", "dep:url", "windmill-common/parquet", "windmill-worker/parquet"]
prometheus = ["windmill-common/prometheus", "windmill-queue/prometheus", "dep:prometheus", "windmill-worker/prometheus"]
openidconnect = ["dep:openidconnect"]
@@ -54,7 +54,6 @@ tokio-stream.workspace = true
anyhow.workspace = true
argon2.workspace = true
axum.workspace = true
half = { workspace = true, optional = true}
futures.workspace = true
git-version.workspace = true
tower.workspace = true

View File

@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.485.3
version: 1.486.1
title: Windmill API
contact:
@@ -13430,8 +13430,6 @@ components:
type: string
raw_code:
type: string
leaf_job:
$ref: "#/components/schemas/LeafJob"
canceled:
type: boolean
canceled_by:
@@ -16731,30 +16729,4 @@ components:
channel_name:
type: string
description: Microsoft Teams channel name
minLength: 1
LeafJob:
type: object
additionalProperties:
oneOf:
- type: object
properties:
SingleJob:
type: string
format: uuid
description: UUID of an individual job
required:
- SingleJob
additionalProperties: false
- type: object
properties:
ListJob:
type: array
items:
type: string
format: uuid
description: Array of UUIDs for multiple related jobs
required:
- ListJob
additionalProperties: false
description: Map of keys to JobResult, which can be either SingleJob (individual UUID) or ListJob (array of UUIDs)
minLength: 1

View File

@@ -253,7 +253,7 @@ impl ModelInstance {
let token_ids = Tensor::new(&tokens[..], &Device::Cpu)?.unsqueeze(0)?;
let token_type_ids = token_ids.zeros_like()?;
let embedding = self.model.forward(&token_ids, &token_type_ids)?;
let embedding = self.model.forward(&token_ids, &token_type_ids, None)?;
let embedding = (embedding.sum(1)? / embedding.dim(1)? as f64)?;
let embedding = normalize_l2(&embedding)?;

View File

@@ -12,27 +12,168 @@ use rmcp::{
service::{RequestContext, RoleServer},
Error,
};
use serde::Serialize;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use sql_builder::prelude::*;
use sqlx::FromRow;
use tokio::try_join;
use tokio_util::sync::CancellationToken;
use windmill_common::db::UserDB;
use windmill_common::scripts::Schema;
use windmill_common::worker::to_raw_value;
use windmill_common::DB;
use windmill_common::{DB, HUB_BASE_URL};
use windmill_common::scripts::{get_full_hub_script_by_path, Schema};
use crate::db::ApiAuthed;
use crate::jobs::{
run_wait_result_flow_by_path_internal, run_wait_result_script_by_path_internal, RunJobQuery,
};
use windmill_common::utils::StripPath;
use crate::HTTP_CLIENT;
use windmill_common::utils::{query_elems_from_hub, StripPath};
/// Transforms the path for workspace scripts/flows.
///
/// This function takes a path and a type string.
/// It then formats the transformed path with the type prefix.
/// This is used when listing, because we can't have names with slashes.
/// Because we replace slashes with underscores, we also need to escape underscores.
///
/// # Parameters
/// - `path`: The path to transform.
/// - `type_str`: The type of the item (script or flow).
///
/// # Returns
/// - `String`: The transformed path.
fn transform_path(path: &str, type_str: &str) -> String {
// Only apply special underscore escaping for paths starting with "f/"
let transformed = if path.starts_with("f/") {
let escaped_path = path.replace('_', "__");
escaped_path.replace('/', "_")
} else {
path.replace('/', "_")
};
// first letter of type_str is used as prefix, only one letter to avoid reaching 60 char name limit
format!("{}-{}", &type_str[..1], transformed)
}
fn convert_schema_to_schema_type(schema: Option<Schema>) -> SchemaType {
let schema_obj = if let Some(ref s) = schema {
match serde_json::from_str::<SchemaType>(s.0.get()) {
Ok(val) => val,
Err(_) => SchemaType::default(),
}
} else {
SchemaType::default()
};
schema_obj
}
trait ToolableItem {
fn get_path_or_id(&self) -> String;
fn get_summary(&self) -> &str;
fn get_description(&self) -> &str;
fn get_schema(&self) -> SchemaType;
fn is_hub(&self) -> bool;
fn item_type(&self) -> &'static str;
fn get_integration_type(&self) -> Option<String>;
}
impl ToolableItem for ScriptInfo {
fn get_path_or_id(&self) -> String {
transform_path(&self.path, "script")
}
fn get_summary(&self) -> &str {
self.summary.as_deref().unwrap_or("No summary")
}
fn get_description(&self) -> &str {
self.description.as_deref().unwrap_or("No description")
}
fn get_schema(&self) -> SchemaType {
convert_schema_to_schema_type(self.schema.clone())
}
fn is_hub(&self) -> bool {
false
}
fn item_type(&self) -> &'static str {
"script"
}
fn get_integration_type(&self) -> Option<String> {
None
}
}
impl ToolableItem for FlowInfo {
fn get_path_or_id(&self) -> String {
transform_path(&self.path, "flow")
}
fn get_summary(&self) -> &str {
self.summary.as_deref().unwrap_or("No summary")
}
fn get_description(&self) -> &str {
self.description.as_deref().unwrap_or("No description")
}
fn get_schema(&self) -> SchemaType {
convert_schema_to_schema_type(self.schema.clone())
}
fn is_hub(&self) -> bool {
false
}
fn item_type(&self) -> &'static str {
"flow"
}
fn get_integration_type(&self) -> Option<String> {
None
}
}
impl ToolableItem for HubScriptInfo {
fn get_path_or_id(&self) -> String {
let id = self.version_id;
let summary = self.summary.as_deref().unwrap_or("No summary");
format!("hs-{}-{}", id, summary.replace(" ", "_"))
}
fn get_summary(&self) -> &str {
self.summary.as_deref().unwrap_or("No summary")
}
fn get_description(&self) -> &str {
self.description.as_deref().unwrap_or("No description")
}
fn get_schema(&self) -> SchemaType {
match serde_json::from_value::<SchemaType>(self.schema.clone().unwrap_or_default()) {
Ok(schema_type) => schema_type,
Err(_) => SchemaType::default(),
}
}
fn is_hub(&self) -> bool {
true
}
fn item_type(&self) -> &'static str {
"script"
}
fn get_integration_type(&self) -> Option<String> {
self.app.clone()
}
}
#[derive(Clone)]
pub struct Runner {}
#[derive(serde::Deserialize, serde::Serialize)]
#[derive(Serialize, Deserialize, Debug)]
struct HubResponse {
asks: Vec<HubScriptInfo>,
}
#[derive(Serialize, Deserialize, Debug)]
struct HubScriptInfo {
version_id: u64,
summary: Option<String>,
description: Option<String>,
schema: Option<Value>,
app: Option<String>,
}
#[derive(Serialize, FromRow, Deserialize, Debug, Clone)]
struct SchemaType {
r#type: String,
properties: std::collections::HashMap<String, serde_json::Value>,
@@ -94,7 +235,7 @@ impl Runner {
authed: &ApiAuthed,
workspace_id: &str,
item_type: &str,
) -> Result<ItemSchema, Error> {
) -> Result<Option<Schema>, Error> {
let mut sqlb = SqlBuilder::select_from(&format!("{} as o", item_type));
sqlb.fields(&["o.schema"]);
sqlb.and_where("o.path = ?".bind(&path));
@@ -110,7 +251,7 @@ impl Runner {
.begin(authed)
.await
.map_err(|_e| Error::internal_error("failed to begin transaction", None))?;
let rows = sqlx::query_as::<_, ItemSchema>(&sql)
let item = sqlx::query_as::<_, ItemSchema>(&sql)
.fetch_one(&mut *tx)
.await
.map_err(|_e| {
@@ -120,27 +261,29 @@ impl Runner {
tx.commit()
.await
.map_err(|_e| Error::internal_error("failed to commit transaction", None))?;
Ok(rows)
Ok(item.schema)
}
fn transform_path(path: &str, type_str: &str) -> Result<String, String> {
if type_str != "script" && type_str != "flow" {
return Err(format!("Invalid type: {}", type_str));
}
// Only apply special underscore escaping for paths starting with "f/"
let transformed = if path.starts_with("f/") {
let escaped_path = path.replace('_', "__");
escaped_path.replace('/', "_")
/// Reverses the transformation of a path.
///
/// This function takes a transformed path and reverses the transformation applied by `transform_path`.
/// It checks if the path starts with "h" (indicating a Hub script) and removes the prefix if present.
/// It then determines the type of the item (script or flow) based on the prefix.
/// This is used in call_tool to get the original path, and the type of the item.
///
/// # Parameters
/// - `transformed_path`: The transformed path to reverse.
///
/// # Returns
/// - `Result<(&str, String, bool), String>`: A tuple containing the original path, the type of the item, and a boolean indicating if it's a Hub script.
/// - `Err(String)`: If the path is invalid.
fn reverse_transform(transformed_path: &str) -> Result<(&str, String, bool), String> {
let is_hub = transformed_path.starts_with("h");
let transformed_path = if is_hub {
transformed_path[1..].to_string()
} else {
path.replace('/', "_")
transformed_path.to_string()
};
// first letter of type_str is used as prefix, only one letter to avoid reaching 60 char name limit
Ok(format!("{}-{}", &type_str[..1], transformed))
}
fn reverse_transform(transformed_path: &str) -> Result<(&str, String), String> {
let type_str = if transformed_path.starts_with("s-") {
"script"
} else if transformed_path.starts_with("f-") {
@@ -157,7 +300,10 @@ impl Runner {
// Check if this path was previously transformed with special underscore handling
let is_special_path = mangled_path.starts_with("f_");
let original_path = if is_special_path {
let original_path = if is_hub {
let parts = mangled_path.split("-").collect::<Vec<&str>>();
parts[0].to_string()
} else if is_special_path {
const TEMP_PLACEHOLDER: &str = "@@UNDERSCORE@@";
let path_with_placeholder = mangled_path.replace("__", TEMP_PLACEHOLDER);
let path_with_slashes = path_with_placeholder.replace('_', "/");
@@ -166,7 +312,7 @@ impl Runner {
mangled_path.replacen('_', "/", 2)
};
Ok((type_str, original_path))
Ok((type_str, original_path, is_hub))
}
async fn inner_get_resources_types(
@@ -280,6 +426,52 @@ impl Runner {
Ok(rows)
}
async fn inner_get_scripts_from_hub(
db: &DB,
scope_integrations: Option<&str>,
) -> Result<Vec<HubScriptInfo>, Error> {
let query_params = Some(vec![
("limit", "100".to_string()),
("with_schema", "true".to_string()),
("apps", scope_integrations.unwrap_or("").to_string()),
]);
let url = format!("{}/scripts/top", *HUB_BASE_URL.read().await);
let (_status_code, _headers, response) =
query_elems_from_hub(&HTTP_CLIENT, &url, query_params, &db)
.await
.map_err(|e| {
tracing::error!("Failed to get items from hub: {}", e);
Error::internal_error(format!("Failed to get items from hub: {}", e), None)
})?;
let body_bytes = to_bytes(response, usize::MAX).await.map_err(|e| {
tracing::error!("Failed to read response body: {}", e);
Error::internal_error(format!("Failed to read response body: {}", e), None)
})?;
let body_str = String::from_utf8(body_bytes.to_vec()).map_err(|e| {
tracing::error!("Failed to decode response body: {}", e);
Error::internal_error(format!("Failed to decode response body: {}", e), None)
})?;
let hub_response: HubResponse = serde_json::from_str(&body_str).map_err(|e| {
tracing::error!("Failed to parse hub response: {}", e);
Error::internal_error(format!("Failed to parse hub response: {}", e), None)
})?;
Ok(hub_response.asks)
}
/// Transforms a value if it's an object.
///
/// This function takes a key and a value, and a schema object.
/// If the value is a string that starts with "$res:", it returns the value as is.
/// Otherwise, it checks if the key is defined in the schema and if it's an object type.
/// If it is, it transforms the value to a string. This is because some clients do not support object types.
/// # Parameters
/// - `key`: The key of the value to transform.
/// - `value`: The value to transform.
/// - `schema_obj`: The schema object.
///
/// # Returns
/// - `Value`: The transformed value.
fn transform_value_if_object(
key: &str,
value: &Value,
@@ -315,6 +507,16 @@ impl Runner {
value.clone()
}
/// Reverses the transformation of a key.
///
/// This function takes a transformed key and a schema object.
/// It then reverses the transformation applied by `apply_key_transformation`. This can be subject to collisions, but it's unlikely and is ok for our use case.
/// # Parameters
/// - `transformed_key`: The transformed key to reverse.
/// - `schema_obj`: The schema object.
///
/// # Returns
/// - `String`: The original key.
fn reverse_transform_key(transformed_key: &str, schema_obj: &Option<SchemaType>) -> String {
let schema_obj = match schema_obj {
Some(s) => s,
@@ -338,6 +540,16 @@ impl Runner {
transformed_key.to_string()
}
/// Applies a key transformation to a key.
///
/// This function takes a key and replaces spaces with underscores.
/// It also removes any characters that are not alphanumeric or underscores.
/// This is used when listing, because we can't have names with spaces or special characters in the schema properties.
/// # Parameters
/// - `key`: The key to transform.
///
/// # Returns
/// - `String`: The transformed key.
fn apply_key_transformation(key: &str) -> String {
key.replace(' ', "_")
.chars()
@@ -345,18 +557,32 @@ impl Runner {
.collect::<String>()
}
/// Transforms the schema for resources.
///
/// This function takes a schema and a database connection, and attempts to transform the schema for resources.
/// It replaces invalid characters in property keys with underscores and converts object properties to strings.
/// It also fetches resource type information and adds it to the description of resource properties.
///
/// # Parameters
/// - `schema`: The schema to transform.
/// - `user_db`: The database connection.
/// - `authed`: The authenticated user.
/// - `w_id`: The workspace ID.
/// - `resources_cache`: A mutable reference to the resources cache.
/// - `resources_types`: A reference to the resource types.
///
/// # Returns
/// - `Result<SchemaType, Error>`: The transformed schema.
/// - `Err(Error)`: If the transformation fails.
async fn transform_schema_for_resources(
schema: &Schema,
schema: &SchemaType,
user_db: &UserDB,
authed: &ApiAuthed,
w_id: &str,
resources_cache: &mut HashMap<String, Vec<ResourceInfo>>,
resources_types: &Vec<ResourceType>,
) -> Result<SchemaType, Error> {
let mut schema_obj: SchemaType = match serde_json::from_str(schema.0.get()) {
Ok(val) => val,
Err(_) => SchemaType::default(),
};
let mut schema_obj: SchemaType = schema.clone();
// replace invalid char in property key with underscore
let replacements: Vec<(String, String, serde_json::Value)> = schema_obj
@@ -489,9 +715,133 @@ impl Runner {
Ok(schema_obj)
}
/// Fetches the schema for a Hub script.
///
/// This function takes a script path and a database connection, and attempts to fetch the schema for the script.
/// It strips the path to remove any leading slashes, and then attempts to retrieve the full script using `get_full_hub_script_by_path`.
/// If successful, it converts the schema string to a `Schema` object.
/// If the schema cannot be converted, it logs a warning and returns `None`.
///
/// # Parameters
/// - `path`: The path of the script to fetch the schema for.
/// - `db`: The database connection.
///
/// # Returns
/// - `Ok(Option<Schema>)`: The schema if found, otherwise `None`.
/// - `Err(Error)`: If the request fails.
async fn get_hub_script_schema(path: &str, db: &DB) -> Result<Option<Schema>, Error> {
let strip_path = StripPath(path.to_string());
let res = get_full_hub_script_by_path(strip_path, &HTTP_CLIENT, Some(db))
.await
.map_err(|e| {
tracing::error!("Failed to get hub script: {}", e);
Error::internal_error(format!("Failed to get hub script: {}", e), None)
})?;
match serde_json::from_str::<Schema>(res.schema.get()) {
Ok(schema) => Ok(Some(schema)),
Err(e) => {
tracing::warn!("Failed to convert schema: {}", e);
Ok(None)
}
}
}
/// Creates a `Tool` from a `ToolableItem`.
///
/// This function takes an item that implements the `ToolableItem` trait and converts it into an RMCP `Tool`.
/// It handles both workspace scripts/flows and Hub scripts differently, depending on the item type.
///
/// # Parameters
/// - `item`: The item to convert to a `Tool`.
/// - `user_db`: The database connection.
/// - `authed`: The authenticated user.
/// - `workspace_id`: The workspace ID.
/// - `resources_cache`: A mutable reference to the resources cache.
/// - `resources_types`: A reference to the resource types.
///
/// # Returns
/// - `Ok(Tool)`: The created `Tool`.
async fn create_tool_from_item<T: ToolableItem>(
item: &T,
user_db: &UserDB,
authed: &ApiAuthed,
workspace_id: &str,
resources_cache: &mut HashMap<String, Vec<ResourceInfo>>,
resources_types: &Vec<ResourceType>,
) -> Result<Tool, Error> {
let is_hub = item.is_hub();
let path = item.get_path_or_id();
let item_type = item.item_type();
let description = format!(
"This is a {} named `{}` with the following description: `{}`.{}",
item_type,
item.get_summary(),
item.get_description(),
if is_hub {
format!(
" It is a tool used for the following app: {}",
item.get_integration_type()
.unwrap_or("No integration type".to_string())
)
} else {
"".to_string()
}
);
let schema_obj = Runner::transform_schema_for_resources(
&item.get_schema(),
user_db,
authed,
&workspace_id,
resources_cache,
&resources_types,
)
.await?;
let input_schema_map = match serde_json::to_value(schema_obj) {
Ok(Value::Object(map)) => map,
Ok(_) => {
tracing::warn!("Schema object for tool '{}' did not serialize to a JSON object, using empty schema.", path);
serde_json::Map::new()
}
Err(e) => {
tracing::error!(
"Failed to serialize schema object for tool '{}': {}. Using empty schema.",
path,
e
);
serde_json::Map::new()
}
};
Ok(Tool {
name: Cow::Owned(path),
description: Some(Cow::Owned(description)),
input_schema: Arc::new(input_schema_map),
annotations: None,
})
}
}
impl ServerHandler for Runner {
/// Handles the `CallTool` request from the MCP client.
///
/// This involves:
/// 1. Parsing arguments and extracting context (DB, Auth).
/// 2. Reversing the tool name (`request.name`) to get the original path and type using `reverse_transform`.
/// 3. Handling Hub scripts: If identified as a Hub script, searches the Hub for the actual script ID.
/// 4. Fetching the schema for the item (needed for argument transformation).
/// 5. Transforming incoming arguments:
/// - Reversing key transformations (e.g., `user_input` back to `user input`).
/// - Parsing stringified JSON objects back into JSON values based on schema type.
/// 6. Executing the corresponding script or flow using internal Windmill runners.
/// 7. Formatting the execution result into an RMCP `CallToolResult`.
///
/// # Parameters
/// - `request`: The `CallToolRequestParam` containing the tool name and arguments.
/// - `context`: The `RequestContext` providing access to workspace ID, DB connections, auth info.
///
/// # Returns
/// - `Ok(CallToolResult)`: On successful execution, containing the output.
/// - `Err(Error)`: If any step fails (parsing, DB access, execution, reversing transform, hub search).
async fn call_tool(
&self,
request: CallToolRequestParam,
@@ -520,14 +870,17 @@ impl ServerHandler for Runner {
.ok_or_else(|| Error::internal_error("UserDB not found", None))?;
let args = parse_args(request.arguments)?;
let (tool_type, path) = Runner::reverse_transform(&request.name).unwrap_or_default();
let (tool_type, path, is_hub) =
Runner::reverse_transform(&request.name).unwrap_or_default();
let item_info =
let item_schema = if is_hub {
Runner::get_hub_script_schema(&format!("hub/{}", path), db).await?
} else {
Runner::get_item_schema(&path, user_db, authed, &context.workspace_id, &tool_type)
.await?;
.await?
};
let schema = item_info.schema;
let schema_obj = if let Some(ref s) = schema {
let schema_obj = if let Some(ref s) = item_schema {
match serde_json::from_str::<SchemaType>(s.0.get()) {
Ok(val) => Some(val),
Err(e) => {
@@ -542,7 +895,7 @@ impl ServerHandler for Runner {
let push_args = if let Value::Object(map) = args.clone() {
let mut args_hash = HashMap::new();
for (k, v) in map {
// need to transform back the key to the original key
// need to transform back the key without invalid characters to the original key
let original_key = Runner::reverse_transform_key(&k, &schema_obj);
// object properties are transformed to string because some client does not support object, might change in the future
@@ -555,7 +908,11 @@ impl ServerHandler for Runner {
};
let w_id = context.workspace_id.clone();
let script_or_flow_path = StripPath(path);
let script_or_flow_path = if is_hub {
StripPath(format!("hub/{}", path))
} else {
StripPath(path)
};
let run_query = RunJobQuery::default();
let result = if tool_type == "script" {
@@ -603,12 +960,31 @@ impl ServerHandler for Runner {
}
}
/// Fetches available tools (scripts, flows, hub scripts) based on the user's scope.
///
/// - Determines scope (all, favorites, hub-specific) from auth token.
/// - Fetches relevant items (workspace scripts/flows, hub scripts) concurrently.
/// - Fetches resource type information needed for schema enrichment.
/// - Transforms each item into an RMCP `Tool` definition, including schema adjustments
/// (like resource description enrichment and object->string conversion).
///
/// # Parameters
/// - `_request`: Optional pagination parameters (currently ignored).
/// - `_context`: The `RequestContext` providing workspace ID, DB, auth.
///
/// # Returns
/// - `Ok(ListToolsResult)`: A list of `Tool` definitions. Pagination is not yet implemented.
/// - `Err(Error)`: If fetching data from DB or Hub fails.
async fn list_tools(
&self,
_request: Option<PaginatedRequestParam>,
mut _context: RequestContext<RoleServer>,
) -> Result<ListToolsResult, Error> {
let workspace_id = _context.workspace_id.clone();
let db = _context
.req_extensions
.get::<DB>()
.ok_or_else(|| Error::internal_error("DB not found", None))?;
let user_db = _context
.req_extensions
.get::<UserDB>()
@@ -617,11 +993,27 @@ impl ServerHandler for Runner {
.req_extensions
.get::<ApiAuthed>()
.ok_or_else(|| Error::internal_error("ApiAuthed not found", None))?;
let scope = authed
let owned_scope = authed.scopes.as_ref().and_then(|scopes| {
scopes
.iter()
.find(|scope| scope.starts_with("mcp:") && !scope.contains("hub"))
});
let hub_scope = authed
.scopes
.as_ref()
.and_then(|scopes| scopes.iter().find(|scope| scope.starts_with("mcp:")));
let scope_type = scope.map_or("all", |scope| scope.split(":").last().unwrap_or("all"));
.and_then(|scopes| scopes.iter().find(|scope| scope.starts_with("mcp:hub")));
let scope_type = owned_scope.map_or("all", |scope| {
let parts = scope.split(":").collect::<Vec<&str>>();
parts[1]
});
let scope_integrations = hub_scope.and_then(|scope| {
let parts = scope.split(":").collect::<Vec<&str>>();
if parts.len() == 3 {
Some(parts[2])
} else {
None
}
});
let scripts_fn = Runner::inner_get_items::<ScriptInfo>(
user_db,
@@ -633,84 +1025,62 @@ impl ServerHandler for Runner {
let flows_fn =
Runner::inner_get_items::<FlowInfo>(user_db, authed, &workspace_id, scope_type, "flow");
let resources_types_fn = Runner::inner_get_resources_types(user_db, authed, &workspace_id);
let (scripts, flows, resources_types) =
try_join!(scripts_fn, flows_fn, resources_types_fn)?;
let hub_scripts_fn = Runner::inner_get_scripts_from_hub(db, scope_integrations.as_deref());
let (scripts, flows, resources_types, hub_scripts) = if scope_integrations.is_some() {
let (scripts, flows, resources_types, hub_scripts) =
try_join!(scripts_fn, flows_fn, resources_types_fn, hub_scripts_fn)?;
(scripts, flows, resources_types, hub_scripts)
} else {
let (scripts, flows, resources_types) =
try_join!(scripts_fn, flows_fn, resources_types_fn)?;
(scripts, flows, resources_types, vec![])
};
let mut resources_cache: HashMap<String, Vec<ResourceInfo>> = HashMap::new();
let mut tools: Vec<Tool> = Vec::new();
let mut script_tools: Vec<Tool> = Vec::with_capacity(scripts.len());
for script in scripts {
let name = Runner::transform_path(&script.path, "script").unwrap_or_default();
let description = format!(
"This is a script named `{}` with the following description: `{}`.",
script.summary.as_deref().unwrap_or("No summary"),
script.description.as_deref().unwrap_or("No description")
);
let schema_obj = if let Some(schema) = script.schema {
Runner::transform_schema_for_resources(
&schema,
tools.push(
Runner::create_tool_from_item(
&script,
user_db,
authed,
&workspace_id,
&mut resources_cache,
&resources_types,
)
.await?
} else {
SchemaType::default()
};
script_tools.push(Tool {
name: Cow::Owned(name),
description: Some(Cow::Owned(description)),
input_schema: {
let value = serde_json::to_value(schema_obj).unwrap_or_default();
if let serde_json::Value::Object(map) = value {
Arc::new(map)
} else {
Arc::new(serde_json::Map::new())
}
},
annotations: None,
});
.await?,
);
}
let mut flow_tools: Vec<Tool> = Vec::with_capacity(flows.len());
for flow in flows {
let name = Runner::transform_path(&flow.path, "flow").unwrap_or_default();
let description = format!(
"This is a flow named `{}` with the following description: `{}`.",
flow.summary.as_deref().unwrap_or("No summary"),
flow.description.as_deref().unwrap_or("No description")
);
let schema_obj = if let Some(schema) = flow.schema {
Runner::transform_schema_for_resources(
&schema,
tools.push(
Runner::create_tool_from_item(
&flow,
user_db,
authed,
&workspace_id,
&mut resources_cache,
&resources_types,
)
.await?
} else {
SchemaType::default()
};
flow_tools.push(Tool {
name: Cow::Owned(name),
description: Some(Cow::Owned(description)),
input_schema: {
let value = serde_json::to_value(schema_obj).unwrap_or_default();
if let serde_json::Value::Object(map) = value {
Arc::new(map)
} else {
Arc::new(serde_json::Map::new())
}
},
annotations: None,
});
.await?,
);
}
for hub_script in hub_scripts {
tools.push(
Runner::create_tool_from_item(
&hub_script,
user_db,
authed,
&workspace_id,
&mut resources_cache,
&resources_types,
)
.await?,
);
}
let tools = [script_tools, flow_tools].concat();
Ok(ListToolsResult { tools, next_cursor: None })
}

View File

@@ -265,9 +265,12 @@ async fn list_scripts(
if lq.show_archived.unwrap_or(false) {
sqlb.and_where_eq(
"o.created_at",
"(select max(created_at) from script where o.path = path
AND workspace_id = ?)"
"o.ctid",
"(SELECT ctid FROM script
WHERE path = o.path
AND workspace_id = ?
ORDER BY created_at DESC
LIMIT 1)"
.bind(&w_id),
);
sqlb.and_where_eq("archived", true);
@@ -650,8 +653,13 @@ async fn create_script_internal<'c>(
) {
Some(String::new())
} else {
ns.lock
.and_then(|e| if e.is_empty() { None } else { Some(e) })
ns.lock.as_ref().and_then(|e| {
if e.is_empty() {
None
} else {
Some(e.to_string())
}
})
};
let needs_lock_gen = lock.is_none() && codebase.is_none();
@@ -901,6 +909,7 @@ async fn create_script_internal<'c>(
let permissioned_as2 = permissioned_as.clone();
let script_path2 = script_path.clone();
let parent_path = p_path_opt.clone();
let lock = ns.lock.clone();
let deployment_message = ns.deployment_message.clone();
let content = ns.content.clone();
let language = ns.language.clone();
@@ -920,6 +929,7 @@ async fn create_script_internal<'c>(
&authed2.email,
&authed2.username,
&permissioned_as2,
lock,
)
.await
{
@@ -982,7 +992,7 @@ async fn get_script_by_path(
AND favorite.usr = $3
WHERE s.path = $1
AND s.workspace_id = $2
AND s.created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2)",
ORDER BY s.created_at DESC LIMIT 1",
)
.bind(path)
.bind(w_id)
@@ -991,9 +1001,7 @@ async fn get_script_by_path(
.await?
} else {
sqlx::query_as::<_, ScriptWithStarred>(
"SELECT *, NULL as starred FROM script WHERE path = $1 AND workspace_id = $2 \
AND created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND \
workspace_id = $2)",
"SELECT *, NULL as starred FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1",
)
.bind(path)
.bind(w_id)
@@ -1033,9 +1041,8 @@ async fn get_script_by_path_w_draft(
let script_o = sqlx::query_as::<_, ScriptWDraft>(
"SELECT hash, script.path, summary, description, content, language, kind, tag, schema, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, ws_error_handler_muted, draft.value as draft, dedicated_worker, priority, restart_unless_cancelled, delete_after_use, timeout, concurrency_key, visible_to_runner_only, no_main_func, has_preprocessor, on_behalf_of_email FROM script LEFT JOIN draft ON
script.path = draft.path AND script.workspace_id = draft.workspace_id AND draft.typ = 'script'
WHERE script.path = $1 AND script.workspace_id = $2 \
AND script.created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND \
workspace_id = $2)",
WHERE script.path = $1 AND script.workspace_id = $2
ORDER BY script.created_at DESC LIMIT 1",
)
.bind(path)
.bind(w_id)
@@ -1057,7 +1064,7 @@ async fn get_script_history(
"SELECT s.hash as hash, dm.deployment_msg as deployment_msg
FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash
WHERE s.workspace_id = $1 AND s.path = $2
ORDER by created_at DESC",
ORDER by s.created_at DESC",
w_id,
path.to_path(),
)
@@ -1085,7 +1092,7 @@ async fn get_latest_version(
"SELECT s.hash as hash, dm.deployment_msg as deployment_msg
FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash
WHERE s.workspace_id = $1 AND s.path = $2
ORDER by created_at DESC",
ORDER by s.created_at DESC LIMIT 1",
w_id,
path.to_path(),
)
@@ -1181,7 +1188,15 @@ async fn toggle_workspace_error_handler(
match error_handler_maybe {
Some(_) => {
sqlx::query_scalar!(
"UPDATE script SET ws_error_handler_muted = $3 WHERE workspace_id = $2 AND path = $1 AND created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2)",
"UPDATE script
SET ws_error_handler_muted = $3
WHERE ctid = (
SELECT ctid FROM script
WHERE path = $1 AND workspace_id = $2
ORDER BY created_at DESC
LIMIT 1
)
",
path.to_path(),
w_id,
req.muted,
@@ -1202,6 +1217,7 @@ async fn toggle_workspace_error_handler(
async fn get_tokened_raw_script_by_path(
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path((w_id, token, path)): Path<(String, String, StripPath)>,
Extension(cache): Extension<Arc<AuthCache>>,
) -> Result<String> {
@@ -1209,7 +1225,13 @@ async fn get_tokened_raw_script_by_path(
.get_authed(Some(w_id.clone()), &token)
.await
.ok_or_else(|| Error::NotAuthorized("Invalid token".to_string()))?;
return raw_script_by_path(authed, Extension(user_db), Path((w_id, path))).await;
return raw_script_by_path(
authed,
Extension(user_db),
Extension(db),
Path((w_id, path)),
)
.await;
}
async fn get_empty_ts_script_by_path() -> String {
@@ -1219,22 +1241,25 @@ async fn get_empty_ts_script_by_path() -> String {
async fn raw_script_by_path(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path((w_id, path)): Path<(String, StripPath)>,
) -> Result<String> {
raw_script_by_path_internal(path, user_db, authed, w_id, false).await
raw_script_by_path_internal(path, user_db, db, authed, w_id, false).await
}
async fn raw_script_by_path_unpinned(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path((w_id, path)): Path<(String, StripPath)>,
) -> Result<String> {
raw_script_by_path_internal(path, user_db, authed, w_id, true).await
raw_script_by_path_internal(path, user_db, db, authed, w_id, true).await
}
async fn raw_script_by_path_internal(
path: StripPath,
user_db: UserDB,
db: DB,
authed: ApiAuthed,
w_id: String,
unpin: bool,
@@ -1260,10 +1285,7 @@ async fn raw_script_by_path_internal(
let mut tx = user_db.begin(&authed).await?;
let content_o = sqlx::query_scalar!(
"SELECT content FROM script WHERE path = $1 AND workspace_id = $2 \
AND
created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND archived = false AND \
workspace_id = $2)",
"SELECT content FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false ORDER BY created_at DESC LIMIT 1",
path,
w_id
)
@@ -1271,6 +1293,22 @@ async fn raw_script_by_path_internal(
.await?;
tx.commit().await?;
if content_o.is_none() {
let exists = sqlx::query_scalar!(
"SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1)",
path,
w_id
)
.fetch_one(&db)
.await?;
if exists.unwrap_or(false) {
return Err(Error::NotFound(format!(
"Script {path} not visible to {} but exists",
authed.username
)));
}
}
let content = not_found_if_none(content_o, "Script", path)?;
if unpin {
@@ -1287,8 +1325,7 @@ async fn exists_script_by_path(
let path = path.to_path();
let exists = sqlx::query_scalar!(
"SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 AND
created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2))",
"SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1)",
path,
w_id
)
@@ -1405,9 +1442,7 @@ pub async fn require_is_writer(authed: &ApiAuthed, path: &str, w_id: &str, db: D
path,
w_id,
db,
"SELECT extra_perms FROM script WHERE path = $1 AND workspace_id = $2 \
AND created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND \
workspace_id = $2)",
"SELECT extra_perms FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1",
"script",
)
.await;

View File

@@ -33,6 +33,8 @@ impl<B> OnResponse<B> for MyOnResponse {
let status = response.status().as_u16();
if response.status().is_success() || response.status().is_redirection() {
tracing::info!(latency = latency, status = status, "response")
} else if response.status().as_u16() == 404 {
tracing::warn!(latency = latency, status = status, "response")
} else {
tracing::error!(latency = latency, status = status, "response")
}

View File

@@ -76,6 +76,8 @@ quick_cache.workspace = true
pin-project-lite.workspace = true
futures.workspace = true
tempfile.workspace = true
systemstat.workspace = true
size.workspace = true
opentelemetry-semantic-conventions = { workspace = true, optional = true }
opentelemetry-otlp = { workspace = true, optional = true }

View File

@@ -98,3 +98,9 @@ pub async fn worker_groups_alerts(_db: &DB) {}
#[cfg(feature = "enterprise")]
pub async fn jobs_waiting_alerts(_db: &DB) {}
#[cfg(feature = "enterprise")]
pub async fn low_disk_alerts(_db: &DB, _server_mode: bool, _worker_mode: bool, _workers: Vec<String>) {
// Implementation is not open source
}

View File

@@ -22,7 +22,8 @@ use crate::{
error::Error,
more_serde::{default_empty_string, default_id, default_null, default_true, is_default},
scripts::{Schema, ScriptHash, ScriptLang},
worker::{to_raw_value, Connection}, DB,
worker::{to_raw_value, Connection},
DB,
};
#[derive(Serialize, Deserialize, sqlx::FromRow)]

View File

@@ -37,6 +37,7 @@ pub const HUB_BASE_URL_SETTING: &str = "hub_base_url";
pub const HUB_ACCESSIBLE_URL_SETTING: &str = "hub_accessible_url";
pub const CRITICAL_ERROR_CHANNELS_SETTING: &str = "critical_error_channels";
pub const CRITICAL_ALERT_MUTE_UI_SETTING: &str = "critical_alert_mute_ui";
pub const CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING: &str = "critical_alerts_on_db_oversize";
pub const DEV_INSTANCE_SETTING: &str = "dev_instance";
pub const JWT_SECRET_SETTING: &str = "jwt_secret";
pub const EMAIL_DOMAIN_SETTING: &str = "email_domain";

View File

@@ -115,6 +115,7 @@ lazy_static::lazy_static! {
pub static ref CRITICAL_ERROR_CHANNELS: Arc<RwLock<Vec<CriticalErrorChannel>>> = Arc::new(RwLock::new(vec![]));
pub static ref CRITICAL_ALERTS_ON_DB_OVERSIZE: Arc<RwLock<Option<f32>>> = Arc::new(RwLock::new(None));
pub static ref JOB_RETENTION_SECS: Arc<RwLock<i64>> = Arc::new(RwLock::new(0));

View File

@@ -265,7 +265,7 @@ pub struct ScriptHistoryUpdate {
pub deployment_msg: Option<String>,
}
#[derive(Serialize, Deserialize, Debug, sqlx::Type)]
#[derive(Serialize, Deserialize, Debug, sqlx::Type, Clone)]
#[sqlx(transparent)]
#[serde(transparent)]
pub struct Schema(pub sqlx::types::Json<Box<serde_json::value::RawValue>>);

View File

@@ -858,18 +858,16 @@ pub async fn handle_ansible_job(
let mut nsjail_extra_mounts = vec![];
if let Some(r) = reqs.as_ref() {
if let Some(db) = conn.as_sql() {
nsjail_extra_mounts = create_file_resources(
&job.id,
&job.workspace_id,
job_dir,
interpolated_args.as_ref(),
&r,
&client,
db,
)
.await?;
}
nsjail_extra_mounts = create_file_resources(
&job.id,
&job.workspace_id,
job_dir,
interpolated_args.as_ref(),
&r,
&client,
conn,
)
.await?;
for repo in &r.git_repos {
append_logs(
@@ -935,8 +933,7 @@ pub async fn handle_ansible_job(
if let Some(collections) = r.roles_and_collections.as_ref() {
let empty = String::new();
let (lockfile, logs) =
req_lockfiles
let (lockfile, logs) = req_lockfiles
.as_ref()
.map(|r| (&r.collections_and_roles, &r.collections_and_roles_logs))
.unwrap_or((collections, &empty));
@@ -1162,7 +1159,7 @@ async fn create_file_resources(
args: Option<&HashMap<String, Box<RawValue>>>,
r: &AnsibleRequirements,
client: &crate::AuthedClient,
db: &sqlx::Pool<sqlx::Postgres>,
conn: &Connection,
) -> error::Result<Vec<String>> {
let mut logs = String::new();
let mut nsjail_mounts: Vec<String> = vec![];
@@ -1232,7 +1229,7 @@ async fn create_file_resources(
file_res.target_path, file_res.resource_path
));
}
append_logs(job_id, w_id, logs, &Connection::Sql(db.clone())).await;
append_logs(job_id, w_id, logs, conn).await;
Ok(nsjail_mounts)
}

View File

@@ -42,7 +42,7 @@ use windmill_common::{
error::{self, Result},
get_latest_hash_for_path,
scripts::ScriptLang,
worker::{exists_in_cache, save_cache, write_file, Connection, DISABLE_BUNDLING},
worker::{exists_in_cache, save_cache, to_raw_value, write_file, Connection, DISABLE_BUNDLING},
DB,
};
@@ -111,7 +111,7 @@ pub async fn gen_bun_lockfile(
let mut empty_deps = false;
if let Some(raw_deps) = raw_deps {
if let Some(raw_deps) = raw_deps.as_ref() {
gen_bunfig(job_dir).await?;
write_file(job_dir, "package.json", raw_deps.as_str())?;
} else {
@@ -201,10 +201,21 @@ pub async fn gen_bun_lockfile(
}
if export_pkg {
let mut content = "".to_string();
let mut content;
{
let mut file = File::open(format!("{job_dir}/package.json")).await?;
file.read_to_string(&mut content).await?;
let mut buf = String::default();
file.read_to_string(&mut buf).await?;
if raw_deps.is_some() {
let mut json_map: HashMap<String, Box<RawValue>> = serde_json::from_str(&buf)?;
json_map.insert(
"generatedFromPackageJson".to_string(),
to_raw_value(&"true".to_string()),
);
content = serde_json::to_string_pretty(&json_map)?;
} else {
content = buf;
}
}
if !npm_mode {
#[cfg(any(target_os = "linux", target_os = "macos"))]

View File

@@ -77,6 +77,7 @@ use crate::{
start_child_process, OccupancyMetrics,
},
handle_child::handle_child,
worker_lockfiles::LOCKFILE_GENERATED_FROM_REQUIREMENTS_TXT,
worker_utils::ping_job_status,
AuthedClient, DISABLE_NSJAIL, DISABLE_NUSER, HOME_ENV, INSTANCE_PYTHON_VERSION, NSJAIL_PATH,
PATH_ENV, PIP_EXTRA_INDEX_URL, PIP_INDEX_URL, PROXY_ENVS, PY_INSTALL_DIR, TZ_ENV, UV_CACHE_DIR,
@@ -2303,8 +2304,16 @@ fn split_requirements(requirements: &str) -> Vec<&str> {
/// Check requirements/lockfile to figure out python version assigned to it.
fn get_pyv_from_requirements_lines(requirements_lines: &[&str]) -> PyVersion {
// If script is deployed we can try to parse first line to get assigned version
let index = if requirements_lines.get(0).map_or(false, |line| {
line.starts_with(LOCKFILE_GENERATED_FROM_REQUIREMENTS_TXT)
}) {
1
} else {
0
};
if let Some(v) = requirements_lines
.get(0)
.get(index)
.and_then(|line| PyVersion::parse_version(*line))
{
// We have valid assigned version, we use it

View File

@@ -65,17 +65,17 @@ pub async fn update_script_dependency_map(
relative_imports: Vec<String>,
) -> error::Result<()> {
let importer_kind = "script";
let mut tx = db.begin().await?;
tx = clear_dependency_parent_path(parent_path, script_path, w_id, importer_kind, tx).await?;
tx = clear_dependency_map_for_item(script_path, w_id, importer_kind, tx, &None).await?;
if !relative_imports.is_empty() {
let mut logs = "".to_string();
logs.push_str("\n--- RELATIVE IMPORTS ---\n\n");
logs.push_str(&relative_imports.join("\n"));
let mut tx = db.begin().await?;
tx =
clear_dependency_parent_path(parent_path, script_path, w_id, importer_kind, tx).await?;
tx = clear_dependency_map_for_item(script_path, w_id, importer_kind, tx, &None).await?;
tx = add_relative_imports_to_dependency_map(
script_path,
w_id,
@@ -86,9 +86,10 @@ pub async fn update_script_dependency_map(
None,
)
.await?;
tx.commit().await?;
append_logs(job_id, w_id, logs, &db.into()).await;
}
tx.commit().await?;
Ok(())
}
@@ -381,6 +382,7 @@ pub async fn handle_dependency_job(
&job.permissioned_as_email,
&job.created_by,
&job.permissioned_as,
None,
)
.await?;
@@ -433,18 +435,42 @@ pub async fn process_relative_imports(
permissioned_as_email: &str,
created_by: &str,
permissioned_as: &str,
lock: Option<String>,
) -> error::Result<()> {
let relative_imports = extract_relative_imports(&code, script_path, script_lang);
if let Some(relative_imports) = relative_imports {
update_script_dependency_map(
&job_id.unwrap_or_else(|| Uuid::nil()),
db,
w_id,
&parent_path,
script_path,
relative_imports,
)
.await?;
if (script_lang.is_some_and(|v| v == ScriptLang::Bun)
&& lock
.as_ref()
.is_some_and(|v| v.contains("generatedFromPackageJson")))
|| (script_lang.is_some_and(|v| v == ScriptLang::Python3)
&& lock
.as_ref()
.is_some_and(|v| v.starts_with(LOCKFILE_GENERATED_FROM_REQUIREMENTS_TXT)))
{
// if the lock file is generated from a package.json/requirements.txt, we need to clear the dependency map
// because we do not want to have dependencies be recomputed automatically. Empty relative imports passed
// to update_script_dependency_map will clear the dependency map.
update_script_dependency_map(
&job_id.unwrap_or_else(|| Uuid::nil()),
db,
w_id,
&parent_path,
script_path,
vec![],
)
.await?;
} else {
update_script_dependency_map(
&job_id.unwrap_or_else(|| Uuid::nil()),
db,
w_id,
&parent_path,
script_path,
relative_imports,
)
.await?;
}
let already_visited = args
.map(|x| {
x.get("already_visited")
@@ -2050,6 +2076,8 @@ async fn ansible_dep(
serde_json::to_string(&ansible_lockfile).map_err(|e| e.into())
}
pub const LOCKFILE_GENERATED_FROM_REQUIREMENTS_TXT: &str = "# from requirements.txt";
async fn capture_dependency_job(
job_id: &Uuid,
job_language: &ScriptLang,
@@ -2118,6 +2146,13 @@ async fn capture_dependency_job(
anns,
)
.await
.map(|res| {
if raw_deps {
format!("{}\n{}", LOCKFILE_GENERATED_FROM_REQUIREMENTS_TXT, res)
} else {
res
}
})
}
}
ScriptLang::Ansible => {

View File

@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.485.3";
export const VERSION = "v1.486.1";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({

View File

@@ -63,7 +63,7 @@ export {
// }
// });
export const VERSION = "1.485.3";
export const VERSION = "1.486.1";
const command = new Command()
.name("wmill")

View File

@@ -17,6 +17,7 @@
extensions = [
"rust-src" # for rust-analyzer
"rust-analyzer"
"rustfmt"
];
};
buildInputs = with pkgs; [
@@ -183,6 +184,7 @@
ANSIBLE_PLAYBOOK_PATH = "${pkgs.ansible}/bin/ansible-playbook";
ANSIBLE_GALAXY_PATH = "${pkgs.ansible}/bin/ansible-galaxy";
RUST_LOG = "debug";
SQLX_OFFLINE = "true";
};
packages.default = self.packages.${system}.windmill;
packages.windmill-client = pkgs.buildNpmPackage {

View File

@@ -1,12 +1,12 @@
{
"name": "windmill-components",
"version": "1.485.3",
"version": "1.486.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "windmill-components",
"version": "1.485.3",
"version": "1.486.1",
"hasInstallScript": true,
"license": "AGPL-3.0",
"dependencies": {

View File

@@ -1,6 +1,6 @@
{
"name": "windmill-components",
"version": "1.485.3",
"version": "1.486.1",
"scripts": {
"dev": "vite dev",
"build": "vite build",

View File

@@ -103,7 +103,9 @@
}}
disabled={!$enterpriseLicense || !$superadmin}
>
Save {#if !$superadmin} <span class="text-2xs text-tertiary">superadmin only</span> {/if}
Save {#if !$superadmin}
<span class="text-2xs text-tertiary">superadmin only</span>
{/if}
</Button>
<span class="text-2xs text-tertiary"

View File

@@ -334,15 +334,15 @@
latestKeyRenewalAttempt.result === 'success'
? 'text-green-600'
: isTrial
? 'text-yellow-600'
: 'text-red-600'
? 'text-yellow-600'
: 'text-red-600'
)}
>
{latestKeyRenewalAttempt.result === 'success'
? 'Latest key renewal succeeded'
: isTrial
? 'Latest key renewal ignored because in trial'
: 'Latest key renewal failed'}
? 'Latest key renewal ignored because in trial'
: 'Latest key renewal failed'}
on {attemptedAt}
</span>
</div>
@@ -928,6 +928,30 @@
{:else if setting.fieldType == 'object_store_config'}
<ObjectStoreConfigSettings bind:bucket_config={$values[setting.key]} />
<div class="mb-6"></div>
{:else if setting.fieldType == 'critical_alerts_on_db_oversize'}
{#if $values[setting.key]}
<div class="flex flex-row flex-wrap gap-2 p-0 items-center">
<div class="p-1">
<Toggle
disabled={!$enterpriseLicense}
bind:checked={$values[setting.key].enabled}
/>
</div>
{#if $values[setting.key].enabled}
<label class="block shrink min-w-0">
<input
type="number"
placeholder={setting.placeholder}
bind:value={$values[setting.key].value}
/>
</label>
<label class="block">
<span class="text-primary font-semibold text-sm">GB</span>
</label>
{/if}
</div>
<div class="mb-6"></div>
{/if}
{:else if setting.fieldType == 'number'}
<input
type="number"

View File

@@ -293,7 +293,7 @@
tooltip="This event is triggered when the script runs successfully."
items={Object.keys($runnableComponents).filter((_id) => _id !== id)}
bind:value={
() => hiddenInlineScript.script.recomputeIds,
() => hiddenInlineScript.script.recomputeIds ?? [],
(v) => {
if ($app.hiddenInlineScripts[hiddenInlineScript.index]) {
$app.hiddenInlineScripts[hiddenInlineScript.index].recomputeIds = v

View File

@@ -4,12 +4,15 @@
import MultiSelect from '$lib/components/multiselect/MultiSelectWrapper.svelte'
import { twMerge } from 'tailwind-merge'
export let items: string[]
export let value: string[] | undefined = undefined
export let title: string
export let tooltip: string
let width = 0
let { items, value = $bindable(), title, tooltip } = $props<{
items: string[]
value: string[] | undefined
title: string
tooltip: string
}>()
let width = $state(0)
const inputWidth = 280
</script>

View File

@@ -26,6 +26,7 @@ export interface Setting {
| 'license_key'
| 'object_store_config'
| 'critical_error_channels'
| 'critical_alerts_on_db_oversize'
| 'slack_connect'
| 'smtp_connect'
| 'indexer_rates'
@@ -353,6 +354,15 @@ export const settings: Record<string, Setting[]> = {
fieldType: 'smtp_connect',
storage: 'setting',
ee_only: ''
},
{
label: 'Alert on DB oversize',
key: 'critical_alerts_on_db_oversize',
description: 'Alert if DB grows more than specified size',
fieldType: 'critical_alerts_on_db_oversize',
placeholder: '100',
storage: 'setting',
ee_only: ''
}
],
'OTEL/Prom': [

View File

@@ -1,30 +1,41 @@
<script lang="ts">
// @ts-ignore
import Portal from '$lib/components/Portal.svelte'
import { createFloatingActions } from 'svelte-floating-ui'
import { tick } from 'svelte'
import { offset, flip, shift } from 'svelte-floating-ui/dom'
import MultiSelect from '$lib/components/multiselect/MultiSelect.svelte'
import DarkModeObserver from '../DarkModeObserver.svelte'
import { deepEqual } from 'fast-equals'
let {
items,
value = $bindable(),
placeholder = undefined,
target = undefined,
topPlacement = false,
allowUserOptions = undefined
} = $props<{
items: any[]
value?: string[]
placeholder?: string
target?: string | HTMLElement
topPlacement?: boolean
allowUserOptions?: boolean | 'append'
}>()
$effect.pre(() => { if (value === undefined) value = [] })
export let items: any[]
let propValue: string[] | undefined = []
export { propValue as value }
$: value = structuredClone(propValue)
export let placeholder: string | undefined = undefined
export let target: string | HTMLElement | undefined = undefined
export let topPlacement = false
export let allowUserOptions: boolean | 'append' | undefined = undefined
const [floatingRef, floatingContent] = createFloatingActions({
strategy: 'absolute',
placement: topPlacement ? 'top-start' : 'bottom-start',
middleware: [offset(5), flip(), shift()]
})
let outerDiv: HTMLDivElement | undefined = undefined
let portalRef: HTMLDivElement | undefined = undefined
let outerDiv = $state<HTMLDivElement | undefined>(undefined)
let portalRef = $state<HTMLDivElement | undefined>(undefined)
let darkMode = $state(false)
let w = $state(0)
let open = $state(false)
function moveOptionsToPortal() {
// Find ul element with class 'options' within the outerDiv
const ul = outerDiv?.querySelector('.options')
@@ -34,17 +45,13 @@
}
}
$: if (portalRef && outerDiv && (allowUserOptions || items?.length > 0)) {
tick().then(() => {
moveOptionsToPortal()
})
}
// bg-indigo-100 text-indigo-800 dark:bg-indigo-200 dark:text-indigo-900
let darkMode: boolean = false
let w = 0
let open: boolean = false
$effect(() => {
if (portalRef && outerDiv && (allowUserOptions || items?.length > 0)) {
tick().then(() => {
moveOptionsToPortal()
})
}
})
</script>
<DarkModeObserver bind:darkMode />
@@ -62,10 +69,14 @@
--sms-focus-border={'none'}
--sms-selected-bg={darkMode ? '#c7d2fe' : '#e0e7ff'}
--sms-selected-text-color={darkMode ? '#312e81' : '#3730a3'}
bind:selected={value}
on:change={() => {
propValue = value
}}
bind:selected={
() => [...value],
(newVal) => {
if (!deepEqual(value, newVal)) {
value = newVal
}
}
}
{placeholder}
options={items}
on:close={() => {
@@ -77,15 +88,19 @@
let:option
disableRemoveAll
>
<!-- needed because portal doesn't work for mouseup event en mobile -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<div
class="w-full text-sm"
on:mouseup|stopPropagation
on:pointerdown|stopPropagation={(e) => {
role="option"
tabindex="0"
onmouseup={(e) => {
e.stopPropagation()
}}
onpointerdown={(e) => {
e.stopPropagation()
let newe = new MouseEvent('mouseup')
e.target?.['parentElement']?.dispatchEvent(newe)
}}
aria-selected={value?.includes(option)}
>
{option}
</div>
@@ -93,13 +108,17 @@
</div>
<Portal {target} name="multi-select">
<div use:floatingContent class="z5000" hidden={!open}>
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<!-- svelte-ignore a11y_click_events_have_key_events -->
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div
bind:this={portalRef}
class="multiselect"
style={`min-width: ${w}px;`}
on:click|stopPropagation
onclick={(e) => {
e.stopPropagation()
}}
role="listbox"
tabindex="0"
></div>
</div>
</Portal>

View File

@@ -2,7 +2,7 @@
import TableCustom from '$lib/components/TableCustom.svelte'
import { displayDate, copyToClipboard } from '$lib/utils'
import type { TruncatedToken, NewToken } from '$lib/gen'
import { UserService } from '$lib/gen'
import { IntegrationService, UserService } from '$lib/gen'
import { Button } from '$lib/components/common'
import { Clipboard, Plus } from 'lucide-svelte'
import { workspaceStore, userWorkspaces, type UserWorkspace } from '$lib/stores'
@@ -12,6 +12,7 @@
import Toggle from '../Toggle.svelte'
import ClipboardPanel from '../details/ClipboardPanel.svelte'
import { sendUserToast } from '$lib/toast'
import MultiSelectWrapper from '../multiselect/MultiSelectWrapper.svelte'
// --- Props ---
interface Props {
@@ -37,10 +38,14 @@
let newToken = $state<string | undefined>(undefined)
let newTokenExpiration = $state<number | undefined>(undefined)
let newTokenWorkspace = $state<string | undefined>(defaultNewTokenWorkspace)
let newMcpApps = $state<string[]>([])
let displayCreateToken = $state(scopes != undefined)
let mcpCreationMode = $state(false)
let newMcpScope = $state('favorites')
let newMcpToken = $state<string | undefined>(undefined)
let loadingApps = $state(false)
let errorFetchApps = $state(false)
let allApps = $state<string[]>([])
function ensureCurrentWorkspaceIncluded(
workspacesList: UserWorkspace[],
@@ -67,6 +72,14 @@
listTokens()
})
$effect(() => {
if (mcpCreationMode) {
getAllApps()
} else {
newMcpApps = []
}
})
// --- Functions ---
async function createToken(mcpMode: boolean = false): Promise<void> {
try {
@@ -75,7 +88,13 @@
date = new Date(new Date().getTime() + newTokenExpiration * 1000)
}
let tokenScopes = mcpMode ? [`mcp:${newMcpScope}`] : scopes
let tokenScopes = scopes
if (mcpMode) {
tokenScopes = [`mcp:${newMcpScope}`]
if (newMcpApps.length > 0) {
tokenScopes.push(`mcp:hub:${newMcpApps.join(',')}`)
}
}
const createdToken = await UserService.createToken({
requestBody: {
@@ -107,6 +126,7 @@
newMcpToken = undefined
newToken = undefined
newTokenExpiration = undefined
newMcpApps = []
newTokenLabel = type === 'mcpUrl' ? 'MCP token' : undefined
}
@@ -141,6 +161,26 @@
tokenPage -= 1
listTokens()
}
async function getAllApps() {
if (allApps.length > 0) {
return
}
try {
loadingApps = true
allApps = (
await IntegrationService.listHubIntegrations({
kind: 'script'
})
).map((x) => x.name)
} catch (err) {
console.error('Hub is not available')
allApps = []
errorFetchApps = true
} finally {
loadingApps = false
}
}
</script>
<div class="grid grid-cols-2 pt-8 pb-1" class:pt-8={scopes == undefined}>
@@ -192,66 +232,103 @@
{#if displayCreateToken}
<div class="py-3 px-3 border rounded-md mb-6 bg-surface-secondary min-w-min">
<h3 class="pb-3 font-semibold">Add a new token</h3>
{#if scopes != undefined}
{#each scopes as scope}
<div class="flex flex-col mb-4">
<label for="label">Scope</label>
<input disabled type="text" value={scope} />
</div>
{/each}
{/if}
{#if showMcpMode}
<Toggle
on:change={(e) => {
mcpCreationMode = e.detail
if (e.detail) {
newTokenLabel = 'MCP token'
newTokenExpiration = undefined
newTokenWorkspace = $workspaceStore
} else {
newTokenLabel = undefined
newTokenExpiration = undefined
newTokenWorkspace = defaultNewTokenWorkspace
}
}}
checked={mcpCreationMode}
options={{
right: 'Generate MCP URL',
rightTooltip:
'Generate a new MCP URL to make your scripts and flows available as tools through your LLM clients.',
rightDocumentationLink: 'https://www.windmill.dev/docs/core_concepts/mcp'
}}
class="mb-4"
size="xs"
/>
<div class="mb-4 flex flex-row flex-shrink-0">
<Toggle
on:change={(e) => {
mcpCreationMode = e.detail
if (e.detail) {
newTokenLabel = 'MCP token'
newTokenExpiration = undefined
newTokenWorkspace = $workspaceStore
} else {
newTokenLabel = undefined
newTokenExpiration = undefined
newTokenWorkspace = defaultNewTokenWorkspace
}
}}
checked={mcpCreationMode}
options={{
right: 'Generate MCP URL',
rightTooltip:
'Generate a new MCP URL to make your scripts and flows available as tools through your LLM clients.',
rightDocumentationLink: 'https://www.windmill.dev/docs/core_concepts/mcp'
}}
size="xs"
/>
</div>
{/if}
<div class="flex flex-row flex-wrap gap-2 w-full justify-between">
{#if scopes != undefined}
<div class="mb-4">
<span class="block mb-1">Scope</span>
{#each scopes as scope}
<input disabled type="text" value={scope} class="mb-2 w-full" />
{/each}
</div>
{/if}
<div class="grid grid-cols-1 md:grid-cols-2 gap-4">
{#if mcpCreationMode}
<div class="flex flex-col">
<label for="label">Scope</label>
<div>
<span class="block mb-1">Scope</span>
<ToggleButtonGroup bind:selected={newMcpScope} allowEmpty={false} let:item>
<ToggleButton {item} value="favorites" label="Favorites Only" />
<ToggleButton {item} value="all" label="All Resources" />
<ToggleButton
{item}
value="favorites"
label="Favorites only"
tooltip="Make only your favorite scripts and flows available as tools"
/>
<ToggleButton
{item}
value="all"
label="All scripts/flows"
tooltip="Make all your scripts and flows available as tools"
/>
</ToggleButtonGroup>
</div>
<div class="flex flex-col">
<label for="label">Workspace</label>
<select bind:value={newTokenWorkspace} disabled={workspaces.length === 1}>
<div>
<span class="block mb-1">Hub scripts (optional)</span>
{#if loadingApps}
<div>Loading...</div>
{:else if errorFetchApps}
<div>Error fetching apps</div>
{:else}
<MultiSelectWrapper
items={allApps}
placeholder="Select apps"
bind:value={newMcpApps}
/>
{/if}
</div>
<div>
<span class="block mb-1">Workspace</span>
<select
bind:value={newTokenWorkspace}
disabled={workspaces.length === 1}
class="w-full"
>
{#each workspaces as workspace}
<option value={workspace.id}>{workspace.name}</option>
{/each}
</select>
</div>
{/if}
<div class="flex flex-col">
<label for="label">Label <span class="text-xs text-tertiary">(optional)</span></label>
<input type="text" bind:value={newTokenLabel} />
<div>
<span class="block mb-1">Label <span class="text-xs text-tertiary">(optional)</span></span
>
<input type="text" bind:value={newTokenLabel} class="w-full" />
</div>
<div class="flex flex-col">
<label for="expires"
>Expires In &nbsp;<span class="text-xs text-tertiary">(optional)</span>
</label>
<select bind:value={newTokenExpiration} disabled={mcpCreationMode}>
<div>
<span class="block mb-1"
>Expires In <span class="text-xs text-tertiary">(optional)</span></span
>
<select bind:value={newTokenExpiration} disabled={mcpCreationMode} class="w-full">
<option value={undefined}>No expiration</option>
<option value={15 * 60}>15m</option>
<option value={30 * 60}>30m</option>
@@ -262,15 +339,23 @@
<option value={90 * 24 * 60 * 60}>90d</option>
</select>
</div>
<div class="flex items-end">
<Button
btnClasses="!mt-2"
on:click={() => createToken(mcpCreationMode)}
disabled={mcpCreationMode && newTokenWorkspace === undefined}
>
New token
</Button>
</div>
</div>
<div class="mt-4 flex justify-end gap-2 flex-row">
<Button
on:click={() => {
mcpCreationMode = false
displayCreateToken = false
}}
>
Cancel
</Button>
<Button
on:click={() => createToken(mcpCreationMode)}
disabled={mcpCreationMode && newTokenWorkspace == undefined}
>
New token
</Button>
</div>
</div>
{/if}

View File

@@ -133,7 +133,7 @@
} else {
await goto(rd ?? '/')
}
console.log('Workspace selected, going to', rd)
console.log('Workspace selected going to ' + (rd ? `rd: ${rd}` : 'home'))
} catch (e) {
console.error('Error going to', rd, e)
window.location.reload()

View File

@@ -163,6 +163,7 @@ class TestAgentWorkers(unittest.TestCase):
def test_create_agent_token(self):
token = self._agent_token
print(f"Agent token tests for token: {token}")
self.assertIsNotNone(token)
# JWT tokens have the format: jwt_agent_<prefix>_<token>

View File

@@ -415,5 +415,5 @@ class WindmillClient:
raise Exception(response.content.decode())
token = response.content.decode().strip('"')
print(f"Created agent token: {token[:15]}...{token[-15:]}")
print(f"Created agent token: {token}")
return token

View File

@@ -4,8 +4,8 @@ verify_ssl = true
name = "pypi"
[packages]
wmill = ">=1.485.3"
wmill_pg = ">=1.485.3"
wmill = ">=1.486.1"
wmill_pg = ">=1.486.1"
sendgrid = "*"
mysql-connector-python = "*"
pymongo = "*"

View File

@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.485.3
version: 1.486.1
title: OpenFlow Spec
contact:
name: Ruben Fiszel

View File

@@ -12,7 +12,7 @@
RootModule = 'WindmillClient.psm1'
# Version number of this module.
ModuleVersion = '1.485.3'
ModuleVersion = '1.486.1'
# Supported PSEditions
# CompatiblePSEditions = @()

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "wmill"
version = "1.485.3"
version = "1.486.1"
description = "A client library for accessing Windmill server wrapping the Windmill client API"
license = "Apache-2.0"
homepage = "https://windmill.dev"

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "wmill-pg"
version = "1.485.3"
version = "1.486.1"
description = "An extension client for the wmill client library focused on pg"
license = "Apache-2.0"
homepage = "https://windmill.dev"

View File

@@ -1,6 +1,6 @@
{
"name": "@windmill/windmill",
"version": "1.485.3",
"version": "1.486.1",
"exports": "./src/index.ts",
"publish": {
"exclude": ["!src", "./s3Types.ts", "./client.ts"]

View File

@@ -1,7 +1,7 @@
{
"name": "windmill-client",
"description": "Windmill SDK client for browsers and Node.js",
"version": "1.485.3",
"version": "1.486.1",
"author": "Ruben Fiszel",
"license": "Apache 2.0",
"devDependencies": {

View File

@@ -1 +1 @@
1.485.3
1.486.1