Files
windmill/backend/windmill-worker/loader.bun.windows.js
hugocasa c28314f424 feat: runner groups for shared-process multi-script dedicated workers (#8434)
* feat: add runner groups for shared-process multi-script dedicated workers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: unify dedicated worker and runner group wrappers into single multi-script wrapper

Replace per-language single-script wrappers with the unified load/exec/exec_preprocess/end
protocol. Each start_worker() now writes scripts to scripts/<safe_name>/ and uses
generate_multi_script_wrapper(). handle_dedicated_process() sends load: on start and
exec: per job instead of raw JSON args.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: merge runner groups into dedicated workers with inline arg metadata

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to match EE branch

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: gate EE-only functions behind cfg(feature = "private") to fix OSS dead_code errors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: auto-detect runner groups from workspace dependency annotations

- New endpoint GET /scripts/list_dedicated_with_deps: returns dedicated
  scripts with parsed workspace dependency names from content annotations
- Frontend: show dep badges in DedicatedWorkersSelector with links to
  workspace settings, warn when referenced dep doesn't exist, group
  scripts sharing deps into "Shared runner" sections
- Remove manual "Runner groups" tab and RunnerGroupSelector component
- Remove runner_groups from WorkerConfigOpt/WorkerConfig (auto-detected)
- Fix Node.js single dedicated workers: transpile main.ts -> main.js via
  Bun.build so the multi-script wrapper's dynamic import() works under Node
- Add package.json with type:module in scripts dir to silence Node warning

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: unify dedicated worker wrappers with baked-in codegen and routing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add e2e tests for multi-script dedicated worker routing (bun, deno, python)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove dead generate_dedicated_worker_wrapper function

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add dependency installation to runner groups + make dep functions pub(crate)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: prevent bun loader from intercepting absolute paths within cwd

When a plugin's onResolve returns an absolute path, Bun re-invokes
the resolver with that path. The loader was then routing it through
the remote URL resolver, breaking runner group script imports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use _wm_ prefix for runner group scripts to avoid bun loader interception

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: extract DENO_UNSTABLE_ARGS constant to avoid repeating flags

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: regenerate system prompts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: gate private-only exports behind cfg(feature = "private") for OSS build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: move format strings before handle_dedicated_process to fix lifetime

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: regenerate sqlx offline cache

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix sqlx

* fix: skip empty lines in deno e2e tests (double newline from console.log + '\n')

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use dict() instead of {{}} in python wrapper to avoid set literal

{{{{}}}} in format!() produces {{}} which Python interprets as an
empty set, not a dict. Use dict() which is unambiguous.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: remove deno from runner groups and associated tests

Deno resolves dependencies at runtime via URLs/import maps, so there's
no shared node_modules/pip install to benefit from runner groups.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: revert deno wrapper to inline old-style with exec: protocol

Since deno doesn't support runner groups, the unified multi-script
wrapper is unnecessary. Reverted to the old inline wrapper from main
but adapted to use the exec:<path>:<args> protocol.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: extract deno wrapper into reusable function and add e2e tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use codebase presence (not nodejs annotation) to determine wrapper import extension

On main, codebase scripts import ./main.js (pre-bundled JS).
The wrapper_ext was incorrectly based on annotation.nodejs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: improve dedicated workers UI - combine lists, better badges, tooltips

- Merge shared runners section with selected tags into one unified list
- Move language tag to right side of selector for alignment
- Change dep badge color from dark-gray to indigo
- Add tooltip on yellow warning badge explaining missing workspace dep

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: group shared runners visually in dedicated workers list

- Runner groups shown with a header (Shared runner · language · dep badge)
- Scripts in the same group nested under the header
- Standalone scripts/flows shown after groups
- Used Svelte snippet for reusable tag row rendering

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: improve visual separation between shared runner groups and standalone items

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: give standalone runners same header style as shared runners

- Each standalone script/flow gets its own header row with bg-surface-secondary
- Header shows "Dedicated runner" / "Flow runner" label, dep link, language badge
- Shared runner header: swapped language and dep badge positions
- Dep shown as inline link instead of badge in headers for cleaner look

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: inline standalone runner path in header, language badge on right edge, no max height

- Standalone items: path shown directly in header row (no sub-row)
- Language badge placed after flex-1 spacer (right-aligned)
- Removed max-h-64 overflow constraint from the list

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: consistent badges across runner list - dep+language on right, depBadge snippet

- Shared runner scripts: show (workspace) and language badge on right
- Standalone items: dep badges and language badge on right (after flex-1)
- Shared runner header: dep badge and language badge on right
- Extract depBadge snippet to deduplicate dep badge rendering
- Picker selector also uses depBadge snippet

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: show language badge on standalone items, hide from shared runner sub-items

- Fetch script language from API when not available from workspace deps
- Hide dep+language badges from tagRow when script is inside a runner group
  (already shown in the group header)
- Standalone items now always show language badge

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: differentiate badge colors - gray for language, indigo for workspace deps

Matches codebase convention: gray for metadata (like script hashes),
indigo for linkable features/entities.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use transparent (bordered) badge for language - visible on all backgrounds

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use gray badge for language everywhere

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert skills.ts and AI files, add _wm_ exclusion to Windows loader

- Revert cli/src/guidance/skills.ts to main (not our change)
- Revert AI provider formatting changes (not our change)
- Add _wm_ prefix exclusion to loader.bun.windows.js filterResolve

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update ee-repo-ref and regenerate system prompts after merge

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: use DISTINCT ON in list_dedicated_with_deps to dedup at DB level

Avoids fetching all script versions and deduplicating in Rust.
Addresses PR review feedback.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use sqlx query! macro for list_dedicated_with_deps and regenerate cache

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: dedicated worker review fixes and test coverage

- Fix Python relative imports in dedicated workers (write loader.py, add
  import loader to wrapper when needed)
- Move Python colon parsing inside try/except to prevent crashes on
  malformed stdin
- Add indexOf guard in Bun/Deno wrappers for malformed protocol messages
- Add stderr logging for unrecognized stdin commands in all wrappers
- Remove asyncio handling from Python wrapper (consistent with normal path)
- Add exec_preprocess protocol tests for Bun, Deno, and Python
- Add argument transformation tests (dates, bytes, kwargs, sentinel)
- Add relative import detection test for Python wrapper
- Add PreprocessedArgs variant to DedicatedWorkerResult test helper

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove symlink from git and gate has_relative_imports behind private feature

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update ee-repo-ref for dedicated_worker_ee.rs changes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add mixed exec+preprocess test to use ProtocolCmd::Exec variant

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove hanging deno missing-preprocessor test

The Deno wrapper only generates the exec_preprocess handler when the
script has a preprocessor function. Without one, the message is
unrecognized and the test hangs reading stdout.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 182943e5ad9bf2a905ccdf07d4e346437fb329a9

This commit updates the EE repository reference after PR #466 was merged in windmill-ee-private.

Previous ee-repo-ref: 995f701fe3754be6260fc6b679e5de8fc636e68a

New ee-repo-ref: 182943e5ad9bf2a905ccdf07d4e346437fb329a9

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-03-25 15:13:04 +00:00

147 lines
5.7 KiB
JavaScript

// Injected by backend: maps normalized paths to temp storage hashes (or null)
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
// Windows-specific bun loader that uses a virtual "windmill-url" namespace instead
// of writing .url files to disk. This avoids Windows path issues (backslashes in
// resolve(), 8.3 short filenames, drive letter prefixes). The virtual namespace
// approach is likely better on all fronts but we keep the original .url-file loader
// on Linux to avoid breaking back-compat.
const p = {
name: "windmill-relative-resolver",
async setup(build) {
const { readFileSync } = await import("fs");
const { resolve } = await import("node:path");
const base_internal_url = "BASE_INTERNAL_URL".replace(
"localhost",
"127.0.0.1"
);
const w_id = "W_ID";
const current_path = "CURRENT_PATH";
const token = "TOKEN";
const cdir = resolve("./");
const cdirNoPrivate = cdir.replace(/^\/private/, ""); // for macos
// Normalize path to forward slashes to match Bun's resolver output on Windows
const cdirFwd = cdir.replace(/\\/g, "/");
const cdirPosix = cdirFwd.replace(/^[a-zA-Z]:/, "");
const filterResolve = new RegExp(
`^(?!\\.\/main\\.ts)(?!\\.\/_wm_)(?!${cdirFwd}\/main\\.ts)(?!${cdirFwd}\/_wm_)(?!${cdirPosix}\/main\\.ts)(?!${cdirPosix}\/_wm_)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$`
);
let cdirNodeModules = `${cdirFwd}/node_modules/`;
const filterLoad = new RegExp(`^${cdir}\/main\\.ts$`);
const transpiler = new Bun.Transpiler({
loader: "ts",
});
function replaceRelativeImports(code) {
const imports = transpiler.scanImports(code);
for (const imp of imports) {
if (imp.kind == "import-statement") {
if (
(imp.path.startsWith(".") ||
imp.path.startsWith("/u/") ||
imp.path.startsWith("/f/")) &&
!imp.path.endsWith(".ts")
) {
code = code.replaceAll(imp.path, imp.path + ".ts");
}
}
}
return {
contents: code,
};
}
function normalizePath(rawPath) {
return rawPath.split("/").reduce((acc, seg) => {
if (seg === "..") acc.pop();
else if (seg !== "." && seg !== "") acc.push(seg);
return acc;
}, []).join("/");
}
// Resolve a windmill script import path relative to an importer path.
// Bun on Windows may prefix args with "windmill-url:" or strip leading "/".
function resolveWindmillImport(importerPath, importPath) {
const path = importPath.replace(/^windmill-url:/, "").replace(/^\//, "");
const isAbsolute = path.startsWith("f/") || path.startsWith("u/");
const endExt = path.endsWith(".ts") ? "" : ".ts";
const rawScriptPath = isAbsolute
? `${path}${endExt}`
: `${importerPath}/../${path}${endExt}`;
const normalized = normalizePath(rawScriptPath);
// Look up temp script hash (keys are extensionless paths)
const lookupPath = normalized.replace(/\.ts$/, "");
const hash = TEMP_SCRIPT_REFS?.[lookupPath];
// Encode hash in the path so onLoad can extract it and append to fetch URL
const resolvedPath = hash ? `${normalized}?temp_script_hash=${hash}` : normalized;
return { path: resolvedPath, namespace: "windmill-url" };
}
build.onLoad({ filter: filterLoad }, async (args) => {
const code = readFileSync(args.path, "utf8");
return replaceRelativeImports(code);
});
// Load windmill scripts by fetching from the API
build.onLoad({ filter: /.*/, namespace: "windmill-url" }, async (args) => {
// Extract temp_script_hash if embedded in the path by resolveWindmillImport
const [scriptPath, queryString] = args.path.replace(/^windmill-url:/, "").split("?");
const hashParam = queryString?.startsWith("temp_script_hash=")
? queryString.replace("temp_script_hash=", "")
: undefined;
const url = `${base_internal_url}/api/w/${w_id}/scripts/RAW_GET_ENDPOINT/p/${scriptPath}`
+ (hashParam ? `?temp_script_hash=${hashParam}` : "");
const req = await fetch(url, {
method: "GET",
headers: {
Authorization: "Bearer " + token,
},
});
if (!req.ok) {
throw new Error(
`Failed to find relative import at ${url} (status ${req.status})`
);
}
const contents = await req.text();
return {
contents: replaceRelativeImports(contents).contents,
loader: "tsx",
};
});
// Resolve windmill script imports from the file namespace (e.g. from main.ts)
build.onResolve({ filter: filterResolve }, (args) => {
const importerFwd = args.importer?.replace(/\\/g, "/") ?? "";
if (importerFwd.startsWith(cdirNodeModules)) {
return undefined;
}
// Check if the import resolves to a local module file (written by write_module_files)
if (args.path.startsWith(".")) {
const cwdPath = resolve(cdir, args.path);
try {
readFileSync(cwdPath);
return { path: cwdPath };
} catch {}
}
const isMainTs =
args.importer == "./main.ts" || importerFwd.endsWith("/main.ts");
const file_path = isMainTs
? current_path
: importerFwd.replace(cdirFwd + "/", "");
return resolveWindmillImport(file_path, args.path);
});
// Resolve nested imports from within windmill-url modules
build.onResolve({ filter: /\.ts$/, namespace: "windmill-url" }, (args) => {
// Strip any query string from the importer path before resolving
const importer = args.importer.replace(/^windmill-url:/, "").split("?")[0];
return resolveWindmillImport(importer, args.path);
});
},
};