* feat: add runner groups for shared-process multi-script dedicated workers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: unify dedicated worker and runner group wrappers into single multi-script wrapper Replace per-language single-script wrappers with the unified load/exec/exec_preprocess/end protocol. Each start_worker() now writes scripts to scripts/<safe_name>/ and uses generate_multi_script_wrapper(). handle_dedicated_process() sends load: on start and exec: per job instead of raw JSON args. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: merge runner groups into dedicated workers with inline arg metadata Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to match EE branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: gate EE-only functions behind cfg(feature = "private") to fix OSS dead_code errors Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: auto-detect runner groups from workspace dependency annotations - New endpoint GET /scripts/list_dedicated_with_deps: returns dedicated scripts with parsed workspace dependency names from content annotations - Frontend: show dep badges in DedicatedWorkersSelector with links to workspace settings, warn when referenced dep doesn't exist, group scripts sharing deps into "Shared runner" sections - Remove manual "Runner groups" tab and RunnerGroupSelector component - Remove runner_groups from WorkerConfigOpt/WorkerConfig (auto-detected) - Fix Node.js single dedicated workers: transpile main.ts -> main.js via Bun.build so the multi-script wrapper's dynamic import() works under Node - Add package.json with type:module in scripts dir to silence Node warning Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: unify dedicated worker wrappers with baked-in codegen and routing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add e2e tests for multi-script dedicated worker routing (bun, deno, python) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove dead generate_dedicated_worker_wrapper function Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dependency installation to runner groups + make dep functions pub(crate) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: prevent bun loader from intercepting absolute paths within cwd When a plugin's onResolve returns an absolute path, Bun re-invokes the resolver with that path. The loader was then routing it through the remote URL resolver, breaking runner group script imports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use _wm_ prefix for runner group scripts to avoid bun loader interception Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: extract DENO_UNSTABLE_ARGS constant to avoid repeating flags Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: regenerate system prompts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: gate private-only exports behind cfg(feature = "private") for OSS build Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move format strings before handle_dedicated_process to fix lifetime Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: regenerate sqlx offline cache Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix sqlx * fix: skip empty lines in deno e2e tests (double newline from console.log + '\n') Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use dict() instead of {{}} in python wrapper to avoid set literal {{{{}}}} in format!() produces {{}} which Python interprets as an empty set, not a dict. Use dict() which is unambiguous. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: remove deno from runner groups and associated tests Deno resolves dependencies at runtime via URLs/import maps, so there's no shared node_modules/pip install to benefit from runner groups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: revert deno wrapper to inline old-style with exec: protocol Since deno doesn't support runner groups, the unified multi-script wrapper is unnecessary. Reverted to the old inline wrapper from main but adapted to use the exec:<path>:<args> protocol. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: extract deno wrapper into reusable function and add e2e tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use codebase presence (not nodejs annotation) to determine wrapper import extension On main, codebase scripts import ./main.js (pre-bundled JS). The wrapper_ext was incorrectly based on annotation.nodejs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: improve dedicated workers UI - combine lists, better badges, tooltips - Merge shared runners section with selected tags into one unified list - Move language tag to right side of selector for alignment - Change dep badge color from dark-gray to indigo - Add tooltip on yellow warning badge explaining missing workspace dep Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: group shared runners visually in dedicated workers list - Runner groups shown with a header (Shared runner · language · dep badge) - Scripts in the same group nested under the header - Standalone scripts/flows shown after groups - Used Svelte snippet for reusable tag row rendering Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: improve visual separation between shared runner groups and standalone items Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: give standalone runners same header style as shared runners - Each standalone script/flow gets its own header row with bg-surface-secondary - Header shows "Dedicated runner" / "Flow runner" label, dep link, language badge - Shared runner header: swapped language and dep badge positions - Dep shown as inline link instead of badge in headers for cleaner look Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: inline standalone runner path in header, language badge on right edge, no max height - Standalone items: path shown directly in header row (no sub-row) - Language badge placed after flex-1 spacer (right-aligned) - Removed max-h-64 overflow constraint from the list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: consistent badges across runner list - dep+language on right, depBadge snippet - Shared runner scripts: show (workspace) and language badge on right - Standalone items: dep badges and language badge on right (after flex-1) - Shared runner header: dep badge and language badge on right - Extract depBadge snippet to deduplicate dep badge rendering - Picker selector also uses depBadge snippet Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: show language badge on standalone items, hide from shared runner sub-items - Fetch script language from API when not available from workspace deps - Hide dep+language badges from tagRow when script is inside a runner group (already shown in the group header) - Standalone items now always show language badge Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: differentiate badge colors - gray for language, indigo for workspace deps Matches codebase convention: gray for metadata (like script hashes), indigo for linkable features/entities. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use transparent (bordered) badge for language - visible on all backgrounds Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use gray badge for language everywhere Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert skills.ts and AI files, add _wm_ exclusion to Windows loader - Revert cli/src/guidance/skills.ts to main (not our change) - Revert AI provider formatting changes (not our change) - Add _wm_ prefix exclusion to loader.bun.windows.js filterResolve Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update ee-repo-ref and regenerate system prompts after merge Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: use DISTINCT ON in list_dedicated_with_deps to dedup at DB level Avoids fetching all script versions and deduplicating in Rust. Addresses PR review feedback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use sqlx query! macro for list_dedicated_with_deps and regenerate cache Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: dedicated worker review fixes and test coverage - Fix Python relative imports in dedicated workers (write loader.py, add import loader to wrapper when needed) - Move Python colon parsing inside try/except to prevent crashes on malformed stdin - Add indexOf guard in Bun/Deno wrappers for malformed protocol messages - Add stderr logging for unrecognized stdin commands in all wrappers - Remove asyncio handling from Python wrapper (consistent with normal path) - Add exec_preprocess protocol tests for Bun, Deno, and Python - Add argument transformation tests (dates, bytes, kwargs, sentinel) - Add relative import detection test for Python wrapper - Add PreprocessedArgs variant to DedicatedWorkerResult test helper Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove symlink from git and gate has_relative_imports behind private feature Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update ee-repo-ref for dedicated_worker_ee.rs changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add mixed exec+preprocess test to use ProtocolCmd::Exec variant Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove hanging deno missing-preprocessor test The Deno wrapper only generates the exec_preprocess handler when the script has a preprocessor function. Without one, the message is unrecognized and the test hangs reading stdout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 182943e5ad9bf2a905ccdf07d4e346437fb329a9 This commit updates the EE repository reference after PR #466 was merged in windmill-ee-private. Previous ee-repo-ref: 995f701fe3754be6260fc6b679e5de8fc636e68a New ee-repo-ref: 182943e5ad9bf2a905ccdf07d4e346437fb329a9 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
123 lines
4.1 KiB
JavaScript
123 lines
4.1 KiB
JavaScript
// Injected by backend: maps normalized paths to temp storage hashes (or null)
|
|
const TEMP_SCRIPT_REFS = TEMP_SCRIPT_REFS_PLACEHOLDER;
|
|
|
|
const p = {
|
|
name: "windmill-relative-resolver",
|
|
async setup(build) {
|
|
const { writeFileSync, readFileSync, mkdirSync } = await import("fs");
|
|
const { dirname, resolve, join } = await import("node:path");
|
|
|
|
const base_internal_url = "BASE_INTERNAL_URL".replace(
|
|
"localhost",
|
|
"127.0.0.1"
|
|
);
|
|
|
|
const w_id = "W_ID";
|
|
const current_path = "CURRENT_PATH";
|
|
const token = "TOKEN";
|
|
|
|
const cdir = resolve("./");
|
|
const cdirNoPrivate = cdir.replace(/^\/private/, ""); // for macos
|
|
// On Windows, normalize path to POSIX format to match args.path from Bun's resolver
|
|
const cdirPosix = cdir.replace(/\\/g, "/").replace(/^[a-zA-Z]:/, "");
|
|
const filterResolve = new RegExp(
|
|
`^(?!\\.\/main\\.ts)(?!\\.\/_wm_)(?!${cdir}\/main\\.ts)(?!${cdir}\/_wm_)(?!${cdirPosix}\/main\\.ts)(?!${cdirPosix}\/_wm_)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$`
|
|
);
|
|
|
|
let cdirNodeModules = `${cdir}/node_modules/`;
|
|
|
|
const filterLoad = new RegExp(`^${cdir}\/main\\.ts$`);
|
|
const transpiler = new Bun.Transpiler({
|
|
loader: "ts",
|
|
});
|
|
|
|
function replaceRelativeImports(code) {
|
|
const imports = transpiler.scanImports(code);
|
|
for (const imp of imports) {
|
|
if (imp.kind == "import-statement") {
|
|
if (
|
|
(imp.path.startsWith(".") ||
|
|
imp.path.startsWith("/u/") ||
|
|
imp.path.startsWith("/f/")) &&
|
|
!imp.path.endsWith(".ts")
|
|
) {
|
|
code = code.replaceAll(imp.path, imp.path + ".ts");
|
|
}
|
|
}
|
|
}
|
|
return {
|
|
contents: code,
|
|
};
|
|
}
|
|
|
|
build.onLoad({ filter: filterLoad }, async (args) => {
|
|
const code = readFileSync(args.path, "utf8");
|
|
return replaceRelativeImports(code);
|
|
});
|
|
|
|
build.onLoad({ filter: /.*\.url$/ }, async (args) => {
|
|
const url = readFileSync(args.path, "utf8");
|
|
const req = await fetch(url, {
|
|
method: "GET",
|
|
headers: {
|
|
Authorization: "Bearer " + token,
|
|
},
|
|
});
|
|
if (!req.ok) {
|
|
throw new Error(
|
|
`Failed to find relative import at ${url}`,
|
|
req.statusText
|
|
);
|
|
}
|
|
const contents = await req.text();
|
|
return {
|
|
contents: replaceRelativeImports(contents).contents,
|
|
loader: "tsx",
|
|
};
|
|
});
|
|
|
|
build.onResolve({ filter: filterResolve }, (args) => {
|
|
if (args.importer?.startsWith(cdirNodeModules)) {
|
|
return undefined;
|
|
}
|
|
|
|
// Check if the import resolves to a local module file (written by write_module_files).
|
|
// Only check relative paths — absolute/bare specifiers should fall through to the
|
|
// remote resolver, matching the Windows loader pattern.
|
|
if (args.path.startsWith(".")) {
|
|
const localPath = resolve(cdir, args.path);
|
|
try {
|
|
readFileSync(localPath);
|
|
return { path: localPath };
|
|
} catch {}
|
|
}
|
|
|
|
const file_path =
|
|
args.importer == "./main.ts" || args.importer == resolve("./main.ts")
|
|
? current_path
|
|
: args.importer.replace(cdir + "/", "");
|
|
|
|
const isRelative = !args.path.startsWith("/");
|
|
const endExt = args.path.endsWith(".ts") ? "" : ".ts";
|
|
const pathNoExt = args.path.replace(/\.ts$/, "");
|
|
|
|
// Lookup temp script hash
|
|
const normalized = (isRelative ? join(dirname(file_path), pathNoExt) : pathNoExt.slice(1)).replace(/\\/g, "/");
|
|
const hash = TEMP_SCRIPT_REFS?.[normalized];
|
|
|
|
const url = (isRelative
|
|
? `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${file_path}/../${args.path}${endExt}`
|
|
: `${base_internal_url}/api/w/${w_id}/scripts/raw_unpinned/p/${args.path}${endExt}`
|
|
) + (hash ? `?temp_script_hash=${hash}` : "");
|
|
const file = isRelative
|
|
? resolve("./" + file_path + "/../" + args.path + ".url")
|
|
: resolve("./" + args.path + ".url");
|
|
mkdirSync(dirname(file), { recursive: true });
|
|
writeFileSync(file, url);
|
|
return {
|
|
path: file,
|
|
};
|
|
});
|
|
},
|
|
};
|