worker-batch-pull-write
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ruben Fiszel
|
0317d5891c |
feat: add powershell common parameters support (#8683)
* feat: add powershell common parameters support (-Verbose, -Debug, -ErrorAction, -WhatIf) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add powershell common params to script editor test panel Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: detect CmdletBinding from code instead of schema in script editor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: ignore commented-out CmdletBinding in powershell detection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use preference variables for -Verbose/-Debug instead of CLI args Verbose/Debug output goes to PowerShell stream 4/5 which isn't captured by the 2>&1 redirect. Setting $VerbosePreference/$DebugPreference in the wrapper scope propagates to child scripts and output flows through the host to stderr, which Windmill captures as logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use *>&1 to capture all powershell streams including verbose/debug The previous 2>&1 only captured error stream. Verbose (stream 4) and debug (stream 5) output was silently lost. Using *>&1 redirects all streams to success stream so they flow through Tee-Object into logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use targeted stream redirects (4>&1 5>&1 2>&1) instead of *>&1 *>&1 breaks $PSCmdlet.ShouldProcess() by redirecting internal streams. Only redirect verbose (4), debug (5), and error (2) to success stream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert to 2>&1 redirect — stream 4/5 redirects break powershell Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use 4>&1 5>&1 for verbose/debug capture, remove WhatIf support Stream 4/5 redirects capture verbose/debug in the pipeline. WhatIf is removed because $PSCmdlet.ShouldProcess() doesn't work when scripts are invoked through Windmill's wrapper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: redirect verbose/debug to files to keep result pipeline clean Verbose (4) and debug (5) streams are redirected to separate log files during script execution, then output via Write-Host after the script completes. This keeps them out of the Tee-Object pipeline (used for result extraction) while still showing them in the job logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: output verbose/debug to stderr via Console.Error for log capture Write-Host goes to stdout which gets mixed with result output and truncated by OSS log threshold. Using [Console]::Error.WriteLine() writes to stderr which Windmill captures separately as logs, with VERBOSE:/DEBUG: prefixes for clarity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: redirect script output to file only, send verbose/debug to stdout The OSS log storage has a 9KB threshold. Previously, Tee-Object sent the full JSON result to both stdout (logs) and the pipe file, eating the log budget. Now script output goes only to the pipe file (> $pipe), and only verbose/debug messages go to stdout for the log viewer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: preserve original Tee-Object behavior, append verbose/debug after Keep the original wrapper behavior (Tee-Object to stdout + pipe file). Only add 4>verbose.log 5>debug.log to capture those streams, and output them at the end of logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: inject preference vars into main.ps1 instead of CLI args Passing -Verbose/-Debug as CLI args causes PowerShell module loading to emit verbose noise. Instead, inject $VerbosePreference/$DebugPreference inside main.ps1's try block so they only affect user code. Stream 4/5 are still redirected to files in the wrapper for log output. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore common param toggles from previous job args on Run Again Extract _wm_ps_* keys from loaded args and initialize the toggle states in PowerShellCommonParams. Also strip them from main args so they don't appear as unknown schema form inputs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: show active common param badges when section is collapsed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: inject ErrorAction as preference variable instead of CLI arg -ErrorAction as a CLI arg only affects the caller, not the script's internal error handling. Setting $ErrorActionPreference inside main.ps1 correctly overrides the default 'Stop' behavior for the user's code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: ensure full backward compatibility with existing powershell scripts - Only filter common param names when [CmdletBinding()] is present (without it, $Verbose etc. are regular user-defined parameters) - Only add 4>verbose.log 5>debug.log and log output lines when common params are actually enabled — original wrapper is unchanged otherwise Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: lighter styling for common params section Replaced heavy Section component with a subtle inline chevron toggle labeled "Common parameters". Smaller text, secondary color, indented options. Badges still show when collapsed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rename section to CmdletBinding parameters Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add ..Default::default() to windmill-parser-r (new parser from main) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: missing comma in graphql parser test + merge main Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add missing commas before ..Default::default() in parser tests Merge from main brought test constructors with formatting issues from the original automated script (missing comma between last field and ..Default::default()). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore comment markers in nu parser test that script broke Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review — ignore commented CmdletBinding, clear stale params 1. Parser: strip comment lines before detecting [CmdletBinding()] to avoid false positives from commented-out attributes 2. RunForm: always assign psCommonParams (not just when non-empty) so stale settings from a previous run don't leak into later runs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
31d6660d56 |
feat: script module mode with CLI sync, preview, and WAC UI improvements (#8380)
* feat: add script module mode with folder model for Bun and Python Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing modules field to RawCode in bun_executor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * sqlx * feat: enrich WAC templates with checkpoint and replay semantics Add prominent comments explaining that all computation must happen inside task/step/taskScript or it will be replayed on resume/retry. Clarify that waitForApproval does not hold a worker and that approve/reject URLs are available in the timeline step details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): script module sync idempotency, per-module hash tracking, and preview support - Fix pull→push idempotency: use `??` instead of `||` for module lock field so empty strings are preserved (matches API's `lock: ""`) - Add per-module hash tracking in wmill-lock.yaml following the flow inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes) - Selective module lock regeneration: only regenerate locks for modules whose content actually changed, not all modules - Use unfiltered rawWorkspaceDependencies for module hashes to match what updateModuleLocks passes to fetchScriptLock - Show changed module names in stale script output for clarity - Add module support to `script preview` command: read modules from __mod/ folder and pass them in the preview API request - Add preview tests for taskScript pattern (flat and folder layout) - Update test assertion for module stale detection output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation - Reorder WAC template buttons: TypeScript before Python in ScriptBuilder, CreateActionsScript, and CreateActionsFlow - Remove dropdown items from +Script button (simplify to direct link) - Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer - Add module tab rename: pencil icon on hover opens popover with validation, fixed-width icon container prevents layout shift Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: remaining module-mode changes from working branch - Backend parser updates for WAC detection - CLI sync/types updates for raw app path and module support - Frontend UI polish (Dev.svelte, ScriptRow, script hash page) - Test fixture updates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(cli): add test for module modification detection in generate-metadata Verifies that modifying a single module file re-triggers stale detection and only the changed module is listed, not all modules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): critical fixes from PR review - Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use "windmill-client" import instead of absolute filesystem path - Fix missed no_main_func → auto_kind rename in parser TS test - Add modules column to clone_script SQL (windmill-common and windmill-api-workspaces) so cloned scripts retain their modules - Add modules: None to RawCode structs in worker tests - Restore complete sqlx cache (merge main's cache + our new queries) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix clone warning treated as error in CI Change `.clone()` on double reference to `*k` dereference in scripts.rs hash implementation. Update sqlx cache with new query hashes from modified clone_script SQL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use published parser wasm versions for CI build The local file:// paths for windmill-parser-wasm-py and windmill-parser-wasm-ts don't exist in the Cloudflare Pages build environment. Revert to published npm versions (1.655.0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): update parser wasm packages to 1.657.2 Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py v1.657.2 which include auto_kind/WAC detection changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): regenerate package-lock.json for npm ci compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use main's lockfile as base, update only parser wasm packages Regenerating package-lock.json from scratch pulled different dependency versions causing svelte-check type errors. Instead, start from main's lockfile and only update the two changed packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): add modules column to fetch_script_for_update query The Script<SR> struct has a modules field (FromRow), but fetch_script_for_update didn't SELECT modules, causing a runtime error "no column found for name: modules" when the worker processed dependency jobs. This was the root cause of the relock_skip test timeout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix script module execution for Python and Bun - Fix modules not passed through job queue: inject _MODULES into PushArgs.extra when pushing Code jobs so worker can extract them - Fix Python module imports: use relative imports (from .helper) and add sys.path.insert for module directory in wrapper - Fix Python tests: use relative imports and empty lock to prevent pip from resolving module names as packages - Add local file check in Bun loader for module resolution - Ignore Bun module test (bundle mode loader integration tracked separately) - Add missing modules column to fetch_script_for_update query Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): remove unnecessary empty lock in Python module tests Relative imports (from .helper) are not parsed as pip packages, so the empty lock workaround is not needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix module execution for Python and Bun — all tests pass Python modules: - Use relative imports (from .helper import greet) since scripts run as packages - Add sys.path.insert for module directory in wrapper to ensure local modules take precedence over pip packages with same name Bun modules: - Use bundled output (./out/main.js) as wrapper import when modules are present — the bundled output has module content inlined by Bun.build, avoiding runtime loader resolution issues - Add local file check in loader.bun.js onResolve to short-circuit API URL resolution for module files on disk Job queue: - Inject _MODULES into PushArgs.extra when pushing Code jobs so the worker can extract them at execution time Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: address PR review — simplify, fix correctness, remove dead code Critical fixes: - Replace all CLI `no_main_func` references with `auto_kind` (string) to match the backend migration and API changes - Remove duplicated `compute_python_module_dir` in worker.rs, use the canonical version from python_executor.rs High priority: - Auto-create `__init__.py` in intermediate directories for nested Python modules so imports like `from .utils.math import add` work without users manually creating __init__.py files - Remove redundant `sys_path_insert` — relative imports use Python's package system, not sys.path Medium: - Fix lock file base name extraction: use regex to strip only the final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")` which breaks for files like `helper.test.ts` Simplification: - Remove dead `{#if false}` Popover block in ScriptEditor.svelte - Guard loader.bun.js local file check to only run for relative paths (matching the Windows loader pattern) - Add clarifying comment on Bun dual mechanism (build + run phases) - Add maintenance comment on manual Hash impl for NewScript Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: final review fixes — stale cleanup, baseName, auto_kind export - Fix sync.ts baseName extraction using indexOf(".") → regex (same fix as script.ts/metadata.ts, missed this instance) - Add stale module file cleanup in writeModulesToDisk: removes files from __mod/ that are no longer in the modules map before writing, fixing the pull→push cycle that couldn't delete modules - Log warning when _MODULES serialization fails in job push instead of silently dropping modules - Use strict equality (===) for auto_kind comparison - Exclude auto_kind from workspace export — it is auto-detected by the parser at deploy time from script content Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): remove auto_kind from push, comparison, and metadata auto_kind is auto-detected by the parser at deploy time, so the CLI should not send it, compare it, or write it to script.yaml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove erroneously added backend/backend/.sqlx directory Duplicate .sqlx cache was committed at the wrong nested path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback + fix CI dead_code warning Frontend (ScriptEditor.svelte): - Fix switchToMain() missing lastSyncedCode update — prevents stale code sync on external changes while editing a module tab - Fix formatAction saving module code to main script's localStorage draft — now saves main code when on a module tab - Fix non-null assertion on inferModuleLang in renameModule — fall back to original language instead of force unwrap - Remove redundant activeModuleTab truthy check in runTest CLI (script.ts): - Clean up empty directories after removing stale module files in writeModulesToDisk Backend: - Add path traversal guard in write_module_files — reject module paths containing ".." - Fix dead_code warning on auto_kind field in workspace export struct Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): improve auto_kind UX + address review findings - Rename "Include without main function" toggle to "Include library scripts" in script list (ItemsList.svelte) - Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip - Filter module file extensions by main script language — Python scripts only allow .py modules, TypeScript only .ts, etc. - Split flushModuleState into flushModuleContent (no UI side-effect) and flushModuleState (flush + reset tab), reducing duplication - Dynamic placeholder and hint text in add module popover based on main script language Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
pyranota
|
11dd4118ce |
feat: add ruby support (#5939)
* implement single line pin * make panic-safe * use pin even if multiple modules imported withing single statement * add repins and make imports respect pins * keep all pins * Allow multiple pins * add comments + handle stuff more safely * fix fully qualified imports * remove ignore * sort nested * apply unique to output requirements list * fix typo * remove mut * update sqlx * sort imports * sort imports * fix formatter and format * feat: add ruby support * fix * update sqlx * add ruby to flake.nix Signed-off-by: pyranota <pyra@duck.com> * upgrade par_install_language_dependencies helper rename fields + allow passing custom payload Signed-off-by: pyranota <pyra@duck.com> * change migrations Signed-off-by: pyranota <pyra@duck.com> * make full pipeline work! Signed-off-by: pyranota <pyra@duck.com> * do a bit fixing * stage Signed-off-by: pyranota <pyra@duck.com> * add ruby in dockerfiles Signed-off-by: pyranota <pyra@duck.com> * implement auto-require + 'windmill/inline' Signed-off-by: pyranota <pyra@duck.com> * create 'windmill/inline' only if does not exist Signed-off-by: pyranota <pyra@duck.com> * add nsjail Signed-off-by: pyranota <pyra@duck.com> * update global settings Signed-off-by: pyranota <pyra@duck.com> * add ruby icon * generate lockfile on deploy (but broken) Signed-off-by: pyranota <pyra@duck.com> * fix(frontend): display deployed script lockfile and other info Signed-off-by: pyranota <pyra@duck.com> * feat: add mini-wmill client Methods: - get_resource - get_variable Additionally add shortcuts on frontend Signed-off-by: pyranota <pyra@duck.com> * add nit warning Signed-off-by: pyranota <pyra@duck.com> * support private repos credentials Signed-off-by: pyranota <pyra@duck.com> * fix compilation error Signed-off-by: pyranota <pyra@duck.com> * found rustc bug + refactor universal installer Signed-off-by: pyranota <pyra@duck.com> * more refactor + fixes Signed-off-by: pyranota <pyra@duck.com> * bug fixing + polishing Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * change 'windmill/inline' Signed-off-by: pyranota <pyra@duck.com> * delegate env_clear to universal installer Signed-off-by: pyranota <pyra@duck.com> * update init script Signed-off-by: pyranota <pyra@duck.com> * add cli support Signed-off-by: pyranota <pyra@duck.com> * unhardcode frontend Signed-off-by: pyranota <pyra@duck.com> * fix cli Signed-off-by: pyranota <pyra@duck.com> * fix init script Signed-off-by: pyranota <pyra@duck.com> * make it work on ms windows Signed-off-by: pyranota <pyra@duck.com> * do not clean env in the end Signed-off-by: pyranota <pyra@duck.com> * fix windows (again) Signed-off-by: pyranota <pyra@duck.com> * include PROXY_ENVS (untested) Signed-off-by: pyranota <pyra@duck.com> * feat: self-signed certs + proxy support (tested) Signed-off-by: pyranota <pyra@duck.com> * flake: pin ruby to 3.4 * merge follow-ups Signed-off-by: pyranota <pyra@duck.com> * remove obsolete libc crate Signed-off-by: pyranota <pyra@duck.com> * cleanup parser Signed-off-by: pyranota <pyra@duck.com> * fix java ci Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fix nits Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-worker/src/universal_pkg_installer.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/ruby_executor.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * add nsjail for lock stage Signed-off-by: pyranota <pyra@duck.com> * fix merge Signed-off-by: pyranota <pyra@duck.com> * Update common.rs * regen windmill-api-client Signed-off-by: pyranota <pyra@duck.com> * fix tests Signed-off-by: pyranota <pyra@duck.com> * fix tests again Signed-off-by: pyranota <pyra@duck.com> * reference published ruby parser Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> |