kalugin66/windmill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12,840 Commits 513 Branches 1,362 Tags
main
Commit Graph

5 Commits

Author SHA1 Message Date
Ruben Fiszel
7485a5db04 remove dead legacyBehaviour param from metadata functions (#8695) 
The legacyBehaviour parameter on generateFlowLockInternal,
generateAppLocksInternal, and generateScriptMetadataInternal was never
passed as true — the tree parameter alone determines the code path.
Replace `!legacyBehaviour && tree` with just `tree` and remove the
param from all call sites. getRawWorkspaceDependencies keeps its
legacyBehaviour param since it has a real effect there.

Also adds 6 integration tests covering generate-metadata lockfile
generation and idempotency for scripts, flows, and apps.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 12:01:34 +00:00
Pyra
9643006f1e feat(cli): better stale scripts detection #3 (#8480) 
* fix

Signed-off-by: pyranota <pyra@duck.com>

* reduce tests

Signed-off-by: pyranota <pyra@duck.com>

* update

Signed-off-by: pyranota <pyra@duck.com>

* fix

Signed-off-by: pyranota <pyra@duck.com>

* update

Signed-off-by: pyranota <pyra@duck.com>

* WIP: stash changes after merge with origin/main

* Delete backend/parsers/windmill-parser-wasm/Cargo.lock

* reset cargo.toml

* feat(cli): integrate dependency tree into generate-metadata command

- Add isDirectlyStale field to DependencyNode for staleness tracking
- Update addScript to accept itemType, folder, isRawApp, isDirectlyStale
- Update propagateStaleness to use isDirectlyStale field instead of parameter
- Handlers now determine staleness and pass it to tree.addScript
- generate-metadata calls propagateStaleness() and populates staleItems from tree
- Pass legacyBehaviour=false and tree to handlers during generation phase

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): store originalPath in tree for correct handler invocation

Scripts need the path with extension to be passed to the handler.
Added originalPath field to DependencyNode to track this.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix parsers

Signed-off-by: pyranota <pyra@duck.com>

* rever sqlx removal

* update sqlx

* feat: make py-imports parser WASM-compatible and add as separate WASM package

Gate heavy deps (sqlx, windmill-common, async-recursion, toml, pep440_rs,
tracing) behind cfg(not(wasm32)). Make parse_code_for_imports,
parse_relative_imports, NImport, and ImportPin public. Remove duplicate
import_parser from parser-py (reset to origin/main). Add py-imports-parser
feature to windmill-parser-wasm and py-imports target to build.nu.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* safer return

* update

* fix: CLI metadata fixes - folder filter, staleness detection, WASM py-imports setup

- Fix lazy_static cfg gating for WASM compatibility (split into separate blocks)
- Fix folder argument filter to match specific file paths (not just directories)
- Fix staleness detection to use checkHash with conf (includes module hashes)
- Convert relative_imports_skip tests from Deno to bun APIs
- Add windmill-parser-wasm-py-imports to CLI and build-npm dependencies
- Relax module stale test to not require per-module change detail in output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore temp_script_refs parameter in parse_python_imports

Re-adds the temp_script_refs parameter that was lost when resetting
py-imports crate to origin/main. This enables resolving relative imports
from not-yet-deployed scripts during CLI lock generation.

* fixes

* extend testsuit

* update ee repo ref

* fix: diff endpoint bytea cast, upload only mismatched scripts

- Add POST /scripts/raw_temp/diff endpoint to batch-compare local content
  hashes against deployed versions using Postgres sha256()
- Use convert_to(content, 'UTF8') instead of content::bytea to avoid
  failure on scripts containing backslash sequences (e.g. \n)
- CLI now diffs all scripts against deployed, uploads only mismatched ones
- propagateStaleness no longer deletes non-stale nodes (needed for diff)
- Suppress verbose log.info messages during metadata generation
- Add E2E tests for locally modified and unpushed helper scripts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* rework

* sqlx

* fixes

* add index

* expand tests

* fix flows

* archive script before executing

* disable tests for ci

* skip Python-dependent E2E tests on CI

Tests requiring the python backend feature are skipped when
CI_MINIMAL_FEATURES=true since CI builds with zip-only features.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: make flow fixture lock optional and reset nonDottedPaths after tests

Flow fixtures no longer emit an empty lock file by default. The lockContent
parameter controls whether a lock: "!inline ..." line appears in flow.yaml.
This prevents flows from appearing "up-to-date" when they should be processed
by generate-metadata.

Also adds afterAll to reset setNonDottedPaths(false) so global state doesn't
leak between test files when run together.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* debug: add error logging in withTestBackend to diagnose CI failures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* debug: add --bail 1 to CI test runner to show full error on first failure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* debug: include CLI stdout/stderr in assertion message for workspace deps test

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: set WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES in test backend

The workspace deps feature requires workers to report their version, but
in test/CI there are no separate workers (standalone mode). The version
check fails because workers haven't had time to ping yet. Setting this
env var bypasses the version check.

Also reverts --bail 1 from CI workflow now that the root cause is fixed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* debug: add --bail 1 to Windows CI and assertion messages for Windows failure diagnosis

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: replace TEMP_SCRIPT_REFS_PLACEHOLDER in bun builder tests

The loader.bun.js now includes a TEMP_SCRIPT_REFS_PLACEHOLDER that must
be replaced before execution. The builder tests were missing this
replacement, causing all 6 bun_builder_tests to fail.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use cdirFwd in Windows loader filterLoad regex

Raw cdir (with backslashes) interpolated into RegExp causes \r to
become carriage return and \w to become word-char, so filterLoad
never matches main.ts. This prevents replaceRelativeImports from
running, leaving bare relative imports like "./script_b" in the
bundled output, which scanImports then misparses as package ".".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: Windows filterLoad regex + graceful fallback for old backends

- Fix filterLoad in loader.bun.windows.js to match both native backslash
  and forward-slash paths from Bun's resolver by escaping cdir for regex
- Wrap uploadScripts in try/catch so generate-metadata degrades gracefully
  when the backend lacks /raw_temp endpoints (locks use deployed versions)
- Add TODO for missing TEMP_SCRIPT_REFS support in Windows loader

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* debug: add loader/builder debug logging for Windows CI diagnosis

Temporary console.log statements to understand:
- What path Bun passes to onLoad for main.ts
- Whether filterLoad regex matches
- Whether replaceRelativeImports fires
- What the bundled output contains
- What imports scanImports extracts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: trigger CI for cli path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: trigger CI via workflow file change

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add TEMP_SCRIPT_REFS to Windows loader, use .ts extensions in test imports

- Add TEMP_SCRIPT_REFS_PLACEHOLDER support to loader.bun.windows.js
  (mirrors loader.bun.js) so CLI lock generation can resolve imports
  from locally-modified scripts on Windows
- Use .ts extensions in all test relative imports to work around the
  Windows filterLoad regex bug (replaceRelativeImports doesn't fire
  on Windows, so extensionless imports fail)
- Remove unused uploadSucceeded variable

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove debug logging from loader_builder.bun.js

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove windmill-parser-wasm-py-imports from frontend package.json

This dependency is only needed by the CLI, not the frontend.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* debug: add temp_script_refs logging for Windows CI investigation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: remove --bail 1 from Windows CLI tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: normalize backslashes in folder filter treePath lookup (Windows)

On Windows, item.path (originalPath) uses backslashes but tree keys
use forward slashes. The isRelevant filter's touchesFolder call
passed the unnormalized path to traverseTransitive, which couldn't
find the node. This caused cross-folder importers to be excluded
from generate-metadata when a folder argument was specified.

Also removes debug logging from previous commit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update cli-tests.yml

* fix: normalize backslashes in strict-folder-boundaries warning message (Windows)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee-repo-ref to fe8f0d1d7448464c98474d994e6492c0a45e8e38

This commit updates the EE repository reference after PR #467 was merged in windmill-ee-private.

Previous ee-repo-ref: 03e6eaf950776c96b9581848a583af9ad735be60

New ee-repo-ref: fe8f0d1d7448464c98474d994e6492c0a45e8e38

Automated by sync-ee-ref workflow.

* revert cli-tests.yml

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2026-03-23 18:20:19 +00:00
Ruben Fiszel
31d6660d56 feat: script module mode with CLI sync, preview, and WAC UI improvements (#8380) 
* feat: add script module mode with folder model for Bun and Python

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add missing modules field to RawCode in bun_executor

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* feat: enrich WAC templates with checkpoint and replay semantics

Add prominent comments explaining that all computation must happen
inside task/step/taskScript or it will be replayed on resume/retry.
Clarify that waitForApproval does not hold a worker and that
approve/reject URLs are available in the timeline step details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(cli): script module sync idempotency, per-module hash tracking, and preview support

- Fix pull→push idempotency: use `??` instead of `||` for module lock
  field so empty strings are preserved (matches API's `lock: ""`)
- Add per-module hash tracking in wmill-lock.yaml following the flow
  inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes)
- Selective module lock regeneration: only regenerate locks for modules
  whose content actually changed, not all modules
- Use unfiltered rawWorkspaceDependencies for module hashes to match
  what updateModuleLocks passes to fetchScriptLock
- Show changed module names in stale script output for clarity
- Add module support to `script preview` command: read modules from
  __mod/ folder and pass them in the preview API request
- Add preview tests for taskScript pattern (flat and folder layout)
- Update test assertion for module stale detection output

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation

- Reorder WAC template buttons: TypeScript before Python in
  ScriptBuilder, CreateActionsScript, and CreateActionsFlow
- Remove dropdown items from +Script button (simplify to direct link)
- Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer
- Add module tab rename: pencil icon on hover opens popover with
  validation, fixed-width icon container prevents layout shift

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: remaining module-mode changes from working branch

- Backend parser updates for WAC detection
- CLI sync/types updates for raw app path and module support
- Frontend UI polish (Dev.svelte, ScriptRow, script hash page)
- Test fixture updates

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(cli): add test for module modification detection in generate-metadata

Verifies that modifying a single module file re-triggers stale
detection and only the changed module is listed, not all modules.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): critical fixes from PR review

- Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use
  "windmill-client" import instead of absolute filesystem path
- Fix missed no_main_func → auto_kind rename in parser TS test
- Add modules column to clone_script SQL (windmill-common and
  windmill-api-workspaces) so cloned scripts retain their modules
- Add modules: None to RawCode structs in worker tests
- Restore complete sqlx cache (merge main's cache + our new queries)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix clone warning treated as error in CI

Change `.clone()` on double reference to `*k` dereference in
scripts.rs hash implementation. Update sqlx cache with new query
hashes from modified clone_script SQL.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): use published parser wasm versions for CI build

The local file:// paths for windmill-parser-wasm-py and
windmill-parser-wasm-ts don't exist in the Cloudflare Pages build
environment. Revert to published npm versions (1.655.0).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): update parser wasm packages to 1.657.2

Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py
v1.657.2 which include auto_kind/WAC detection changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): regenerate package-lock.json for npm ci compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): use main's lockfile as base, update only parser wasm packages

Regenerating package-lock.json from scratch pulled different dependency
versions causing svelte-check type errors. Instead, start from main's
lockfile and only update the two changed packages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): add modules column to fetch_script_for_update query

The Script<SR> struct has a modules field (FromRow), but
fetch_script_for_update didn't SELECT modules, causing a runtime
error "no column found for name: modules" when the worker processed
dependency jobs. This was the root cause of the relock_skip test
timeout.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix script module execution for Python and Bun

- Fix modules not passed through job queue: inject _MODULES into
  PushArgs.extra when pushing Code jobs so worker can extract them
- Fix Python module imports: use relative imports (from .helper)
  and add sys.path.insert for module directory in wrapper
- Fix Python tests: use relative imports and empty lock to prevent
  pip from resolving module names as packages
- Add local file check in Bun loader for module resolution
- Ignore Bun module test (bundle mode loader integration tracked
  separately)
- Add missing modules column to fetch_script_for_update query

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): remove unnecessary empty lock in Python module tests

Relative imports (from .helper) are not parsed as pip packages,
so the empty lock workaround is not needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix module execution for Python and Bun — all tests pass

Python modules:
- Use relative imports (from .helper import greet) since scripts run
  as packages
- Add sys.path.insert for module directory in wrapper to ensure local
  modules take precedence over pip packages with same name

Bun modules:
- Use bundled output (./out/main.js) as wrapper import when modules
  are present — the bundled output has module content inlined by
  Bun.build, avoiding runtime loader resolution issues
- Add local file check in loader.bun.js onResolve to short-circuit
  API URL resolution for module files on disk

Job queue:
- Inject _MODULES into PushArgs.extra when pushing Code jobs so
  the worker can extract them at execution time

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address PR review — simplify, fix correctness, remove dead code

Critical fixes:
- Replace all CLI `no_main_func` references with `auto_kind` (string)
  to match the backend migration and API changes
- Remove duplicated `compute_python_module_dir` in worker.rs, use
  the canonical version from python_executor.rs

High priority:
- Auto-create `__init__.py` in intermediate directories for nested
  Python modules so imports like `from .utils.math import add` work
  without users manually creating __init__.py files
- Remove redundant `sys_path_insert` — relative imports use Python's
  package system, not sys.path

Medium:
- Fix lock file base name extraction: use regex to strip only the
  final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")`
  which breaks for files like `helper.test.ts`

Simplification:
- Remove dead `{#if false}` Popover block in ScriptEditor.svelte
- Guard loader.bun.js local file check to only run for relative paths
  (matching the Windows loader pattern)
- Add clarifying comment on Bun dual mechanism (build + run phases)
- Add maintenance comment on manual Hash impl for NewScript

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: final review fixes — stale cleanup, baseName, auto_kind export

- Fix sync.ts baseName extraction using indexOf(".") → regex
  (same fix as script.ts/metadata.ts, missed this instance)
- Add stale module file cleanup in writeModulesToDisk: removes files
  from __mod/ that are no longer in the modules map before writing,
  fixing the pull→push cycle that couldn't delete modules
- Log warning when _MODULES serialization fails in job push instead
  of silently dropping modules
- Use strict equality (===) for auto_kind comparison
- Exclude auto_kind from workspace export — it is auto-detected by
  the parser at deploy time from script content

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): remove auto_kind from push, comparison, and metadata

auto_kind is auto-detected by the parser at deploy time, so the CLI
should not send it, compare it, or write it to script.yaml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove erroneously added backend/backend/.sqlx directory

Duplicate .sqlx cache was committed at the wrong nested path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review feedback + fix CI dead_code warning

Frontend (ScriptEditor.svelte):
- Fix switchToMain() missing lastSyncedCode update — prevents stale
  code sync on external changes while editing a module tab
- Fix formatAction saving module code to main script's localStorage
  draft — now saves main code when on a module tab
- Fix non-null assertion on inferModuleLang in renameModule — fall
  back to original language instead of force unwrap
- Remove redundant activeModuleTab truthy check in runTest

CLI (script.ts):
- Clean up empty directories after removing stale module files in
  writeModulesToDisk

Backend:
- Add path traversal guard in write_module_files — reject module
  paths containing ".."
- Fix dead_code warning on auto_kind field in workspace export struct

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(frontend): improve auto_kind UX + address review findings

- Rename "Include without main function" toggle to "Include library
  scripts" in script list (ItemsList.svelte)
- Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip
- Filter module file extensions by main script language — Python
  scripts only allow .py modules, TypeScript only .ts, etc.
- Split flushModuleState into flushModuleContent (no UI side-effect)
  and flushModuleState (flush + reset tab), reducing duplication
- Dynamic placeholder and hint text in add module popover based on
  main script language

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 01:20:09 +00:00
Travis Pew
eb03ebbb04 fix(cli): Fix nonDottedPaths handling in cli flow lock generation (#8375) 
* fix(cli): preserve non-dotted flow lock filenames

* test(cli): add non-dotted path tests for generate-metadata and sync pull

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-15 04:36:04 +00:00
Pyra
4c2c165a5b feat(cli): add unified generate-metadata command (#8335) 
* feat(cli): add unified generate-metadata command

- Add generate-metadata command that calls script, flow, and app handlers
- Export generateLocks from flow.ts and generateMetadata from script.ts
- Add deprecation warnings to individual metadata commands

* feat(cli): improve unified generate-metadata command

- Use internal handlers for single-pass collection of stale items
- Add --dry-run flag to show what would be updated
- Fix WASM parser init deprecation warning
- Add comprehensive tests for all flags
- Match original handler behavior for per-item messages

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(cli): add skip flags and comprehensive tests for generate-metadata

- Add --skip-scripts, --skip-flows, --skip-apps flags for granular control
- --schema-only now properly skips flows and apps (they only have locks)
- Dynamic "Checking X, Y, Z..." message based on what's being processed
- Show warning when all types are skipped
- Add comprehensive tests for all flags:
  - --dry-run shows stale items without updating
  - --schema-only only processes scripts
  - --skip-scripts, --skip-flows, --skip-apps work correctly
  - skipping all types shows warning
  - 'All metadata up-to-date' when nothing to update

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* improve output

Signed-off-by: pyranota <pyra@duck.com>

* refactor(cli): add shared test fixtures with cross-links

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(cli): add folder argument to generate-metadata command

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-13 06:10:28 +00:00