Commit Graph

248 Commits

Author SHA1 Message Date
Ruben Fiszel
5f0ef936d1 feat: add sandbox annotations, volume mounts, for AI sandbox starting with claude (#8058) 2026-03-05 06:19:51 +00:00
hugocasa
a111653c6d fix: don't insert underscore after digit in PascalCase to snake_case conversion (#8184)
* fix: don't insert underscore after digit in PascalCase to snake_case conversion (#7934)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* update parsers

* remove unused wasms + fix build

* update cli lock

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-02 16:02:48 +00:00
Diego Imbert
762fd3d993 Fix python datatable client requiring explicit types (#8086)
* Support arg type decl in postgres

* Python datatable client no longer requires explicit arg typing

* compilation fix

* Set correct type in statement exec

* reset to main

* Explicit pg arg types

* remove code duplication

* update parser js

* FLOAT8 doesn't have space

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-02-28 07:08:02 +00:00
Diego Imbert
095505136c fix: Handle CTEs and local tables in SQL asset parser (#8131)
* Handle CTEs and local tables in SQL asset parser

* also handle CREATE VIEW

* Update package regex version
2026-02-28 07:04:19 +00:00
Diego Imbert
87f3de9ae5 feat: Support column detection on S3 objects in DuckDB (#8018)
* Support column detection on S3 objects in DuckDB

* Compilation fix

* support direct s3 path without read_parquet()

* package update

* npm i
2026-02-20 20:34:53 +00:00
Ruben Fiszel
6bf544f507 refactor: extract object store into dedicated crate with filesystem backend (#7996)
* refactor: extract object store code into windmill-object-store crate with filesystem backend

Consolidate all object_store-dependent code from windmill-common into a new
windmill-object-store crate. Add a filesystem-backed object store implementation
using LocalFileSystem for dev/testing without cloud credentials. Includes 30
comprehensive tests covering render_endpoint, lfs_to_object_store_resource,
duckdb_connection_settings, error mapping, and filesystem-backed integration tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* all

* all

* all

* fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 12:24:45 +00:00
Pyra
cd06bd25e1 nit(python): add mapping for azure-identity #7922 2026-02-12 10:07:21 +00:00
Luigi
8e19f9652d Added serpapi import handling (#7871)
Added the handling for serpapi import. The name of the Python library to be installed is google-search-results.

Source: https://pypi.org/project/google-search-results/
2026-02-10 09:59:35 +00:00
Ruben Fiszel
ff70a4e9d1 fix: parse Python datetime.datetime and datetime.date type annotations (#7856)
* fix: parse Python datetime.datetime and datetime.date type annotations correctly

The Python parser only matched ExprKind::Name for type annotations, so
`datetime.datetime` (an Attribute expression) silently fell through to
Typ::Unknown and no datetime picker was shown in the UI.

- Extend parse_expr to resolve `datetime.*` attribute access (alongside
  the existing `wmill.*` handling)
- Add Typ::Date variant for `datetime.date` → JSON schema format "date"
- Update python worker to import and convert `date.fromisoformat()`
- Update argSigToJsonSchemaType, AI types, schema validation, and SQL
  datatype wasm for the new Date variant

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* all

* all

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 15:17:26 +00:00
Diego Imbert
3476ef4b9c fix: fix DB Manager not working with db resources with 4+ path segments (#7809)
* support more than 3 path segments

* Fix explore db resource not working with 4+ path segments

* don't assume 3 segments

* ?table= syntax impl

* update parsers

* more nit fixes

* fix sql query

* claude nit

* Update SQLx metadata

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-02-05 14:00:15 +00:00
Diego Imbert
c9ee675de8 Fix asset parser integration test (#7794)
* fix ci

* Fix TS asset parser CI
2026-02-04 18:22:04 +00:00
Diego Imbert
c760a33ccd fix ci (#7790) 2026-02-04 16:44:06 +00:00
Diego Imbert
0caa533fbd feat: column-level asset tracking for ducklake and datatables (#7774)
* Refactor 1

* claude tmp1

* fixes1

* support for insert and update

* Fix returning

* docs: add nuanced mutex selection guidance to Rust backend skill (#7737)

Add "Mutex Selection in Async Code" section explaining when to use
std::sync::Mutex vs tokio::sync::Mutex based on official Tokio docs.
std::sync::Mutex is preferred for data protection as it's faster;
tokio::sync::Mutex only needed when holding locks across .await points.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): revert findCodebase change that broke ../shared codebases (#7740)

* fix(cli): revert findCodebase relative_path check that broke ../shared codebases

The previous change added a check to ensure script paths start with the
codebase's relative_path. However, this broke cases where relative_path
uses parent directory references (e.g., "../shared") because:

1. path.join normalizes paths, so "/project/../shared/f/script.ts" becomes
   "/shared/f/script.ts"
2. FSFSElement strips the cwd prefix, resulting in "f/script.ts"
3. The check "f/script.ts".startsWith("../shared/") failed

The original behavior was correct - relative_path indicates where to find
codebase files, while includes/excludes patterns match against the normalized
paths that get passed during sync.

Fixes regression reported in #7729 comments.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(cli): add preview test for codebase with imports

Tests that codebase bundling correctly includes imported modules,
which is the key functionality needed for ../shared codebases.
The test creates a helper module and a main script that imports
from it, then verifies the bundled script executes correctly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* chore(main): release 1.621.2 (#7735)

* chore(main): release 1.621.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* feat: workspace dedicated workers (#7741)

* feat: workspace dedicated workers

* ref

* chore: update ee-repo-ref to a18ac31062ac092cb9a5fc87629e217d97f4911d

This commit updates the EE repository reference after PR #398 was merged in windmill-ee-private.

Previous ee-repo-ref: 98cfe3fef764d9d815d326d5056c734a03689d33

New ee-repo-ref: a18ac31062ac092cb9a5fc87629e217d97f4911d

Automated by sync-ee-ref workflow.

* fix(frontend): workspace script in flow steps

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* fix: remove uuid-ossp extension requirement for RDS compatibility

The uuid-ossp extension was created in the first migration but never
actually used - the codebase uses gen_random_uuid() which is built-in
to PostgreSQL 13+. This allows Windmill to run on AWS RDS where
application users may not have CREATE SCHEMA privileges.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: indexer build error (#7744)

* fix: indexer build error

* prepare sqlx

* Remove changes from Cargo.toml

* refactor: remove seed parameter from AI chat completions (#7745)

* better claude

* refactor: remove seed parameter from AI chat completions

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* fix: visibility bug on deployment UI (issue when renaming items) + add tracking of folders and resource types (#7739)

* fix: Raw apps deployment UI (and merge UI)

* Add folders and resource tpyes to merge UI

* claude first pass on adding the new arg for h_deploy_metadata

* Add missing argument to handle_deployment_metadata in all its calls

* Add support for folders and resource types in merge UI

* Update eereporef for CI

* Update ee repo

* Add migration to reset cached diff with potential artifacts

* fix type in frontend

* Preapare sqlx

* Remove unused import and logs

* update ee-repo

* Update eerepo

* chore: update ee-repo-ref to aca38475afd2cafaf63f4bbffc65be9437d57d86

This commit updates the EE repository reference after PR #397 was merged in windmill-ee-private.

Previous ee-repo-ref: 19c64cf8c61d83f45047b37660054b29658cd403

New ee-repo-ref: aca38475afd2cafaf63f4bbffc65be9437d57d86

Automated by sync-ee-ref workflow.

* Make integration  test for workspace comparisons

* Update SQLx metadata

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* fix: require AGENT_TOKEN and BASE_INTERNAL_URL for agent mode

- Add AgentConfig struct to validate required env vars on startup
- Change build_agent_http_client to require explicit token and URL
- Remove DEFAULT_BASE_INTERNAL_URL fallback (no more silent localhost:8000)
- Exit immediately if agent cannot connect to server on initial load
- Update integration tests to use dynamic port for BASE_INTERNAL_URL

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: forward teams error to client (#7746)

* fix: forward teams error to client

* chore: update ee-repo-ref to 9a3d71f2c6a41ed4d17111a8c05d8e1d4933898d

This commit updates the EE repository reference after PR #400 was merged in windmill-ee-private.

Previous ee-repo-ref: 25d35a8de1cd70e281dc876e51cd30402580b5c0

New ee-repo-ref: 9a3d71f2c6a41ed4d17111a8c05d8e1d4933898d

Automated by sync-ee-ref workflow.

* fix

* fix

* fix

* al

* sqlx

* sqlx

* all

* all

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

* feat: add token usage tracking to AI agent output (#7738)

* feat: add token usage tracking to AI agent output

Add TokenUsage struct to track input/output/cache tokens from AI providers.
Currently implemented for Bedrock provider, with infrastructure in place
for other providers. Usage is included in the AI agent result alongside
output and messages when available.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add token usage extraction for Anthropic provider

Extract usage from message_delta SSE event and convert to TokenUsage.
Includes input_tokens, output_tokens, cache_read_input_tokens, and
cache_creation_input_tokens (mapped to cache_write_input_tokens).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add token usage extraction for Google AI/Gemini provider

Extract usage from usageMetadata in Gemini SSE events and convert to TokenUsage.
Maps promptTokenCount -> input_tokens, candidatesTokenCount -> output_tokens,
totalTokenCount -> total_tokens.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add token usage extraction for OpenAI Responses API provider

Extract usage from response.completed SSE event and convert to TokenUsage.
Maps input_tokens, output_tokens, and total_tokens directly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add token usage extraction for Azure OpenAI / Chat Completions API

Add stream_options.include_usage to request and parse usage from final
SSE chunk for providers using the standard OpenAI Chat Completions API
(Azure OpenAI, Mistral, DeepSeek, Groq, TogetherAI, CustomAI).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: cleanup token usage tracking - remove unused Image usage field and accumulate across iterations

- Remove unused `usage` field from ParsedResponse::Image variant
- Add TokenUsage::accumulate() method to sum usage across agent iterations
- Accumulate input/output/total/cache tokens instead of replacing with last iteration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: remove verbose debug logging from AI providers

Remove tracing::info!("[debug] ...") statements that were too verbose
for production. These logged raw events on every streaming event.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add retry mechanism for OpenAI-compatible providers without stream_options support

Some OpenAI-compatible providers don't support the stream_options parameter
for usage tracking. This adds a retry mechanism that:
- First attempts the request with stream_options.include_usage
- If it fails with 400 and error mentions stream_options/include_usage,
  automatically retries without the parameter

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: remove unnecessary text parsing overhead in image response handlers

Revert debugging changes that read response as text before parsing JSON.
Using response.json() directly is more efficient.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: centralize TokenUsage conversion with constructor methods

Add new(), from_input_output(), and with_cache() constructors to TokenUsage
to eliminate duplicate conversion logic across providers. Also fixes potential
truncation in Bedrock cache token conversion by using i32::try_from with
fallback to i32::MAX.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: simplify Anthropic usage extraction and add Default derive

- Use idiomatic `if let` pattern instead of `is_some()` check for usage extraction
- Add Default derive to OpenAIChatUsage for consistency with other usage structs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use saturating_add to prevent overflow in token accumulation

In long-running agents with many iterations, token counts could
potentially overflow. Using saturating_add ensures values cap at
i32::MAX instead of wrapping around.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* better claude

* nit

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* chore(main): release 1.622.0 (#7742)

* chore(main): release 1.622.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* fix npm check

* fix: add schema compatibility layer for MCP clients like n8n (#7747)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* nit ui nextcloud triggers (#7749)

* feat: add PocketID OAuth provider support (#7318)

* feat(oauth): add Pocket-ID OAuth provider component

- Implements PocketIdSetting.svelte following Keycloak pattern
- Configures OIDC endpoints for Pocket-ID (/authorize, /api/oidc/*)
- Supports standard OIDC scopes (openid, profile, email)
- Uses passkey-only authentication via Pocket-ID

Refs #5678

* feat(oauth): register Pocket-ID in SSO provider list

- Import PocketIdSetting component
- Add Pocket-ID to provider list in SSO tab
- Update exclusion filter to prevent duplicate custom entries

Refs #5678

* fix(oauth): add missing PocketID icon and fix component integration

- Create PocketIdIcon.svelte component with user profile icon
- Register pocket-id in APP_TO_ICON_COMPONENT mapping
- Fix PocketIdSetting to use IconedResourceType pattern matching other OAuth providers

This resolves the issue where PocketID toggle was not appearing in SSO settings.

Refs #5678

* refactor: migrate PocketIdSetting to Svelte 5 runes syntax

- Use $props() with $bindable() for reactive prop binding
- Use $state() for local reactive state
- Use $derived() for computed values
- Use $effect() for reactive side effects
- Replace on:change with onchange event handler
- Pre-populate base URL from existing config when editing
- Clean up bracket notation to dot notation for value properties

Addresses reviewer feedback

* fix: rename pocket-id to pocketid for naming convention compliance

Change identifier from 'pocket-id' to 'pocketid' to match Windmill's naming convention.
No OAuth provider uses hyphens - all custom SSO providers (keycloak, authentik, authelia,
kanidm, zitadel) use no separator.

Changes:
- AuthSettings.svelte: oauths['pocket-id'] → oauths['pocketid'] (2 locations)
- PocketIdSetting.svelte: name={'pocket-id'} → name={'pocketid'}
- icons/index.ts: 'pocket-id': PocketIdIcon → pocketid: PocketIdIcon

Note: PocketID does not need oauth_connect.json entry as it's a custom SSO provider
with user-configured endpoints, similar to Keycloak/Authentik.

Addresses reviewer feedback

* fix: use TextInput component for consistency

---------

Co-authored-by: hugocasa <hugo@casademont.ch>

* fix: preserve script envs field during sync push

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* nit frontend fix

* chore(main): release 1.623.0 (#7748)

* chore(main): release 1.623.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* fix: resolve infinite effect loop in PocketIdSetting component (#7753)

* fix: prevent retention cleanup from deleting jobs of active flows (#7755)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* chore(main): release 1.623.1 (#7754)

* chore(main): release 1.623.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>

* feat: default to quickjs on ce for flow eval (#7756)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* feat: runtime assets (#7656)

* Runtime assets

* Nits

* Revert "Nits"

This reverts commit 3031a2ddd1.

* detection_kinds

* don't delete runtime assets

* Show latest executions

* conditional unique idx

* nit status

* refactor

* nit refactor

* prepare sql

* Detect assets in complex JSON input objects

* false positive prevent

* nit

* redundant idx

* Update frontend/src/lib/components/assets/AssetsUsageDrawer.svelte

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Update backend/migrations/20260122134517_runtime_assets.up.sql

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* runtime assets are inserted in a loop

* nit

* nit fix

* Don't use lazy static

* fix compilation

* nits

* missing on conflict do nothing

* add index

* Fix max n logic

* created at

* nits

* remove pagination

* sqlx prepare

* Only detect resource assets in input

* get_runtime_asset_sender()

* use global get_runtime_asset_sender to avoid prop drilling

* nit refactor : register_runtime_asset

* get job_id from token

* job as a usage kind

* fixes

* ee

* nit refactor

* merge access types when same job uses same asset multiple times

* Refactor to support wmill s3 API

* nit

* parse_wmill_sdk_sql_assets refactor

* Detect datatable and ducklake usage

* nit order by

* Join with v2_job

* better UI

* add sequential id for cursor pagination

* useInfiniteQuery

* useScrollToBottom

* sql index

* claude code stash

* migration fixes

* Infinite scroll UI

* nit

* style nit

* runtime asset created at

* Asset filters

* fix usage kind filter

* also check runnable_path for jobs when filtering

* better filters

* avoid flickering

* debounced filters

* nit

* tooltips

* fix: update AssetUsage type to match new ListAssetsResponse structure

The ListAssetsResponse changed from an array to an object with an 'assets' property.
Updated the type extraction accordingly.

Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>

* sqlx prepare

* Delete .claude/hooks/.symlink-manifest

* unnecessary dep

* nit refactor

* nit comment

* nit naming

* CI fix attempt 1

* ee ref

* nit remove alerts

* nit

* chore: update ee-repo-ref to 138a4f5f868f3bded5bb7cb77b222b532c07e4af

This commit updates the EE repository reference after PR #395 was merged in windmill-ee-private.

Previous ee-repo-ref: 7d3a21d53066726e97dfea9f117373299bc9318c

New ee-repo-ref: 138a4f5f868f3bded5bb7cb77b222b532c07e4af

Automated by sync-ee-ref workflow.

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* fix: remove $schema field from Google AI output schema requests (#7765)

* fix: remove $schema field from Google AI output schema requests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: add $schema field to all output schema integration tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: remove $schema field from Google AI tool parameter schemas

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: add workspace script tool test for AI agents

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* fix: Fix app multiselect not refreshing result when creating element (#7766)

* Fix returning

* asset columns are saved and displayed in the assets page

* runtime assets column detectionz

* frontend nits

* update regex parsers

* UI nits

* Display asset columns in flow graph

* Column hint directly in asset node

* nit bg

* sqlx prepare

* ee repo ref

* chore: update ee-repo-ref to 66a68df97e8c65c498b28f302a365ab8687cad9e

This commit updates the EE repository reference after PR #402 was merged in windmill-ee-private.

Previous ee-repo-ref: 0a32bc104cbaec9664a4d7cb1565823722c875a1

New ee-repo-ref: 66a68df97e8c65c498b28f302a365ab8687cad9e

Automated by sync-ee-ref workflow.

---------

Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
Co-authored-by: hugocasa <hugo@casademont.ch>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: wendrul <53628737+wendrul@users.noreply.github.com>
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>
Co-authored-by: Devdatta Talele <50290838+devdattatalele@users.noreply.github.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>
2026-02-04 16:27:53 +00:00
Diego Imbert
635a24f82c feat: runtime assets (#7656)
* Runtime assets

* Nits

* Revert "Nits"

This reverts commit 3031a2ddd1.

* detection_kinds

* don't delete runtime assets

* Show latest executions

* conditional unique idx

* nit status

* refactor

* nit refactor

* prepare sql

* Detect assets in complex JSON input objects

* false positive prevent

* nit

* redundant idx

* Update frontend/src/lib/components/assets/AssetsUsageDrawer.svelte

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Update backend/migrations/20260122134517_runtime_assets.up.sql

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* runtime assets are inserted in a loop

* nit

* nit fix

* Don't use lazy static

* fix compilation

* nits

* missing on conflict do nothing

* add index

* Fix max n logic

* created at

* nits

* remove pagination

* sqlx prepare

* Only detect resource assets in input

* get_runtime_asset_sender()

* use global get_runtime_asset_sender to avoid prop drilling

* nit refactor : register_runtime_asset

* get job_id from token

* job as a usage kind

* fixes

* ee

* nit refactor

* merge access types when same job uses same asset multiple times

* Refactor to support wmill s3 API

* nit

* parse_wmill_sdk_sql_assets refactor

* Detect datatable and ducklake usage

* nit order by

* Join with v2_job

* better UI

* add sequential id for cursor pagination

* useInfiniteQuery

* useScrollToBottom

* sql index

* claude code stash

* migration fixes

* Infinite scroll UI

* nit

* style nit

* runtime asset created at

* Asset filters

* fix usage kind filter

* also check runnable_path for jobs when filtering

* better filters

* avoid flickering

* debounced filters

* nit

* tooltips

* fix: update AssetUsage type to match new ListAssetsResponse structure

The ListAssetsResponse changed from an array to an object with an 'assets' property.
Updated the type extraction accordingly.

Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>

* sqlx prepare

* Delete .claude/hooks/.symlink-manifest

* unnecessary dep

* nit refactor

* nit comment

* nit naming

* CI fix attempt 1

* ee ref

* nit remove alerts

* nit

* chore: update ee-repo-ref to 138a4f5f868f3bded5bb7cb77b222b532c07e4af

This commit updates the EE repository reference after PR #395 was merged in windmill-ee-private.

Previous ee-repo-ref: 7d3a21d53066726e97dfea9f117373299bc9318c

New ee-repo-ref: 138a4f5f868f3bded5bb7cb77b222b532c07e4af

Automated by sync-ee-ref workflow.

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-02-02 10:09:40 +00:00
Maxime Thiebaut
00fc86d099 Add grpc entry to mapping in mapping.rs (#7616)
Map `grpc` to [`grpcio`](https://pypi.org/project/grpcio/).
2026-01-20 06:00:10 +00:00
hugocasa
0f2b417ff5 feat: add Pydantic BaseModel and dataclass support for Python type inference (#7497)
* feat: add Pydantic BaseModel and dataclass support for Python type inference

- Add AST-based detection of Pydantic BaseModel inheritance patterns
- Add AST-based detection of @dataclass decorator (all variants)
- Implement recursive field schema extraction with type inference
- Add thread-safe stack-based module storage for nested parsing
- Add RAII cleanup guard to ensure memory safety on all code paths
- Add security limits: 200 fields max, 10 recursion levels max
- Add comprehensive test coverage: 3 new tests for Pydantic/dataclass
- Maintain 100% backward compatibility with existing type system

This enables ML/AI practitioners to use Pydantic models as function
parameters with automatic UI generation from model schemas.

Implementation highlights:
- Zero code execution: Pure AST analysis for safety
- Thread-safe: Stack-based storage prevents race conditions
- Memory-safe: RAII pattern guarantees cleanup
- Security-hardened: Field count and recursion depth limits
- Performance-optimized: Depth-limited recursion, lazy parsing

Test results: All 12 tests passing (9 existing + 3 new)

Closes #4700

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix: improve Pydantic/dataclass parser robustness and error handling

This commit addresses critical bugs and improves error handling in the
Python parser for Pydantic BaseModel and dataclass support.

## Critical Fixes

1. **Thread-local storage RAII pattern**: Fixed bug where parse failures
   could leave the module stack in an inconsistent state. Now uses proper
   functional composition with .ok().map() to ensure cleanup always happens.

2. **Recursion depth warnings**: Added explicit warning messages when the
   recursion depth limit (10 levels) is reached during type extraction.
   Made the limit a named constant for clarity.

3. **Unsupported type warnings**: Added informative warning messages for
   unsupported type annotations (Union types and forward references) to
   help users understand why their types aren't being inferred.

## Improvements

- Added 10 comprehensive test cases covering:
  - Empty Pydantic models
  - List[T] and Optional[T] types
  - Dataclass with decorator arguments
  - Dict types
  - Regular classes (non-model types)
  - Invalid syntax handling
  - Datetime fields
  - Multiple model definitions
  - Nested models

- All 21 tests pass successfully

## Testing

Verified that:
- Parser handles malformed code gracefully
- RAII cleanup works correctly with early returns
- Warning messages are clear and actionable
- No memory leaks or panics

Closes #4700

* refactor: Separate Pydantic/dataclass code into dedicated module. Created src/pydantic_parser.rs with thread-local storage, model detection, and type extraction logic. Moved 12 Pydantic tests to tests/pydantic_tests.rs and removed duplicate code from lib.rs. All 21 tests passing.

* opti and publish

---------

Co-authored-by: Devdatta Talele <devtalele0@gmail.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-06 18:48:04 +00:00
Diego Imbert
3affbb3321 feat: type-checked data tables v0 (#7381)
* data tables settings ui

* install runed

* zod 4 fixes

* use new toJSONSchema

* Migrate ducklake catalogs to more generic custom instance databases

* fix compilation

* Safety conversion for old duckdb ffi

* data tables settings

* ts client basis

* inline run works

* datatables work

* Revert "datatables work"

This reverts commit 6e1588d59e.

* datatables work (without leaking pg credentials)

* println

* separate sqlUtils.ts

* nit

* Separate custom instance db Select and Wizard components

* nit

* nit wording

* add tags to custom instance dbs

* error when trying to use ducklake as datatable or opposite

* show status in dropdown

* data table instance setup works

* sqk function for ducklake

* factorize logic

* fix temp reactivity

* Data table assetexplore

* Migrate S3 permissions to modal

* Revert "Migrate S3 permissions to modal"

This reverts commit 0631d03cb0.

* nit query -> fetch

* Custom instance setup new look

* run_language_executor separate fn

* run_inline param

* nit wording

* Better typed client

* Data tables display as assets in frontend

* asset db icon

* nit

* cleaner errors

* nit

* Fix sed calls in mac

* run_inline_script_preview in python client

* basic python datatable client

* datatable and datalake parser in python

* ducklake client python

* nit fix

* Fix migration producing NULL instead of {} when no custom databases

* merge conflict fail

* python ducklake client arg fix

* parse or infer sql types in ts client

* ts asset parser, detect datatable & ducklake R/W

* fix sql repl for other read ops than select

* export type SqlTemplateFunction

* rename list_custom_instance_pg_databases

* typecheck datatable and ducklake name in Typescript

* Fix typecheck datatable and ducklake in TS

* declare module overriding instead of extending

* infer_sql_type in python client

* SqlQuery object in python

* fix merge conflicts

* update const_format

* CI fix

* factor out to var_identifiers

* sqlx prepare

* unnecessary security (admin is required)

* clearer comment

* ee repo ref

* nit snake case

* claude step 1: detect var declarations

* move detect_sql_access_type to common mod

* claude step 2: detect when saved vars are queried

* Revert "claude step 2: detect when saved vars are queried"

This reverts commit 1e1f930568.

* Revert "claude step 1: detect var declarations"

This reverts commit f866f4819d.

* remove ducklake/datatable and default

* detect data table assigns in var_identifiers

* Python parser successfully infers R/W/RW from ducklake / datatable

* still register ducklake/datatable if not used as unknown R/W

* Go to settings button in Assets Dropdown on not found

* nit

* sqlx prepare fail

* manual fix, somehow sqlx prepare won't do it

* fix frontend ci

* ee repo ref

* ducklake_user doesnt exist in unit tests

* nit fix

* ui nit

* nit

* nit missing clone

* fork ducklakes and datatables

* fix surface hover bug

* stupid mistake

* better deeply reactive mutable derived

* Ducklake picker

* Editor bar data tables

* DuckDB supports datatables

* datatable in duckdb asset parser

* duckdb asset parser var_identifiers

* Revert "duckdb asset parser var_identifiers"

This reverts commit 88068b1a77.

* sqlx prepare

* Box pin in test_workflow_as_code to fix stack overflow

* stash

* sql asset parser parses most s3 literals

* nit

* Detect attach + handle returning RW

* detect assets used with dot notation

* detect implicit access with USE dl; syntax

* Add assets as unknown if var was never used

* Support default ducklake/datatable main in parser

* ignore asset parsing errors in frontend (avoid flow layout shift)

* super weird duplication (merge conflict ?)

* nits

* fix duckdb parser detecting too much as asset when RW ctx is unknown

* fix transparent assets btn

* missing arg

* nit styling

* asset parser specific table parsing

* fix resource specific table parsing

* More concise asset display in flows + better icons

* fix assets page filtering out resources with added table

* Fix frontend to support specific table assets

* Open DB Manager to specific table

* Specific table parser in Python and TS + unit tests

* Fix UPDATE setting access to None

* fix flow edge rendering on top of output picker

* python parser fix var override bug

* add ts test

* fix compilation

* sqlx prepare

* update parsers version

* fix missing schema key onDelete

* Grant permission to create schemas in custom instance databases

* Update pg query to return empty schemas

* Create schema

* Select nits

* support schemas in sql parser

* ts parser handle schema with sql parser result

* detect .schema() syntax

* detect schema syntax in python

* support .schema() in ts and py SDK

* open db manager to specific schema

* support reassignment in ts parser

* nit better unitest

* : syntax in ts

* datatable:schema syntax in python

* fix client py

* nit select dropdown darkmode

* object | null fetchOne

* ts client nits

* parse_sql_client_name fn

* getImportWmillTsStatement refactor in EditorBar

* text to json() in python client

* update parser versions

* pkg lock

* Sql query details in TS asset parser

* code transformation with type parameter in Editor

* Custom Language Worker, code substition works !

* Error marker mapping works

* hover info is correct

* completions work correctly

* other overrides

* type inference kinda works

* Position mapping tests

* refactor prepare_queries

* Refactor PgDatabase to share common code

* Pgdatabase in prepare_queries

* TokioPgConnection refactor

* refactor prepare_queries

* type parameter to sql function

* Fix deadlock

* nit fix

* Fix worker async call freezing because of svelte Proxy

* Force worker to recompute when we set queries

* nit refactor

* nits console logs

* wait that ts worker initialize

* monaco change file version

* update diagnostics

* Refactor for errors

* Show SQL errors in Monaco

* improve sdk

* cleaning refactor + MapResource + usePreparedAssetSqlQueries

* Fixes

* Fix error position mapping

* cache in typescript worker

* fix insert no values

* don't inject type if already present

* Support schema in prepare queries

* update parsers

* ChangeOnDeepInequality

* inferAsset ScriptEditor usage refactor

* sql query typecheck work in flow editor

* Assets and SQL Query check in Raw App Inline Editor

* pkg lock

* Fix DatatableSqlTemplateFunction nit

* prepare query schema nit

* duplicate diagnostics

* nit getScriptVersion mock

* Reprepare queries when switching workspaces

* nit fix

* nit fix

* fetch_one_scalar and execute in python client

* limit pg_connections

* -- prepare flag in postgres

* skip serializing

* fix destructuring undefined

* Prepare queries in workers instead of backend

* nit

* Execute search_path instructions normally

* nit fix

* Fix SET search_path issue in prepare

* only support preparing single-statement queries for now

* update parsers

* safety

* better remove_comments

* Fix getQueryStmtCountHeuristic

* getQueryStmtCountHeuristic tests

* comment out failing tests

* Fix getQueryStmtCountHeuristic impl

* only datatable
2025-12-20 08:51:07 +00:00
Avigail Royzenberg
da500fcf3e fix: Python Enum types generate proper dropdown schemas with descriptions (#7400)
* Fix Python Enum and Literal schema generation with docstring descriptions

- Extract Enum class definitions and their string values
- Parse docstring Args: sections for parameter descriptions
- Map Enum type annotations to string enums with proper values
- Handle Enum.VALUE default values correctly
- Store descriptions in Arg.otyp field
- Add test case for enum with docstring parsing

* perf: optimize enum parser and fix default value handling

- Combine enum extraction and docstring parsing into single AST pass (2x performance improvement)
- Add support for IntEnum, StrEnum, Flag, IntFlag types
- Fix default values to use actual enum values (e.g., 'red') instead of member names (e.g., 'RED')
- Improve docstring parsing robustness with proper indentation tracking
- Clean up code structure with EnumInfo type for better maintainability

All tests pass. This addresses code review feedback for performance and correctness.

* perf: implement true lazy evaluation for enum parsing

- Only parse metadata when unknown types encountered
- Two-pass approach: parse types first, extract only if needed
- Zero overhead for scripts without enums
- Keyword checks + prepass filtering when extraction needed
2025-12-19 20:41:34 +00:00
Diego Imbert
75fdc2cdc9 feat: data table schemas (#7353)
* data tables settings ui

* install runed

* zod 4 fixes

* use new toJSONSchema

* Migrate ducklake catalogs to more generic custom instance databases

* fix compilation

* Safety conversion for old duckdb ffi

* data tables settings

* ts client basis

* inline run works

* datatables work

* Revert "datatables work"

This reverts commit 6e1588d59e.

* datatables work (without leaking pg credentials)

* println

* separate sqlUtils.ts

* nit

* Separate custom instance db Select and Wizard components

* nit

* nit wording

* add tags to custom instance dbs

* error when trying to use ducklake as datatable or opposite

* show status in dropdown

* data table instance setup works

* sqk function for ducklake

* factorize logic

* fix temp reactivity

* Data table assetexplore

* Migrate S3 permissions to modal

* Revert "Migrate S3 permissions to modal"

This reverts commit 0631d03cb0.

* nit query -> fetch

* Custom instance setup new look

* run_language_executor separate fn

* run_inline param

* nit wording

* Better typed client

* Data tables display as assets in frontend

* asset db icon

* nit

* cleaner errors

* nit

* Fix sed calls in mac

* run_inline_script_preview in python client

* basic python datatable client

* datatable and datalake parser in python

* ducklake client python

* nit fix

* Fix migration producing NULL instead of {} when no custom databases

* merge conflict fail

* python ducklake client arg fix

* parse or infer sql types in ts client

* ts asset parser, detect datatable & ducklake R/W

* fix sql repl for other read ops than select

* export type SqlTemplateFunction

* rename list_custom_instance_pg_databases

* typecheck datatable and ducklake name in Typescript

* Fix typecheck datatable and ducklake in TS

* declare module overriding instead of extending

* infer_sql_type in python client

* SqlQuery object in python

* fix merge conflicts

* update const_format

* CI fix

* factor out to var_identifiers

* sqlx prepare

* unnecessary security (admin is required)

* clearer comment

* ee repo ref

* nit snake case

* claude step 1: detect var declarations

* move detect_sql_access_type to common mod

* claude step 2: detect when saved vars are queried

* Revert "claude step 2: detect when saved vars are queried"

This reverts commit 1e1f930568.

* Revert "claude step 1: detect var declarations"

This reverts commit f866f4819d.

* remove ducklake/datatable and default

* detect data table assigns in var_identifiers

* Python parser successfully infers R/W/RW from ducklake / datatable

* still register ducklake/datatable if not used as unknown R/W

* Go to settings button in Assets Dropdown on not found

* nit

* sqlx prepare fail

* manual fix, somehow sqlx prepare won't do it

* fix frontend ci

* ee repo ref

* ducklake_user doesnt exist in unit tests

* nit fix

* ui nit

* nit

* nit missing clone

* fork ducklakes and datatables

* fix surface hover bug

* stupid mistake

* better deeply reactive mutable derived

* Ducklake picker

* Editor bar data tables

* DuckDB supports datatables

* datatable in duckdb asset parser

* duckdb asset parser var_identifiers

* Revert "duckdb asset parser var_identifiers"

This reverts commit 88068b1a77.

* sqlx prepare

* Box pin in test_workflow_as_code to fix stack overflow

* stash

* sql asset parser parses most s3 literals

* nit

* Detect attach + handle returning RW

* detect assets used with dot notation

* detect implicit access with USE dl; syntax

* Add assets as unknown if var was never used

* Support default ducklake/datatable main in parser

* ignore asset parsing errors in frontend (avoid flow layout shift)

* super weird duplication (merge conflict ?)

* nits

* fix duckdb parser detecting too much as asset when RW ctx is unknown

* fix transparent assets btn

* missing arg

* nit styling

* asset parser specific table parsing

* fix resource specific table parsing

* More concise asset display in flows + better icons

* fix assets page filtering out resources with added table

* Fix frontend to support specific table assets

* Open DB Manager to specific table

* Specific table parser in Python and TS + unit tests

* Fix UPDATE setting access to None

* fix flow edge rendering on top of output picker

* python parser fix var override bug

* add ts test

* fix compilation

* sqlx prepare

* update parsers version

* fix missing schema key onDelete

* Grant permission to create schemas in custom instance databases

* Update pg query to return empty schemas

* Create schema

* Select nits

* support schemas in sql parser

* ts parser handle schema with sql parser result

* detect .schema() syntax

* detect schema syntax in python

* support .schema() in ts and py SDK

* open db manager to specific schema

* support reassignment in ts parser

* nit better unitest

* : syntax in ts

* datatable:schema syntax in python

* fix client py

* nit select dropdown darkmode

* object | null fetchOne

* ts client nits

* parse_sql_client_name fn

* getImportWmillTsStatement refactor in EditorBar

* text to json() in python client

* update parser versions

* pkg lock
2025-12-16 21:16:42 +00:00
Diego Imbert
c6c7f3415a Specific asset tables (#7323)
* data tables settings ui

* install runed

* zod 4 fixes

* use new toJSONSchema

* Migrate ducklake catalogs to more generic custom instance databases

* fix compilation

* Safety conversion for old duckdb ffi

* data tables settings

* ts client basis

* inline run works

* datatables work

* Revert "datatables work"

This reverts commit 6e1588d59e.

* datatables work (without leaking pg credentials)

* println

* separate sqlUtils.ts

* nit

* Separate custom instance db Select and Wizard components

* nit

* nit wording

* add tags to custom instance dbs

* error when trying to use ducklake as datatable or opposite

* show status in dropdown

* data table instance setup works

* sqk function for ducklake

* factorize logic

* fix temp reactivity

* Data table assetexplore

* Migrate S3 permissions to modal

* Revert "Migrate S3 permissions to modal"

This reverts commit 0631d03cb0.

* nit query -> fetch

* Custom instance setup new look

* run_language_executor separate fn

* run_inline param

* nit wording

* Better typed client

* Data tables display as assets in frontend

* asset db icon

* nit

* cleaner errors

* nit

* Fix sed calls in mac

* run_inline_script_preview in python client

* basic python datatable client

* datatable and datalake parser in python

* ducklake client python

* nit fix

* Fix migration producing NULL instead of {} when no custom databases

* merge conflict fail

* python ducklake client arg fix

* parse or infer sql types in ts client

* ts asset parser, detect datatable & ducklake R/W

* fix sql repl for other read ops than select

* export type SqlTemplateFunction

* rename list_custom_instance_pg_databases

* typecheck datatable and ducklake name in Typescript

* Fix typecheck datatable and ducklake in TS

* declare module overriding instead of extending

* infer_sql_type in python client

* SqlQuery object in python

* fix merge conflicts

* update const_format

* CI fix

* factor out to var_identifiers

* sqlx prepare

* unnecessary security (admin is required)

* clearer comment

* ee repo ref

* nit snake case

* claude step 1: detect var declarations

* move detect_sql_access_type to common mod

* claude step 2: detect when saved vars are queried

* Revert "claude step 2: detect when saved vars are queried"

This reverts commit 1e1f930568.

* Revert "claude step 1: detect var declarations"

This reverts commit f866f4819d.

* remove ducklake/datatable and default

* detect data table assigns in var_identifiers

* Python parser successfully infers R/W/RW from ducklake / datatable

* still register ducklake/datatable if not used as unknown R/W

* Go to settings button in Assets Dropdown on not found

* nit

* sqlx prepare fail

* manual fix, somehow sqlx prepare won't do it

* fix frontend ci

* ee repo ref

* ducklake_user doesnt exist in unit tests

* nit fix

* ui nit

* nit

* nit missing clone

* fork ducklakes and datatables

* fix surface hover bug

* stupid mistake

* better deeply reactive mutable derived

* Ducklake picker

* Editor bar data tables

* DuckDB supports datatables

* datatable in duckdb asset parser

* duckdb asset parser var_identifiers

* Revert "duckdb asset parser var_identifiers"

This reverts commit 88068b1a77.

* sqlx prepare

* Box pin in test_workflow_as_code to fix stack overflow

* stash

* sql asset parser parses most s3 literals

* nit

* Detect attach + handle returning RW

* detect assets used with dot notation

* detect implicit access with USE dl; syntax

* Add assets as unknown if var was never used

* Support default ducklake/datatable main in parser

* ignore asset parsing errors in frontend (avoid flow layout shift)

* super weird duplication (merge conflict ?)

* nits

* fix duckdb parser detecting too much as asset when RW ctx is unknown

* fix transparent assets btn

* missing arg

* nit styling

* asset parser specific table parsing

* fix resource specific table parsing

* More concise asset display in flows + better icons

* fix assets page filtering out resources with added table

* Fix frontend to support specific table assets

* Open DB Manager to specific table

* Specific table parser in Python and TS + unit tests

* Fix UPDATE setting access to None

* fix flow edge rendering on top of output picker

* python parser fix var override bug

* add ts test

* fix compilation

* sqlx prepare

* update parsers version

* fix missing schema key onDelete
2025-12-11 13:34:17 +00:00
Diego Imbert
9bbab3321e feat: Data tables (#7226)
* data tables settings ui

* install runed

* zod 4 fixes

* use new toJSONSchema

* Migrate ducklake catalogs to more generic custom instance databases

* fix compilation

* Safety conversion for old duckdb ffi

* data tables settings

* ts client basis

* inline run works

* datatables work

* Revert "datatables work"

This reverts commit 6e1588d59e.

* datatables work (without leaking pg credentials)

* println

* separate sqlUtils.ts

* nit

* Separate custom instance db Select and Wizard components

* nit

* nit wording

* add tags to custom instance dbs

* error when trying to use ducklake as datatable or opposite

* show status in dropdown

* data table instance setup works

* sqk function for ducklake

* factorize logic

* fix temp reactivity

* Data table assetexplore

* Migrate S3 permissions to modal

* Revert "Migrate S3 permissions to modal"

This reverts commit 0631d03cb0.

* nit query -> fetch

* Custom instance setup new look

* run_language_executor separate fn

* run_inline param

* nit wording

* Better typed client

* Data tables display as assets in frontend

* asset db icon

* nit

* cleaner errors

* nit

* Fix sed calls in mac

* run_inline_script_preview in python client

* basic python datatable client

* datatable and datalake parser in python

* ducklake client python

* nit fix

* Fix migration producing NULL instead of {} when no custom databases

* merge conflict fail

* python ducklake client arg fix

* parse or infer sql types in ts client

* ts asset parser, detect datatable & ducklake R/W

* fix sql repl for other read ops than select

* export type SqlTemplateFunction

* rename list_custom_instance_pg_databases

* typecheck datatable and ducklake name in Typescript

* Fix typecheck datatable and ducklake in TS

* declare module overriding instead of extending

* infer_sql_type in python client

* SqlQuery object in python

* fix merge conflicts

* update const_format

* CI fix

* factor out to var_identifiers

* sqlx prepare

* unnecessary security (admin is required)

* clearer comment

* ee repo ref

* nit snake case

* claude step 1: detect var declarations

* move detect_sql_access_type to common mod

* claude step 2: detect when saved vars are queried

* Revert "claude step 2: detect when saved vars are queried"

This reverts commit 1e1f930568.

* Revert "claude step 1: detect var declarations"

This reverts commit f866f4819d.

* remove ducklake/datatable and default

* detect data table assigns in var_identifiers

* Python parser successfully infers R/W/RW from ducklake / datatable

* still register ducklake/datatable if not used as unknown R/W

* Go to settings button in Assets Dropdown on not found

* nit

* sqlx prepare fail

* manual fix, somehow sqlx prepare won't do it

* fix frontend ci

* ee repo ref

* ducklake_user doesnt exist in unit tests

* nit fix

* ui nit

* nit

* nit missing clone

* fork ducklakes and datatables

* fix surface hover bug

* stupid mistake

* better deeply reactive mutable derived

* Ducklake picker

* Editor bar data tables

* DuckDB supports datatables

* datatable in duckdb asset parser

* duckdb asset parser var_identifiers

* Revert "duckdb asset parser var_identifiers"

This reverts commit 88068b1a77.

* sqlx prepare

* Box pin in test_workflow_as_code to fix stack overflow

* go to settings button

* ee repo ref

* fix compilation

* wording nit
2025-12-05 23:08:58 +00:00
Pyra
d38c96db36 feat: workspace dependencies (#7124)
* commit raw requirements

Signed-off-by: pyranota <pyra@duck.com>

* raw requirements

Signed-off-by: pyranota <pyra@duck.com>

* implement `parse_annotation`

Signed-off-by: pyranota <pyra@duck.com>

* more progress on wdeps

Signed-off-by: pyranota <pyra@duck.com>

* more progress

Signed-off-by: pyranota <pyra@duck.com>

* fixes

Signed-off-by: pyranota <pyra@duck.com>

* more progress

Signed-off-by: pyranota <pyra@duck.com>

* fixes

Signed-off-by: pyranota <pyra@duck.com>

* cli improvements + raw deps

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fix python versions

Signed-off-by: pyranota <pyra@duck.com>

* progress

Signed-off-by: pyranota <pyra@duck.com>

* update :)

Signed-off-by: pyranota <pyra@duck.com>

* add MaybeLock

Signed-off-by: pyranota <pyra@duck.com>

* go WIP

Signed-off-by: pyranota <pyra@duck.com>

* fix python ignoring py version from requirements

Signed-off-by: pyranota <pyra@duck.com>

* optimize php

Signed-off-by: pyranota <pyra@duck.com>

* require admin to alter

Signed-off-by: pyranota <pyra@duck.com>

* fix(cli): flow generateLocks raw deps

Signed-off-by: pyranota <pyra@duck.com>

* progress in checklist

Signed-off-by: pyranota <pyra@duck.com>

* fix agent workers

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* nit: remove default features

Signed-off-by: pyranota <pyra@duck.com>

* oh-wow

Signed-off-by: pyranota <pyra@duck.com>

* remove dbg!

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* add indexes

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* nits

Signed-off-by: pyranota <pyra@duck.com>

* remove todos

Signed-off-by: pyranota <pyra@duck.com>

* fix cli

Signed-off-by: pyranota <pyra@duck.com>

* add debug flag

Signed-off-by: pyranota <pyra@duck.com>

* cli: remove noise

Signed-off-by: pyranota <pyra@duck.com>

* fix cli

Signed-off-by: pyranota <pyra@duck.com>

* remove todos

Signed-off-by: pyranota <pyra@duck.com>

* trigger deps correctly

Signed-off-by: pyranota <pyra@duck.com>

* fix frontend

Signed-off-by: pyranota <pyra@duck.com>

* fix frontend again

Signed-off-by: pyranota <pyra@duck.com>

* finally fix frontend

Signed-off-by: pyranota <pyra@duck.com>

* ee repo ref

Signed-off-by: pyranota <pyra@duck.com>

* fix all

Signed-off-by: pyranota <pyra@duck.com>

* more fixes...

Signed-off-by: pyranota <pyra@duck.com>

* remove test

Signed-off-by: pyranota <pyra@duck.com>

* Update backend-test.yml

* comment out legacy test

Signed-off-by: pyranota <pyra@duck.com>

* fix ci

Signed-off-by: pyranota <pyra@duck.com>

* fix ci?

Signed-off-by: pyranota <pyra@duck.com>

* comment out thing

Signed-off-by: pyranota <pyra@duck.com>

* ignore test

Signed-off-by: pyranota <pyra@duck.com>

* ci

Signed-off-by: pyranota <pyra@duck.com>

* base fixture

Signed-off-by: pyranota <pyra@duck.com>

* fix regression

Signed-off-by: pyranota <pyra@duck.com>

* fix docs links

Signed-off-by: pyranota <pyra@duck.com>

* update min version

Signed-off-by: pyranota <pyra@duck.com>

* simplify

* implement cache for get_latest

Signed-off-by: pyranota <pyra@duck.com>

* move to workspace settings

Signed-off-by: pyranota <pyra@duck.com>

* sqlx + migration

Signed-off-by: pyranota <pyra@duck.com>

* more migrations

Signed-off-by: pyranota <pyra@duck.com>

* use box pin

Signed-off-by: pyranota <pyra@duck.com>

* nit

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-11-28 16:00:11 +00:00
HugoCasa
67895abc9f add option to build wasm packages for nodejs 2025-11-26 16:20:16 +01:00
hugocasa
b66e038a0f feat: add support for validateset in pwsh (#7158) 2025-11-17 11:29:58 +00:00
hugocasa
c16bef8f29 feat: add support for switch and attributes in pwsh params (#7143) 2025-11-14 17:28:54 +00:00
hugocasa
51cba95d39 fix(backend): improve pwsh param block parsing (#7096)
* fix(backend): replace regex with one-pass algorithm for PowerShell param parsing

The previous regex-based approach used  which would
fail when encountering nested parentheses inside the param block, such as
 or .

This commit replaces the regex with a proper one-pass parser that:
- Tracks parenthesis depth to correctly handle nesting
- Respects string quotes (both single and double quotes)
- Handles PowerShell's backtick escape character
- Works correctly with complex default values and function calls

Changes:
- Removed RE_POWERSHELL_PARAM regex constant
- Added extract_powershell_param_block() function for extracting param contents
- Added extract_powershell_param_block_full() function for extracting full param block
- Updated parse_powershell_file() in windmill-parser-bash
- Updated pwsh_executor.rs to use the new extractor
- Added comprehensive test cases for nested parens and quoted strings

Fixes #7079

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>

* refactor(backend): combine PowerShell param extraction functions

Merged extract_powershell_param_block() and extract_powershell_param_block_full()
into a single function with a boolean parameter to control output format:
- include_keyword=false returns just contents between parentheses
- include_keyword=true returns full param(...) block

This eliminates code duplication while maintaining all existing functionality.

Co-authored-by: hugocasa <hugocasa@users.noreply.github.com>

* fix(backend): improve pwsh param block parsing

* chore: publish parser

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: hugocasa <hugocasa@users.noreply.github.com>
2025-11-10 15:08:20 +00:00
dieriba
fc5034e94d fix: preproccessor ui and expanding preprocessor support (#6872)
* preproccessor php

* fix

* ok

* remove folder

* chore: publish parser

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-11-10 11:22:31 +00:00
Ruben Fiszel
a9b99e9467 fix(cli): fix generate-metadata wasm parsers 2025-10-27 21:12:40 +00:00
Ruben Fiszel
5b1ab610b7 improve stack size usage (#6936)
* oom

* final

* final
2025-10-24 22:08:54 +00:00
hugocasa
4b26def0cd chore(parsers): publish bash parser with CRLF handling (#6905) 2025-10-23 08:33:41 +00:00
Vivek Chavan
fa76105d14 fix: detect preprocessor in re-exported named exports (#6899)
Fixes #6894

The TypeScript parser now correctly detects preprocessor functions
that are re-exported from other modules using named exports like:
  export { preprocessor } from "./other_module";

Previously, only function declarations were detected. Now the parser
also checks ExportNamed AST nodes for any specifier named 'preprocessor'.

This allows developers to easily reuse preprocessor functions across
multiple scripts without the workaround of wrapping them in a new function.

Added comprehensive tests covering:
- Simple re-export: export { preprocessor } from "./other"
- Re-export with renaming: export { preprocessor as preprocessor }
- Mixed exports: export { foo, preprocessor, bar }
- Negative case: exports without preprocessor
2025-10-23 07:51:57 +00:00
Vivek Chavan
5529736784 fix(parser): Handle CRLF line endings in bash and PowerShell parsers (#6889)
Fixes #6867

The bash and PowerShell parsers were not properly handling CRLF (Windows-style)
line endings when parsing script arguments. The regex patterns were only
matching LF line endings, causing scripts with CRLF to fail metadata generation.

Changes:
- Updated RE_BASH regex to optionally match \r before line end (\r?$)
- Updated RE_POWERSHELL_ARGS regex to optionally match \r (\r?)
- Added test case test_parse_bash_sig_with_crlf() to verify CRLF handling

This fix ensures that bash scripts created on Windows (or with editors
using CRLF) will correctly parse arguments and generate proper metadata
via 'wmill script generate-metadata' command.

Python parser was verified to work correctly as it uses an AST parser
that inherently handles line endings properly.
2025-10-22 06:21:38 +00:00
wendrul
83337b2035 Ansible repo update hubPath + fixes (#6843)
* Allow ssh auth

* ssh changes pt2

* Fix unused

* Update package.json and hubPaths
2025-10-16 18:24:41 +00:00
wendrul
32fae7a10c feat: ansible playbook execution git repo mode (repo viewer + UI utils) (#6831)
* Improve minio on flake.nix

* Add first asset parsing logic for ansible

* Correct html gt sign

* Decouple s3 file picker from drawer

* Factor duplicate code into snippet

* Update S3FilePickerInner to be compatible

* Fix pane shrinking issue

* Git repo viewer

* Change GitRepoViewer

* Endpoints for git repo visualizer

* Move git repo viewer to its own component

* Add button to populate git repo viewer

* Update parser yaml for new ansisble features (repo viewer)

* Reflect parser changes for ansible

* Add button to add the git repo mode of declaration for ansible

* Factor function

* Playbook + inventories into the drawer

* Add button to add inventories from s3

* Move tests to lib.rs

* Inventory loading from s3

* Move get github app token logic to be reused by ansible

* Update parser and ansible executor

* Use the correct path for inventories

* Add nushell to flake for wasm builds

* Add published parser

* Update hubPaths with clone and upload to s3

* Update ee-repo to the branch ref

* Fix npm run check

* Update cargo.lock

* Change labels on buttons

* Remove debug log

* Update ee-repo-ref

* Fix ee issues

* Update ee-repo ref

* Fix typo

* Fix ee

* Update ee-repo-ref

* Fix missing imports

* Unused var

* Fix typo

* Layout improvents

* Fix typos

* Remove unused function and log

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-10-16 09:08:18 +00:00
Pyra
e2feba391c feat(rust): add resource types (#5843)
* feat(rust): add resource types

* write tests

* copy from go-resources

* implement frontend for rust

* update

* update cli

* update tests

* update

Signed-off-by: pyranota <pyra@duck.com>

* fix package-lock

* fix ci

* update parser

Signed-off-by: pyranota <pyra@duck.com>

* Update frontend/src/lib/components/Editor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-10-13 19:15:44 +00:00
Diego Imbert
59cdb141c3 NULL Toggle in InsertRow drawer (#6729)
* NULL toggle in InsertRow

* fix long type parsing in postgres

* nits

* graphite catch

* lazy_static

* support for time/timestamp/tz long forms in pg parser

* graphite suggestion
2025-10-02 10:10:53 +00:00
hugocasa
898eb6231b feat(backend): array and object params support in pwsh (#6706)
* feat(backend): array and object params support in pwsh

* add bool and improve arg conversion

* fix bash

* update parser

* sqlx
2025-09-30 12:25:12 +00:00
Ruben Fiszel
8518b7c897 check 2025-09-20 00:59:03 +00:00
Diego Imbert
673b4d2a4c Advanced S3 permissionning (#6617)
* Advanced permission rules UI

* stash

* first iteration for s3 rule parsing

* Move to glob based approach

* cache expiry

* fix popover positioning

* ee

* unused imports

* forgot windmill_uploads

* Check S3 permissions for apps

* nit

* typo

* ee repo ref

* forgot get_workspace_s3_resource_and_check_paths in oss
2025-09-17 12:15:33 +00:00
dieriba
f3f330dd2a feat: windmill dyn multiselect (#6488)
* done

* fix

* better

* typo

* use old key

* chore: publish pkgs and update deps

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-09-05 07:10:01 +00:00
claude[bot]
8c1be78f61 fix(parser): detect assets in Python named args (#6518) 2025-09-05 06:28:08 +00:00
Ruben Fiszel
97ed4a539b nits cleanup + faster script index #6450 2025-08-23 22:42:39 +00:00
pyranota
11dd4118ce feat: add ruby support (#5939)
* implement single line pin

* make panic-safe

* use pin even if multiple modules imported withing single statement

* add repins and make imports respect pins

* keep all pins

* Allow multiple pins

* add comments + handle stuff more safely

* fix fully qualified imports

* remove ignore

* sort nested

* apply unique to output requirements list

* fix typo

* remove mut

* update sqlx

* sort imports

* sort imports

* fix formatter and format

* feat: add ruby support

* fix

* update sqlx

* add ruby to flake.nix

Signed-off-by: pyranota <pyra@duck.com>

* upgrade par_install_language_dependencies helper

rename fields + allow passing custom payload

Signed-off-by: pyranota <pyra@duck.com>

* change migrations

Signed-off-by: pyranota <pyra@duck.com>

* make full pipeline work!

Signed-off-by: pyranota <pyra@duck.com>

* do a bit fixing

* stage

Signed-off-by: pyranota <pyra@duck.com>

* add ruby in dockerfiles

Signed-off-by: pyranota <pyra@duck.com>

* implement auto-require + 'windmill/inline'

Signed-off-by: pyranota <pyra@duck.com>

* create 'windmill/inline' only if does not exist

Signed-off-by: pyranota <pyra@duck.com>

* add nsjail

Signed-off-by: pyranota <pyra@duck.com>

* update global settings

Signed-off-by: pyranota <pyra@duck.com>

* add ruby icon

* generate lockfile on deploy (but broken)

Signed-off-by: pyranota <pyra@duck.com>

* fix(frontend): display deployed script lockfile and other info

Signed-off-by: pyranota <pyra@duck.com>

* feat: add mini-wmill client

Methods:
- get_resource
- get_variable

Additionally add shortcuts on frontend

Signed-off-by: pyranota <pyra@duck.com>

* add nit warning

Signed-off-by: pyranota <pyra@duck.com>

* support private repos credentials

Signed-off-by: pyranota <pyra@duck.com>

* fix compilation error

Signed-off-by: pyranota <pyra@duck.com>

* found rustc bug + refactor universal installer

Signed-off-by: pyranota <pyra@duck.com>

* more refactor + fixes

Signed-off-by: pyranota <pyra@duck.com>

* bug fixing + polishing

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* change 'windmill/inline'

Signed-off-by: pyranota <pyra@duck.com>

* delegate env_clear to universal installer

Signed-off-by: pyranota <pyra@duck.com>

* update init script

Signed-off-by: pyranota <pyra@duck.com>

* add cli support

Signed-off-by: pyranota <pyra@duck.com>

* unhardcode frontend

Signed-off-by: pyranota <pyra@duck.com>

* fix cli

Signed-off-by: pyranota <pyra@duck.com>

* fix init script

Signed-off-by: pyranota <pyra@duck.com>

* make it work on ms windows

Signed-off-by: pyranota <pyra@duck.com>

* do not clean env in the end

Signed-off-by: pyranota <pyra@duck.com>

* fix windows (again)

Signed-off-by: pyranota <pyra@duck.com>

* include PROXY_ENVS (untested)

Signed-off-by: pyranota <pyra@duck.com>

* feat: self-signed certs + proxy support (tested)

Signed-off-by: pyranota <pyra@duck.com>

* flake: pin ruby to 3.4

* merge follow-ups

Signed-off-by: pyranota <pyra@duck.com>

* remove obsolete libc crate

Signed-off-by: pyranota <pyra@duck.com>

* cleanup parser

Signed-off-by: pyranota <pyra@duck.com>

* fix java ci

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fix nits

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/universal_pkg_installer.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/ruby_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* add nsjail for lock stage

Signed-off-by: pyranota <pyra@duck.com>

* fix merge

Signed-off-by: pyranota <pyra@duck.com>

* Update common.rs

* regen windmill-api-client

Signed-off-by: pyranota <pyra@duck.com>

* fix tests

Signed-off-by: pyranota <pyra@duck.com>

* fix tests again

Signed-off-by: pyranota <pyra@duck.com>

* reference published ruby parser

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-08-15 13:03:32 +00:00
Diego Imbert
e6f1211d31 feat: Ducklake native support (#6268)
* upgrade duckdb

* basic ducklake works

* ducklake works with custom db catalogs

* fix: pwsh skip already installed modules outside of cache (#6037)

* improve query performance of user stats

* separate ducklake_catalog db

* ducklake settings

* DucklakeSettings frontend

* Ducklake ws settings saved in backend

* fetch ducklake catalog resource

* Ducklake works with configured s3 storage

* Ducklake as asset

* ducklake asset icon

* Fix duckdb array and object args not working properly (#6254)

* Fix bug with comments in duckdb

* Avoid multiple queries when doing ATTACH ducklake

* trunc sig no longer needed now that comments are trimmed

* cache DuckdbConnectionSettingsResponse

* duplicated code

* transform_attach_ducklake contributes to duckdb_connection_settings_cache

* eliminate the need for used_storages

* nit

* cleaner management of the bigquery credentials file

* DBManagerDrawer refactor to prepare for Ducklake

* get ducklake schema

* implement delete for ducklake

* load column metadata for ducklake

* Select query works for ducklake, basic db explorer works !

* duckdb count query

* Support all db ops for ducklake

* clean migrations

* SQL repl for Ducklake

* fix broken database studio

* nit

* assert function

* Ducklake in Editor Bar

* default ducklake syntax + allow extra args

* DucklakeCatalogWizard UI

* nit + remove extra $

* modal when databases do not exist

* cannot be windmill

* Ducklake works safely with instance database

* Avoid sending instance db credentials on network

* resource leak security

* remove fetch_attach_db_conn_str

* prevent instance pg password leak

* hide asset usage count when not available

* case unsensitivity duckdb

* warnings

* disable instance catalog

* use shorthand syntax when inserting with EditorBar

* Instance ducklake catalog is now safe to use

* use safer argon2 pwd

* update package json parsers

* update package json

* better msgs

* tooltips

* disable explore button until saved

* nit

* fix warnings

* better ducklake_user password management

* nit

* Sanitize passwords from errors in ducklake

* DisplayResult broken in job result

* remove superadmin requirement to check databases_exist

* duckdb_connection_settings_v2_inner

* Ducklake works on agent worker (finally)

* ci

* #[allow(dead_code)]

* fix openapi missing response

* Separate +Database button for DuckDB in EditorBar

* Fix dropdown in ducklake settings

* Attempt to fix migration race condition in CI

* update sqlx failing for some offline queries

* avoid temp password for ducklake_user

* nits

* ducklake settings nits

* update duckdb default script

* fix sql repl resetting text on refresh

* avoid pgcrypto extension

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-08-06 13:55:36 +00:00
dieriba
9931311650 fix: resource-type-ts-parser (#6289) 2025-07-29 14:53:24 +00:00
dieriba
10befb995d feat: local type references parsing support for main function args (#5995)
* add base struct

* feat resolve interface and type declarion in entrypoint param's function

* nits

* fix reset dependencies

* update package

* fix handle infinite recursion

* add depth level and handle enum for referenced type

* nits

* nits

* nits

* perf

* fix

* done

* fix schema form cache inconsistency

* fix default type and nits

* remove

* update Object typ for parser

* one level ref from from parent when resolving types and use format for resource

* update cli and use resource type

* nits

* update parsers

* fix: use specific parser versions

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-07-23 10:05:32 +02:00
Ruben Fiszel
91849baf34 fix: include export from ts relative import tracking 2025-07-22 22:58:49 +00:00
Diego Imbert
2062a634f6 Fix duckdb array and object args not working properly (#6254) 2025-07-22 15:34:38 +00:00
pyranota
3efc329fc0 build: combine build-pkgs*.sh into build.nu (#6231)
* build: combine build-pkgs* into build.nu

Signed-off-by: pyranota <pyra@duck.com>

* nit refactor

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
2025-07-21 12:11:17 +00:00
Diego Imbert
2ab5345e61 Assets refactor (#6217)
* Moved logic to FlowAssetsProvider

* Remove assetsMap in flow

* do not parse everything on mount + only check for missing assets fields

* add assets field in backend

* remove fallbackAccessTypes

* better structure and less queries / parsing

* Fix assets not showing when pulling raw_flow from jobs

* flow assets ctx for job run

* Fix transitive assets fetching

* Fix input args asset node

* enablePathScriptAndFlowAssets flag

* edit btn for variable

* untrack refresh

* move parseInputArgsAssets

* Assets tab in runs

* Update FlowStatusViewerInner to svelte 5 + fix asset sync bug

* avoid toast error on bad resource

* fetch res metadata for input arg asset

* Job assets viewer in run page

* r/w selector

* remove indigo badge

* store alt_access_type state in ScriptEditor

* Don't parse assets in flow script editor

* Add alt_access_type in backend

* show Read as selected by default to avoid giving the feeling of having made a decision

* keep alt_access_type when reparsing in flow raw scripts

* Remove variable asset kind, and save assets for scripts

* remove all backend asset parsing

* R/W/RW selector button nits

* fix insert into assets not saving alt access type

* support named arguments in python asset parser

* improve asset usage drawer R/W indicator

* update legacy $res: syntax

* reactivity issue

* remove last variable asset stuff

* sqlx prepare

* tooltip explainer

* deprecated variable asset nit
2025-07-17 22:15:01 +00:00