Use .get("storage") instead of direct key access to prevent KeyError
when s3object doesn't contain the optional storage field.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* data tables settings ui
* install runed
* zod 4 fixes
* use new toJSONSchema
* Migrate ducklake catalogs to more generic custom instance databases
* fix compilation
* Safety conversion for old duckdb ffi
* data tables settings
* ts client basis
* inline run works
* datatables work
* Revert "datatables work"
This reverts commit 6e1588d59e.
* datatables work (without leaking pg credentials)
* println
* separate sqlUtils.ts
* nit
* Separate custom instance db Select and Wizard components
* nit
* nit wording
* add tags to custom instance dbs
* error when trying to use ducklake as datatable or opposite
* show status in dropdown
* data table instance setup works
* sqk function for ducklake
* factorize logic
* fix temp reactivity
* Data table assetexplore
* Migrate S3 permissions to modal
* Revert "Migrate S3 permissions to modal"
This reverts commit 0631d03cb0.
* nit query -> fetch
* Custom instance setup new look
* run_language_executor separate fn
* run_inline param
* nit wording
* Better typed client
* Data tables display as assets in frontend
* asset db icon
* nit
* cleaner errors
* nit
* Fix sed calls in mac
* run_inline_script_preview in python client
* basic python datatable client
* datatable and datalake parser in python
* ducklake client python
* nit fix
* Fix migration producing NULL instead of {} when no custom databases
* merge conflict fail
* python ducklake client arg fix
* parse or infer sql types in ts client
* ts asset parser, detect datatable & ducklake R/W
* fix sql repl for other read ops than select
* export type SqlTemplateFunction
* rename list_custom_instance_pg_databases
* typecheck datatable and ducklake name in Typescript
* Fix typecheck datatable and ducklake in TS
* declare module overriding instead of extending
* infer_sql_type in python client
* SqlQuery object in python
* fix merge conflicts
* update const_format
* CI fix
* factor out to var_identifiers
* sqlx prepare
* unnecessary security (admin is required)
* clearer comment
* ee repo ref
* nit snake case
* claude step 1: detect var declarations
* move detect_sql_access_type to common mod
* claude step 2: detect when saved vars are queried
* Revert "claude step 2: detect when saved vars are queried"
This reverts commit 1e1f930568.
* Revert "claude step 1: detect var declarations"
This reverts commit f866f4819d.
* remove ducklake/datatable and default
* detect data table assigns in var_identifiers
* Python parser successfully infers R/W/RW from ducklake / datatable
* still register ducklake/datatable if not used as unknown R/W
* Go to settings button in Assets Dropdown on not found
* nit
* sqlx prepare fail
* manual fix, somehow sqlx prepare won't do it
* fix frontend ci
* ee repo ref
* ducklake_user doesnt exist in unit tests
* nit fix
* ui nit
* nit
* nit missing clone
* fork ducklakes and datatables
* fix surface hover bug
* stupid mistake
* better deeply reactive mutable derived
* Ducklake picker
* Editor bar data tables
* DuckDB supports datatables
* datatable in duckdb asset parser
* duckdb asset parser var_identifiers
* Revert "duckdb asset parser var_identifiers"
This reverts commit 88068b1a77.
* sqlx prepare
* Box pin in test_workflow_as_code to fix stack overflow
* stash
* sql asset parser parses most s3 literals
* nit
* Detect attach + handle returning RW
* detect assets used with dot notation
* detect implicit access with USE dl; syntax
* Add assets as unknown if var was never used
* Support default ducklake/datatable main in parser
* ignore asset parsing errors in frontend (avoid flow layout shift)
* super weird duplication (merge conflict ?)
* nits
* fix duckdb parser detecting too much as asset when RW ctx is unknown
* fix transparent assets btn
* missing arg
* nit styling
* asset parser specific table parsing
* fix resource specific table parsing
* More concise asset display in flows + better icons
* fix assets page filtering out resources with added table
* Fix frontend to support specific table assets
* Open DB Manager to specific table
* Specific table parser in Python and TS + unit tests
* Fix UPDATE setting access to None
* fix flow edge rendering on top of output picker
* python parser fix var override bug
* add ts test
* fix compilation
* sqlx prepare
* update parsers version
* fix missing schema key onDelete
* Grant permission to create schemas in custom instance databases
* Update pg query to return empty schemas
* Create schema
* Select nits
* support schemas in sql parser
* ts parser handle schema with sql parser result
* detect .schema() syntax
* detect schema syntax in python
* support .schema() in ts and py SDK
* open db manager to specific schema
* support reassignment in ts parser
* nit better unitest
* : syntax in ts
* datatable:schema syntax in python
* fix client py
* nit select dropdown darkmode
* object | null fetchOne
* ts client nits
* parse_sql_client_name fn
* getImportWmillTsStatement refactor in EditorBar
* text to json() in python client
* update parser versions
* pkg lock
* Sql query details in TS asset parser
* code transformation with type parameter in Editor
* Custom Language Worker, code substition works !
* Error marker mapping works
* hover info is correct
* completions work correctly
* other overrides
* type inference kinda works
* Position mapping tests
* refactor prepare_queries
* Refactor PgDatabase to share common code
* Pgdatabase in prepare_queries
* TokioPgConnection refactor
* refactor prepare_queries
* type parameter to sql function
* Fix deadlock
* nit fix
* Fix worker async call freezing because of svelte Proxy
* Force worker to recompute when we set queries
* nit refactor
* nits console logs
* wait that ts worker initialize
* monaco change file version
* update diagnostics
* Refactor for errors
* Show SQL errors in Monaco
* improve sdk
* cleaning refactor + MapResource + usePreparedAssetSqlQueries
* Fixes
* Fix error position mapping
* cache in typescript worker
* fix insert no values
* don't inject type if already present
* Support schema in prepare queries
* update parsers
* ChangeOnDeepInequality
* inferAsset ScriptEditor usage refactor
* sql query typecheck work in flow editor
* Assets and SQL Query check in Raw App Inline Editor
* pkg lock
* Fix DatatableSqlTemplateFunction nit
* prepare query schema nit
* duplicate diagnostics
* nit getScriptVersion mock
* Reprepare queries when switching workspaces
* nit fix
* nit fix
* fetch_one_scalar and execute in python client
* limit pg_connections
* -- prepare flag in postgres
* skip serializing
* fix destructuring undefined
* Prepare queries in workers instead of backend
* nit
* Execute search_path instructions normally
* nit fix
* Fix SET search_path issue in prepare
* only support preparing single-statement queries for now
* update parsers
* safety
* better remove_comments
* Fix getQueryStmtCountHeuristic
* getQueryStmtCountHeuristic tests
* comment out failing tests
* Fix getQueryStmtCountHeuristic impl
* only datatable
* data tables settings ui
* install runed
* zod 4 fixes
* use new toJSONSchema
* Migrate ducklake catalogs to more generic custom instance databases
* fix compilation
* Safety conversion for old duckdb ffi
* data tables settings
* ts client basis
* inline run works
* datatables work
* Revert "datatables work"
This reverts commit 6e1588d59e.
* datatables work (without leaking pg credentials)
* println
* separate sqlUtils.ts
* nit
* Separate custom instance db Select and Wizard components
* nit
* nit wording
* add tags to custom instance dbs
* error when trying to use ducklake as datatable or opposite
* show status in dropdown
* data table instance setup works
* sqk function for ducklake
* factorize logic
* fix temp reactivity
* Data table assetexplore
* Migrate S3 permissions to modal
* Revert "Migrate S3 permissions to modal"
This reverts commit 0631d03cb0.
* nit query -> fetch
* Custom instance setup new look
* run_language_executor separate fn
* run_inline param
* nit wording
* Better typed client
* Data tables display as assets in frontend
* asset db icon
* nit
* cleaner errors
* nit
* Fix sed calls in mac
* run_inline_script_preview in python client
* basic python datatable client
* datatable and datalake parser in python
* ducklake client python
* nit fix
* Fix migration producing NULL instead of {} when no custom databases
* merge conflict fail
* python ducklake client arg fix
* parse or infer sql types in ts client
* ts asset parser, detect datatable & ducklake R/W
* fix sql repl for other read ops than select
* export type SqlTemplateFunction
* rename list_custom_instance_pg_databases
* typecheck datatable and ducklake name in Typescript
* Fix typecheck datatable and ducklake in TS
* declare module overriding instead of extending
* infer_sql_type in python client
* SqlQuery object in python
* fix merge conflicts
* update const_format
* CI fix
* factor out to var_identifiers
* sqlx prepare
* unnecessary security (admin is required)
* clearer comment
* ee repo ref
* nit snake case
* claude step 1: detect var declarations
* move detect_sql_access_type to common mod
* claude step 2: detect when saved vars are queried
* Revert "claude step 2: detect when saved vars are queried"
This reverts commit 1e1f930568.
* Revert "claude step 1: detect var declarations"
This reverts commit f866f4819d.
* remove ducklake/datatable and default
* detect data table assigns in var_identifiers
* Python parser successfully infers R/W/RW from ducklake / datatable
* still register ducklake/datatable if not used as unknown R/W
* Go to settings button in Assets Dropdown on not found
* nit
* sqlx prepare fail
* manual fix, somehow sqlx prepare won't do it
* fix frontend ci
* ee repo ref
* ducklake_user doesnt exist in unit tests
* nit fix
* ui nit
* nit
* nit missing clone
* fork ducklakes and datatables
* fix surface hover bug
* stupid mistake
* better deeply reactive mutable derived
* Ducklake picker
* Editor bar data tables
* DuckDB supports datatables
* datatable in duckdb asset parser
* duckdb asset parser var_identifiers
* Revert "duckdb asset parser var_identifiers"
This reverts commit 88068b1a77.
* sqlx prepare
* Box pin in test_workflow_as_code to fix stack overflow
* stash
* sql asset parser parses most s3 literals
* nit
* Detect attach + handle returning RW
* detect assets used with dot notation
* detect implicit access with USE dl; syntax
* Add assets as unknown if var was never used
* Support default ducklake/datatable main in parser
* ignore asset parsing errors in frontend (avoid flow layout shift)
* super weird duplication (merge conflict ?)
* nits
* fix duckdb parser detecting too much as asset when RW ctx is unknown
* fix transparent assets btn
* missing arg
* nit styling
* asset parser specific table parsing
* fix resource specific table parsing
* More concise asset display in flows + better icons
* fix assets page filtering out resources with added table
* Fix frontend to support specific table assets
* Open DB Manager to specific table
* Specific table parser in Python and TS + unit tests
* Fix UPDATE setting access to None
* fix flow edge rendering on top of output picker
* python parser fix var override bug
* add ts test
* fix compilation
* sqlx prepare
* update parsers version
* fix missing schema key onDelete
* Grant permission to create schemas in custom instance databases
* Update pg query to return empty schemas
* Create schema
* Select nits
* support schemas in sql parser
* ts parser handle schema with sql parser result
* detect .schema() syntax
* detect schema syntax in python
* support .schema() in ts and py SDK
* open db manager to specific schema
* support reassignment in ts parser
* nit better unitest
* : syntax in ts
* datatable:schema syntax in python
* fix client py
* nit select dropdown darkmode
* object | null fetchOne
* ts client nits
* parse_sql_client_name fn
* getImportWmillTsStatement refactor in EditorBar
* text to json() in python client
* update parser versions
* pkg lock
* export S3Object + URI / Record in TS SDK
* stash getS3SignedPublicUrls
* getPresignedS3PublicUrls in TS client
* update python client for get_presigned_s3_public_urls
* data tables settings ui
* install runed
* zod 4 fixes
* use new toJSONSchema
* Migrate ducklake catalogs to more generic custom instance databases
* fix compilation
* Safety conversion for old duckdb ffi
* data tables settings
* ts client basis
* inline run works
* datatables work
* Revert "datatables work"
This reverts commit 6e1588d59e.
* datatables work (without leaking pg credentials)
* println
* separate sqlUtils.ts
* nit
* Separate custom instance db Select and Wizard components
* nit
* nit wording
* add tags to custom instance dbs
* error when trying to use ducklake as datatable or opposite
* show status in dropdown
* data table instance setup works
* sqk function for ducklake
* factorize logic
* fix temp reactivity
* Data table assetexplore
* Migrate S3 permissions to modal
* Revert "Migrate S3 permissions to modal"
This reverts commit 0631d03cb0.
* nit query -> fetch
* Custom instance setup new look
* run_language_executor separate fn
* run_inline param
* nit wording
* Better typed client
* Data tables display as assets in frontend
* asset db icon
* nit
* cleaner errors
* nit
* Fix sed calls in mac
* run_inline_script_preview in python client
* basic python datatable client
* datatable and datalake parser in python
* ducklake client python
* nit fix
* Fix migration producing NULL instead of {} when no custom databases
* merge conflict fail
* python ducklake client arg fix
* parse or infer sql types in ts client
* ts asset parser, detect datatable & ducklake R/W
* fix sql repl for other read ops than select
* export type SqlTemplateFunction
* rename list_custom_instance_pg_databases
* typecheck datatable and ducklake name in Typescript
* Fix typecheck datatable and ducklake in TS
* declare module overriding instead of extending
* infer_sql_type in python client
* SqlQuery object in python
* fix merge conflicts
* update const_format
* CI fix
* factor out to var_identifiers
* sqlx prepare
* unnecessary security (admin is required)
* clearer comment
* ee repo ref
* nit snake case
* claude step 1: detect var declarations
* move detect_sql_access_type to common mod
* claude step 2: detect when saved vars are queried
* Revert "claude step 2: detect when saved vars are queried"
This reverts commit 1e1f930568.
* Revert "claude step 1: detect var declarations"
This reverts commit f866f4819d.
* remove ducklake/datatable and default
* detect data table assigns in var_identifiers
* Python parser successfully infers R/W/RW from ducklake / datatable
* still register ducklake/datatable if not used as unknown R/W
* Go to settings button in Assets Dropdown on not found
* nit
* sqlx prepare fail
* manual fix, somehow sqlx prepare won't do it
* fix frontend ci
* ee repo ref
* ducklake_user doesnt exist in unit tests
* nit fix
* ui nit
* nit
* nit missing clone
* fork ducklakes and datatables
* fix surface hover bug
* stupid mistake
* better deeply reactive mutable derived
* Ducklake picker
* Editor bar data tables
* DuckDB supports datatables
* datatable in duckdb asset parser
* duckdb asset parser var_identifiers
* Revert "duckdb asset parser var_identifiers"
This reverts commit 88068b1a77.
* sqlx prepare
* Box pin in test_workflow_as_code to fix stack overflow
* go to settings button
* ee repo ref
* fix compilation
* wording nit
The usage example in the load_s3_file_reader docstring incorrectly showed `wmill.load_s3_file(...)`. Updated it to `wmill.load_s3_file_reader(...)` to match the actual method being documented.
* feat: Add runScriptByPath and runScriptByHash methods to SDK clients
- Add runScriptByPath and runScriptByHash methods to TypeScript client
- Add run_script_by_path and run_script_by_hash methods to Python client
- Split functionality from existing methods that took both path and hash parameters
- Add deprecation warnings to existing run_script methods
- Maintain backward compatibility while encouraging use of focused methods
Closes#6251
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
* Refactor SDK methods to eliminate code duplication
- Extract common logic into internal helper methods
- Python client: _run_script_async_internal() and _run_script_internal()
- TypeScript client: _runScriptAsyncInternal() and _runScriptInternal()
- Eliminate duplicated parameter processing and HTTP setup
- Maintain exact same public API surface and functionality
- Reduce lines of code while preserving all existing behaviors
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
* fix ts and dev.nu for python
* trade warnings.warn for logging.warning
Signed-off-by: pyranota <pyra@duck.com>
---------
Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <rubenfiszel@users.noreply.github.com>
Co-authored-by: pyranota <pyra@duck.com>
* assets migration
* parse assets (duckdb)
* iterate on assets
* S3 object Preview
* remove pagination
* filterText
* better occurence list
* tweak
* assets in JobPreview
* clone impl
* AssetsDetectedBadge
* improve DbManagerButton + asset dropdown button
* edit resource btn
* warning when incorrect resource
* +Resource in DuckDB
* +S3 Object editor bar
* nit fix rename
* flow asset badge
* More Generic OnChange
* Highlight assets used in modules
* Show occurence count in flow
* Better UX, avoid moving parts
* nit
* Asset nodes
* move to dedicated Asset ctx
* fix layoutNodes not handling first assetsMap
* explore asset btn in flow asset node
* correct offset
* single computeAssetNodes function
* Fix y positioning of nodes with assets
* resource editor
* write mode node (ui)
* accessType in ctx + fix insert button positioning
* right positioning when mixing read and write nodes
* right positioning when mixing R and W assets
* Better layout fix algorithm
* listAssetsByUsage and asset nodes on transitive usages
* refactor + remove linkAssets
* Refactor to allow for custom R/W modes
* AssetsDropdownButton in flow script editor
* R/W/RW selection and changes node pos in flow
* layoutNodes doesnt need recompute now
* fix wrong assumption that nodes recompute when assets change
* r/w/rw multi toggle
* MultiToggle cool animation + clearable
* rename + 1px nit
* remove mini toggle button group, use ToggleButtonGroup
* Combinator parser that detects R / W asset context
* nit fix missing flex-1
* missing order by
* better ui indication for access type
* special x offset case when only one asset node for clarity
* parse getResource in TS with swc ecma parser
* support load and write s3 detection in TS
* Python asset parser
* support wmill api calls without special $res: or s3:// syntax
* detect out of context asset uris python
* do not use access type override when not ambiguous in flow graph
* parse_assets match case in rust
* AsRef<str> refactor
* From impl
* Save flow assets
* Save script asset usages + fixes + save fallback access types
* asset sub icon
* max total asset node width to avoid overlap
* small refactor
* don't parse comments in duckdb assets
* fix assets clearing on parse error
* fix script asset save in wrong place
* load initial asset fallback access types
* support variables
* ui fixes
* Support S3Object as URI in TS client
* support new syntax in python client
* Support +S3Object in EditorBar for TS and python
* Reduce resource requests in assets page
* import windmill client when necessary
* update s3Types.d.ts
* nit fix
* Show input resources and s3 objects as assets
* improve asset icons
* DarkModeObserver refactor
* asset page tabs
* Moved resource variables and s3object pages to assets tabs
* fetch resource usages
* Get variables usages
* move assets usage dropdown to component
* Revert "move assets usage dropdown to component"
This reverts commit 622ea4ab12.
* Revert "Get variables usages"
This reverts commit b11ced4e29.
* Revert "fetch resource usages"
This reverts commit aa5187ad4b.
* Revert "Moved resource variables and s3object pages to assets tabs"
This reverts commit 4430487be4.
* Revert "asset page tabs"
This reverts commit dacc2f0da5.
* move assets usage dropdown to component
* asset icon in asset pages
* tooltip
* details
* Storage selector in S3 File Picker
* make edge less opaque
* Refactor computeAssetNodes to separate in and out nodes
* AssetsOverflowedNode
* nits
* fix assets not being parsed in flows sometimes
* show asset kind and resource_type
* ui nits
* support res:// in duckdb
* add banner for old deployments
* Fix permissionning
* fix broken disable /enable all
* assets page view permission for operators
* Disable ExploreAssetButton for operators
* asset kind as subtitle
* do not spam getResource in assets page. prob. revert fail
* update assets page on workspace change
* reload storage names on ws change
* delete assets on archive / deletion
* sqlx prepare
* missing update when updating user
* add indexes on asset
* better message
* missing loadInit: false
* dead code
* use transaction
* typo
* update package.json
* update package.json
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* Allow setting progress explicitly from script body.
This feature exposes:
* `getProgress`
* `setProgress`
* `incProgress`
API in TypeScript client (python is coming soon).
NOTE: Progress cannot be out of range 0..100 and cannot decrease.
With exposed APIs there is also UI changes, so progress can be shown for individual jobs as well.
For optimization reasons, jobs start to ask for progress only after N-seconds of execution.
* feat: Add `shell.nix`
If you dont have anything but nix, dont worry, run nix-shell in root, or activate with direnv and get all needed dependencies
NOTE: You will still need docker
* feat: Add `dev.nu` to typescript client
Little helper function, allowing developer to work on ts client easier.
To use:
`./dev.nu watch`
Now add import of windmill in body of your script and `//nobundle` on top of the file
Edit ts client in your favourite editor and hit save. Script will do the rest.
* Cleanup files
* Fix: Failed to deserialize query string: missing field `get_progress`
* perf: Implement non-naive polling mechanism for getting job progress
* Add independant delay for getProgress
Problem in `TestJobLoader`:
There should be 2 delays:
One until we find our first progress (every 5s)
Once we found our first progress, we can do it every second
* nit: Use `query_scalar!` instead of `query_as`
* Fix: Sql error, no rows returned by a query that expected to return at least one row
* refactor: Remove global CSS for JobProgressBar
* Change UI for progress of flow subjobs
* Replace `Step 1` with `Running` in ProgressBar for individual jobs
* Remove `incProgress`
incProgress is not very usefull and error-prone
* perf: Set metric only for jobs that are actually using it
(https://github.com/windmill-labs/windmill/pull/4373#discussion_r1759843773)
* Offload registering progress from clients to server
* Add `jobId?` argument to typescript-client's `setProgress` and `getProgress`
Allows to set progress of other jobs and flows,
if jobId specified, than flow id will be inferred automatically.
Could be used by SDK.
* Add `Error::MetricNotFound` for better error handling
* Fix: Make `JobProgressBar` display in red when failed
* Add persistant progress bar
Now you can reload the page after job is done and progress will be still there
* Allow succeeded individual job's progress bar stick to 100%
* Add python support
* nit: Remove usage of undefined variable in python-client
* Add `async` in ts client (for error handling)
* nit(frontend): Remove unused import
* Dont load JobProgressBar when it is not needed
* nit: npm check fix
* cargo sqlx prepare
* fix sqlx
---------
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
