Commit Graph

65 Commits

Author SHA1 Message Date
Diego Imbert
fdefd4be93 feat: duckdb sql lang support (#5761) 2025-05-27 15:52:57 +02:00
Ruben Fiszel
5b96bccedd feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
Diego Imbert
b29c6e7636 fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response

* Return S3Object instead of path string
2025-05-16 19:28:57 +02:00
Diego Imbert
c7886ea07a feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate

* S3 works with new syntax

* snowflake s3 streaming support

* postgres s3 support

* fix postgres stream format

* mysql s3 streaming

* mssql s3 streaming

* new s3 mode syntax

* optional folder param

* rename folder to prefix

* json_stream_arr_values

* cargo toml rollback

* convert_ndjson with datafusion

* format conversion kinda works

* Fixed not finishing the datafusion writer

* support for pg and mssql

* fix file ext

* bigquery conversion and works with s3 streaming

* fix s3 flag parser

* snowflake s3 streaming support

* factor out duplicate code

* remove anyhow

* Err case for parse s3 mode

* Send error to mpsc

* bigquery s3 streaming fix for huge queries

* remove extra stuff

* snowflake s3 streaming support

* small regex mistake

* cfg(not(feature = "parquet"))

* fix CI (unused import)

* error handling fix (graphite)
2025-05-13 10:19:44 +02:00
Ruben Fiszel
a3c76fb10c fix: improve error display of nativets exceptions 2025-05-11 11:35:31 +02:00
Ruben Fiszel
63fa499015 feat: agent workers v2 using http (#5588) 2025-04-10 00:56:11 +00:00
Ruben Fiszel
3edca4bc91 perf: optimize number of queries needed for job run (#5504)
* optPerf

* update sqlx

* update sqlx
2025-03-24 20:12:26 +01:00
wendrul
38ee0183aa feat: unsafe parameters for sql queries (table names, column names) (#5488)
* Make schema validation struct

Schema Validation rules that are constructed from the schema or from the
MainArgSig(TODO).

* Make other validator builder

* Fail dependency job like with lockfile failing for schema validator

* Add last types + tests

* Remove unused dependency

* fix typos

* Migration ID was colliding with another, changed it manually

* Add Oneof + other fixes

* fix: cache for querying scripts correclty handles ScriptMetadata

* Add cache for schema validation from main arg sig

* Prepare sqlx

* Remove default features

* Feature flags

* WIP: unsafe sql params for sql langauges

* Fix down migration table name

* cleanup: put validation logic inside a function

* Refactor to cache the should_validate boolean

Changed the schemavalidators cache to take in an
Option<SchemaValidator>, effectively storing the `should_validate_schema` information.

Also pass the schema when avaialble to construct the schema validator

* Add other job kinds to u8 cache key just in case

* Change sql languages to all get arguments as Values instead of RawValue

* Only cache if not preview

* Add last sql languages and some CI fixes

* Rename after typo on `sanitized`

* Finish rename

* Remove unused import

* Fix wrong test

* Add newly published regex parser version

* Remove default features from cargo.toml

* Change to a cleaner syntax for the interpolated args

* Update republished parser
2025-03-20 20:02:25 +01:00
Ruben Fiszel
880db319e8 feat: filter by worker + backend perf opt (#5489)
* all

* all

* all

* refactor

* sqlx

* ref

* impr

* fix

* fix

* fix

* ee private
2025-03-19 09:55:53 +01:00
Ruben Fiszel
eb2ffee4ec pg nits 2025-03-14 12:13:12 +01:00
Ruben Fiszel
8e95bc3972 fix: avoid lock contention for native workers on cached connection (#5481) 2025-03-14 11:58:03 +01:00
Ruben Fiszel
2c5d07f3d9 feat: docker runtime 2024-12-09 18:02:18 +01:00
HugoCasa
17d893315b fix: revert bool to text support in pg (#4727) 2024-11-16 10:01:16 +01:00
HugoCasa
d999e91583 unbreak nullable inputs in pg (#4726) 2024-11-15 22:27:28 +01:00
HugoCasa
9ad1afc8f2 unbreak ::text for json in pg (#4722)
* unbreak ::text for json in pg

* better error handling
2024-11-15 18:35:21 +01:00
HugoCasa
921e3217c6 improve array params handling in postgres (#4605) 2024-10-30 14:44:42 +01:00
Ruben Fiszel
85de44cb93 feat: autoscaling v0 (#4593)
* all

* all

all

all

all

all

all

all

all

all

* all

* all

* nits

* nits
2024-10-28 00:47:19 +01:00
pyranota
c3ba832444 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel
bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa
97cdafac86 feat: add return_last_result annotation to sql (#4443) 2024-09-25 18:11:11 +02:00
Ruben Fiszel
fc6e480e56 fix: postgres scripts that take longer than 20s do not timeout anymore 2024-09-18 14:32:16 +02:00
Ruben Fiszel
bef1bcb2a4 fix: timeout in pg executor on postgresql connection after 20s 2024-09-12 18:49:43 +02:00
HugoCasa
8942e80ebb fix: handle time with tz col type in pg (#4239) 2024-08-14 13:25:56 +02:00
HugoCasa
2b9b4eedc3 feat: multi sql statement with pg fix (#4134)
* Revert "Revert "feat: multi statement sql (#4104)" (#4133)"

This reverts commit c578f05e2d.

* fix: sort pg args
2024-07-26 09:23:39 +02:00
Ruben Fiszel
c578f05e2d Revert "feat: multi statement sql (#4104)" (#4133)
This reverts commit 5bc0e96171.
2024-07-26 01:24:03 +02:00
HugoCasa
5bc0e96171 feat: multi statement sql (#4104)
* feat: multi statement pg

* fix: add other flavors

* feat: make pg params start at 1 and sequential

* fix: improve sql statement parsing

* add tests

* fix: allow no semi in last statement

* fix: merge conflict

* fix: minor improvement

* fix: parser version
2024-07-25 20:46:04 +02:00
HugoCasa
6da5e15903 feat: add support for bytea in pg (#3926)
* feat: add support for bytea in pg

* fix: editor nits
2024-06-18 17:38:46 +02:00
Ruben Fiszel
6a332bc30f fix: make all error strings more verbose 2024-05-25 22:15:15 +02:00
Ruben Fiszel
75730d410d rm go bin if cloud hosted 2024-04-26 13:42:24 +02:00
HugoCasa
e8905b2734 feat: pg add json support (#3620) 2024-04-26 13:28:15 +02:00
Ruben Fiszel
8cb1e4846c increase pg size limit for cloud 2024-04-21 10:20:50 +02:00
Ruben Fiszel
da3ded6196 fix: measure memory usage on postgres scripts 2024-04-18 01:05:16 +02:00
HugoCasa
c2d18a6d6f feat keep sql columns ordering (#3444)
* feat: keep pg columns ordering

* feat: pass column order through flow status

* feat: sql ordering for all sql langs

* fix: sqlx build on mac + rename to _metadata

* Update JobPreview.svelte

* Update +page.svelte

* Update +page.svelte

* Update LogPanel.svelte

* Update +page.svelte

* Update LogPanel.svelte

* Update JobPreview.svelte

* fix: build

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-03-22 10:34:32 +01:00
HugoCasa
c0104641f0 feat: add bit support in pg (#3407) 2024-03-14 10:40:20 +01:00
HugoCasa
c03c73797b fix: pg timstamptz param (#3364) 2024-03-06 12:50:01 +01:00
HugoCasa
1ee45a1f7d feat: allow to pin database in sql scripts (#3304) 2024-02-27 19:59:52 +01:00
Ruben Fiszel
d414e9fff9 refactor future wrapper for queries 2024-02-19 22:11:45 +01:00
Ruben Fiszel
37e80ca1a4 fix(psql): add mem peak to postgresql 2024-02-19 20:51:09 +01:00
Ruben Fiszel
fa9d5af10e respect query timeout for pg queries 2024-02-19 20:11:28 +01:00
Ruben Fiszel
52b2b5048d fix: prevent postgres timeout abuse 2024-02-19 19:59:43 +01:00
HugoCasa
af1b4c0dde fix: add timestamp array support in pg (#3229) 2024-02-15 17:18:41 +01:00
HugoCasa
d7834f3694 fix: pg coerce nb to string (#3127) 2024-02-01 14:08:14 +01:00
Faton Ramadani
35756cf411 feat(frontend): DB Explorer (#2892)
* feat(frontend): Make table cell editable

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): clean up

* fix(frontend): extract db schemas

* feat(frontend): v0 done

* feat(frontend): v0 done

* feat(frontend): v0 done

* feat(frontend): fix insert

* feat(frontend): fix insert

* feat(frontend): remove temp data

* feat(frontend): align insert button to the right

* feat(frontend): rework columns

* feat(frontend): rework insert

* feat(frontend): rework insert

* feat(frontend): rework insert

* feat(frontend): rework insert

* feat(frontend): fix jsonb display

* feat(frontend): fix reloading issues

* feat(frontend): fix reloading issues

* feat(frontend): fix reacticity issue

* feat(frontend): fix insert

* feat(frontend): support nullable default value

* feat(frontend): fix build

* update

* update

* db studio v0

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-01-06 17:27:15 +01:00
HugoCasa
3cfcf28c57 fix: postgres ssl mode (#2861) 2023-12-15 15:15:55 +01:00
Ruben Fiszel
b9f7fd882a fix: support interval in pg 2023-12-08 11:47:41 +01:00
Ruben Fiszel
056d03b386 fix: support INET in pg 2023-11-30 20:18:43 +01:00
Ruben Fiszel
a285ca24b0 feat: cache postgres connection (#2621)
* feat: cache pg connection

* fix wmill dev

* adjust timings

* don't keep connection alive if not latest con

* reduce sleep
2023-11-13 19:40:25 +01:00
Ruben Fiszel
2ee57f3f84 feat: refactor entirely json processing in favor or rawjson to handle larger payloads (#2446)
* raw json

* raw json

* raw json

* raw json

* the big rawjson rewrite

* resolve merge conflicts

* all

* progress

* progress

* all compile

* remove mut

* remove mut

* remove mut

* arc queuedjob

* finish

* finish

* finish

* fixes

* progress

* fix result

* small fixes

* small fixes

* fix tests

* fix tests

* fix tests

* optimize eval timeout

* optimize eval timeout

* optimize eval timeout

* handle big args and result in get jobs

* improve args

* fix

* fix

* fix tests

* fix tests

* done

* add bigquery to native picker

* add bigquery

---------

Co-authored-by: gbouv <guillaume@windmill.dev>
2023-10-16 10:46:55 +02:00
HugoCasa
9002a0e05a fix: add numeric, array and date types (#2379)
* fix: add numeric, array and date types

* fix: update parser npm
2023-10-05 09:13:56 +02:00
HugoCasa
8319f5d0e5 fix: add reserved variables in args (#2371)
* fix: add reserved variables in args

* fix: only get job when necessary
2023-10-03 18:14:37 +02:00