Commit Graph

206 Commits

Author SHA1 Message Date
Ruben Fiszel
c7dfd0ebc4 fix suspend count for early message + delete resume_job 2022-10-10 20:17:49 +02:00
Ruben Fiszel
f6e794cc5f add failed_jobs to retrystatus 2022-10-10 13:41:07 +02:00
Ruben Fiszel
e6cad4df7b specify branch_chosen in openflow 2022-10-10 12:59:16 +02:00
Ruben Fiszel
cc07a6b7e4 feat(backend): implement new OpenFlow module Branches (#692)
* branches

* iterate

* branches

* progress

* progress

* progress

* done

* adapt frontend

* adapt frontend

* sqlx
2022-10-09 23:12:05 +02:00
Ruben Fiszel
7bdbfec71a feat: support struct in Go as script parameters #705 2022-10-08 20:03:08 +02:00
Ruben Fiszel
b99cb2e299 improve error messages 2022-10-08 14:36:21 +02:00
Ruben Fiszel
3f8f0eb878 improve error messages 2022-10-08 14:22:34 +02:00
Ruben Fiszel
6cb3fbc8b7 feat: add go LSP (#699)
* lsp

* progress

* 1.36.0
2022-10-08 13:28:46 +02:00
Ruben Fiszel
6ebedfc5fb feat(backend): separate properly logs from result 2022-10-07 19:54:44 +02:00
Ruben Fiszel
22c4a3b375 fix(backend): go lock dependency with no requirements 2022-10-07 16:11:45 +02:00
Ruben Fiszel
a08436622b feat: sleep for arbitrary number of seconds statically or with a javascript expression (#691)
* progress

* progress

* fix serde

* fix frontend

* fix frontend
2022-10-07 04:09:32 +02:00
Ruben Fiszel
feb5aadcc3 only stop wait on child on ok changes values 2022-10-06 23:37:10 +02:00
Ruben Fiszel
8accb59a8c fix: get info about kill reason 2022-10-06 23:25:49 +02:00
Ruben Fiszel
b31e72a620 fix: get info about kill reason 2022-10-06 23:15:27 +02:00
Ruben Fiszel
2c5b32bdb7 fix(backend): same_worker uses the same folder even within loops 2022-10-06 21:39:07 +02:00
Ruben Fiszel
1b2994ac8a add GOMEMLIMIT to 2Gb 2022-10-06 05:13:06 +02:00
Ruben Fiszel
f4caa4ffa6 feat: implement same_worker openflow attribute for running flow all in one go + sharing folder /shared (#689)
* init

* progress

* all

* sqlx

* remove unecessary test

* done

* frontend

* frontend
2022-10-06 04:59:09 +02:00
Ruben Fiszel
91ce66e7fe spurrious printlns 2022-10-04 14:44:56 +02:00
Ruben Fiszel
e385eb54af progress 2022-10-04 08:40:01 +02:00
Ruben Fiszel
79636e6d86 progress 2022-10-04 08:40:01 +02:00
Ruben Fiszel
89a27837af minor UI fixes 2022-10-02 21:35:07 +02:00
Ruben Fiszel
612f7272a9 feat(backend): add WM_BASE_URL 2022-10-02 18:53:05 +02:00
Ruben Fiszel
6fe5b8d6b7 fix(backend): allow for now payload on resume GET 2022-10-02 05:14:36 +02:00
Ruben Fiszel
945a750c6b fix(go): make lines align with appended code 2022-10-02 01:28:14 +02:00
Ruben Fiszel
eabd835807 fix(go): inner_main is in a separate file rather than wrapped 2022-10-02 01:22:33 +02:00
Ruben Fiszel
d863b1ed90 feat(backend): add WM_FLOW_JOB_ID 2022-10-01 09:02:00 +02:00
Ruben Fiszel
eaadad6143 hotfix trusted host for pip-compile 2022-09-28 09:17:08 +02:00
Ruben Fiszel
ae3f86db11 feat: add private registries pip (#636)
* add private registries pip

* add private registries pip

* fix

* fix

* fix

* echo parameters
2022-09-28 08:47:43 +02:00
Ruben Fiszel
b87a498087 do not restart flow as zombie job 2022-09-26 00:45:26 +02:00
Ruben Fiszel
4044edf8d6 refactor logs panel everywhere applicable (#625)
* init

* init

* progress

* progress

* refactor logs entirely

* fix sqlx
2022-09-25 18:03:51 +02:00
Ruben Fiszel
3a530c8566 fix cancel job 2022-09-25 16:36:32 +02:00
sqwishy
e7a6c1b99f reorganize handle_child (#606)
* reorganize handle_child

There were a couple issues with the current implementation:

1. When reading stdout and stderr from the child, as soon as we hit EOF
   on one we would stop reading from both (line 1420). This could lead
   to the return value not being read from the job program.

2. Lines read from stdout and stderr are put into a channel and read
   elsewhere with `rx.recv()` (line 1497) but that channel isn't read
   until empty.  It is only read in the `while !done.load(...)` (line
   1449) loop and that loop can stop after any `.store(true, ...)`.
   Which happens when the child exits, when the job is cancelled, when
   either stdout or stderr reach EOF...

   This can be verified by putting `dbg!(rx.recv().await)` or a similar
   assertion after the while loop before returning from that function.
   It shows the channel still containing log lines on rare occasions.

I was pretty careful in this to maintain the current behaviour; adding
comments to express intention.

One difference in this is that some regular intervals (cancel check and
ping update) should be more regular?

Before...

> at 00ms wait for 10ms
> at 10ms do things for 3ms
> at 13ms wait again for *10ms*
> at 23ms do things again ...

With change...

> at 00ms wait for 10ms
> at 10ms do things for 3ms
> at 13ms wait again but for *7ms*
> at 20ms do things again ...

Which I'm guessing is preferable but I could be wrong.

* renames; interpolate values in log messages

* do `append_logs()` in tokio::task

* tokio::time::interval & close pipe after limit

* clean up comments
2022-09-24 22:55:52 +02:00
sqwishy
47fba21256 remove delay between spawning each worker (#614) 2022-09-21 21:05:19 -07:00
Ruben Fiszel
e1f54832fa more details on error handling job 2022-09-18 10:11:21 +02:00
sqwishy
126dd24c71 feat(backend): flow suspend resume (#522)
Flow observes `suspend` setting and will wait for resume messages sent for the job before continuing to the next step in a flow.

Adds endpoints under workspaces at `/jobs/<cancel|resume>/<job-uuid>` to either cancel or resume the job with a payload. For POST requests to the endpoint, payload is a JSON document. For GET requests to the endpoints, the payload is a base64url encoded JSON document as the value of the payload query parameter.
2022-09-14 11:46:57 -07:00
Ruben Fiszel
39918a9bb1 feat: implement go support (#571)
* progress

* progress

* all in one

* frontend

* small nits

* go job test

* go.sum is optional

* add golang-go to backend test image

Co-authored-by: sqwishy <somebody@froghat.ca>
2022-09-13 21:14:21 +02:00
Ruben Fiszel
110a25f6f8 fix: iterator input transform is made more generic (#524)
* fix: iterator expr is standardized with regular input transforms

* fix: iterator expr is standardized with regular input transforms

* v2
2022-09-13 20:11:00 +02:00
ex0ns
6f09405c2d feat(job): run job by hash (#551)
It was possible to run them using a webhook, but not through an endpoint.
This PR aims to fix that so the user can target a specific version of
the script to run.
2022-09-11 16:39:48 +02:00
Ruben Fiszel
e9abcffdd1 feat: is_trigger is just a type tag, soon to include failure and command (#523)
* script kind

* all

* init code flow

* kind: trigger

* kind: trigger
2022-09-05 09:55:04 +02:00
Ruben Fiszel
482dc808ae logs around fetching job 2022-09-04 13:30:48 +02:00
Ruben Fiszel
12e2a31776 logs around job execution contain job id 2022-09-04 13:11:43 +02:00
Ruben Fiszel
b6725dceca use local time for timeout 2022-09-04 13:05:39 +02:00
Ruben Fiszel
1bc12179c7 fix: last ping is set when the job is started avoiding erronous restart 2022-09-04 12:51:55 +02:00
sqwishy
d69d002b82 flow step retry feature (#493)
* flow step retry feature

* comparison constant on right side for clarity

* raise high retry values when starting a flow

also renamed duration to interval to be more specific about the retry
interval/period between tries or attempts

* add flow retry to openflow openapi

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2022-09-03 20:38:31 +02:00
Ruben Fiszel
cf7209bdb9 feat: clean openflow spec v1 (#491)
* clean api 2

* the rest

* clean tests

* stop_after_if test

* unbox `modules: Vec<FlowModule>` in `ForloopFlow`

* migrate

* initFlow stop_after_if_expr and skip_if_stopped

* s/migrateInitTransform/migrateFlowModule

I didn't read the name before... oops

* sql migration for openflow changes

* fix frontend migration code

Co-authored-by: sqwishy <somebody@froghat.ca>
2022-09-02 01:26:39 +02:00
Ruben Fiszel
76f8165b82 pass language to completed job 2022-08-25 14:55:38 +02:00
Ruben Fiszel
b8b1cc83ae list flow steps only on the tab 'all' 2022-08-23 15:18:44 +02:00
Ruben Fiszel
9498975fee add base_url to DENO_AUTH_TOKENS 2022-08-22 02:47:44 +02:00
Ruben Fiszel
35277160a6 feat: pass bearerToken as queryArg 2022-08-21 22:39:19 +02:00
Ruben Fiszel
b9dfbfa2d8 feat: implicit types infered from default parameters 2022-08-21 22:01:24 +02:00