Commit Graph

216 Commits

Author SHA1 Message Date
Ruben Fiszel
5707dfe4f1 add go-client mvp + stabilize openapi (#732) 2022-10-14 16:26:51 +02:00
Ruben Fiszel
444750d594 improve supercache 2022-10-13 03:04:55 +02:00
Ruben Fiszel
0cf9f0c8e2 improve supercache 2022-10-13 01:23:08 +02:00
Ruben Fiszel
c1cc60045a improve supercache 2022-10-13 00:47:00 +02:00
Ruben Fiszel
d116218a0e improve supercache 2022-10-13 00:15:21 +02:00
Ruben Fiszel
889965feb3 fix supercache in presence of other deps 2022-10-12 23:28:50 +02:00
Jakub Kołodziejczak
7e35d9989a feat(backend): supercache for python heavy dependencies in alpha 2022-10-12 23:02:27 +02:00
Ruben Fiszel
2cc0fe7bd7 fix handle shared path not existing because job was resumed on another worker 2022-10-12 20:09:07 +02:00
Ruben Fiszel
1438898617 fix suspend + sleep with same worker 2022-10-12 15:56:15 +02:00
Ruben Fiszel
ff7fb0f6f3 feat: secure suspended resume event + configurable timeout (#721)
* fix suspend count for early message + delete resume_job

* all

* done everything

* done everything
2022-10-11 22:55:14 +02:00
Ruben Fiszel
c7dfd0ebc4 fix suspend count for early message + delete resume_job 2022-10-10 20:17:49 +02:00
Ruben Fiszel
f6e794cc5f add failed_jobs to retrystatus 2022-10-10 13:41:07 +02:00
Ruben Fiszel
e6cad4df7b specify branch_chosen in openflow 2022-10-10 12:59:16 +02:00
Ruben Fiszel
cc07a6b7e4 feat(backend): implement new OpenFlow module Branches (#692)
* branches

* iterate

* branches

* progress

* progress

* progress

* done

* adapt frontend

* adapt frontend

* sqlx
2022-10-09 23:12:05 +02:00
Ruben Fiszel
7bdbfec71a feat: support struct in Go as script parameters #705 2022-10-08 20:03:08 +02:00
Ruben Fiszel
b99cb2e299 improve error messages 2022-10-08 14:36:21 +02:00
Ruben Fiszel
3f8f0eb878 improve error messages 2022-10-08 14:22:34 +02:00
Ruben Fiszel
6cb3fbc8b7 feat: add go LSP (#699)
* lsp

* progress

* 1.36.0
2022-10-08 13:28:46 +02:00
Ruben Fiszel
6ebedfc5fb feat(backend): separate properly logs from result 2022-10-07 19:54:44 +02:00
Ruben Fiszel
22c4a3b375 fix(backend): go lock dependency with no requirements 2022-10-07 16:11:45 +02:00
Ruben Fiszel
a08436622b feat: sleep for arbitrary number of seconds statically or with a javascript expression (#691)
* progress

* progress

* fix serde

* fix frontend

* fix frontend
2022-10-07 04:09:32 +02:00
Ruben Fiszel
feb5aadcc3 only stop wait on child on ok changes values 2022-10-06 23:37:10 +02:00
Ruben Fiszel
8accb59a8c fix: get info about kill reason 2022-10-06 23:25:49 +02:00
Ruben Fiszel
b31e72a620 fix: get info about kill reason 2022-10-06 23:15:27 +02:00
Ruben Fiszel
2c5b32bdb7 fix(backend): same_worker uses the same folder even within loops 2022-10-06 21:39:07 +02:00
Ruben Fiszel
1b2994ac8a add GOMEMLIMIT to 2Gb 2022-10-06 05:13:06 +02:00
Ruben Fiszel
f4caa4ffa6 feat: implement same_worker openflow attribute for running flow all in one go + sharing folder /shared (#689)
* init

* progress

* all

* sqlx

* remove unecessary test

* done

* frontend

* frontend
2022-10-06 04:59:09 +02:00
Ruben Fiszel
91ce66e7fe spurrious printlns 2022-10-04 14:44:56 +02:00
Ruben Fiszel
e385eb54af progress 2022-10-04 08:40:01 +02:00
Ruben Fiszel
79636e6d86 progress 2022-10-04 08:40:01 +02:00
Ruben Fiszel
89a27837af minor UI fixes 2022-10-02 21:35:07 +02:00
Ruben Fiszel
612f7272a9 feat(backend): add WM_BASE_URL 2022-10-02 18:53:05 +02:00
Ruben Fiszel
6fe5b8d6b7 fix(backend): allow for now payload on resume GET 2022-10-02 05:14:36 +02:00
Ruben Fiszel
945a750c6b fix(go): make lines align with appended code 2022-10-02 01:28:14 +02:00
Ruben Fiszel
eabd835807 fix(go): inner_main is in a separate file rather than wrapped 2022-10-02 01:22:33 +02:00
Ruben Fiszel
d863b1ed90 feat(backend): add WM_FLOW_JOB_ID 2022-10-01 09:02:00 +02:00
Ruben Fiszel
eaadad6143 hotfix trusted host for pip-compile 2022-09-28 09:17:08 +02:00
Ruben Fiszel
ae3f86db11 feat: add private registries pip (#636)
* add private registries pip

* add private registries pip

* fix

* fix

* fix

* echo parameters
2022-09-28 08:47:43 +02:00
Ruben Fiszel
b87a498087 do not restart flow as zombie job 2022-09-26 00:45:26 +02:00
Ruben Fiszel
4044edf8d6 refactor logs panel everywhere applicable (#625)
* init

* init

* progress

* progress

* refactor logs entirely

* fix sqlx
2022-09-25 18:03:51 +02:00
Ruben Fiszel
3a530c8566 fix cancel job 2022-09-25 16:36:32 +02:00
sqwishy
e7a6c1b99f reorganize handle_child (#606)
* reorganize handle_child

There were a couple issues with the current implementation:

1. When reading stdout and stderr from the child, as soon as we hit EOF
   on one we would stop reading from both (line 1420). This could lead
   to the return value not being read from the job program.

2. Lines read from stdout and stderr are put into a channel and read
   elsewhere with `rx.recv()` (line 1497) but that channel isn't read
   until empty.  It is only read in the `while !done.load(...)` (line
   1449) loop and that loop can stop after any `.store(true, ...)`.
   Which happens when the child exits, when the job is cancelled, when
   either stdout or stderr reach EOF...

   This can be verified by putting `dbg!(rx.recv().await)` or a similar
   assertion after the while loop before returning from that function.
   It shows the channel still containing log lines on rare occasions.

I was pretty careful in this to maintain the current behaviour; adding
comments to express intention.

One difference in this is that some regular intervals (cancel check and
ping update) should be more regular?

Before...

> at 00ms wait for 10ms
> at 10ms do things for 3ms
> at 13ms wait again for *10ms*
> at 23ms do things again ...

With change...

> at 00ms wait for 10ms
> at 10ms do things for 3ms
> at 13ms wait again but for *7ms*
> at 20ms do things again ...

Which I'm guessing is preferable but I could be wrong.

* renames; interpolate values in log messages

* do `append_logs()` in tokio::task

* tokio::time::interval & close pipe after limit

* clean up comments
2022-09-24 22:55:52 +02:00
sqwishy
47fba21256 remove delay between spawning each worker (#614) 2022-09-21 21:05:19 -07:00
Ruben Fiszel
e1f54832fa more details on error handling job 2022-09-18 10:11:21 +02:00
sqwishy
126dd24c71 feat(backend): flow suspend resume (#522)
Flow observes `suspend` setting and will wait for resume messages sent for the job before continuing to the next step in a flow.

Adds endpoints under workspaces at `/jobs/<cancel|resume>/<job-uuid>` to either cancel or resume the job with a payload. For POST requests to the endpoint, payload is a JSON document. For GET requests to the endpoints, the payload is a base64url encoded JSON document as the value of the payload query parameter.
2022-09-14 11:46:57 -07:00
Ruben Fiszel
39918a9bb1 feat: implement go support (#571)
* progress

* progress

* all in one

* frontend

* small nits

* go job test

* go.sum is optional

* add golang-go to backend test image

Co-authored-by: sqwishy <somebody@froghat.ca>
2022-09-13 21:14:21 +02:00
Ruben Fiszel
110a25f6f8 fix: iterator input transform is made more generic (#524)
* fix: iterator expr is standardized with regular input transforms

* fix: iterator expr is standardized with regular input transforms

* v2
2022-09-13 20:11:00 +02:00
ex0ns
6f09405c2d feat(job): run job by hash (#551)
It was possible to run them using a webhook, but not through an endpoint.
This PR aims to fix that so the user can target a specific version of
the script to run.
2022-09-11 16:39:48 +02:00
Ruben Fiszel
e9abcffdd1 feat: is_trigger is just a type tag, soon to include failure and command (#523)
* script kind

* all

* init code flow

* kind: trigger

* kind: trigger
2022-09-05 09:55:04 +02:00
Ruben Fiszel
482dc808ae logs around fetching job 2022-09-04 13:30:48 +02:00