Commit Graph

258 Commits

Author SHA1 Message Date
Ruben Fiszel
00b70d9aaa feat(backend): increase timeout for premium workspace 2023-03-26 18:56:40 +02:00
Ruben Fiszel
47a7f7163a feat(backend): introduce RESTART_ZOMBIE_JOBS and ZOMBIE_JOB_TIMEOUT 2023-03-23 12:47:49 +01:00
Ryan Rich
944795f6ee feat(deno): add support for custom npm repo (#1291)
* feat(deno): add support for custom npm repo

* feat(deno): add support for custom npm repo

* feat(deno): fix comment being on wrong line

* feat(deno): review feedback
2023-03-14 22:23:44 +01:00
Ryan Rich
580388ce19 Add support for binding server listener to a specific IP address (#1253) 2023-03-02 08:01:50 +01:00
Ruben Fiszel
05f568fb8c display startup info in all cases 2023-02-28 17:52:47 +01:00
Ruben Fiszel
c0b87cc7d7 send a webhook on new user + lazy static refactor (#1203)
* supercharge

* supercharge

* progress

* progress

* display config

* display config

* display config

* display config

* fix extensions

* fix build

* disable nsjail = false for test
2023-02-15 10:59:50 +01:00
Ruben Fiszel
3f318b4ac4 add license key check 2023-01-15 12:06:37 +01:00
Ruben Fiszel
cd7bc33b6a re-enable picking groups 2022-12-20 07:18:10 +01:00
Ruben Fiszel
876e140626 change the default number of connections to 3 for workers 2022-12-19 18:59:46 +01:00
Ruben Fiszel
4daf5c18fa usage tracker v3 2022-12-12 01:11:44 +01:00
Ruben Fiszel
8a4282ef6a disable nsjail by default 2022-11-25 16:31:15 +01:00
Ruben Fiszel
8846ca5856 feat: supercache extended to all version 2022-11-18 20:17:23 +01:00
Kai Jellinghaus
bd09884955 feat(backend): sync cache features on all workers [enterprise] (#907)
* Periodic Job

* Add write lock & bash execution

* Use env for bash script

* Enterprise Feature

* Only time based

* Cleanup

* Move code

* Use rclone sync

* WIP

* Use buildarg

* Remove unecessary deps

* 10 Minute Intervals

* Revert testing change

* Manually pull rclone

* fix rclone

* Initial sync

* Update worker.rs

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2022-11-18 17:20:39 +01:00
Kai Jellinghaus
0034129a30 restructure the entire backend layout using workspaces (#815) 2022-10-29 11:58:06 +02:00
Ruben Fiszel
8475e31740 UX flow improvements + fix collecting results for loops in frontend + fix Identity behavior in backend 2022-10-27 18:50:32 +02:00
Ruben Fiszel
188171126c fix export to tarball 2022-10-27 00:43:51 +02:00
Ruben Fiszel
0199007d46 improve python parsing of imports 2022-10-27 00:19:17 +02:00
Ruben Fiszel
9a67607b20 feat: payload capture of json to initialize flow input (#655)
* capture table and some endpoints for json pastebin

* only retain user's most recent captures

* merge conflict

* sqlx data

* fix migration

Co-authored-by: sqwishy <somebody@froghat.ca>
2022-10-25 00:54:53 +02:00
Ruben Fiszel
40bfc486c7 add resume_id to job_resume + add identity module 2022-10-24 14:48:28 +02:00
Ruben Fiszel
0f4ffd29dd fix get_suspended_job_flow module finding 2022-10-24 10:37:21 +02:00
Ruben Fiszel
e7d000a6d7 fix get_suspended_job_flow 2022-10-24 10:18:23 +02:00
Ruben Fiszel
17d9f38d30 fix(deno-client): add approver 2022-10-24 02:24:44 +02:00
Ruben Fiszel
a0b2c9e77d feat: add approver to approval step 2022-10-24 01:50:39 +02:00
Ruben Fiszel
884edd7715 fix: improve approval flow with approval page 2022-10-22 13:55:12 +02:00
Ruben Fiszel
90c7b0c89d fix send tx for kill 2022-10-22 00:34:36 +02:00
Ruben Fiszel
e0b86e958f feat(backend): propagate cancel instantly to all flow jobs if any 2022-10-21 22:00:52 +02:00
Ruben Fiszel
8f40d3b117 fix import script from hub 2022-10-21 19:38:33 +02:00
Kai Jellinghaus
ac84b76909 fix(backend): avoid mem leak on interval #786 2022-10-21 12:15:03 +02:00
Ruben Fiszel
8ec33c5e16 feat: support running and publishing go, python scripts to the hub (#779) 2022-10-20 20:16:08 +02:00
Kai Jellinghaus
3fb5ec08b0 Add more tracing (#773) 2022-10-20 07:40:58 +02:00
Kai Jellinghaus
55d3f38b8b More Metrics (#735)
* Expose Worker Tokio Metrics

* Add metrics tracking the job queue

This can be used to (approximate) the queue length
This can be used to (approximate) the running jobs

* Remove testing code

* Rename metrics to exclude TOTAL_

* Sleep in worker metrics loop

* Add jobs_executed

* Add zombie job metrics

* Add uptime metric

* Fix metric naming

* Remove poll stats

* unify worker execution metrics

* Rename jobs_* metrics

* Rename metrics to match exposed name

* Remove leftover import

* Rename variables to match further

* Fix Merge error

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2022-10-19 10:15:42 +02:00
Ruben Fiszel
2350a39335 remove erronous pull created_by 2022-10-19 04:10:41 +02:00
Ruben Fiszel
c03c3afcb4 make get job more reliable 2022-10-19 03:54:41 +02:00
Ruben Fiszel
f4ff43ec49 nits 2022-10-19 03:41:56 +02:00
Ruben Fiszel
45a6976d52 feat(backend): atomic moving queue -> complete and delete (#771)
* feat(backend): atomic move to queue + delete

* fix

* fix

* fix

* fix
2022-10-19 03:16:15 +02:00
Ruben Fiszel
955cc4104a fix(backend): reschedule flow at first step end (#746)
* fix(backend): reschedule flow at first step end

* fix

* fix

* fix

* fix

* sqlx prepare
2022-10-18 17:06:39 +02:00
Ruben Fiszel
57600ab873 feat: add result by id to fetch result from any node (#769)
* feat: add result by id to fetch result from any node

* sqlx prepare

* fix

* fix

* review
2022-10-18 17:04:13 +02:00
Ruben Fiszel
af9e1f4479 feat: add ids to modules + input_transform lowered to flowmodulevalue (#768)
* feat: add ids to modules + inputTransforms from module to module value

* all

* delete obsolete input transforms
2022-10-17 22:34:12 +02:00
Ruben Fiszel
a5aad947e6 feat(backend): add branch all (#751)
* feat(backend): add branch all

* progress

* all
2022-10-17 17:31:31 +02:00
Ruben Fiszel
9e0c2d759b feat: InProgress forloop_jobs -> flow_jobs to unify with branchAll 2022-10-16 16:04:24 +02:00
Ruben Fiszel
b1805699c9 feat(backend): rework forloop flow job arg passing + reimplement branchone using flows 2022-10-16 15:56:41 +02:00
Jakub Kołodziejczak
ed108de1b9 chore(backend): allow disabling nsjail for unit tests (#742)
UC: as a developer I want to be able to test code w/o nsjail

bug: flaky test `test_go_job`
```
running 1 test
test worker::tests::test_go_job ... FAILED

failures:

---- worker::tests::test_go_job stdout ----
received killpill for worker 0
thread 'worker::tests::test_go_job' panicked at 'assertion failed: `(left == right)`
  left: `Object {"error": String("Error during execution of the script:\n\ngo: error obtaining buildID for go tool compile: pipe2: too many open files\ngo: error obtaining buildID for go tool compile: pipe2: too many open files\ngo: error obtaining buildID for go tool compile: pipe2: too many open files")}`,
 right: `String("hello world")`', src/worker.rs:2434:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```

workaround: disable nsjail
2022-10-15 16:08:05 +02:00
Ruben Fiszel
5707dfe4f1 add go-client mvp + stabilize openapi (#732) 2022-10-14 16:26:51 +02:00
Ruben Fiszel
444750d594 improve supercache 2022-10-13 03:04:55 +02:00
Ruben Fiszel
0cf9f0c8e2 improve supercache 2022-10-13 01:23:08 +02:00
Ruben Fiszel
c1cc60045a improve supercache 2022-10-13 00:47:00 +02:00
Ruben Fiszel
d116218a0e improve supercache 2022-10-13 00:15:21 +02:00
Ruben Fiszel
889965feb3 fix supercache in presence of other deps 2022-10-12 23:28:50 +02:00
Jakub Kołodziejczak
7e35d9989a feat(backend): supercache for python heavy dependencies in alpha 2022-10-12 23:02:27 +02:00
Ruben Fiszel
2cc0fe7bd7 fix handle shared path not existing because job was resumed on another worker 2022-10-12 20:09:07 +02:00