bbb397b6ad
* fix: reuse outer tx for schedule push in commit_completed_job Instead of calling handle_maybe_scheduled_job(db) which opens its own connections (peak=3), inline the schedule push using a savepoint on the outer transaction. Auth is fetched via the tx connection using fetch_authed_from_permissioned_as_conn, and push_scheduled_job runs on a savepoint so failures roll back only the push, not the completion. On push failure: savepoint rolls back, schedule is disabled on the outer tx, and the zombie return path is preserved if disabling also fails. Peak connections drop from 3 to 1 (or 2 on cold RunnableSettings cache). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * fix: extract shared try_schedule_next_job to unify schedule push paths Replace the two diverging schedule-push implementations (inlined in commit_completed_job and standalone handle_maybe_scheduled_job) with a single try_schedule_next_job that reuses the caller's transaction via savepoints. This eliminates extra pool connection usage in the worker_flow.rs path and ensures consistent retry/error semantics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add failpoint markers to try_schedule_next_job Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove plan.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove inner retry loop from try_schedule_next_job, add caller-level retries The 10-retry x 5s-sleep loop inside try_schedule_next_job held locks on v2_job_completed/v2_job_queue for up to ~45s when running inside the outer commit_completed_job transaction. Now try_schedule_next_job makes a single attempt and returns errors to the caller. Non-retryable errors (QuotaExceeded, NotFound) disable the schedule immediately inside the function. Transient errors are returned for the caller to retry: - commit_completed_job path: outer backon retry (10x3s) retries the entire transaction including the schedule push, so no locks are held during sleep. - handle_flow path: new backon retry (10x3s) wraps begin/push/commit with a fresh transaction per attempt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: clear push_err after successful schedule disable to prevent stuck schedules When try_schedule_next_job disables the schedule for non-retryable errors (NotFound, QuotaExceeded), clear the error so the caller commits the tx (persisting the disable). Previously, the error propagated up, causing the tx to be dropped and rolling back the disable — leaving the schedule permanently enabled but broken. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add 5s timeout on push_scheduled_job, clean up handle_flow error handling - Add tokio::time::timeout(5s) around push_scheduled_job inside try_schedule_next_job to bound worst-case lock holding per attempt - Remove unreachable QuotaExceeded/NotFound match arms in handle_flow (these errors are handled internally by try_schedule_next_job) - Add report_error_to_workspace_handler_or_critical_side_channel in handle_flow when post-exhaustion schedule disable fails Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: return SchedulePushZombieError when both schedule push and disable fail When handle_flow cannot push the next scheduled job AND cannot disable the schedule, return a SchedulePushZombieError so the worker leaves the flow job in the queue for zombie detection to restart. This prevents stuck schedules where neither the next tick was pushed nor the schedule was disabled. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
53 lines
1.2 KiB
TOML
53 lines
1.2 KiB
TOML
[package]
|
|
name = "windmill-queue"
|
|
version.workspace = true
|
|
authors.workspace = true
|
|
edition.workspace = true
|
|
|
|
[lib]
|
|
name = "windmill_queue"
|
|
path = "src/lib.rs"
|
|
|
|
[features]
|
|
default = []
|
|
private = []
|
|
enterprise = ["windmill-common/enterprise"]
|
|
cloud = []
|
|
benchmark = ["windmill-common/benchmark"]
|
|
failpoints = []
|
|
prometheus = ["dep:prometheus"]
|
|
smtp = []
|
|
|
|
[dependencies]
|
|
windmill-audit.workspace = true
|
|
windmill-common = { workspace = true, default-features = false }
|
|
anyhow.workspace = true
|
|
hmac.workspace = true
|
|
sql-builder.workspace = true
|
|
sqlx.workspace = true
|
|
tracing.workspace = true
|
|
serde.workspace = true
|
|
serde_json.workspace = true
|
|
ulid.workspace = true
|
|
uuid.workspace = true
|
|
chrono.workspace = true
|
|
chrono-tz.workspace = true
|
|
hex.workspace = true
|
|
reqwest.workspace = true
|
|
lazy_static.workspace = true
|
|
prometheus = { workspace = true, optional = true }
|
|
cron.workspace = true
|
|
tokio.workspace = true
|
|
futures-core.workspace = true
|
|
futures.workspace = true
|
|
itertools.workspace = true
|
|
async-recursion.workspace = true
|
|
axum.workspace = true
|
|
serde_urlencoded.workspace = true
|
|
regex.workspace = true
|
|
backon.workspace = true
|
|
quick_cache.workspace = true
|
|
thiserror.workspace = true
|
|
dashmap.workspace = true
|
|
once_cell.workspace = true
|