Files
windmill/backend/tests/delete_after_secs.rs
Ruben Fiszel 2d18a68099 feat: add scheduled job deletion with configurable retention period (#8753)
* feat: add scheduled job deletion with configurable retention period

Extends delete_after_use with delete_after_secs to enable configurable
retention periods for job args/result/logs. At completion, jobs can be
scheduled for future deletion via a new job_delete_schedule table,
processed by a monitor task. Supports per-script, per-flow, and
per-flow-step configuration. Backward compatible.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add integration tests, revert query! macros, fix review issues

- Add integration tests for resolve_delete_after_secs, schedule_job_deletion,
  flow-level and module-level delete_after_secs, backward compat
- Revert sqlx::query() back to sqlx::query!() macros for compile-time safety
- Regenerate sqlx offline cache
- Fix FlowModule/NewScript/FlowValue constructions in all test files
- Fix autoscaling_ee.rs for updated script_path_to_payload return type

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref.txt for autoscaling_ee fix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: gate cleanup_scheduled_job_deletions behind enterprise feature

Prevents dead_code warning (which CI treats as error via -D warnings)
when compiling without enterprise feature.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: regenerate sqlx cache after merge with main

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review feedback on scheduled deletion

- Monitor: roll back transaction on any cleanup error so schedule rows
  survive for retry on next cycle (instead of best-effort then discard)
- Migration: add FK with ON DELETE CASCADE to job_delete_schedule.job_id
  to prevent orphan rows when jobs are deleted through other means
- Simplify bool-to-Option conversion with .then_some(true)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: stop setting delete_after_use alongside delete_after_secs

No mixed-version deployment scenario exists, so delete_after_secs alone
is sufficient. The backend's resolve_delete_after_secs handles
(None, Some(secs)) correctly without needing delete_after_use set.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: remove delete_after_use from public API surface

Remove delete_after_use from OpenAPI spec, API client, runtime client,
and workspace export. Only delete_after_secs is exposed going forward.

The field remains in Rust backend types with #[serde(skip_serializing)]
for backward-compatible deserialization of existing scripts/flows that
were saved with delete_after_use: true.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 1d4b7a31fc115d6aba8640f7cd3fd5a01abe6806

This commit updates the EE repository reference after PR #519 was merged in windmill-ee-private.

Previous ee-repo-ref: 9eba09a13b778caafc6ae65098b90e53c91984d3

New ee-repo-ref: 1d4b7a31fc115d6aba8640f7cd3fd5a01abe6806

Automated by sync-ee-ref workflow.

* fix: regenerate system prompts, remove unused import

- Regenerate auto-generated system prompts after openflow schema change
- Remove unused serde_json::json import in test file (CI -D warnings)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: insert dummy v2_job row in schedule tests for FK constraint

The job_delete_schedule table has a FK to v2_job, so tests need a
real v2_job row before inserting into the schedule table.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: trigger CI re-run

* fix: remove heavy flow integration tests to avoid CI worker contention

The flow integration tests spawn workers that compete for CPU with
the existing relock_skip tests under --test-threads=10, causing
consistent 60s timeouts in CI. Keep only the lightweight unit tests
and DB integration tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore correct ee-repo-ref for our branch

The ref was overwritten to main's EE ref during a rebase. Restore to
our branch's EE commit that includes the autoscaling tuple fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: retrigger CI on fresh runner

* fix: remove FK constraint from job_delete_schedule to unblock CI

The FK with ON DELETE CASCADE to v2_job may have caused performance
overhead during test DB setup (each sqlx::test creates a fresh DB
with all migrations). Remove the FK — orphan schedule rows are
harmlessly cleaned by the monitor.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ee-ref

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-04-08 04:15:28 +00:00

93 lines
3.2 KiB
Rust

use sqlx::{Pool, Postgres};
use windmill_common::jobs::{resolve_delete_after_secs, schedule_job_deletion};
use windmill_test_utils::*;
// ---------------------------------------------------------------------------
// Unit tests for resolve_delete_after_secs
// ---------------------------------------------------------------------------
#[test]
fn test_resolve_no_deletion() {
assert_eq!(resolve_delete_after_secs(None, None), None);
assert_eq!(resolve_delete_after_secs(Some(false), None), None);
}
#[test]
fn test_resolve_immediate_backward_compat() {
// delete_after_use=true with no secs → immediate (0)
assert_eq!(resolve_delete_after_secs(Some(true), None), Some(0));
}
#[test]
fn test_resolve_explicit_secs() {
assert_eq!(resolve_delete_after_secs(None, Some(0)), Some(0));
assert_eq!(resolve_delete_after_secs(None, Some(3600)), Some(3600));
assert_eq!(resolve_delete_after_secs(Some(true), Some(60)), Some(60));
assert_eq!(resolve_delete_after_secs(Some(false), Some(120)), Some(120));
}
#[test]
fn test_resolve_rejects_negative() {
assert_eq!(resolve_delete_after_secs(None, Some(-1)), None);
assert_eq!(resolve_delete_after_secs(Some(true), Some(-100)), None);
}
// ---------------------------------------------------------------------------
// Integration: schedule_job_deletion inserts into job_delete_schedule
// ---------------------------------------------------------------------------
#[sqlx::test(fixtures("base"))]
async fn test_schedule_job_deletion_inserts_row(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let job_id = uuid::Uuid::new_v4();
schedule_job_deletion(&db, job_id, "test-workspace", 3600).await?;
let row = sqlx::query_as::<_, (uuid::Uuid, String)>(
"SELECT job_id, workspace_id FROM job_delete_schedule WHERE job_id = $1",
)
.bind(job_id)
.fetch_one(&db)
.await?;
assert_eq!(row.0, job_id);
assert_eq!(row.1, "test-workspace");
// Verify delete_at is approximately now + 3600s
let delete_at: chrono::DateTime<chrono::Utc> =
sqlx::query_scalar("SELECT delete_at FROM job_delete_schedule WHERE job_id = $1")
.bind(job_id)
.fetch_one(&db)
.await?;
let expected_min = chrono::Utc::now() + chrono::Duration::seconds(3500);
let expected_max = chrono::Utc::now() + chrono::Duration::seconds(3700);
assert!(
delete_at > expected_min && delete_at < expected_max,
"delete_at should be ~1 hour from now, got {delete_at}"
);
Ok(())
}
#[sqlx::test(fixtures("base"))]
async fn test_schedule_job_deletion_is_idempotent(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let job_id = uuid::Uuid::new_v4();
schedule_job_deletion(&db, job_id, "test-workspace", 60).await?;
// Second call should not error (ON CONFLICT DO NOTHING)
schedule_job_deletion(&db, job_id, "test-workspace", 120).await?;
let count: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM job_delete_schedule WHERE job_id = $1")
.bind(job_id)
.fetch_one(&db)
.await?;
assert_eq!(count, 1, "should have exactly one row (idempotent)");
Ok(())
}