Testing Guide for Windmill CLI

Running Tests

# Run unit tests only (fast — no backend, no database, no cargo build)
bun run test:unit

# Run all tests (unit + integration — requires PostgreSQL + cargo)
DATABASE_URL=postgres://postgres:changeme@localhost:5432 bun run test

# Run specific test files
bun test test/sync_pull_push.test.ts
bun test test/workspace_conflicts_unit.test.ts

Test Categories

Unit tests (`*_unit.test.ts`)

Pure local tests — no backend, no database. Uses bunfig.unit.toml (no preload).

Examples: git_unit, lint_command_unit, tar_creation_unit, workspace_conflicts_unit

Integration tests

Require a running backend and PostgreSQL. The setup.ts preload builds the backend binary and starts a shared backend instance.

Examples: sync_pull_push, dev_server, standalone_commands

AI Benchmark Caveats

The repo-level benchmark CLI lives under ai_evals/, but it currently depends on mocked frontend flow execution in a few places. Treat flow benchmark passes as artifact-shape signal, not full runtime correctness, when either of these apply:

deterministic flow validation does not currently reject syntactically invalid rawscript module bodies
frontend benchmark calls to test_run_flow and test_run_step return mocked completed jobs for mock-job-id-* workspaces instead of executing the flow

If a prompt change depends on flow wiring or script runtime behavior, verify it with additional validation or a real run before trusting the benchmark result.

Environment Variables

Variable	Purpose	Default
`DATABASE_URL`	PostgreSQL connection string (without database name)	`postgres://postgres:changeme@localhost:5432`
`TEST_BACKEND`	`cargo` or `docker`	`cargo`
`CI_MINIMAL_FEATURES`	`true` for CI mode (zip-only features)	unset
`EE_LICENSE_KEY`	Enterprise license for EE feature tests	unset
`TEST_FEATURES`	Additional cargo features (comma-separated)	unset
`TEST_CLI_RUNTIME`	`node` to test npm package	unset
`UNIT_ONLY`	`1` to skip backend setup in preload (used by `test:unit`)	unset
`VERBOSE`	`1` for backend process output	unset

Cleanup

Stale test databases (windmill_test_*) and orphaned backend processes from previous crashed runs are automatically cleaned up when starting a new test run.

To manually check for leftovers:

# Check for stale test databases
psql postgres://postgres:changeme@localhost:5432/postgres -c \
  "SELECT datname FROM pg_database WHERE datname LIKE 'windmill_test_%';"

# Check for orphaned backend processes
ps aux | grep "target/debug/windmill" | grep -v grep

2.7 KiB Raw Permalink Blame History