all

Explores adding constraint-based job assignment where workers declare
properties (e.g., gpu, mem, region) and jobs specify constraints.

Key insight: with max 3 properties, constraints can be encoded as tags
using canonical format (e.g., "c:gpu=a100,mem=high"), preserving the
existing efficient IN-clause query pattern. Workers generate all 2^3=8
combinations they can satisfy.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

backend/2026-01-18-local-command-caveatcaveat-the-messages-below-w.txt

@@ -0,0 +1,219 @@
+                                                                                
+ ▐▛███▜▌   Claude Code v2.1.12                                                  
+▝▜█████▛▘  Opus 4.5 · Claude Max                                                
+  ▘▘ ▝▝    ~/windmill/backend                                                   
+                                                                                
+❯ /clear                                                                        
+  ⎿  (no content)                                                               
+                                                                                
+❯ If you look at how job assignment is done, it's done based on a tag which     
+  allow workers to do a super efficient pull. It works well and nice but I      
+  wonder if it would be possible to do a constraint assignment one instead      
+  where workers declare some property, a,b,c,d with some value, and job         
+  says that a job has constraints in terms of a,b,c,d with and or query. If     
+  it's impossible to make fast, maybe we could say that it's strict equal       
+  and always AND                                                                
+                                                                                
+● This is an interesting architectural question. Let me explore how the current 
+  tag-based job assignment works to understand the performance implications and 
+  feasibility of a constraint-based approach.                                   
+                                                                                
+● Explore(Explore job assignment mechanism)                                     
+  ⎿  Done (35 tool uses · 71.8k tokens · 1m 5s)                                 
+                                                                                
+● Based on my exploration, here's how the current system works and my analysis  
+  of constraint-based assignment:                                               
+                                                                                
+  Current Tag System                                                            
+                                                                                
+  The efficiency comes from:                                                    
+  1. Single tag column on v2_job_queue                                          
+  2. B-tree index on (priority DESC, scheduled_for, tag) WHERE running = false  
+  3. Simple query: WHERE running = false AND tag IN ('tag1', 'tag2', ...) AND   
+  scheduled_for <= now()                                                        
+  4. FOR UPDATE SKIP LOCKED for race-free claiming                              
+                                                                                
+  The IN clause with a B-tree index is extremely efficient because PostgreSQL   
+  can do index-only scans.                                                      
+                                                                                
+  Constraint-Based Assignment Options                                           
+                                                                                
+  Option 1: JSONB containment (AND/OR support)                                  
+                                                                                
+  -- Workers store properties                                                   
+  ALTER TABLE worker_ping ADD COLUMN properties JSONB;                          
+  -- Jobs store constraints                                                     
+  ALTER TABLE v2_job_queue ADD COLUMN constraints JSONB;                        
+                                                                                
+  -- Pull query becomes                                                         
+  SELECT id FROM v2_job_queue                                                   
+  WHERE running = false                                                         
+    AND (constraints IS NULL OR worker_properties @> constraints)               
+  ORDER BY priority DESC NULLS LAST, scheduled_for                              
+  FOR UPDATE SKIP LOCKED                                                        
+                                                                                
+  Problem: This inverts the query direction. Currently workers pull jobs        
+  matching their tags. With JSONB, you'd need to check if worker properties     
+  satisfy job constraints, but the worker properties aren't in the job table.   
+  You'd need a JOIN or subquery, killing performance.                           
+                                                                                
+  Option 2: Strict AND with equality - Constraint Hash                          
+                                                                                
+  If constraints are always AND with strict equality, you can compute a         
+  canonical hash:                                                               
+                                                                                
+  // Job with constraints {a: "gpu", b: "high-mem"}                             
+  // Gets constraint_hash = hash("a=gpu&b=high-mem")                            
+                                                                                
+  // Worker with properties {a: "gpu", b: "high-mem", c: "fast-disk"}           
+  // Pre-computes ALL hashes it can satisfy:                                    
+  // - hash("") (no constraints)                                                
+  // - hash("a=gpu")                                                            
+  // - hash("b=high-mem")                                                       
+  // - hash("c=fast-disk")                                                      
+  // - hash("a=gpu&b=high-mem")                                                 
+  // - hash("a=gpu&c=fast-disk")                                                
+  // - ... (all 2^n combinations)                                               
+                                                                                
+  Pull query stays efficient:                                                   
+  WHERE constraint_hash IN ('hash1', 'hash2', ...)                              
+                                                                                
+  Downside: 2^n hashes per worker. With 4 properties = 16 hashes (manageable).  
+  With 8 = 256 (getting large).                                                 
+                                                                                
+  Option 3: Fixed constraint columns (pragmatic)                                
+                                                                                
+  ALTER TABLE v2_job_queue ADD COLUMN constraint_gpu VARCHAR;                   
+  ALTER TABLE v2_job_queue ADD COLUMN constraint_memory VARCHAR;                
+  ALTER TABLE v2_job_queue ADD COLUMN constraint_region VARCHAR;                
+                                                                                
+  CREATE INDEX queue_constraints ON v2_job_queue                                
+    (running, constraint_gpu, constraint_memory, constraint_region, priority    
+  DESC, scheduled_for)                                                          
+    WHERE running = false;                                                      
+                                                                                
+  Workers generate query:                                                       
+  WHERE running = false                                                         
+    AND (constraint_gpu IS NULL OR constraint_gpu = 'nvidia-a100')              
+    AND (constraint_memory IS NULL OR constraint_memory IN ('high', 'medium'))  
+    ...                                                                         
+                                                                                
+  Downside: The IS NULL OR pattern doesn't index-scan as cleanly. PostgreSQL    
+  might fall back to bitmap scans.                                              
+                                                                                
+  Option 4: Keep tags, but make them composite                                  
+                                                                                
+  Instead of changing the architecture, allow composite tags:                   
+  tag = "gpu:a100,mem:high,region:us-east"                                      
+                                                                                
+  Workers declare which composite tags they handle. Jobs specify their required 
+  composite tag.                                                                
+                                                                                
+  Downside: Loses flexibility - can't match partial constraints.                
+                                                                                
+  My Recommendation                                                             
+                                                                                
+  For strict AND with equality, the constraint hash approach (Option 2) is the  
+  cleanest because it preserves the current query structure. The 2^n blowup is  
+  acceptable if you limit to ~5-6 constraint dimensions.                        
+                                                                                
+  Would you like me to prototype what this would look like in the codebase? I   
+  could show:                                                                   
+  1. Schema changes                                                             
+  2. How workers would compute their satisfiable constraint hashes              
+  3. How jobs would compute their constraint hash at creation time              
+                                                                                
+✻ Cooked for 2m 5s                                                              
+                                                                                
+❯ couldn't hash be a tag in the end? What if I say that at most 3               
+  properties can be set globally ?                                              
+                                                                                
+● Exactly right. With 3 properties max, workers compute at most 2³ = 8          
+  constraint tags. The existing system stays unchanged:                         
+                                                                                
+  Example:                                                                      
+                                                                                
+  Global properties: gpu, mem, region                                           
+                                                                                
+  Worker declares: {gpu: "a100", mem: "high"} (no region)                       
+                                                                                
+  Worker generates these tags:                                                  
+  c:                          # satisfies jobs with no constraints              
+  c:gpu=a100                                                                    
+  c:mem=high                                                                    
+  c:gpu=a100,mem=high                                                           
+                                                                                
+  Job with constraints {gpu: "a100", mem: "high"} gets tag:                     
+  c:gpu=a100,mem=high                                                           
+                                                                                
+  The existing tag IN (...) query works unchanged. No schema changes, no index  
+  changes.                                                                      
+                                                                                
+  Key implementation details:                                                   
+                                                                                
+  1. Canonical ordering - always sort keys alphabetically so {mem: "high", gpu: 
+  "a100"} produces the same tag as {gpu: "a100", mem: "high"}                   
+  2. Workers without a property can't satisfy jobs requiring it - they simply   
+  don't generate tags containing that property                                  
+  3. Jobs without constraints get tag c: (or could coexist with regular tags    
+  like deno, python3)                                                           
+  4. Coexistence - constraint tags could be a separate dimension from language  
+  tags, or you could combine them: deno,c:gpu=a100,mem=high                     
+                                                                                
+  Want me to sketch out what the tag generation logic would look like in Rust?  
+                                                                                
+❯ create a feature branch, put your notes on it, then push that branch          
+                                                                                
+● Bash(cd /home/rfiszel/windmill && git checkout -b                             
+      feature/constraint-based-job-assignment)                                  
+  ⎿  Switched to a new branch 'feature/constraint-based-job-assignment'         
+  ⎿  Shell cwd was reset to /home/rfiszel/windmill/backend                      
+                                                                                
+● Write(docs/design/constraint-based-job-assignment.md)                         
+  ⎿  Wrote 265 lines to docs/design/constraint-based-job-assignment.md          
+      1 # Constraint-Based Job Assignment                                       
+      2 ## Overview                                                             
+      3 This document explores adding constraint-based job assignment to        
+        Windmill, allowing workers to declare properties and jobs to spec       
+        ify constraints that must be satisfied.                                 
+      4 ## Current System                                                       
+      5 Jobs are assigned based on a single `tag` column in `v2_job_queue       
+        `. Workers declare which tags they handle, and pull jobs using:         
+     … +255 lines (ctrl+o to expand)                                            
+                                                                                
+● Bash(cd /home/rfiszel/windmill && git add                                     
+      backend/docs/design/constraint-based-job-assignment.md && git commit -m   
+      "docs: add design document for constraint-based job…)                     
+  ⎿ [feature/constraint-based-job-assignment 8c34c65d8e] docs: add design d     
+    ocument for constraint-based job assignment                                 
+     1 file changed, 264 insertions(+)                                          
+    … +2 lines (ctrl+o to expand)                                               
+                                                                                
+  ⎿  Shell cwd was reset to /home/rfiszel/windmill/backend                      
+                                                                                
+● Bash(cd /home/rfiszel/windmill && git push -u origin                          
+      feature/constraint-based-job-assignment)                                  
+  ⎿ branch 'feature/constraint-based-job-assignment' set up to track 'origi     
+    n/feature/constraint-based-job-assignment'.                                 
+    remote:                                                                     
+    … +14 lines (ctrl+o to expand)                                              
+                                                                                
+  ⎿  Shell cwd was reset to /home/rfiszel/windmill/backend                      
+                                                                                
+● Done. Branch feature/constraint-based-job-assignment pushed with the design   
+  document at backend/docs/design/constraint-based-job-assignment.md.           
+                                                                                
+  The document covers:                                                          
+  - Current tag system analysis                                                 
+  - Constraint tag format (c:gpu=a100,mem=high)                                 
+  - Worker tag generation (all 2^n combinations)                                
+  - Schema changes needed                                                       
+  - Rust implementation sketch for tag generation                               
+  - Query modification                                                          
+  - Performance analysis                                                        
+                                                                                
+  PR link: https://github.com/windmill-labs/windmill/pull/new/feature/constraint
+  -based-job-assignment                                                         
+                                                                                
+✻ Crunched for 54s                                                              
+

backend/docs/design/constraint-based-job-assignment.md

@@ -0,0 +1,264 @@
+# Constraint-Based Job Assignment
+
+## Overview
+
+This document explores adding constraint-based job assignment to Windmill, allowing workers to declare properties and jobs to specify constraints that must be satisfied.
+
+## Current System
+
+Jobs are assigned based on a single `tag` column in `v2_job_queue`. Workers declare which tags they handle, and pull jobs using:
+
+```sql
+SELECT id FROM v2_job_queue
+WHERE running = false
+  AND tag IN ('tag1', 'tag2', ...)
+  AND scheduled_for <= now()
+ORDER BY priority DESC NULLS LAST, scheduled_for
+FOR UPDATE SKIP LOCKED
+LIMIT 1
+```
+
+This is highly efficient due to:
+- B-tree index on `(priority DESC, scheduled_for, tag) WHERE running = false`
+- Simple `IN` clause allows index-only scans
+- `FOR UPDATE SKIP LOCKED` for race-free claiming
+
+## Proposed: Constraint Tags
+
+### Concept
+
+Instead of modifying the query structure, encode constraints as tags using a canonical hash/string format. This preserves the existing efficient query pattern.
+
+### Constraints
+
+- **Limit**: At most 3 global properties can be defined (e.g., `gpu`, `mem`, `region`)
+- **Matching**: Always AND with strict equality
+- Workers compute all 2³ = 8 possible constraint tag combinations they can satisfy
+
+### Example
+
+**Global properties defined**: `gpu`, `mem`, `region`
+
+**Worker declares properties**:
+```json
+{
+  "gpu": "a100",
+  "mem": "high"
+}
+```
+
+**Worker generates constraint tags** (all combinations it can satisfy):
+```
+c:                          # no constraints
+c:gpu=a100
+c:mem=high
+c:gpu=a100,mem=high
+```
+
+Note: Worker doesn't generate tags with `region` because it doesn't have that property.
+
+**Job with constraints**:
+```json
+{
+  "gpu": "a100",
+  "mem": "high"
+}
+```
+
+**Job gets tag**: `c:gpu=a100,mem=high`
+
+The existing `tag IN (...)` query matches this job to the worker.
+
+### Tag Format
+
+Constraint tags use a canonical format:
+- Prefix: `c:` to distinguish from regular tags
+- Key-value pairs: `key=value`
+- Separator: `,` between pairs
+- **Alphabetical ordering**: Keys are always sorted alphabetically for canonical representation
+
+Examples:
+- No constraints: `c:`
+- Single constraint: `c:gpu=a100`
+- Multiple constraints: `c:gpu=a100,mem=high` (gpu before mem alphabetically)
+
+### Coexistence with Language Tags
+
+Two options:
+
+**Option A: Separate tag dimension**
+- Jobs have both `tag` (language) and `constraint_tag`
+- Query: `WHERE tag IN (...) AND (constraint_tag IS NULL OR constraint_tag IN (...))`
+- Requires schema change and new index
+
+**Option B: Combined tags**
+- Combine language and constraints: `deno,c:gpu=a100,mem=high`
+- Workers generate cartesian product of language tags × constraint tags
+- No schema change, but more tags per worker
+
+**Recommendation**: Option A is cleaner for separation of concerns.
+
+## Implementation Plan
+
+### 1. Schema Changes
+
+```sql
+-- Add constraint_tag column
+ALTER TABLE v2_job_queue ADD COLUMN constraint_tag VARCHAR;
+
+-- Add index for constraint-aware queries
+CREATE INDEX queue_sort_constraint ON v2_job_queue
+  (priority DESC NULLS LAST, scheduled_for, tag, constraint_tag)
+  WHERE running = false;
+```
+
+### 2. Worker Configuration
+
+Extend `WorkerConfig` to include constraint properties:
+
+```rust
+// windmill-common/src/worker.rs
+
+pub struct WorkerConfig {
+    pub worker_tags: Vec<String>,
+    pub priority_tags_sorted: Vec<PriorityTags>,
+    // New field
+    pub constraint_properties: HashMap<String, String>,
+}
+```
+
+### 3. Constraint Tag Generation
+
+```rust
+// windmill-common/src/constraints.rs
+
+use itertools::Itertools;
+
+/// Global constraint property keys (max 3)
+pub const CONSTRAINT_KEYS: &[&str] = &["gpu", "mem", "region"];
+
+/// Generate all constraint tags a worker can satisfy
+pub fn generate_worker_constraint_tags(
+    properties: &HashMap<String, String>
+) -> Vec<String> {
+    let mut tags = Vec::new();
+
+    // Get the properties this worker has (filtered to valid keys)
+    let worker_props: Vec<(&str, &str)> = CONSTRAINT_KEYS
+        .iter()
+        .filter_map(|&key| {
+            properties.get(key).map(|v| (key, v.as_str()))
+        })
+        .collect();
+
+    // Generate all 2^n combinations (power set)
+    for size in 0..=worker_props.len() {
+        for combo in worker_props.iter().combinations(size) {
+            let tag = format_constraint_tag(&combo);
+            tags.push(tag);
+        }
+    }
+
+    tags
+}
+
+/// Generate the constraint tag for a job
+pub fn generate_job_constraint_tag(
+    constraints: &HashMap<String, String>
+) -> String {
+    let mut pairs: Vec<(&str, &str)> = constraints
+        .iter()
+        .filter(|(k, _)| CONSTRAINT_KEYS.contains(&k.as_str()))
+        .map(|(k, v)| (k.as_str(), v.as_str()))
+        .collect();
+
+    // Sort alphabetically for canonical form
+    pairs.sort_by_key(|(k, _)| *k);
+
+    format_constraint_tag(&pairs.iter().map(|(k, v)| (k, v)).collect::<Vec<_>>())
+}
+
+fn format_constraint_tag(pairs: &[(&str, &str)]) -> String {
+    if pairs.is_empty() {
+        "c:".to_string()
+    } else {
+        let inner = pairs
+            .iter()
+            .map(|(k, v)| format!("{}={}", k, v))
+            .join(",");
+        format!("c:{}", inner)
+    }
+}
+```
+
+### 4. Query Generation Update
+
+```rust
+// windmill-common/src/worker.rs
+
+pub fn make_pull_query(tags: &[String], constraint_tags: &[String]) -> String {
+    let tags_sql = tags.iter().map(|x| format!("'{x}'")).join(", ");
+    let constraint_sql = constraint_tags.iter().map(|x| format!("'{x}'")).join(", ");
+
+    format_pull_query(format!(
+        "SELECT id
+        FROM v2_job_queue
+        WHERE running = false
+          AND tag IN ({})
+          AND (constraint_tag IS NULL OR constraint_tag IN ({}))
+          AND scheduled_for <= now()
+        ORDER BY priority DESC NULLS LAST, scheduled_for
+        FOR UPDATE SKIP LOCKED
+        LIMIT 1",
+        tags_sql,
+        constraint_sql
+    ))
+}
+```
+
+### 5. Job Creation Update
+
+When creating a job with constraints, compute the constraint tag:
+
+```rust
+// windmill-queue/src/jobs.rs
+
+// In push() or create_job():
+let constraint_tag = if let Some(constraints) = &job_constraints {
+    Some(generate_job_constraint_tag(constraints))
+} else {
+    None
+};
+```
+
+## Performance Analysis
+
+### Query Efficiency
+
+- **No degradation**: The `constraint_tag IN (...)` clause uses the same efficient index pattern
+- **Partial index**: `WHERE running = false` still applies
+- **Tag count**: Workers have at most `|language_tags| + 8` tags (8 constraint combinations max)
+
+### Index Considerations
+
+With the composite index on `(priority, scheduled_for, tag, constraint_tag)`:
+- PostgreSQL can efficiently filter on all columns
+- The `OR constraint_tag IS NULL` handles jobs without constraints
+
+### Scalability
+
+- 3 properties = 8 constraint tags per worker (manageable)
+- 4 properties = 16 constraint tags (still reasonable)
+- 5+ properties = consider alternative approaches
+
+## Future Extensions
+
+1. **OR support**: Could be added by generating multiple constraint tags for a single job
+2. **Inequality constraints**: Would require different approach (not compatible with tag model)
+3. **Dynamic properties**: Workers could update properties at runtime
+
+## Open Questions
+
+1. Should constraint properties be configured globally or per-workspace?
+2. How to handle workers that don't declare any properties? (Treat as matching `c:` only)
+3. Should there be validation that job constraints only use defined property keys?