Failing Tests

This file tracks benchmark cases that still fail or need follow-up validation.

Flow

flow-test6-ai-agent-tools Latest failing run: ai_evals/results/2026-04-09T11-25-24.107Z__flow Issues: final output does not include the actions or tool-result details the prompt asks for open_support_ticket contains a syntax bug
flow-test7-simple-modification Latest failing run: ai_evals/results/2026-04-09T11-25-24.107Z__flow Issues: validate_data was added, but the failure behavior still does not match the requested contract save_results throws instead of returning a graceful structured result
flow-test11-preprocessor-and-failure-handler Latest failing run: ai_evals/results/2026-04-09T11-25-24.107Z__flow Issues: the model creates regular preprocessor and failure modules it does not use Windmill's special top-level preprocessor_module and failure_module

flow-test4-order-processing-loop Full-suite failing run: ai_evals/results/2026-04-09T11-25-24.107Z__flow Follow-up passing run after prompt improvement: ai_evals/results/2026-04-09T13-29-15.877Z__flow Note: this case failed on invalid branchone downstream result access it passed after adding explicit branch-output guidance to the flow prompt rerun the full flow suite to confirm the fix holds in the broader benchmark