windmill/integration_tests/ai_agent_tests/README.md

AI Agents Integration Tests

Integration tests for Windmill AI agents using the preview_flow endpoint.

Quick Start

1. Create and activate a virtual environment:

   cd integration_tests/ai_agent_tests
   python -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Create .env file with your API keys:

   OPENAI_API_KEY=sk-...
   ANTHROPIC_API_KEY=sk-ant-...
   GOOGLE_AI_API_KEY=...
   OPENROUTER_API_KEY=sk-or-...
   BEDROCK_API_KEY=...
   

4. Run a single test to verify:

   pytest test_basic_completion.py::TestOpenAI::test_openai_completion -v -s
   

Prerequisites

1. A running Windmill instance (default: http://localhost:8000)
2. Python 3.10+
3. API keys for the providers you want to test

How It Works

Provider Setup

The setup_providers fixture automatically creates Windmill variables and resources from your environment variables:

Provider	Variable Path	Resource Path	Resource Type
OpenAI	u/admin/openai_api_key	u/admin/openai	openai
Anthropic	u/admin/anthropic_api_key	u/admin/anthropic	anthropic
Google AI	u/admin/google_ai_api_key	u/admin/googleai	googleai
OpenRouter	u/admin/openrouter_api_key	u/admin/openrouter	openrouter
Bedrock	u/admin/bedrock_api_key	u/admin/bedrock	aws_bedrock

Tools

Tools use rawscript with inline content instead of creating actual scripts. This means:

• No script creation/deployment needed
• Scripts are embedded directly in the flow definition
• Faster test execution

Running Tests

All commands assume you're in the integration_tests/ai_agent_tests directory with the venv activated.

Run all AI agent tests

pytest . -v -s

Run only basic completion tests

pytest test_basic_completion.py -v -s

Run only tool calling tests

pytest test_tool_calling.py -v -s

Run tests for a specific provider

# Only Anthropic tests
pytest . -v -s -k "anthropic"

# Only OpenAI tests
pytest . -v -s -k "openai"

Run a single test

# OpenAI basic completion
pytest test_basic_completion.py::TestOpenAI::test_openai_completion -v -s

# Anthropic basic completion
pytest test_basic_completion.py::TestAnthropic::test_anthropic_completion -v -s

# Parametrized test for one provider
pytest test_basic_completion.py::TestBasicCompletion::test_simple_prompt[openai] -v -s

Test Structure

• conftest.py - Test fixtures and utilities
  • AIAgentTestClient - HTTP client for preview_flow
  • create_ai_agent_flow() - Creates AI agent flow definitions
  • create_rawscript_tool() - Creates inline script tools
  • setup_providers - Sets up variables and resources
• providers.py - Provider configurations (OpenAI, Anthropic, Google AI, Bedrock, OpenRouter)
• test_basic_completion.py - Basic AI completion tests
• test_tool_calling.py - Tool calling tests with inline scripts

Troubleshooting

Tests fail with authentication error

Make sure Windmill is running and you can login with the default credentials:

• Email: admin@windmill.dev
• Password: changeme

Tests skip due to missing API keys

Ensure the environment variables are set correctly:

echo $OPENAI_API_KEY

Resource creation fails

If resources already exist with different values, you may need to delete them manually via the Windmill UI or API before running tests again.