kalugin66/windmill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
wmill-script-python
windmill/integration_tests/ai_agent_tests
History
centdix d95e4db8f3 feat(ai): add AWS bedrock session token support (#7908) 
* Add AWS Bedrock session token support in API and worker

* Add Bedrock auth mode integration tests for AI agents

* Split Bedrock integration test env vars for IAM and session creds

* cleaning

* Add masked Bedrock bearer-token debug info logs

* Revert "Add masked Bedrock bearer-token debug info logs"

This reverts commit 6b2fc5e7c2d5b1c6db81f416a4439941a084108c.

* cleaning
2026-02-11 19:29:58 +00:00
..
__init__.py
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00
.env.example
feat(ai): add AWS bedrock session token support (#7908)
2026-02-11 19:29:58 +00:00
conftest.py
feat(ai): add AWS bedrock session token support (#7908)
2026-02-11 19:29:58 +00:00
providers.py
feat(ai): add AWS bedrock session token support (#7908)
2026-02-11 19:29:58 +00:00
README.md
feat(ai): add AWS bedrock session token support (#7908)
2026-02-11 19:29:58 +00:00
requirements.txt
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00
test_bedrock_auth_modes.py
feat(ai): add AWS bedrock session token support (#7908)
2026-02-11 19:29:58 +00:00
test_completion_params.py
fix param for openai (#7483)
2026-01-05 10:56:27 +00:00
test_image_output.py
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00
test_image.webp
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00
test_memory.py
fix(aiagent): fix memory with openai (#7506)
2026-01-06 16:17:29 +00:00
test_output_schema.py
fix: remove $schema field from Google AI output schema requests (#7765)
2026-02-03 01:04:36 +00:00
test_streaming.py
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00
test_tool_calling.py
fix: remove $schema field from Google AI output schema requests (#7765)
2026-02-03 01:04:36 +00:00
test_user_images.py
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00
test_websearch.py
feat(ai): add websearch tool for AI agents (#7399)
2025-12-22 11:41:28 +00:00

README.md

AI Agents Integration Tests

Integration tests for Windmill AI agents using the preview_flow endpoint.

Quick Start

  1. Create and activate a virtual environment:

    cd integration_tests/ai_agent_tests
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

  2. Install dependencies:

    pip install -r requirements.txt

  3. Create .env file with your API keys:

    OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_AI_API_KEY=...
OPENROUTER_API_KEY=sk-or-...
BEDROCK_API_KEY=...
BEDROCK_IAM_ACCESS_KEY_ID=...
BEDROCK_IAM_SECRET_ACCESS_KEY=...
BEDROCK_SESSION_ACCESS_KEY_ID=...
BEDROCK_SESSION_SECRET_ACCESS_KEY=...
BEDROCK_SESSION_TOKEN=...  # optional, only for session-token Bedrock test
AWS_ACCESS_KEY_ID=...      # optional, for env-fallback Bedrock test
AWS_SECRET_ACCESS_KEY=...  # optional, for env-fallback Bedrock test
AWS_SESSION_TOKEN=...      # optional, required for env-fallback if AWS_ACCESS_KEY_ID is ASIA...
BEDROCK_REGION=us-east-1  # optional

  4. Run a single test to verify:

    pytest test_basic_completion.py::TestOpenAI::test_openai_completion -v -s

Prerequisites

  1. A running Windmill instance (default: http://localhost:8000)
  2. Python 3.10+
  3. API keys for the providers you want to test

How It Works

Provider Setup

The setup_providers fixture automatically creates Windmill variables and resources from your environment variables:

Provider Variable Path Resource Path Resource Type
OpenAI u/admin/openai_api_key u/admin/openai openai
Anthropic u/admin/anthropic_api_key u/admin/anthropic anthropic
Google AI u/admin/google_ai_api_key u/admin/googleai googleai
OpenRouter u/admin/openrouter_api_key u/admin/openrouter openrouter
Bedrock (api key) u/admin/bedrock_api_key u/admin/bedrock aws_bedrock
Bedrock (IAM) u/admin/bedrock_iam_access_key_id, u/admin/bedrock_iam_secret_access_key u/admin/bedrock_iam aws_bedrock
Bedrock (IAM + session) u/admin/bedrock_session_access_key_id, u/admin/bedrock_session_secret_access_key, u/admin/bedrock_session_token u/admin/bedrock_iam_session aws_bedrock
Bedrock (environment fallback) none (uses worker/api AWS env) u/admin/bedrock_env aws_bedrock

Tools

Tools use rawscript with inline content instead of creating actual scripts. This means:

  • No script creation/deployment needed
  • Scripts are embedded directly in the flow definition
  • Faster test execution

Running Tests

All commands assume you're in the integration_tests/ai_agent_tests directory with the venv activated.

Run all AI agent tests

pytest . -v -s

Run only basic completion tests

pytest test_basic_completion.py -v -s

Run only tool calling tests

pytest test_tool_calling.py -v -s

Run tests for a specific provider

# Only Anthropic tests
pytest . -v -s -k "anthropic"

# Only OpenAI tests
pytest . -v -s -k "openai"

Run a single test

# OpenAI basic completion
pytest test_basic_completion.py::TestOpenAI::test_openai_completion -v -s

# Anthropic basic completion
pytest test_basic_completion.py::TestAnthropic::test_anthropic_completion -v -s

# Parametrized test for one provider
pytest test_basic_completion.py::TestBasicCompletion::test_simple_prompt[openai] -v -s

Test Structure

  • conftest.py - Test fixtures and utilities
    • AIAgentTestClient - HTTP client for preview_flow
    • create_ai_agent_flow() - Creates AI agent flow definitions
    • create_rawscript_tool() - Creates inline script tools
    • setup_providers - Sets up variables and resources
  • providers.py - Provider configurations (OpenAI, Anthropic, Google AI, Bedrock, OpenRouter)
  • test_basic_completion.py - Basic AI completion tests
  • test_tool_calling.py - Tool calling tests with inline scripts

Troubleshooting

Tests fail with authentication error

Make sure Windmill is running and you can login with the default credentials:

  • Email: admin@windmill.dev
  • Password: changeme

Tests skip due to missing API keys

Ensure the environment variables are set correctly:

echo $OPENAI_API_KEY

Resource creation fails

If resources already exist with different values, you may need to delete them manually via the Windmill UI or API before running tests again.

Reference in New Issue View Git Blame Copy Permalink