This directory contains integration tests for the entire system, from input (images, ETH transactions, and etc.) to output (robot commands).
Instead of a monolithic test runner, we use a configuration-based approach where each test case is defined in its own JSON5 configuration file. This allows:
- Testing different VLM implementations independently
- Using different API keys for different tests
- Easier debugging by separating tests
data/: Test data filestest_cases/: Individual test case configurationsimages/: Test images for VLM testing
uv run pytest -m "integration" tests/integration/test_case_runner.py -vuv run pytest -m "integration" -s --log-cli-level=INFO tests/integration/test_case_runner.py -vTEST_CASE="coco_indoor_detection" uv run pytest -m "integration" tests/integration/test_case_runner.py::test_specific_case -v- Create a new JSON5 file in
data/test_cases/following the format in existing files - Add any necessary test images to
data/images/ - Run your test case to verify it works correctly
{
// Test case metadata
"name": "test name",
"description": "test description",
"hertz": 1,
"system_prompt_base": "...",
"system_governance": "...",
"system_prompt_examples": "...'",
"agent_inputs": [...],
"cortex_llm": {...},
"agent_actions": [
{
"name": "move",
"llm_label": "move",
"connector": "ros2"
},
{
"name": "speak",
"llm_label": "speak",
"connector": "ros2"
},
{
"name": "face",
"llm_label": "emotion",
"connector": "ros2"
}
],
"api_key": "openmind_free",
// Input data
"input": {
"images": ["../vlm_test/image1.png", "../vlm_test/image2.png"],
},
// Expected output
"expected": {
"movement": "turn_left | turn_right | move_forward",
"keywords": ["person", "furniture", "indoor"],
"minimum_score": 0.7 // Minimum score required to pass
}
}