Pydantic validation error: observation content exceeds 1000 character limit #385

Problem

API request failed with Pydantic validation error when returning an EntityResponse with an observation that exceeds the 1000 character limit.

Evidence

Timestamp: 2025-10-20T04:55:14.804886Z
Service: api
Trace ID: 0199fff8ba7498bba067d43b47f7cc93

Error:

pydantic_core._pydantic_core.ValidationError: 1 validation error for EntityResponse
observations.0.content
  Value should have at most 1000 items after validation, not 1458 
  [type=too_long, input_value='{\\n  "$schema": "http://...nalProperties": true\\n}', input_type=str]

Analysis

The EntityResponse schema has a validation constraint limiting observation content to 1000 characters, but an entity has an observation with 1458 characters. This causes the API response to fail validation.

Affected field: observations.0.content
Current limit: 1000 characters
Actual size: 1458 characters

Impact

• API request fails with 500-level error
• User cannot retrieve the entity via API
• Breaks client applications expecting valid responses

Root Cause

Likely one of:

1. Schema mismatch: Database allows longer content than API schema
2. Migration issue: Old data created before validation was added
3. Import issue: Data imported without validation

Fix Options

1. Increase schema limit: Raise the 1000-character limit to accommodate longer observations
2. Truncate on response: Automatically truncate long observations in API responses
3. Validation on write: Enforce limit when creating/updating observations
4. Data cleanup: Find and fix/truncate observations exceeding the limit

Recommended Approach

1. Query database to find all observations exceeding 1000 characters
2. Decide on appropriate limit (1000 may be too restrictive)
3. Update schema validation to match database constraints
4. Add validation on write to prevent future violations

Investigation Needed

• What's the reasonable maximum observation length?
• How many observations in production exceed 1000 characters?
• Should we truncate or increase the limit?