On March 31st 2026, 4:50 AM to 8:50 AM UTC, a small subset of customers experienced HTTP 500 errors when using the Data Service Entity Query API in multiple regions.
The issue occurred only for requests containing a user-defined filter configuration when the API is used directly, distinguishing it from filters generated by platform consumption surfaces (e.g., Activities).
For affected customers, queries with this specific payload request failed, while all other requests continued to function normally. No data loss or data inconsistency occurred.
The issue was triggered for a subset of requests with a specific payload pattern ("filterGroups": [null]) that was valid but not handled correctly by the system.
A recently introduced logging component did not properly handle this input. As a result, it failed during processing and caused the API to return an HTTP 500 error.
This exposed a gap in how the system handles certain uncommon but valid inputs. Additionally, because this logging runs as part of the request flow, its failure impacted the overall request instead of being isolated.
The issue was initially identified through an automated Sev2 alert and was also reported by a customer.
During the investigation, the engineering team identified a common payload pattern across failing requests and traced the issue to the query analytics telemetry path.
Upon detection, the team identified the affected execution path and applied mitigation by disabling the related functionality via a configuration change.
This action immediately stopped the errors and restored normal API behavior. The issue was mitigated within 30 minutes.
The following actions are being taken to prevent a recurrence: