Multiple Regions - Agents - Agent Builder might not load for some customers.

Customer Impact

Following a deployment, some customers experienced issues where the Agent Builder frontend failed to load.

• 8 organizations encountered the issue multiple times
• 65 organizations experienced the issue exactly once

The issue was self-resolving upon clearing browser cache and refreshing the page. No further occurrences were observed after mitigation, and the status page was updated and resolved.

Root Cause

The incident was caused by browser caching of a stale remote entry file (remoteEntry.js) following a deployment.

After the deployment, the remote entry file was updated on the server, but browsers continued to serve the previously cached version. The stale cached file referenced assets that no longer existed on the server, resulting in 404 errors when the browser attempted to load them, causing the Agent Builder frontend to fail to load.

The cache was not properly invalidated upon deployment, meaning the problem only appeared immediately after a deployment and resolved itself as browser caches expired.

Detection

The issue was reported by a customer. The failed requests returned 404 status codes, which did not trigger any existing alerts on our side.

This represents a monitoring gap: 404 errors on frontend asset requests were not covered by our alerting rules, meaning the issue went undetected until a customer reported it.

Response

Upon detection:

1. The engineering team identified the root cause as stale browser caching of the remoteEntry.js file after the deployment.
2. Immediate mitigation was communicated: affected customers could resolve the issue by disabling cache and refreshing the page.
3. Deployments were paused to prevent further occurrences until a fix was merged into the release branch.
4. The status page was updated and resolved once the root cause was confirmed and mitigation verified.

Follow-Up

To prevent similar incidents in the future, we are taking the following steps:

• Deployment process improvement: Ensuring that future deployments properly invalidate cached frontend assets to prevent stale file references.