Incident Root Cause Analysis and Post-Mortem

Incident Summary:

Incident Date: 7 June 2023

Stoplight experienced an issue where workspaces with custom domains were unable to login via the custom domain URL due to a CORS error.  Workspaces were still available for login via their *.stoplight.io domains.  The issue was resolved 9 hours and 35 minutes after initial urgent support case escalation.

Timeline of Events:

Wednesday, June 7, 2023

• 14:22 CDT - Stoplight released a change to production which affected login for workspaces accessed via custom domains
• 16:41 CDT - Initial indication of potential issue based on receipt of non-urgent support case, issue investigation begins
• 23:41 CDT - Initial indication of broader issue impact based on receipt of first Urgent/P1 support case

Wednesday, June 7, 2023

• 08:06 CDT - Investigation concludes, root cause identified, Stoplight initiates release roll-back process
• 09:16 CDT -  Production roll-back completed, issue resolved

Root Cause:

• Stoplight experienced an issue where workspaces with custom domains were unable to login via the custom domain URL due to a CORS error.
• The issue was not caught before making it into production by Stoplight’s build validation testing and was not immediately caught after release to production by Stoplight’s stack and log monitoring instrumentation.
• While Stoplight initiated incident response mechanisms to investigate and address the issue, it was not immediately clear of the broader impact to customer workspaces until urgent support cases began coming in after standard operating hours.

Remediation Actions

• Stoplight will be adding non stoplight.io and stoplight-dev.io custom domains for pre-release development and qa login testing
• Stoplight is implementing additional pre-release end to end tests for service-auth CORS operations and custom domains and adding production metrics/alerts for 500s for login URLs
• Stoplight is revising incident response mechanisms to better identify broader issue impact and improve time to resolution for non-business-hour urgent incidents.