IRM Loading Issues

Incident Report for Visp

Postmortem

Subject: Brief IRM Outage Resolved – Root Cause and Prevention Plan

Dear ISPs,

We want to inform you of a brief service disruption that occurred this morning affecting the IRM system:

  • Issue: IRM displayed a blank screen upon loading
  • Start Time: 8:37 AM EST
  • End Time: 9:20 AM EST

Root Cause:
During a recent process upgrade, the backend service endpoint was inadvertently routed to a load balancer instead of the correct Elastic Beanstalk environment during a dev process upgrade. This misconfiguration led to a 404 error, causing the IRM to load a blank page.

Resolution:
Our team swiftly identified the issue and restored service by reverting the endpoint configuration to Elastic Beanstalk.

Preventive Measures:
We are implementing a new URL verification step in our CI/CD pipeline to automatically detect and prevent misrouted endpoints during future deployments.
We sincerely apologize for the inconvenience this may have caused. Thank you for your continued trust as we enhance system reliability and performance.

Posted May 09, 2025 - 21:15 UTC

Resolved

We are pleased to report that the issues causing inaccessibility for the IRM feature has been resolved.

Following our investigation and collaboration with the development teams, the necessary fixes have been implemented and verified.

Users should now be able to access and use the IRM system without encountering the reported problems.

Thank you for your patience and understanding as we worked to resolve this issue. If you still encounter any difficulties, please reach out to the Visp Client Success team.

Note: If you encounter loading errors, please refresh your browser and re-login (or clear your cache and re-login).
Posted May 09, 2025 - 13:28 UTC

Investigating

We've received multiple reports regarding the IRM being inaccessible or fails to load. We understand this is impacting several users. Please be assured that our team is actively investigating this issue. We will provide updates as more information becomes available.
Posted May 09, 2025 - 13:08 UTC
This incident affected: VISP.