Payment Processing Issue

Incident Report for Visp

Postmortem

On March 10th, 2025, an issue occurred at 9:00 AM PST, resulting in payment processing and login issues. The issue was resolved at 11:35 AM PST on the same day. 

What Happened?

On March 10, some users could not process payments and experienced login difficulties. After investigating, the team discovered that a security tool on the server mistakenly blocked the database connection, preventing transactions and causing login issues.

Why Did This Happen?

The team upgraded the MySQL database to enhance query execution, strengthen security, and improve overall performance. However, a long-standing tool on the server, which had operated without issues for years, failed to update its settings correctly. As a result, it mistakenly blocked the database IP despite being correctly added to the trusted IP list. This disruption in communication between the UBO server and the database led to payment processing failures and login issues.

Impact of the Issue:

  • Payment Processing: You may have been unable to complete payment transactions during the affected period.
  • Login Issues: You may have experienced intermittent difficulties logging into your UBO (legacy desktop) account.

What We Did to Fix It:

  • To prevent further disruptions, we disabled the faulty tool and strengthened our security measures by implementing an early detection system. This system provides real-time alerts to our team, ensuring quick identification and resolution of potential issues. 

What We Are Doing to Prevent This in the Future:

  • Upgrading Security Systems: The team is actively reviewing and implementing alternative, more reliable security solutions. This new system will prevent blocking essential services while maintaining robust security.
  • Enhanced Monitoring and Alerts: The team is implementing comprehensive logging and automated alerts to monitor blocked IP addresses. This will allow us to identify issues quickly, minimize disruptions, and resolve them as quickly as possible.

Our Commitment to You:

We understand the importance of reliable and secure access to UBO. We are committed to providing you with stable and dependable service and are taking the necessary steps to ensure that similar issues are mitigated and even prevented in the future.

We appreciate your patience and understanding as we improve the systems. If you have any further questions or concerns, feel free to contact your Visp Client Success team.

Posted Mar 11, 2025 - 00:00 UTC

Resolved

The issue affecting payment processing has been resolved. You can now process payments as normal. Our team is actively monitoring the stability of the fixes to ensure continued smooth operation.

A detailed root cause analysis will be provided once the investigation is fully completed.

Thank you for your patience.
Posted Mar 10, 2025 - 17:56 UTC

Investigating

We've received reports from some users that there are delays when they try to make payments, and sometimes it leads to an error. We advise users to post payments only once, to prevent any duplicated payment postings. Our team is aware of the issue and is actively investigating the root cause. We'll keep you updated as we learn more. Thanks for your patience and understanding!
Posted Mar 10, 2025 - 16:04 UTC
This incident affected: VISP.