Incident: Invoice Generation Issue – May 17th & 18th

🛠️ Post-Incident Summary: Invoice Generation Issue (May 17–18)

What Happened:
Some ISPs did not receive their invoices as expected on May 17th and 18th. Other automated tasks in the system were also affected during this time.

✅ Root Cause

Behind the scenes, we were working on system upgrades earlier in the week to improve performance and ensure future compatibility. However, the upgrade was only partly applied at first, and when the final changes went live on May 16, they unintentionally caused scheduled tasks—like invoice creation—to stop running.

These tasks failed quietly, so the problem wasn’t immediately visible.

🔧 What We Did to Fix It

• We manually created the missing invoices and made sure affected workflows were completed.
• We updated the system to better handle errors and ensure tasks don’t fail silently.
• We added tools that help us monitor these tasks so we can catch any future issues quickly.
• We improved how the system handles connections and memory to avoid similar problems.

📌 What’s Next

• We are now closely monitoring all automated processes.
• Additional checks are in place to ensure all updates are applied fully and correctly in the future.
• We’re committed to preventing similar incidents and responding even faster if they occur.

We sincerely apologize for the inconvenience and thank you for your patience as we worked through this.

If you have any remaining concerns, feel free to reach out to your Visp Client Success team.