Is there an YOOBIC outage?
YOOBIC status: Systems Active
Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for YOOBIC and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for YOOBIC and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for YOOBIC.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| YOOBIC | Active |
| Help Center | Active |
| Integration Center | Active |
| MySQL Reporting Database | Active |
| Public API | Active |
| Scheduler | Active |
| YOOBIC applications | Active |
View the latest incidents for YOOBIC and check for official updates:
Description: ## Summary On 24 July 2023, our application experienced degraded performance with slower response times for our customers. The incident was caused by a higher server CPU usage, due to a scheduled task calculating aggregation metrics. This incident report provides a detailed account of the timeline, resolution, root cause analysis, impact, corrective actions, and conclusion related to the outage. ## Timeline Events that happened on the 24th of July, 2023: * 07:40am \(UTC\): Our server monitoring system detected higher CPU usage and increased response times. * 07:50am: The issue was escalated to the CTO, who initiated a crisis conference call with the Head of Backend and other Backend engineers for investigation. * 08:00am: The initial investigation focused on determining if the issue was related to a scheduled task, impacting the load on the database, utilizing analytics tools. * 08:05am: The team decided to kill the scheduled task. * 08:10am: The load stopped and the performance progressively returned to normal * 08:15am: The performance was back to normal, and the outage was resolved. ## Root cause analysis The root cause of the issue was traced back to a scheduled task impacting MongoDB database load, and causing the outage. ## Impact During the outage, our application experienced higher response times for some customers, including invalidation of their token forcing a logout, because the token could not be refreshed. It is probable that the same scheduled task played a role in the outage of the 17th July and we are actively monitoring that task in the coming weeks. ## Corrective actions To prevent similar incidents in the future, we have implemented the following corrective actions: * Dispatch the load of the scheduled task on the secondary nodes of MongoDB. * Adding new indexes to ease the workload of the scheduled task. * Evaluating the benefit of adding dedicated secondary nodes to MongoDB for this type of calculation unloading the workload from the primary nodes. ## Conclusion We apologize for the inconvenience caused by the outage. Our team worked diligently to identify and resolve the issue promptly. We are taking the necessary steps to prevent similar incidents in the future and improve the overall stability and reliability of our application. We understand the critical nature of our services for frontline workers, and we are committed to continuously improving our infrastructure and processes to ensure the best possible experience for our valued customers. If you have any further questions or concerns, please don't hesitate to contact our support team. Thank you for your understanding.
Status: Postmortem
Impact: Minor | Started At: July 24, 2023, 7:53 a.m.
Description: # Incident report 17 July 2023 ## Summary On 17 July 2023, our SaaS application experienced an outage resulting in high response times for our customers. The incident was caused by a query plan rebuilt, triggered by a new index created on the mission collection in the previous days. This incident report provides a detailed account of the timeline, resolution, root cause analysis, impact, corrective actions, and conclusion related to the outage. ## Timeline Events that happened on the 17th of July, 2023: * 08:05am \(UTC\): Our server monitoring system detected higher CPU usage and increased response times. * 08:15am: The issue was escalated to the CTO, who initiated a crisis conference call with the Head of Backend and other Backend engineers for investigation. * 08:20am: Initial investigation focused on determining if the issue was related to MongoDB or Heroku, utilizing analytics tools. * 08:25am: It was confirmed that the problem was on the MongoDB side, and further analysis revealed the new index on the mission collection as the root cause. * 08:28am: The team began rebuilding the query plan on the server side. * 08:30am: The performance progressively returned to normal, and the outage was resolved. ## Root cause analysis The outage was caused by the unexpected rebuilt of the query plan, triggered by the implementation of a new index on the mission collection designed to improve the performance of specific queries on the mission collection. This rebuilt was specific to the production environment context and was not triggered during previous tests in staging environments. The outage was resolved by forcing ramp up of queries using the index and rebuilding the query plan on the server side. These actions were taken promptly to restore normal performance. ## Impact During the outage, our SaaS application experienced high response times, with the 95th percentile exceeding 20 seconds. ## Corrective actions To prevent similar incidents in the future, the following corrective actions have been taken: * A thorough review process will be implemented for any index creation or modification, ensuring ramp up query plan building right after deployment. ## Conclusion We apologize for the inconvenience caused by the outage and resulting in higher response times. The incident was promptly addressed by our engineering team, who successfully identified the root cause and implemented corrective actions to restore normal performance. We remain committed to delivering a reliable and efficient SaaS application for our frontline worker customers and will continue to take proactive measures to prevent future occurrences.
Status: Postmortem
Impact: Major | Started At: July 17, 2023, 8:13 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
