Limble outages and incidents
Outage and incident data over the last 30 days for Limble.
There have been 0 outages or incidents for Limble in the last 30 days.
Severity Breakdown:
Tired of searching for status updates?
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowComponents and Services Monitored for Limble
Outlogger tracks the status of these components for Xero:
Limble Fuse
| Component | Status |
|---|---|
| Limble CMMS API | Active |
| Limble CMMS Authentication Service | Active |
| Limble CMMS Authentication Service Backend | Active |
| Limble CMMS Marketing Website | Active |
| Limble CMMS Web Application | Active |
| Limble Fuse | Active |
| Limble Fuse System | Active |
Latest Limble outages and incidents.
View the latest incidents for Limble and check for official updates:
Some customers having issues logging in
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: June 17, 2024, 3:33 p.m.
Updates:
-
Time: June 17, 2024, 6:46 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: June 17, 2024, 6:45 p.m.Status: MonitoringUpdate: We are continuing to monitor for any further issues.
-
Time: June 17, 2024, 3:56 p.m.Status: MonitoringUpdate: A fix has been implemented and we are monitoring the results.
-
Time: June 17, 2024, 3:33 p.m.Status: InvestigatingUpdate: We are currently investigating this issue.
Some customers not able to access the application
Description: Issue Resolved
Status: Resolved
Impact: Major | Started At: May 20, 2024, noon
Updates:
-
Time: May 20, 2024, 5:13 p.m.Status: ResolvedUpdate: Issue Resolved
Some Customers unable to access the application
Description: Customers are reporting a blank screen when logging in.
Status: Resolved
Impact: None | Started At: May 16, 2024, 10 a.m.
Updates:
-
Time: May 22, 2024, 1:11 p.m.Status: ResolvedUpdate: Customers are reporting a blank screen when logging in.
We are aware of an issue that's causing some customers to not be able to access the application
Description: **Date:** May 14, 2024 **Status:** Resolved **Summary** On May 14, 2024, our service experienced a downtime event due to multiple frontend versions being served concurrently. This incident was caused by a discrepancy in our build process that led to different JavaScript chunks being deployed despite using the same Git commit. Immediate actions were taken to redeploy the code and resolve the issue, and measures have been implemented to prevent recurrence. **Impact** The downtime affected all users attempting to access our site, resulting in an inability to load the app. This incident primarily impacted customers on the main version of the app, while those on Canary were unaffected. **Root Causes** The incident was triggered by a container restart that resulted in different JavaScript chunks being served. Our build process, though using the same Git commit, produced non-idempotent results, causing one of our webApp containers to serve incorrect chunks. **Resolution and Improvements** Immediate actions included redeploying the rollback branch, ensuring all containers served the correct chunks, and implementing a fix to skip build/push for the container image if the SHA tag already exists. Additionally, the following improvements are planned: * **Logging Enhancements** * **Container Health Checks** * **Immutable Tags** * **Build Process Update** * **Nginx Configuration** **Description of Events** * **6:04 AM MST:** Incident detected * **6:15 AM MST:** Investigation and communication initiated. * **6:46 AM MST:** Redeployment initiated using rollback branch. * **6:51 AM MST:** Successful redeployment and app loading restored. * **7:50 AM MST:** Incident resolved
Status: Postmortem
Impact: Critical | Started At: May 14, 2024, 12:21 p.m.
Updates:
-
Time: May 20, 2024, 7:42 p.m.Status: PostmortemUpdate: **Date:** May 14, 2024 **Status:** Resolved **Summary** On May 14, 2024, our service experienced a downtime event due to multiple frontend versions being served concurrently. This incident was caused by a discrepancy in our build process that led to different JavaScript chunks being deployed despite using the same Git commit. Immediate actions were taken to redeploy the code and resolve the issue, and measures have been implemented to prevent recurrence. **Impact** The downtime affected all users attempting to access our site, resulting in an inability to load the app. This incident primarily impacted customers on the main version of the app, while those on Canary were unaffected. **Root Causes** The incident was triggered by a container restart that resulted in different JavaScript chunks being served. Our build process, though using the same Git commit, produced non-idempotent results, causing one of our webApp containers to serve incorrect chunks. **Resolution and Improvements** Immediate actions included redeploying the rollback branch, ensuring all containers served the correct chunks, and implementing a fix to skip build/push for the container image if the SHA tag already exists. Additionally, the following improvements are planned: * **Logging Enhancements** * **Container Health Checks** * **Immutable Tags** * **Build Process Update** * **Nginx Configuration** **Description of Events** * **6:04 AM MST:** Incident detected * **6:15 AM MST:** Investigation and communication initiated. * **6:46 AM MST:** Redeployment initiated using rollback branch. * **6:51 AM MST:** Successful redeployment and app loading restored. * **7:50 AM MST:** Incident resolved
-
Time: May 14, 2024, 3:25 p.m.Status: ResolvedUpdate: Services are recovering and we have identified the cause of the outage. We are working to implement a fix to prevent the same thing from happening in the future.
-
Time: May 14, 2024, 12:21 p.m.Status: InvestigatingUpdate: We are currently investigating this issue.
Users unable to access the limble applications
Description: **Date**: April 26, 2024 **Status**: Resolved # Summary On April 26th, 2024, our service experienced a downtime event due to an SSL certificate failure, affecting most of our customers from approximately 2:20 AM to 3:00 AM MST. The issue stemmed from recent changes in the rules around SSL certificate auto-renewal by our SSL provider, which were not fully accounted for in our system. As a result, we have enhanced our alerting systems and updated our infrastructure configurations to prevent similar issues in the future. # Impact The downtime affected all users attempting to access our site without a cached version of the SSL certificate, resulting in about 40 minutes of service disruption. # Root Causes The primary cause of the downtime was a change in the auto-renewal rules for SSL certificates by our provider, which led to an unexpected, silent expiration. Although a fix was previously developed, it was not yet fully propagated across the entire Limble infrastructure. # Resolution and Improvements Immediate actions were taken to renew the SSL certificate and mitigate the situation. Following the incident, we fully integrated the auto-renewal fix and enhanced our monitoring and alerting capabilities to detect similar issues ahead of potential disruptions. # Description of Events: * **2:18 AM MST:** SSL certificate for [limblecmms.com](http://forlimblecmms.com/) and some of its subdomains. * **2:20 AM MST:** Service disruption detected. * **2:25 AM MST:** The on-call team was alerted to the issue. * **2:33 AM MST:** Critical alert alarm raised. * **2:44 AM MST:** Official incident declaration posted on our status page. * **3:03 AM MST:** New SSL certificate procured and implemented; service restored.
Status: Postmortem
Impact: Critical | Started At: April 26, 2024, 8:43 a.m.
Updates:
-
Time: May 3, 2024, 2:50 p.m.Status: PostmortemUpdate: **Date**: April 26, 2024 **Status**: Resolved # Summary On April 26th, 2024, our service experienced a downtime event due to an SSL certificate failure, affecting most of our customers from approximately 2:20 AM to 3:00 AM MST. The issue stemmed from recent changes in the rules around SSL certificate auto-renewal by our SSL provider, which were not fully accounted for in our system. As a result, we have enhanced our alerting systems and updated our infrastructure configurations to prevent similar issues in the future. # Impact The downtime affected all users attempting to access our site without a cached version of the SSL certificate, resulting in about 40 minutes of service disruption. # Root Causes The primary cause of the downtime was a change in the auto-renewal rules for SSL certificates by our provider, which led to an unexpected, silent expiration. Although a fix was previously developed, it was not yet fully propagated across the entire Limble infrastructure. # Resolution and Improvements Immediate actions were taken to renew the SSL certificate and mitigate the situation. Following the incident, we fully integrated the auto-renewal fix and enhanced our monitoring and alerting capabilities to detect similar issues ahead of potential disruptions. # Description of Events: * **2:18 AM MST:** SSL certificate for [limblecmms.com](http://forlimblecmms.com/) and some of its subdomains. * **2:20 AM MST:** Service disruption detected. * **2:25 AM MST:** The on-call team was alerted to the issue. * **2:33 AM MST:** Critical alert alarm raised. * **2:44 AM MST:** Official incident declaration posted on our status page. * **3:03 AM MST:** New SSL certificate procured and implemented; service restored.
-
Time: April 26, 2024, 9:35 a.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: April 26, 2024, 9:05 a.m.Status: MonitoringUpdate: We have implemented a fix to address this issue and are monitoring the results.
-
Time: April 26, 2024, 8:43 a.m.Status: InvestigatingUpdate: Currently we are aware an an issue where limble applications appear to be down. We are investigating.
