Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for RebelMouse and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for RebelMouse.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
AWS ec2-us-east-1 | Active |
AWS elb-us-east-1 | Active |
AWS RDS | Active |
AWS route53 | Active |
AWS s3-us-standard | Active |
AWS ses-us-east-1 | Active |
Braintree API | Active |
Braintree PayPal Processing | Active |
CDN | Active |
Celery | Active |
Content Delivery API | Active |
Discovery | Active |
EKS Cluster | Active |
Active | |
Fastly Amsterdam (AMS) | Active |
Fastly Hong Kong (HKG) | Active |
Fastly London (LHR) | Active |
Fastly Los Angeles (LAX) | Active |
Fastly New York (JFK) | Active |
Fastly Sydney (SYD) | Active |
Full Platform | Active |
Google Apps Analytics | Active |
Logged In Users | Active |
Media | Active |
Mongo Cluster | Active |
Pharos | Active |
RabbitMQ | Active |
Redis Cluster | Active |
Sentry Dashboard | Active |
Stats | Active |
Talaria | Active |
Active | |
WFE | Active |
View the latest incidents for RebelMouse and check for official updates:
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Oct. 19, 2024, 8:30 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Sept. 5, 2024, 9:54 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Aug. 8, 2024, 2:07 p.m.
Description: ## Chronology of the incident \(UTC\) * Jul 25, 2024 9:56 AM Update for mitigation of message broker \(Amazon MQ for RabbitMQ\) connectivity issues was deployed to all applications. * Jul 27, 2024 1:10 AM RabbitMQ instances experienced a sudden 40% increase in CPU utilization due to an unexpected number of external requests. * Jul 27, 2024 1:12 AM Applications started to observe increased latencies and errors due to RabbitMQ degradation. To mitigate the issue, applications were automatically restarted based on the new connectivity monitoring update. * Jul 27, 2024 1:12 AM Platform began experiencing periodic service unavailability affecting the logged-in users caused by applications restarting. * Jul 27, 2024 2:20 AM Rollback of updates related to RabbitMQ connectivity was initiated to stabilize the platform. * Jul 27, 2024 2:25 AM Platform recovered after the rollback of updates related to RabbitMQ connectivity. ## The impact of the incident * Periodic service unavailability affecting the logged-in user experience of the RebelMouse application. * Data processing delays occurred during the incident. ## The underlying cause if known The root cause of the incident was the newly deployed RabbitMQ connectivity update. The update introduced a real-time RabbitMQ connectivity monitoring and restart mechanism to ensure system reliability. However, the update coincided with an unexpected increase in external requests, causing a significant load on the RabbitMQ instances. ## Actions taken & Preventive Measures * Alerts have been set up to notify the team if RabbitMQ CPU consumption exceeds a certain threshold. * Changes to connectivity monitoring will be re-evaluated to prevent similar incidents in the future. * Consider replacing RabbitMQ with an AWS managed solutions to improve stability
Status: Postmortem
Impact: Minor | Started At: July 27, 2024, 1:27 a.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: July 12, 2024, 10:20 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.