Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Molecule and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Molecule.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
Europe | Active |
Amazon Web Services - Databases | Active |
Amazon Web Services - DNS | Active |
Amazon Web Services - EC2 | Active |
Amazon Web Services - EKS | Active |
Amazon Web Services - File Storage | Active |
Auth0 User Authentication | Active |
Molecule Europe | Active |
US | Active |
Amazon Web Services - Databases | Active |
Amazon Web Services - DNS | Active |
Amazon Web Services - EC2 | Active |
Amazon Web Services - EKS | Active |
Auth0 User Authentication | Active |
Molecule | Active |
View the latest incidents for Molecule and check for official updates:
Description: This incident has been resolved. We have not seen any issues with reconciliations or any 3rd party data.
Status: Resolved
Impact: None | Started At: July 19, 2024, 9:10 p.m.
Description: # **Incident Report: Incomplete Valuation Data - Resolved** **Date of Resolution:** July 1, 2024 ## **Summary** On Monday morning, July 1, 2024, an issue where certain trades were missing valuation entries was identified and addressed the same day. We apologize for any inconvenience this may have caused and appreciate your understanding. ## **What happened?** The Molecule application has a feature that limits how far into the future valuation entries will be created when trades are made. However, there was no mechanism to create entries for skipped as\_of dates when they became relevant. These previously excluded entries became relevant for the first time on July 1, and some customers began reporting the issue on Friday, June 28. ## **Why** * When implementing the as\_of\_date limits for entry building, a corresponding job to build skipped entries before they became relevant was not available to be run in production. * The system did not include sufficient checks to identify these issues ahead of time. ## **Corrective/Preventive Actions** * A script was deployed immediately to generate missing entries, addressing the issue and correcting the problem for impacted customers as soon as possible. This script will run daily to ensure all entries are built until we remove the limit for good. * An internal check has been created to catch missing entries ahead of time. We apologize for any inconvenience caused and appreciate your understanding as we work to improve our systems and prevent similar issues in the future.
Status: Postmortem
Impact: None | Started At: July 1, 2024, 7:16 p.m.
Description: **Summary** On May 10, 2024, from 08:03 AM to 08:09 AM CST, a brief service interruption affected our US-based users. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding. **What happened?** The system became unavailable to US users for approximately 6 minutes due to excessive frontend resource requests that resulted in unresponsive web application nodes. **Why?** The outage was triggered by an unusual surge in resource demand due to excessive frontend resource requests to the Molecule web application. This placed an unexpectedly high load on our systems, leading to unresponsive web application nodes. **Corrective/Preventive Actions** To enhance our system's reliability and efficiency, we are implementing the following measures: * Enhanced System Monitoring: We have upgraded our monitoring capabilities to detect and address anomalies more effectively before affecting service availability. * Scalability Improvements: We are increasing the scalability of our web servers to handle spikes in demand more fluidly. * Rate Limiting Enhancements: We plan to improve our rate-limiting measures to prevent any single user or service from making excessive requests in a short time frame. * Software Updates: A specific software issue that allowed for unnecessary resource consumption to help alleviate the issue was discovered and fixed. Once again, we thank you for your patience and understanding as we strive to ensure the reliability of our services.
Status: Postmortem
Impact: None | Started At: May 10, 2024, 1:16 p.m.
Description: **Summary** On April 29, 2024, from 11:34 AM to 12:05 PM CST, a service interruption affected our US-based users. We apologize for any inconvenience. **What happened?** The system was intermittently unavailable to US-based users for approximately 30 minutes. **Why?** The outage was triggered by a large trade upload, which consumed significant system resources. We mitigated the issue by scaling down the heavy resource consumption to allow the system to get back to a healthy state. **Corrective/Preventive Actions** We are committed to providing and continuously improving reliable and efficient software, and this incident has highlighted areas for improvement. To prevent future occurrences of similar issues, we are taking the following steps: 1. **Software Updates**: We are correcting the issue that caused the heavy consumption of resources. 2. **Enhanced System Monitoring:** We are focused on improving our system monitoring to detect and respond to such anomalies before they impact service availability. 3. **Performance Enhancements:** We will roll out several performance improvements over the next few weeks to enhance system resource utilization efficiency. Thank you for your patience.
Status: Postmortem
Impact: Critical | Started At: April 29, 2024, 4:44 p.m.
Description: **Summary** On the morning of March 5, 2024, there were some delays related to prior day market data being copied to the current day. The incident was first noticed at approximately 8:00 AM CST and was fully resolved by 12:00 PM CST. We apologize for the interruption, and would like to share a summary of the incident. **What Happened** You may have noticed that some or all market data from the previous day did not get copied to the current day, resulting in incorrect current day valuations. **Why?** A bug was discovered that created duplicates of certain market prices. The result of these duplicates was the inability to successfully copy the prior day's market prices due to database constraints. **Preventing This Type of Issue** The bug that caused this issue has been fixed. Thank you for your patience, and once again––we’re sorry for the interruption.
Status: Postmortem
Impact: None | Started At: March 5, 2024, 3:45 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.