Time: Nov. 18, 2019, 11:10 p.m.

Status: Postmortem

Update: We wanted to share our research and results related to this incident in late October. Thank you for your patience at that time--here’s what we found. #### The First Event On Tuesday, October 22, in the afternoon, Amazon Web Services \(AWS\) began experiencing a DDoS \(distributed denial of service\) attack. To be clear, this is not an attempted intrusion – it’s an attempt to slow down a provider by inundating it with traffic. Amazon has confirmed this in e-mails to customers, but has not posted about this incident, for some reason. AWS, our datacenter, largely stayed up during this incident, but experienced intermittent seconds of downtime. #### Knock-On Effects These seconds of downtime caused pieces of our infrastructure \(our Kubernetes masters, specifically\) to reboot. This is routine and shouldn’t cause major issues. However, due to a misconfiguration, these servers had conflicting information about how many servers they were each managing. This issue had not previously manifested, because these servers had not recently needed a reboot. As a result, two separate masters began spinning up and spinning down additional servers, thus causing occasional issues during the late evening. Increasing our minimum server count fixed the issue temporarily, and the root cause was found and fixed, that weekend. #### Knock-On Effects We learned that we should fully reboot our servers as part of our routine maintenance, to catch issues like this. We plan to do this going forward. We apologize for the intermittent warnings and slowness we experienced the week of October 22, and plan to avoid this type of incident in the future.

Time: Oct. 23, 2019, 2:21 p.m.

Status: Resolved

Update: We were notified by Amazon that all of their network-related issues have been resolved. The Molecule app has been stable all morning and we do not expect that to change. Of course, we will continue to monitor all performance closely.

Time: Oct. 23, 2019, 3:17 a.m.

Status: Monitoring

Update: Amazon, Molecule's primary provider of network and infrastructure services, is currently experiencing network-related issues that are causing instability of the Molecule app. We are continuing to monitor it closely and are investigating possible infrastructure modifications to stabilize the app. The app is currently fully operational and updates will be provided if that status changes.

Time: Nov. 18, 2019, 11:10 p.m.

Status: Postmortem

Update: We wanted to share our research and results related to this incident in late October. Thank you for your patience at that time--here’s what we found. #### The First Event On Tuesday, October 22, in the afternoon, Amazon Web Services \(AWS\) began experiencing a DDoS \(distributed denial of service\) attack. To be clear, this is not an attempted intrusion – it’s an attempt to slow down a provider by inundating it with traffic. Amazon has confirmed this in e-mails to customers, but has not posted about this incident, for some reason. AWS, our datacenter, largely stayed up during this incident, but experienced intermittent seconds of downtime. #### Knock-On Effects These seconds of downtime caused pieces of our infrastructure \(our Kubernetes masters, specifically\) to reboot. This is routine and shouldn’t cause major issues. However, due to a misconfiguration, these servers had conflicting information about how many servers they were each managing. This issue had not previously manifested, because these servers had not recently needed a reboot. As a result, two separate masters began spinning up and spinning down additional servers, thus causing occasional issues during the late evening. Increasing our minimum server count fixed the issue temporarily, and the root cause was found and fixed, that weekend. #### Knock-On Effects We learned that we should fully reboot our servers as part of our routine maintenance, to catch issues like this. We plan to do this going forward. We apologize for the intermittent warnings and slowness we experienced the week of October 22, and plan to avoid this type of incident in the future.

Time: Oct. 23, 2019, 2:21 p.m.

Status: Resolved

Update: We were notified by Amazon that all of their network-related issues have been resolved. The Molecule app has been stable all morning and we do not expect that to change. Of course, we will continue to monitor all performance closely.

Time: Oct. 23, 2019, 3:17 a.m.

Status: Monitoring

Update: Amazon, Molecule's primary provider of network and infrastructure services, is currently experiencing network-related issues that are causing instability of the Molecule app. We are continuing to monitor it closely and are investigating possible infrastructure modifications to stabilize the app. The app is currently fully operational and updates will be provided if that status changes.

Time: Oct. 22, 2019, 10:01 p.m.

Status: Resolved

Update: Amazon registered a DNS fault, and the Molecule application was unavailable for approximately 7 minutes.

Time: Oct. 22, 2019, 10:01 p.m.

Status: Resolved

Update: Amazon registered a DNS fault, and the Molecule application was unavailable for approximately 7 minutes.

Time: Oct. 14, 2019, 6:05 p.m.

Status: Resolved

Update: As a result of an ICE outage, there is a possibility that trades were not captured in Molecule for a brief period of time this morning. We have reset all connections with ICE and have requested trade history. All deals should currently be captured in Molecule. Please contact the customer success team at [email protected] if you find any missing trades.