Time: Feb. 5, 2019, 10:35 p.m.

Status: Resolved

Update: There was a brief gap this morning with our corporate website hosted on HubSpot as a new SSL certificate was being deployed and the existing SSL certificate was being revoked. During this gap, users would see in their web browser that the website was not secured with an active SSL certificate. We have since rectified the situation and have deployed the new active SSL certificate. We apologize for any inconvenience or concern this may have caused.

Time: Feb. 7, 2019, 12:53 a.m.

Status: Postmortem

Update: RADAR immediately asked for a root cause on i\).. how the issue was caused and ii\) why it took so long to fix. HubSpot responded with the following: HubSpot spent some time investigating the flow of events for RADAR's particular case. HubSpot uses JIRA to track engineering issues \(filed by customer support\). HubSpot recently rolled out a new "Customer Urgency" system to flag relevant and painful issues to the owning engineering leads of a particular area of the product. In this scenario, a JIRA ticket was actually **never** filed. Rather, a series of Slack messages were sent trying to find the correct people/teams to look into RADAR's urgent issue. This is actually against HubSpot’s own internal protocol and is likely one of the driving reasons why the issue took so long to get to the correct people. For obvious reasons, individual messaging is less than ideal, as conversations on progress and the issue itself were siloed in 1:1 conversations, so as people went in an out of meetings, etc. the full picture was lost and not put in front of the right people. If the JIRA ticket had been created with appropriate details and urgency, it likely would have been addressed much quicker, as everyone internally would have had visibility to the scope and details of the issue, and the engineering owners of the issue would have been flagged to address the critical issue. In addition, when the request to update the TLS settings for RADAR was selected for work, the operational team did not verify that RADAR had a custom domain and this information was required to be added to the certificate and wasn’t. HubSpot will be ensuring feedback is given to the involved parties, as well as make sure our "critical" protocol internal instructions properly reflect the correct protocol to ensure this support flow does not occur again.

Time: Feb. 7, 2019, 12:45 a.m.

Status: Resolved

Update: After requesting that HubSpot tighten their TLS protocol and cipher suites to be more secure, HubSpot misconfigured the settings causing an error page to appear in the client's web browser stating "your connection is not private" and directing visitors to exit using the "Back to safety" button, or to click on the "Advanced" button and proceed to RADAR's corporate website with the warning that it is unsafe. After 4 hours of troubleshooting, HubSpot fixed their mistake and reconfigure TLS 1.2 correctly. We sat down with HubSpot to have a retrospective and root cause discussion regard this incident and how they were to prevent this in the future. We apologize for the inconvenience this may have caused.

Time: Feb. 7, 2019, 12:53 a.m.

Status: Postmortem

Update: RADAR immediately asked for a root cause on i\).. how the issue was caused and ii\) why it took so long to fix. HubSpot responded with the following: HubSpot spent some time investigating the flow of events for RADAR's particular case. HubSpot uses JIRA to track engineering issues \(filed by customer support\). HubSpot recently rolled out a new "Customer Urgency" system to flag relevant and painful issues to the owning engineering leads of a particular area of the product. In this scenario, a JIRA ticket was actually **never** filed. Rather, a series of Slack messages were sent trying to find the correct people/teams to look into RADAR's urgent issue. This is actually against HubSpot’s own internal protocol and is likely one of the driving reasons why the issue took so long to get to the correct people. For obvious reasons, individual messaging is less than ideal, as conversations on progress and the issue itself were siloed in 1:1 conversations, so as people went in an out of meetings, etc. the full picture was lost and not put in front of the right people. If the JIRA ticket had been created with appropriate details and urgency, it likely would have been addressed much quicker, as everyone internally would have had visibility to the scope and details of the issue, and the engineering owners of the issue would have been flagged to address the critical issue. In addition, when the request to update the TLS settings for RADAR was selected for work, the operational team did not verify that RADAR had a custom domain and this information was required to be added to the certificate and wasn’t. HubSpot will be ensuring feedback is given to the involved parties, as well as make sure our "critical" protocol internal instructions properly reflect the correct protocol to ensure this support flow does not occur again.

Time: Feb. 7, 2019, 12:45 a.m.

Status: Resolved

Update: After requesting that HubSpot tighten their TLS protocol and cipher suites to be more secure, HubSpot misconfigured the settings causing an error page to appear in the client's web browser stating "your connection is not private" and directing visitors to exit using the "Back to safety" button, or to click on the "Advanced" button and proceed to RADAR's corporate website with the warning that it is unsafe. After 4 hours of troubleshooting, HubSpot fixed their mistake and reconfigure TLS 1.2 correctly. We sat down with HubSpot to have a retrospective and root cause discussion regard this incident and how they were to prevent this in the future. We apologize for the inconvenience this may have caused.

Time: Dec. 14, 2018, 6:28 p.m.

Status: Resolved

Update: We had an unhealthy API instance, that wasn't detected by our Amazon EC2 health checks. The issue was intermittent because some requests were hitting one of the healthy API instances. We removed the unhealthy instance manually from the autoscaling group and it has since been replaced with a healthy instance. All services are back to full function. Ticket RADDEV-13610.

Time: Oct. 2, 2018, 5:36 p.m.

Status: Resolved

Update: On June 14th starting at 5:10am customers started to experience 500 level errors when creating or updating incidents. During this time, users could still view incidents, report on incidents and perform other work with other parts of the application including the law library; no other functionality was impacted except the ability to create or update existing incidents. Customer support was contacted approximately at 5:30am and after troubleshooting escalated to operations team around 6:35am. Operations team determined it was caused by a known, but obscure issue, with Postgres 9.6.1. RADAR was put into maintenance mode at 7:50am and the database was patched. RADAR was placed back online at 8:06am with full services restored.