Is there an AchieveIt outage?
AchieveIt status: Systems Active
Last checked: 10 minutes ago
Get notified about any outages, downtime or incidents for AchieveIt and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 10 minutes ago
Get notified about any outages, downtime or incidents for AchieveIt and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for AchieveIt.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Web Application - Commercial Environment | Active |
| Web Application - US Government Environment | Active |
View the latest incidents for AchieveIt and check for official updates:
Description: ## **Summary** On Monday, November 29, 2021 at approximately 15:56 UTC we became aware that the SSL certificate for the domain `gov-api.achieveit.com` expired and was not properly updated with a new certificate. This domain is used by the AchieveIt web application to send and retrieve data between the browser and the server. When the SSL certificate expired, the user’s browser began rejecting requests to that domain causing required data connections to fail. This interrupted the service for most customers using the system in our US Government environment. We remediated the problem by manually updating the SSL certificate and by 16:40 UTC all service was fully restored. ## **Root Cause** We use an automated process to update the SSL certificate for each service as that certificate approaches its expiration. Most of those SSL certificates have a one year valid lifespan and are renewed between 15-30 days before the expiration date. In the the case of the `gov-api.achieveit.com` domain, the SSL certificate was updated as expected but the process that makes the certificate available to use failed. This resulted in the previous version of the certificate being used after it expired and subsequently browsers contacting the `gov-api.achieveit.com` domain rejecting the responses due to the expired certificate. Our investigation uncovered that the root cause was a missing permission that caused the process failure. Upon correcting the missing permission, we were able to trigger the automated certificate update successfully. ## **Mitigation Actions** In order to reduce the likelihood of a similar failure interrupting the service in the future, we have implemented additional monitoring to detect expired SSL certificates. We also corrected the missing permission to ensure that future automated certificate renewals are propagated successfully.
Status: Postmortem
Impact: Critical | Started At: Nov. 29, 2021, 3:56 p.m.
Description: ## **Summary** On Monday, November 29, 2021 at approximately 15:56 UTC we became aware that the SSL certificate for the domain `gov-api.achieveit.com` expired and was not properly updated with a new certificate. This domain is used by the AchieveIt web application to send and retrieve data between the browser and the server. When the SSL certificate expired, the user’s browser began rejecting requests to that domain causing required data connections to fail. This interrupted the service for most customers using the system in our US Government environment. We remediated the problem by manually updating the SSL certificate and by 16:40 UTC all service was fully restored. ## **Root Cause** We use an automated process to update the SSL certificate for each service as that certificate approaches its expiration. Most of those SSL certificates have a one year valid lifespan and are renewed between 15-30 days before the expiration date. In the the case of the `gov-api.achieveit.com` domain, the SSL certificate was updated as expected but the process that makes the certificate available to use failed. This resulted in the previous version of the certificate being used after it expired and subsequently browsers contacting the `gov-api.achieveit.com` domain rejecting the responses due to the expired certificate. Our investigation uncovered that the root cause was a missing permission that caused the process failure. Upon correcting the missing permission, we were able to trigger the automated certificate update successfully. ## **Mitigation Actions** In order to reduce the likelihood of a similar failure interrupting the service in the future, we have implemented additional monitoring to detect expired SSL certificates. We also corrected the missing permission to ensure that future automated certificate renewals are propagated successfully.
Status: Postmortem
Impact: Critical | Started At: Nov. 29, 2021, 3:56 p.m.
Description: ## Summary On Thursday, September 2, 2021 from approximately 14:06 UTC until 16:22 UTC our US commercial and government hosting environments both experienced a service interruption related to a configuration error in the service we use to manage our domain name and DNSSEC configuration, GoDaddy. We observed that our monitoring tools reported increasing errors of DNS queries for domain names used by our application, such as my.achieveit.com, during the 136 minutes of the interruption. As the errors increased, it is possible that some users were not able to reach the system. At approximately 16:20 UTC we cleared the interruption by manually resetting the state on our DNSSEC configuration, and this resulted in queries succeeding across all our monitoring regions within minutes. ## Root Cause We confirmed with GoDaddy support that on September 2, 2021, they experienced a failure in their managed DNSSEC service that caused some of the regular updates to DNSSEC records to not succeed. We observed that the outcome of this failure was that one of RRSIG signing key used to sign the DNSKEY records for the achieveit.com domain was not properly rotated and thus expired at 14:05 UTC on September 2. When the key expired, DNS resolvers that were handling queries for our domain began to error out due to the invalid key. Although this was a good indication that DNSSEC protection of our domain was working as desired, in this case it was due to an operational error and not a security fault. After investigating the issue and working with GoDaddy support to understand the root cause, we decided to toggle the DNSSEC configuration in an attempt to reset the expiration on the RRSIG signing key. At approximately 16:20 UTC we toggle the configuration and immediately observed that the RRSIG expiration was updated and DNS queries began to succeed again. Within a minutes, we observed that all our DNS monitoring errors cleared worldwide. ## Mitigation Actions The main mitigation action was taken by GoDaddy: they verified that they released patches for the DNSSEC system on September 2 and September 3 to correct the failure. AchieveIt will continue to monitor the rotation cycles of our DNSSEC records to ensure the fix GoDaddy put in place works as expected.
Status: Postmortem
Impact: Major | Started At: Sept. 2, 2021, 3 p.m.
Description: ## Summary On Thursday, September 2, 2021 from approximately 14:06 UTC until 16:22 UTC our US commercial and government hosting environments both experienced a service interruption related to a configuration error in the service we use to manage our domain name and DNSSEC configuration, GoDaddy. We observed that our monitoring tools reported increasing errors of DNS queries for domain names used by our application, such as my.achieveit.com, during the 136 minutes of the interruption. As the errors increased, it is possible that some users were not able to reach the system. At approximately 16:20 UTC we cleared the interruption by manually resetting the state on our DNSSEC configuration, and this resulted in queries succeeding across all our monitoring regions within minutes. ## Root Cause We confirmed with GoDaddy support that on September 2, 2021, they experienced a failure in their managed DNSSEC service that caused some of the regular updates to DNSSEC records to not succeed. We observed that the outcome of this failure was that one of RRSIG signing key used to sign the DNSKEY records for the achieveit.com domain was not properly rotated and thus expired at 14:05 UTC on September 2. When the key expired, DNS resolvers that were handling queries for our domain began to error out due to the invalid key. Although this was a good indication that DNSSEC protection of our domain was working as desired, in this case it was due to an operational error and not a security fault. After investigating the issue and working with GoDaddy support to understand the root cause, we decided to toggle the DNSSEC configuration in an attempt to reset the expiration on the RRSIG signing key. At approximately 16:20 UTC we toggle the configuration and immediately observed that the RRSIG expiration was updated and DNS queries began to succeed again. Within a minutes, we observed that all our DNS monitoring errors cleared worldwide. ## Mitigation Actions The main mitigation action was taken by GoDaddy: they verified that they released patches for the DNSSEC system on September 2 and September 3 to correct the failure. AchieveIt will continue to monitor the rotation cycles of our DNSSEC records to ensure the fix GoDaddy put in place works as expected.
Status: Postmortem
Impact: Major | Started At: Sept. 2, 2021, 3 p.m.
Description: As of 21:36 UTC (5:36 PM EDT), Microsoft has reported that it has mitigated the performance degradation on its CDN services. We have observed that AchieveIt application behavior has returned to normal. We will continue to monitor system performance, but we believe the issue is now resolved.
Status: Resolved
Impact: Minor | Started At: July 12, 2021, 7:40 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
