Time: Aug. 17, 2022, 9:04 a.m.

Status: Postmortem

Update: We have published a postmortem response to yesterday’s incident on our blog, available here: [https://blog.edsmart.com/critical-incident-response](https://blog.edsmart.com/critical-incident-response)

Time: Aug. 16, 2022, 6:35 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 16, 2022, 6:34 a.m.

Status: Monitoring

Update: (Editing to change the Incident Status to Resolved.) Service is now restored on the EdSmart Platform. We are awaiting a Root Cause Analysis (RCA) from the Microsoft Azure Support Team to explain how their infrastructure caused this significant outage whilst preventing us from activating failover processes. We will continue to monitor and work to implement measures to eliminate or reduce the impact of this problem should it happen again. Please continue to subscribe to https://status.edsmart.com for updates. If you are still experiencing issues, or would like a detailed RCA, please contact [email protected]. We apologise for the inconvenience this may have caused and appreciate your patience and understanding today.

Time: Aug. 16, 2022, 6:24 a.m.

Status: Monitoring

Update: Service is now restored on the EdSmart Platform. We are awaiting a Root Cause Analysis (RCA) from the Microsoft Azure Support Team to explain how their infrastructure caused this significant outage whilst preventing us from activating failover processes. We will continue to monitor and work to implement measures to eliminate or reduce the impact of this problem should it happen again. Please continue to subscribe to https://status.edsmart.com for updates. If you are still experiencing issues, or would like a detailed RCA, please contact [email protected]. We apologise for the inconvenience this may have caused and appreciate your patience and understanding today.

Time: Aug. 16, 2022, 4:30 a.m.

Status: Monitoring

Update: Updating to reflect that API and Application services are now running.

Time: Aug. 16, 2022, 4:28 a.m.

Status: Monitoring

Update: Our database is now running and we are restoring services on the EdSmart Platform and running diagnostic tests. You may experience some performance issues while systems start back up.

Time: Aug. 16, 2022, 4:15 a.m.

Status: Identified

Update: Updating outage information to show that the API is unavailable in addition to the application.

Time: Aug. 16, 2022, 4:10 a.m.

Status: Identified

Update: Microsoft Azure Support Engineers are working to cancel the stalled auto scaling operation. We are awaiting further updates from the Azure Support team and will continue to provide updates via https://status.edsmart.com.

Time: Aug. 16, 2022, 2:55 a.m.

Status: Identified

Update: The issue lies with the scale up/down functions in the Azure database platform. This is functionality we have been using without incident for some 8 years. The most recent automated scale up operation stalled and our engineering team were not able to manually resolve. We have found some references online, indicating that this is a very rare but known issue with Azure SQL. The nature of the issue also appears to have sidestepped our monitoring, for example, database connection monitoring, which did not trigger. It seems Azure 'thinks' the database is available when actually it isn't. The engineering team is dealing directly with senior account and engineering management at Microsoft Azure. In the meantime, they are also working to make the database available via other means, as the accepted Azure restore methods for failover are also unavailable due to the current issue.

Time: Aug. 16, 2022, 1:44 a.m.

Status: Investigating

Update: Microsoft Azure Support are still investigating the issue, it’s ranked at the highest priority in their support system and we are in constant contact with their account and technical management teams.

Time: Aug. 16, 2022, 12:06 a.m.

Status: Investigating

Update: Microsoft Azure Support have escalated management for this critical issue and are working with EdSmart to identify and resolve the problem urgently. Please continue to subscribe to https://status.edsmart.com/ for updates. Thank you for your understanding.

Time: Aug. 15, 2022, 10:51 p.m.

Status: Investigating

Update: Our Support and Engineering teams are aware of an issue with our Microsoft Azure SQL Database as of 6:30 am local time. We are investigating the issue with highest priority and have contacted Microsoft Support. Please continue to subscribe to status.edsmart.com for updates. We apologise for the inconvenience this may have caused and we endeavour to restore service as soon as possible. We appreciate your patience and understanding during this time.

Time: July 28, 2022, 6:20 a.m.

Status: Resolved

Update: The intermittent performance issues have been resolved. The platform has been steady since 10:00AM AEST. The EdSmart team will continue to monitor the platform closely over the next few days. If we identify any further issues we will advise via our Status Page, status.edsmart.com. If you are still experiencing issues, or would like a detailed root cause analysis, please contact [email protected].

Time: July 28, 2022, 2:45 a.m.

Status: Monitoring

Update: We have observed significant improvement in platform performance after changes made by our engineering team and will continue to monitor. We have also implemented automated handling of similar database issues to reduce their impact in the future. Please continue to subscribe to status.edsmart.com for updates. If you are still experiencing issues, or would like a detailed root cause analysis, please contact [email protected]. We apologise for the inconvenience this may have caused and appreciate your patience and understanding during this time.

Time: July 28, 2022, 2:41 a.m.

Status: Monitoring

Update: Automated fixes implemented, improvement in performance noted.

Time: July 27, 2022, 11:35 p.m.

Status: Investigating

Update: Hi, Our Support and Engineering teams are aware of intermittent performance issues with EdSmart. This is our highest priority and we have made a few configuration changes on our database which should start to have a noticeable impact on performance. We are continuing our investigation on the issue to identify the root cause. Please continue to subscribe to status.edsmart.com for updates. We apologise for the inconvenience this may have caused and we endeavour to rectify this as soon as possible. We appreciate your patience and understanding during this time.

Time: Jan. 20, 2022, 7:15 a.m.

Status: Resolved

Update: Earlier EdSmart had an incident where Form Template data was missing and as a result, the ability to use the application was limited. We have fully restored all Template data and have verified the issue is resolved. If you encounter any further issues or would like a detailed root cause analysis, please contact [email protected]. We apologise for any inconvenience.

Time: Jan. 20, 2022, 5:29 a.m.

Status: Monitoring

Update: We are aware of an issue with Form Templates missing data. Our Engineering team are putting remediations in place to resolve this issue. We will provide an update once EdSmart functionality is fully restored. We apologise for any inconvenience.

Time: Jan. 20, 2022, 4:56 a.m.

Status: Investigating

Update: We are aware of an issue where Form Templates are missing data. This will lead to an issue where users will not able able to create Forms/Slips/Broadcasts. Please bear with us while our team investigate and resolve this incident. We apologise for any inconvenience.

Time: April 15, 2021, 3:51 a.m.

Status: Resolved

Update: We have been monitoring platform performance and no issues have been recorded in over two weeks. We are marking this incident as resolved. If you experience any issues in the future please contact our Support team at [email protected].

Time: March 30, 2021, 12:10 a.m.

Status: Monitoring

Update: The Engineering Team has taken steps to address the recent platform performance which has since produced positive performance stability. The team continues to monitor for any issues and will respond accordingly.

Time: March 29, 2021, 3:55 a.m.

Status: Monitoring

Update: Our Engineering have increased capacity on the database to account for high levels of activity on the platform. This may have resulted in loss of access to EdSmart for 1-2 minutes. Response times are now starting to improve as a result.

Time: March 29, 2021, 3:44 a.m.

Status: Monitoring

Update: Our Engineering team have put in place several improvements to platform performance, and we have seen stability on the platform for the last month. However, we are aware of a slowdown on the platform this afternoon. It appears that this relates to a high load on the database, corresponding with a spike in sending and responding to Slips and Broadcasts. Response times are now improving and we will continue to monitor performance, in addition to broader efforts to improve performance on the platform.

Time: Feb. 10, 2021, 6:40 a.m.

Status: Monitoring

Update: We have put in place several measures to address the intermittent performance degradation we have been experiencing recently, including optimisation of database resources and rescheduling of background functions. We have also increased the scope of proactive monitoring, and implemented new tracing capabilities to help with performance analysis and improvement in the short term. We believe that these changes will have a significant impact, however our investigations are ongoing and we will provide further updates as they are available.

Time: Feb. 9, 2021, 6:14 a.m.

Status: Investigating

Update: At approximately 1.50pm and again at 3.30pm AEST today our primary database became saturated with a very large number of concurrent connections. The effect for the end user using EdSmart is a significant slow down in performance whilst the database infrastructure endeavours to service all the requests. This occurred on a previous occasion and our Engineering team has been engaged in a series of mitigation activities, including conferencing with domain experts at our hosting company, Microsoft Azure. The slow down lasts a relatively short window, but obviously for people using EdSmart at the time it leads to frustrations over long load times. EdSmart usage is some 50% above the same time next year, and other than these couple of short blips, the technology platform has been stable and performing as normal. Our Engineering team, with help from Microsoft, is continuing to work to identify the source of the issue.

Time: Feb. 9, 2021, 5:17 a.m.

Status: Investigating

Update: We are aware of a slowdown on the platform. Our Engineering team is investigating the issue. We apologise for any inconvenience. We will be providing updates with any developments.

Time: Feb. 1, 2021, 10:51 p.m.

Status: Resolved

Update: When trying to open a Slip, users would see an error, ‘Sorry, can’t find that slip,’ or a blank Slip in their browser. This was due to a configuration change in the EdSmart API. The change has been revised and the issue is now resolved. We apologise for the issue, and if you experience any further problems please contact Support at [email protected].

Time: Feb. 1, 2021, 9:58 p.m.

Status: Monitoring

Update: We have applied a fix for the issue and are monitoring for any further issues. We sincerely apologise for any inconvenience.

Time: Feb. 1, 2021, 9:40 p.m.

Status: Investigating

Update: We are aware of an issue where parents receive an error, "Sorry, cannot find Slip" when trying to open Slips. Our Engineering team is working to resolve the issue ASAP. We will provide further updates shortly.