Time: Feb. 28, 2019, 12:32 a.m.

Status: Resolved

Update: Our engineering team has resolved this issue. Parent initiated forms and Schoolbox integration pages are now functional. The issue was an effect of security measures we put in place following a Pen Test and we have corrected the problem.

Time: Feb. 25, 2019, 5:21 a.m.

Status: Investigating

Update: We are aware of issues relating to parents getting timed out intermittently of the Parent Initiated Forms page and Schoolbox integration page. This presents in the form of parents seeing the EdSmart login screen. Engineering is currently working on resolving this issue. In the meantime, if you experience this issue or have parents raising this issue, please email [email protected] and we can update you as soon as the issue is resolved.

Time: March 12, 2019, 7:30 a.m.

Status: Resolved

Update: Our Engineering team have deployed a new outbound email module. This redesign has enabled us to fix the two issues causing the duplicate messages - messages being placed in our outbound queue multiple times, as well as our outbound message processing not coping with the volume of messages to send. Now, the duplicate email issue is resolved, and in addition, average times from an email is dropped into the outbound queue, to when it is sent, are a significant improvement on the previous system. We also now can deliver emails faster; better handle and log errors; and scale way beyond our current peak email loads.

Time: Feb. 11, 2019, 11:10 p.m.

Status: Identified

Update: Our engineering team is continuing to work on a fix for the duplicate emails, which will be around transaction handling to ensure messages that have been sent are removed properly from the queue, particularly when the system is under load.

Time: Feb. 8, 2019, 4:01 a.m.

Status: Identified

Update: While monitoring the results of the fixes we made for duplicate emails last weekend, we are seeing a few duplicate emails being sent for slips and broadcasts, particularly to staff. This is at a much smaller scale than before, and also appears to be related to times of peak load on our database. Engineering's plan is to address our school import processing, which will reduce load, and in parallel, they are working to resolve the bugs related to duplicate emails.

Time: June 11, 2019, 2:46 a.m.

Status: Resolved

Update: A number of updates to the import processing have been implemented, and overnight imports and processing are stable. For any further queries regarding imports please contact Support via help.edsmart.com.

Time: Feb. 14, 2019, 4:12 a.m.

Status: Identified

Update: In general imports are stable after some rescheduling of some import tasks, but we are manage a group of individual use cases and assisting schools where imports are delayed. Engineering continues to consider, review and implement improvements as monitoring identifies bottlenecks.

Time: Feb. 11, 2019, 11:13 p.m.

Status: Identified

Update: Our engineering team is continuing to work on improvements to import processing. Symptoms of delayed import processing include the following - - List appearing as empty - Slips or Broadcasts appearing to have no recipients - Students appearing to have no Contacts In the meantime, if you experience these symptoms and need to send slips or broadcasts urgently, email [email protected] and we will run a manual sync for your school.

Time: Feb. 8, 2019, 3:52 a.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Feb. 8, 2019, 3:16 a.m.

Status: Investigating

Update: We are aware of issues with delays in school data importing and processing into EdSmart. We have reached a point where the import jobs are running behind and not keeping up with the queue of data. Symptoms of delayed import processing include the following - - List appearing as empty - Slips or Broadcasts appearing to have no recipients - Students appearing to have no Contacts Engineering is currently working re-tuning our import and processing jobs will continue during the weekend to have improvements in place for early next week. In the meantime, if you experience these symptoms and need to send slips or broadcasts urgently, email [email protected] and we will run a manual sync for your school.

Time: Feb. 4, 2019, 5:25 a.m.

Status: Resolved

Update: Queues for Slips and Broadcasts have been processing normally for the last 2 days.

Time: Feb. 2, 2019, 4:30 a.m.

Status: Monitoring

Update: Queued Slips and Broadcasts have been resumed, after identifying and removing duplicates. We are continuing work on resolving the root cause. We have also implemented alerts to notify us of duplicates, and will make best endeavours to remove any that occur while we are in the process of fixing the issue.

Time: Feb. 1, 2019, 6:26 a.m.

Status: Monitoring

Update: Slips and Broadcasts are currently paused from sending due to the duplicate email issue. Once we resume sending, we will monitor the backlog and ensure everything is sent.

Time: Feb. 1, 2019, 5:20 a.m.

Status: Monitoring

Update: We are aware that some Slips and Broadcasts scheduled today are still in 'Queued' state and have not sent. The backlog of Slips and Broadcasts is gradually being processed. We are monitoring the queue to make sure nothing is missed.

Time: Feb. 4, 2019, 5:24 a.m.

Status: Resolved

Update: After implementing some changes on Saturday and monitoring over the last 2 days, we are no longer seeing repeated emails being sent for the same slip or broadcast. Specific steps taken include: * Re-organising the sending of Slips module with better multi-threading to avoid potentially the module being overwhelmed when under extreme load in an individual thread. Basically better spreading the load across threads better so more work can be done simultaneously. * Adding better fail safes to the send module to make sure the Slip has not been sent already. * Adding new monitoring so that even if all changes have not been effective we are alerted if something seems out of the ordinary. Since those changes were implemented late on Saturday we've processed a great deal of Slips including ones we'd held back from last week. We have not seem a recurrence of the issue.

Time: Feb. 2, 2019, 3:23 p.m.

Status: Monitoring

Update: Amendments to messaging modules and routines designed to mitigate the root cause have been deployed, We will continue to monitor.

Time: Feb. 2, 2019, 10:57 a.m.

Status: Monitoring

Update: Slips and Broadcasts have resumed sending, after identifying and removing duplicates. We are continuing work on resolving the root cause. We have also implemented alerts to notify us of duplicates, and will make best endeavours to remove any that occur while we are in the process of fixing the issue.

Time: Feb. 1, 2019, 6:07 a.m.

Status: Monitoring

Update: We are once again seeing repeated emails being sent for slips and broadcasts, and have made the call to pause them from sending while we do further investigation into the root cause. We will provide an update once we have resumed sending.

Time: Jan. 31, 2019, 11:16 p.m.

Status: Monitoring

Update: Summary and update on this issue - Over the past 24 hours we’ve experienced a fault in the platform which resulted in a limited number of forms resulting in duplicate email notifications to parents. The issue appears to emanate from the module which processes scheduled Slips/Broadcasts and generates the resulting email notifications. The issue only appears when the platform is under extreme load, it is not a fault which manifests every time, making tracing and diagnosis difficult. The issue existed only for a short time, but did result in some recipients seeing multiple copies of notifications for the same Slip/Broadcast. When the issue first became apparent we, for a time, paused outbound notifications to allow analysis of the data. Over the past few hours we have worked through queued messages to, where possible, identify and remove duplicates. Remediation actions have included changes to our database scaling configuration; a minor adjustment to the scheduled process; and close examination of the module in question. At this time the platform is operating normally and all outbound messages are being delivered. The engineering team is continuing to monitor the platform’s performance and review the code.

Time: Jan. 31, 2019, 3:54 p.m.

Status: Identified

Update: Email queues are clearing, backlogs should be out in next few minutes. We continue to work on the root cause, but have several areas within one particular module being closely reviewed.

Time: Jan. 31, 2019, 2:04 a.m.

Status: Investigating

Update: Engineering has temporarily suspended emails for slips and broadcasts from being sent, while the root cause for duplicate emails is being investigated. Once a fix has been applied, we will resume sending emails.

Time: Jan. 31, 2019, 1:32 a.m.

Status: Investigating

Update: Engineering currently aims to put a stop to the repeated emails for the affected slips. We will advise when that has been done.

Time: Jan. 31, 2019, 12:52 a.m.

Status: Investigating

Update: We have received reports of parents receiving repeated emails for slips and broadcasts. Engineering is investigating and working to resolve the issue. We will provide updates here for this issue.