-
Time: April 26, 2023, 6:27 p.m.
Status: Resolved
Update: We are closing this incident after running at full performance for the day. We believe we are on track to fully resolve and prevent this from occurring going forward. We will send an email update with more information about the issue and its resolution and prevention measures we are putting into place.
-
Time: April 26, 2023, 6:27 p.m.
Status: Resolved
Update: We are closing this incident after running at full performance for the day. We believe we are on track to fully resolve and prevent this from occurring going forward. We will send an email update with more information about the issue and its resolution and prevention measures we are putting into place.
-
Time: April 26, 2023, 12:30 p.m.
Status: Monitoring
Update: The first wave of corrective actions have taken place and all servers and services are back online. All inbound mail queues have processed and our general performance stats are back to normal. Now that the root cause is understood, we will work closely with our datacenter and storage teams to put monitoring in place that should alert us before it occurs in the future.
-
Time: April 26, 2023, 8:25 a.m.
Status: Monitoring
Update: As per the recommendation of our storage vendor and datacenter teams, we executed a plan which seems to have had positive impacts on performance overall. We still have some work to do, but we remain cautiously optimistic.
-
Time: April 26, 2023, 8:25 a.m.
Status: Monitoring
Update: As per the recommendation of our storage vendor and datacenter teams, we executed a plan which seems to have had positive impacts on performance overall. We still have some work to do, but we remain cautiously optimistic.
-
Time: April 26, 2023, 3:45 a.m.
Status: Identified
Update: The storage subsystem has been confirmed to be the issue with performance. We are executing a plan based on the storage vendor's recommendations. Providing your service and keeping everything running is our number one priority. Please know that we are working to get this completed as soon as possible in order to get everything back to normal operation. Thanks for your continued understanding.
-
Time: April 26, 2023, 3:45 a.m.
Status: Identified
Update: The storage subsystem has been confirmed to be the issue with performance. We are executing a plan based on the storage vendor's recommendations. Providing your service and keeping everything running is our number one priority. Please know that we are working to get this completed as soon as possible in order to get everything back to normal operation. Thanks for your continued understanding.
-
Time: April 26, 2023, 12:10 a.m.
Status: Identified
Update: Our performance continues to be degraded. Our datacenter has identified that the storage subsystem is doing a low-level filesystem check. During that filesystem check, all other operations operate degraded. They are analyzing with the storage vendor to understand why the check is impacting performance so greatly. This is a normal operation that runs ever other week and isn't new to our system. Mail is delivering very slowly to your inboxes and will continue to queue up during this degradation. At present, we do not have an ETA, but hope to have things operational through the night.
-
Time: April 26, 2023, 12:10 a.m.
Status: Identified
Update: Our performance continues to be degraded. Our datacenter has identified that the storage subsystem is doing a low-level filesystem check. During that filesystem check, all other operations operate degraded. They are analyzing with the storage vendor to understand why the check is impacting performance so greatly. This is a normal operation that runs ever other week and isn't new to our system. Mail is delivering very slowly to your inboxes and will continue to queue up during this degradation. At present, we do not have an ETA, but hope to have things operational through the night.
-
Time: April 25, 2023, 10:14 p.m.
Status: Identified
Update: We have updated the status to critical since performance remains severely degraded and we have two mailstores that are performing filesystem repairs. We will continue to keep you updated here.
-
Time: April 25, 2023, 10:14 p.m.
Status: Identified
Update: We have updated the status to critical since performance remains severely degraded and we have two mailstores that are performing filesystem repairs. We will continue to keep you updated here.
-
Time: April 25, 2023, 8:18 p.m.
Status: Identified
Update: Hello Everyone. Our datacenter and storage teams continue to analyze the situation. Given the current situation, we may need to do an emergency maintenance tonight on the storage subsystem. Please note that email is being queued up on our servers and is tricking through.
The queues will be delivered once the issue resolves.
-
Time: April 25, 2023, 8:18 p.m.
Status: Identified
Update: Hello Everyone. Our datacenter and storage teams continue to analyze the situation. Given the current situation, we may need to do an emergency maintenance tonight on the storage subsystem. Please note that email is being queued up on our servers and is tricking through.
The queues will be delivered once the issue resolves.
-
Time: April 25, 2023, 6:18 p.m.
Status: Identified
Update: Our performance degradation continues. Our storage vendor continues to analyze and we are monitoring the situation. Everything is up and running, but running degraded. Thanks for your patience while we work through this.
-
Time: April 25, 2023, 6:18 p.m.
Status: Identified
Update: Our performance degradation continues. Our storage vendor continues to analyze and we are monitoring the situation. Everything is up and running, but running degraded. Thanks for your patience while we work through this.
-
Time: April 25, 2023, 3:48 p.m.
Status: Identified
Update: Thanks for your patience. Our storage engineers are analyzing. All services are up, albeit running degraded.
-
Time: April 25, 2023, 3:48 p.m.
Status: Identified
Update: Thanks for your patience. Our storage engineers are analyzing. All services are up, albeit running degraded.
-
Time: April 25, 2023, 2:18 p.m.
Status: Identified
Update: The degradation continues. We are prepping for a brief outage to reboot our storage. We believe that the short outage will be far better than an extended degradation.
-
Time: April 25, 2023, 2:18 p.m.
Status: Identified
Update: The degradation continues. We are prepping for a brief outage to reboot our storage. We believe that the short outage will be far better than an extended degradation.
-
Time: April 25, 2023, 12:58 p.m.
Status: Identified
Update: BlueTie's email service is experiencing a performance degradation due to the storage subsystem. We are actively monitoring and may need to engage in our storage vendor to resolve. All services are operational, but performance degraded.
