-
Time: Aug. 26, 2024, 7:55 a.m.
Status: Postmortem
Update: A postmortem is now available at [
[email protected]](mailto:
[email protected]).
-
Time: Aug. 16, 2024, 4:59 a.m.
Status: Resolved
Update: Metrics stayed at normal levels since it last update. We're closing the incident and will follow-up with our partners for measures to prevent this from happening again.
-
Time: Aug. 15, 2024, 11:06 p.m.
Status: Monitoring
Update: We are back to normal levels, will continue to monitor overnight and close the incident tomorrow morning German time if it remains stable.
-
Time: Aug. 15, 2024, 10:52 p.m.
Status: Monitoring
Update: We continue to monitor it, we are close to the normal values.
-
Time: Aug. 15, 2024, 10:42 p.m.
Status: Identified
Update: The emergency maintenance that was started by our connectivity provider's partner at around 19 UTC has improved the situation according to our internal graphs. The missing connections have dropped to a tenth of the maximum missing connections. However, they are still above the normal level. the trend is now clearly and continuously positive. The INVERS teams will continue to monitor the issue and keep you up to date. We are downgrading the impact classification by one level.
-
Time: Aug. 15, 2024, 9:19 p.m.
Status: Identified
Update: The error rates are slowly improving, but because there are currently hundreds of thousands of devices sending data, the incoming data traffic has to be controlled, so it doesn't happen all at once.
-
Time: Aug. 15, 2024, 8:29 p.m.
Status: Identified
Update: We are seeing further improvements in error rates, but we are not out of the woods yet.
-
Time: Aug. 15, 2024, 7:29 p.m.
Status: Identified
Update: The restoration of the server system is unfortunately delayed, so we do not yet see any improvement. We will stay tuned.
-
Time: Aug. 15, 2024, 7:10 p.m.
Status: Identified
Update: The planned maintenance window is over, we will check with our partners whether the emergency maintenance was successful and post further updates.
-
Time: Aug. 15, 2024, 6:49 p.m.
Status: Identified
Update: The Partner Carrier is still engaged in their emergency maintenance operation. Updates will provided once they are available.
-
Time: Aug. 15, 2024, 5:51 p.m.
Status: Identified
Update: Emergency maintenance has been started and should be completed by around 19:00 UTC. All teams on our side that can ensure a proper restart are ready. We have also activated additional resources to be able to process the traffic as quickly as possible.
-
Time: Aug. 15, 2024, 5:12 p.m.
Status: Investigating
Update: This is a Europe-wide outage of one of the main roaming platforms used to connect users in other countries to their home operator. It is currently also affecting a massive number of users of internet services abroad, beyond car sharing or M2M services. That is why all available resources are working on the topic.
We have been informed that the partner is now planning a major emergency maintenance which should bring operations back to normal. We are seeing a slight easing, but are waiting for our monitoring teams to confirm the improvement.
-
Time: Aug. 15, 2024, 3:32 p.m.
Status: Investigating
Update: We are still in close contact with our partners, but there is still no reliable information on when the problem will be finally resolved. We are still active with all teams and will remain so.
-
Time: Aug. 15, 2024, 2:35 p.m.
Status: Investigating
Update: The work on the specific network component is still ongoing, unfortunately there is no estimate of how long it will take. All teams will continue to work on the issue until it is finally resolved. We sincerely apologize for any inconvenience caused. CloudBoxx 1020 units are still not affected.
-
Time: Aug. 15, 2024, 1:43 p.m.
Status: Investigating
Update: C-level has been involved for some time, all the necessary teams are working on the issue, but there is still no ETA. We will continue to post updates here.
-
Time: Aug. 15, 2024, 12:47 p.m.
Status: Investigating
Update: The partner continues to work on the specific solution, which has not yet been fully implemented.
-
Time: Aug. 15, 2024, 11:58 a.m.
Status: Investigating
Update: The partner of the connectivity provider found the specific issue in the network, just confirmed, and they are working on a specific fix.
-
Time: Aug. 15, 2024, 11:48 a.m.
Status: Investigating
Update: We are continuing to investigate this issue.
-
Time: Aug. 15, 2024, 11:20 a.m.
Status: Investigating
Update: Other teams have been brought in, there are no visible changes yet, we are working hard on the issue
-
Time: Aug. 15, 2024, 10:50 a.m.
Status: Investigating
Update: The increase in faulty connections has leveled off, but the number of errors is not yet decreasing. The teams are continuing to work on the cause and we will keep you up to date.
-
Time: Aug. 15, 2024, 10:27 a.m.
Status: Investigating
Update: There is an outage at a partner of our connectivity service provider. This has been confirmed by the partner. The emergency teams are working on a quick solution, INVERS teams are monitoring it from our side. we will post regular updates here.
-
Time: Aug. 15, 2024, 10:14 a.m.
Status: Investigating
Update: We see an increase in missing heartbeats, we are in contact with the Connectivity Partner to solve it as soon as possible
-
Time: Aug. 15, 2024, 8:25 a.m.
Status: Investigating
Update: Some CloudBoxx units have a slightly longer response time, we are working with our partners on this topic and will let you know if there is anything new.
