Time: March 28, 2018, 11:34 a.m.

Status: Resolved

Update: We rolled back our workaround settings. All services are up and running. The incident has been resolved. Our customers will receive an incident report via e-mail.

Time: March 28, 2018, 7:47 a.m.

Status: Monitoring

Update: Last night our telcos restored the destroyed lines. All services are recovered, within the next hours we deprovision our workarounds which will happen without any interuptions. During the day our customers will receive an incident report via e-mail.

Time: March 27, 2018, 3:57 p.m.

Status: Monitoring

Update: Our telcos still work on restoring the destroyed lines. Repairing seems to be more difficult than expected, previously announced estimated restoral times (late afternoon) have been postponed. The telcos estimate service restoration in the morning hours. Our on call duty continues monitoring the situation and will take action if needed.

Time: March 27, 2018, 1:37 p.m.

Status: Monitoring

Update: The telcos still work hard to re-establish all lines. We'll keep you up to date.

Time: March 27, 2018, 10:11 a.m.

Status: Monitoring

Update: Our telcos reported that the preparation work on the fiber lines have been finished, also splicing on one end is done. Splicing on the second side just started.

Time: March 27, 2018, 8:17 a.m.

Status: Monitoring

Update: The telcos are stil working on bringing up the destroyed lines again. There were delays because local authorities barred access at night. We expect updates from our telcos before noon.

Time: March 26, 2018, 2:59 p.m.

Status: Monitoring

Update: We monitor the situation and expect the fire damage to be repaired this night. We'll enable our default connection as soon as the lines are up and stable - assumably tomorrow in the morning.

Time: March 26, 2018, 2:15 p.m.

Status: Monitoring

Update: We implemented all possible workarounds to mitigate the outage. We expect the broken connections to be repaired at night.

Time: March 26, 2018, 1:55 p.m.

Status: Identified

Update: IPv6 is active again for all our sites.

Time: March 26, 2018, 1:06 p.m.

Status: Identified

Update: Cause of the outage is a fire at a bridge in Berlin which is used by many telcos for ducting. Connections (IPv4) Berlin-Frankfurt and Berlin-Amsterdam are up using a workaround. MPLS services still are affected, we are working on it.

Time: March 26, 2018, 12:59 p.m.

Status: Identified

Update: We implemented a workaround to reestablish the connection between Berlin and Amsterdam.

Time: March 26, 2018, 12:44 p.m.

Status: Identified

Update: We implemented a workaround to reestablish the connection between Berlin and Frankfurt.

Time: March 26, 2018, 12:23 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: March 26, 2018, 12:21 p.m.

Status: Investigating

Update: A bigger outage in a duct close to our Berlin datacenter caused an interuption of several transits and connections. The duct is used by several telco providers. Currently the redundant connections Berlin <-> Frankfurt and Berlin <-> Amsterdam are down. Additionaly two transits in Berlin are currently down. We are closely working with the affected telcos and are about to reconnect Amsterdam and Frankfurt in alternative ways.

Time: March 26, 2018, 11:51 a.m.

Status: Investigating

Update: We currently have a network outage in Berlin, probably caused by a fiber interruption. Several lines are affected. We work hard on the incident and keep you updated.

Time: March 28, 2018, 11:34 a.m.

Status: Resolved

Update: We rolled back our workaround settings. All services are up and running. The incident has been resolved. Our customers will receive an incident report via e-mail.

Time: March 28, 2018, 7:47 a.m.

Status: Monitoring

Update: Last night our telcos restored the destroyed lines. All services are recovered, within the next hours we deprovision our workarounds which will happen without any interuptions. During the day our customers will receive an incident report via e-mail.

Time: March 27, 2018, 3:57 p.m.

Status: Monitoring

Update: Our telcos still work on restoring the destroyed lines. Repairing seems to be more difficult than expected, previously announced estimated restoral times (late afternoon) have been postponed. The telcos estimate service restoration in the morning hours. Our on call duty continues monitoring the situation and will take action if needed.

Time: March 27, 2018, 1:37 p.m.

Status: Monitoring

Update: The telcos still work hard to re-establish all lines. We'll keep you up to date.

Time: March 27, 2018, 10:11 a.m.

Status: Monitoring

Update: Our telcos reported that the preparation work on the fiber lines have been finished, also splicing on one end is done. Splicing on the second side just started.

Time: March 27, 2018, 8:17 a.m.

Status: Monitoring

Update: The telcos are stil working on bringing up the destroyed lines again. There were delays because local authorities barred access at night. We expect updates from our telcos before noon.

Time: March 26, 2018, 2:59 p.m.

Status: Monitoring

Update: We monitor the situation and expect the fire damage to be repaired this night. We'll enable our default connection as soon as the lines are up and stable - assumably tomorrow in the morning.

Time: March 26, 2018, 2:15 p.m.

Status: Monitoring

Update: We implemented all possible workarounds to mitigate the outage. We expect the broken connections to be repaired at night.

Time: March 26, 2018, 1:55 p.m.

Status: Identified

Update: IPv6 is active again for all our sites.

Time: March 26, 2018, 1:06 p.m.

Status: Identified

Update: Cause of the outage is a fire at a bridge in Berlin which is used by many telcos for ducting. Connections (IPv4) Berlin-Frankfurt and Berlin-Amsterdam are up using a workaround. MPLS services still are affected, we are working on it.

Time: March 26, 2018, 12:59 p.m.

Status: Identified

Update: We implemented a workaround to reestablish the connection between Berlin and Amsterdam.

Time: March 26, 2018, 12:44 p.m.

Status: Identified

Update: We implemented a workaround to reestablish the connection between Berlin and Frankfurt.

Time: March 26, 2018, 12:23 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: March 26, 2018, 12:21 p.m.

Status: Investigating

Update: A bigger outage in a duct close to our Berlin datacenter caused an interuption of several transits and connections. The duct is used by several telco providers. Currently the redundant connections Berlin <-> Frankfurt and Berlin <-> Amsterdam are down. Additionaly two transits in Berlin are currently down. We are closely working with the affected telcos and are about to reconnect Amsterdam and Frankfurt in alternative ways.

Time: March 26, 2018, 11:51 a.m.

Status: Investigating

Update: We currently have a network outage in Berlin, probably caused by a fiber interruption. Several lines are affected. We work hard on the incident and keep you updated.

Time: Feb. 2, 2018, 2:04 p.m.

Status: Resolved

Update: English version below Update zu Spectre/Meltdown Sehr geehrte Damen und Herren, im Zusammenhang mit den als Spectre und Meltdown bekannten Sicherheitslücken können wir Ihnen folgendes Update geben: Wir sehen die Bedrohung für dedizierte Umgebungen als nicht kritisch an. Unsere Customer Teams werden in Abstimmung mit den Kunden im Rahmen der Patchzyklen entsprechende Updates einspielen. Stärker betroffen von Spectre und Meltdown sind Umgebungen, die von mehreren Kunden genutzt werden, wie die *umCloud. Wir werden hier umfangreiche Maßnahmen ergreifen. Dazu gehören Änderungen an der Infrastruktur, Aktualisierungen von Cloud-Software, Hypervisoren und den virtuellen Maschinen. In diesem Zusammenhang werden wir direkt auf unsere Kunden zugehen. Unsere öffentlichen Informationen via E-Mail und https://status.unbelievable-machine.com/ zu Spectre und Meltdown stellen wir mit diesem Update ein. Bei Fragen wenden Sie sich bitte an Ihr Customer Team. ---- Update on Spectre/Meltdown Dear customers, please read our updates regarding the security bugs known as Spectre and Meltdown: We classify the threat for dedicated environments as non-critical. Our customer teams will contact our customers regarding updates during our regular patch cycles. Shared infrastructure which is used by multiple customers (as *umCloud) is stronger affected by Spectre and Meltdown. We plan extensive activities, among them infrastructural changes, updates of cloud software, hypervisors and virtual machines. We will contact our *umCloud customers directly regarding these activities. Please note that we stop releasing public information about Spectre and Meltdown via email and https://status.unbelievable-machine.com/. For questions please contact your customer team.

Time: Jan. 24, 2018, 3:37 p.m.

Status: Investigating

Update: English version below Update zu Spectre/Meltdown Sehr geehrte Damen und Herren, derzeit führen wir in Zusammenarbeit mit unseren Partnern und Herstellern Aktualisierungen auf unseren Systemen durch, um die als Spectre bzw. Meltdown Bug bekannte Sicherheitslücke zu beheben. Allerdings stufen sowohl unsere Sicherheitsexperten als auch die Experten bekannter Sicherheitsfirmen oder regierungsnaher Institute die Bedrohung durch die Sicherheitslücke als deutlich geringer ein als die Medienresonanz vermuten lässt. Beispielsweise wird die Lücke vom Information Technology Laboratory (ITL) mit einem mittleren Wert von 5.6 eingeschätzt, wobei der Maximalwert 10 wäre. https://nvd.nist.gov/vuln/detail/CVE-2017-5753 https://nvd.nist.gov/vuln/detail/CVE-2017-5715 https://nvd.nist.gov/vuln/detail/CVE-2017-5754 Nichtsdestotrotz nehmen wir auch diese Sicherheitsbedrohung sehr ernst und behandeln sie im Rahmen unserer nach ISO27001 zertifizierten Sicherheits- und Patchmanagementprozesse. Wir konzentrieren uns dabei zunächst auf Systemumgebungen, in welchen mehrere Kunden nebeneinander betrieben werden, wie z. B. zentrale Netzwerkkomponenten oder die *umCloud. Wir gehen so vor, dass wir die Stabilität und Performance der Systeme nicht beeinträchtigen. Sobald neue Informationen vorliegen, werden wir Sie über das weitere Vorgehen in Kenntnis setzen. --- Update on Spectre/Meltdown Dear customers, we are currently updating our systems to address the security bug known under the name Spectre and Meltdown, in collaboration with our partners and hardware providers. However, our own security experts as well as renowned security companies and government-related institutions classify this bug as a lower risk than the media coverage may suggest. Information Technology Laboratory (ITL) for example assesses the risk with a medium value of 5.6; 10 being the highest risk. https://nvd.nist.gov/vuln/detail/CVE-2017-5753 https://nvd.nist.gov/vuln/detail/CVE-2017-5715 https://nvd.nist.gov/vuln/detail/CVE-2017-5754 Nonetheless we take this security issue very seriously and treat it according to our ISO 27001 certified security and patch management processes. We are focussing first on system environments in which multiple customers are run together, such as central networking components or *umCloud. We do it in such way that the stability and performance of the systems do not get affected. As soon as we have new information on the issue we will update you on how we proceed.

Time: Jan. 9, 2018, 3:52 p.m.

Status: Investigating

Update: -english version below- wir möchten Ihnen ein kurzes Update zum Stand von Meltdown und Spectre geben: Wir analysieren weiterhin die Systeme und warten noch auf Veröffentlichung der relevanten Patches und Aktualisierung verschiedener Hersteller. Die ersten internen Serversysteme wurden von uns mittlerweile gepatcht, wir gehen hierbei nach einem risikobasierten Ansatz vor. Sobald neue Informationen oder Patches der Hersteller vorliegen, werden wir Sie über das weitere Vorgehen in Kenntnis setzen. --- Dear customer, As a short update on the status of the vulnerabilities Meltdown and Spectre: We are still investigating our systems and are waiting for patches and updates being released. Meanwhile we updated the first internal servers according to a risk based process. As soon as new information or patches are available we will keep you updated.

Time: Jan. 5, 2018, 3:13 p.m.

Status: Investigating

Update: -english version below- Sehr geehrte Damen und Herren, es wurde ein kritischer Bug in den Prozessoren von INTEL, AMD und ARM veröffentlicht. Dieser Bug ermöglicht es möglicherweise Daten unerlaubt von Servern auszulesen. Details dazu finden sich unter folgenden Links: https://www.heise.de/newsticker/meldung/Prozessor-Luecken-Meltdown-und-Spectre-Intel-und-ARM-fuehren-betroffene-Prozessoren-auf-Nvidia-3934667.html https://newsroom.intel.de/news-releases/intel-issues-updates-protect-systems-security-exploits/ https://newsroom.intel.de/news-releases/intel-responds-security-research-findings/ https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html Aktuell prüfen unsere Sicherheitsexperten alle vorliegenden Informationen und bewerten die Schwachstellen hinsichtlich ihrer Kritikalität. Weiterhin bereiten wir Maßnahmen zur Behebung der Schwachstellen vor. Sobald neue Informationen oder Patches der Hersteller vorliegen, werden wir Sie über das weitere Vorgehen in Kenntnis setzen. --- Dear customer, as you might have read in the news there is a critical bug for INTEL, AMD and ARM processors. Under certain circumstances it allows unauthorized access to stored data in the processors memory. Find more details at: https://newsroom.intel.de/news-releases/intel-issues-updates-protect-systems-security-exploits/ https://newsroom.intel.de/news-releases/intel-responds-security-research-findings/ https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html Our security experts are currently analyzing all available informations and evaluate the vulnerabilities in terms of criticality. Further on we are working on an action plan to clear out the vulnerabilities. As soon as new informations or patches are available we will keep you updated.

Time: Feb. 2, 2018, 2:04 p.m.

Status: Resolved

Update: English version below Update zu Spectre/Meltdown Sehr geehrte Damen und Herren, im Zusammenhang mit den als Spectre und Meltdown bekannten Sicherheitslücken können wir Ihnen folgendes Update geben: Wir sehen die Bedrohung für dedizierte Umgebungen als nicht kritisch an. Unsere Customer Teams werden in Abstimmung mit den Kunden im Rahmen der Patchzyklen entsprechende Updates einspielen. Stärker betroffen von Spectre und Meltdown sind Umgebungen, die von mehreren Kunden genutzt werden, wie die *umCloud. Wir werden hier umfangreiche Maßnahmen ergreifen. Dazu gehören Änderungen an der Infrastruktur, Aktualisierungen von Cloud-Software, Hypervisoren und den virtuellen Maschinen. In diesem Zusammenhang werden wir direkt auf unsere Kunden zugehen. Unsere öffentlichen Informationen via E-Mail und https://status.unbelievable-machine.com/ zu Spectre und Meltdown stellen wir mit diesem Update ein. Bei Fragen wenden Sie sich bitte an Ihr Customer Team. ---- Update on Spectre/Meltdown Dear customers, please read our updates regarding the security bugs known as Spectre and Meltdown: We classify the threat for dedicated environments as non-critical. Our customer teams will contact our customers regarding updates during our regular patch cycles. Shared infrastructure which is used by multiple customers (as *umCloud) is stronger affected by Spectre and Meltdown. We plan extensive activities, among them infrastructural changes, updates of cloud software, hypervisors and virtual machines. We will contact our *umCloud customers directly regarding these activities. Please note that we stop releasing public information about Spectre and Meltdown via email and https://status.unbelievable-machine.com/. For questions please contact your customer team.

Time: Jan. 24, 2018, 3:37 p.m.

Status: Investigating

Update: English version below Update zu Spectre/Meltdown Sehr geehrte Damen und Herren, derzeit führen wir in Zusammenarbeit mit unseren Partnern und Herstellern Aktualisierungen auf unseren Systemen durch, um die als Spectre bzw. Meltdown Bug bekannte Sicherheitslücke zu beheben. Allerdings stufen sowohl unsere Sicherheitsexperten als auch die Experten bekannter Sicherheitsfirmen oder regierungsnaher Institute die Bedrohung durch die Sicherheitslücke als deutlich geringer ein als die Medienresonanz vermuten lässt. Beispielsweise wird die Lücke vom Information Technology Laboratory (ITL) mit einem mittleren Wert von 5.6 eingeschätzt, wobei der Maximalwert 10 wäre. https://nvd.nist.gov/vuln/detail/CVE-2017-5753 https://nvd.nist.gov/vuln/detail/CVE-2017-5715 https://nvd.nist.gov/vuln/detail/CVE-2017-5754 Nichtsdestotrotz nehmen wir auch diese Sicherheitsbedrohung sehr ernst und behandeln sie im Rahmen unserer nach ISO27001 zertifizierten Sicherheits- und Patchmanagementprozesse. Wir konzentrieren uns dabei zunächst auf Systemumgebungen, in welchen mehrere Kunden nebeneinander betrieben werden, wie z. B. zentrale Netzwerkkomponenten oder die *umCloud. Wir gehen so vor, dass wir die Stabilität und Performance der Systeme nicht beeinträchtigen. Sobald neue Informationen vorliegen, werden wir Sie über das weitere Vorgehen in Kenntnis setzen. --- Update on Spectre/Meltdown Dear customers, we are currently updating our systems to address the security bug known under the name Spectre and Meltdown, in collaboration with our partners and hardware providers. However, our own security experts as well as renowned security companies and government-related institutions classify this bug as a lower risk than the media coverage may suggest. Information Technology Laboratory (ITL) for example assesses the risk with a medium value of 5.6; 10 being the highest risk. https://nvd.nist.gov/vuln/detail/CVE-2017-5753 https://nvd.nist.gov/vuln/detail/CVE-2017-5715 https://nvd.nist.gov/vuln/detail/CVE-2017-5754 Nonetheless we take this security issue very seriously and treat it according to our ISO 27001 certified security and patch management processes. We are focussing first on system environments in which multiple customers are run together, such as central networking components or *umCloud. We do it in such way that the stability and performance of the systems do not get affected. As soon as we have new information on the issue we will update you on how we proceed.

Time: Jan. 9, 2018, 3:52 p.m.

Status: Investigating

Update: -english version below- wir möchten Ihnen ein kurzes Update zum Stand von Meltdown und Spectre geben: Wir analysieren weiterhin die Systeme und warten noch auf Veröffentlichung der relevanten Patches und Aktualisierung verschiedener Hersteller. Die ersten internen Serversysteme wurden von uns mittlerweile gepatcht, wir gehen hierbei nach einem risikobasierten Ansatz vor. Sobald neue Informationen oder Patches der Hersteller vorliegen, werden wir Sie über das weitere Vorgehen in Kenntnis setzen. --- Dear customer, As a short update on the status of the vulnerabilities Meltdown and Spectre: We are still investigating our systems and are waiting for patches and updates being released. Meanwhile we updated the first internal servers according to a risk based process. As soon as new information or patches are available we will keep you updated.

Time: Jan. 5, 2018, 3:13 p.m.

Status: Investigating

Update: -english version below- Sehr geehrte Damen und Herren, es wurde ein kritischer Bug in den Prozessoren von INTEL, AMD und ARM veröffentlicht. Dieser Bug ermöglicht es möglicherweise Daten unerlaubt von Servern auszulesen. Details dazu finden sich unter folgenden Links: https://www.heise.de/newsticker/meldung/Prozessor-Luecken-Meltdown-und-Spectre-Intel-und-ARM-fuehren-betroffene-Prozessoren-auf-Nvidia-3934667.html https://newsroom.intel.de/news-releases/intel-issues-updates-protect-systems-security-exploits/ https://newsroom.intel.de/news-releases/intel-responds-security-research-findings/ https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html Aktuell prüfen unsere Sicherheitsexperten alle vorliegenden Informationen und bewerten die Schwachstellen hinsichtlich ihrer Kritikalität. Weiterhin bereiten wir Maßnahmen zur Behebung der Schwachstellen vor. Sobald neue Informationen oder Patches der Hersteller vorliegen, werden wir Sie über das weitere Vorgehen in Kenntnis setzen. --- Dear customer, as you might have read in the news there is a critical bug for INTEL, AMD and ARM processors. Under certain circumstances it allows unauthorized access to stored data in the processors memory. Find more details at: https://newsroom.intel.de/news-releases/intel-issues-updates-protect-systems-security-exploits/ https://newsroom.intel.de/news-releases/intel-responds-security-research-findings/ https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html Our security experts are currently analyzing all available informations and evaluate the vulnerabilities in terms of criticality. Further on we are working on an action plan to clear out the vulnerabilities. As soon as new informations or patches are available we will keep you updated.

Time: Dec. 20, 2017, 1:23 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 20, 2017, 1:23 p.m.

Status: Identified

Update: Dear *umCloud customers, Our cloud service is up and running. Incident solve. We are still reviewing the services of the affected VMs.

Time: Dec. 20, 2017, 1:02 p.m.

Status: Identified

Update: We are still recovering some degraded VMs. New update will be posted asap.

Time: Dec. 20, 2017, 11:48 a.m.

Status: Identified

Update: We identified the root cause. Now we start to continue the firmware upgrade. After finishing we will recover all affected services.

Time: Dec. 20, 2017, 11:24 a.m.

Status: Identified

Update: We are working at full speed to solve this issue and will keep you posted. The following services are affected. Disks on the datastores: - SAN1-S2 - SAN1-S4

Time: Dec. 20, 2017, 11:15 a.m.

Status: Investigating

Update: We're currently experiencing a degradation of our *umCloud service. We're analyzing the incident and are still busy working on recovering all services.