Network dip

One of our upstream transits suffered an outage at 19.05. This caused network re-routing to happen and may have been felt as momentary network dip.
Traffic has been re-routed to failovers. We’re monitoring the networks.

When: 19:05 – 19:10 (upstream has failed over)
Impact: 5min re-routing, traffic dip
What: 46.253.192.0/20

Update: Some more network dips occured during the evening. The network upstream suspected of causing high CPU usage in our cores have been isolated. We’re continuing to monitor the situation.

When: 20.30, 23.30 , duration (roughly 2-3min per incident)

Driftstörning / Outage

Start: 09:00
End: 09:15

Fault: Optical fiber outage causing network disruption of internet traffic.
Status: Partially remedied (failover path in use).
Work is being done to find the fault and remedy it.

Datastorage is not affected by the outage (multipathing).

Update:
At 10:40 backup and primary paths are now recovered. Full redundancy has been restored.

Further investigation into the failed optical fiber will be made.

Work is being undertaken to improve and speed up the fail-over of fiber redundancy for inter-DC connectivity.

Details:
At 09.00 a switch failure occurred in core infrastructure which burnt optical fiber transceivers.

Remediation: By 09.15 the fault had been found and all network paths had been migrated to secondary fiber connections.

Planned steps:
The faulty switch has been replaced and redundancy restored. We’re planning to perform maintenance work (future scheduled window) to implement new protocols for automated fail-over of fiber paths.