Global Maintenance – Limited Impact

Dec 15, 18:09 UTC
Completed – The scheduled maintenance has been completed.

Sep 18, 16:31 UTC
Update – Changes are now largely complete around the world with some limited changes still to be made. Customers in those specific sites will be contacted directly with more specific updates.

All up this project ran two weeks longer than expected.

We have adjusted more than 600,000 lines of config in aggregate across all routers worldwide.
We have removed an additional 170,000 lines of config considered technical debt caused by the last 5 years of growth.
We have added more than 350 100G ports to the network to support internal infrastructure and various services.
We have added 84 external 100G ports to the network over the last two months.

Most things should be returning to normal, albeit there may be some routing regressions or suspect paths may be active.

If routing looks sub-optimal, please either reach out to noc@globalsecurelayer.com or via your related support avenues if you’re not a direct customer. We are actively reviewing global routing at this stage to make widespread improvements.

Sep 2, 01:27 UTC
Update – Work is still progressing.

North America has been complete and stable since the 23rd.

At this stage it looks like we are going to run right up to our previous estimation/deadline and run 1-2 days over for global completion, pending any significant issues that arise.

We have begun the migration of cities in Australia today, we do expect there to be some scenic routing while some other dependent sites are migrated through the week.

We apologize in advance for the headaches this causes, we hope that the scenic routing is short lived through this week.

Please still reach out with traces to your contacts or to noc@globalsecurelayer.com if something is incorrect, it may be something we can repair immediately, or if not we can advise with more specifics on when it will be restored.

Aug 11, 09:18 UTC
Update – At approximately 9:00AM UTC some automation systems were updated to account for changes within the new network.

This caused adverse affects to a number of locations, most notably Sydney, Frankfurt & Tokyo.

A large amount of traffic was re-routed for approximately 6 minutes with some others black holed for roughly 1 minute.

Apologies for this inconvenience, This event should not re-occur.

Aug 10, 02:40 UTC
Update – During turnup of Pittsburgh some traffic entering the network over an IXP was incorrectly backhauled to a remote DDoS scrubber for a period of time today.

This has been rectified.

Aug 7, 03:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 7, 02:24 UTC
Scheduled – Over the past six months, we have diligently worked on upgrading our global network to accommodate new products, additional features, enhanced customization for our technical customers, and to significantly improve the quality of life and reliability in regions prone to frequent cascading cable cuts and infrastructure failures.

With our preparations now complete, we are ready to implement the more noticeable changes throughout the month of August, with the project expected to conclude in the first week of September.

During this period, users may observe the following temporary changes:
– Occasional incorrect routing or re-routing to nearby locations.
– Reduction in peak DDoS mitigation capacity on certain days.
– Bypassing of some locations for network traffic.
– Slightly higher than expected latency between specific regions.

We want to reassure you that, apart from the latency caused by the use of different paths, there should be no significant underlying throughput or performance issues during this maintenance period.

If you notice an incorrect route persisting for more than 24 hours, we kindly request you to inform us by sending an email to noc@globalsecurelayer.com, regardless of whether you are a direct customer or not.

Your feedback is essential for maintaining the integrity and performance of our network. If you have any concerns, feel free to reach out to us.

Leave a Reply

Your email address will not be published. Required fields are marked *