Internap Datacenter Down

Resolved

Internap is indicating this issue is fully resolved, and all Lightning Base systems have been running normally. We are going to close this incident, if you are seeing any problems with a site please open a ticket at https://secure.lightningbase.com .

Monitoring

Our internal monitoring is now reporting all systems as up. We are currently checking services manually, servers appear to have continued running during the entire outage as this was only a loss of network connectivity (as opposed to a power outage or other physical impact to the datacenter).

We will complete internal checks and will begin emailing clients that had opened tickets. This incident will remain open in a Monitoring state at least until Internap has pushed out a status update indicating things are resolved, as it is possible there will be higher latency or network issues as the repairs are finalized.

Problem Identified

We have received an update from Internap. They are indicating there is a complex multi-path fiber failure in the Chicago area, and crews have been dispatched to multiple locations to resolve the issue.

We will be continuing to monitor services here, both our systems directly impacted by the outage and sites impacted by the associated downtime at wordpress.org and api.wordpress.org.

If you have any questions, please open a ticket in our https://secure.lightningbase.com client area.

Monitoring

The outage at Internap is continuing to affect Lightning Base servers that are hosted there.

This is causing outages for a variety of other sites/services as well, as it is a large datacenter. This appears to include wordpress.org, and api.wordpress.org, which show Internap IP addresses. Issues with api.wordpress.org are impacting performance of WordPress administration areas and potentially other parts of sites.

We are continuing to monitor the status here and will ensure all services are running normally as Internap comes back up.

Assessed

One of our datacenter partners, https://internap.com is experiencing an outage at their Chicago location that hosts our older US-based systems. They are aware of the problem and are working on a solution, we will update this page as we learn more.

6 Affected Services: