RESOLVED: Nagios Monitoring Outage

EVENT START DATE: 01/20/2015
EVENT START TIME:
12:54 PM CST

EVENT END DATE: 01/20/2015
EVENT END TIME:
01:08 PM CST

EVENT LOCATION: MCI01

EVENT DETAILS:
Our Nagios monitoring services were momentarily interrupted during the time frame above. All monitoring services have since been restored.

MCI01 SCHEDULED NETWORK MAINTENANCE 25-JAN-2015 03:00 CST

EVENT START DATE: 01/25/2015
EVENT START TIME: 03:00 CST

EVENT END DATE: 01/25/2015
EVENT END TIME: 06:00 CST

EVENT LOCATION: MCI01

EVENT DETAILS:
Datapipe Infrastructure Engineering has been informed of vendor maintenance on one of the redundant transit links servicing the facility.  Customers may experience brief periods of sub-optimal routing and/or increased latency as a result of this work.

Scheduled Maintenance

EVENT START DATE: 1/10/2015
EVENT START TIME: 00:01 AM CST

EVENT END DATE: 1/10/2015
EVENT END TIME: 06:00 AM CST

EVENT LOCATION:  ATC

EVENT DETAILS:
Datapipe will be performing scheduled maintenance on our shared virtual and cloud environments in our ATC Datacenter.  No customer impact is expected to result from this maintenance.

MCI01 SCHEDULED NETWORK MAINTENANCE 12-FEB-2015 03:00 CST

EVENT START DATE: 02/12/2015
EVENT START TIME: 03:00 CST

EVENT END DATE: 02/12/2015
EVENT END TIME: 06:00 CST

EVENT LOCATION: MCI01

EVENT DETAILS:
Datapipe Infrastructure Engineering has been informed of vendor maintenance on one of the redundant transit links servicing the facility.  No customer impact is expected as traffic is routed through other providers.

ORD01 SCHEDULED NETWORK MAINTENANCE — 08-JAN-2015 00:00 CST

EVENT START DATE: 01/08/2015
EVENT START TIME: 00:00 CST

EVENT END DATE: 01/08/2015
EVENT END TIME: 06:00 CST

EVENT LOCATION: ORD01

EVENT DETAILS:
Datapipe Infrastructure Engineering has been informed of vendor maintenance on one of the redundant transit links servicing the facility.  No customer impact is expected as traffic is routed through other providers.

ORD01 SCHEDULED NETWORK MAINTENANCE — 07-JAN-2015 00:00 CST

EVENT START DATE: 01/07/2015
EVENT START TIME: 00:00 CST

EVENT END DATE: 01/07/2015
EVENT END TIME: 06:00 CST

EVENT LOCATION: ORD01

EVENT DETAILS:
Datapipe Infrastructure Engineering will be performing power maintenance on the redundant power supplies of one of our core routers.  No customer impact is expected to result from this maintenance.

Unscheduled Virtual Machine Maintenance

EVENT START DATE: 01/03/2015
EVENT START TIME: 00:00 CST

EVENT END DATE: 01/03/2015
EVENT END TIME: 18:00 CST

EVENT LOCATION:  Midwest

EVENT DETAILS:
Datapipe will be performing emergency maintenance on our shared virtual and cloud environments in our midwest region.  Customer virtual machines may be rebooted as part of this maintenance. Downtime for individual virtual machines should be brief.

Critical Event – Cedar Falls Network Connectivity Issue

EVENT START DATE: 12/31/2014
EVENT START TIME: 6:10 AM CT

EVENT END DATE: 12/31/2014
EVENT END TIME: 4:51 PM CT

EVENT LOCATION: Cedar Falls


4:51 PM CT – Our Service Provider has notified us that the issue has been completely resolved. Testing performed on our end reflects the same. Please feel free to contact our support team if any performance or connectivity issues still exist. – ISSUE IS NOW RESOLVED

2:31 PM CT – Initial findings of service restoration have been noticed. Environments may start to receive connectivity, though there may be intermittent drops while the overall work is being completed for repairing the severed fiber.

2:15 PM CT – Our Service Provider has informed us that there was a fiber cut that caused the connectivity issues into the Cedar Falls datacenter. They are currently in process of repairing the fiber connection and hope to have the connectivity repaired by 5PM CT. This outage was found to have impacted a portion of our support ticket system, requiring any requests for support to be sent through the LayeredTech Portal or by calling directly into our support team ((816)421-1420). We’ll continue to give updates as they become available.

12:14 PM CT – Our Service Provider has informed us that it located the issue with connectivity and have a team working on restoring service. An ETA is still unavailable though we’ll continue to provide updates as they become available.

10:50 AM CT – Our Service Provider is continuing their troubleshooting to reestablish connectivity to the region. An ETA for resolution has not been given, though we have been assured the issue is being worked on high priority. We’ll continue to give updates as they become available.

08:50 AM CT – Our Service Provider confirmed an issue within Iowa that is causing impact to the Cedar Falls location.  They are working to resolve the issue. Updates will continue to be posted as they become available.

08:20 AM CT – Currently, we are working with our Service Provider for the Cedar Falls location as all troubleshooting so far points to an upstream provider issue. Further details will be provided as soon as they are available.

06:43 AM CT – We are currently experiencing a network connectivity issue at the Cedar Falls Datacenter location impacting a portion of clients hosted at the facility. We currently have our engineers troubleshooting to find root cause as quickly as possible. Updates will be posted as soon as they are available.

SCHEDULED NETWORK MAINTENANCE — 10-DEC-2014 00:00 CST

EVENT START DATE: 12/10/2014
EVENT START TIME: 00:00 CST

EVENT END DATE: 12/10/2014
EVENT END TIME: 06:00 CST

EVENT LOCATION:  Midwest

EVENT DETAILS:
Datapipe Infrastructure Engineering will be performing maintenance on backbone routers as part of a network improvement process for our midwest environments.  No customer impact is expected to result from this maintenance.

INTERNAL ERP TICKETING SYSTEM MAINTENANCE — 06-DEC-2014 10:00 CST

EVENT START DATE: 12/06/2014
EVENT START TIME: 10:00 AM CST

EVENT END DATE: 12/06/2014
EVENT END TIME: 11:00 AM CST

EVENT LOCATION: Ticketing System (AnswerTrack) and All Client Portal’s (Cloud, Pentagon, LT Portal)

EVENT DETAILS: The Layered Tech Infrastructure Team will be upgrading our ERP/Ticketing System. During this scheduled maintenance our ticket system (AnswerTrack) will be offline and we will be unable to generate new tickets or respond to existing tickets. If you experience an outage (P1 Incident) during this maintenance window, please contact our support number (888-952-4888, option 2) for immediate support. We will be monitoring our Nagios system and will call the contacts listed for your company to notify you of any P1 Incidents that may occur. If you create a ticket through your Portal or send in an email to support@layeredtech.com your email will be queued in the system and after the maintenance has ended tickets will be updated or created. Those emails/tickets will not be responded to until after the scheduled maintenance has been completed. Client environments will NOT be impacted during this maintenance.