(Resolved) Scheduled Maintenance: ALO01 (Cedar Falls) 07-25-2010 00:00 – 07-25-2010 06:00 CST

EVENT START DATE: 07-25-2010

EVENT START TIME: 00:00 CST

EVENT END DATE: 07-25-2010

EVENT END TIME: 06:00 CST

EVENT IMPACT: ALO01 (Cedar Falls)

EVENT DETAILS:

On Sunday 7/25/2010 at 00:00 CST, LT Network Engineering will be performing scheduled maintenance in the Cedar Falls data center to migrate aggregate customer connectivity to a new network device.  Customers in the Cedar Falls data center will experience brief periods of outage as connectivity is moved to the new device in a phased and controlled migration.  Recovery for customers affected during this migration should be rapid and complete with no residual affect following resumption of normal connectivity.

2010
07/15
POSTED BY
mfritsche
CATEGORY
Network
announcements
data center
TAGS

Comments Off

(Resolved) Unplanned Event: ORD02 UPS System

EVENT START DATE: 07-12-2010
EVENT START TIME: 20:16 CST

EVENT END DATE:  07-13-2010
EVENT END TIME: 05:53 CST

EVENT IMPACT: UPS Unavailable

EVENT DETAILS:

During this time, the UPS system for our Chicago facility is on Maintenance Bypass.  IBX Site staff will be onsite to monitor the UPS and assist customers with any issues. In the unlikely event of a commercial power outage customers will see a momentary impact of services and IBX site staff will assist customers with any needs. There is no impact to customer services at this time and none is expected. We are doing everything to expedite the repairs of the UPS and will provide another update tomorrow morning at 9am.

UPDATE 23:40 07/12/2010: Our UPS Vendor has arrived on site and has begun repairs to UPS 1 and UPS 2. Further updates will be posted as the repairs progress.

UPDATE 05:53 07/13/2010:  Our Vendor has completed repairs and testing of UPS units 1 and 2. The UPS system is now online and operational. There has been no impact to customer services and this event is now considered resolved and closed.

2010
07/12
POSTED BY
gnelson
CATEGORY
announcements
data center
Comments Off

(Resolved) Unplanned Event: Defcon Managed Support Line

EVENT START DATE: 07-09-2010

EVENT START TIME: 15:00 CST

EVENT END DATE:  07-9-2010

EVENT END TIME: 15:45 CST

EVENT IMPACT: Toll free and auto routing of inbound Defcon Managed Support line

EVENT DETAILS:

At approximately 3pm on Friday, July 9th, incoming calls to our Defcon Managed Support line began experiencing routing problems or fast-busy responses.  This is impacting customers needing to contact our Managed Support team for updates on open issues or to report new issues via phone.  We apologies for this inconvenience and are working with our providers to correct the call routing and restore full phone services.

Access to all technical support, including Managed Support, remains available via https://pentagon.layeredtech.com.

Thank you for your patience while we work through this issue.

UPDATE 15:45 07/9/2010: The inbound call routing issues have been resolved.  Thank you again for your patience while we worked to resolve this issue.

2010
07/09
POSTED BY
nmarple
CATEGORY
Uncategorized
Comments Off

(Completed) Event Notification: ALO01 (Cedar Falls, IA) 07-11-2010 00:00 CST

EVENT START DATE: 07-11-2010

EVENT START TIME: 00:00 CST

EVENT END DATE:  07-11-2010

EVENT END TIME: 06:00 CST

EVENT IMPACT: Connectivity stability in the ALO01-Cedar Falls data center affecting multiple customers

EVENT DETAILS:

On Sunday 7/11/2010 at 00:00 CST, LT Network Engineering will be performing scheduled maintenance in the Cedar Falls data center to provide improved customer routing.  Customers in the Cedar Falls data center will experience brief periods of outage as customer connectivity is aggregated in a phased and controlled migration.  Recovery for customers affected during this migration should be rapid and complete with no residual affect following resumption of normal connectivity.

2010
07/09
POSTED BY
mfritsche
CATEGORY
Uncategorized
TAGS

Comments Off

Event Notification: ALO01 (Cedar Falls) UPS Quarterly Maintenance 29-JUL-10

START DATE: 29 JULY 2010
START TIME: 20:00 (08:00PM) Central Time

END DATE: 30 JULY 2010
END TIME: 04:00 (04:00AM) Central Time

WORK PURPOSE: UPS Quarterly Maintenance
IMPACT: Prairie Lakes Data Center

EVENT DETAILS:

TEAM Technologies will be performing its regular UPS quarterly preventative maintenance from 20:00 (8pm) July 29th to 04:00(4am) July 30th.  The data center will be transferred to generator power for the duration of the maintenance window. No disruption to customer power is expected during the maintenance window.  LayeredTech personnel will also be on-site to monitor the maintenance.  If there are any questions regarding this maintenance, please open a support request via Pentagon (https://pentagon.layeredtech.com).

2010
07/06
POSTED BY
gnelson
CATEGORY
announcements
data center
Comments Off

(RESOLVED) Event Notification: ALO01 (Cedar Falls, IA) 06-30-2010 14:55 CST

EVENT START DATE: 06-30-2010

EVENT START TIME: 14:56 CST

EVENT END DATE: 06-30-2010

EVENT END TIME: 15:02 CST

EVENT IMPACT: Connectivity instability in the ALO01-Cedar Falls data center

EVENT DETAILS:

At 14:56 CST on 6/30/2010, LT experienced an event that has resulted in some instability for some users in the Cedar Falls data center.  LT Network Engineering is aware of this situation and is working at this time to identify and resolve the detected instability.  Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue.

EVENT UPDATE:  This issue has been resolved.  A failure condition on a switch port resulted in downstream customers experiencing a brief period of impact while the redundant connection took over.  LT Network Engineering has reviewed the status of the connected devices and finds no further issue.  If corrective maintenance is required it will be scheduled during a standard maintenance window.
Thank you for your patience while this issue was quickly

2010
06/30
POSTED BY
mfritsche
CATEGORY
Network
data center
TAGS

Comments Off

Event Notification: ORD01 (Chicago) Generator Maintenance 11-AUG-10

EVENT START DATE: 08-11-2010
EVENT START TIME: 18:00 CST

EVENT END DATE: 08-12-2010
EVENT END TIME: 00:00 CST

EVENT IMPACT: ORD01 (Chicago)

EVENT DETAILS:

CoreSite will be performing quarterly checks on the facility generators.  This includes performing a 30 minute loaded run.  Best practices and procedures will be employed throughout this event and we would like to ensure that all relevant IDC tenants are aware of the activity.  LayeredTech personnel will also be on-site to monitor the maintenance.  If there are any questions regarding this maintenance, please open a support request via Pentagon (https://pentagon.layeredtech.com).

2010
06/29
POSTED BY
gnelson
CATEGORY
announcements
data center
Comments Off

(RESOLVED) Network Incident Report: ALO (Cedar Falls) 27-JUN-2010 22:00

Network Incident Report: ALO (Cedar Falls) 27-JUN-2010 22:00

EVENT START DATE: 27-06-2010

EVENT START TIME: 22:00 CDT

EVENT END DATE: 28-06-2010

EVENT END TIME: : 01:20 CDT

EVENT IMPACT: Connectivity stability in the ALO-Cedar Falls data center impacting multiple customers.

EVENT DETAILS: At 22:00CDT 27-06-2010 Layered Tech NOC observed connectivity issues impacting the Cedar Falls data center.  Network Services is working to identify the source of the issue and a resolution.  Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue.  Our apologies for the inconvenience.

UPDATE: As of 22:40 LT Network Services is continuing to work to isolate the cause of the connectivity stability issues in Cedar Falls.  Steps are being taken to isolate and mitigate the impact to the various segments of the network and to return stability as soon as possible.  At this time there continues to be some instability that is manifesting itself for customers as latency/packet loss and loss of connectivity to various hosts.  We will continue to update this posting as progress is made, and appreciate your patience while we work to resolve this issue.

UPDATE: 23:40, LT Network Service has identified a DDOS event which is impacting portions of the Cedar Falls network.  Efforts to isolate the source and targets of this attack are underway.  Customers should begin seeing an improvement as these efforts continue.  We appreciate your patience in this matter as we work to fully resolve the issue.

UPDATE: As of approximately 01:00 CDT, LT Network Services was able to identify the various sources and targets of a DDOS which was of significant size and scope to have impacted multiple portions of the Cedar Falls network.  External sources of the attack and internal targets have been null routed to eliminate the impact to the remainder of our customer base in this facility.  Overall network performance and stability has returned to normal as a result of these efforts.

Again, we appreciate everyone’s patience while we worked to identify and resolve this issue.  If further updates are necessary they will be posted here.

UPDATE: 28-06-2010 9:45 CDT Layeredtech monitored a return of the DDOS pattern which preceeded the earlier issues.  Measures have been taken to isolate the traffic in question in order to protect the overall network health and ensure continued access and performance for our customers installed in the Cedar Falls data center.  We continue to work with our upstream providers to implement a permanent solution to this issue.  As further updates are necessary we will post them on LTStatus.

2010
06/27
POSTED BY
nmarple
CATEGORY
Network
Comments Off

(Completed) Scheduled Maintenance: ORD02 (Chicago) 06-26-2010 00:00 – 06-26-2010 04:00 CST

EVENT START DATE: 06-26-2010

EVENT START TIME: 00:00 CST

EVENT END DATE: 06-26-2010

EVENT END TIME: 04:00 CST

EVENT IMPACT: ORD02 (Chicago)

EVENT DETAILS:

On Saturday 06/26/2010 at 00:00 CST, Internap Network Services, the primary enterprise Internet Service Provider for Layered Technologies, will be performing a scheduled maintenance on one of our circuits in the Chicago office.  To mitigate any problems with connectivity that may result from this maintenance, LT Network Engineering will move our traffic to our other Internap connection for the duration of that maintenance.  While no outage is expected as a result of this scheduled maintenance due to multiple routing redundancies that LT maintains, customers may experience brief periods of higher latency due to route convergence.  LT Network Engineering will be monitoring this situation closely in the event of any unexpected routing anomalies.

2010
06/23
POSTED BY
mfritsche
CATEGORY
Network
announcements
data center
TAGS

Comments Off

(Completed) Scheduled Maintenance: Grid 21-7 07-11-2010 00:00 CDT

EVENT START DATE: 07-11-10

EVENT START TIME: 00:00 CDT

EVENT END DATE: 07-11-10

EVENT END TIME: : 05:00 CDT

EVENT IMPACT: Virtual Private Servers hosted within the 72.233.48.0/24 and 72.233.51.0/24 ranges.

EVENT DETAILS: At 00:00CDT 07-11-2010 Layered Tech Operations will be upgrading Grid 21-7 to AppLogic 2.8.9 which is the latest production release. This upgrade process requires that all instances be brought offline until the upgrade has been completed.

UPDATE: This maintenance activity has concluded and all VPS’s are online. If your VPS is not functioning properly and is located within this IP Range please create a Support request via Pentagon.

2010
06/15
POSTED BY
dmyers
CATEGORY
host services
Comments Off