DynStatus

Degraded Performance in Chicago

Starting at approximately 1530 UTC today we experienced routing issues with one of our bandwidth providers in our Chicago datacenter. Our Operations staff took measures to work around the issue until it could be resolved. This had an impact on Dynect DNS services as some traffic was routed to datacenters in other regions.

By 1630 UTC, regular service was restored.

Degraded Performance in Hong Kong

Due to a routing issue with an upstream provider, which began at approximately 1725 UTC, users may notice a degraded performance in DNS resolution and monitoring from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 1830 UTC, routing issues upstream have been resolved.

Bandwidth Provider Maintenance in Palo Alto, CA

On October 30, between 0600-1400 UTC, our bandwidth providers will be performing maintenance on our connections in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers with US West regional definitions a small percentage of responses may return your global pool. We do not recommend any changes to your GSLB regional settings.

Bandwidth Provider Maintenance in Newark, NJ

On October 18, between 0300-1100 UTC, our bandwidth providers will be performing maintenance on our connections in our Newark, NJ data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US East site definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

Internet Routing Issues

At approximately 1915 UTC today routing issues involving AS 701 (MCI Communications Services, Inc. d/b/a Verizon Business) resulted in some of our sites being unavailable to part of the Internet. The problem appeared to be localized to the northeastern United States. Due to the diversity of our connections to the Internet and hosting locations, we successfully routed around the outage.

As of approximately 1950 UTC, the issue appears to have been resolved, and connectivity restored.

Dynect Platform Monitoring Degradation

As of 0900 UTC, we’ve been made aware of degradations of service with our Dynect Platform Monitoring service. Our teams are actively investigating the issue.

Update: During a DDOS, our monitoring agents were unable to properly report their results. This resulted in some monitoring services reporting incorrect results. After mitigation of the attack we were able to resynchronize the monitoring services.

As of 1015 UTC all monitoring services have returned to normal. We apologize for any false negatives that may have been reported during this time.

Update 2: During a DDOS, from approximately 0840 UTC to 0930 UTC DNS responses from 2 of our nameservers were impacted. Our other nameservers were not impacted at all, and responded flawlessly during this time.

All DNS responses are back to normal at this time.

Statistics Reporting Maintenance Window

On Monday, Sept 28, 2009 from 1330 UTC to 1400 UTC we will performing the final stage of our Statistics and Reporting upgrades. During this maintenance window there will be a brief period of time that graph retrieval via the web interface will be unavailable.

We apologize for any inconvenience.

Bandwidth Provider Maintenance in Ashburn, VA

On October 1, between 0700-1000 UTC, one of our bandwidth providers will be performing maintenance on their router in our Ashburn, VA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your Ashburn, VA site definitions are not necessary as this provider does not handle anycast traffic.

Dynect DNS and monitoring services at this site will continue to function.

Bandwidth Provider Maintenance in Palo Alto, CA

On September 28, between 1000-1400 UTC, one of our bandwidth providers will be performing maintenance on their router in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your Palo Alto, CA site definitions are not necessary as this provider does not handle anycast traffic.

Dynect DNS and monitoring services at this site will continue to function.

Dynect Traffic Management Interface Upgrade

During our regular maintenance window on Tuesday Sept 22, 2009 (0900 UTC) we will be releasing an updated version of our Traffic Management user interface which makes a change in the ruleset definition interface.

We will be replacing the ruleset definitions from location specific to regional definitions. The new user interface will display the following regions.

• US East (combines Newark and Ashburn rulesets)
• US Central
• US West
• Asia
• EU West
• EU Central
• EU East

What does this mean to you? If you have defined your Traffic Management Service with just a single Global Ruleset you will see no changes in the way the current service works.

For the regional location which will be combining two existing city locations the results will be that those rules will be combined into a single ruleset. If both cities have matching rulesets you will not notice any change in the way DNS is handled.

This modification to the Global ruleset definitions will provide you with easier ruleset definitions as we add more individual locations to our network.

Routing Reconvergence in Amsterdam 2

At approximately 23:45 UTC, a routing event in our Amsterdam 2 point of presence caused BGP routing re-convergence, creating a 1 minute degradation of service for ns2.pXX.dynect.net and ns4.pXX.dynect.net. Our Operations team quickly investigated the issue and isolated the problem to upstream service provider maintenance, and has mitigated the issue. Traffic in this region was gracefully handled by our ns1.pXX.dynect.net and ns3.pXX.dynect.net nodes.

Dynect Statistic Reporting Maintenance Outage

Between 1300 and 1900 UTC on September 14, 2009, we will be performing maintenance on the servers for our statistics reporting. During this time reporting and graphing of Dynect statistics will be unavailable.

We will continue to be collecting the data for statistical reporting, there will be no loss of query information. For a short period following this maintenance window, statistical reporting will be delayed as the server processes the backlog of queries from that time.

No other services will be effected during this event.

Update: Data transfer for this maintenance window is taking longer than expected. We are extending this Statistics Reporting Outage until 0000 UTC Sept 15, 2009.

Update 2: As of 22:40 UTC Sept 14, 2009 this maintenance window has closed. The statistics server is available again. There will be a short time over the next several hours while the query logs are processed that the data presented will be incomplete.