DynStatus

EveryDNS: System Migration Update

As of 11-March-2010, 17:00 UTC, www.everydns.com, dyn.everydns.net, axfr.everydns.net, and the main EveryDNS databases have been migrated to Dyn Inc. operated systems. Over the coming weeks, we will be integrating user support and administrative systems. Response to support inquiries may be delayed, however, we are working diligently to respond to all EveryDNS as soon as possible.

Network Capacity Upgrades - London

As part of our ongoing efforts to expand and harden the Dyn Inc. infrastructure, we will be upgrading our transit links to NTT Communications at our London Point of Presence on Thursday, March 11, starting at approximately 15:00 UTC. No impact to production systems is expected as this work is fully scripted and coordinated, however, to complete this update, some routing adjustments may occur, and customers may experience slightly degraded network performance as we migrate to the new links. These disturbances should last no longer than 5 minutes.
Update: Upgrade Complete

EveryDNS: System Migration to Dyn, Inc. Infrastructure

On Sunday, March 7, we will be migrating the EveryDNS Website and Dynamic DNS functionality to Dyn Inc. infrastructure. No impact to DNS services is expected, although users may experience a small window of time where www.everydns.com and dyn.everydns.net are unavailable.

Update: This work will be completed on Monday, March 8th, at approximately 17:30 UTC.

Network Capacity Upgrades - Palo Alto, Chicago, and Frankfurt

As part of our ongoing efforts to expand and harden the Dyn Inc. infrastructure, we will be upgrading our transit links to NTT Communications at our Palo Alto, Chicago, and Frankfurt Points of Presence on Friday, March 5, starting at approximately 20:30 UTC. No impact to production systems is expected as this work is fully scripted and coordinated, however, to complete these updates, some routing adjustments may occur, and customers may experience slightly degraded network performance as we migrate to the new links. These disturbances should last no longer than 5 minutes per PoP.

Update: Frankfurt upgrade is now scheduled for Monday March 8 at approximately 20:30 UTC.

EveryDNS: New IP Address for axfr.everydns.net

To All EveryDNS Users,
In preparation for the migration of the EveryDNS userbase to Dyn Inc systems, we will be migrating axfr.everydns.net to a new IP address. We will be starting up the AXFR processing daemon at 208.78.69.112 to test connectivity to various EveryDNS users. Users are encouraged to permit the current axfr.everydns.net IP address (71.6.202.218) and the new IP address (208.78.69.112) to permit zone transfers.

Router Maintenance in Hong Kong

On March 4th, between 0100-0300 UTC, we will be performing router maintenance in our Hong Kong data center. This will result in an outage of our ns5.dyndns.org and ns5.mydyndns.org nameservers. The other four dyndns.org and mydyndns.org will continue to serve our Dynamic, Custom, and Secondary DNS services normally. There is also the possibility of degraded performance for Internet Guide as queries for that service are routed to other datacenters. No other DynDNS services will be affected.

Update: This maintenance has been extended by 30 minutes to 0330 UTC. We apologize for any inconvenience this may cause.

Update: Maintenance is complete as of 0325 UTC.

Router Maintenance in Hong Kong

On March 4th, between 0100-0300 UTC, we will be performing router maintenance in our Hong Kong data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers with Asia regional definitions a small percentage of responses may return your global pool. We do not recommend any changes to your GSLB regional settings.

Update: This maintenance has been extended by 30 minutes to 0330 UTC. We apologize for any inconvenience this may cause.

Update: Maintenance is complete as of 0325 UTC.

Network Capacity Upgrades - Ashburn and Amsterdam

As part of our ongoing efforts to expand and harden the Dyn Inc. infrastructure, we will be upgrading our transit links to NTT Communications at our Ashburn and Amsterdam Points of Presence on Wednesday, March 3, starting at approximately 20:30 UTC. No impact to production systems is expected as this work is fully scripted and coordinated, however, to complete these updates, some routing adjustments may occur, and customers may experience slightly degraded network performance as we migrate to the new links. These disturbances should last no longer than 5 minutes per PoP.

Update: Planned maintenance in these PoPs is complete as of 21:00 UTC.

Dynect Monitoring Service Interruption

Beginning at approximately 2230 UTC 2010-02-22 a database issue on our Dynect Platform database server caused degraded performance for monitoring result processing.

As our operations and engineering team works to resolve this issue, monitoring services for Load Balance, Failover and Dynect Traffic Management services are disabled.

This event is in no way related to yesterday’s status posting regarding monitoring services, this is a different event.

UPDATE: As of 0120 UTC 2010-02-23 all services are again operational. We apologize for this interruption in service. Our system engineers are following best practices for ensuring data integrity and working to minimize the impact as much as possible.

DynDNS.com Cart Addition Problem

Beginning at approximately 0800 UTC, users were unable to add new items to their shopping carts on DynDNS.com.

UPDATE: As of 12:00 UTC, the problem was identified and corrected. Purchases made on DynDNS.com are now processing normally. Any users affected by the problem have had their shopping carts automatically repopulated by 16:00 UTC, and received an email informing them that they can continue their order.

Dynect Monitoring Disruption

Beginning at approximately 18:05 UTC, a database maintenance processes caused disruption in the Dynect Platform monitoring and reporting system.

UPDATE: As of 20:45 UTC, Dyn Inc. Operations located the source of the issue and restored all monitoring and reporting systems. During this time period, Dynect Platform customers utilizing our Failover, Load Balancing, and Traffic Management functions did not have their IP endpoints monitored for failure. All systems are functioning normally at this time. An Incident Report will be forwarded to all customers via e-mail shortly.

Hardware Malfunction in Palo Alto Data Center

Starting at approx 1332UTC we had a hardware malfunction in our Palo Alto Data center. DynDNS.com customers may have seen degraded performance in DNS resolution from the West Coast of the United States.

As of 1415UTC our operations staff have corrected the issue, and all traffic to the site is moving normally.

Dynect Platform services being monitored from this site were automatically configured to be monitored from the next closest location. There was no interruption to Dynect Platform DNS as requests to this site were answered from another of our Anycast nodes. Regionalized Dynect Traffic Management services saw no downtime as queries were resolved from other Anycasted nameservers within that region.

Bandwidth Provider Maintenance in Chicago, IL

On February 18, between 0800-1100 UTC, one of our bandwidth providers will be performing maintenance on their router in our Chicago, IL data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US Central definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

MailHop E-Mail Delivery Delays

Due to an extremely high amount of incoming e-mail being queued on our MailHop Forward, Relay, and BackupMX products, we are seeing intermittent delays for some customer e-mail. We are taking steps to correct the problem, and will update this post at regular intervals with status information. No customer e-mail has been lost, everything will be delivered. This does not affect MailHop Outbound, nor any of our other products.

We apologize for any inconvenience this may have caused.

Update: As of approx. 1900 UTC, the queued mail is continuing to drain. We are continuing to monitor the situation.

Update 2: As of approx. 2300 UTC, we are continuing to deliver the backlogged mail. Most of the mail has been delivered at this point, and we expect it to be completely caught up in the next couple of hours.

More EveryDNS DDoS

We’ve noticed a large DDoS attack targeting EveryDNS this afternoon. While we do not operate the infrastructure or network that powers EveryDNS, we are working hard with them to get back up and running. We are moving towards creating a parallel infrastructure to make EveryDNS more reliable (why we acquired them, in the first place), but have yet to do so. If you are affected by the outage and want to transfer things over to DynDNS.com, please create a DynDNS.com account and contact support@dyndns.com to set things up.

Bandwidth Provider Maintenance in Newark, NJ

On February 9, between 1900-2100 UTC, one of our bandwidth providers will be performing maintenance on their router in our Newark, NJ data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US East definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

Webhop Outage

As of 1033 UTC, Webhop was the target of a distributed denial of service (DDoS) attack. This did not cause any issues with DNS, Web, checkip.dyndns.org, and NIC update services. The Operations team has resolved the problem. Service returned at 1237 UTC.

Site maintenance in Newark, NJ - Canceled

On January 28, between 1900 and 2000 UTC, we will be performing site maintenance at our Newark, NJ data center. In order to avoid any interruptions to the Dynect monitoring and DNS services, and DynTLD DNS services, we will be removing this site from the Dynect and DynTLD anycast pools during this period. DNS and monitoring services will continue to be served from the other locations across our network. For Dynect traffic management customers, changes to your Newark site definitions are not necessary as this traffic will be routed to other anycast sites.

Update: maintenance for this site has been canceled.

Bandwidth Provider Maintenance in Ashburn, VA

On February 1, between 0700-1000 UTC, one of our bandwidth providers will be performing maintenance on their router in our Ashburn, VA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US East definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

Degraded DNS Resolution from Hong Kong

Due to a routing issue with an upstream provider, which began at approximately 2335 UTC, users may notice a degraded performance in DNS resolution from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 0240 UTC, routing issues upstream have been resolved.

Degraded Performance in Hong Kong

Due to an issue with an upstream provider, which began at approximately 2335 UTC, users may notice a degraded performance in DNS resolution and monitoring from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 0240 UTC, routing issues upstream have been resolved.

EveryDNS Outages (As Observed)

For the last hour, we have been tracking reports of EveryDNS outages. While we do not operate the infrastructure or network that powers EveryDNS, we are in contact with them and working to get back in order. If you are affected by the outage and want to transfer things over to DynDNS.com, please create a DynDNS.com account and contact support@dyndns.com to get things going.

OpenSRS outage

Beginning at approximately 2206 UTC, we started receiving errors via our interface with OpenSRS covering the following TLDs: .mobi, .com.cn, .net.cn, .org.cn, .biz, .org.uk, .me.uk, .info, .tv, and .name. Currently, there is no estimated time to repair. Changes and registrations to domains under those domains are currently unavailable.

Update: As of 2300 UTC, all services are back to normal.

Emergency Bandwidth Provider Maintenance in London

On January 13, between 0200-0500 UTC, one of our bandwidth providers will be performing maintenance on their router in our London data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths.

Dynect DNS and monitoring services at this site will continue to function.