yesterdays outage

[wesintl]

If you're interested...

10/17/07 22:04:31
We are currently experiencing intermittent network connectivity issues, in which large DNS servers (Yahoo, AOL, Verizon) are periodically unable to resolve domains on our network. We are working directly with CISCO to identify and resolve this issue as soon as possible. System status will be updated when further information becomes available or when the issue has been resolved. We sincerely apologize for any inconvenience this has caused you and appreciate your continued patience.
10/17/07 17:08:53
At approximately 3:20pm EST today (10/17/2007), we experienced an internal network hardware malfunction. While EasyCGI is fully redundant, this malfunction caused routing problems that resulted in temporary downtime for certain pieces of our network. Because of this downtime you will have experienced slow response times or random inability to utilize services provided by EasyCGI. EasyCGI was immediately aware of this issue and was working to restore services. Service for most customers was completely restored by 4:00pm EST, while a small portion of customers experienced problems until 4:29pm EST. We apologize for any inconvenience this has caused and thank you for your patience.

 

[Uncle Ben]

As usual Wes, THANK YOU for taking care of us! We rarely have many forum or web issues (except trying to keep Arthog in the dark ) and it's reassuring to know your got it handled! You have a thankless type of job and I want to make sure you know you are appreciated!

 

[wesintl]

10/18/07 01:32:47
At approximately 3:20pm EST yesterday (10/17/2007), we experienced an internal network hardware malfunction. While EasyCGI is fully redundant, this malfunction caused routing problems that resulted in temporary downtime for certain pieces of our network. Because of this downtime you will have experienced slow response times or random inability to utilize services provided by EasyCGI. EasyCGI was immediately aware of this issue and was working to restore services. Service for most customers was completely restored by 4:00pm EST, while a small portion of customers experienced problems through the evening. These later issues involved large DNS servers (Yahoo, AOL, Verizon) and their periodic inability to resolve domains on our network. We worked directly with CISCO to identify and resolve this issue as soon as possible. All network connectivity was fully restored by 12:30am EST on 10/18/2007. We sincerely apologize for any inconvenience this has caused you and appreciate your patience

 

[bskey]

As usual Wes, THANK YOU for taking care of us! We rarely have many forum or web issues (except trying to keep Arthog in the dark ) and it's reassuring to know your got it handled! You have a thankless type of job and I want to make sure you know you are appreciated!

+1! I sure am glad you are here to do this Wes, and reading your posts about what happened was linened to listening to Tiwanese men cat call hookers in Las Vegas.... I just dont get it!

Thanks! I really enjoy this site and the people here; very sincere!

 

[Hulk]

Wes, thank you for all the work you do on the web site. The club wouldn't be what it is without your magic!

 

[corsair23]

As usual Wes, THANK YOU for taking care of us! We rarely have many forum or web issues (except trying to keep Arthog in the dark ) and it's reassuring to know your got it handled! You have a thankless type of job and I want to make sure you know you are appreciated!

x2...All that internet/cloud stuff just gives me a headache

 

[powderpig]

The other could not of said it better. thanks robbie

 

[nakman]

Man I was lost last night, but knew you were on it. thanks, Wes!

 

[Hulk]

Whatever they fixed, it may have been in the process of failing for a while. The web site is running 3X faster for me today than usual.

 

[wesintl]

10/18/07 14:21:11
Dear Easy CGI Customer, As many of our customers already know, our network experienced a series of problems yesterday that resulted in downtime. We did announce the situation yesterday with several system status messages that were posted in our control panel and emailed to all customers who subscribe to our system status mailing list, but I would like to formally explain what happened in more detail and the actions that were taken to resolve the issues. Yesterday at 3:20 PM EST, a switch on our network malfunctioned and created a problem that took down our entire network for 40 minutes. At 4:00 PM EST, we remedied this situation and began bringing the network back up in segments. By 4:30 PM EST, all parts of our network were back online. Almost immediately after this problem was resolved, we began receiving an extremely high influx of DNS traffic from large ISPs and Email providers such as AOL, Yahoo, Verizon, and Comcast. An immediate spike of email and DNS related traffic is expected after a period of network downtime, and mail and network requests are queued during the outage, but the volume in this particular case was higher than normal. Because of this influx of DNS queries, our Cisco firewall began dropping queries from high volume DNS servers as if those servers were making an attempt to attack our network. Without a valid DNS response from our Name Servers, web browsers and email applications are unable to function properly. This issue was immediately recognized and required the intervention of Cisco technical support to resolve. Both myself, our Vice President, Ted Beckwith, as well as our Network Engineers and several of Cisco's highest level of support technicians were working on this issue during its entire duration. At 12:30 AM EST this morning, we successfully implemented a work-around for the problem and at 1:30 AM EST the problem was fully resolved. During this time, many of our phone lines dropped offline to the heavy call load, which produced busy signals for many customers trying to call in to our support staff. I am communicating with you today to let you know that we are here for you 24x7, and every action we take on our network is done to maximize uptime and make sure that your services are up and running. When unexpected events occur, we are able to draw on a wealth of knowledge, both from internal and external sources, always working to minimize the impact on customer services. When unfortunate circumstances come together as they did yesterday, we must ask for your understanding and hope you do not lose confidence in our ability to provide a first class hosting service. We are determined to continue offering great service to our customers. Thank you again for your patience and understanding. Dan Richfield President, Easy CGI

 

[nakman]

Was it just me? or was the forum down last night?

 

[wesintl]

I wasn't online.. dunno. If it was probably not long. I didn't see anything in the log.