We are in the process of mitigating a large scale DDoS attack against our global DNS platform. We expect service to return to normal very shortly. Stay tuned and let me know if you have any questions. ted@namecheap.com
Just for future reference, it's usually considered a good idea to put your status page on completely independent infrastructure so that it stays up even when the rest of your stuff goes down. A status page that doesn't work during an outage isn't particularly useful.
Good point. The status page is on another cloud but since this is a DNS issue, the subdomain is down. In the future, we'll investigate running this page on a secondary DNS.
According to whois for namecheap.com, the DNS for that domain is hosted on dynect.net, and "host status.namecheap.com" resolves just fine (to 204.232.212.56), so it does not appear to be a DNS issue that is preventing your status page from working.
Perhaps also investigate allowing customers to slave their own secondary DNS, too? (That is, allow AXFRs.)
This has been a feature requested for, as far as I can tell from the support forums, four years now. It should be possible to make this allowed/disallowed per-zone, and as far as I'm concerned, I don't care who is able to download my tiny zone file. It would allow me to add more diversified DNS servers in the face of things like a DoS against Namecheap.
Posting up-to-the-minute updates on Twitter is also a good idea. Lots of tweets come back from a search for "Namecheap" & you want to be sure you're a part of that conversation!
Cloudflare has also been nailed the last 3 weeks in a row causing outages. Just search Twitter for cloudflare DDOS or NTP or etc. etc. etc. At the end of the day there are currently so many slave machines out there we are all vulnerable. It's just the nature of things. At least outages are only temporary. It's much better than the early days were we'd be down for 12 hours at a time. https://twitter.com/search?q=cloudflare%20attack&src=typd
I suspect the hackers are targeting one of namecheap's customers, not namecheap directly. Because that's usually the case, a good approach is to not give all customers the exact same nameservers.
I agree. We left Network Solutions b/c they were hit 3 times in less than a year. Maybe the website targets moved to Namecheap, so they were targeted also?
Yes please, thank you :) All my domains are down now but I understand how shitty (the panic!) it feels to have things going down. I'll be checking this.
I tried this and now for some reason my apache server is returning the default page. also I think it wiped out my mailserver settings -- my mail is hosted on namecheap. i guess i gotta wait till everything is fixed.
ETA how long it will take for the transfer to be completed? I know that editing host records usually is instantaneous (unlike other providers), but we're talking about changing the DNS servers here.
I had a couple of domains to try this on. One domain switched over within a minute. The other has been almost an hour. I'm assuming the TLD makes a difference?
Was going to switch to route 53 while you guys were down and switch back, but the page says it'll take a day...Might as well just wait at that point. I know it's panic mode over there, but some kind of failover record would be awesome for when this happens in the future(or an option for one).
Everything related to DNS says it'll take a day but that's "worst case"... I've successfully transferred DNS for 5 Namecheap-registered domains to Route 53 since this all started a couple hours ago, and they're now up & running smoothly.
Their "update DNS server" page was acting a bit wonky, kept saying some of my nameservers were invalid when they weren't, but I eventually got them all switched.
This isn't a dig against Namecheap, it sounds like this attack is pretty bad, but for important domains Route53 just seems like a much better setup (geographically dispersed, different nameservers for each hosted zone, etc).
Hi, I work on Route 53, can you clarify; are you seeing a problem from your registrar when you try to move to Route 53? We're also always keen to hear about features customers would like.
It's definitely on our list, but in the meantime there is https://code.google.com/p/route53d/ which will let you translate an AXFR/IXFR from a bind server directly into Route 53 authenticated calls.
Does it take the same amount of time to complete that switching to another provider's DNS server takes?
Many of us already switched to other DNS servers during the panic, and we'd like to know whether you'd suggest we reverse that change in order to jump to v1 instead for a quicker result. Thank you.
You need to provide some gift to your customers for this downtime. I am using NC for 10 years, every time on bad issues I continue to use. But this outage very bad, I lost money...
While it's not a customer friendly viewpoint, it's not exactly what happened here. You have to consider what is being provided and how much was paid into it for said service. Had they been paying a lot of money for solid DNS I could agree with you, but they weren't.
It's a completely free service. With an extremely low price for domains.
Even though it's become almost a standard for domain registrars to provide DNS, it's still free. I've been using NC for years for my business (also a free service) and the downtime for what it provides has been mostly minimal.
Now if I was paying namecheap specifically for enterprise DNS something like $10/month or $40/year and it included DDoS protection then yes I should be compensated. Namecheap gave people what they paid for - a domain, the DNS is just a really great perk.
If namecheap chose to stop assigning all customers the same list of DNS servers, it would benefit namecheap as well. There's 200+ people in the queue for live chat right now :)
It's not DDOS mitigation or H/A, or some other high end feature.
The only way this would be successful is if the "customer IP's" were spread accross separate networks, and the announcement for the attacked network was sent somewhere else.
Assigning customers across lots of IP's in the same /24 isn't going to do anything. A volumetric attack is still going to succeed there.
In this landscape of ever-dwindling portable IPv4 subnets, it's harder and harder to get a /24. You won't get an assignment if your justification is "to fight a DDoS."
Are you idiot or kid? We are doing professional business and we were using namecheap DNS with registered domains (not free dns). The main point is protection, they provide this service FREE (with or without domain) yeah good but where is protection where is responsibility?
I just suggested something, if money important for me, I didn't move all of my domains from namecheap to another provider :)
That protection and responsibility is defined in your contract or registration agreement with NameCheap. You can consider DNS or DDoS protection NameCheap's responsibility, and they'll decide -- like any business -- which one they can afford.
While offering some gift or compensation for ANY negative incident -- especially one outside anyone's control -- is nice, that's not something everyone can afford under ANY circumstance.
If they messed up your cheeseburger, I could see this. How does this help this situation? It doesn't. Namecheap should take that money and invest in their infrastructure.
Weird, I haven't researched DNS as well as I should have. I always lived under the impression that there was this extensive DNS cache network where intermediaries responded to queries with cached results from root DNS servers.
Instead, the second that this DDos hits is the second we have websites stopping working.
How is it that in this day and age we can't have distributed caches of DNS entries at our providers of full dns databases. I mean there can't be more than like a few billion dns entries in the world total, which fits easily in a modern desktop computers RAM.
If that is an underestimate, I can't believe a single modern server wouldn't be able to mirror the world's DNS queries for at least a providers worth of users.
Yes, perhaps there lies our folly. It's the choice between being flexible in ability to move our servers really quick, or being tolerant of DNS servers going down.
I sort of hoped that a DNS client would just use an expired DNS result in case the servers would not respond, but perhaps that is naieve/dumb.
How would one add (say) AWS Route53 as a secondary DNS?
I assume you'd make sure the DNS records are the same in both DNS portals; and then add Route53 as 3rd & 4th nameservers with the first and second still being Namecheap?
At first I thought the same. I went in to edit records, and I hit save. I checked again and it was working. I assumed there was just a delay of a minute or so and that clicking save was a coincidence.
Well even if you switch to a different (working) nameserver, the old nameserver will still be cached all over the place so it will still appear down for many people.
I recently switched most of my domains to DNSMadeEasy because they are constantly in the top for speed[1], provide a top tier anycast network and for what you get are a great value.
If you want speed and readability I suggest switching to a paid DNS provider.
Best of luck to their support team. Outages can make tech support's life miserable. If you call in, just remember the person on the other side of the phone has likely been yelled at all morning for something that wasn't their fault. Totally reasonable to be upset at the situation, just don't take it out on the tech you're talking to!
I am monitoring a few servers with DNS records. And the last week I have found all the servers unresponsive (by DNS, not tried directly) from time to time. And after an extensive amount of troubleshooting I am unable to find a problem.
I don't know how this website works, but I can't see the latest posts at the top of the page! I'm looking for the latest info on the issue. Are you up and running? Should I move back to v2? Thanks
If it's absolutely critical that your users get service right now, it might be a good idea to at least prepare a migration to other DNS servers, like perhaps those of Linode. If the situation doesn't improve within an hour or so, it might be that they don't have a good way to deal with it, and the outtage might take long, depending on the depth of the DDoSers pockets.
Instead of putting all of your eggs in (yet another) one basket, for a site with critical availability requirements I would distribute DNS over multiple providers. If any one provider goes down it is less likely to hurt overall site availability.
Yep. I had been hesitating on moving more domains over to Route53 because I thought the per-zone pricing was fixed at $0.50 per domain, but it actually scales really well. I'm going to move the rest over to Route53 as soon as I have a chance.
We're using Route 53 in production to handle hundreds of millions of queries per month. We're happy with it, and the price is reasonable. Also, service checks with failover is heavenly.
If you don't use namecheap's dns servers, dns requests for your domain do not go to namecheap for redirection. They don't go to namecheap at all. Your specified dns servers get registered with the top level domain servers.
No. The zone file lives on the nameserver (DNS server) which Route 53 provides. Namescheap registers a list of nameservers for your domain with the TLD.
An over simplified example for looking up "news.ycombinator.com":
1. First query the TLD nameserver for all ".com" domains asking for the authoritative nameserver(s) for "ycombinator.com".
2. Next query that nameserver for "news.ycombinator.com".
But, the first entry point is namecheap. The domain is at Namecheap, so it will not find the zone file at Route 53, if Namecheap does not send it there. Right?
No, registrar's are your portal to updating data in TLDs, but those TLDs are each operated separately. .com and .net are operated by Verisign, for example.
