Happy to report nothing short of smooth sailing since the big DDos attack back in March.
All client sites have been moved to new IP addresses and only a few of my old websites remain on the old IP address. I received a message at about 11pm from my system admin who suggested that the attack is over and that he had removed the IP block from the afflicted IP address. With in 15 minutes I had another message from him explaining that the attack started back up. Of course this is a bummer but it is great confirmation that the efforts I put in place to fix the problem for all of my clients worked.
I have also been told by several of the system administrators I have communicated with that they have never seen an attack so persistant. Usually DDos attacks last a short while and when they go away they almost never come back.
I am just glad I have everything set up the way I do. If I did not have a team watching over my server 24/7/365, I would not have caught this so early and been able to take action to get my clients sites back up in less than 12 hours.
I have completed the change of IP addresses for all clients. I will continue to monitor things and if there are any other issues that come up I will update all of you again.
Over the next few hours the remaining sites should start to come back online. Depending on the internet service providers (ISPs) it could take as little as 30 minutes and as much as 24 hour hours. So far I have been noticing that in the Modesto area it’s taking less than an hour to propagate.
I am truly sorry for the inconvenience. There is nothing that I could have done to prevent it, however if this should happen again in the future I know exactly what to do to get changes made quickly.
If any of you are interested in your own unique IP address which would separate you from the current shared IP address, please let me know.
If you have any questions at all, please contact me at http://jeradhill.com/contact or give me a call directly.
Me and Rockstar continue to work on the issues at hand. Many of our clients websites are back up and running on their new IP addresses already. As I explained in the previous post, some internet service providers take longer to update than others.
If you have any questions at all, please email me jerad@jeradhill.com or give me a call.
I am currently reassigning my clients one by one to different IP addresses. This attack is coming at one website domain name. We are not sure which website at this time. However, moving websites to different IP addresses will help us figure out which website is the one being attacked. It is likely that when I move that particular website over to a new IP, the attack will follow it. If the attack does move I am not sure if it will let go of the original IP address or attack both. However I am moving websites to different IP addresses and moving the websites that I would think could be attacked to their own IP addresses. Like I said, I have no clue which website the attack is pointed at. IP addresses are not cheap and I have been adding them like crazy. After we are in the clear I will probably be making additional adjustments as well as making it an option for my clients to add an IP address to their hosting package so that they have their own IP that is not shared with other users.
As I make these IP changes you may begin to see your website come back up. There are many internet service providers in our area and all over the world and each of their systems have to update. This can take minutes to a full day.
This is an explanation from my system admin of where we are at and what possibly may have happened.
DDoS attacks don't have much to do with your server's configuration.
Usually the reason that these attacks happen is that you have a site that is
running on your server that either offended someone, or is being extorted by
someone who has the resources to amass an attack.
A fair number of computers out there, be it someone who leaves their computer
on at home, in an office, or at a university, with malicious software
installed, can take a server down with their combined resources. Usually these
infected systems "phone home" to an IRC chatroom or the like, where
an individual can issue commands to attack a target.
Despite the mechanics, there is simply no way to filter this traffic out. 4
Gbps of attack traffic outmatches your server's network link by 40x.
I'm sorry, but we're going to have to leave your server's IP
blocked until we see the DoS end.
I am now able to access the server and have moved it to a different IP address. However, all of our clients are still on the effected IP address. I have just purchased additional IP addresses and will begin moving websites over one by one. Once I do this it could take 6-12 hours for the website to show up across all networks due to the IP address having to propagate. In most cases it will happen rather quickly. I will continue to update as I make progress.
Jerad
This email is going out to all of my hosting clients so I apologize for not addressing you by name.
Last night at around 9pm the servers came under attack via a DDos Request Attack (Denial of Service Attack). What this means is that someone wrote a script that would direct 1000’s of computers that were probably infected with malware to try and ping our server. The only thing a DDos attack attempts to do is make the servers unavailable. It is not out to collect information from websites or anything of that nature, simply just to bring a network to it’s knees. In this case it did. Myself as well as a team of network professionals are working on this to block the IP addresses that are driving this force at our network. We have yet to determine which website they were attempting to shut down and we probably will not know until we can get the server back up on it’s own and look through log files. I have been up all night working on this and have been able to get some of the IPs attacking the server blocked As of 10pm last night there were 100’s. Myself and the network administration team I have working with me on this will continue to work through today. What is great about the company that I work with to manage the health and wellbeing of my servers is that they have 24 hour support and a very knowledgeable staff that answers all of my questions. They have yet to fail me in the 8 years I have been with them.
Some of you are on private IP addresses because we have a secure certificate on your website, your websites are still up and running. One of the safeguards that I will be putting in place over the next few weeks is to spread all of you over several IP addresses so that if an issue like this happens again it will effect less of you. In the 12 years I have been doing this I have never dealt with an attack of this magnitude. Thankfully, because I take so many other safeguards, we have not lost any data, nor was any data breached.
Most of us have our email managed through Google Apps service which means that email is up and not effected by this issue. The rest of you are on your own email addresses with your own carriers so of course your email is not effected either.
I will continue to update all of you. I have set up a temporary website that I can easily post updates to from my computer and my iPhone. I will also further explain what a DDos attack on the website.
The location of that website is: http://jeradhillstudios.tumblr.com
I am truly sorry about this. There is nothing that I can do except try to block these addresses and hope the rest of them stop attacking. This won’t last forever. If you have any questions feel free to email me or give me a call. I will be in meetings from 11am until 2pm today.
