Monthly Archives: May 2012

Web Crawlers Looking for an Unsecured Server

Recently I checked my server’s access log to see who had been visiting my site. There is not a ton of content yet so I was shocked to see that I had over 3,872 requests since it’s launch. I can understand; I’m a pretty big deal, right?

Unfortunately a lot of the page requests were from crawlers. Your first thought may be “Google bots”, however they were only a minor player.  Most of the web crawlers were “hack” bots, whose sole purpose was to break into my site. The requests would almost always be looking for unsecured parts of the site. Here’s what a typical part of the access log looks like:

Snippet of a "hack" bot's requests

Most of the requested files don’t even exist. The bot is just guessing random admin-esque urls. There are a ton of these attacks. I took the time to compile some basic info on them and found out a few interesting things.

    • The total amount of requests related to attacks was about 20% of all requests.
    • Most attacks made about 40 requests before giving up ( None succeeded)
    • Most attacks lasted between 5 and 20 seconds
    • Attacks originated from the following countries:
      • USA, Poland, Singapore, Canada, South Korea, Japan, Australia, Taiwan, Brazil, Thailand, Germany, Venezuela
    • The U.S. had the most attacks, 3.
    • Interestingly some of the worlds largest countries by population were absent including: China(1), India(2), Indonesia(4), Pakistan(6), Nigeria(7), and Russia(8). (Population is not a good predictor of the “attacking” countries, however, GDP per capita is. Nine of the thirteen “attacking” countries had GDP per capita in the top 25.)

I remember when I installed the server there were a lot of disclaimers about maintaining security. When reading them, my first thought was always something like “Who’s trying to get into my tiny website? Not many people will even know I’m here.” I was wrong. The web is filled with crawlers, many of which are scanning for a free server. Check out a snippet from my access_log to get real glimpse at what even a tiny server like this one gets requests for.