Archive
Why can’t web crawlers use some ethics…and some intelligence?
So I’ve been up since 4am this morning battling what is essentially a Distributed Denial of Service attack…basically a bunch of computers sending requests to our web servers over and over and over again. After two hours of battling, the culprit was found and disabled.
This company offers to crawl data on websites via some customizable code. However, their business practices are definitely questionable. A google search is most enlightening. This web crawler hit our site over 7,000 times in a 10 minute span. And based on that Google search, we are not the only ones.
Now there are a couple of things I simply don’t understand. First of all, who’s the genius at 80legs.com that thinks hitting any site on the web at this volume is a good idea? I understand that they have a business, that they are selling crawling technology, but how much do they expect to sell if the end result of implementing their crawler is the immediate block of the crawler by the unwitting victim? Certainly whoever is paying them to crawl our site is now going to be disappointed.
Second, why would anyone think that this sort of crawling is ethical in this day and age of botnets and hackers? If I was building a business based on this technology I would at a minimum make sure targets could remove themselves from the line of fire (80legs claims it does so but it doesn’t work…they don’t honor robots.txt like they say the do), and make sure my bot speed was within reason speed wise. Google, Bing, and Yahoo all crawl the web without causing mass chaos and overwhelmed servers. Certainly if you have the intelligence to write a crawler, you have the intelligence to throttle a crawler.
Or maybe my standards are too high.
Why I am disappointed in my new Nexus
So I posted a tweet about this, but upon further review with my colleague Drew, I’ve got a pretty decent handle on what bugs me about my new Samsung Nexus Droid. Everyone has congratulated me on getting the next top of the line phone but for me, while that’s all great, the convenience factor of this phone is dramatically limited. It’s almost as if someone decided convenience wasn’t necessary on the ultimate device of convenience…your phone.
Let’s start with silencing your phone. On my old Droid 2, on my home screen, even before the unlock screen, I had a slider I could use to instantly go into vibrate mode. Very handy when I got texts or phone calls during meetings…immediate ignore and then I could put the phone back into standard mode without worrying about settings. In order to put my phone into vibrate mode, I have to unlock the phone and go do three or four clicks deep. That’s unacceptable in my opinion. And yes, I get that there’s a volume control on the phone, but I don’t want volume control; I want instant vibrate mode. Maybe there’s an app for that?
Speaking of apps, I had to download the beta version of Swipe to get my swipe on; it doesn’t come native on the Nexus. Took me a while to figure out why I couldn’t find it and I eventually put the app on the phone. Smart move, yep, taking away one of the input methods.
The systemic screens are unintuitive as well…my first phone call only showed me the phone icon, with no indication as to what I should do to ignore or accept the call. I had to do some experimentation to figure out the slide techniques to do so, but there was certainly no hint on the UI that I had actions available to me.
With that, I can admit to you that the screen on the Nexus is gorgeous, and I like some of the new features such as the foldering on the homepage, I feel like my convenience has been snatched from me like an ice cream cone in a hurricane, and I’ve been left all wet and hungry. Maybe next time I’ll ask a few more questions and not take so much for granted, eh?
Welcome To WordPress!
Hi folks!
I’ve decided to move from Blogger to WordPress, primarily as a public place to house my resume and some other information about that I want to be public. Feel free to follow or friend me as you wish; you can find social media links on my About Me page.
If you are an employer seeking to speak to me, please refer to my resume for information about me, or connect with me via LinkedIn.
Thanks!
