Archive

Archive for July, 2012

Why can’t web crawlers use some ethics…and some intelligence?

July 17, 2012 Leave a comment

So I’ve been up since 4am this morning battling what is essentially a Distributed Denial of Service attack…basically a bunch of computers sending requests to our web servers over and over and over again. After two hours of battling, the culprit was found and disabled.

http://www.80legs.com

This company offers to crawl data on websites via some customizable code. However, their business practices are definitely questionable. A google search is most enlightening.  This web crawler hit our site over 7,000 times in a 10 minute span. And based on that Google search, we are not the only ones.

Now there are a couple of things I simply don’t understand. First of all, who’s the genius at 80legs.com that thinks hitting any site on the web at this volume is a good idea? I understand that they have a business, that they are selling crawling technology, but how much do they expect to sell if the end result of implementing their crawler is the immediate block of the crawler by the unwitting victim? Certainly whoever is paying them to crawl our site is now going to be disappointed.

Second, why would anyone think that this sort of crawling is ethical in this day and age of botnets and hackers? If I was building a business based on this technology I would at a minimum make sure targets could remove themselves from the line of fire (80legs claims it does so but it doesn’t work…they don’t honor robots.txt like they say the do), and make sure my bot speed was within reason speed wise. Google, Bing, and Yahoo all crawl the web without causing mass chaos and overwhelmed servers. Certainly if you have the intelligence to write a crawler, you have the intelligence to throttle a crawler.

Or maybe my standards are too high.

Advertisements

Last Night’s Crazy Dream

July 14, 2012 Leave a comment

So last night I had one of those crazy dreams…I was working at some major retailer in some major city, professing my undying affection for one my high school crushes, who had just arrived in the store in a panic-room style elevator. She was returning my advances, only to tell me she was part of a revolution about to attack the city, from the basement, with an army of high tech weapons and a bunch of revolutionary friends, and she asked me to join her, which, of course, I did “for love” or some such sentimental reason.

Then Elijah woke up thirsty and wanted his water refilled. At 4:30am. Sweet.

One of those real “real” dreams where you think it’s really happening, and the only reason I’m posting it is because it was so crazy …and it sounds like the plot to a great story or maybe even book, and I don’t want to forget it…

P.S. I was a lot younger in the dream too…and even had all my hair back. Been having a lot of those lately.

Categories: Dreams

Why I am disappointed in my new Nexus

July 12, 2012 Leave a comment

So I posted a tweet about this, but upon further review with my colleague Drew, I’ve got a pretty decent handle on what bugs me about my new Samsung Nexus Droid. Everyone has congratulated me on getting the next top of the line phone but for me, while that’s all great, the convenience factor of this phone is dramatically limited. It’s almost as if someone decided convenience wasn’t necessary on the ultimate device of convenience…your phone.

Let’s start with silencing your phone. On my old Droid 2, on my home screen, even before the unlock screen, I had a slider I could use to instantly go into vibrate mode. Very handy when I got texts or phone calls during meetings…immediate ignore and then I could put the phone back into standard mode without worrying about settings. In order to put my phone into vibrate mode, I have to unlock the phone and go do three or four clicks deep. That’s unacceptable in my opinion. And yes, I get that there’s a volume control on the phone, but I don’t want volume control; I want instant vibrate mode. Maybe there’s an app for that?

Speaking of apps, I had to download the beta version of Swipe to get my swipe on; it doesn’t come native on the Nexus. Took me a while to figure out why I couldn’t find it and I eventually put the app on the phone. Smart move, yep, taking away one of the input methods.

The systemic screens are unintuitive as well…my first phone call only showed me the phone icon, with no indication as to what I should do to ignore or accept the call. I had to do some experimentation to figure out the slide techniques to do so, but there was certainly no hint on the UI that I had actions available to me.

With that, I can admit to you that the screen on the Nexus is gorgeous, and I like some of the new features such as the foldering on the homepage, I feel like my convenience has been snatched from me like an ice cream cone in a hurricane, and I’ve been left all wet and hungry. Maybe next time I’ll ask a few more questions and not take so much for granted, eh?

Categories: Informational, LinkedIn, Mobile