Author Archive

Customer service gone bad: How Honda almost won the battle, but then lost it and the war

December 12, 2013 9 comments

If you’ve followed me on Facebook, then you are aware of an ongoing battle I (and many others) have had with Honda about the quality of their paint jobs on their new Civic models from 2006 until today. I can still remember how excited I was to buy a new Civic in ’06. The car was much roomier than comparable models and it’s gotten excellent gas mileage, sometimes as high as 40mpg on the highway. I have about 115k miles on it in 7 years and it still runs beautifully.

But about two years ago the paint started to flake along the roof, and it progressively got worse. I reached out to my local Honda dealer, Sunset Honda in San Luis Obispo, and was basically told that I “park my car in the sun too much”. I responded with, “So does everyone else around me, it’s California…what else can I do?” Basically I was told the problem was mine.

I reached out to the dealer who sold me the car (Penske in Ontario) and while they were much nicer, they were clearly not interested in repainting the car. I got a couple of quotes myself on a repaint, one of which was more than the actual original sticker price of the car.

I reminded both dealerships that I was a good Honda customer, and was soon to be in the market for a mini-van, and the Odyssey caught my eye…but not if it was going to look like broken eggshells in 4 or 5 years. If they weren’t going to stand behind their product I wouldn’t buy another vehicle from them. There was no way that this problem was naturally occurring and the only logical explanation was a faulty paint job.



Over the next couple of years my car continued to get worse, and after some soul searching, I decided that I’d never recoup the value of my car if I spent that kind of money on a paint job. Since it still ran well, I decided I’d run it into the ground instead, try to squeeze another 6 or 7 years out of it. In the meantime, I commiserated with other Honda owners in the same situation and came to realize something.

There were thousands of Hondas out there just like mine, all with faulty, flaking paint. And Honda continued to blame sunlight and waxing and UV rays and whatever it could come up with to refuse to address that it obviously had a defective product.

Of course in the meantime, since my car was no longer under warranty, I stopped going to Sunset Honda for service. There’s several hundred dollars of revenue a year down the drain right there.

About two months ago, I received a surprise in the mail, so innocuous that we almost didn’t open it. It was a letter from Honda extending the warranty on paint for Civics like mine and offering to paint the car at no charge!

Hallelujah, right?

Well, sort of. See, while they were offering to paint it, they were going to make sure that it was as inconvenient as possible to do so. They only offered a six month time frame to get the repaint done. That was ok, but when I called the dealership to schedule, they said that they would need to have the car for two weeks to do the paint job, and no, they couldn’t expedite that.

Ok, two weeks isn’t bad if I have a loaner car…but no. Since, in the words of the Honda service technician, they were doing me a favor and helping me out by repainting my car, they were not going to offer a loaner.

Super. Would they reimburse the cost of a rental?


So basically, I had to endure the inconvenience and cost of replacing my transportation while they “helped me out”.

I solved that problem by scheduling my paint job over a time where I was either out of work on vacation or traveling out of the area to minimize inconvenience on my family.

Intriguingly, while waiting for my car to get painted, I discovered that there was a class-action lawsuit filed against Honda ( . So much for the idea that they had decided a repaint was the honorable thing to do, eh?

Once my car was painted, I wanted to pick it up on a Saturday, mostly because I didn’t want to have to interrupt work or my family schedule to do so. I got a call from Sunset Honda saying that I could pick it up, but it was extremely inconvenient for them because they don’t like to have repaints picked up while the primary technician was out of the office.

Making it crystal clear that my convenience was unimportant to them. Sweet.

But I picked it up on Saturday anyway, since I’m a rebel that way. The technician on duty at Sunset handed me my completed paperwork and then congratulated me on the repaint. I told him it looked fine. And he then graced me with this fantastic statement completely lacking in self-awareness:

“What other car company would repaint your car for free at 114,000 miles?”

Let’s see. One that also failed to paint it properly the first time? One that thinks they are
doing me a huge favor by fixing their faulty product? One that also completely missed the boat and a chance to gain a bunch of loyal customers by stepping up and fixing it right away? One that has now completely lost me as a customer?

Yeah, that company.

At the very least, the car pretty much looks like new. They painted everything except the bumpers.


You’d think that’s the end of it, but no, today the seal at the top of my windshield has started to come off…as near as I can tell they didn’t seal it completely. I’ll be taking that to my personal mechanic, because the last thing I want is for Sunset Honda (or any other Honda) to be involved in my automotive business. I’ll take my next $30k+ purchase to someone else.

One can only imagine, based on the number of other people I know with this problem, that there are now thousands of dissatisfied customers out there. I see Hondas with bad paint jobs every day now when I am out and about. And when I show the photos to my friends, they say they start seeing Hondas in the same situation. One wonders what that does to people who might have been considering buying a Civic or an Accord.

Or an Odyssey.

Well played, Honda. Well played.

Considering a new Code Camp talk

December 11, 2013 Leave a comment

So I’m considering getting back into the ring at code camps. For the past several years I’ve held out of them mostly due to more qualified people handling a lot of the topics, and wonderful personal things like my kids and family taking up quite a bit of time (seriously!). But lately I’ve felt the urge to get back into the swing of it.

One of the things I think is underrepresented, and will likely become the basis of my next talk, is Website Operations, and how you can support that concept via architecture and developer philosophy. This would cover things like email delivery and storage, cache management, web farm management, logging and debugging, change management and tracking, GA versus local web analytics, and so forth. Wondering if people would consider that interesting or boring?

Be careful interacting with any objects retrieved from cache

February 9, 2013 2 comments

Good evening all,

Just wanted to offer some background on a recent problem we’ve run into in several different web applications. When it occurs it can be difficult to diagnose and difficult to track down once it is diagnosed, but generally you can end up with a fairly easy fix. The problem revolves around storing List- or IEnumerable-based objects in in-memory cache in C#.

Our web applications generally have a robust caching mechanism built in to minimize reads to databases as well as reads to disk. You can store file contents or information about users in cache under the expectation that those won’t change very often. Many times things like lists of countries, or states, or property attributes would end up in cache as well; when is the last time the list of U.S. states changed? So instead of going to the database to get states when you need them, they become an obvious candidate for storing a List in some sort of cache.

The problem, then, is two-fold. Assuming that your List is in memory, your caching mechanism is likely to hand back the list when is asked for out of cache. Typically that’s mistake number one, because you don’t want to hand back the list itself, you want to hand back a new copy of the list. Now I realize that sounds like wasteful processing, to create a new copy every time something is requested from cache, but here’s a scenario and what can happen.

Let’s say that you have two web applications, one a standard desktop web site, and the other a mobile web site. Both of the them use a List to populate search forms. You decide that you’d like to sort the state list on the mobile site based on distance from the mobile phone’s location, which is different behavior than the desktop site.

Within minutes of starting up your desktop site, you notice that the state list in your search form appears to be in a random order.

What just happened?

Well, the mobile version of your website needed a list of states, sorted to meet its needs. Something like this:

List<State> states = CacheManager.Get<List<State>>(stateCacheCategory, stateCacheKey);
states.Sort((c, d) =>
   return GetDistance(c.Latitude, c.Longitude, localLat, localLong)
      .CompareTo(GetDistance(d.Latitude, d.Longitude, localLat, localLong));

If you can imagine, with a few supporting functions, this would get the states from cache and then sort them by distance from the phone’s latitude and longitude.

Seems harmless enough, right?

Except for when you haven’t created a new object from the list in cache. If you don’t do that, you hand back a reference to the cached item. This sorting function would execute against the object directly in memory, and replace it with a new, sorted version.

And suddenly the cached copy feeding your desktop website is handing back a list sorted by some random mobile visitor. This has happened to us a couple of times between our mobile and desktop web applications.

The way to resolve this is to create a copy of the item coming back from cache, so that you are not operating directly on the in-memory object:

List<State> states = CacheManager.Get<List<State>>(stateCacheCategory, stateCacheKey).ToList();
states.Sort((c, d) =>
   return GetDistance(c.Latitude, c.Longitude, localLat, localLong)
      .CompareTo(GetDistance(d.Latitude, d.Longitude, localLat, localLong));

The second scenario revolves around updating or adding items to the cache, and the appropriate locking semantics that must take place in a multi-threaded web world. If you don’t properly lock your cached objects, you run the risk of threadlocking during an update or add, sending your web application into a spiraling dance of death as threads lock and new ones are created simply to have those lock as well. This recently happened to us where a web server would suddenly create hundreds of threads and eventually crash, or create no new threads, but have the request count on the web server go through the roof waiting for threads to unlock.

Let’s take this code here. Let’s assume we’re storing in memory items in a dictionary of dictionaries.

public void AddToCache(string category, string key, object item)
   if (cacheDictionary == null)
      cacheDictionary = new Dictionary<string, Dictionary<string, object>>();

   cacheDictionary[category].Add(key, item);

Now we can assume there are also methods to get something from cache, and to delete something from cache. Under heavy load, we could assume that if a large item is being added to the cache dictionary, that other requests might also try to add this item, or try to get this item while the add is occurring. This would result in thread collisions and contention as requests tried to read from an object locked for writing without the appropriate locking semantics to held the threads know when to wait.

There are two ways to get around this. The first is to wrap your code with a generic object lock like this.

object _addLock = new object();
public void AddToCache(string category, string key, object item)
      if (cacheDictionary == null)
         cacheDictionary = new Dictionary<string, Dictionary<string, object>>();

      cacheDictionary[category].Add(key, item);

The second would be use an actual ReaderWriterLock. These types of locks have an advantage over the locking semantic above. The locking semantic above blocks all threads when the lock object is locked. A ReaderWriterLock allows data to be read by multiple threads at the same time, only blocking when a write is going to occur. So that would look like this:

private ReaderWriterLock _readerWriterLock = new ReaderWriterLock();
public void AddToCache(string category, string key, object item)
      if (cacheDictionary == null)
         cacheDictionary = new Dictionary<string, Dictionary<string, object>>();

      cacheDictionary[category].Add(key, item);
public object GetFromCache(string category, string key)
   if (cacheDictionary == null)
      cacheDictionary = new Dictionary<string, Dictionary<string, object>>();

   var item = cacheDictionary[category][key];
   return item;

This would allow for multiple threads to read the data while the add would lock for writing, causing the read threads to appropriately wait for the write to end. Without these locking semantics in place, you can put yourself in a poor position if your server ends up deadlocking on shared cached resources.