I've been to Louisville multiple times. Things I enjoyed: the Seelbach hotel, Jazz, Louisville sports.
There jazz clubs on the east side of downtown, Triple A baseball, NCAA championship basketball and BCS Bowl Championship football all downtown. Maybe you just didn't know where to go.
Things I didn't enjoy: food (lines all day outside the Spaghetti Factory), the entertainment center (forget the name of the road, but it's outside the downtown core, and it's nothing but strip mall hell),
Bardstown Road is pretty long and there are some dodgy ends to it however, the section that is closest to downtown has dozens of nationally reviewed (and recommended) restaurants. Again, I don't know what you're used to - as you haven't said where it is that you live - but, 4th Street Live is happening 52 weekends a year and most weekdays during good weather.
and the taxi service (my lord, unreal, wait all night for a taxi, and when they show up, they are invariably lunatics).
You got me - there aren't a lot of taxis in Louisville. I guess that's due to most of us driving ourselves rather than relying on public transportation. Clearly you're right - I know that all the big travel mags list number of taxis as the most important criteria of a city's worth...
I do believe there is absolutely no reason to go to Ville for a visit unless you're there to see Ville sports (or, in my case, a conference at the university).