There is MUCH more apparent return on investment from better speakers, than there is from an OK amp to a good amp.
However, I find that people who don't think of themselves as experts often hear differences better than those who do. I ascribe it to the phenomenon Malcolm Gladwell describes in Blink when talking about taste tests. When our brains are trying to deconstruct and dissect an experience - taste or sound to me are interchangeable on this - we do much better when we AREN'T thinking about the why of it, and when we have no expectation of being an expert at it. Just how it is. We can determine what's enjoyable pretty easily. You'd be amazed at how many customer's wives, who had firm beliefs that they didn't hear differences, noticed differences better than their husbands.