Lots of good comments...
Just to add some to the story - We were literally the only people he was waiting on because we were out of town and it was kind of an early time to eat dinner (around 5). I think 2 other tables were being served at the time.
Since we are out of town, I won't ever go back again.
I could have asked for change, but I just felt he made a dumb move.