The waiter should have learned a lesson. I don't think you made a mistake. He did not go out of his way to ask you nor did he go out of his way to give you smaller bills so why should you go out of your way to give him full credit and a full 20%+ tip when he made no effort on his part?
The waiter has a job of which he gets paid for by his employer and that is to service his customers. For extra service he may get a tip.