I agree, the 335i is a completely different machine. Test drive it and you will see the difference. Sure it is very pricey but the car is worth every penny in my book.
However, from my point of view, the question is whether you can drive the car to the "full" ie can you use all that performance that it can offer. Don't get me wrong on this, the M3s and 911s of this world are great to drive but in terms of entertainment value, you would really enjoy getting the most out of your car.
Although I may sound bias, there is a difference between 323 and 325, the question is whether you think its worth that extra 100k. I test drove both of the cars and after much struggle, plunge for the 325. It is a very personal choice, that's why I would recommend you to test drive before putting down the hammer. Your car is also relatively new, so I would recommend to keep it but see if there are any after-market mods that can freshen up the car a bit for your enterainment.