Oh I completely agree that as an experience a nice crisp sunny day makes it far far nicer.
Hurtling down Hangar straight trying to spot a braking point through a wall of spray from that talented git in the Nissan Micra is not everyone's idea of a fun day out. But it does learn you car control which is actually much more relevant to everyday driving
I would also argue that it is, somewhat counter-intuitively, actually safer in the wet. There's nothing like bolting on a set of cut slicks in the dry to make you acutely aware that the huge corner speeds you can achieve will result in a veritable party 7 of bad news if it goes wrong. Driving to wet conditions means that if it goes wahoonie shaped you are travelling somewhat slower giving you more time to run through you options (before shutting your eyes and screaming
).
Plus incidents generally start much more slowly giving you more chance of catching it
I'm certainly not trying to say you should be out doing a rain dance before every track day but if it is inclement don't see it as the end of any possible enjoyment from the day