Have you though about using two sets of rings, one on front of speaker and one on rear of door panel, that sandwiches the door panel and speaker mounting ring?
I guess I'm layering it like this from outside of panel to inside of door:
speaker grill --- adapter ring --- speaker --- door panel --- adapter ring
Then you would use a nut and bolt to sandwich the speaker and door panel between the two rings. No glue needed.