There isn't a hard number.
I bet it's a rough ride in your car without five people. The springs are going to be compressed even more = less available damper travel, so you'll be constantly rebounding off the bumpstops. It's more than unpleasant. I agree it's probably not a good idea to take five people in your car.
2011 BMW E92 335i - Le Mans Blue/Black, 6MT, M Sport package