There isn't a hard number.
I bet it's a rough ride in your car without five people. The springs are going to be compressed even more = less available damper travel, so you'll be constantly rebounding off the bumpstops. It's more than unpleasant. I agree it's probably not a good idea to take five people in your car.
2011 BMW 335i E92 - Le Mans Blue, 6MT, M Sport. JRZ RS Pro, Quaife LSD, BMW Perf Exhaust