There isn't a hard number.
I bet it's a rough ride in your car without five people. The springs are going to be compressed even more = less available damper travel, so you'll be constantly rebounding off the bumpstops. It's more than unpleasant. I agree it's probably not a good idea to take five people in your car.
2011 BMW E92 335i, 6MT, Le Mans blue - Quaife LSD, JRZ RS Pro, JRZ camber plates, Eibach ERS, full M3 susp, custom toe arms, Powergrid end-links, SS brake lines, brake ducting, Michelin PSS, UUC SSK, ETS FMIC, ER CP, Perf. Exhaust, VAC RSIK, Sparcos; work by VAC Motorsports in Philadelphia