It could a boost leak, or boost control. Start with the obvious. Tighten hoses, clamps whatever you have.
If the problem persists then move on to boost control, vacuum line, solenoids could be the issue. But they cost 200$ so you don;t want to swap them in before you are sure. If you know someone with an N54 you can swap them with him and see if the issue goes away.
SOLD --'07 E90 335i, PTF COBB E50, HFS4, ER CP, ETS 7", AA DP, Vishnu Exhaust, DCI, ACT, Nitto Nt555
SOLD '10 E92 335i M, 6MT COBB v3, ER IC, ER CP, VRSF DP, Fuel-it st2, Mfactory LSD