It's pretty easy to guesstimate it. "Build" the front of the box out of some cardboard, then use packing peanuts to fill the void between the front of the box and the trunk liner. Once that area is full (don't compress it, just dump it in there), transfer the peanuts to a cardboard box of known dimensions. Make sure the peanuts are level in the box, then measure the approximate volume that the peanuts take up in the box.
Some people prefer to measure the peanuts out first, then fill, but I think that's a backwards way of going. I want to know how much space I have, then pick a subwoofer for that space - not the other way around.