Your 4 GB is gonna fill up in a month if you do anything other than just save games. Unless you plan on only playing and getting downloads for one or two games, then you'll be wishing you got the 250 gig. Even my 20 GB original 360 HDD filled up after about a year, only downloaded a couple of map packs for Halo and CoD etc, plus some addons for Oblivion.. But, like you said if you're seriously only gonna play and download stuff for one game, it'll be fine. I thought I wouldn't need much space, but then new games and new map packs just kept coming out!
On the bright side, the new 360's come with built in wireless so you don't have to blow an extra 80 bucks for the wireless adapter like I did.