From what I know there is no such thing in VirtualBox, You need to specify how much memory each guest system may have, at least for VirtualBox 3.x, there was recently release VirtualBox 4.x so maybe there You will find something similar (memory ballooning).