Turns out that my VM was running in what's apparently know as the "green turtle mode".However, the resizing process is damn slow. Resizing takes about 15 seconds (no joke!).
This is my first time working with VirtualBox. So far I worked with VMware Workstation and Hyper-V (and of course trusty bhyve on FreeBSD, but that doesn't apply here). I understand that those are proprietary solutions but I can't imagine that VirtualBox resizing with a FreeBSD guest would be that slow. What is the limiting factor here?
The VM was not using the underlying CPU-enabled virtualization technology (VT-x or AMD-v). This was caused by the fact that my Windows machine also had Hyper-V installed/enabled (Windows Feature).
After disabling Hyper-V, VirtualBox was able to get a hold of the CPU virtualization features and the VM ran much smoothly in general - especially with x11/xorg.
The resizing is also much faster in that case.