The problem was really annoying when I need to ssh to some Linux servers from the Ubuntu virtualbox guest. As I remember, it started after I updated the VirtualBox platform to 4.3.26 or updated the guest from 12.04 to 14.04. The fix was surprisingly simple. See https://forums.virtualbox.org/viewtopic.php?f=6&t=66086#p313103
With the default Intel adapter, ssh connections were always dropped. When switching to the first non-Intel adapter, the issue was gone.