Windows 10 & Virtualbox’s “VERR_VRDP_PROTOCOL_ERROR” Error

I adopted Windows 10 on release day, wanting to be one of the first to get my arms around the retail copy of the new OS. I was excited to play with Windows 10 before my friends, partially so I would be available in case they wanted help with their upgrades.

I’m glad I installed Windows 10 right away, since the OS has been wonderful for me so far. Cortana works very well for my limited usage (though I’d understand if others decide to opt-out, depending on their use cases). Meanwhile, the UI design changes are really positive for my desktop usage, and I’ve noticed almost zero compatibility issues.

The big issue that came up this weekend happened when I was trying to run some updates on my Virtualbox-hosted servers. I run many small virtual servers on a headless install of Virtualbox in Debian Linux, and this setup has worked very well for me for quite a while. I can configure Virtualbox to use VRDP to allow me to connect to the machines, as if I were using IPMI or an IP KVM, and administer everything from the comfort of my main home workstation.

virtualbox error verr_vrdp_protocol_error

Unfortunately, Windows 10’s version of the Terminal Services Client (mstsc.exe) requires negotiation in a specific way that Virtualbox does not support at this time, and there isn’t a way I can find to change it. You can, if you’re so inclined, switch the VRDE module from RDP to VNC (see the Virtualbox manual on how to do that).

Instead, in the hopes that Virtualbox would be fixed shortly thereafter to support the new protocol version (0x00080005), I decided to try to find a better way from my workstation to connect to my servers in the short term.

Here’s what I did: 

The big problem with getting RDP clients working on Windows is that most vendors will interface directly with the OS-provided RDP library that ships with the default Windows install. This means almost every single “3rd party” RDP software available for Windows will exhibit the same core problem with the protocol mismatch as mstsc.exe itself.

Instead, I went to FreeRDP, a full-stack open source implementation of the RDP protocol. To my understanding, it’s not entirely feature-complete to the extent that the Microsoft client is, but it will certainly work for my purposes. Cloudbase even has nightly builds of both the development and stable branches available for download. This client uses its own RDP library, and successfully connects to Virtualbox VRDP displays. The FreeRDP client even uses the same execution syntax as mstsc.exe, so for users that have their RDP settings in shortcuts, it’s just a matter of pointing the shortcut at the new executable.

If you have any thoughts on this solution, or alternate ideas, please comment below! 

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s