If you create a bad connection it will try to connect indefinitely, there is no option to cancel connection.  If you try to remove while it's connecting it won't let you because it is currently connecting.  The only way to close it is to close VS Code.