I was getting the same error but with the vSphere client installed on a recent OS version (i.e. not XP or Windows Server 2003).
Same scenario: getting the error only when sorting with more than 38 VMs in the view.
This was occuring, in my case, because of a DNS issue. I could not resolve the myvcenter.mydomain.com server so simply putting an entry in the HOSTS file resolved the issue.