The “brains v. bot” 80,000-hand contest has ended. The four, very human players won precisely $732,713. Three of four ended ahead and one lost. Interestingly, while I and almost anyone looking at that bottom line, would claim victory for the brains, the folks at Carnegie Mellon University who wrote the software for Claudico (the AI — as noted before, everyone names their AIs) called it a “statistical tie.”
You can go here for more detail but the guts of Tuomas Sandholm’s argument is that since over $170 million was wagered this win represents less than a half of one percent ROI — a pretty pathetic win rate.
Sandholm heads the research group at CMU and surely knows his stats so he also realizes that when numbers get this big you do not need to be very far from the mid-point for a small difference to be statistically reliable. Since the game was played using the “duplicate” method which reduces the impact of the random turn of the cards, the theoretical variance around the zero-point will be smaller than if a truly random (Gaussian) distribution were being used for comparison.
I’ll leave it to others to work out the p-values here but my guess is that Sandholm is grasping at straws. A formal analysis will likely show that the brains won handily.
But, no matter. Claudico played remarkably well. No limit hold-em, even when played heads-up, is an astonishingly complex and difficult game. This is why so many of the top online pros focus on it. Small differences in skill get mapped into significant wins.
It was a lot of fun tapping into the online feed from time to time. I, and a lot of other poker players with intellectual interests and a fondness for the theoretical foundations of the game, will be looking for the next iteration of a poker ‘bot — which, I have no doubt, will have its software tweaked and play an even stronger game.