In 2012, the hacking team DefCon 949 disclosed their method to break Google’s reCaptcha. They used weaknesses in the version dedicated to visually impaired persons. End of 2014, Google replaced its letter-warping version with a user-friendlier version. It is based on the recognition of a set of images illustrating an object within a set of nine images.
At Black Hat Asia 2016, S. Sivakorn, J. Polakis and A. Keromytis from Columbia disclosed a method to break this visual captcha. They used many tools, but the core of the attack is the use of image annotation services, such as Google Reverse Image Search (GRIS) or Clarifai. These tools return a best guess description of the image, i.e., a list of potential tags. For instance, for the picture of a go-ban illustrating the blog post about AlphaGo, Clarifai returns chess, desktop, strategy, wood, balance, no person, table, and game, whereas GRIS returns go game. They use many tricks to increase the efficiency. My preferred one is to use GRIS to locate a high-resolution instance of each proposed challenge. They discovered that the accuracy of these annotation services decreased with the resolution of the submitted image.
They obtained a 70% accuracy for Google reCaptcha and 83.5% for Facebook’s version.
Sivakorn, Suphannee, Jason Polakis, and Angelos D. Keromytis, “I’m Not a Human: Breaking the Google reCaptcha” presented at Black Hat Asia, Singapore, 2016.