Ask Your Question

iOS game "Letterpress" cheat - Crop, ColorDistance, binarize, OCR, and more

asked 2013-03-24 18:25:12 -0500

JHawkins gravatar image

Call me ambitious, or foolish, but I thought a good way to learn Python and SimpleCV would be to tackle a project end-to-end and learn as I go. I am very new to programming in general, and I have tried to make as much progress as I could without asking questions that are in the FAQ. :)

Here is my project: Letterpress Cheater

  1. [√] user obtains screenshot from iOS game Letterpress (currently building around iPad-sized screenshot)
  2. [√] use SimpleCV to crop out the game board
  3. [√] use SimpleCV to identify which letters have one of 5 backgrounds; dark and light red, gray, and dark and light blue
  4. [√] return the letters with dark blue backgrounds, light blue backgrounds, gray... you get this idea
  5. [currently on this step] perform OCR on the letters
  6. [ ] run the entire collection of letters through dictionary...
  7. [ ] ... giving preference to letter in the light red area to steal from opponent
  8. [ ] ... or to the gray areas to capture unused letters
  9. [ ] ... or in the light blue areas to expand your collection of "locked" letters, which are dark blue

That's the whole concept, and it's what I started coding this morning.

My primary question: am I going about this the right way? Am I missing something obvious?

Here is how far I have gotten:

import tesseract

## name of Letterpress screenshot to use
screenshot = Image("/Users/jesse/Desktop/IMG_0095.PNG")

## switch between different board sizes here
board_big = screenshot.crop(198,708,1140,1140)
board_small = board_big.scale(width=int(board_big.width*.50), height=int(board_big.height*.50))
board = board_small

## dictionary of Letterpress color names and cooresponding variables
letterpressDict = { \
"darkBlue" :["darkBlue" , "board_darkBlue" , (0  , 162, 255), "board_darkBlue_bin" ], \
"lightBlue":["lightBlue", "board_lightBlue", (120, 200, 245), "board_lightBlue_bin"], \
"gray"     :["gray"     , "board_gray"     , (232, 231, 228), "board_gray_bin"     ], \
"darkRed"  :["darkRed"  , "board_darkRed"  , (255, 67 , 47 ), "board_darkRed_bin"  ], \
"lightRed" :["lightRed" , "board_lightRed" , (247, 153, 141), "board_lightRed_bin" ] \

## stepping through dictionary, identify the letters that match the specificed color
for colorName, theList in letterpressDict.items():
    theList[1] = board.colorDistance(theList[2])
    theList[3] = theList[1].binarize(50)

    theList[3].save("/Users/jesse/Desktop/" + theList[0] + ".png")

    ## theList[3].readText()

print "done"

The readText() line that I have commented out vexes me - it crashes Python Notebook when I run it.

Getting to this point, I tried a much more cumbersome method where I would use tesseract from the command line via a call to os.system(). I would pass the PNGs I made via SimpleCV to tesseract using -psm 10. The results were 70%-80% accurate, which I think I could improve with training on the font used in Letterpress. I stopped pursuing this path when I was finally able to access tesseract from within Python (OS X Mountain Lion installs are tricky!) and skip all the file creation mess... I think.

I've attached an image that shows the results I have been able to obtain thus far. Everything is scaled down to save bandwidth. The individual squares that contain each letter are 228 px x 228 px in the full size image, and are very sharp ... (more)

edit retag flag offensive close merge delete

2 Answers

Sort by » oldest newest most voted

answered 2013-03-25 09:41:39 -0500

kscottz gravatar image

This is a super cool project. I am not super familiar with the game you mention so please forgive me if I make some incorrect assumptions.

I would like to know a little bit more about the error you are getting with tesseract. The pytesseract bindings are a bit wonky and I think what we are doing is saving your image as a temp file, calling out to the command line tesseract reader, and then returning the results. This might be overkill for your application. Tesseract doesn't always return the best results either. More often than not you are going to get extraneous white space characters back or tesseract will match your character with some character with a dipthong or some other mark.

Using tesseract to match single characters is overkill in my opinion. What I would do to solve your character matching problem is to create a custom dictionary of templates for each character you want to find. What you would then do is iterate over the dictionary and calculate some heuristic of the match quality of an input character to your dictionary characters. A brain dead simple heuristic is to binarize and invert your input and the dictionary characters (so they are white characters on a black background), perform some sort of match operation (subtract or multiply would work well), and then calculate the mean color. The higher or lower the mean color value the better the match. It is worth noting that your templates and test images will need to be the same size, you probably could scale them down significantly to speed things up.

Otherwise your image processing pipeline appears to be solid. Please keep us posted on your progress. We would love to see the finished results. Also we are going to lower the karma needed to post links so that problem should be resolved shortly.

edit flag offensive delete link more

answered 2013-03-25 09:57:41 -0500

JHawkins gravatar image

updated 2013-04-02 16:45:28 -0500

\/\/\/\/\/\/ Update #4 \/\/\/\/\/\/

Huge progress!

I am now able to detect the letter and the background color of each of the 25 tiles. The character recognition is not perfect, however. It appears that the problem is related to Letterpress. Some letter appear differently in different screenshots. I'll continue to work on this.

If you have comments, please share them!

I am also learning Git, so the project has moved to

import os
import glob
from collections import defaultdict # remove if the color grouping doesn't work
from SimpleCV import *

## 1. load screenshot and break it into individual tiles

## name of Letterpress screenshot to use
screenshot = Image("./screenshots/3.PNG")

## crop to the game board
board = screenshot.crop(198,708,1140,1140)
## have a look at the boards, if you want

## setup an empty dictionary to hold the names and crop location of all 25 tiles as well as variables to store crop location
tileMaker = {}
locx = 0
locy = 0

## populate the dictionary with a unique name for each tile (tile_r#c#) and it's cooresponding crop location (each cell is 228 px square)
for foo in range(1,6):
    for bar in range(1,6):
        tileMaker["tile_r" + str(foo) + "c" + str(bar)] = "board.crop(" + str(locx) + "," + str(locy) + ",228,228)"
        locx += 228
    locx = 0
    locy += 228
## have a look at the dictionary, if you want
#print tiles

## create and save the tiles
for key, value in tileMaker.iteritems():
    exec(str(key) + " = " + str(value))
    exec(str(key) + ".save(\"./myLetters/" + str(key) + ".png\")")
    ## have a look at the tiles, if you want
    #exec(str(key) + ".show()")

## 2. detect the background color and letter of each tile

## dictionary of Letterpress RGB color values  ## NOTE: there are two grays!
lpColorValues = { \
"dark blue" : "(0.0, 162.0, 255.0)", \
"light blue" : "(120.0, 200.0, 245.0)", \
"gray_1" : "(233.0, 232.0, 229.0)", \
"gray_2" : "(230.0, 229.0, 226.0)", \
"light red" : "(247.0, 153.0, 141.0)", \
"dark red" : "(255.0, 67.0, 47.0)"}

## create dictionary of myLetters by glob'ing the directory
myLetters = dict()
myLettersPath = "./myLetters/" ## update this as needed to match your system
extension = "*.png"
directory = os.path.join(myLettersPath, extension)
myFiles = glob.glob(directory)

## get the background color of each tile
for file in myFiles:
    ## get the color of a pixel a few in and down from the top left corner
    color = Image(file).getPixel(10,10)
    ## loop through the Letterpress RGB color values to find a match
    for eachColor in lpColorValues:
        if str(color) == str(lpColorValues[eachColor]):
            color = eachColor
    newMyImg = Image(file)
    ## add color data to myLetters dictionary

## create dictionary of testLetters by glob'ing the directory
testLetters = dict()
testLettersPath = "./testLetters/" ## update this as needed to match your system
extension = "*.png"
directory = os.path.join(testLettersPath, extension)
testFiles = glob.glob(directory)

## get the letter shown in each testLetter tile from the filename
for file in testFiles:
    cut = file.find(".png")
    letterName = file[cut-1:cut]
    newTestImg = Image(file)
    ## add ...
edit flag offensive delete link more



This looks about right. You will want to test your dictionary to make sure it works well. I suggest you look at the confusion matrix class in the MachineLearning subdirectory. It will help you calculate stats on your matches.

kscottz gravatar imagekscottz ( 2013-03-25 10:15:17 -0500 )edit

Will do. Once this project is complete, I'd like to write a blog post about it and provide sample images for other people that are trying to learn along the path. -- FYI, I do not see a comment button below your original answer, otherwise I would have posted there. Thanks again!

JHawkins gravatar imageJHawkins ( 2013-03-25 10:25:59 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2013-03-24 18:25:12 -0500

Seen: 990 times

Last updated: Apr 02 '13