1, Introduction to PyAutoGui

PyAutoGUI is a powerful UI automation library of Python. Its purpose is to automatically control mouse and keyboard operations with programs. It is mainly used to realize UI automation on the PC side.

① it has the functions of mouse control, keyboard operation, screenshot, picture positioning, message dialog box, window operation, etc;  
② there are practical tools such as countdown, mouse coordinate color capture applet, UI automation Mini language and so on;  
③ there are fail safe, general pause and other mechanisms.  

2, PyAutoGui installation

cmd install command:

pip install PyAutoGUI

If the python or pip installation fails, the following is a solution:

        1. Access website: PyAutoGUI · PyPI

        2. Download PyAutoGui with the suffix tar GZ compressed package

        3. Output cmd on the decompression directory and press enter

        4. Enter the command: Python setup py install

3, Four solutions to unstable image location

1. Scheme introduction

        1. Fuzzy positioning: with the help of opencv to improve the recognition rate, the confidence parameter is added to the local function, that is, the recognition accuracy. When the confidence is smaller, the positioning accuracy will be lower, so as to realize fuzzy positioning.

        2. Gray matching: add the grayscale parameter to the local function. When grayscale=True, the color in the image and screenshot will be desaturated, which can solve the problem of image positioning failure caused by subtle color differences caused by different saturation of the display.

        3. Specify range: adding the region parameter to the local function can control the map finding range, so as to improve the map finding efficiency.

        4. Multi image positioning: icon s may have different display effects in different scenes. Multiple images with different display effects can be classified as one event, and multiple images can be searched circularly. Locating one image can locate the whole event.

2. Fuzzy query

opencv installation command:

pip install opencv-python

In pyautogui The confidence parameter is added to the locateonscreen() function. When the value of confidence determines the accuracy

>>>t = locateOnScreen('images/test1.png', confidence=0.9)
>>>print(center(t))

Point(x=873, y=1056)

3. Gray matching

In pyautogui grayscale=True is added to the locateonscreen() function to realize grayscale matching

>>>t = locateOnScreen('images/test1.png', grayscale=True)
>>>print(center(t))

Point(x=873, y=1056)

4. Specified range:

Adding the region parameter to the local function can control the map finding range, so as to improve the map finding efficiency.

region(x,y,width,height), where x and y are the coordinates of the upper left corner of the range, and width and height are the width and height of the range

>>>t = locateOnScreen('images/test1.png', region=(0,940,1919,939))
>>>print(center(t))

Point(x=873, y=1056)

5. Multi map positioning

You can cut multiple different graphs for an icon. Each graph represents the icon. You can locate the icon as long as you locate one of the graphs.

The locateonscreen() function is encapsulated twice, and multiple pictures are separated by '|' to realize circular map finding

from pyautogui import *

#Cut the string by '|'
def word_cut(args):
    tup = []
    if '|' in args:
        re1 = args.split('|')
        return re1
    else:
        tup.append(args)
        return tuple(tup)

#Judge whether the image is found. If it is found, it returns True. If it is not found, it skips
def assertPIC(args):
    if locateOnScreen(args) == None:
        pass
    else:
        return True

#Cycle to find the map, return to the image center point if found, and print not found if not found
def img_locat(args):
    arg = word_cut(args)
    for i in range(len(arg)):
        if assertPIC(arg[i]):
            return center(locateOnScreen(arg[i]))
        else:
            print('not found')

#test
print(img_locat('images/test.png|images/test1.png'))

The result is: