GUI Automation Using Python

Python is popularly known as the language of the future. It is a powerful, high level language which can perform complex tasks with few lines of code.

In this article, I will show how to automate movements of mouse using pyautogui module in python. It does not come preloaded with python. So to install it, open cmd (type cmd in windows search and click first result). It shows C:/Users/yourname as the current directory. You need to change the current directory to the ‘Scripts’ folder at the place where you have installed python (something like C:/Python/Scripts in my PC). To change the current directory, type the following in cmd:


cd C:/Python/Scripts

Now to install pyautogui, type the following in cmd:


pip install pyautogui

Wait until the package is installed. Now it can be used to write python code.

Open a text editor and type:


import pyautogui

to import all its functions.

Controlling mouse movements using pyautogui:

Python tracks and controls mouse using coordinate system of screen. Suppose the resolution of your screen is 1023×767, then your screen’s coordinate system looks like :

to know your screen resolution, type the following in a text editor:


import pyautogui

print(pyautogui.size())

Save this file with .py extension, and then run the file.

This python code use size() function to output your screen resolution in x,y format:

output: (1023, 768)

To move the mouse, use function moveTo() in pyautogui module.  Type:


import pyautogui
pyautogui.moveTo(100, 100, duration= 1)

This code uses moveTo() function, which takes x and y coordinates, and an optional duration argument. This function moves your mouse pointer from it’s current location to x,y coordinate, and takes time as specified by duration argument to do so.

Save and run this python script to see your mouse pointer magically moving from its current location to coordinates (100, 100) , taking 1 second in this process.

Using moveRel() function:

moveRel() function moves the mouse pointer relative to its previous position. Type:


import pyautogui
pyautogui.moveRel(0, 50, duration=1)

This code will move mouse pointer at (0, 50) relative to its original position. For example, if mouse position before running the code was (1000, 1000), then this code will move the pointer to coordinates (1000, 1050) in duration 1 second.

Getting current mouse position:

use position() function to get current position of the mouse pointer. Type:


import pyautogui
print(pyautogui.position())

output: (500, 600)

or wherever your mouse was residing at the time of executing the program.

Clicking and dragging the mouse:

type:


import pyautogui
pyautogui.click(100, 100)

This code performs a typical mouse click at the location (100, 100).

We have two functions associated with drag operation of mouse, dragTo and dragRel. they perform similar to moveTo and moveRel functions, except they hold the left mouse button while moving, thus initiating a drag. This functionality can be used at various places, like moving a dialog box, or drawing something automatically using pencil tool in MS Paint. For example to draw a square in paint, type the following code:


import time #a module which has functions related to time. It can be installed using cmd command: pip install time, in the same way as pyautogui.
import pyautogui
time.sleep(10)   //makes program execution pause for 10 sec
pyautogui.moveTo(1000, 1000, duration=1) //moves mouse to 1000, 1000.
pyautogui.dragRel(100, 0, duration=1)/*drags mouse 100, 0 relative to its previous position, thus dragging it to 1100, 1000*/
pyautogui.dragRel(0, 100, duration=1)
pyautogui.dragRel(-100, 0, duration=1)
pyautogui.dragRel(0, -100, duration=1)

before running the code, open MS paint in background with pencil tool selected. Now run the code, quickly switch to MS paint before 10 seconds (since we have given 10 second pause time using sleep() function before running the program).

After 10 seconds, you will see a square being drawn in MS paint, with its top left edge at 1000,1000 and edge length 100 pixels.

Using scroll() function:

scroll function takes no. of pixels as argument, and scrolls the screen up to given number of pixels.


import pyautogui
pyautogui.scroll(200)

This code scrolls the active screen up to 200 pixels.

Controlling the keyboard:

Automate typing of a string:

You can automate typing of string by using typewrite() function. just pass the string which you want to type as argument of this function. Ex:


import pyautogui
pyautogui.click(100, 100)
pyautogui.typewrite(“hello Geeks!”)

suppose a text field was present at coordinates 100, 100 on screen, then this code will click the text field to make it active and type “hello Geeks!” in it.

Passing key names:

You can pass key names separately through typewrite() function. For example:


import pyautogui
pyautogui.typewrite([“a”, “left”, “ctrlleft”])

This code is automatic equivalent of typing “a”, pressing left arrow key, and pressing left control key.

Pressing hotkey combinations:

Use hotkey() function to press combination of keys like ctrl-c, ctrl-a etc. Ex:


import pyautogui
pyautogui.hotkey(“ctrlleft”, “a”)

This code is automatic equivalent of pressing left ctrl and “a”simultaneously. Thus in windows, this will result in selection of all text present on screen.

Hence we finish with the basic introduction of GUI automation using python. If you want to read more about it, read the Pyautogui documentation here. For more cool articles, stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *