Python Friday #144: Automate Browsers With Selenium (Part 1)

With Selenium ready and a solution in place to download drivers with ease, we now can start with the interesting part and automate our browser.

This post is part of my journey to learn Python. You can find the other parts of this series here. You find the code for this post in my PythonFriday repository on GitHub.

 

Preparation

Make sure that you have Selenium installed on your machine and that you use the download manager to get the right drivers.

 

Navigate to a site

We can tell Selenium to navigate to a web site with the get() method:

When the browser got the page we requested, we can start to interact with it and ask for its title:

This gives us the title of the page and we can make sure that we are at the right place:

DuckDuckGo — Privacy, simplified.

 

Find elements

We can find all elements of a page with a combination of the right selectors. First, we need to import the By module to get access to the selectors:

This import allows us to search elements by different approaches:

• By.ID – search for the id attribute
• By.TAG_NAME – search for the HTML tag name of an element
• By.CLASS_NAME – search for the class name of an element
• By.LINK_TEXT – search using the exact link text
• By.PARTIAL_LINK_TEXT – search using parts of the link text
• By.NAME – search in the name attribute of an element
• By.XPATH – search with an XPath expression
• By.CSS_SELECTOR – search with the CSS selector syntax

To get all elements with an H1 tag, we can use the method find_elements() method to find them and show the text between the H1 tags with this code:

Unfortunately for us, there are plenty of H1 in the site and most have no text:

H1: []
H1: []
H1: []
H1: [Tired of being tracked online? We can help.]
H1: [We don’t store your
personal information. Ever.]
H1: []
H1: []
H1: []
H1: []

If we know that there is only ever one element or we are just interested in the first one, we can use the find_element() method (singular):

Unlike to C#, the find_element() method does not throw an exception if it finds multiple matching elements.

 

Fill input forms

We can use the selectors to get an input form element and write text to it with the send_keys() method:

We can see the filled-out search form in the Browser:

To fill out a whole form, we need to repeat this for each field.

 

Click buttons

To submit a form, we need to find the submit button and click it:

 

Next

This covers the most important parts to automate your browser. As soon as you try that on a real web page, you will notice that this is not quite enough to tackle all edge cases. Thus, next week we look at some ways to mitigate the most common obstacles.