overview-of-python-web-scraping-tools-18-728

Introduction to Twill – A Simple tool for automated testing of Web Automations

Posted on Posted in Open Source, Software Testing, Test Automation

Twill is an open source library that allows the user to browse the Web from standard command line interface or Python scripts. This library is written in Python and can be extremely useful for test automation for web applications. Unlike many test automation tools for web application testing, Twill does not launch or need a browser. Twill works from the command line (or from your Python script) and allows you to perform standard operations like navigating to specific pages, use forms, cookies and so on.

As mentioned earlier, Twill does not launch a real browser and hence its usage as test automation tool is limited. Twill can not be used to test rendering of pages, functionality related to Java scripts and so on. Tools like Selenium are better suited for that purpose. Twill is better suited for validating functionality which can be exercised using the HTTP request/response and static analysis.

After installing Twill (Can be downloaded from http://darcs.idyll.org/~t/projects/twill-0.9.tar.gz), it can be used from scripts as well as command line. Twill shell can be launched from the command line using the twill-sh command. If your installation is right and Twill is present in your path, twill-sh will open Twill in interactive mode and show following information in the command line –

-------------------------------------

-= Welcome to Twill! =-

current page: *empty page*

>>

————————————-

‘>>’ shows that Twill is ready and waiting for the commands. Commands in Twill are very simple and are self-explanatory. Typical commands in Twill might look on the lines of –

go http://www.TestingGeek.com (This will take you to the TestingGeek home page)

showforms (This will show all the forms present on the current page)

Even in interactive mode, Twill offers many commands which are extremely useful for web testing. There are commands for browsing websites (go, back, reload, follow), assertions (code, find, notfind, url, title ), display (showlinks, showforms, save_html …), forms (submit, formvalue, formclear …), cookies (show_cookies, clear_cookies, load_cookies …), debugging, other useful commands like agen, reset_browser and so on. These commands are explained nicely at this page.

Using Twill in interactive mode is very useful for quick tests, but its usage is still limited. To realize the power of Twill and see how useful it can be, you should explore its usage from Python script. Twill, when used with other Python libraries like BeautifulSoup (Converts raw HTML source into DOM tree like structure) can become an extremely useful tool and can be used for testing specific patterns for many web applications.

When using Twill from Python, you can either import all the twill commands using ‘from twill.commands import *’ or get a browser object using ‘from twill import get_browser’. Irrespective of how you use it, all the commands are still available in both the modes.

Following code snippet will show various commands and how they are being used from Python scripts –

-----------------------------------------------------

# Import all the commands from Twill

from twill.commands import*
go(url) # Go to a specifc URL

try:
code(200) # Assert whether status code for this URL is 200 or not.

save_html(“temp.html”) # Save raw HTML of the page in this file

link_list = list(showlinks()) # Get all the links present in this page

except:
pass # What should be done if status code is not 200

—————————————————–

In next article, I will share how Twill and BeautifulSoup libraries can be used together in Python scripts to create a small crawler for testing specific patterns in the web applications.

Please follow and like us: