Test Automation DataDriven Framework Using Selenium Webdriver and TestNG

Tool Automation Comparison

HTML Report for Test automation using Selenium and Ruby

I had to start test automation for my project using any open source tool. After a series of analysis and comparison, we decided to go with Selenium(Selenium RC) and Ruby... (...selection of 'Ruby' is due to the obvious reasons that, I am more familiar in Ruby ;) ).

As usual, the first thing that hit when we start automation with an open source tool, is the kind of result/report tool provides after test run. And Selenium RC (with Ruby script), didn't generate any test result/report by default. And I will have to go with Rspec reporting. But instead of digging much into Rspec, I have written an HTML logger class (using the default logger class of 'Watir' as a base).

As I have not used Rspec reporting before, I can neither suggest you which is better nor do a comparison... But this solved my purpose, and I just thought of sharing what I do...

Okay enough with story... Here I am with the HTML logger class : "Logger_class.rb", written in Ruby for generating result/report of automation testing. This can be used to generate test result for 'Selenium' with Ruby (yea.., of course this would works for 'Watir' as well).

###----------------------------------------###
# --- HTML Logger Class --- #
# Purpose:
# To log the test results of automated tests into an HTML report
#-----------------------------------------#

class HTMLLogger

def initialize(test_result_path)

time = Time.now.strftime('%m_%d_%Y_%H_%M')
@logfile = File.new(File.join(test_result_path, "/Test_Result_#{time}.html"), "w")
@logfile.puts "<html>"
@logfile.puts " <title>Logon2 Automation Test Results</title>"
@logfile.puts " <head><style type='text/css'>"
@logfile.puts " body{margin: 4; padding: 0;}"
@logfile.puts " h2{font-family:Verdana, Arial, Helvetica, sans-serif; font-size:16px; font-weight:bold; color:#333333}"
@logfile.puts " p{font-family:Verdana, Arial, Helvetica, sans-serif; font-size:11px; font-weight:bold; color:#333333}"
@logfile.puts " .tr_summary{border-width: 1px; border-style: solid; border-color: #163B69; background-color: #163B69; color: white; font-size: 12px; font-family: Verdana; text-decoration: none; font-weight: bold; }"
@logfile.puts " th {border-width: 1px; border-style: solid; border-color: #163B69; background-color: #163B69; color: white; font-size: 12px; font-family: Verdana; text-decoration: none; font-weight: bold; }"
@logfile.puts " .tr_pass {border-width: 1px; border-style: solid; border-color: #163B69; color: #666666; font-size: 11px; font-family: Verdana; text-decoration: none; font-weight: normal; }"
@logfile.puts " .tr_fail {background-color:#FFFF99; border-width: 1px; border-style: solid; border-color: #163B69; color: #FF0000; font-size: 11px; font-family: Verdana; text-decoration: none; font-weight:bold; }"
@logfile.puts " </style></head>"
@logfile.puts " <body>"
@logfile.puts " </table><h2>Test suite results</h2>"
@logfile.puts " <table id='suiteTable' border='1' cellpadding='1' cellspacing='0' width='100%'>"
@logfile.puts " <th>Sl.No</th><th>TC.ID</th><th>Test Case Name</th><th>Result</th><th>Screen Shot</th><th>Comments</th>"
$tc_sl_no = 0

end

def log_result(tc_id, name, test_status, screen, notes)

$tc_sl_no = $tc_sl_no+1
if test_status == "Pass"
@logfile.puts "<tr class='tr_pass'>"
@logfile.puts "<td>#{$tc_sl_no}</td>"
@logfile.puts "<td>#{tc_id}</td>"
@logfile.puts "<td>#{name}</td>"
@logfile.puts "<td>#{test_status}</td>"
@logfile.puts "<td>#{screen}</td>"
@logfile.puts "<td>#{notes}</td>"
else
@logfile.puts "<tr class='tr_fail'>"
@logfile.puts "<td>#{$tc_sl_no}</td>"
@logfile.puts "<td>#{tc_id}</td>"
@logfile.puts "<td>#{name}</td>"
@logfile.puts "<td>#{test_status}</td>"
@logfile.puts "<td>#{screen}</td>"
@logfile.puts "<td>#{notes}</td>"
end

end

def end_log(tc_total, tc_executed, tc_pass, tc_fail, tc_notrun)

@logfile.puts "</table>"
@logfile.puts "<table>"
@logfile.puts " <h2>Test Summary</h2>"
@logfile.puts " <table id='summary' border='0' cellpadding='1' cellspacing='2' width='25%'>"
@logfile.puts " <tr class='tr_summary'><td><strong>Total No. of Test Cases in Suite</strong></td><td>#{tc_total}</td></tr>"
@logfile.puts " <tr><td></td><td></td><td></td></tr>"
@logfile.puts " <tr class='tr_summary'><td><strong>Number of Test Cases Executed</strong></td><td>#{tc_executed}</td></tr>"
@logfile.puts " <tr><td></td><td></td><td></td></tr>"
@logfile.puts " <tr class='tr_summary'><td><strong>Number of Test Cases Passed</strong></td><td>#{tc_pass}</td></tr>"
@logfile.puts " <tr><td></td><td></td><td></td></tr>"
@logfile.puts " <tr class='tr_summary'><td><strong>Number of Test Cases Failed</strong></td><td>#{tc_fail}</td></tr>"
@logfile.puts " <tr><td></td><td></td><td></td></tr>"
@logfile.puts " <tr class='tr_summary'><td><strong>Number of Test Cases Not Run</strong></td><td>#{tc_notrun}</td></tr>"
@logfile.puts" </table>"
@logfile.puts"<p> </p></body>"
@logfile.puts"</html>>"
@logfile.close

end

end
###----------------------------------------###

How to Use:

1. Download the file "Logger_class.rb" from the link given below in Sample Script section (or)
Copy the Logger_class script from above and save as .rb file("Logger_class.rb").

2. Paste the "Logger_class.rb" somewhere into your test script directory.

3. Require this "Logger_class.rb" in your suit file and initialize/create a HTML result file for your test. All this could be done, just adding two lines of code in your main test script rb file (refer the sample test script in the link).

require "{your directory path}/Logger_class' #Example : require 'C:/Test scripts/Logger_class'
$logger = HTMLLogger.new(File.expand_path(File.dirname(__FILE__))) #filePrefix_name

4. Function call to log the result at the end of each test script. Add the below command at the end of each test case script.
$logger.log_result($tc_id, $tc_name, $tc_status, screenshot_name, actual_result)
NOTE : If you are using setup-teardown, you can add this part in your tear down method.

5. Function call to add test summary to your html logger. Add the below line at the end of your main test script file or as a last test in your suit.
$logger.end_log($tc_total, tc_executed, $tc_pass, $tc_fail, $tc_notrun)

Sample Selenium Script:

Logger class can be downloaded from : http://docs.google.com/Doc?id=d7wdxgx_9dzpjrddn

A Sample test script, which uses logger class for report generation can be found here : http://docs.google.com/Doc?id=d7wdxgx_10f9ktx9gk

NOTE : Both files are saved as txt files, download and save with .rb extension

Sample Result:

And here, you can find a sample HTML result generated...

http://docs.google.com/Doc?id=d7wdxgx_8r2twnfmb

See if it helps you..!

All the best..!

Issues and resolutions with Watir 1.6.2

Issue 1 :

If we have 'require 'watir/contrib/enabled_popup'" in test script, then the below error would be shown.

c:/ruby/lib/ruby/gems/1.8/gems/watir-1.6.2/lib/watir/contrib/ enabled_popup.rb:5: uninitialized constant Watir::PageContainer::Win32 (NameError) from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'

Resolution from watir Development Team :

require 'watir/ie' after 'require watir'. Watir team has added the require statement to the enabled popup script itself, so this workaround will not be required with the next version of watir

Issue 2:

If we have 'include Watir' in your test script the below error may be shown when running test.
c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.2/lib/active_support/core_ext/module/introspection.rb:57:in `const_get': no such file to load -- safariwatir (MissingSourceFile)

Resolution from watir Development Team :

Remove 'include Watir' from their scripts. Almost every one who use 'include watir'does so out of confusion. They think that it is like "include" in Perl or "import" in Java, but it isn't. Specifically, Java's import is a naming convenience that affects the current file. Ruby's include is a naming conventience that affects the current file AND EVERY OTHER RUBY FILE. what it actually does it take everything in the Watir module and add them to the base object that is used by every other Ruby class, module and library. It is much, much worse than using global variables.
End of rant. In this case it is running into a conflict with the activesupport package, which Watir recently started using to help simplify our code base.

Automation Testing with Watir

If you are looking for an open source tool for automating the testing of your web application, then let me suggest Watir for you...

Watir is a functional automation Testing tool for web based application. Watir stands for Web Application Testing In Ruby and it is pronounced as “Water”. Watir uses Ruby as the programming language, it is a full featured object oriented language. Ruby gives the power to connect to databases, read data files, export XML and structure your code into reusable libraries.

Watir is a Ruby library for writing test scripts for web applications. It works with Internet Explorer on Windows, via its COM interface. Watir is currently being ported to support Firefox and Safari. It is a Ruby library which drives Internet Explorer the same way people do, clicks links, fills in forms, and presses buttons etc...

Advantages of Watir:

• Open Source Tool
• Simple and easy to use/learn and powerful tool.
• Good support for automating normal web based applications
• Use Ruby - full featured object oriented language. Ruby gives the power to connect to databases, read data files, export XML and structure your code into reusable libraries.
• Good tool for Data Driven or Keyword driven Test approach and it support accessing and reading from MS Excel or OpenOffice for Data driven tests.
• Watir API is richer
• Very Active development/enhancements happening for Tool with frequent releases; Also the Watir has a growing support/community on net.

Limitations of Watir:

• The record and playback option is very limited with the Watir recorder. It does not support record and playback of web applications that has a frame.
• Does not support Activex Plugin Components in your web pages.
• Does not suppport JAVA Applets in web pages.
• Does not suppport Macromedia Flash or other plugin applications

Installation :

First you will have to install Ruby 1.8.5 and then Watir. Refer the site http://wtr.rubyforge.org/install.html for finding the correct version of Ruby and Watir and installing it to the machine.

Normally for object recognisation, we use IE developers Tool bar. This also can be downloaded and installed from the same above link.

Start using Watir:

Watir has a simple syntax and that is one of the reason, I mantioned it is "Simple and easy to use/learn" tool. So of the basic syntax are given below...

• ie=IE.new #-- To open a new IE window
• ie.goto("http//www.google.com ") #-- To navigate to google page.
• ie.text_field(:name,'name').set('value') #-- For accessing text field in a web page
• ie.select_list(:name,'name').select('value') #-- For accessing dropdown list in a web page
• ie.button(:name,'name').click #-- For clicking button in a web page
• ie.link(:text,'text').click #-- For clicking link in a web page
• ie.frame(:name,"someFrame").text_field(:name,'name').set('value') #-- For accessing text field inside a frame in a web page

One more thing, I would like to tell you when you start working with Watir is about IRB. You can try out test script ideas interactively with irb. To start IRB on Windows, select start > Run... and type irb in the Open field. Click OK, and a DOS console should open with a command prompt. Require the Watir library in IRB and enter commands to start driving a web browser interactively, it looks like this:

In Depth :
With the the above listed, you would be able to write test scripts for your web application.. and now, let us get a little in depth.. here few details on all you would require to design a complete automation suit with Watir. As we have very little documentation for Watir, probably the below details would reduce your time for doing research in finding out.

Screenshot capturing:

There is an inbuild method for capturing screenshot of web application and it is given below. This can be called when a test case fails, so that screenshot of failure will be captured and saved in the same directory from which you run the script.

screen_capture()

Generating Results Files:

There can be two kind of results files. Along with the default TXT logger, you can create an XML result file with some modification in the logger class as described below.

class XMLLogger < level =" Logger::INFO" datetime_format = "%d-%b-%Y %H:%M:%S" logfile =" File.new(xmlFileName," version="'1.0'" encoding="'ISO-8859-1'?">"
@logfile.puts ""
@logfile.puts ""
end

#hackish method to log results without aid of REXML or other Ruby XML tools
def log_results(number, tc_number, name, test_status, screen, notes)
log("XML file output: S.No: " + number + ", TC No: " + tc_number + ", Test Case Name: " + name + ", Status: " + test_status + ", Screen Shot: " + screen + ", Comments: " + notes + "")
@logfile.puts''
@logfile.puts''
@logfile.puts tc_number
@logfile.puts''
@logfile.puts''
@logfile.puts name
@logfile.puts''
@logfile.puts''
@logfile.puts test_status
@logfile.puts''
@logfile.puts''
@logfile.puts screen
@logfile.puts''
@logfile.puts''
@logfile.puts notes
@logfile.puts''
@logfile.puts""
end
#hack to close the tag and the file
def end_log
@logfile.puts""
@logfile.close
end
end

How to handle pop-up:

Here a sample script which handle the pop-up and click "ok" button in the pop-up that shows up when we click the button with index "1" in the ie.

$ie.button(:index,1).click_no_wait
click_popup_ok_button

def click_popup_ok_button
begin
hwnd = $ie.enabled_popup(10)
if(hwnd)
wc = WinClicker.new
wc.clickWindowsButton_hwnd(hwnd, "OK")
end
rescue Watir::Exception::TimeOutException
$logger.log("The Popup is not shown.")
result_fail
end
end

Data base connection:

Ruby has different methods of database connectivity.

Database driver:

The syntax for connecting mysql database with database driver is below.

require 'mysql'
mysql = Mysql.init()
mysql.connect('server','username','password')
mysql.select_db("schema")
a = mysql.query("query;")
a.each do record
b = record[0]
print b
end

Similary you can connect any other DB also.

DBI:

The DBI connectivity interface is a database independent wrapper to make SQL calls. DBI is an interface to the low level database drivers. DBI will NOT work unless you install the low level database drivers.

dbh = DBI.connect('DBI:Mysql:test', 'username', 'password')

If you face any trouble in making the DB connection works, get the mysql.so and libmysql.dll etc files as instructed in http://rubyforge.org/forum/message.php?msg_id=4293

Have a try with Watir and I hope you will enjoy it.

Performance Testing - An Overview

Performance Testing, Load Testing, Stress Testing, Volume Testing etc are terms which are normally used interchangeably. Here a humble attempt to provide details what each of these are exactly and how to approach! (Not sure how far I succeed in this attempt...)

Performance Testing : The performance testing is to determine the speed or effectiveness of a computer, network or software program or application. The goal of performance testing is to identify the bottlenecks of your application performance and tune/optimize it. Performance testing means the entire process of testing the performance of application which includes load testing, endurance testing, stress testing, etc.

Load Testing : A load test is usually conducted to understand how the application behaves under a specific load. Say in case of a web application, this load can be the number of concurrent users accessing the application.

Stress Testing : The intention of stress test is to break the application and determine the robustness of application at the extreme load. The main focus of stress testing is to analyze the failure & recovery of application when the load is more than expected or double to the expected load.

Performance Testing can be done at different levels listed below

• At the application level: Performance Testing Engineers can use profilers to spot inefficiencies in the performance of code
• At the database level: Performance Testing Engineers can use database-specific profilers or JDBC requests or query optimizers to identify the performance leakage at database level.
• At the Server level: The performance issues at this level can be identified by monitoring the hardware resources such as CPU, memory, swap, disk I/O.
• At the network level: This will mostly be not in scope of test team. network engineers can use packet sniffers, network protocol analyzers such as ethereal etc to identify performance issues at this level.

Pre-requisites for Performance Testing : A Stable build of an application is the first requisite for going for performance Testing. It is advisable to have separate test environment for performance testing which must resemble to the production/live environment as much as possible.

Defining the Performance Goal and Objective :

A completely defined set of expectations is essential for meaningful performance testing. If you don't know where you want to be in terms of the performance of your application, then it effect in which direction you take the application with performance testing. So the first thing is to define the expectation, goal and objective of performance testing. For example, the below could be some the items that requires to be defined well for a web application at this stage.

• The maximum number of concurrent users or HTTP connections, the system should support for accessing the pages in application.
• The maximum number of concurrent users, performs the critical transactions at a time.
• The acceptable response time(Maximum value) for loading normal pages in application(in seconds)
• The acceptable response time(Maximum value) for complex processes and generating reports in application(in seconds)
• The expected time for processing a request with different volume of data (say search of 100/200/500/1000 employees)

Once you know where you want to be, you can start on your way there…!

Performance Tool Evaluation:

There are tools which are capable of simulating the HTTP/HTTPS requests generated by hundreds or even thousands of simultaneous users and thus allow you to test performance of application under such situations. Some of the tools available are LoadRunner, SilkPerformer, WebLoad, JMeter from Jakarta, OpenSTA from CYRANO, QEngine from AdventNet, WAPT from SoftLogica etc. There are many attributes which has to be considered during the performance testing tool evaluation and some of them are...

• Supporting Protocols(say HTTP/HTTPS etc) and supporting platforms
• Maximum virtual users allowed
• Record and playback feature
• Supporting scripting languages
• Server Performance/Data base Performance Monitoring
• IP Spoofing and Proxy Support of the tool
• Support of Distributed Load Testing
• Performance Result/Reports provided by the tool
• Support of real world performance testing and configurable user think time

Performance Testing Approach:

The simple and generalized approach for doing performance testing with any Tool can be as detailed below.

Identifying Key Scenarios : At this step, we will identify all possible application scenarios and transactions that are going to be performance/load/stress tested.

Identifying Work Load : Here, we will be identifying the work load that we want to apply to the scenarios identified at above step. This step also includes identifying how the profile should be created for the identified scenarios.

Preparing the Scripts : Using the tool, we will capture the each scenarios identified in above step and generate the test scripts. The scripts recorded by tool will be require modifications according to the need of the test. At this stage, we will also add required assertions, result capturing listeners, controllers etc to the recorded scripts.

Configuring the Test : Using the Performance tool, we will be configuring how the requests will be run against the server, like number of concurrent users, ramp time period, Loop count etc. In some cases, it is good to configure running different profiles at a time, and also add user think timing in order to suit it to a more practical way.

Performance Test Excecution : Once we have created all the test scripts and configured all the settings, we will perform the testing at this phase and capture the results.

Reporting and Analysis : At the end of the test execution, we need to gather all the test reports provided by tool including the total number of requests, requests per second and failures. There’s also additional detail about each link and average request times of each link. The basic result data that we look for in any performance test will be

• Throughput versus user load.
• Response time versus user load.

Throughput: Throughput is the number of requests that can be served by our application per unit time. It can vary depending upon the load (number of users) and the type of user activity applied to the server. This can be obtained from the Requests per Second.

Response Time (Latency): Response time is the amount of time taken to respond to a request. The latency measured at the client includes the request queue, plus the time taken by the server to complete the execution of the request and the network latency.

Other than the above, there are many other result data you get from the tool. Once we capture the data and analyze the captured data with respect to our application's performance objectives, the next level is performance tuning.

Performance Tuning :

When the results of the test indicate that performance of the system does not match the expected goals, it is time for performance tuning. The performance Tuning should be starting with the application and the database. You want to make sure your code runs as efficiently as possible and your database is optimized on a given OS/hardware configurations.

After tuning the application and the database, if the system still doesn't meet its expected goals in terms of performance, then a wide range of tuning procedures is available at the all the levels discussed above. Below are some examples of things you can do to enhance the performance of a Web application when the bottleneck is beyond the application code:

• Scale the Web server farm horizontally via load balancing
• Scale the database servers horizontally and split them into read/write servers and read-only servers, then load balance the read-only servers
• Scale the Web and database servers vertically, by adding more hardware resources (CPU, RAM, disks)
• Increase the available network bandwidth

I believe, I have tried to put together all points atleast at a high level. Thanks to Google for it's excellent search engine built, with out that it could have been highly difficult for me to collect/learn these details!

:)