Generic Questions to RFP on Performance Testing

 I have listed some generic questions which can be asked to a potential customer in response to an RFP on performance testing

• Will you provide any performance data that has been gathered on the existing application? If you don’t have the data on hand, will you help make it available?
• How many Performance contractors will be involved in the project team?
• Are there any metrics for success or expectations for future volume? Does the organization have a performance testing technology that they regularly use? Does the Application already have service levels and performance levels defined?
• What is the total number of users that will access the application?
• Description of your existing software systems, e.g., Web server, database, application server, network operating system, messaging, monitoring, network management, etc.
• Description of in-house IT staff, including their areas of expertise, be it networking, systems management, development, etc.
• Can the application provide high availability systems? What is considered high availability?
• How is load balancing performed?
• Does the application have processes in place to monitor CPU usage, file system usage, memory usage, and network performance?
• What are the tools does your organization have?
• How many and what percent of applications/projects are performance tested with tool (LoadRunner) before they are launched into production

 

Solution to Common Siebel Script Errors in LoadRunner

The following section covers common issues that are most often encountered.

Issues

1. Wrong field values

The server returns an error like the following:

@0`0`3`3``0`UC`1`Status`Error`SWEC`8`0`2`Errors`0`2`0`Level0`0`
ErrMsg` Wrong field values or value types detected in field Request
Type. Please re-enter your field values. If you need additional
assistance, please refer to the documentation.
(SBL-UIF-00299)`ErrCode`21336`2`0`.

Cause
The values (parameters) in Siebel fields of a request are invalid. The reason would be values are hard coded while recording.

Fix
Manually correlate the field values from previous requests.

2. Back or Refresh Error

The Siebel application displays an error applet indicating the following:

We detected an Error which may have occurred for on or more of the following reasons:
We are unable to process your request. This is most likely because you used the
browser BACK or REFRESH button to get to this point.
and fix
Cause
This issue can be caused by the following conditions:
• The SWETS was not parameterized for the current request.
• The SWEC was not correlated correctly for the current request.
• The request was submitted twice to the Siebel server without the SWEC being updated.
• The frame was not created on the server, possibly because the SWEMethod has changed since the script was recorded. A previous request should have set up a frame for the browser to download.
Fix
To resolve this issue, try the following fixes:
• Make sure that SWETS is parameterized.
• Make sure that you reset the SWE Count to the SWEC value of the first request in the iteration
at the beginning of the iteration.
If these fixes do not resolve the issue, rerecord the script.

3. No Content HTTP Response

The server returns an error like the following:

HTTP/1.1 204 No Content
Server: Microsoft-IIS/5.0
Date: Fri, 31 Jan 2003 21:52:30 GMT
Content-Language: en
Cache-Control: no-cache

Cause
The row ID is not properly correlated.

Fix
Manually correlate the row ID.

4. Same Values Error

The server returns an error like the following:

@0`0`3`3``0`UC`1`Status`Error`SWEC`10`0`1`Errors`0`2`0`Level0`0`
ErrMsg`The same values for 'Name' already exist. If you would like to
 enter a new record, pleaseensure that the field values are unique.
`ErrCode`28591`

Cause
One of the values in this request (in the previous code example, it is the value for the Name field)
is the same as a value in another row in the database table.

Fix
You must replace this value with a unique value for each iteration for each user. The recommended
solution is to use the row ID parameter for the value; this causes the value to be unique.

5. Restoring the Context Error

The server returns an error like the following:

@0`0`3`3``0`UC`1`Status`Error`SWEC`9`0`1`Errors`0`2`0`Level0`0`
ErrMsg`An error happened during restoring the context for requested
location`ErrCode`27631`

Cause
The row ID is not properly correlated.

Fix
Manually correlate the row ID.

6. Cannot Locate Record Error

The server returns an error like the following:

@0`0`3`3``0`UC`1`Status`Error`SWEC`23`0`2`Errors`0`2`0`Level0`0`
ErrMsg`Cannot locate record within view: Contact Detail - Opportunities
View applet: Opportunity List Applet.`ErrCode`27573`

Cause
The input name SWERowId does not contain a row ID for a record on the Web page. This input name
should have been parameterized. The parameter's source value may have changed its location.

Fix
Manually correlate the row ID.

7. End of File Error

The server returns an error like the following:

@0`0`3`3``0`UC`1`Status`Error`SWEC`28`0`1`Errors`0`2`0`Level0`0`
ErrMsg`An end of file error has occurred. Please continue or ask your
systems administrator to check your application configuration if the
problem persists.`ErrCode`28601`

Cause
The input name SWERowId does not contain a row ID for a record on the Web page. This input name
should have been parameterized. The parameter's source value may have changed its location.

Fix
Manually correlate the row ID.

Performance Assessment Methodology

The below approach will help you if  you are planning to act as a consultant for a performance testing engagement and provide recomendations to your customer.

The attached diagram will detail  the assessment approach for the following topics

• Study the existing test processes
• Study Architecture
• Gather Business Information
• Define test Model
• Define tests 

Performance Assessment Methodology

           

                

How to convert LoadRunner Siebel-Web Protocol scripts to Web Protocol

1. Open the LoadRunner .usr file in Notepad

2. Check the following lines in the notepad

[General]


Type=Multi


AdditionalTypes=Siebel_Web


ActiveTypes=Siebel_Web


GenerateTypes=Siebel_Web



3. Replace Siebel_web with QTWeb

[General]


Type=Multi


AdditionalTypes=QTWeb


ActiveTypes=QTWeb


GenerateTypes=QTWeb


RecordedProtocols=QTWeb



Controller no longer requires separate license for Siebel-Web. Web user license is sufficient to execute the scripts

Setting LoadRunner Header File Path

runneLoadRunner automatically compiles all the header files present in LoadRunner\include directory.


But it required header files need to be updated regularly after changes into the LoadRunner\include directory.



You can change the load runner properties to include your own header files folder

Please follow the step below one by one:

1. Browse through C:\Program Files\Mercury\LoadRunner\dat folder and search mdrv.dat file

2. Create a back up of the file in the same directory before making changes

3. Open the mdrv.dat file and search for [lrun_api] text in the file

4. You will reach the below section

[lrun_api]


ExtPriorityType=internal


WINNT_EXT_LIBS=lrun50.dll


WIN95_EXT_LIBS=lrun50.dll


LINUX_EXT_LIBS=libLrun50.so


SOLARIS_EXT_LIBS=libLrun50.so


HPUX_EXT_LIBS=libLrun50.sl


AIX_EXT_LIBS=libLrun50.so


LibCfgFunc=LrunApi_configure


UtilityExt=ParamEngine,Transaction,vusr_log,faserver,run_time_context


ExtIncludeFiles=lrun.h


ActiveScriptItems=Message:Mercury.Lrvb.LrMessage.1,Timing:Mercury.Lrvb.LrTiming2.1,Transaction:Mercury.Lrvb.LrTransaction2.1


ExtMessageQueue=0


SecurityRequirementsFiles=AllowedFunctions.asl


SecurityMode=On


5. Now add a new line to this section as follow


[lrun_api]


ExtPriorityType=internal


WINNT_EXT_LIBS=lrun50.dll


WIN95_EXT_LIBS=lrun50.dll


LINUX_EXT_LIBS=libLrun50.so


SOLARIS_EXT_LIBS=libLrun50.so


HPUX_EXT_LIBS=libLrun50.sl


AIX_EXT_LIBS=libLrun50.so


LibCfgFunc=LrunApi_configure


UtilityExt=ParamEngine,Transaction,vusr_log,faserver,run_time_context


ExtIncludeFiles=lrun.h


ActiveScriptItems=Message:Mercury.Lrvb.LrMessage.1,Timing:Mercury.Lrvb.LrTiming2.1,Transaction:Mercury.Lrvb.LrTransaction2.1


ExtCmdLine=-compile_flags C:\testscripts\ExternalHeaderFilePath


ExtMessageQueue=0


SecurityRequirementsFiles=AllowedFunctions.asl


SecurityMode=On

6. After making the changes, save the mdrv.dat file.

7. Take a sample script and execute the test once.

End to End Performance Test Approach - Part 2

In continuation with my earlier post
http://performancetestinginfo.blogspot.com/2009/10/end-to-end-performance-test-apprach.html
Test Plan & Strategy:

During the test planning activities, gather the performance test objectives, calculate resource estimations and project timelines, and review the Architecture with individual team members and determine the types of tests required to test the application.

Capture the project specific information for each of the projects in the test plan as per the following template. This information will help us to co related the performance test results with system configurations. Performance test results will vary with system configuration changes.

Project Name	XYZ
Application Background	XYZ is an online web based application which is used by the customer to place orders related to various products. System currently getting upgraded from oracle 10g to oracle 11i
Type of Project:	Seibel Web application/ SAP GUI
Application Technology	dot Net, IIS web server, C #, VB
Hardware Platform involved and OS	App Server - windows XP, 2 GB ram, 4 cpu DB server - Windows XP SP2, 3 GB, 10 CPU
Database	Oracle
Third party tools

Create a workload model which covers the list of scenarios identified for the performance testing along the SLA’s and the user load. No of Txns and No of Concurrent Users will be derived from the volumetric analysis

S. No.	Transaction/Script	Online/Batch	No. of Concurrent Users	Response time	No. of Txns.
1	Scenario 1	O	9	< 10 secs	12
2	Scenario 2	O	4	< 1 secs	12
3	Scenario 3	O	15	< 2 secs	143
4	Scenario 4	O	4	< 13 secs	20
5	Scenario 5	O	2	< 4 secs	20
6	Scenario 6	B	3	< 5 secs	3
7	Scenario 7	B/O	1	< 2 secs	4
8	Scenario 8	O	1	< 4 secs	5

Identify the different types of tests required for testing the application based on the requirement analysis.

In Load test,  measure server response times to verify if the application can sustain expected maximum number of concurrent users and expected maximum size of the database.

In Stress test, measure server response times at varying loads starting from low load (low number of concurrent users), medium load (average number of concurrent users) through high load (expected maximum number of concurrent users until unacceptable levels of response times are experienced) to validate application's stability and validity.

In Endurance test, test the application for longer durations with half the average system load to detect the possible memory leaks in the system.

A detailed test plan canl be laid out using the information captured during the requirements gathering phase and share it with the development/Business team and take their inputs for the final approval. Test plan should include the following (but not limited to):

• Scope
• Test Approach
• Test Objectives
• Test Environment setup and requirements
• Types of tests
• Transaction mix
• Workload Scenario
• Identify Monitors
• Scheduling ( Testing sequence , Test cycles)
• Data setup (Data required by the Test tool, not the Application data)

Test Design & Execution

During the test design phase, validate the existing scripts and develop new functionalities based on the workload model and also validate and update the data required for the test environment identified during the test plan and also analyze script failures with the intent of finding their root cause so that we can debug our scripts effectively. We should also collect any application related errors found during the script validations and share with the development team.

During the execution process, first try to validate the scripts are pointing to the correct environment and performance metrics to be captured are properly configured in the environment. We should also validate load generator machines are up and working fine.

Each script should be run individually several times to validate that the script has been developed correctly. These tests may reveal performance problems that will need to be addressed.

Mixed load test can be carried out for the identified scenarios consisting of all transactions, online and batch, according to the workload mix discussed earlier. The load tests have to be run multiple times to ensure that the testing process is repeatable and also configure all the performance metrics in the load testing tool prior to the start of the test.

Result Analysis and Reporting

 Focus on analysis, monitoring, identifying bottle necks and proving recommendations, thus providing an end-to-end performance solution for the complete application.

Send the test reports   from various tests results with  conclusions based on those results, and also with consolidated data that supports those conclusions. And also do analysis, comparisons, and details of how the results were obtained.

At the end of each run of the Performance Test, a report should be produced. The test report should have comprehensive data collected from various sources presented in a single document.

For each of the test cases, the following response times should be reported: arithmetic mean, standard deviation, 90th percentile response time and other percentiles as necessary. In addition, each test case also report the total number of transactions executed, the time period over which the transactions were executed, number of errors and number of retries.

Collect comprehensive set of system data and tabulated in the test report for each run. The data that will be collected will include CPU utilization, memory utilization – system wide and per process, DB statistics.

End to End Performance Test Approach - Part 1

The purpose of this post is to show end to end approach for implementing performance testing. This document covers different phases of performance testing and approach to follow for successful performance test

Requirement Gathering:


During the Requirement Analysis phase it is important to assess and understand the nature of the application and the environment in which testing and monitoring should be performed. In addition to this, identify the resource requirements and plan based on the application. Also the existing Non-functional Requirement (NFR) would be discussed and understood by having discussion with the Design & Development team with respect to SLA’s, number of concurrent users and volumetric information. Also, wherever there is any specific information lacking, the same should be discussed with Design & Development team to reach a mutual agreement and definition of the same.

Performance team should also analyze system volume metrics over specified period of time in production to identify load patterns, work load behavior, Peak user load etc in the system.


Identify the peak period during volume metric analysis and also transaction arrival rate along with the user concurrency to simulate in the test environment. Create a workload model for the peak load with transaction mix along the user load based on the volume metric analysis. This will also help us in identifying the types of tests required for testing the application. Below is a sample template of the workload model for deriving the peak load.

Workload Model

Decoding concepts of Performance Engineering

In most of the situations we have a tendency of using the formula’s without understanding the concepts behind it, because of which we may not be able to apply the laws correctly where it is required. 

The following example will help in understanding the some of the performance rules

A system was subjected to 10 requests in a span of 2 min and got the responses for all requests during this period and Utilization of the system during the time frame was observed to be at 50%.

Request Arrival rate for that system is  10/2=5 requests/Min

Throughput = 10/2 = 5 Responses/Min

in the above example, system utilized 50% of its resources for serving 5 requests. For Each request, it consumed 50%/5 of system resources. This value is called service demand of that resource

From the above example the formula for service demand can be derived as below
Service Demand  =Utilization/Throughput. 
similarly Utilization = Throughput * Service Demand - This is also called utilization Law

The utilization of a resource is equal to the product of the throughput of that resource and the average service demand at that resource. 

Working with LoadRunner Sybase Ctlib protocol

The goal of this document is to bring together necessary information to help those users who are involved in scripting Client Server Database protocols using LoadRunner.


The document can be used as a basis for scripting, enhancing and debugging any of the following protocols

Ø Sybase CTlib
Ø Sybase DTlib
Ø Informix
Ø MS SQL Server
Ø Oracle 2-Tier
Ø ODBC
Ø DB2 CLI
Ø ERP/CRM Siebel Vuser scripts


This document is based on my experiences using LoadRunner with Sybase Ctlib protocol

Click here to download the document