13. Continuous Integration

Setting up a Jenkins CI server using the DataHunter sample Jobs

In this chapter we go through the setup of a Jenkins Windows installation, including jobs to run the DataHunter P&V test on a single server, and the setup for a distributed test.

This is meant as demonstration on how to build a full testing environment, so everything is assumed to be built on the one Windows machine (excepting the distributed test). We also assume

  • you have have downloaded the Mark59 GitHub repo at https://github.com/mark-5-9/mark59 on your machine at C:\gitrepo\mark-5-9\mark59 (there are only be a few path changes needed if you decide to use the mark59-wip GitHub repo instead).

  • you have have downloaded the Mark59 GitHub repo at https://github.com/mark-5-9/mark59-xtras

    on your machine at C:\gitrepo\mark-5-9\mark59-xtras (this mark59-xtras repo contains artefacts used to configure and build the Jenkins server).

CI Setup Instructions

Install the Jenkins Server.

  • From the Jenkins download page, download the latest 'Long-term Support' 'Generic Java package (.war)'.

  • Copy the jenkins.war file to C:_Jenkins\JENKINS_HOME

  • Open a command prompt window and execute the following:

  • c: SET JENKINS_HOME=c:\_Jenkins\JENKINS_HOME cd %JENKINS_HOME% java -Dhudson.model.DirectoryBrowserSupport.CSP="" -jar jenkins.war --httpPort=8082 Note that Jenkins by default starts up using port 8080. The default port is overridden in the above command using the--httpPort option (as ports 8080 & 8081 are used by Trend Analysis and DataHunter components by default)

  • You should see a message “Jenkins is fully up and running” upon successful installation.

  • Confirm Jenkins Home at URL: http://localhost:8082

  • Accessing Jenkins for the first instance requires you to unlock it using the auto generated password. After doing this, make sure to create an admin user account, e.g. admin.

Install the required Jenkins plugins

  • As part of the initial setup, you will be asked to either:

    • Install suggested Plugins / Select Plugins to Install (or)

    • Configure proxy / Skip Plugin Installation. This most probably would be the case if you are in a corporate site where in you see the message "This Jenkins instance appears to be offline". This can be easily solved by configuring the proxy server by accessing http://localhost:8082/pluginManager/advanced.

  • We suggest installing the following list of plugins to be able to run a continuous Integration test (several are required for the supplied Jobs to work properly). You can remove or Install additional plugins via the Manage Jenkins > Manage Plugins page in Jenkins (note that same may be pre-installed):

    • OWASP Markup Formatter

    • Log Parser

    • Parameterized Trigger

    • Mask Passwords

    • Email Extension

    • Build Timeout

    • Timestamper

    • Green Balls

    • HTML Publisher

    • Node and Label parameter

    • Git

Jenkins System Configuration

Once you are logged in, you will see these options under 'Manage Jenkins' -> 'Configure System'

  • In Global Properties, set the following Environment Variable name/value pairs :

    • JAVA_HOME : Set this to the relevant JDK (e.g. C:\Program Files\Java\jdk-11)

    • MAVEN_HOME : where your maven is (e.g. C:\apache-maven-3.5.3)

    • path : %JAVA_HOME%\bin;%MAVEN_HOME%\bin;%PATH%

  • In Jenkins Location, set 'System Admin e-mail address' to something similar to PV.Jenkins<doNotReply@mycorp.com>

  • In Extended E-mail Notification, set the following values:

    • SMTP server : your SMTP server

    • Default Content Type : Plain Text (text/plain)

    • Default Recipients : your email address All other values can be left as they are Hint : You can use the E-mail Notification section (the one just below) in conjunction with 'Test configuration by sending test e-mail' to check you have the correct details for your SMTP server.

  • In Console Output Parsing, set the following Parsing Rules:

    • Log Parse Rules Generic for JMeter and Vugen C:\gitrepo\mark-5-9\mark59-xtras\Jenkins_ParsingRules\LogParseRulesGenericJMeterAndVugen.txt

    • Log Parse Rules for JMeter Execute C:\gitrepo\mark-5-9\mark59-xtras\Jenkins_ParsingRules\LogParseRulesJmeterExecute.txt

    • Log Parse Rules for LoadRunner TruClient C:\gitrepo\mark-5-9\mark59-xtras\Jenkins_ParsingRules\LogParseRulesLoadRunnerTruClient.txt

Jenkins Global Security

Once you are logged in, you will see these options under 'Manage Jenkins' -> 'Configure Global Security'

  • In Authorization, under 'Logged-in users can do anything, tick 'Allow anonymous read access'

  • Set Markup Formatter to 'Safe HTML'

  • In Agents, set the TCP port for inbound agents to fixed and enter a free port (e.g. 8099)

Create Jenkins Jobs

  • Stop the Jenkins server (ctrl+C in the Command Prompt window) Hint: Also close any browser tabs with Jenkins URL (localhost:8082) loaded - otherwise you may get confusing errors on re-start.

  • Copy the folders under C:\gitrepo\mark-5-9\mark59-xtras\Jenkins_Jobs\jobs (8 folders, corresponding to the 8 jobs we are going to create), to C:\_Jenkins\JENKINS_HOME\jobs

  • Restart the Jenkins server

  • You should see the jobs listed on the Jenkins home page

Set Up and Run the Mailer Job

This section sets up and verifies the provided sample mailer job. Ensure the SMTP server details have been entered in the System Configuration as described above. Then:

  • Create a New View called DataHunter, listing at least the DataHunter_Result job and DataHunterDistributed_Result job

  • Copy the 'email-templates' folder under C:\gitrepo\mark-5-9\mark59-xtras\Jenkins_email-templates to C:_Jenkins\JENKINS_HOME

  • In the _Mailer_DataHunter job configuration, ensure the 'Project Recipient List' is your email address. The rest of the configuration should be okay.

  • Run the _Mailer_DataHunter job. You should get an email in you inbox titled "DataHunter Continuous Performance testing". Note at time of writing a successful job had in its the job console output messages like : No emails were triggered.... followed by Email was triggered for: Success. Appears to be a quirk with the current version of the mailer.

  • That email will most likely look a bit odd, as it has not picked up the the icons from the job workspace. To fix this copy all the images in C:_Jenkins\JENKINS_HOME\jobs\_Mailer_DataHunter\workspace to C:_Jenkins\JENKINS_HOME\workspace\_Mailer_DataHunter

  • Run the the _Mailer_DataHunter job again. You should get an email in you inbox that now includes the images:

  • If you wish, change the template, so that, for example, it has the correct signature for your team.

Once a mail job has run at least once, you can play around with the "Email Template Testing" option to test template changes. 'Jelly/Groovy Template File Name' using the supplied file is PV_CI_Results_With_History_By_View.template

Continuous Integration Job Flow General Design

The Continuous Integration (CI) test is designed to include the following 4 integrated jobs.

Verify Job (ProjectName_Check)

CI Test starts with a verification job which is a short duration shakeout test to verify the test scripts & test environment. Generally, the verification job will run the same test plan as the main test execution, but for a shorter time. Also the number of users per thread executed is usually reduced by using job parameters (the Users parameter in the DataHunter Check jobs). A successful verification job is marked by a green build and proceeded by the Execute Job , which starts the actual JMeter test (unless it's a distributed test). A red build for a verification job indicates a failure. Flags are set to indicate job status, which in the case of a failed verify job are passed and used to mark the Execute & Results job as Failure with a red build.

Start JMeter Server (ProjectName_DistributedJmeterServices)

Only required for Distributed tests. This job start JMeter server instances on remote servers required for a distributed testing. The Master and the remote servers are defined in this job with the following parameters:

  • MasterIP – IP Address of the JMeter master server

  • DistributedServersList - Comma delimited list of remote slave server names followed by their IP address (e.g. slavesrvr1:111.222.333.444)

Execute Job (ProjectName_Execute)

The execute Job could be a peak load test, a stress test or a long duration soak JMeter test. The configuration for a distributed test is similar to a single node test except that distributed test needs an additional parameter “DistributedIPsList” in the execute job, which allows you to include the list of (comma separated) IP Addresses to run the test plan from. It’s important to remember that the Master Instance should be listed as the first entry in the DistributedIPsList parameter.

Also, for a single node test (or a parameter that only needs to be know to the master), the -J options is used to define the parameter on the JMeter command line, but for a distributed test parameter the -G option has to be used (e.g. -GForceTxnFailPercent=0).

Results Job (ProjectName_Results)

Upon successful completion of test execution, the JMeter Summary Report (csv or xml output file) is used to map results into the Trend Analysis Web Application and compared against the defined SLAs. Once the result is processed, a JMeter HTML Report(s) are generated and can be viewed with the "Transactions Report" link. In addition to this, there may be separate reports generated for CPU Utilization, Memory & Data points or a combined All Metrics Report which can be accessed from the Results job.

Flags passed through the job stream to the Results job get used to set a failure, if the Verify failed. These flags are also used by the "Mailer" job to produce the correct icons related to the stage of the job stream that failed.

The Mailer Job (_Mailer_JenkinsViewName)

The mailer job provided in the samples isn't a part of the DataHunter testing job flows, it has been included to demonstrate a possible method to inform users of how their tests have performed. We tend to run the mailer jobs each morning to give everyone visibility of how overnight test runs went.

The email template PV_CI_Results_With_History_By_View.template looks for jobs ending with _result (case insensitive) in the Jenkins View whose name corresponds with the end portion of the Mailer job name. The sample mailer job is named _Mailer_DataHunter, so will look for jobs ending with _result in the DataHunter view. It uses flags (as described above in the Results Job section above) to work out the status of the last run. It also provides a history of up to the previous 14 runs for the test.

Executing the DataHunter Jenkins Job Flows

The DataHunter Single Server Test Job Flow

If everything has set up correctly as described above, the only requirements before triggering the DataHunter_Check job, to start a run are:

  • a local copy of the DataHunter application is running at http://localhost:8081/dataHunter/

  • a install of JMeter is set up at C:\apache-jmeter which can run the DataHunter selenium test plan

Follow the instructions in Executing the Selenium-JMeter Sample Scripts in JMeter if you don't have these requirements set up already.

Clicking on the 'Build with Parameters' you should see parameterization has been set as shown below. Click 'Build' to start the run.

Upon successful completion you should of seen the job sequence DataHunter_Check > DataHunter_Execute > DataHunter_Result execute. DataHunter_Result does not necessarily need to be green for the flow to of succeeded. SLA's may not of been met, changing the outcome status of the Results Job.

The DataHunter Distributed Test Job Flow

To run the DataHunter Distributed test, another machine on your network (subnet actually), and some significant setup is required. The Jenkins setup is done using 'configure' panels in these jobs :

  • DataHunterDistributed_Check : An parameter is required to define the URL of the DataHunter instance being executed So in the parameter:

    • ExtraParam2: the 'MASTERPC' part of the default value DataHunterUrlHostPort=http://MASTERPC:8081 should be changed to the machine running the DataHunter application. Actually, in this check job you could get away with just using 'localhost', but just to keep the jobs consistent its a good idea to use the machine name.

  • DataHunterDistributed_DistributedJmeterServices : This job starts the JMeter 'servers', a requirement of JMeter distributed testing. So parameter:

    • DistributedServersList: needs to be updated from SLAVEPC: to the machine and IP values for your slave.

    • MasterIP: the IP of the master machine

  • DataHunterDistributed_Execute : The IPs of all server running the test need to be listed, and a non-localhost URL of the DataHunter is required, so parameters:

    • DistributedIPsList need to be updated from -R, to the master and slave machine IPs

    • ExtraParam2 (-GDataHunterUrlHostPort=http://MASTERPC:8081) needs the MASTERPC value set as discussed for the Check job. - note the change from -J... to -G. -G is required when you wish to pass parameters to the slave machines.

Also, you will need to ensure all the necessary requirements for JMeter distributed testing, such as firewall rules and so on are met.

If you have never done it before, getting JMeter distributed testing to work can take some effort. There is good information in the JMeter user guide, but it can't cover all the security issues, etc you may run into. For example, when we tried to set up this demo using our work desktops (instead of using servers), we run into issues with the corporate anti-Virus software blocking ports. When we tried to run a distributed test using a 'master' machine connected to the corporate network via cable and a slave using a wireless connection, the slave would not connect because is was on a different subnet (breaking a RMI rule)

The main JMeter user guide references:

So once everything has set up correctly as described above, the only other requirement before triggering the DataHunterDistributed_Check job to start the run, is that a copy of the DataHunter application at the correct URL is running.

Upon successful completion you should of seen the job sequence DataHunterDistributed_Check>DataHunterDistributed_DistributedJmeterServices > DataHunterDistributed_Execute> DataHunterDistributed_Result execute. DataHunterDistributed_Result does not necessarily need to be green for the flow to of succeeded. SLA's may not of been met, changing the outcome status of the Results Job.