Today I want to cover a topic of how to compare results of two test runs in JMeter. As there is nothing provided out of the box solution, I have seen a few solutions in my career as performance tester. The one presented here I like the most.
A goal behind the solution presented below was to have:
- a simple graph,
- clearly showing how two test runs compare,
- even for a complex test comprising lots of request and transactions.
The idea for this chart came from one of my colleagues from a time when I worked at Objectivity. I remember that back in the day it was quite a hassle to obtain similar chart in Excel, thankfully the version below can be generated in no-time in R.
For me it is a great tool to quickly compare and assess response times for two JMeter test runs without overwhelming you with the amount of data:
So what do we have here:
- a scatter plot
- x-axis is a response time in baseline test
- y-axis brings you a percentage of improvement/degradation in the run compared to the baseline test
- there are separate data points for requests and transactions
- there are separate regression lines for requests and transactions to check for trends in data
I will also describe other additions to the chart showing more useful information in the next post.
UPDATE: Check out some extensions to this graph in another post.
How to generate the JMeter test results comparison chart?
The chart is generated in R and I have already written a short note on what you’ll need in terms of tooling in a previous post.
Once you have the tooling in place, this is the script that you need to run:
# Load packages library("ggplot2") library("dplyr") library("plyr") # Load JMeter results data for baseline test run and compared test run jmeter_results_baseline <- read.csv("C:\\Path\\To\\JMeter's\\baseline\\results.jtl",header=T, sep=",",stringsAsFactors = FALSE) jmeter_results_compared <- read.csv("C:\\Path\\To\\JMeter's\\compared\\run\\results.jtl",header=T, sep=",",stringsAsFactors = FALSE) # Aggregate loaded data summarized_data_baseline <- ddply(jmeter_results_baseline, c("label"), summarise, sample_number = length(elapsed), mean = mean(elapsed), q90 = quantile(elapsed,probs=0.9), q95 = quantile(elapsed,probs=0.95) ) summarized_data_compared <- ddply(jmeter_results_compared, c("label"), summarise, sample_number = length(elapsed), mean = mean(elapsed), q90 = quantile(elapsed,probs=0.9), q95 = quantile(elapsed,probs=0.95) ) # Merge the results for both of the test runs merged_results <- merge(summarized_data_baseline,summarized_data_compared, by="label", suffixes = c(".baseline",".compared")) # Quantify the improvement/degradation factor in the compared test run merged_results <- mutate(merged_results,improvement_factor=ifelse(q90.baseline > q90.compared,abs((q90.compared/q90.baseline)-1), -((q90.compared/q90.baseline) - 1))) # Add the distinction between the requests and transactions trasaction_identifier <- "TR" # Make the distinction -> when label starts with 'trasaction_identifier', then it's a transaction merged_results <- mutate(merged_results,sample_type=factor(ifelse(startsWith(as.character(label),trasaction_identifier),"transaction","request"))) # Construct the comparison plot plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type)) + geom_point(size=4) + geom_smooth( method="lm", se=F) + scale_y_continuous("Improvement / degradation in the compared run [%]", labels = scales::percent) + xlab("90th percentile of response time in baseline test [ms]") + scale_color_brewer(name="Sample type",palette="Set1") + labs(title="Comparison of two test runs",subtitle="Complementary JMeter results graph",caption="PerfTuned.com") + theme(axis.line = element_line(size = 2, colour = "grey"),axis.text = element_text(size = rel(1.1)),legend.position = "bottom", title = element_text(size = rel(1.1),face="bold")) print(plot_comparison) # Save the plot to file ggsave("C:\\path\\to\\save\\graph.pdf",device="pdf")