Today I want to cover a topic of how to compare results of two test runs in JMeter. As there is nothing provided out of the box solution, I have seen a few solutions in my career as performance tester. The one presented here I like the most.

 

A goal behind the solution presented below was to have:

  • a simple graph,
  • clearly showing how two test runs compare,
  • even for a complex test comprising lots of request and transactions.

 
The idea for this chart came from one of my colleagues from a time when I worked at Objectivity. I remember that back in the day it was quite a hassle to obtain similar chart in Excel, thankfully the version below can be generated in no-time in R.

For me it is a great tool to quickly compare and assess response times for two JMeter test runs without overwhelming you with the amount of data:

 

Compare results of two JMeter test runs

 

So what do we have here:

  • a scatter plot
  • x-axis is a response time in baseline test
  • y-axis brings you a percentage of improvement/degradation in the run compared to the baseline test
  • there are separate data points for requests and transactions
  • there are separate regression lines for requests and transactions to check for trends in data

I will also describe other additions to the chart showing more useful information in the next post.
 
UPDATE: Check out some extensions to this graph in another post.
 

How to generate the JMeter test results comparison chart?

 
The chart is generated in R and I have already written a short note on what you’ll need in terms of tooling in a previous post.
 
Once you have the tooling in place, this is the script that you need to run:
 

# Load packages
library("ggplot2")
library("dplyr")
library("plyr")

# Load JMeter results data for baseline test run and compared test run
jmeter_results_baseline <- read.csv("C:\\Path\\To\\JMeter's\\baseline\\results.jtl",header=T, sep=",",stringsAsFactors = FALSE)

jmeter_results_compared <- read.csv("C:\\Path\\To\\JMeter's\\compared\\run\\results.jtl",header=T, sep=",",stringsAsFactors = FALSE)

# Aggregate loaded data 
summarized_data_baseline <- ddply(jmeter_results_baseline, c("label"), summarise,
                                  sample_number = length(elapsed),
                                  mean = mean(elapsed),
                                  q90 = quantile(elapsed,probs=0.9),
                                  q95 = quantile(elapsed,probs=0.95)
)

summarized_data_compared <- ddply(jmeter_results_compared, c("label"), summarise,
                                  sample_number = length(elapsed),
                                  mean = mean(elapsed),
                                  q90 = quantile(elapsed,probs=0.9),
                                  q95 = quantile(elapsed,probs=0.95)
)

# Merge the results for both of the test runs
merged_results <- merge(summarized_data_baseline,summarized_data_compared, by="label", suffixes = c(".baseline",".compared"))

# Quantify the improvement/degradation factor in the compared test run
merged_results <- mutate(merged_results,improvement_factor=ifelse(q90.baseline > q90.compared,abs((q90.compared/q90.baseline)-1),  -((q90.compared/q90.baseline) - 1)))

# Add the distinction between the requests and transactions
trasaction_identifier <- "TR" # Make the distinction -> when label starts with 'trasaction_identifier', then it's a transaction
merged_results <- mutate(merged_results,sample_type=factor(ifelse(startsWith(as.character(label),trasaction_identifier),"transaction","request")))


# Construct the comparison plot
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type)) + 
  geom_point(size=4) + 
  geom_smooth( method="lm", se=F) +
  scale_y_continuous("Improvement / degradation in the compared run [%]", labels = scales::percent) +
  xlab("90th percentile of response time in baseline test [ms]") +
  scale_color_brewer(name="Sample type",palette="Set1") +
  labs(title="Comparison of two test runs",subtitle="Complementary JMeter results graph",caption="PerfTuned.com") +
  theme(axis.line = element_line(size = 2, colour = "grey"),axis.text = element_text(size = rel(1.1)),legend.position = "bottom", title = element_text(size = rel(1.1),face="bold"))

print(plot_comparison)

# Save the plot to file
ggsave("C:\\path\\to\\save\\graph.pdf",device="pdf")