Today I want to cover a topic of how to compare results of two test runs in JMeter. As there is nothing provided out of the box solution, I have seen a few solutions in my career as performance tester. The one presented here I like the most.

A goal behind the solution presented below was to have:

• a simple graph,
• clearly showing how two test runs compare,
• even for a complex test comprising lots of request and transactions.

The idea for this chart came from one of my colleagues from a time when I worked at Objectivity. I remember that back in the day it was quite a hassle to obtain similar chart in Excel, thankfully the version below can be generated in no-time in R.

For me it is a great tool to quickly compare and assess response times for two JMeter test runs without overwhelming you with the amount of data:

So what do we have here:

• a scatter plot
• x-axis is a response time in baseline test
• y-axis brings you a percentage of improvement/degradation in the run compared to the baseline test
• there are separate data points for requests and transactions
• there are separate regression lines for requests and transactions to check for trends in data

I will also describe other additions to the chart showing more useful information in the next post.

UPDATE: Check out some extensions to this graph in another post.

### How to generate the JMeter test results comparison chart?

The chart is generated in R and I have already written a short note on what you’ll need in terms of tooling in a previous post.

Once you have the tooling in place, this is the script that you need to run:

# Load packages
library("ggplot2")
library("dplyr")
library("plyr")

# Load JMeter results data for baseline test run and compared test run

summarized_data_baseline <- ddply(jmeter_results_baseline, c("label"), summarise,
sample_number = length(elapsed),
mean = mean(elapsed),
q90 = quantile(elapsed,probs=0.9),
q95 = quantile(elapsed,probs=0.95)
)

summarized_data_compared <- ddply(jmeter_results_compared, c("label"), summarise,
sample_number = length(elapsed),
mean = mean(elapsed),
q90 = quantile(elapsed,probs=0.9),
q95 = quantile(elapsed,probs=0.95)
)

# Merge the results for both of the test runs
merged_results <- merge(summarized_data_baseline,summarized_data_compared, by="label", suffixes = c(".baseline",".compared"))

# Quantify the improvement/degradation factor in the compared test run
merged_results <- mutate(merged_results,improvement_factor=ifelse(q90.baseline > q90.compared,abs((q90.compared/q90.baseline)-1),  -((q90.compared/q90.baseline) - 1)))

# Add the distinction between the requests and transactions
trasaction_identifier <- "TR" # Make the distinction -> when label starts with 'trasaction_identifier', then it's a transaction
merged_results <- mutate(merged_results,sample_type=factor(ifelse(startsWith(as.character(label),trasaction_identifier),"transaction","request")))

# Construct the comparison plot
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type)) +
geom_point(size=4) +
geom_smooth( method="lm", se=F) +
scale_y_continuous("Improvement / degradation in the compared run [%]", labels = scales::percent) +
xlab("90th percentile of response time in baseline test [ms]") +
scale_color_brewer(name="Sample type",palette="Set1") +
labs(title="Comparison of two test runs",subtitle="Complementary JMeter results graph",caption="PerfTuned.com") +
theme(axis.line = element_line(size = 2, colour = "grey"),axis.text = element_text(size = rel(1.1)),legend.position = "bottom", title = element_text(size = rel(1.1),face="bold"))

print(plot_comparison)

# Save the plot to file
ggsave("C:\\path\\to\\save\\graph.pdf",device="pdf")