In the last post I’ve written how you can quickly generate a chart that compares results of two JMeter tests. Now, let’s add some more info to this graph.
 
This is where we finished the last time and what we will be improving today:
 
Compare results of two JMeter test runs

Difference in number of samples between two runs

 
One of most valuable pieces of information that we can add on top of the ‘default’ chart is to show difference in number of executed samples between the compared tests. If there is a difference that looks suspicious, then it might be a clear indication that one of the compared test runs is not valid.

 

Below you can check an example of such chart. Size of a data point denotes a difference in number of sample for particular request. The bigger the dot, the bigger the disparity, also check chart’s legend for the value in percents.

 

compare two JMeter runs and show difference in number of samples

 
As you can see the biggest data points equal 20% difference here and, as there is also 30% improvement in response time for these samples, one should perhaps check compared tests for some issues.
 
In order to generate such chart we need to execute the following R script. Marked in red are changes to the ‘default’ chart code that was developed in previous post.

# Load packages
library("ggplot2")
library("dplyr")
library("plyr")

# Load JMeter results data for baseline test run and compared test run
jmeter_results_baseline <- read.csv("C:\\Path\\To\\JMeter's\\baseline\\results.jtl",header=T, sep=",",stringsAsFactors = FALSE)

jmeter_results_compared <- read.csv("C:\\Path\\To\\JMeter's\\compared\\run\\results.jtl",header=T, sep=",",stringsAsFactors = FALSE)

# Aggregate loaded data - we are using the 90th percentile: q90 later on
summarized_data_baseline <- ddply(jmeter_results_baseline, c("label"), summarise,
                                  sample_number = length(elapsed),
                                  mean = mean(elapsed),
                                  q90 = quantile(elapsed,probs=0.9),
                                  q95 = quantile(elapsed,probs=0.95)
)

summarized_data_compared <- ddply(jmeter_results_compared, c("label"), summarise,
                                  sample_number = length(elapsed),
                                  mean = mean(elapsed),
                                  q90 = quantile(elapsed,probs=0.9),
                                  q95 = quantile(elapsed,probs=0.95)
)

# Merge the results for both of the test runs
merged_results <- merge(summarized_data_baseline,summarized_data_compared, by="label", suffixes = c(".baseline",".compared"))

# Quantify the improvement/degradation factor in the compared test run
merged_results <- mutate(merged_results,improvement_factor=ifelse(q90.baseline > q90.compared,abs((q90.compared/q90.baseline)-1),  -((q90.compared/q90.baseline) - 1)))

# Add the distinction between the requests and transactions
transaction_identifier <- "TR" # Make the distinction -> when label starts with 'transaction_identifier', then it's a transaction
merged_results <- mutate(merged_results,sample_type=factor(ifelse(startsWith(as.character(label),transaction_identifier),"transaction","request")))

# Add the difference in response times calculation
merged_results <- mutate(merged_results,sample_size_difference=abs(sample_number.baseline-sample_number.compared)/sample_number.baseline*100)

# Construct the comparison plot
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type, size=sample_size_difference)) + 
  geom_point() + 
  scale_size("Difference in number of samples [%]") +
  geom_smooth( method="lm", se=F) +
  scale_y_continuous("Improvement / degradation in the compared run [%]", labels = scales::percent) +
  xlab("90th percentile of response time in baseline test [ms]") +
  scale_color_brewer(name="Sample type",palette="Set1") +
  labs(title="Comparison of two test runs",subtitle="Complementary JMeter results graph",caption="PerfTuned.com") +
  theme(axis.line = element_line(size = 2, colour = "grey"),axis.text = element_text(size = rel(1.1)),legend.position = "bottom", title = element_text(size = rel(1.1),face="bold"))

print(plot_comparison)

# Save the plot to file
ggsave("C:\\path\\to\\save\\graph.pdf",device="pdf")

 

Tackling huge spread in response times

There might be a case when the response times for your requests and transactions differ hugely, e.g. most are in range of 3-5 seconds, but some fall in the proximity of 40 seconds like below. That makes the chart less comprehensible.

 

compare JMeter runs - when there are huge spreads in response times between transactions

 

Fear not! We can easily make the chart more readable by changing the X-axis to log10 scale:

compare 2 JMeter runs with x-axis changed to log10 scale

To change x-axis to log10 you simply need to add one more line to your plot definition:

...
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type,size=sample_size_difference)) + 
  scale_x_log10() +
  geom_point() + 
...

 

You could also spot a color change in the graph. You are able to achieve that by changing the palette set (e.g. to one of mentioned here) or by directly inserting the color RGB definition in hex format as shown below:

...
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type,size=sample_size_difference)) + 
 ...
 # commented palette usage below
 # scale_color_brewer(name="Sample type",palette="Set1") +
 scale_color_manual(name="Sample type",values = c("#E7B800","#00AFBB")) +
...

 

Printing Request and Transaction names

Another useful feature might be showing the requests and transaction names next to the data points on the chart. That way you will at once spot what has improved and what degraded. I will show you two ways to achieve that.

 

First one involves a method from ‘ggplot2’ package used already to generate the chart which gives output like presented below:

compare results of two JMeter tests, show transaction names
 
To get such chart you only need to add one line with geom_text configuration to your chart definition. You can play with the options to find the best solution for your needs, I disabled text overlapping and decided to tilt the labels a bit on Y-axis.

...
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type)) + 
 geom_text(aes(label=label), check_overlap = T,nudge_y = 0.03) +
...

 

The other method involves installation of a separate package ‘ggrepel’. But then, you can get a really nice result with labels being automatically put next to each other to avoid overlapping. Check it out:

 

compare results of two JMeter tests, show transaction names

 

Here are the needed code snippets:

 

# Installing the new package - do it only once
install.packages("ggrepel")
...
# Loading the new library
library("ggrepel")

...
plot_comparison <- ggplot(merged_results,aes(x=q90.baseline,y=improvement_factor,colour=sample_type, group=sample_type,size=sample_size_difference)) + 
  ggrepel::geom_text_repel(aes(label = label), color = "black", size = 3, segment.color = "grey") +
  ...


Finish line?

There are still some other avenues to follow in regards to showing some extra stuff on the comparison chart, so there might be more posts coming on this topic in the future.

Of course, let me know if you have some ideas as well!