Few years ago, JMeter has added a great html report that could be generated after a test run.
Even though the report is constantly improving, it’s still far from being perfect. I like the variety of graphs a lot, however, in my opinion, there is an issue with the graphs being only available with a specific metric, that cannot be chosen to suit user’s needs. Some examples of it:
- response time per request/transaction over the time of the test can only be presented with an average,
- transaction throughput is presented in transactions per second, which is fine, but if you choose to pace your script with constant throughput timer component, you specify pacing in transaction per minute. Hence after the test you are not able to easily see on the graph whether you reached the requested transaction rate.
Time vs threads graph
Few weeks ago I was asked to test scalability of a certain functionality in the system depending on number of users concurrently using it. I’ve created a script to test various levels in a single test and thought of the Time vs Threads graph being a really good solution to present the conclusions.
This is how the graph looked like:
When presenting it to the customer I was immediately asked about the variability in the results (turned out that there were scientists on the client’s side). While I had seen some standard deviation and percentile statistics, they were rather calculated for the whole test, not on each load level, so were not much helpful here.
To rectify that I had to come up with different solution.
R to the rescue
Having previously worked with R language, I knew that it would be a quick and easy solution to this conundrum. In just few steps I was able to process the JMeter results to obtain the needed metrics and plot them on a nice looking graphs:
- first one is similar to JMeter’s “Average response time vs threads”, but has also standard deviation,
- other one is showing a 95th percentile of response time across all level loads.
Below is the R script that you can use to create such graphs and make your demanding customer content. In case you don’t want to visualize some of your requests or transactions just add your transactions in ‘remove not needed data’ code section below or process the .jtl file beforehand.
# install the library packages - only needed to be done once install.packages("ggplot2") install.packages("plyr") # import the libraries library("ggplot2") library("plyr") # load the data jmeter_results <- read.csv("C:\\Path\\To\\Your\\Results.jtl",header=T, sep=",") # group data by "label" and "allThreads" columns (sample name and number of active threads respectively), compute mean, standard deviation, percentiles summarized_data <- ddply(jmeter_results, c("label","allThreads"), summarise, N = length(elapsed), avg = mean(elapsed), sd = sd(elapsed), p90 = quantile(elapsed,probs=0.9), p95 = quantile(elapsed,probs=0.95) ) # remove not needed stuff (if applicable) summarized_data <- summarized_data[summarized_data$label != "010_JustWarmingUpHere" & summarized_data$label != "070_NothingImportant", ] # move the position of data points a bit in order for the standard deviation bars not overlap pd <- position_dodge(0.5) # construct the graph of average response time vs threads plot <- ggplot(summarized_data, aes(x=allThreads, y=avg,colour=label, group=label)) + geom_errorbar(aes(ymin=(ifelse(mean-sd>0,mean-sd,0)), ymax=avg+sd),width=2, position=pd) + geom_line(position=pd, size=1.5) + geom_point(position=pd, size=3, shape=21, fill="white") + scale_color_brewer(name="Transaction Name",palette="Set1") + # when showing more than 9 data groups the following might be needed instead: scale_color_hue(name="Transaction Name",l=60) ggtitle("Average Response Time (with SD) vs Number of Concurrent Users") + scale_y_continuous("Average Response Time [ms]",breaks=0:20*500) + scale_x_reverse("Number of concurrent users",breaks=c(1:20),minor_breaks=NULL) + theme_gray() + theme(axis.line = element_line(size = 2, colour = "grey"),axis.text = element_text(size = rel(1.1)),legend.position = "bottom", title = element_text(face="bold")) # show the graph print(plot) # alternatively construct 95th percentile plot plot_percentile <- ggplot(summarized_data, aes(x=allThreads, y=p95,colour=label, group=label)) + geom_line(size=1.5) + geom_point(size=3, shape=21, fill="white") + scale_color_brewer(name="Transaction Name",palette="Set1") + ggtitle("95th percentile Response Time vs Number of Concurrent Users") + scale_y_continuous("95th percentile Response Time [ms]",breaks=0:20*500) + scale_x_reverse("Number of concurrent users",breaks=c(1:20),minor_breaks=NULL) + theme_gray() + theme(axis.line = element_line(size = 2, colour = "grey"),axis.text = element_text(size = rel(1.1)),legend.position = "bottom", title = element_text(face="bold")) # print the plot print(plot_percentile)