Response:

Hello! This #WebTortoise post was written 2012-OCT-29 at 03:30 PM ET (about #WebTortoise).

Main Points

#- Percentile-based Histograms complement time-based line charts, and should be part of your Web Performance data analysis arsenal.

#- Percentile-based Histograms look at overall Performance, independent of time.

#- Percentile-based Histograms are a good way to look at A/B test results (In Web Tortoise World, we’re looking at a change affecting various Response Times. Did the change improve or worsen Response Times?).

#- Excel Chart: Excel Hockey Stick Chart.

#- Excel VBA: Add drop lines to chart in this story: http://t.co/zSKTAy4Q

#- Define: What is a Percentile?

Story

In this story, we’re looking at whether a change in Apache improved or worsened our overall Webpage Response Times. We started by looking at a traditional time-based line chart, but there were periods during the day where the Performance lines oscillated around each other. Because of this, we couldn’t really get an idea of the overall Performance change. So, we then decided to look at a percentile-based Histogram (because we wanted an idea about the overall change (if in fact there was one)).

After making the change in Apache, measurements ran for a week. In the below chart, we’re looking at the Earlier week VS the Latter week, where the X axis dimension is based on Percentages and the Y axis is the actual Percentile values.

Reading the Chart

Here’s how you read the above chart:  Example, forty percent % of the Webpage Response Times, for both the Before and After week, were just under 1,000 ms (find 40% on the X axis, and follow it up to the corresponding Y axis value).  You can start to see the delta in the Before and After weeks get larger and larger as you move further right along the X axis.

NOTICE how the Median values are nearly the same.  If you were looking at only a single calculation versus this above full distribution, you may have said there was no Performance change at all!

Note download the excel sheet here to see the actual Percentile values.

If we’re eyeballing the above chart, might say right around 50-55’ish percent of the Response Times were about the same (or very close), before and after our change was made.  However, as we move further right along the Percentage X axis, we notice that for about 40-45’ish percent of our users, they are getting a slower Response Time!

Download the excel sheet to see the actual values for all the Percentiles.  But here are the 99% for each “Before” and “After” week:
In the “Before” week, 99% of our Response Times were below 2,299 ms.  In the “After” week, 99% of our page loads were below 4,022 ms.  We can confidently say the change we made for our A/B Performance testing *worsened* our overall Performance.

Document Complete / OnLoad:

_The following is optional reading material._

Download the excel sheet here:  https://docs.google.com/file/d/0B9n5Sarv4oonYUJFaTVRMlZpZDQ/edit?usp=sharing

LinkedIn: http://www.linkedin.com/in/leovasiliou

Twitter: @LvasiLiou

#CatchpointUser #KeynoteUser #GomezUser #ExcelOverlayChart #Overlay #CompareToPrevious #PatternChange #WebPerformance #ExcelHockeyStickChart #PercentileBasedHistogram