Web Tortoise

2012-Oct-29

The Web Performance Hockey Stick Chart!

Response:

Hello! This #WebTortoise post was written 2012-OCT-29 at 03:30 PM ET (about #WebTortoise).

Main Points

#- Percentile-based Histograms complement time-based line charts, and should be part of your Web Performance data analysis arsenal.

#- Percentile-based Histograms look at overall Performance, independent of time.

#- Percentile-based Histograms are a good way to look at A/B test results (In Web Tortoise World, we’re looking at a change affecting various Response Times. Did the change improve or worsen Response Times?).

#- Excel Chart: Excel Hockey Stick Chart.

#- Excel VBA: Add drop lines to chart in this story: http://t.co/zSKTAy4Q

#- Define: What is a Percentile?

Story

In this story, we’re looking at whether a change in Apache improved or worsened our overall Webpage Response Times. We started by looking at a traditional time-based line chart, but there were periods during the day where the Performance lines oscillated around each other. Because of this, we couldn’t really get an idea of the overall Performance change. So, we then decided to look at a percentile-based Histogram (because we wanted an idea about the overall change (if in fact there was one)).

After making the change in Apache, measurements ran for a week. In the below chart, we’re looking at the Earlier week VS the Latter week, where the X axis dimension is based on Percentages and the Y axis is the actual Percentile values.

WebTortoise Excel Hockey Stick Chart

Reading the Chart

Here’s how you read the above chart:  Example, forty percent % of the Webpage Response Times, for both the Before and After week, were just under 1,000 ms (find 40% on the X axis, and follow it up to the corresponding Y axis value).  You can start to see the delta in the Before and After weeks get larger and larger as you move further right along the X axis.

NOTICE how the Median values are nearly the same.  If you were looking at only a single calculation versus this above full distribution, you may have said there was no Performance change at all!

Note download the excel sheet here to see the actual Percentile values.

If we’re eyeballing the above chart, might say right around 50-55’ish percent of the Response Times were about the same (or very close), before and after our change was made.  However, as we move further right along the Percentage X axis, we notice that for about 40-45’ish percent of our users, they are getting a slower Response Time!

Download the excel sheet to see the actual values for all the Percentiles.  But here are the 99% for each “Before” and “After” week:
In the “Before” week, 99% of our Response Times were below 2,299 ms.  In the “After” week, 99% of our page loads were below 4,022 ms.  We can confidently say the change we made for our A/B Performance testing *worsened* our overall Performance.

Document Complete / OnLoad:

_The following is optional reading material._

Download the excel sheet here:  https://docs.google.com/file/d/0B9n5Sarv4oonYUJFaTVRMlZpZDQ/edit?usp=sharing

LinkedIn: http://www.linkedin.com/in/leovasiliou

Twitter: @LvasiLiou

#CatchpointUser #KeynoteUser #GomezUser #ExcelOverlayChart #Overlay #CompareToPrevious #PatternChange #WebPerformance #ExcelHockeyStickChart #PercentileBasedHistogram

2012-Sep-20

Hey Hey, Check the Overlay!

Response:

Hello!  This #WebTortoise post was written 2012-SEP-20 at 03:43 PM ET (about #WebTortoise).

Main Points

#- Use programs like Excel or Tableau to get additional formatting value from your third-party-provided chart data.

#- Continually measure website Performance to identify Pattern Changes, be them by minute of hour, by hour of day, by day of week or by some other dimension.  Point-in-time measurements have value, where continually measuring will tell more of the story.

#- Use overlay charts (a.k.a. “Compare to Previous” charts) to complement time-based line charts.   Use overlay charts to also identify especially subtle Pattern Changes.

Story

Time-based line charts are probably the most popular chart type (at least in Web Tortoise world).  In them, some period of time will be along the X axis where we read it from the left to the right, matching to a corresponding Y axis value as we go.  The chart could be showing anything (e.g. from website response time by the hour to number of website hits by the day) but they are all read the same way.

When reading a time-based line chart, will generally look at the value for a given period and compare to the previous value.  Was it the same as the previous value?  Was it more or less than the previous value?  If the value was different, by how much?  Is there a blip, or is there a sustained Pattern Change?

The identification of sustained Pattern Change is the focus of this Web Tortoise post.  The other day, was fortunate to discover a sustained Pattern Change in one of my customer’s key metrics.  The problem is the Pattern Change was so subtle, it almost went unnoticed!  In this Performance Index chart (screenshot taken from Google Analytics), able to identify which day the Pattern Change occurred?  How about which hour?  No?  Then “Check the Overlay!”

Note this chart is showing a Performance Index and the values of the Y axis are materially irrelevant.  What’s to be focused on is the general shape(s) of the daily patterns.

First, download the chart data and load into Excel; then apply custom formatting.  Might say the below chart is prettier with additional formatting, but still can’t quite identify the day or hour of the Pattern Change.

Second, take the above time-based line chart and turn it into an Overlay chart.  Since this chart is for 2-weeks, by the hour, there are 336 data points (24 hours in a day * 14 days = 336).  To create the overlay, take the latter 168 data points and plot along the Primary (bottom) left-to-right X axis; take the earlier 168 data points and plot along the Secondary (top) left-to-right X axis.

Note in the below Overlay chart, using both Primary and Secondary axis titles takes up a lot of real estate.  May remove the Secondary axis title if comfortable doing so.  Note I also chose to override Excel’s auto-sizing of the chart to ensure its length and width were using the Golden Ratio.  Here again, this takes up a lot of real estate, but adjust accordingly.

Because we used some semi-advanced Excel formatting features, because we continually measured Performance and because we Checked the Overlay!, we are able to report the sustained Pattern Change started on Wednesday, August 15th in the 08:00 AM PT hour.  At this point, consider engaging Release, Change or some other Management Function to figure out why.

Document Complete / OnLoad:

_The following is optional reading material._

Download the Excel file(s) here:  https://docs.google.com/open?id=0B9n5Sarv4oonYW8zQ0NjS2lqM2c

LinkedIn:  http://www.linkedin.com/in/leovasiliou

Twitter:  @LvasiLiou

#CatchpointUser #KeynoteUser #GomezUser #ExcelOverlayChart #Overlay #CompareToPrevious #PatternChange

Blog at WordPress.com.