Planning for Peak with HCL

This blog is a companion piece to the Holiday Readiness Podcast by Stephanie Heyward and Andon Salvarinov from HCL Commerce Advisory Services. Listen to the first episode here.

One of the most critical aspects of preparing for any peak online period is the up-front planning of your stress/load testing.

It is quite a challenge to identify and determine all the facets that would lead to proper preparation and successful testing.  For years, the best practices around this planning we have shared in our Performance Test Plan Template. Let’s review quicky the key elements in this document that are critical to the overall success.

Understanding the Goals.

Understanding the volumes that your ecommerce site needs to sustain, and be responsive and operational, are key information to the success. These volume metrics we frequently refer to as NFRs, or Non-functional Requirements.  NFRs should capture the key metrics that describe how much the ecommerce site should be able to sustain and at what operational resource consumption. This should be targeting the peak load, usually measured within a 1h timeframe.

For example the key metrics that are defining the NFRs are:

–         Orders/hr, Pageviews/hr and Sessions/hr

–         CPU utilisation for each server type in the cluster

–         Response times of each individual page.

This should be planned and determined carefully in joint effort between the technical/operations group as well as marketing and business groups. The process usually starts with understanding the previous year peak NFRs, that can be easily captured from the existing analytics on the ecommerce site.

These numbers should be adjusted with the business plans for the coming peak as well as taking into considerations the marketing events and campaigns that will influence the peak hour NFRs.

Capturing and Understanding the User Behavior

Understanding the user behavior on the site is the second key piece of information in the successful planning and preparation for your peak. This information is usually collected by the existing analytics tools for your ecommerce site.   These analytics help define what we typically call the workload mix.  For example:

• The ratio of users that visit the site , for example guest vs registered
• The percentage distribution by page type by user type within the shopping flow. Grouped together you should consider the following pages:  Home, Category, Category List, Faceted Search, Keyword Search, Add To Cart , Shopping Cart, Checkout, Login, Logout etc.
• The Abandoned cart ratio (or ratio between all the orders created and  orders completed)
• Personalisation specifics, that the eCommerce site may have customized.

This behavior needs to be captured in the workload testing scripts, to trustfully represent the workload on the site.

Once your goals have been documented and you’ve determined the right workload for your testing scripts, and you confirmed the scripts themselves represent user behaviors, then it’s time to focus on the test itself.

Variations in Testing

Once the goals are set, many customer want to rush straight into stress/load testing.  However, it is important to consider all the variations of testing you really should do.  First you should benchmark your performance under normal load in order to have a comparison to performance at peak load.  Second, you need to know at what point degradation may start to occur beyond your peak goals and for how long the site tolerate the loads you are anticipating.

The following variations should be considered:

1. Single user testing, where the scripts are validated and errors caused by the scripts are cleared.
2. Minimal Configuration testing, where the workload is sent against minimal configuration of the eCommerce site. This workload would be a fraction of the total workload. The testing effort would focus on the Functional Correctness on the site, and making sure that the minimal system can sustain the capacity it needs to sustain.
3. Full System test, where we would test the fully operational cluster under full workload targeted for the Holliday.
4. Full System test and operation procedures that are anticipated and/or required for the Holliday
5. Longer test (aka Soak test) on the full system , going for 12 to 24 hours uninterrupted. The goal of this test is to test longer term stability of the cluster.

To help you with these steps, please use our Performance Test Planfor additional information.