Brian, the digital marketing manager responsible for web operations, has been working with his UX team to optimize the Request a Quote process on the website. This process is the most important lead generator in his marketing organization as it is present on every product web page and generates millions of dollars in marketing contribution. Brian and his UX team have implemented some great ideas in the new process, which has been live for some time now. Brian’s manager Joan, the director of digital marketing has asked Brian to show the impact of the optimization on the conversion rate with a high degree of confidence.

While Brian can look at the conversion rate before and after the change on the form, he is not 100% sure whether the results are statistically significant.

Brian seeks the help of a data analyst, Juan, who has advanced knowledge of statistics to run an analysis and determine whether the conversion rate changes are statistically significant. Let’s see what they discover.

Goal

Determining whether the optimization of the website quoting process led to an improvement in the conversion rate of web traffic on product web pages to quotes (since quotes are a leading indicator of orders).

An important KPI monitored in marketing is the conversion rate on the quoting process. The conversion rate is defined as how many unique visits on the product web page resulted in quotes.

Conversion rate = # of web quotes for a product / # of unique visits on the product web page in that same time period

Imagine a situation of changing the quoting process to improve the user experience.

Hypothesis testing can be used to determine whether the new quoting process will have a statistically significant improvement in the conversion rate. This is a critical business decision because web quotes are a strong leading indicator of orders and revenue. If web quotes decline, it might lead to a decline in revenue in the future.

Pain Points

Sometimes just measuring changes in the conversion rate might not be enough. The data before and after might have a different sample size, sampling errors, or sampling mix (traffic variations by products, countries, etc.). As a result, one of the two situations will develop:

• Seeing an improvement in the conversion rate, but not sure whether it is statistically significant (valid) or caused by some other reason
• Not seeing an improvement in the conversion rate, and not sure whether the decline is caused by some other factor

Idea

Scientific decision-making (hypothesis testing) can be used to determine whether the new quoting process had a statistically significant improvement on the conversion rate before making the changes permanent. 

Decision to be Made

Did the new quoting process lead to an improvement in the conversion rate?

Learn more about how to prove or disprove this hypothesis in our course Using Data Analytics for Decision-Making in B2B Marketing.