Sometimes when looking at the results from survey data, we see something that makes us say ‘huh?’ or ‘that doesn’t look right’. When the odd results persist after verifying the data were processed correctly (always a good practice), there is typically still a logical answer that can be uncovered after doing some digging. Sometimes the answer lies with something that we will call ‘unusual statistical phenomena.’  This is part 2 of a series that will look at some of these interesting – or confounding – effects that do pop up now and then in real survey research data.

This time we will look at an unusual phenomenon that can occur when doing something typically considered fairly mundane – testing for statistical significance between percentages. An example will help to illustrate this phenomenon which periodically causes us to question stat testing results.

Let’s say we have fielded the same survey for two different brands. One part of the survey collects respondent opinions of the test brand using a battery of attribute statements with a 5-point agreement scale. The base size for each survey was 300.

Stat testing was conducted between results for the two brands for Top Box percentages on each of the attribute statements. However, some of the results are questionable. Specifically, for the attribute “Is Unique and Different” Brand B’s score was higher than Brand A’s by 4 percentage points, which was statistically significant at the 90% confidence level (denoted by the “A” in the chart below); while for the attribute “Is a Brand I Can Trust” Brand B’s score was higher than Brand A’s by 6 percentage points, which was NOT statistically significant at the 90% confidence level. How could this be!

How can a difference of 4 points be statistically significant while a difference of 6 points is not, even with the same base sizes? To understand how this can happen, let’s first look at the basics of how a statistical test for comparing percentages works.

First, a t-value is computed according to this formula:

Then this t-value is compared to a critical value. If the t-value exceeds the critical value then we say that the difference between the percentages is statistically significant.  The critical value is based on the chosen confidence level and the base sizes of the samples from which the percentages were derived.

In our example, we chose the 90% confidence level for both statistical tests and the base sizes are the same, so the critical value for both tests is the same. We also know the difference between the percentages (the numerator of our equation) is what appears anomalous as the difference of 4 led to a t-value that exceeded the critical value, while the difference of 6 did not exceed the critical value. Therefore, the issue must lie with the Standard Error of the Difference.

Let’s next examine what a Standard Error represents. Our surveys were fielded among a sample of the overall population. If we sample among women 18 to 49 in the United States, we will infer that our results are representative of the entire population of interest, which is all women 18 to 49 in the United States. However, it is unlikely that the measures we compute from the sample (such as the percentage that say Brand A “is a brand I can trust”) will be exactly the same as the percentage would be if we could ask everyone in the entire population of interest.  There is some uncertainty in the result because we are asking it of only a subset of the population. The Standard Error is a measure of the size of this uncertainty for a given metric.

In our equation, the denominator is the Standard Error of the Difference between the percentages. While not precisely correct, the Standard Error of the Difference can be thought of as the sum of the individual Standard Errors for the two percentages being subtracted (the actual value will be somewhat less due to taking squares and square roots). As the graph below illustrates, the Standard Error for a percentage is a function not only of the sample size, but also of the size of the percentage itself.

Specifically, for any given sample size the Standard Error is largest for values around 50% and decreases as values approach either 0% or 100%. For a base size of 100 (the dark blue line), the Standard Error is close to 5 for percentages near 50%, but decreases close to 2 for very small or very large percentages.  You can think about this as it being harder to estimate the percent incidence of a characteristic of a population when around half the population has that characteristic versus when almost all (or almost none) of the population has that characteristic.

In our example, the percentages for Is a Brand I Can Trust are close to 50%, so at a base size of 300 the individual Standard Errors would each be a little under 3. In contrast the percentages for Is Unique and Different are around 10%, so at a base size of 300 the Standard Errors would each be around 1.5.  That’s a big difference!

It follows that the Standard Error of the Difference for Is a Brand I Can Trust would be much larger than for Is Unique and Different. In fact, the actual values are 4.08 for Is a Brand I Can Trust and 2.34 for Is Unique and Different. Again, a big difference. If we divide the differences in the percentages by these values for Standard Error of the Difference, we get t-values of 1.47 and 1.71, respectively. Given the critical value is approximately 1.65, we see that the t-value for the difference of 6 is below the critical value (hence not statistically significant); while the t-value for the difference of 4 is above the critical value (hence is statistically significant).

Hopefully this takes some of the mystery out of stat testing and helps in understanding why what can appear to be anomalous results may actually be correct.

The Journal of Advertising Research just published a terrific paper that MSW was proud to participate in, titled…

Effectiveness and Efficiency of TV’s Brand-Building Power: A Historical Review – Why the Persuasion Rating Point (PRP) Is a More Accurate Metric than the GRP
Frank Findley, Kelly Johnson, Douglas Crang, David W. Stewart
DOI: 10.2501/JAR-2020-011 Published 1 June 2020

ABSTRACT
This article examines the effectiveness of television advertising and changes in television-audience response in the United States since the 1980s. It concludes that television remains one of the most effective platforms for advertising, despite the rise of digital media and new technological developments. On a single, quality exposure basis, television advertising continues to be highly effective, although the rate of delivery of advertising selling power per gross rating point (GRP) has declined, but the decline is mitigated by the increasing number of households in the United States. Television advertising remains effective despite the potential increase in distracted viewing, but advertisers need to manage the quality of their messages and the media weight of their advertising more carefully than in the past. The persuasion rating point (PRP) offers an accurate measure of that effectiveness.

Contact us for a full reprint of the article.

In times of uncertainty, economic, health or otherwise, many companies reduce or eliminate “discretionary expenses”.  It’s very easy to say now is not the time to think about Marketing, put brand plans on hold and freeze all spending – cut research & development, cut new product introductions, and cut advertising spend.  Are these prudent moves to keep a company operating profitably?

As the world is in the grip of this Coronavirus pandemic and many of us are hunkering down at home, taking care of our families, trying to occupy the kids and trying to stay calm; C-suite executives from all over the world are struggling with this very question.  For many of them marketing is the last thing on their mind.

Unfortunately, we’ve seen this before; this isn’t the first or the last time that we’ll face a crisis like this.  There’s a lot of precedent to lean on and we as marketers and market researchers have had the opportunity to reflect and study the events of the past and learn from them.

This is just a small sampling of recent “challenging” times:

Year	Type	Name
2019	Health	Wuhan coronavirus outbreak
2018	Natural	Camp Fire
2017	Natural	Hurricane Maria
2016	Health	MERS
2015	Health	Zika virus outbreak
2014	Health	Ebola virus epidemic in West Africa
2013	Health	H7N9 (Avian flu)
2012	Natural	Hurricane Sandy
2011	Health	Tōhoku earthquake and tsunami
2010	Health	Haiti earthquake
2009	Natural	2009 Samoa earthquake and tsunami
2009	Health	N1H1 (Influenza)
2009	Health	Pandemic H1N1/09 Influenza
2007	Economic	Subprime Mortgage Crisis
2005	Natural	Hurricane Katrina
2004	Natural	Indian Ocean earthquake and tsunami
2004	Natural	Hurricane Ivan
2004	Natural	Hurricane Frances
2004	Natural	Hurricane Charley
2004	Health	Avian influenza (H5N1), sometimes avian flu, and commonly bird flu
2003	Health	Severe acute respiratory syndrome (SARS)
2002	Economic	Dotcom Bubble
2002	Health	SARS
2001	Terror	9/11
2001	Health	Anthrax attacks in the United States, also known as Amerithrax
1996	Health	Bovine spongiform encephalopathy (BSE), commonly known as mad-cow disease
1995	Natural	Chicago heat wave of 1995
1994	Natural	Northridge earthquake
1993	Natural	Storm of the Century
1992	Natural	Hurricane Andrew
1987	Economic	Black Monday (Stock Market Crash)
1981	Health	HIV/AIDs epedemic

Having studied these events, research shows that scaling back, not only has significant downside but, in many cases, represents enormous missed opportunity.

History has repeatedly demonstrated that now is not the time to scale back. Companies that have the most success during challenging times such as these, are the ones that are able to maintain their presence among consumers and continue to build their brands regardless of temporary conditions.

The current Coronavirus crisis is a health crisis that is morphing into a full-blown economic crisis that is exacerbated by the fact that it has come on the heels of and effectively ended one of the longest running bull markets in history.  That said, it is a temporary condition that will pass.  Unfortunately, it will likely get worse before it gets better, but rest assured it will pass.  What we have seen during past crises is that companies that step up are rewarded after the recovery.

A common assumption is that, customers will be spending less; and therefore, money spent on advertising will be wasted. The fact is families will continue to seek food, entertainment, prescriptions, autos and other goods and services.  And even if they do scale backing spending somewhat, they will then exhibit pent up demand during the recovery period.

During the 2007-2008 economic crisis that was kicked off by the Subprime Mortgage Crisis the Commerce Department reported that consumer spending rose 0.4 percent in January, 2008. This continued throughout the first quarter of 2008 as retail sales in the United States increased 0.4 percent for the week ended March 15. On a year-over-year basis, sales grew 1.6 percent (The International Council of Shopping Centers-USB).

Most research on the topic suggests advertising in difficult economic times can actually provide significant opportunity to capture market share.

In a study of U.S. recessions, analyzing 600 companies from 1980 to 1985, the results showed that business-to-business firms that maintained or increased their advertising expenditures during the 1981-1982 recession averaged significantly higher sales growth, both during the recession and for the following three years, than those that eliminated or decreased advertising. By 1985, sales of companies that were aggressive recession advertisers had risen 256% over those that didn’t keep up their advertising. (McGraw-Hill Research)

Businesses that aggressively increased media advertising expenditures during the last recession (just 25% of all businesses) increased their market share 2 1/2 times the average for all businesses in the post-recession period (CARR Report, Aug 13 2001).

Another study documented a 1.5 point increase in market share among businesses increasing ad spending during recessionary periods. By contrast, during expansion periods, 80 percent of businesses increased advertising budgets with no improvement in market share, since most competitors did the same thing (Cahners and SPI, 2002).

Of course, not all brands have the option to increase their ad spending during times of uncertainty. However, by maintaining ad spending and spending more intelligently, while competitors are cutting back, you will increase your voice and ultimately pick up business in doing so.

One thing to note about this current crisis, which is a combination of health and economic, is that the economic side is acting differently than in previous recession conditions.  In a recent conversation, a colleague of ours, Frank Findley, Director of MASB made an interesting observation and Frank knows our data well – he used to work for us.

During true recessions, the behavior we’ve seen has primarily been about price and buying down.  People choosing to conserve; spending and buying less.  In this situation we are seeing distribution disruptions.  Empty shelves, stores closed, people afraid to go to the stores.  People aren’t able to find their favorite brands in some categories on the shelf because they are sold out and/or people are hoarding.  This is forcing switching behavior.  Forcing them to buy (and thus try) other brands.  This is a very dangerous situation for brands – it looks like sales are through the roof – but with different customers than they had before the crisis and as people try the other competitors they might not switch back.  Add to this that when it does end there is a possibility of a stock-up phenomenon where people won’t continue to buy but use up what they’ve hoarded.  This will create a hiatus after which people may be more open to trial.

Brands really need to think about what comes next and take actions to shore up brand preference and brand loyalty.  And at the same time, one thing that is especially critical for brands to do is, fill the distribution channels as quickly as possible even if that increases operational expenses.  Make sure that customers can find their preferred brand to ward off forced switching/forced trial.

How can you maintain advertising and come out ahead?

Regardless of the economic times, responsible advertisers should work to get the most out of their advertising investments by applying proven methods to increase the ROI of their ad expenditures. During difficult economic times, it is even more critical for advertisers to avoid the trap of making decisions that have proven to have adverse impact on sales; such as not refreshing creative, wasting money by flighting versus continuous spending, and running away from the reach benefits of TV.

Since the reality is that despite strong objections by marketers, corporate marketing spending is not going to increase (and is likely going to be cut at some level during a recession), the key question is “If I have to cut back or maintain ad spending, how do I ensure my expenditures are effective and put my brand ahead in the marketplace?”

Here’s what you should be doing immediately:

1.  Quickly reevaluate your messaging.  Don’t just blindly plow forward with your pre crisis messaging.  Make sure campaigns and messaging don’t look insensitive, opportunistic, completely inappropriate or in bad taste given the climate of the crisis.  For example, if you were Coors Light you probably wouldn’t want to run the “Official Beer of” March Madness campaign you had planned in light of the cancellation of virtually all sports in the country.

2.  Test new messaging to make sure it is appropriate given the climate of the crisis.  Again, making sure new messaging isn’t seen as insensitive, opportunistic, inappropriate or in bad taste.

3.  Track progress of new messaging and the recovery.  New messaging that’s particularly timely and related to or focused on the crisis can wear out faster than typical messaging or along the same curve as the recovery so you’ll need to be ready with a recovery message.

4.  Track for the environment.  Your current tracker might not be designed to capture the nuances of the current environment.  Make sure the cadence is right and that you’re capturing:

•  Saliency

• Relevance

• Perceptions

• Preference

• Purchasing

Most importantly make sure that your tracker is built with the MSW Trigger System without it, brands are flying blind.

What you should do next; 4 steps to understanding how to make what you have work to maximum capacity.

1.  Aggressively utilize 15-second ads – but make sure they are “good” ones:

•  15-second ads are just over half the price of 30-second ads, but on average three-quarters the strength (so, you might want to use multiple 15-second ads to maintain on-air strength).

•  Over 25% of the time, 15-second ads are as strong or stronger than their 30-second counterpart (when you know you have this opportunity, you win).

•  Use “cut-down” versions from 30-second ads to reduce production costs in multiple execution plans while simultaneously boosting GRPs – but when cutting down the ads, clearly understand the drivers of success and don’t cut them.

•  Use original 15-second ads to replace 30-second ads in single execution plans (build your 15second ads “from scratch” with the end in mind that they will be 15-second ads).

•  15-second ads are best used when there are clear-cut benefits/messages and when the goal is to maintain brand awareness and loyalty.

2.  Understand the persuasive life of an ad and manage the wear-out of your advertising:

•  As money is spent behind an ad, its selling power decreases in a predictable fashion – leading to diminishing sales returns. If you know the starting strength of the ads in your campaign, you can plan to maximize delivery of selling power and ad refreshment without wasting money.

•  A very slight difference can result in a “new” ad; therefore, ad poolouts are a very cost effective way to replace ads without producing totally new ads.

•  In contrast, ads are frequently replaced prematurely, when there is plenty of power remaining. If you know the starting strength of your ads, you can avoid wasting money on replacing ads too soon.

•  Know your end point business objectives. If you know the strength of your ads, you can marry spending patterns to project likelihood of hitting your goals via simulation technology.

3.  Take advantage of emerging touchpoints and synergies

•  While TV is often the strongest single-reach element of a multi-media plan, other touchpoints can be just as effective at motivating consumers and are often less expensive (e.g., print and web).

•  Take advantage of synergies between executions by placing media spend behind those combinations yielding the greatest total impact.

•  Avoid negative interactions (often the result of unexpected executional issues).

•  Both single and multiple message campaigns can be effective:

ο  Single message when a straightforward brand-differentiating message which appeals to a broad consumer segment has been identified.

ο Multiple messages when brand differentiation cannot be communicated with a single straightforward message.

4.  Flighting vs. continuous airing? Don’t go black!

•  In an attempt to optimize media spending, advertisers sometimes spend advertising in waves figuring that periodic spending in bursts is the key to driving sales. This is not the case.

•  If reducing media spend, avoid flighting, continuous spending even at lower levels is more effective.

Advertise Smarter

To improve the odds of success in this environment, marketers must understand how to make what they’ve got work to maximum capacity.  Doing this will involve expenditures for consumer measurement and technology, but if chosen wisely, the returns are well worth it in understanding what to do once you’ve been asked to maintain or cut your budget.

The end result is simple: when the these times of health and economic uncertainty end, your return on marketing spend will be multiplied.  While your competitors cut back their ad spending, and you maintain yours while making smarter decisions, you increase your voice among them. By knowing this, you are already ahead.

Lastly, for the good of the country and your fellow citizens you should continue marketing to stimulate the economy and keep the economic engine of our country and the global economy running so we all have food on our tables.