Recently, I was asked for some pragmatic guidance: If I’m not a data analyst, how do I know if I’m getting good analytics? In our business, the FDA regulates pharmaceutical promotion. The FDA insists our work presents a fair balance of benefits and risks—that we avoid suggesting any off-label use and that claims be supported by “substantial evidence.” Shouldn’t injecting a dose of analytics into our business be held to an equally high standard?
The short, bad news answer is that there’s no federally approved lab test for good analytics. The good news is that you can look for readily-identifiable symptoms.
Let me take a page from the FDA. My advice is to look for three things: 1) benefits should be accompanied by statements about risk, 2) the analyst should be aware of off-label analytics use, and 3) the analytics conclusions must be supported by “substantial evidence.” I’ll expand on each of these three check-points below.
Balance of Benefits and Risks
To minimize the risks of setting up a bad analysis, the best analysts will work to understand both your data and your business. The classic adage “what’s measured is managed” presupposes your analyst has taken the time to understand what you want to manage in the first place! If they haven’t done this step, then you’re less likely to get the benefit of good analytics.
What’s this look like in practice? They’ll start by asking about your business objectives. Expect to hear questions like: What are your top priority goals? What sort of trade-off are you willing to make between the “quality” of the visitors to your website versus the volume of visits? What steps do your patients typically follow as they learn more about an illness and adopt a treatment course? How long do people typically wait to act on getting a prescription once they’ve decided to take action? What role does the HCP or caregiver play? If you don’t hear questions like these, then something is missing.
Look for analytic support that digs into how your business works and questions assumptions. Once the analyst appreciates and understands these business features, they can suggest the best options and give you guidance regarding possible risks. Ask them what assumptions they have made, and which ones could have the most impact to skew the results. Finally, you’ll also be able to diagnose this issue based on the questions they ask you in the beginning—as much as the answers they report in the end.
Be Careful of Off-Label Use
In pharmacology, off-label use describes a physician using a drug legally for a purpose for which it has not been specifically approved. Since there is rarely a perfect set of data, the real art of analysis is knowing how well you’re generalizing from limited information. Think of it as watching out for an off-label use of a drug. The best analysts ask themselves if any boundary has been crossed in what’s appropriate—and you should make sure they’re asking themselves that question.
For example, New Year’s Eve is a traditional time to make resolutions. So moving health concerns from pre-contemplation (out of sight, out of mind), to contemplation (hmmm…maybe), and on to taking action (let’s do this!) should be easier in January. As a result, your findings using a January advertising pilot might not apply to running the program in the summer. It’s off-label—you’re generalizing the result outside of what you’ve tested.
Or it might partially apply: You’ll still get some lift, but the season might affect the best message to use. Maybe a website offering to help patients find a prescribing doctor is less effective during a summer campaign because they aren’t as encouraged by the season to take action. It’s complicated—there could be a number of impacts. But your task is simpler. You should check to see if your analyst is considering these kinds of problems in his or her calculations. Ask your analyst what his or her thinking is, and what he or she is doing to adjust for it.
In fact, an analyst can often minimize the problem by choosing the right comparison group. Or he or she can tell you what winning promotion to use—what message leads the pack—but can’t guarantee the overall absolute lift in Rx. You may not have a complete answer, but you’re often further ahead with a leading indication compared to not having anything to help you decide your next marketing actions.
In all these cases, your analyst should be thinking through the assumptions about how the program is supposed to affect people, and gauging the implications of any inconsistencies. He or she should especially be looking for seasonal effects, competitor action (who has campaigns running?), and changes in the market (in funding, generics, or even the growing prominence of mobile devices and smartphone search). In short, your analyst should be asking themselves if his or her work still applies, or is it an off-label case.
The classic standard of proof for the FDA is a randomized, double-blind, case control study. With a placebo. I think of this as the “smoking gun, eye-witness testimony, DNA-evidence” courtroom standard. The solid-gold standard for marketing analytics is similar in many key respects, and if you’re getting good analytics support, your analyst should be thinking along the same direction. Listen for it.
You’ll know it if your analyst is talking about “control groups” or “hold-back samples.” Why? Analysts want a group of people for comparison. So they’ll be encouraging you to set aside some patients or HCPs just to see what happens to an “unmarketed” group. If there are seasonal effects impacting your program—or competitor marketing or industry changes—this control group will show those impacts. Then you can be more certain that the improvement you gain from marketing really is due to the marketing, and not some lucky (or unlucky!) change that happened to occur at the same time.
That’s the gold standard—but gold is very expensive. You might not want to give up marketing to potential clients, or it might complicate your marketing execution. For lots of reasons.
A smart analyst won’t be surprised. If you’re getting good support, he or she will suggest some alternates.
Your analyst may set things up so you compare changes in activity to the same period last year. If website visits go up every January by 10%, he or she will use that as the baseline and see if you had a 13% improvement this year. Or the analyst will look at industry norms and see what’s been happening in general to open rates and click-through rates. Or he or she might use internal analysis and show that while 17% of visitors used to go on to download key drug information, now 24% do since the key website prompts were changed.
Your analyst might also divide your patients or HCPs into logical groups, and see how different segments acted. He or she will check out if it was consistent with your theory about how the marketing should affect people. For example, the analyst would check to see if patients in the “action phase” really were heading to the Find-A-Doctor web page more than other segments after being given a stage-of-change appropriate message. An analyst will use “comparison groups” rather than “control groups.” Then you can be more confident of the findings.
In short, analysts look for circumstantial evidence. Just like a court case, no single piece of it will get a conviction. But put enough of it together, and your analyst will create a convincing description that will lead to convincing and useful conclusions. Maybe you won’t have a smoking gun at the end of it, but the circumstantial evidence will still give you results you can trust.