I was going to entitle this "Responding to responses to my response" and decided no, some might take offence (although I trust both Jason Carmel and WindKiller would not). Then I thought, "Hey, wait a second...two responses to a single post? You mean people are reading this stuff?"
Anyway, I decided to respond to both Jason's and WindKiller's comments to my Responding to "Visitor Engagement: Time for a reality check?" in a new blog post rather than continue the thread there (easier for me. Hope it's not too difficult for you out there). I'll be taking my usual format of placing their comments in italics and mine in regular text.
Starting with Jason Carmel's response...Fascinating post, Joseph.
Howdy, Jason! And thanks. I do my best.
I don't pretend to speak for Matt here, but I would submit that the "measurement so you can improve it" premise is valid, if it is unpacked a certain way (I fear that semantics may be leading us to confusion- a subject you would have far more insight into than I).
I would agree with you and I'm pretty sure nothing in my original response (pick your version, Responding to "Visitor Engagement: Time for a reality check?" here on BizmediaScience or Responding to "Visitor Engagement: Time for a reality check?" on TheFutureOf
<UPDATE>
That blog no longer exists in its original form. The blog name is still there and it's no longer what I'd hoped it would be. All of my FutureOf contributions can now be found on TheAnalyticsEcology.
</UPDATE>
) indicated the original premise wasn't valid. I do believe the premise offered was incomplete, though. I'm guessing this might be what you meant by "...if it is unpacked a certain way...". I also believe that incompleteness is a flaw in the reasoning offered. How severe a flaw is dictated by the measurement models in use and the paradigms in which these models are enforced. More colloquially, when throwing the baby out with the bathwater, first make sure a) you won't be needing the child at a later date and b) that the child won't grow up to become a very large, hairy primate.
The goal of measurement for our clients is to take action on the findings and to improve stuff, ...
Same here. Comforting, that, yes? That we're both in the business of providing actionable information to clients, yes?
...much like the goal of a clinical research oncologist may be to discover a drug that cures cancer. However, the doctor's research may not actually improve anything- it could simply (and importantly) add to the body of knowledge that allows for her, her colleagues, her students and her successors to support and refute hypotheses that lead to a proposed treatment.
Accepted.
As you correctly point out, the first step for both web analyst and (I would assume) oncologist is to understand whether something ab initio is measurable. Once a measurement is validated (which is probably the subject of a whole different discussion), the researcher embarks on a series of educated guesses (hypotheses) based on a holistic understanding of the system, to evolve the body of knowledge about that system to the point where (for a website) stuff can be improved, or (in the oncologist's world) a treatment can be derived. In the case of an oncologist, many years of tests (and retests) may be needed before one proposed treatment is offered for clinical trials, while a web analyst will need to perform a great deal of baseline studies and modeling before even proposing an hypothesis aimed at improving a "good" metric.
Hmm...umm... I think there are a couple of flaws in the above. The flaws might be on my interpretation of what is offered. My interpretation of the above might be hindered by having someone on staff with about a dozen or so years of clinical and therapeutic drug discovery research to her credit (along with being cited on a couple of patents. Yes, NextStage has an interesting mix of personnel).
Oncologists come up with a proposed treatment. This then goes through animal trials with animals that elicit the same type of cancer or with tissue models. Then it goes into experimental trials -- a call for volunteers for experimental treatment (to work dosages out, etc) and study. These volunteers know they could die. I doubt if any of us would be in business long if we told prospects, "By the way, acting upon this metric might send your company into financial ruin."
Next the treatment goes to the FDA for clinical trials (blind studies, much grander scale). It goes into the real world. It still goes through a great deal of improvement because it's now being used under conditions that could never be anticipated by any single or group or researchers. At this point the treatment could be removed, modified, etc. In some cases we learn the pharmaceutical is so safe it becomes an OTC (this has yet to happen in oncological studies).
I'll accept your statement if this is the parallel you're drawing and recognize the caveats inherent above.
Also and all cards on the table, all of NextStage's reports came from clients asking us if we could tell them something (how many men visit my site in a day? How many women? Why aren't women buying more online? ...) so in a way NextStage fits more the oncology model than the traditional web analytics model. Also and yes, we do provide suggestions for how to get more men, more women, increase the sales of this, that, ... (The FDA has approved us, so to speak, and we're still improving our treatments).
My main concern with Matt's premise generally, and the "improvement" element specifically (which you touch on in your Factor A example), is that it implies that a test which fails to result in a measurable improvement is a failed test. The negation of an hypothesis can be massively helpful to further a course of study. The Greeks looked at the shape of lunar eclipses and negated the premise that the world was flat. This resulted in a far more focused study of geodesy and geography than would have occurred had that original test been merely inconclusive. By creating and then negating an hypothesis, a good web analyst or test designer should be able to refocus resources on paths that are more likely to bear fruit. These may not be the tests that are given credit down the road for a 20% increase in purchase (as one example), but it is certain that the test which does result in significant improvement could not have happened absent the learnings that came before it. I would imagine that Matt agrees with this as well, but the phrasing of his premise may mislead in that respect.
That's a beautiful and cogent argument, Jason, really and truly, and thank you for it. I'm guessing we're not going to get into the suppression any knowledge that didn't fit into the existing hegemenical political power structure, yes? I ask this question because -- although not a part of the original discussion -- I believe the analogy can't be ignored in the ecology under study.
I do not subscribe to a measurement-for-measurement's-sake policy for web analytics. As Jason Burby is famous for saying, analytics without action has a return-on-investment of zero. But I don't think that a measurement which proves difficult to affect should be shelved. The road from measurement to improvement is often unpredictable, and a good analyst will occasionally investigate seemingly unrelated or unchangeable metrics for the chance to find a new correlation (if not out of sheer curiosity). Most of the analysts I work with at ZAAZ spend a good amount of time investigating the unchangeable and irrefutable either to map the nature of those metrics more clearly or to challenge the assumptions that define them. I find that this is a fantastic way to spend time learning about the system in general, and without question results in a better analyst.
Really? You have any job openings? You need a dedicated researcher with a proven track record of figuring out how to measure the impossible and turn it into profit? Able to answer complex questions with a single word? Look, up in the sky, it's a bird, it's a plane, ...
Which brings me to the point (I think). Measurement should be taken, broadly, with the goal of improvement, but that is accomplished through repeated experimentation to learn about the system. It is through only through iterative, focused, and scientifically rigorous testing and remeasurement of the system that improvements will come.
I think we are in high agreement here.
Web analysts would do well to understand what oncologists have learned for decades-that this is a process that takes time, requires resources, and demands very specialized attention.
Again, high agreement.
[P.S. In no way am I trying to equate the importance of web analytics relative to cancer research. It's an analogy. I hope everyone gets that.]
Duly noted, Jason. And thanks for this great response. I love conversations like this. My concern with the introduction of the oncological research as an analog to web analytics is that rarely (we hope) does a treatment get released to the public in anything but a final form (or so the FDA would have us believe). My understanding of web analytics is that there is no "final form" of a measurement or a metric. The WAA Standards committee is doing their utmost and without an FDA-like authority to impose regulations ...
Now to WindKiller's response...
I think one of Mr. Belkin's problems is that he is only aware of two types of math in the web analytic world. The first is math that can be completed by the average American fourth grader (page views/hour, visitors/sales, etc.). The second is math that incorporates numeric interpretations of subjective data (visit duration > 3.5 minutes = 1,
Just so it's on the record, NextStage is web analytics agnostic. Some client requests require us to use some traditional web analytics data and we don't care what tool is used, only that we can validate the numbers with our own methods. Some of this data is in the forms WindKiller describes above.
However, there is a wonderful spectrum of math that extends beyond the understanding of the average American fourth grader, yet remains useful for understanding and analyzing data. Integrals, standard deviations, differential equations, and various other calculations that can be used to understand real data, but these are calculations whose value may not be readily apparent to the average decision-maker.
This I recognize greatly (see Communicating Science to Business and Vice Versa or Responding to Ms. Pascoe’s 18 July 08 1:17pm comment). What I'll offer is a bit of a compromise with WindKiller's statement; decision makers don't need to know how a suggested action is derived (the possibly complex math involved) only that there's good solid reasoning behind it.
I was going to write "...only that the suggested action has demonstrated an X% success rate in the past in ..." and then was going to get into congruency analysis between test and real-world systems and you get the idea, right?
What this message from Omniture is telling me is their service is not going to perform any mathematical calculations that cannot be performed by even the most dysfunction IT staff.
Umm...ouch! Not even going to touch that.
Please contact NextStage for information regarding presentations and trainings on this and other topics.
Sign up for the NextStage Irregular, our very irregular, definitely frequency-wise and probably topic-wise newsletter.
