This post is a response to Visitor Engagement: Time for a reality check? posted by Matt Belkin, 14 July 08 on Industry Insights The Omniture Blog. A friend sent me the link and asked for my thoughts. I read the post, formed my thoughts, asked some folks to review what I wrote, did a little editing and went to post it today. Sometimes (usually) I don't do links and such very well and they don't show up in comments I leave, hence I'm also posting my rsponse here as I know links and such will be preserved.
Anyway, what follows is what I posted as a comment to "Visitor Engagement: Time for a reality check?" (hopefully with links, etc., in tact).
Hello,If I read the paragraphs under the The Basic Premise of Measurement heading correctly the logical outcome is that the only measurements being made (or provide value) are those that are subject to intervention ("improve").
I have to admit that confuses me. I agree that lots of things are measured with the goal of acting upon the results of the measurement. I also recognize that lots of measurements are made simply to know what they are and with no realistic hopes of being able to modifying what's being measured. I did look in several texts (started with Horst De la Croix's Psychological Measurement and Prediction, went to Sabins' Remote Sensing: Principles and Interpretation (I've often thought this discipline most closely demonstrates concepts and principles that would be valuable to people doing web analytics. I mean, consider Sabins' opening statement to the 2nd edition, "Remote Sensing is broadly defined as collecting and interpreting information about a target without being in physical contact with the object." Admittedly website visitors aren't the subject of the book and the theoretical constructs often apply, I think), skipped over to Fukunaga's Introduction to Statistical Pattern Recognition with lots of side trips into other fields) to find something similar to "The basic premise of measurement is that you want to measure something so you can improve it, if necessary."
The best I could come up with was that the basic premise of measurement is to determine if something is measurable. I think the reframe is valid and leads to some necessary concerns.
Example: Some factor, A, is recognized and known to exist within a system (a website). A measurement is taken. Modifications are made to the system and another measurement is taken. But factor A hasn't changed. Perhaps something else has changed and not factor A, the recognized and known factor of concern.
So we recognize that whatever is being modified based on the above measurement has no effect on factor A hence whatever is being measured is not factor A nor does it contribute substantially to factor A.
Yet still we want to understand factor A and be able to effect it at will. Other measurements are made and only of things we know we can improve. Nothing in our existing measurement framework seems to affect factor A.
Shall we continue? Shall we pursue other metrics? Shall we investigate until we can produce a "this = that" metric that has statistical validity?
"The basic premise of measurement is that you want to measure something so you can improve it, if necessary." Thus the answer is no, we neither pursue nor investigate.
Nothing in our existing repertoire successfully produces a "measurement so you can change it" tool for factor A therefore we can conclude any of a) factor A doesn't exist (even though it is recognized and known), b) it exists and is unmeasurable, c) the existing toolset does not include a tool with the ability to take a measurement of factor A such that changes can be made to the system that affect factor A, ...
However, the fact that what is measurable (and actionable based on those measurements) is changing is quite true. The horizon of metrics meeting the initial conditions of measurable and actionable is, like the observable universe's, growing every moment.
And I freely note that I'm a researcher, not a web analyst. I did ask one of our researchers, someone with a long history of lab work, and they offered that the basic premise of measurement is to determine the fundamental characteristics of what is measured.
"It's pretty simple -- collect data, analyze, improve. I love this because of its simplicity and objectivity." But if the only data being collected and analyzed is based on factors we know we can change then there is no objectivity in the measurement or method involved. The only factors being analyzed are those recognized as changeable in the existing paradigm (the toolset).
Let me offer an example that demonstrates moving from measurable but unactionable to measurable and actionable: We've been intentionally collecting meteorological data for about 250 years. There was nothing in that toolset that could affect climate until recently. We also developed the ability to collect meteorological data going back far into prehistory (I can provide references to paleoclimatology if necessary. A quick search of Science provided some 1012 references going back to 1900).
Thus things were measurable. Without being able to measure them we could never have learned a) that they did indeed exist, b) that they were affecting us or c) how to affect them. Thus does our observable (web analytics) universe grow from measurable but unactionable to measurable and actionable.
"...you introduce a level of abstraction on the data. You "dumb it down" introducing bias and subjectivity." I agree with this to a point and offer Communicating Science to Business and Vice Versa for your reading pleasure. I also recognize that all web analytics solutions offer training and certification on their toolsets.
Are these trainings and certifications offered to insure that all abstraction, bias and subjectivity are removed when a consultant or in-house analyst uses a given toolset? I want to take part in a web analytics tool class where the word "interpret" isn't used.
"So what happens when you start combining metrics into uber-formulas ...? That model breaks, because you introduce a level of abstraction on the data." This also confuses me. Given a formula that has several elements and each element is known to be valid and measurable then the sum of the elements is also valid and measurable. This is basic mathematical logic (distributivity); if x is true and y is true and conservation of units holds then (x + y) is true. This principle (not by name) is mentioned in this post itself when the subject is RFM.
"First, what kind of return frequency is “often” - two visits? Four? Six? That’s subjective. What is “important” content? The home page? An article? A support document? Subjective again. And what is a “long” time on site — 5 minutes, 10 minutes? Perhaps “long” means any visit that exceeds the average for the site that week?" This logic reminds me of one of my favorite jokes:
A man wants to know what 2 + 2 is so he goes to a mathematician and asks, "Professor, what is 2+2?" to which the mathematician replies, "How may decimal points do you want it out to?" He then finds a psychiatrist and asks, "Doctor, what is 2+2" to which the psychiatrist replies, "Interesting. How did your mother treat you when you were two?" He then goes to an economist and asks, "Sir, what is 2+2?" and the economist answers, "What do you want it to be, my boy?"
I do agree that the types of values you reference need to be defined, qualified and quantified (and my experience is that they usually are by the business model of the organization requesting the metrics to be applied) before being used in any measurement and resultant decision process. This is something I believe Nathan Janitz states in his comment(s). I also believe that to suggest subjectivity is not part of the business world -- even when it comes to something as definite as well-defined measurements -- is naive. To that point, suggesting that "often", "important" and "long" implies subjectivity without recognizing that the conversation drawn from was not meant to have any rigor is amusing, hence my offering of the above joke. This goes back to the reference to RFM. If the underlying (or "principle") metrics are valid then the uber-metric is valid.
And I also recognize that my paradigm might not be appropriate here. I do think the core concept of measurement (of anything) is scientific in nature. Ms. Debbie Pascoe writes in her Continuing the Discussion with Joseph Carrabis "Anyone who has ever used a tire gauge or a tape measure has employed a scientific calibration method." Hence I offer Paul Davies' "A scientific claim is taken seriously only if it can be tested by others in a disinterested (not uninterested) way."
The true test of any measurement is that it can be duplicated when attempted under similar situations with similarly calibrated tools. That concept, I believe, is where true simplicity and objectivity will demonstrate themselves first. After that is achieved, companies that make measurement tools will find a way to make the measurements personally relevant to consumers (see Framing Science). This, I think, hearkens back to Ms. Pascoe's comments and thoughts.
Jim Novo offers "Simple to explain to just about anyone..." and I worry if our decisions on usefulness are going to be based on simplicity of explanation. If so, then everyone who doesn't understand solid-state physics, magneto-interference pattern imaging, magneto-optical coding, lasers, satellite telemetry, ... oh, heck, let's just go to basic electricity... throw away your digital cameras, computers, pdas, smartphones, stop watching TV, move out of your homes completely, never again use cars, buses, boats, trains or planes, ...
But (!!!) if the decisions on usefulness are to be based on simplicity of use? That's a completely different story. My experience is that most people don't care (or even care to know) how complex something is "under the covers", they just want to know that "when I do this, that happens" and even more accurately "when I do this, I get what I want" (this is the history of technology concept I often share (see The Long Tail, Part 1, Rocks, Hammers, Competition and How People Get Left Behind or VerizonWireless' 20 year plan)).
Bravo to some and not all of what Steve Jackson adds. In the end, any metric that provides value will survive, even if that value is simply that something is being measured.
I also appreciate Nathan Janitz statements. His statement "...you also can’t catch a mouse without looking at the right information….all of the right information" is, I think, exactly to my point.
Allow me to quote Joe Tragert, Director, Market Development, EBSCC Publishing, 'The question isn't 'Which mouse trap works better?', it's 'Did we kill the mouse?'" You can have all the metrics in the world available to you. And if your goal is still unachieved? Or if none of the available metrics address your goal? Then it is time to either give up your goal or create new and valid methods of measurement that help you achieve that goal.
Please contact NextStage for information regarding presentations and trainings on this and other topics.
Sign up for the NextStage Irregular, our very irregular, definitely frequency-wise and probably topic-wise newsletter.
Fascinating post, Joseph.
I don't pretend to speak for Matt here, but I would submit that the "measurement so you can improve it" premise is valid, if it is unpacked a certain way (I fear that semantics may be leading us to confusion- a subject you would have far more insight into than I).
The goal of measurement for our clients is to take action on the findings and to improve stuff, much like the goal of a clinical research oncologist may be to discover a drug that cures cancer. However, the doctor’s research may not actually improve anything- it could simply (and importantly) add to the body of knowledge that allows for her, her colleagues, her students and her successors to support and refute hypotheses that lead to a proposed treatment.
As you correctly point out, the first step for both web analyst and (I would assume) oncologist is to understand whether something ab initio is measurable. Once a measurement is validated (which is probably the subject of a whole different discussion), the researcher embarks on a series of educated guesses (hypotheses) based on a holistic understanding of the system, to evolve the body of knowledge about that system to the point where (for a website) stuff can be improved, or (in the oncologist’s world) a treatment can be derived. In the case of an oncologist, many years of tests (and retests) may be needed before one proposed treatment is offered for clinical trials, while a web analyst will need to perform a great deal of baseline studies and modeling before even proposing an hypothesis aimed at improving a “good” metric.
My main concern with Matt’s premise generally, and the “improvement” element specifically (which you touch on in your Factor A example), is that it implies that a test which fails to result in a measurable improvement is a failed test. The negation of an hypothesis can be massively helpful to further a course of study. The Greeks looked at the shape of lunar eclipses and negated the premise that the world was flat. This resulted in a far more focused study of geodesy and geography than would have occurred had that original test been merely inconclusive. By creating and then negating an hypothesis, a good web analyst or test designer should be able to refocus resources on paths that are more likely to bear fruit. These may not be the tests that are given credit down the road for a 20% increase in purchase (as one example), but it is certain that the test which does result in significant improvement could not have happened absent the learnings that came before it. I would imagine that Matt agrees with this as well, but the phrasing of his premise may mislead in that respect.
I do not subscribe to a measurement-for-measurement’s-sake policy for web analytics. As Jason Burby is famous for saying, analytics without action has a return-on-investment of zero. But I don’t think that a measurement which proves difficult to affect should be shelved. The road from measurement to improvement is often unpredictable, and a good analyst will occasionally investigate seemingly unrelated or unchangeable metrics for the chance to find a new correlation (if not out of sheer curiosity). Most of the analysts I work with at ZAAZ spend a good amount of time investigating the unchangeable and irrefutable either to map the nature of those metrics more clearly or to challenge the assumptions that define them. I find that this is a fantastic way to spend time learning about the system in general, and without question results in a better analyst.
Which brings me to the point (I think). Measurement should be taken, broadly, with the goal of improvement, but that is accomplished through repeated experimentation to learn about the system. It is through only through iterative, focused, and scientifically rigorous testing and remeasurement of the system that improvements will come. Web analysts would do well to understand what oncologists have learned for decades- that this is a process that takes time, requires resources, and demands very specialized attention.
[P.S. In no way am I trying to equate the importance of web analytics relative to cancer research. It’s an analogy. I hope everyone gets that.]
Posted by: Jason Carmel | August 19, 2008 3:11 PM | Permalink to Comment