MUSINGS: Trust, empirical testing, and evidence in the Audiophile hobby. And the Double Blind Test (DBT) strawman argument.

For this post, let's spend a little bit of time thinking about "levels of evidence" in audiophilia. This is important because it helps determine how we gauge the veracity of the articles we read and how we build confidence as we create our own internal models of the topics covered in this hobby.

Inevitably, if we stick around this hobby of audio gear and products long enough, we'll come across the abbreviation 'DBT' - that dreaded Double Blind Test; that seemingly ultimate methodological scientific "gold standard" in human research to tell us whether an intervention or change in our systems actually resulted in a worthwhile difference.

But do audiophiles actually throw around the comment "You need to do a DBT!" very often? Or insist that "I don't believe that unless a DBT proves it!". The answer is 'NO!'- of course not!

Let's start by zooming out a bit and thinking about the very big picture...

I. The Big Picture

Let's start with the really big picture from 300,000 feet at the edge of space. A "Coles Notes" as it were for some general principles. 🧐

For a moment, let's ask ourselves: "How do we 'know' anything?" To know something implies that we have internalized basic ideas and hopefully incorporated the concepts into something coherent. For passionate audiophiles, it's not really just some random facts like "The capital of Canada is Ottawa."; rather if we really care about something, that knowledge becomes a part of who we are. Perhaps meaningful enough that we wage war (in forums of course) over that belief! 🤯

While there are different ways to look at this, on a personal level, the developmental stages of Erick Erikson could be an interesting model to consider. He places the developmental stage of foundational trust (he calls this conflict that we must negotiate "Trust vs. Mistrust") in that first year of life when we are forced to have faith in mommy and daddy to ensure that we have the nourishment we need, to keep us safe, to know that there is love in our lives. We need this as a foundation to experience security from which we can launch into the later stages with freedom from anxieties so that we can be autonomous, curious, able to embrace a true identity, and by adulthood, be in a place of generativity in raising the next generation, and being productive members of society. Ideally, we aim not just to accumulate wealth or power for ourselves, but something better, "leaving the world a better place" as a broad legacy for the limited time we all have here. By the last phase of life, hopefully we can look back and know what it means to have lived a life of "integrity". Inevitably, what we "know", or "believe", or whom we have "trust" in will have an impact in how we conduct our lives and the effect we have on others.

As noted above, successful development of trust allows us to achieve freedom and confidence. Without the ability to move on from basic anxieties (including some emotional regulatory skills like the ability to self-soothe), we can become stuck in states of insecurity, potentially always looking for ways to satisfy others, perhaps looking to be the center of attention (so one is not abandoned or embarrassed?), unable to be content in who we are, desperately searching for that next "hit". We might look at audiophiles with "neurotic" fantasies as discussed a few years back as examples of a kind of hobbyist who has lost the ability to find contentment, hording all kinds of things (maybe all kinds of ideas as well), as if to satisfy an internal emptiness of "love". Audio hardware neuroticism to the extent that some might end up divorced from the love of music is of course but one of many possible manifestations of foundational psychological insecurities.

So, ideally, as individuals, growing up in a predictable, safe, loving home with parent(s), extended family, and maybe in time a good partner, close friends, caring community will provide the stepping-stones to maturity. However, just having trust - "faith alone" - is not fully mature nor adequate in this unpredictable, complex world where ideas are often false, and not every person possesses noble intent for us. Scams, frauds, cults, peddlers - whether religious, or political, or financial - test our ability to negotiate faith in others and even the "system" as a whole (paranoid, anyone?). Mature adults would be able to maintain our integrity through such swings and influences - no matter what, hopefully our strong early development sustains us, and will provide that balanced foundation.

I'm sure we all have regrets in life, both big and small. Some regrets could be massive. I hope if we've been cheated by the used car salesman or audiophile magic cable peddler, these rank as the smaller things in our lives to recover from. As individuals with a strong foundation, we can learn from such mistakes and securely adapt our worldviews hopefully to be better. Part of that process is in making sure that we accumulate knowledge and trust others wisely. How do we "test" that the knowledge we accumulate is in accordance with reality? How do we know that the people we read/watch/listen to are indeed trustworthy? As I've said over the years, audiophilia is a fascinating microcosmic "testing ground" for those of us drawn to this stuff, practicing these skills and hopefully communicating the understanding maturely to others on this journey (intelligently, gracefully for the most part, among other positive attributes).

For much of human existence, knowledge about the natural world (including what we know these days as physics, material sciences, etc., the things that make up the electronics we buy in this hobby) was simply passed down as a kind of received wisdom of the ancients. Religions, myths, traditions for example told us how things came to be, who we are, and perhaps the ultimate question, what happens to us beyond this life. As kids, Santa Claus perhaps was real because our parents and the culture may have suggested so. As adults, how do we let go of certain "truths" recognizing that the faith in mommy, daddy and Santa we once had might have been developmentally appropriate but we're now free to move on? In the same way, our society can progress taking what is good in religious faith and "conservative values", incorporated into a much more nuanced understanding of the universe.

In the Western world, things changed from the 16-18th Centuries with the Scientific Revolution and Enlightenment as the development of math, physics, chemistry, biology, astronomy paved the way we thought - to be more empirically-based and to question the power of dogmatic authority. This revolution of thinking subsequently flowed into all aspects of society including modern understanding of freedoms, right, laws, etc. beyond knowledge as declarative information around the nature of things in the physical world.

II. Science & The Adequacy of Measurements

Modern knowledge is built on the foundation of the Scientific Method of the 16-17th Century (while Aristotle thought of everything, consider the empiricism of Descartes, Bacon, Hume). At the heart of this is that we know things because we can empirically observe, measure, document, and demonstrate that which we ultimately hold as true. Whether it's as simple as elementary school science classes where we directly observe a phenomenon, or more complex science where we no longer can behold what is empirically measurable. That last bit is important because many things are measurable but might be difficult to fathom, require us to not be overly concrete, or demand that we appreciate statistical analysis (like the value of vaccines on a population level).

On a very basic, ridiculous level, the fallacy of the Flat Earther is a beautiful example. Just because these people might not see the curvature of the Earth doesn't mean it doesn't exist - we can measure it by other means, and it's demonstrable when seen from space. The key is that if we believe something is true beyond subjective opinion, then the scientific individual looks toward ways to document and produce the empirical evidence.

As part of that empirical process, a scientist must also consider the testability and repeatability of the observations. If something is a true property in this universe such that one can share it with others (like telling you that an amplifier provides hi-fi sound quality) and expect others to experience the same thing, then whatever I'm noticing must also be repeatable in order to make my recommendation valuable. (In the past, we have also talked about other aspects like Science and Falsifiability.)

It is then out of this gradual process of scientific understanding that engineering as a discipline developed - engineering is the application of the scientific, mathematical, and practical knowledge into the design, building, and optimization of things that solve some "real-world" problems we have. Engineering involves creativity, there could be esthetic elements also (after all, to make something not look ugly is in itself solving a real-world problem!).

For hobbyists interested in "high-fidelity" reproduction of audio, we know the underlying issue that we're trying to solve - playback transparency given our source music material or signal. Whether a playback device like a streamer, DAC, amplifier, or even speaker is capable of "hi-fi" reproduction can be answered using engineering techniques. Fidelity is defined based on technical targets. Measurements are all we need to answer the question of playback fidelity. Given the level of technical ability in the 21st Century, audio products can be measured to a degree that "high-fidelity" sound is a function of those measurements, listening test are done for confirmation afterwards.

III. Psychology

I know, that last sentence above might sound terribly reductionistic among certain audiophiles.

"What!? You don't believe that listening is the most important part of the esteemed audio designer's work, Archimago?! Just merely for confirmation after the measurements?"

While I agree that ultimately what's important is how we hear the sound, enjoy the music, the art, the issue is that at a time in history when playback fidelity is almost always good enough to be enjoyable at some level, what does it matter the subjective opinion of any one person regardless of whether this is some respected audio designer, high-end reviewer, or euphonophilic salesperson if the engineering targets are reached? Are there not all kinds of variables that could be affecting enjoyment in even more powerful ways than mere sonic fidelity whenever we sit down to listen?

And that's where psychology and emotions play into this and why we as audiophiles get into all kinds of heated debates over this and that endlessly and often meaninglessly. Making judgments and using language to describe which product sounds "better", "worse", "crap", "beautiful" has always been the easiest part of audio reviewing IMO; heck, even AI can easily write plausible opinions. How we make these judgments will always be governed by our limits of perception and cognition as discussed a number of years back. Already, I had talked about the McGurk Effect (paired visual-auditory interaction) in that article, but here's another one to consider:

In many ways this illusion is even more powerful as it's not even based on cross-modal sensory interaction (ie. watching lips move and correlating this to single syllables like "ba" or "fa" as in McGurk), but the direct effect of reading multisyllabic words affecting perception of the complex ambiguous stimuli we hear! Very cool.

Like it or not, our mindset plays such a critical role in perception of sound quality in complex, subconscious, ways. Some of the most powerful applications of psychology is in how we affect others by directing sentiment, shaping thoughts, and ultimately coupled with the profit motive, applications in advertising products especially in the "High End" audio space driven by desire for luxury.

So, unlike pure science and engineering where we can directly perform measurements, how do we study the response of humans to the things they perceive or do not perceive?

Notice how the "amount of information" is greatest at the base
of the pyramid. Just as there are tons of forum posts with countless
opinions! How much of that is really trustworthy and factual?
Quality over quantity, right?
(Source)

IV. Human Research (and DBT)

So, let's have a look at the diagram above regarding the matter of levels of evidence when it comes to human research as we contemplate the complexity of biology, psychological make-up, and expectations.

First, notice that there are many levels of evidence ranging for observational data, to experimental data, and ultimately when we have multiple research trials completed, we could synthesize the data to demonstrate testability, and reproducibility across trials. Each level up builds confidence that the results constitute more trustworthy knowledge. In the audiophile pursuit, there are times where we see summary syntheses such as the mildly positive one on hi-res audio (Reiss, 2016) a number of years back. Within this pyramid of evidence, at best the majority of listening impressions we read about online, in the forums, the magazines and YouTube reviews, belong to one of those lowest tiers - the almost dime-a-dozen "case report". Without at least some level of systematic control of the variables, these case reports cannot be constituted as "studies". These days, most scientific journals do not even bother publishing case reports any more.

[As per the discussion with Propanidid in the comments, I think it's worth being cautious about the value of meta-analyses and systematic reviews at the top of the triangle. While these can be powerful, it depends a lot on the quality of the researchers and do employ techniques which are affected by subjective opinions and choices made.

In my experience, there are situations where results from meta-analyses could be classified as interesting academically, but in clinical practice not useful.]

Blinding/masking is simply the act of taking out those biases in human psychology as discussed in section III. If as audiophiles we truly are just interested in the sound quality (disregarding price/luxury, appearance), then we should openly embrace blind listening tests because this is how we get at the heart of audio quality! But no... For some reason, magazine and YouTube reviewers seem to shun blinding for some odd reason. 🤔 Needless to say, we can imagine why.

The idea of a "double blind test" (DBT) (often in medicine also placebo controlled) is a research method in which both researcher and subject are unaware of the actual stimulus, intervention, or medication being tested, typically implemented within a more complex systematic research structure including fair randomization and often the trials are done in multiple phases (like crossover designs, or an open-label phase). DBTs are highly valued as "gold standard" in intervention trials as they demonstrate rigorous, disciplined application of blinding. They are uncommon, typically require more funding, and represent some of the highest levels of quality. Significant results from such demanding tests on human subjects need to be taken seriously as powerful evidence of effect or not.

As hobbyists, do we need to do double blind tests? Of course not. But from what I can see, many subjectivists seem to want to talk about this kind of testing and then argues against it when as far as I can tell, rather few objectivists demand that we do such things!

Some examples:

Richard Murison - PSAudio/Copper Magazine February 2019:

... We’ve all come across the biggest elephant in the audiophile skeptic’s user manual, the double blind test (DBT), which is inevitably cast as the Gold Standard for the scientific method. And although we’ve also seen countless rebuffs to the DBT argument, they suffer in comparison to the apparent simplicity of the DBT proposition. In the end, nobody ends up satisfied, whichever side of the fence you may sit on.

...

Now, the armchair expert in Seattle doesn’t care either, and just prates on about how this, that, or the other piece of audiophilia is snake oil unless proven otherwise by a DBT. The audio manufacturer, on the other hand, cares deeply. He still needs to be satisfying his customers. It’s an existential challenge for him. And in the absence of a proven scientific method with which to address his toughest design challenges, he has to rely on his experience and skills.

For brevity, I can only quote a small portion of the article. Read through the article though and I think you'll come across some highly questionable claims in there about science and what he thinks we'd find or need to understand. So, who's this "armchair expert in Seattle" calling out snake oil? 😆 Does this fellow call for DBTs to be done? Where?

[At this point in the history of audiophilia, I'd love to know if a writer like Richard Murison has ever called out any snake oil products. I believe that any audiophile writer who cannot name some examples of snake oil in this hobby has either not had enough experience or has vested interest in the Industry and likely is unable to honestly advise consumers.]

Then there's this nugget from the subjective-only glossy review magazine - "The Absolute Sound's Review Methodology: First Principles":

11. Why don’t you do double blind testing?

There are several reasons. Double blind testing to some degree presumes that the comparison of interest is between two pieces of equipment. But, as we have seen, our comparison is with the sound of real instruments and voices in real spaces. For a designer interested in comparing two prototype designs, double blind listening might be useful. In our case, writing for consumers evaluating equipment, there are so many possible comparisons that it is impractical to do them at all, much less double blind. And double-blind listening tends to drag the conversation into relative benchmarking (“speaker A has more bass than speaker B” – okay, but we still have to compare with the absolute sound, so what did we accomplish?)...

The whole concept of "the absolute sound" has been nebulous and impossible to use as a standard so the whole charade of "our comparison is with the sound of real instruments and voices in real spaces" has always been a bit silly (discussed here). Isn't it fascinating that they frame the question purposely as one of "double blind testing"! I think most people are more curious about simply "Why don't you do blind listening?". Again who's asking that TAS do double blinded evaluations?

Here one recently from the world of YouTube - Danny Richie - the crossover parts whisperer, with UFO Believer mentality 😯 - on "Double Blind Tests, Measuring One Speaker, Sound Clips and more! Answering Your Top 10 Questions!"

He's "always asked about this" (DBT)? Is that true? Any of you guys asking him this? Are there many comments in his video episodes from people telling him to "Do a DBT!" anywhere? Perhaps occasionally but is this seriously common enough to be in the "Top 10", demanding to be in large bold font as the cover image for the episode!?

I agree with Danny to a point. As I already said above, doing a proper DBT study is hard work and generally not feasible nor needed for hobbyists. Note that it's not impossible even with basic cable switching, but it does require quite a bit of planning, discipline, and finding the right people for the test.

Notice that he tried to sneak by with saying "it doesn't have to be blind" when it comes to listening tests - no Danny, obviously often blinding is essential unless one is sure the visual bias doesn't impact one's assessment. As far as I can tell, as discussed before, it's pretty clear that Danny is biased in all kinds of way. For him to not be blinded when "listening" to capacitors (of the same value presumably) and claiming confidence in what he hears is ridiculous as there is simply no evidence they make a big difference anyways. Based on these kinds of self-claims with no evidence otherwise, I don't honestly believe he is capable of adequate objectivity. For guys like this, blinding will go a long way to restore credibility given some of the snake oilish claims he has already peddled over the years.

Here's a disturbing comment he made: "If you did do double blind, how do you know which was which?" at 24:00. Hmmm, it appears that Danny does not understand how double blind testing is actually done! 🤣 If he did, he would not be saying such a thing. It's an obvious example of not knowing what he doesn't know, yet seemingly confident to let the cameras roll and broadcast the ignorance!

Even in that short portion on DBT, there are a few other things in his comments that are not true or he's clearly expressing his own insightlessness, but we'll let this pass as I think I've said enough.

In the examples above, we see that it is the subjective-leaning apologists who are the ones who talk about DBT as if this is a topic that comes up often. Speaking as an objective-leaning audiophile, I don't think we're actually too interested in doing double blinding. I honestly don't remember the last time I used the term 'DBT' in my writings. Despite the hundreds of blog posts, this is the first time in more than 10 years addressing this topic directly. I don't think anyone has ever asked me to do a DBT to confirm measurements or my expressed opinions, and I don't even remember the last time on forum posts anyone seriously insisting a DBT is needed to resolve an issue!

As I said above, measurements are more than likely enough.

What I think rational audiophiles do suggest are not DBTs, just "blind tests" in general. Maybe an ABX paradigm to compare the sound and select if we think the "X" sample was the same as "A" or "B". I often talk about single blind listening tests (SBT) which is basically what we do in all the Internet Blind Tests that I've performed on this blog (like this, or this, or this) where the listeners/respondents are blinded but I as the experimenter am not - my job is just to keep quiet so as not to signal a "tell" and bias the results I receive from the listeners. Once awhile, call up an audiophile buddy to come over and maybe switch some cables or switch between inputs without the listener (you) knowing which DAC is playing, evaluating audible differences. None of this should be too scary, terribly difficult, nor necessarily stressful!

[For completeness, there are other testing methodologies used in audio such as MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) in codec testing. Have a look at the ITU-R BS. 1116-3 for detailed subjective test design recommendations that goes well beyond casual YouTube discussions and the typical audiophile forum chats!]

V. In Summary...

Examine thyself, audiophiles. By what process do we accept claims as facts? Have we honestly tried evaluating these engineered products for ourselves using a more scientific, empirical mindset? How highly do we accept the beliefs of others compared to our own experience? It's good and important to trust people of course, but have we taken the time to 'vet' individuals we confer esteem to - especially individuals whose influence on our worldview might result in their personal or financial gain?

Perhaps consider the forms of 'subjectivisms' and whether there are certain ones we need to be particularly mindful of in our audiophile journey. 

Beware of those who raise the supposed "need" for 'DBT'. To me, it seems that more often than not, it's not the objectivists who demand this, but rather it's those subjective-leaning who want to use it as a strawman to attack as if this is in turn some kind of slight against those who want to see objective results. The underlying message these guys want to send being that we somehow should not blind test because blinding is too hard and sighted listening is somehow always adequate. Given the biases inherent in human psychology, such beliefs are ridiculously naïve.

Despite the importance of blinded listening, I certainly do not believe that sighted listening is worthless as there are many situations where an honest (though never truly objective) individual can certainly hear differences and report them accurately. I don't really have any major problems with this most of the time. It's when we're talking about snake oilish items and when people make all kinds of dramatic claims that sighted listening veers into realms of disbelief and can clearly go off the rails (eg. rather old guys like this and this thinking they can hear typically picosecond jitter, eh? Prove it!).

Whether an audiophile likes the sound of their Wilson or that Magico or some Sonus Faber speakers I think isn't a big deal these days because ultimately the room-speaker complex is more important than almost anything else and each of us will be experiencing a different sound in our spaces. We can treat rooms, use DSP, etc. to hopefully adapt a high-performance product enjoyably. But if a person is going to get dramatic about big differences in the 'sound' of cables, their "tube connectors", bizarre CD cleaners, high-priced computer audio streamers, quasi-psychotic tweak claims, or pontificate about the sound of egregiously priced DACs, then yeah, let's get a little more curious to clarify if this is BS!

If we feel we need to get the human listener involved, then figuring out a way to do a single-blind listening test would be likely adequate already.

I hope that most rational audiophiles appreciate that modern, detailed measurements are really all we need to answer the engineering problems which our audio systems were designed to solve. This is especially true for devices like DACs, streamers, and cables that have very straight forward purposes and can be measured more easily (ironically, on forums there tends to be lots of arguments about these). Usually, once the measured data is in, whether you like the "accurate" sound coming out of your DAC for example, or whether you prefer something more "euphonic", becomes then a personal choice that no high-priest high-end Golden Ear reviewer's opinion should affect.

For those less common situations where blinded listening can be useful to supplement the measurements, we can maybe gathering a typical cohort of audiophiles aged 30-50 for a level-controlled, single-blinded, observational study. For example, a fun day of testing might be to compare audibility of a US$200,000+ dCS Varèse stack vs. a comparatively cheap US$900 Topping D90 III Sabre in a high quality sound system over some pizzas and beers! Any high-end dealership with access to the dCS want to host such a blind listening test?

I'll bring the Topping DAC and chip in on the pizza and beer. 😏

To end, here's a nice demonstration of the power of sighted biases as a function of the reported price of an item:

Whether it's a US$19 Japanese strawberry or standard grocery store strawberry, the power of human psychology is an amazing thing when we don't implement at least some kind of "honesty control" to maintain objectivity! In particular, that chef guy from 10:44 reminds me of some audiophiles trying to sound sophisticated describing his subjective experience of the "mastication process". 🙂

[To be clear, while I'm presenting a viewpoint on science that's simplified and at times idealistic, there are certainly some major concerns currently in the scientific literature. Though I respect the power of the scientific, empirical way of thinking, it's still conducted as a human endeavor with all our weaknesses and other foibles. So it's important not to be naïve about the influences that shape the modern scientific enterprise such as Big Pharma and personal ambition.

There's simply a lot of stuff being published these days resulting in a "replication crisis" in many corners of academic research. In the medical world, there have been all kinds of fascinating books about the progress (or lack of) as well as ballooning costs of health care, and related questions of value. A recent book that's easy to pick up for the lay public would be Seamus O'Mahony's Can Medicine Be Cured?: The Corruption of a Profession (2019) if you're interested in reading more.]

--------------------

As usual, these articles end up running much longer than I anticipated when I start writing. So to end, let's have some music! Here's a "virtual concert" for Yello's album Touch Yello (2009) - notice that it's in 4K/60fps High Frame Rate:

Recently, the album was re-released on BluRay Audio for the 15th Anniversary (also available here) with a TrueHD-Atmos (7.1 bed channels, DR15). All 14 tracks remixed along with a 5.1 DTS-HDMA version and Binaural Headphone Mix on the disc. Sounds excellent, especially for those with subs that hit deep and on full multichannel/Atmos systems.

Like most binaural mixes, while the music still for the most part sounds like it's inside my head, some of the elements like the plane fly-by at the end of "Out of Dawn" externalized well through my Sennheiser HD800 headphones.

Alright friends, it's Spring Break time! I'm going to take some time off to enjoy the season with the family.

Hope you're all enjoying the music from your sweet-sounding "hi-fi". 😊

Addendum March 29, 2025:

An example of how blinding takes away the supposed difference in sound quality "heard" this time with an audio engineer and the difference between an RME linear power supply, the RME DPS-2 (MSRP $1500), versus the switch mode stock power supply!

See around 2:33 how unblinded he believes it sounds different. And this latter part with measurements and into the A-B blind listening with a switch:

As expected in the conclusion, it's simply realization of psychological biases.

If it's not too hard to measure, stick with what the measurements tell us unless you have better evidence, audiophiles. Definitely don't sweat the small stuff, you're not missing out!