|Notice that the quest for high fidelity has always been, almost exclusively, a male pursuit. A picture from the Edison Diamond Disc and the "tone tests" done back in the day circa ~1916.|
I've been critical of many audiophile magazines, websites, and YouTube channels because it's hard to believe some of these individuals truly can hear what they report. I know, it's perhaps "rude" to question subjective claims, but unless one questions stuff like this, how are we to differentiate thoughtful critical listening from potentially biased claims of sound quality?
For example, I honestly doubt the folks who claim to hear big differences between cables. I think it's silly that some people insist they hear differences because of unsubstantiated "jitter" effects in digital audio. Those who think "bit-perfect" playback can sound very different have never over the years proved their case. Then there are those subjective reports that sing the praises of "hi-res audio" despite research never showing differences as being anything more than subtle. How about claims of the ability to hear the effect of "audiophile" ethernet switches, or USB noise/jitter devices?! As you can see in the links to these topics, I've tried to address them over the years so when questions come up, I can just point folks to areas covered already.
IMO, with time, more and more nonsense articles and claims have built-up in audiophile magazines and the Internet. I think young audiophiles must have difficulties making heads or tails about what is true anymore while negotiating this massive minefield of questionable information. I would not be surprised that some in this Industry actually prefer the presence of this "fog of subjectivity" because it allows people to make irresponsible claims about anything they want and not have to provide evidence so long as some key reviewers or readers have faith to repeat the belief over time!
That the ongoing majority of audio hardware reviewing these days is purely subjective is a bit of a problem in the context of what I believe to be very poor reliability.
Beyond the pure silliness of some of the bizarre claims above, as discussed a few years back, we do need to be aware of our hearing limitations and how this fits with writing reviews with conviction. Like it or not, a listener's age does play a role. Now that I have turned 50 years old myself, I am mindful of the effects on hearing (among other biological systems!). No, I do not believe I can hear as well compared to my 20's or 30's; to think otherwise would be ridiculous, uninsightful, and frankly dishonest. Looking around, there are some seriously old reviewers out there writing articles about the sound of hi-res DACs and $100,000 speakers, claiming the ability to hear nuances. Some might take offence to my skepticism as being "ageist", but isn't this simply what the science shows as we get older?
The problem with accepting subjective accounts at face value, whether concerns of the age of the individual or simply hard-to-believe claims (like cable differences), is that claims could all be wrong and there's typically no feedback mechanism to help correct these errors of "faith" in the audiophile world. Hey, If I'm going to be skeptical when someone tells me he saw a ghost, UFO or Sasquatch, then I think I have just as much right to question unlikely "audible" testimony!
Having said these things, it would not be fair of me to complain about others if I did not consider whether my own subjective impressions are correct! For this post, let's consider a comment I made a few weeks ago in the review on the Cayin RU6 R-2R DAC:
Putting on my "critical listening hat" for a bit, while usually not a problem, I noticed occasionally audible distortions with lower impedance headphones. Here's a concrete example I came across while rummaging through some music on my phone: with the closed-back lowish impedance AKG K371 headphones (measured here) in a quiet room, turn the Cayin to Low Gain, 90% volume (not too loud), NOS mode. Listen to the first few seconds of track 2 "A Million Dreams" on The Greatest Showman (2017, DR8) soundtrack. With this combination, I noticed during the lower-level piano intro, a subtle "haze"/"hash" has been introduced into the sound. Imagine something like the top end distortion of low bit-rate MP3 but transposed down an octave or two. It's not as strong with "OS" mode but still clearly not as clean as with a higher resolution DAC+amp (like say the Topping D10s with the Drop + THX AAA 789 headphone amp). I suspect this is an example where the non-harmonic distortions we see in the measurements surpassed the threshold for detection and I remember thinking "that sounds different...".
|We can see the LH Labs Geek Out V2 in the right corner of the table further back.|
What you see is the Cayin RU6 plugged into the Raspberry Pi 4 "Touch" USB port (I'm running Volumio on the Pi). I have the track "A Million Dreams" (off The Greatest Showman CD) on a USB stick plugged into the Pi. The headphone output of the Cayin is connected to a test cable; one side (gold color) goes to the AKG K371 (rated 32Ω, see measurements for more details) and the other side is connected to the E1DA Cosmos ADC for recording, set to 2.7Vrms level or ~500-600Ω single-ended impedance - effective impedance seen by the headphone amp would be ~30Ω.
As you can see, I've then connected the Cosmos ADC to my Surface 3 computer where the audio is recorded at 24/96.
This test would be no good if I didn't give you a chance to hear this yourself! Here's the first 1:40 of "A Million Dreams" (portion of the music used as per "fair use" of copyrighted content for educational purposed) recorded from the Cayin RU6 with OverSampling (OS) and NonOverSampling (NOS) settings.
To get the best dynamic range in the recording, I used "High gain", 90% volume on the RU6. Also, I have included the Geek Out V2 DAC recording done at approximately the same output level. I did not adjust the volume between the OS and NOS recordings. The Geek Out V2 recording was normalized to the output level of the Cayin RU6 OS in Adobe Audition.
|The Geek Out V2 recording was normalized in Adobe Audition to the Cayin RU6 OS recording using a slightly different algorithm than Foobar's RMS number, hence the slightly different average value.|
As you can see, all 3 tracks are the same 1:40 length, we see relatively close peak values, and RMS amplitude also similar - within 0.15dB. Average dynamic range is identical at DR12 between the different settings/DAC/headphone amps.
Now, let me cue up a couple of tracks at a time to listen through the ABX Comparator and see if I can ABX the difference:
|Note: Since the tracks are already of similar amplitude, I did not use the ReplayGain setting in ABX Comparator to change volumes since I found that this exacerbated volume differences.|
Let's see the ABX results comparing Cayin OS vs. NOS modes, and between Cayin OS vs. Geek Out V2 (which is also an OverSampling DAC):
I used my desktop playback set-up which is a Topping D10 --> Drop + THX AAA 789 --> modded Dekoni Blue (anomalies easier to hear with Sennheiser HD800). Make sure your system has adequate resolution when listening for differences.
As you can see, differentiating the sound from the Cayin RU6 OS vs. NOS (left printout) could be done quickly - about 2 minutes for 16 trials with 14/16 correct.
I took more time to listen to a passage I thought differentiated the Cayin OS vs. Geek Out V2 (right printout) and the 16 trials took about 5 minutes total with 12/16 correct. Not perfect accuracy, but I think a reasonable demonstration that the files were not hard to tease apart and as expected, it was easier to tell the difference between Cayin's OS/NOS settings than between the more conventional sound of the Cayin OS vs. Geek Out V2.
You too should be able to hear the distortions with the NOS mode as a kind of distortion "haze" I described in the subjective review. It's not a static noise but fluctuates with the music itself, so it's not just an elevated noise floor (although the noise floor is higher). As for the Cayin RU6 OS vs. Geek Out V2, the difference is still present and you should be able to hear that the RU6's R-2R design is more noisy, definitely not as clean as a typical hi-res chip DAC such as the Geek Out V2's ESS SABRE9018AQ2M.
As expected, the Cayin NOS vs. Geek Out V2 was the easiest comparison to perform with 16/16 correct done a few days later even through DirectSound (set at 24/96 as per the test recording):
An interesting subjective effect I've described before with NOS playback is that it tends to sound "fuller", compared to the "leaner" OS mode. I believe this is because of the added harmonic and non-harmonic content to the signal with this DAC. Notice that the Dynamic Range Meter's results for the Cayin NOS recording showed a lower peak value and overall RMS amplitude despite the subjective impression of "fullness".
Another interesting effect I've heard with some NOS DACs is the impression that a reverb or fade trail seems extended or accentuated. Again this should not be interpreted as any kind of "accurate" sound even if some audiophiles might like this effect. I've often wondered with devices where the noise floor is higher (and this I think applies to vinyl as well), as the sound fades out quickly, maybe the mind is able to "fill in" the presence of that sound for a bit longer as it still detects some subtle noise/hiss. This is unlike a truly quiet device where the "fade to silence" is clearly perceived as quick and obvious, leaving more silence between notes and passages resulting in a "more hi-fi" or even "dry" sound. IMO, this is a good thing, it reflects the actual low-noise characteristic of the recording. And in those quiet intervals, we can appreciate the nuances thanks to the improved resolution.
|Cayin RU6 with 1MORE Quad Driver IEM.|
So what does this little demo show?
1. Not all modern DACs sound the same! In particular, the R-2R architecture in the Cayin RU6 is relatively noisy and with the lower-impedance headphones putting more demand on the amp, this can be heard without great difficulty. Note that I have done other recordings with higher impedance headphones (like the AKG Q701) and the distortions are less audible.
The Cayin RU6 performance level (as can be seen objectively) is such that audibility is not difficult. However, most modern DACs have higher resolution and will be much more subtle if not impossible to hear differences when volume controlled.
2. NOS indeed sounds different. Lack of filtering allowing imaging artifacts to pass as well as a -3dB roll-off on the top end can be heard. Ultrasonic artifacts can (and will) cause issues like intermodulation to affect the audible frequencies, depending also on your playback system.
Let me reiterate the idea that "intentional distortions" such as NOS does qualitatively add a flavour to the sound even though it's not strictly "high fidelity". Whether you like this kind of sound is of course an individual choice. This theme is nothing new. In fact, the other day I was reviewing the very first blind test done on this blog back in 2013 comparing high bitrate MP3 vs. FLAC. Even back then, the results suggested some listeners preferred the sound of the MP3 tracks even though we know it's lossy and some nuances/details have been compromised.
My personal philosophy remains that of a preference for transparency when it comes to hardware. Even if we accept that imperfections, whether it be NOS, or high noise level, or elevated harmonic/intermodulation distortions might sound "better" to some audiophiles, for me, my "hi-fi" is still about primarily achieving lowest noise level, least distortion, and accurate temporal performance. If an artist intended the album to sound more "wet" than "dry", more noisy than pristine, then by all means he/she can do that. I just don't want my hardware to insist that all albums sound "wet" or have a uniform minimum noise level that's audible! Tonal colorations, distortions, and higher noise level are IMO antithetical to the goals of "high fidelity" reproduction. As such, the Cayin RU6 is not a "high fidelity" DAC/headphone amp.
3. See, objective measurements were able to demonstrate the differences between DACs, and we can actually hear it subjectively! Who says objective results are useless or don't correlate to sound quality? ;-)
I trust that listening to these samples now can give you some context into what the graphs and numbers were pointing to all along about this DAC if we now go back to the Cayin RU6 review.
4. Whew, I wasn't hallucinating when I wrote the subjective review! ;-)
I believe with a bit of work setting up the test conditions and making recordings of the output of devices like DACs, one can "capture" what was heard to show to readers/viewers if indeed there is something significant to hear. This goes beyond the usual publication of purely subjective claims or even measurements and graphs. I think it would be very cool if reviewers are able to chase down audible anomalies and demonstrate these with high-quality audio clips to contextualize what was heard.
For the vast majority of the time with reasonably high fidelity gear, I suspect differences will be subtle and will not correlate with some of the extremes of language used in the subjective descriptions (especially between devices like DACs). Apart from technical ability, I think pure-subjective reviewers likely will not want to do this as it weakens the prospect of generating a strong impression based on textual descriptions. Likewise, manufacturers looking for hype would likely be unable to achieve the excitement they're looking for.
Notice that I'm recording the demo tracks directly to 24/96, slight editing done in 32-bits, and making this available as a lossless file (like the AMPT recordings).
Many times, we see audiophiles wanting to show differences in sound using lossy YouTube audio. At a gross level, this might help give us a taste of the sound quality. These days, if we're intending to demo high fidelity gear, a YouTube video is obviously inadequate. For example, is this recording truly telling us anything about the equipment? Notice the nasty distortions at a few of the peaks, and lossy distortion artifacts.
My feeling is that subjective reviewers need to once awhile "prove" at least to themselves if not the readership that what they heard is indeed true. Just because a writer/reviewer may have had their hands on a bunch of gear over the years or had copious "experience" doesn't make the person necessarily a great listener nor should we think that age will be any kinder to self-proclaimed "Golden Ears".
I still think at the very least, there should be some kind of minimum standard "certification program" for subjective reviewers (like this). Not likely to happen ever, but if one is serious, listener training I think is a must.
As a final test before returning the Caying RU6 DAC back to AudioPhil, I performed the stepped sine measurement to have a peek at distortion across output levels. I set the device to High Gain, 100% output level, connected to the E1DA Cosmos ADC at 2.7V reference (~600Ω single-ended impedance). Here are the OS and NOS graphs:
|Since this is a USB loopback on my Intel NUC, 8kHz USB noise exacerbates the 8th harmonic. So I used 950Hz instead of 1kHz.|
This is the more "normal" pattern we see with high-resolution DACs these days.
Looking on the Internet (like here, here, here, here), we see many positive reviews of the Cayin RU6 with no recognition of rather serious fidelity limitations. Hey bros, do you guys ever give negative reviews or actually listen critically? Or are reviews just an exercise in marketing meant to push products and make a buck?
Behold the true nature of the Internet these days. Audiophilia being just a microcosm of much of the "information" fed to us in numerous other areas.
Don't worry guys and gals, I'm not paranoid or think they're "out to get us"; just a reminder that we need to be critical about all the stuff we read and see these days. We all need to make sure to sharpen our critical thinking skills, even more so when anyone can say anything online these days without needing to show evidence.
Recognition that the Internet in many ways is very much an advertising tool should not be shocking. Nothing here that would surprise Tim Berners-Lee or the idea that the Internet to a large degree has become a tool "about making markets" whether on a large scale with "Big Data" or implicitly on a micro level with content produced in one's YouTube broadcasts.
On the muddled "Thoughts on reviewing"...
To end, I see that Jim Austin in a recent Stereophile editorial regarding "Thoughts on reviewing" continues to try to make a case that there is an apparent special need for a certain type of subjective evaluation of audio products. I believe this viewpoint is misleading and simply bad for the audiophile hobby.
Austin channels his theater professor and poses these apparent words of wisdom for reviewers: "Look inward before looking outward. How does it make you feel? Why?"
Immediately, I think many seasoned audiophiles will be a bit suspicious about such strong encouragement placed on emotions. Indeed, feelings are important and I certainly do not want to minimize this nor suggest life should be devoid of joys, sadness, or humor! However, when we open the pages of Stereophile to read about audio equipment reviews, are we sure that the process of writing reviews can fairly be lumped together with the output of "an aspiring playwright", the "theater critic", and a "composer"? These latter 3 individuals are either creating or writing about art. An audio equipment reviewer is writing about a technologically manufactured product. There is a wide gulf between sharing the emotional experience of a play, or enjoying an album, and discussing the merits and qualities of an inanimate product made in some kind of factory!
Yes, as humans we can share many experiences across cultures (Beethoven and the Beatles among them). Artistic appreciation is inherently qualitative although we can still measure certain aspects of music like tempo, or dynamic range, or complexity which might correlate to our "feelings" also. Is a product review in that same category as judging or critiquing art!? I don't think so!
The next part where Austin talks about "quantitative, scientific proof" is where the wheels really fall apart in this essay. Austin seems incapable of differentiating between the psychoacoustic science and research of Floyd Toole and quantitative measurements of products! By doing this, the article loses coherence and it becomes impossible to follow what his points are.
Yes, it's true that Toole's research helps educate by allowing for "human preference(s) made scientific". Sure, the scientific approach can produce reliable findings. But what's this about?
But there's a tradeoff. What it gains us in rigor and certainty, it gives up in specificity and incisiveness.
Huh? Tradeoff in what context? I thought we were talking about equipment reviewing here and just a moment ago weren't you expounding on there being a "shared humanity"? Isn't science that can illuminate reliably some parameters of how we as humans listen helping us understand that "shared humanity"? So why are we then pivoting to this:
First, what's measured is an average over a whole population, over all test subjects. Studying a population is necessary to achieve statistical significance—to be confident of the result to some stated precision—and to be certain your result has broad validity, beyond one individual. But what you learn from such research reveals little of interest about any individual in the group.
Suddenly, we're not "sharing" a commonality anymore, but complaining that scientific research (not equipment measurements necessarily) leaves out the "individual" within the group. As a reader trying to understand what the point is when it comes to equipment reviews, I'm confused!
Realize that complaining about not incorporating listener idiosyncrasies is a bit silly. When writing reviews, does the reviewer also not have to consider the audience? Does a reviewer write for the "individual" out there who only listens to death metal and demands >100dBSPL peaks (nothing wrong with this of course!)? Or does the equipment reviewing process also understands and targets the "average audiophile" with their tastes in music and desired sound quality? Within the pages of Stereophile, isn't it normal for a reviewer not to address the needs of that individual who listens only to death metal?
Then it gets even messier:
Second, in most such studies, human emotional response is reduced to a single, blunt concept: Which one do you prefer?
What does that teach us about love, or fate, or the human response to mortality—about deeper human things that almost anyone can learn about by watching a play or listening to a great piece of music? Not much.
Oi! What "such studies" are we talking about!? Since when did scientific studies become just about "a single, blunt concept" even? Did Austin ever actually read the studies and appreciate that research papers are way more nuanced than what he seems to be insinuating here? Don't research papers include statistical analyses and basic concepts like range, standard deviation, and confidence intervals to remind us that humans indeed do have different responses? To make it sound like science blandly is about providing "dull responses" (a phrase used later in the article) speaks volumes about Austin's lack of understanding as he creates the proverbial "strawman" that he tries to knock down; as if the sciences (especially biology, psychology, the social sciences) do not in their own ways also encompass an understanding of idiosyncrasies.
I've previously addressed thoughts like that second paragraph already - the concept of "emotional connection to music". If we're talking about equipment reviews here, tell me, what business does an inanimate piece of equipment, like say a DAC or even a nice tube amp have to do with teaching "love, or fate, or the human response to mortality"? Are those emotional, philosophical, even spiritual concepts something a piece of equipment is supposed to enhance or in any way affect? Or is it that those concepts are actually what the artist expresses in the music and our sound system's job is simply to "get out of the way", be "transparent"?
Personally, I will happily side with the latter idea that inanimate objects are simply incapable of especially enhancing these human thoughts and emotions. The idea that any amplifier can teach us "about love" or endows playback with some kind of "response to mortality" is bizarre and more than a little anthropomorphic even! Would an audio reviewer (based on his inner feelings of course) ever say that "This Pass Labs amplifier taught me more about my own mortality than that McIntosh."? Right about at this point in the article, I was reminded of that line by the antagonist Mugatu in Zoolander: "Doesn't anyone notice this? I feel like I'm taking crazy pills!" ;-)
Well, since he's in the zone, I guess he might as well continue to get even more metaphysical:
We read, or look, or listen, and we feel less alone. But to embrace those responses—to be a part of the human race—we have no choice but to abandon certainty. There are things we know about ourselves and others—about what it means to be human—that we'll never prove. We don't need to prove it because we know it.
My background in science sensitizes me to these issues—which many readers and other reviewers may never question....
I understand what he's trying to get at with the intangibles of human experience. But we're still talking about audio equipment reviews in the pages of Stereophile here, right? Are we sure he's not talking about a term paper on epistemology? I'm not sure how his background in science "sensitizes me to these issues". I have a background in science too and all I see are vague, hand-waving, somewhat disconnected comments with enough holes to make Swiss cheese jealous.
After all that, indeed it looks like he is talking about reviews and pivots back to this (emphasis mine):
Yes, our field—reproduced music—is underpinned by science. But that doesn't mean that the scientific approach to criticism is the only one that's valid. Music reproduction is based in electronics, but music itself, and our experience of it, is based in shared humanity.
Let me get this straight. While there are music/album reviews in Stereophile, most of the time, we're talking about audio reproduction hardware reviews, right? Audio hardware products, last I checked, are a type of consumer electronics (whether mass-produced or artisan-created, they're still a type of electronic machine for consumers). Since when should reviewing electronics hardware, and their ability to reproduce an electrical signal need to inherently include the "music itself"? DACs, amplifiers, speakers, computer streamers, even turntables can be considered works of art in their design, but is the "sound quality", often defined as how closely the device achieves a level of fidelity, worthy of artistic critique?
Consider for example if we read a review of an expensive, glorious, "high end" large screen flat panel OLED TV with amazing dynamic range, infinite contrast and ultra-realistic color tonality. Would an AV reviewer go beyond a "scientific approach" to that TV to also claim that viewing Schindler's List using the product "drove me to tears with such deep pathos in a way I had never experienced that movie before with any of my other large screen TVs!" Isn't that what expressing "feelings" and "shared humanity" might look like in such a product review?
Instead of appropriately attributing thoughts and feelings to the skills and artistry of Spielberg, or the actors, the screenwriter, sound editor, soundtrack composer, or the cinematographer, such an AV reviewer instead would be claiming that the object (TV) itself is the source of emotion and "humanity"! I trust that what the readers are actually looking for and the thrust of a TV review should be about its technical abilities. Nothing wrong with saying that the beautifully smooth black-and-white gradation in Schindler's List added to the ability for the reviewer to be drawn into the humanity of the film, but these "feelings" should not be a central rationale for why you, the reader should buy this phenomenal work of engineering!
As silly as this example might be, isn't this exactly what audiophile subjective reviews do (not just Stereophile, but also TAS, Hi-Fi+ and almost all the other magazines and YouTube videos)? Isn't this what Austin seems to be advocating? When month after month, we read about how these $50,000 speakers or that $20,000 computer streamer perpetually increases the emotional and even spiritual experience of the reviewer, at some point, I would hope audiophiles might just have had enough of this silly form of reviewing! Let's just get back to whether the device represented value for its build quality, usability, and of course technical performance.
The last paragraph in this article is interesting (again, emphasis mine):
The heart of this magazine, though, is in this other realm. It's based on a faith in shared human experience, a belief that what's true for the critic will be true for others, that humans share enough in common that not every insight requires proof. All that's needed is a certain sensitivity, seriousness, and goodwill.
Words like "heart", "faith", "belief", and "not every insight requires proof" all sound sweet. The kind of words the yoga instructor might evoke. Likewise, "sensitivity, seriousness, and goodwill" are fine human characteristics. But is there not a fundamental balance to be found? What about words that help us grasp rationality and reality: "mind", "facts", "knowledge", and "evidence"?
Earlier on in this article, Austin suggests to reviewers that:
Understanding your reaction requires a degree of self-reflection that's neither comfortable nor easy. Even if you're willing to engage in such self-reflection, answers don't come easily.
I would argue that to dig deep into truly honest, self-reflective reviewing, we must always maintain a balance with all those concepts in mind. Otherwise we'll drown in fantasies, wish-fulfillment, and neurotic preoccupations that might have nothing to do with the reality of these inanimate objects being reviewed. So are you truly willing to engage in self-reflection Mr. Austin? Is this comfortable or easy? I hope not because simply stating that "not every insight requires proof" seems to be a rather easy route for scammers and liars intent on financial gain regardless of truth. Let's face it, there are many "myths" in audiophilia that simply need to be addressed with "proof". There is already too much "fog of subjectivity" out there in audiophilia which has allowed the perpetuation of falsehoods, and hyped up questionable claims for years if not decades already. As I said near the start, I believe some in the Industry prefer it like this. Yet I believe this is unacceptable for the long-term health of the hobby, if not fundamentally flawed for honest journalistic ventures.
The implications I think are clear to most audiophiles these days that measurements and empirical examination of supposedly significant audible differences help anchor our beliefs within a reliable foundation. Austin mentions almost in passing the fact that at least Stereophile publishes measurements; the only reason I still have a subscription. Without that objective foundation, one is prone to lose track of reality and focus - as with Austin's IMO weak, and meandering "As We See It" article.
Stay rational, stay real, dear audiophiles. I hope you're enjoying the music - the art - with the full measure of emotions and humanity while using the engineered devices we are blessed to have at our disposal thanks to science.
As per the comment to G:
Spent about 30-40 minutes playing with Harman's "How To Listen" software... Achieved Level 6 with the "Band ID Training" at Trial 26, no practice, and just listening to the "How Long" track. Now that I know what's going on, should not be difficult achieving this with <25 trials the next time.
Would be curious whether others have tried this or what the "par" is for a 50 year old dude!?
Hmmm, it doesn't seem to be saving my progress running this on Windows 11. That means I won't really be able to use this seriously to "progress" over multiple listening sessions...
Also, the software definitely needs to allow the user to bring his/her own music. Gets really tiring listening to unfamiliar songs I might not like. :-(