"High-End" DAC Blind Listening Results - PART II: Results & Analyses

I imagine that the banner graphic above must be so bizarre for some audiophiles! How is it even possible that the audio output from something cheap like the Apple USB-C headphone dongle could be mistaken for the way more expensive Linn digital streamer devices that cost orders of magnitude more?!

Well, of course we can! And if we are to honestly appreciate the difference in sound output between the very cheap and very expensive (we can buy a lot of other stuff, services, and experiences for US$20,000!), IMO, as audiophiles, we must open our minds to such comparisons. The hi-fi audiophile pursuit is not a cult; everything is up for empirical examination regardless of company, price tag, or which heroic personality is attached to said product.

Last week in Part I, I unveiled the identity and discussed those DAC/streamer devices. Let's proceed today with looking at the data from the recent 2024 "High-End" DAC Blind Listening Survey which collected listener impressions for 6 weeks; plenty of time I trust for those motivated to download, listen, and offer their subjective opinions.

As usual, let's go through the data broadly and then let's see if the results can provide some answers for specific questions around audibility, preferences, and listener subgroups. I'll group these questions and evaluations into a number of Sections.

Grab your favorite beverage, have a seat, this is a pretty long one... 🙂

I. Listener Demographics:

To start, as with any test result, we must make sure to contextualize the source of the information. In this case, anonymous Internet responses were collected from audiophiles around the world. As in previous listening tests, most of the respondents are from North America and Europe as shown on this map:

Thanks especially to the American audiophiles who showed up for the test! As you can see from the small bar graph bottom left, we also got a good number from the UK, Germany, France, the Netherlands, and Italy. We got a few from Asia; this is common due to fewer native English speakers and blogs like this one is inaccessible beyond the Great Firewall of China.

It's no surprise that like the General Audiophile Survey in 2023, the vast majority of us audiophiles who bother to test out hardware stuff are men - 97%; thanks to the women who gave this test a try!

And when it comes to age, we're also seeing a similar picture as last year with peak participation in the 51-60 years old group followed closely by those 61-70 years old. This correlates to the demographic we often see at places like audio shows:

Later on, we can analyze the preferences based on "younger" (≤50, 34% respondents) compared to "older" (>50, 65%) age groups to see if there could be differences.

II. What equipment did the listeners use?

I always find it interesting to know what gear was used for testing because this is the stuff you guys own. I suspect that those of you who would try a test like this represents probably a cohort of the more curious audiophiles - not just those who would take what's written in the press at face value or buy expensive goods just for luxury.

For this test, I see that most listeners used headphones:

That definitely makes sense for detailed listening tests. Headphones could be particularly sensitive for low noise, fine details and nuances, there are no room effects and there could be isolation from ambient noise for a better experience of the dynamics (IEM & closed more isolating than open headphones). However, loudspeaker listening provides a different perspective into how we judge characteristics like the soundstage that would be very different from using headphones.

Nice to see 11% of you used both headphones and loudspeakers!

Next, let's consider the overall system price: digital player, (pre)amp, speakers & headphones only:

44% of you used systems between $2k to $10k. 16% over $10k. So the median price would be around the $5,000 mark which I think is a very reasonable amount.

As we would expect, the price of headphone systems would be less on average compared to full loudspeaker systems - notice the skew:

It would be hard to find a reasonable-sounding loudspeaker system for <$500, but this is very much doable with headphones and IEMs.

Of those who listed their gear in the comments section, let's have a look at the kinds of hardware used. I noticed a number of you had the RME ADI-2 DAC and Topping D90SE for example which I only listed once. General computers like Mac Minis, Macbook Pros and Windows PCs are not listed; let's just focus on the specific audio hardware.

Streamers / DAC:

iFi Zen DAC v.2, Topping E30 II, RME ADI-2 DAC FS, Topping E30 Lite DAC, Cambridge DACMagic 100, Gustard DAC X18, SMSL M500 Mk I, THX Onyx DAC, M-Audio Air 192|4, Rega DAC-R, EverSolo Z6, SMSL SU-10, Audio-GD R1, Violectric V800, Topping D90SE, Topping DM7, Chord Mojo 2, Lumin U1 Mini, Topping D20, Chord Hugo TT2+MScaler, Aurender N200, Orchard Audio Pecan Pi+, Topping NX4, Metrum Acoustics ONYX, iFi Zen Stream, Innuos Zenith MK2, Auralic Vega DAC, Ferrum stack (Hypsos, Wandla, OOR), E1DA 9039S, SMSL DL200, SMSL SU-8, Schiit Bifrost 2/64, Wiim Pro Plus, Aune X8 XVIII Magic DAC, Marantz SA-10, Denafrips Ares II, SMSL DO100, Chord Mojo + Poly, TASCAM CD-200SB, Gustard X16, Topping DX1, Oppo UDP-205, NuPrime DAC-10, FiiO K7, Exasound e22, RME ADI-2 Pro FS Black Edition

(Pre)Amplifier:

Audiolab 6000A, Yamaha A-S3200, NAD 350, Cambridge Topaz AM10, NuForce IA7 V3, Violectric DHA V226, Rega Brio, BV Audio PA300SSE + BV Audio Pre1, Marantz PM-11, Schiit Mjolnir 3, Topping A90 Discrete, TubeCube | 7, NAD C298, McIntosh MC275, Roksan K3, Fosi V3, Simaudio Moon Neo 340i, Electrocompaniet ECI 5 MK II, Topping PA5, Hypex NCore, TacT 2150, ZMF Homage, Decware Taboo MkIII, SMSL SH-9, Musical Fidelity KW550, Schiit Freya + Aegir, Topping EHA5, Marantz SR7012, Devialet Expert Pro 140, Topping L30 II, Gustard H16, JDS Atom 2, ZeroZone IRS 2092 monoblocks, Chord SPM 600

Headphones:

AKG K371, AudioTechnica ATH M40x, TRUTHEAR Crinacle Zero:RED IEM, Philips SHP-9500, ZMF Aeolus, Beyerdynamic DT880 600Ω, AKG K702, HifiMan HE1000 Stealth, Dan Clark Audio Stealth, Fiio FD7, Sony IER-Z1R, Sony MDR-Z1R, AKG K553 Pro, Truthear Hexa, Meze Empyrean, FiiO FT3 Suede, HifiMan HE400se, Focal Elegia, Sennheiser HD800S, ABYSS AB1266 Phi TC, Beyerdynamic DT770 Pro, ZMF Atrium Closed, ZMF Bokeh, Beyerdynamic DT 990 Pro, HifiMan HE400i, Apple Airpods Pro Gen2,  STAX DR-L300 Limited, Sony WH-1000XM5,  Truthear Nova, Dan Clark Audio Aeon Closed X, AKG K271, Meze 109 Pro, Sennheiser HD650

Speakers:

JBL L100 Classic, Monitor Audio Bronze 100, Genelec 8351b (with GLM room correction), Altec 14, Wharfedale Dovedale, APS Klasik Studio Monitor, Triangle Altea ESW, Dynavoice DF-5, KEF R900, Acoustic Energy AE1 Active, KEF 105.2 Reference, ProAc Response DT8, Harbeth M30.2 + REL T/9x sub, DIY SEAS-based speakers, Driade Premium 2S, Canton Reference 3.2 DC, Linkwitz LX521.4, Neumann KH310A, Eminent Tech LFT-8b, Dali Grand Coupe, Dynaudio S 1.3 SE, Dali Menuet SE, B&W 802-D3, B&W 683 + SVS SB-1000 sub, ATC SCM40A + Arendal 1723-1S sub, KEF R3 Meta, Spatial Audio Lab M5 Sapphire, Paradigm Premiere 800F + SVS PB-2000 sub, DeVore O Baby, Yamaha C40, Neumann K+H O300, Unity Audio Boulder MKI + JLAudio 12" subs, Magneplanar MG1.7, Martin Logan Ascent

Nice cross-section of hardware in use out in the real world spanning a wide price range.

Clearly many of you spent significant time listening and this shows in the detailed responses including how you set up the gear, and even descriptions of the room and various treatments and specialty cables (not listed).

I noticed 2 resubmissions where the respondent added extra information as a separate entry that I consolidated, otherwise I did not see any irregularities to the submissions suggesting that people "cheated" or data that looked bot-generated.

III. Did listeners get the impression of audible differences between the Test Samples (DACs)?

18% thought that there were no audible differences between the samples and another 25% felt there was "very little difference" thus not worth spending any money to upgrade the DACs.

Remember, we're talking about streamers/DACs varying from US$10 to US$20,000 with the sonic output captured in high-resolution and reproduced using the hardware listed above with the majority of DACs these days capable of better-than-CD quality. Almost half - 43% - of the listeners thought it simply wasn't worth spending more money given the lack of change in the hi-res audio signature captured.

That still means that 57% thought there was a difference in the sound and they would consider upgrading based on the difference. 20% of respondents went beyond and felt they "definitely need(ed) to upgrade" or it was "essential" to upgrade based on what they heard.

IV. Of the listeners who heard a difference, in what device order did they prefer?

Now we start getting into the "meat" of the matter...

Since those who said "No Difference" left the DACs unsorted (A-B-C), let's filter out that group, leaving a total of 86 (82%) responses with preference ranking:

Listeners ranked by #1 being "best" and #3 "worst". Taking an average of all 86 responses, we are able to see that in fact the more expensive and objectively better performing DACs were both able to beat the Apple USB-C headphone dongle! I think this is a nice demonstration that with enough audiophile listeners, we can actually achieve the "power" to detect differences using blind testing.

Surprised?

Notice from the length of the bars that the difference isn't by any means massive. If there is absolutely no difference, all the devices would average out to "2". The question then becomes, is this statistically significant? The "null hypothesis" being that these deviations are just a result of random chance. Let's run this through a 3x3 (3 devices x 3 categories of preferences) χ2 test using the raw data to create the multivariate contingency table. 

Calculated Pearson's χ2 = 6.35, p-value = .175 therefore not significant even if we relax this to p < .10 level of confidence.

So while the overall pattern is not statistically significant by the usual definition, I would say that we're seeing a "trend" worth examining deeper. I know typically we talk about 5% (p < .05) as a threshold, but this is audio so I think it's fine to even relax things a bit.

Let's keep looking and asking questions...

V. Okay then, what if we look only at listeners who thought they heard a bigger difference and felt that the DAC was worth upgrading?

This will filter out the 26 responses that thought they heard a difference but "not worth spending money to upgrade", leaving us with 60 responses to analyze who felt more confident in their choice:

That pattern of preference for the Linn Klimax DSM/2 > Linn Majik DS > Apple USB-C dongle persists with now an even higher ranking for the Linn Klimax DSM/2. So what does the statistical analysis suggest?

χ2 = 12, p-value = .0174 therefore is likely significant with p < .05.

Way to go audiophiles!

I ran various analyses beyond this but since the number of respondents drops precipitously, statistical analysis did not show any further significance for those who expressed an even higher confidence that the DACs they chose were worth upgrading to.

Here's something interesting though - for those 6 who were the most confident that it was "essential" to upgrade the DAC because they heard a major difference, they liked the Linn Majik DS/1 + Dynamik the "best":

With only 6 samples, not unexpectedly, this was not calculated to be a significant pattern. It is however a good reminder that expressing confidence that one heard very audible differences does not mean the preference of the person was towards what we might expect to be the best or most expensive DAC.

VI. Does age play a role in discriminating differences?

As you can see in the Demographics (I) section above, there were 36 participants aged <50 years old. Of this group, 27 of them (75%) felt they heard a difference. In the >50 years old group, there were 69 and a total of 59 (86%) thought they could hear a difference.

Let's have a look at the preferences of the "younger" ≤50 years vs. the "older" >50 years groups who heard a difference:

Interesting, it looks like the younger folks were overall better able to, as a group, differentiate the Linn devices from the Apple USB-C dongle. Statistically not significant however and probably not surprising with younger ears.

VII. Was there a difference between headphone listeners vs. speaker listeners?

Check this out - for the listeners who thought they could hear a difference, let's separate them by whether they used headphones, speakers, or both:

Excellent! That pattern among the headphone listeners is highly significant with χ2 = 18.9, p-value = .00084; clearly much better than the typical p < .05 significance threshold!

To better illustrate the significance of this finding among headphone listeners, let me graph the results for each device based on the distribution of preferences:

Notice how many thought the Apple USB-C was the worst sounding of the 3 devices! In comparison, notice the relative preference for the DSM/2 as either "best" or "middle" subjective quality with few thinking it sounded the "worst". Beautiful!

So, headphone listeners as a cohort were the ones able to best differentiate the Linn DACs as sounding better than the Apple dongle. Notice that the loudspeaker listeners ranked the Linn Majik DS + Dynamik as sounding best to them and the expensive Klimax DSM/2 as being the worst! While the number is small at only 9, the Headphone + Speakers group did think the Klimax DSM/2 sounded best, but still ranked the Apple dongle as second best.

My respect to the headphone listeners as a group!

VIII. How about those with higher priced sound systems?

Already I think from what we see above achieved by the headphone listeners, the price of our hardware might not need to be too expensive to appreciate the difference between DACs. So, how did those who believe they heard a difference using <US$2,000 gear perform compared to those using >US$2,000?

Indeed, it's actually those listeners who overall spent less money that performed better picking the more expensive Linn streamers! Obviously $2,000 isn't a lot of money, but in comparison, while those who spent >US$2,000 were still able to separate out the Linns from the Apple dongle, thought that the less expensive Majik DS edged out the Klimax DSM/2 slightly.

For the subgroup who noted that they heard a difference (even if very small), using systems costing <US$2,000, with an n=35, we get χ2 = 8.06, p-value = .09. I'll leave you to decide if this is significant enough! At this lower price range, clearly we're again likely just seeing the ability of headphone listeners to discern a difference.

How did those with some of the most expensive systems do? Let's look at the 13 who had systems >US$10k:

Yikes. The listeners with more expensive systems ranked the $10 Apple USB-C dongle above the $20k Klimax DSM/2. As expected, the vast majority of these listeners are older (90% between 51-70 years old), 100% males, 92% using loudspeakers.

IX. How did the Musicians and Audio Engineering subgroups do?

While the numbers are small, we can see that the Apple USB-C dongle did really well even among these subgroups who in principle have more ear-training experience and musical knowledge. The audio engineers did show a good preference towards the Linn Klimax DSM/2.

Portrait of The Golden Ear 🙂

X. Any Golden Ears out there!?

As with my other blind listening tests/surveys, I like to look for those individuals who were able to accurately select the "best" components with confidence. For this test then, I'd like to award it to the 3 European listeners from the UK, Sweden, and Russia who selected the C-B-A order of preference (Klimax > Majik > Apple dongle), declared that the DAC upgrade was "definitely" worth it (though not "essential"), and all 3 achieved this while using gear <US$2,000!

What hardware did these guys use?

Macbook Pro M3 (audio at 96kHz) → TRUTHEAR x Crinacle Zero IEM 

E1DA 9039S / SMSL DL200 → Beyerdynamic DT770 Pro and Topping PA5 → speakers (not identified)

Windows PC → Cambridge DACMagic 100 → Cambridge Topaz AM10 → Wharfedale Dovedale speakers

Nice work boys!

XI. Summary

Well done and thanks for all the time spent on the listening, audiophiles!

Although there are other correlations in the data I could look for, I think I've answered the questions I was most curious about based on the 105 of you who sent in your listening impression from around the world. Let's then summarize the most important points:

1. Audibility of DACs, even when devices are as disparate as a US$10 Apple USB-C headphone dongle and US$20k Linn Klimax DSM/2 (based on 2020 MSRP), is not high. (Section III) 18% of respondents felt they could not discern a difference, and another 25% felt that whatever differences they thought they heard were so small, that it wasn't worth spending money on. So in total 43% simply did not think there is a valuable "upgrade" path. On the other end, 20% did feel they heard a difference and thought they "definitely" or even "essentially" needed to upgrade to the better sounding device (even if what they thought was better sounding might not have been the most expensive Linn).

With the actual DACs connected to the same system, it's certainly possible that the effect would be stronger than what was heard by respondents playing back the 24/96 files. However, linnrd and I did some volume-controlled listening on the evening of the test sample recording and to be honest, even with his excellent audio system, the Majik and Klimax sounded basically identical with instantaneous A/B switching while Roon streamed the same music simultaneously to both. Even if we were able to show the Klimax beating the Majik, it would be far from a slam dunk! We can talk more about this next time when discussing subjective impressions.

2. There was a trend towards preferring the objectively higher performing and more expensive devices: $20k Linn Klimax DSM/2 > $3k Linn Majik DS + Dynamik Power Supply > $10 Apple dongle among all 86 listeners who felt they heard a difference. (Section IV)

That trend towards preferring the Linn streamers did seem to hit significance when we consider those who had more confidence in their ranking and thought it might be at least "worth spending money" to upgrade.

This finding I think is also a nod to the idea that in a blind test, audiophiles do overall prefer higher-fidelity gear which correlates to better objective performance (like the 1kHz 0dBFS THD+N), at least in this test.

3. So what subgroup of listeners was best able to rank in the expected order based on price and highest-to-lowest performing devices? Clearly the headphone listeners. (Section VII)

This subgroup of 42 respondents was the "engine" that drove audibility to the point of demonstrating a significant result! They were able to demonstrate the superiority of the more expensive Linn streamers, particularly the Klimax DSM/2, over the Apple USB-C dongle.

4. Was there a correlation towards preference for the more expensive (and higher objective performance) Linn devices with higher-price sound systems? NO! (Section VIII)

Since headphone users clearly discerned the differences better, their systems typically would be priced lower than those listening with full loudspeakers (Section II). To give a concrete number, those who spent <US$2,000 in their audio system were more accurately able to identify the Linn Klimax DSM/2 as the best sounding device with 74% of listeners in that group using headphones only and another 9% using both headphones and speakers to evaluate.

This is important because some audiophiles claim that in order to hear differences with more expensive DACs, one must in a way "match" the price of the system and the component. Clearly the "sound" of the $20k Linn Klimax DSM/2 did not require that listeners have >$20k sound systems to significantly rank as subjectively sounding "better"! A nice headphone system say around $1-2k performed well already these days and I suspect will allow the audiophile to perceive differences to a resolution that their hearing will permit. I saw no evidence among the small group of listeners with much more expensive systems that they preferred the more expensive and higher-resolution Linn Klimax DSM/2.

This suggests that when listening with loudspeakers in typical rooms, there is a lower chance of benefitting from the highest performing DACs (those with highest dynamic range, lowest distortion). This makes sense when thinking about the typical ambient noise, limitations of speaker resolution, and effect from room reflections. As usual, I would highly recommend loudspeaker listeners to focus more on the sound room before getting too excited with very expensive DACs or possibly even speakers in order to maximize quality.

5. I don't see good evidence that age itself played a big part in the results obtained. The differences in Section VI were likely the result of older adults listening to loudspeakers and younger listeners using headphones. Among the 35 who used loudspeakers, 88% were 51 years old or above as compared to only 55% of headphone respondents (n=42) being above 51 years old.

6. You don't need to be a musician or audio engineer.  (Section IX) While the audio engineer group preferred the Linn DSM/2 as sounding the best, neither of these groups in aggregate ranked Linn Klimax DSM/2 > Linn Majik DS > Apple USB-C dongle as one might expect based on objective performance and price.

--------------------

TL;DR: Yes, audiophiles can tell the difference between higher performing DACs like the Linn streamers compared to the Apple USB-C dongle with preference for the Linns. Headphone users were able to differentiate best. And one does not need to spend a lot of money to resolve the differences between the hi-res 24/96, FLAC lossless test files.

Obvious Next Step: Can listeners differentiate the $20k Linn Klimax DSM/2 from a $120 Topping D10s (or $150 D10 Balanced), both devices with THD+N better than -100dB?! Anyone up for a follow-up blind listening survey?! 🤔 The hypothesis being that the objective performance is what's important and correlates with "best" sounding, not the "high-end" price or brand name (counter to Jim Austin's claim as discussed last week).

While this isn't research done in a formal audio lab with trained listeners by researchers with grants, I think what I'm offering here is a realistic evaluation based on the perception of more than 100 audiophiles around the world who donated their time. I hope we as audiophiles are not afraid to ask the questions that need to be asked and make sure that we collect the "crowd sourced" data based on controlled procedures. Employing safeguards against strong biases (like using blind listening) rather than accepting all opinions at face value is essential if we are to find truth (J Gordon Holt was right). Beyond objective measurements which we can collect and compare, this is how as hobbyists, we too can try contributing in a practical way to "science".

As per my usual sequence, next time in Part III let me publish some of the individual comments I received and the more subjective observations from respondents.

--------------------

Checking out Twenty One Pilots' latest album Clancy (2024, DR5 2-channel which is atrocious, DR12 multichannel/Atmos much better) this weekend for some indie pop / alt. rock flavors...

As discussed not long ago, the multichannel downmix is often a much different and IMO superior experience from the 2-channel mix! In comparison to the relatively small difference between DACs, this is way more important.

As usual, albums like this do not benefit from 24-bits! Waste of storage space.

I hope you're all enjoying the music, dear audiophiles!

Addendum - May 27, 2024:

Added graph showing distribution of "best"/"mid"/"worst" by subjective preference for the headphone listeners who reported hearing a difference, showing the nature of that statistically significant finding.