*Surprise!* It's time for another listening survey.
I like doing these once awhile because it's an opportunity for audiophiles to actually listen for differences between devices, resolutions, or even things like filter settings for themselves as opposed to hanging on to the testimonies of others whether in text or expressed in videos. We can argue all day long about whether THD+N of -120dB is better than -85dB, or if a US$20,000 DAC sounds better than a US$500 one; what matters ultimately is in the listening.
[I agree with Taylor Christensen in his excellent article on "subjectivisms" that what we need more of these days is audibility data to address the category of "scientific subjectivism"; not just opinions or measurements.]
This time, I'll be using recordings of the AMPT Test Track from 3 different DACs for you to listen to. These were created based on my standard procedure of recording and processing described in the link. The only difference is that I'll be using the 24-bit version of the AMPT for this survey to maximize higher resolution dynamic range potential from the DACs.
As a blind survey, I will not tell you which DACs these are to minimize bias. What I can say for now is that these are not devices from Chinese brands although anything can be "Made in China" these days (ie. these are not Topping, SMSL, Sabaj, Shanling, Matrix, Gustard, etc.). Some of these devices can be considered "aspirational" DACs or streamers in that the MSRP of the device could be out of reach to many, even priced "ridiculously" high.
Perhaps this fact might encourage you to listen for yourself: the difference between the least expensive and most expensive device is >100 times!
I guess we'll see when the results come out whether we might have to talk about "diminishing returns".
Based on these 24/96 recordings captured using an excellent ADC (RME ADI-2 Pro FS) which is capable of hi-res studio quality, can you hear a difference between these DACs? Here's what to do...
Step 1: Download the hi-res samples here:
Archimago's Musings - 2024 - DAC Listening Survey (24-96, FLAC)
It's a pretty big download - about 515MB. Here at the Musings, we don't do YouTube listening tests, baby... 😎 (If the differences are big enough, sure, we can hear the change even over YouTube.) The 3 samples are in 24/96 FLAC format, recorded from the ADC with very minimal volume adjustment applied. This should easily capture any significant audible difference between these DACs when you play back the Samples with your high-resolution-capable sound system.
Step 2: Have a listen.
Because the AMPT audio files are processed the same way and volume normalized to a target, feel free to use something like the ABX Comparator in Foobar to quickly switch between samples when listening to hone into specific portions where differences can be heard. Note that it is normal for 10 seconds of silence at the start of the track to listen for hum or other noises coming from the DAC output. I would caution not to push the volume too high because the first music track is very dynamic!
The order of the DAC/streamer devices were randomized the old fashioned way - I had my daughter pull the names from a hat.
Here's the Dynamic Range Meter result of the 3 tracks for comparison and confirmation that average RMS amplitudes are well-matched:
DACs will have differences like filter settings, or maybe FPGA processing as the case might be. This can affect values such as the level peaks which could just be the peak at one single sample across the almost 10-minute track so I would caution against using this to make a correlation with sound quality.
There's only at most 0.05dB average RMS level difference between the 3 samples. This difference is clearly small enough that it would not be the main factor to create qualitative bias.
Listen and keep note of your preferences because I'll be asking you to rank the 3 samples from best to worst sounding. Listen for tonality, soundstage, details, dynamics, transient response, instrument placement, "realism", naturalness of vocals, etc...
Feel free to use headphone or speakers. As usual, let me know which you used, or even both!
Step 3: Submit your survey results here:
https://freeonlinesurveys.com/s/LRUv44ja
The questionnaire should be straight forward with enough detail for subgroup analysis. Here's what I'm going to ask:
- How would you rank Samples A, B, C in order from "best" to "worst". Arrange in your preferential order.
- How much difference did you hear - you have a few options from "No difference" to "Very different". I believe it's always important for respondents to have the option to express if they honestly hear no difference.
- Some demographics info - age, gender, musicians, those who do audio engineering work. Good to track these even if it's just for the sake of knowing a little more about the readership and those who partake in audiophile listening surveys!
- What kind of system(s) did you use: headphones, speakers, headphones+speakers.
- Approximate price of your hardware system - include source, DAC, amp, transducers; let's not worry about cable price, tweaks or room treatments - from <US$100 to >US$30k.
- Comments section (optional). Feel free to list which amp / headphones / speakers / DAC you used. Feel free to describe the differences you heard - I'll also list the tracks that make up the AMPT file in that question for reference if you want to specify which musical passage(s) allowed you to hear the most difference. Some listeners like putting their alias here for future reference - for example, if someone forgot how they rated, I can go back and let them know. Otherwise, all data collected anonymously.
If you've never participated in one of my listening surveys, here's your chance to get your audiophile "golden ears" on and your opinions counted! As usual, I'll be unblinding the 3 samples in an article ahead and analyze the data to pick out trends, preferences and the truly "golden eared" among you! 🫡
Like with previous surveys, please refrain from objective analysis discussions at this stage because this is a listening exercise. Objective results and discussions would make more sense once I unblind the devices later on.
So as to give time for everyone to listen and gather your thoughts, let's keep the survey open for 5 weeks or so (close-off date May 15th). I'll be away in early May for a visit to NYC, perfect time to close off after that.
Thank you for your participation in advance!
Let's have fun experiencing and learning.
To end, I just wanted to mention Beyoncé's Cowboy Carter album which dropped last week. Obviously, no album caters to everyone's taste but I really enjoyed this one. Well done cover of "Blackbiird", the pop-country Top 40 made-for-radio "Texas Hold 'Em", and I like this version of "Jolene" way more than good 'ol Dolly's original (blasphemy!).
Interesting idea but I'm not really sure how the additional ADC/DAC loop between the DUTs and the listener is going to be dismissed.
ReplyDeleteHi shoddy,
DeleteYeah there will inevitably be an ADC/DAC loop here but these days that's highly transparent with objectively confirmed hi-res capability and we're capturing the frequency bandwidth to 48kHz which would include the octave about and the filtering ability of the DAC.
My hope is that if there is something to be said about the value of very expensive DACs - things that can cost thousands, even tens of thousands of dollars - as opposed to maybe something in the hundreds of dollars at most, that as a corporate group of audiophiles, we'd be able to detect change in blinded listening to make it worth our while to consider such an expense!
I can see what you mean, but -whilst this reasoning holds good for listening to turntables- I'm not really sure it works for dacs. If the difference between a 10k Dac DUT and the dac you are listening on is that the DUT produces much less distortion, how are you going to hear that? Why will the output *not* be dominated by the response of the listening dac? This is all good clean fun but I don;t think it will persuade anyone. And in this case, the refusal to be persuaded will not be bonkers. It might I suppose make sense if we were listening to different devices' 16/44 reconstruction filters captured at 24/96.
DeleteHey Shoddy,
DeleteYeah that would be true if we believe that most of the difference in sound between say a $250 and $10,000 DAC (not exactly the price difference we're talking about in the actual test) is the level of distortion and a poor-quality DAC will mask this. That's why I ask folks to tell me the cost of their systems so we can perhaps try to make some correlations.
When we read reviews of expensive DACs, I don't think the descriptions are of changes in distortion usually. They're typically about frequency changes like deeper or more authoritative bass, more extended highs, wider sound stage, cleaner tonality, vague subjective descriptions that may or may not apply to anyone else, etc.
For example, the dCS Bartok review in Stereophile by Herb Reichert in August 2023 uses descriptions like "quieter, deeper, more corporeal, less grainy", there are other descriptions like "texture" (?detail?), "atmospherics" (?soundstage, reverb time?), it gets even more ethereal like "wet feel to its clarity" as opposed to being "transistor-dry". I'm personally not sure what some of this is or if it's all just the imaginary projection of an older gentleman, but if it's related to electrical waveforms, I don't think the RME ADC will have difficulty capturing it.
We know that the RME ADC is capable of capturing the playback with flat response (easily to ultrasonic 48kHz with these recordings), at least down to -110dB THD+N when connected to a high-resolution DAC input, better than 100dB channel separation, with a noise floor down in the -150dB range so I don't think distortion is what we're worrying about. No music requires this level of dynamic range either and no speaker/headphone system is capable of that level of detail even if one's DAC is not the greatest!
The question remains. If hi-end reviewers and connoisseurs of $10,000 DACs feel they make a highly significant difference, then I don't see why there would be any problem for the ADC of this quality to capture it, and even with fair quality playback gear, we *should* be able to hear at least a statistical difference between the least and most expensive device I would think given the massive disparity of >100x price difference!
Ideally, it would be good to see a significant bias towards preference for the more expensive device, of course.
Dear Archimago.
ReplyDeleteSpeaking of XTC, try the beginning of Verdi's Otello in this edition. The effect is magnificent.
Greetings from Stefano. https://music.apple.com/it/album/verdi-otello/1452740578?ls
Nice Thing!
DeleteWill need to give this one a try. Have not listened to opera in awhile!
Just had a listen - yup! Great effect on that Otello! Thanks.
DeleteHappy that you like it. XTC is really magical
DeleteYou cannot tell a difference between really expensive DACs you recorded if one listens to them through a normally priced DAC, amplifier and speakers at home. In order to tell apart a DAC which costs $5,000 from another DAC which costs $50,000, one needs to compare them in a system which costs $200,000 and in which no component will be the bottleneck and the weakest link.
ReplyDeleteThat would be true if price of audio components were directly related to their sonic transparency. I think one point of this survey might be to try to test this hypothesis.
DeleteThere is an anecdotal evidence in audiophile press that differences between N-ten thousand dollar tonearms can be heard via the speaker of an iPad (https://www.stereophile.com/content/analog-corner-282-oma-plinth-technics-sp10-sat-cf1-12-tonearm-doshi-v30-phono-stage). It just depends on talents of the listener. Or luck. That's why it is needed to gather some stats.
:-) You're pulling our leg, right fgk?
DeleteI agree with Mikhail. Plenty of anecdotes that this is not the case even if dealers and manufacturers want us to think there's a correlation between price & performance.
Heck, even that link to Jay's Audio Lab video on digital streaming vs. LP playback over YouTube with him using something over $500k "worth" of gear we can easily hear the difference on simple headphones and decent computer speakers as discussed earlier this year:
https://archimago.blogspot.com/2024/01/musings-on-youtube-listening-tests-and.html
I don't think this lack of correlation is in doubt. The question is, can we hear a significant difference that might be in favour of the expensive devices - recorded directly from the DAC in hi-res - between devices that are priced >100x apart?!
I know some of the respondents already are listening and giving me their feedback with playback systems >$30k (as well as almost every price range in between...).
The way you designed your testing is faulty.
DeleteIf one cannot hear a difference, how you'll be able to tell why no difference can be heard? Is it because (1) there is no difference indeed, or simply because (2) the user, while trying to hear a difference, used an instrument (his home audio system) which is not transparent enough to show this difference?
Analogy: if the resolution of your monitor is 480 x 640, it's useless to compare on it 720 p and 1080 p video recordings.
Of course, there is a correlation in general between the price and performance under all other conditions being equal. Assuming the manufacturers are honest and did not inflate their profit margin. If the price is high, it means that the manufacturer had a higher budget and he used better components. Better components = better sound.
DeleteI would have to respectfully disagree with the general premise, fgk.
DeleteOver the years of listening and testing, I have never been too concerned about price correlating with actual resolution of a device. Sure, a <$5 DAC would not perform well (as we saw with the CheapDAC'22 article) or the limitations of the little $30 AGPTEK digital player recently. If anything, there are just as many disappointments on the "high end" (like the Vitus Audio amp from years back at huge costs).
IMO, the analogy of comparing 480P vs. 720P vs. 1080P which is a technical spec would not really apply here. Even when it comes to 1080P TV sets (or more likely 4K these days), the price range can be huge and many inexpensive models would be just as good as very expensive ones to show the improvements in vertical resolution - would anyone claim that a $15,000 65" Beovision Eclipse TV be better resolution-wise than a $2000 Samsung 4K OLED of the same size?
While I keep myself "blind" to the results as they come in so I'm not biased when I respond to comments, I can say that after making these recordings, both myself and the owner of the DACs had our own listening session using an expensive system.
What we're hearing on these hi-res recordings we felt correlated with the listening session with the actual devices, volume-matched down to <0.5dB in his room.
BTW: Notice that I ask listeners to estimate their system price. So in principle there should be a correlation between preference for the most expensive DAC on those higher-priced systems? We'll see!
One more thing, the manufacturers of these devices will be well-recognized and respectable brands that use good components, known in the audiophile world for decades; not some small outfit that sells bespoke devices charging $$$ for being exotic.
Fgk has a point about the user's home DAC being a severely confounding factor. There's nothing to be done about it, but it will be difficult to draw conclusions from a negative result. Still worth doing, in part because there is a chance of a positive result!
DeleteYup Neil,
DeleteWe can talk about it later with further analysis...
First things first - let's collect audibility data to the best of our ability!
Not only the user's home DAC (and also the user's amplifier and the user's speakers AND the room) will be a confounding factor, but also Archi's home audio system (DAC, amplfier, speakers AND the room acoustics) may be not revealing and resolving enough to show the difference between the audited DACs.
DeleteArchi, what kind of amplifier and speakers did you use to reproduced the audio material you offer for testing purposes? If your amplifier and your speakers are, let's assume, 10-20 times less expensive then one of the DACs participating in the testing, why do you expect that YOUR system is not a chokepoint?
The way this survey is being set up is the objectivist's roguery.
Its results will be as trustworthy as the latest Vladimir Putin's "presidential election" results (87%).
We shall see fgk,
DeleteI listened volume-matched with Pass Labs amps and Avantgarde Acoustic speakers at my friend's place which was his spacious listening room. The system has good frequency response and room with low reverb time. Even had the DAC company's expensive cables hooked up (vs. inexpensive interconnects for the least expensive device). Don't know the price of all that stuff but let's say it ain't cheap.
I don't know what's "roguery" here. Once I unblind, you'll see what I mean by the massive price difference. Regardless, the results will still be a data point for audiophiles to consider I think given the music material, the resolution of the ADC capture (compared to so many subjective-leaning folks using YouTube samples!), and for those who want to further analyze, they can always download this to listen for themselves well after the listening survey is closed... By all means, if the test is clearly misleading or wrong, I would love others to show me.
Hi Arch,
ReplyDeleteI have not submitted my final impressions despite spending quite some time listening to the tracks on several systems. Oddly enough, my initial response is that I found it easier to distinguish the tracks when listening to speakers as compared to when listening to my headphones. I found it quite difficult to find a preference using headphones, regardless of the pair I used, finding that test frustrating. Much easier when listening to most of my speakers regardless of amplification. I have D, A and AB. 😊 Both setups employ supposedly quality dacs in the sub 1000 dollar range. I shall plough on!
Cheers
Mike
Thanks Mike for the listening!
DeleteThat's definitely what I want folks to try and ultimately once they've heard what they needed to (or not as the case may be), let me know what they think. Whether it's with speakers or headphones; I think each will have their plusses and minuses - for example, speaker soundstage is vastly different from using headphones whereas headphone "intimacy" for details and noise level likely would be superior.
Yesterday, I came across this John Darko article:
Wishful thinking is endemic to the hi-fi show experience
Where he says:
"Alternatively, we might wish that our “$100 DAC sounds just as good as any $10,000 DAC”. Never mind that a half-decent side-by-side comparison would send this idea into orbit. If we can avoid comparative demos, our wishful thinking remains intact."
What a timely comment given this test! He clearly believes comparisons would be easy (send the idea of just as good "into orbit"?). Let's see how this turns out considering that the relative price points I'm looking at are even greater than the numbers he threw out! I'm not sure which side in these debates have more "wishful thinking"; let's not worry about that at this time and listen for ourselves using the best gear we have.
BTW: I notice that I can't click on links to Darko's articles like the one above from this site. I don't know if this is in general forbidden from Blogspot or just from here. Anyhow, you can always right-click and "Open link in incognito window" to see the post and not let him know where you came from. :-) Silly.
Hi Arch,
DeleteAh yes, reviewers. There is an interesting thread to read on audioscience regarding YouTube reviewer Erin and Eric at Tekton, who was unimpressed by a speaker review. https://www.audiosciencereview.com/forum/index.php?threads/legal-fund-for-reviewers-erin.53577/
As stated by many, I view YouTube reviewers as entertainers, and would not allow their reviews to dictate my purchases. I have seen, read, and heard far too much regarding these reviewers to ever trust them. I do make an exception for you Arch 😉
Hey there Mike,
DeleteYeah I saw the uproar around Tekton this week as well and made a few references to it online at Stereophile and this thread on the Steve Hoffman forum.
Even a cursory look at the nature of that speaker tells us that the Tekton Troubadour will be "measurement challenged" with its large front baffle, multiple tweeters, lack of dispersion control. Mr. Alexander has already said what matters to him is how something sounds... Then he should stick with that and acknowledge that obviously a speaker like this will not measure particularly well yet many people probably would still like it.
No big deal, many Pass Labs amps measure "poorly" and I don't think Wilson speakers measure particularly well but there's a market and people still like the gear or the brand. For a $2,000 pair of speakers (Erin wasn't even trashing it and I think he said he even liked the sound), Eric Alexander should IMO just "own it" and say it's his intent to design such a thing...
No need for bullying litigation tactics or fear the facts of what the measurement system found since probably a lot of it was anticipated.
BTW: This "apology" video that he's not planning to sue anyone from Eric Alexander is interesting.
DeleteNot sure that helps much as he tries to temper the intensity of the word "litigation" :-). Will be interesting to see if he actually publishes measurements he says he would.
Video went "private". :( Would love to hear what he said!
DeleteBTW thanks for the test, just submitted my results! Interesting differences on my system so I'm hoping my hearing is still decent. Great that the samples are long and different music genres represented.
ReplyDelete