Saturday, 26 January 2019

INTERNET BLIND TEST: Do digital audio players sound different? (Playing 16/44.1 music.)


The other day, I was surfing the usual audiophile haunting grounds and came across this poll question on the Steve Hoffman Hardware Forum started a year ago. It reads: "Do You (Think) That Different CD Players Have Their Own Sonic Signature?"

Here are the results when I looked recently. I know it was closed last year, but it looks like it might be re-opened and still running into April if you want to cast a vote:


I thought it's worth spending some time to discuss this very simple question which of course is both understandable and asked frequently among newbies and longtime audiophiles alike. It's an important, and "core" question which can still be highly contentious after all these years! As suggested by the "blind test" title, by the end of this post, I will give you dear readers an opportunity to "cast your vote" and tell me what you think with a poll of our own using test samples to try (if you dare)!

 Let's make this our winter/spring research project...

First, notice the poll result above. Since when do we see audiophiles / music lovers vote 9:1 in the same direction? Almost never, right? So this is essentially a unanimous belief; survey says "Yes", CD players do sound different with their own sonic "signature" proclaimed by 263 of 296 total respondents.

Although the question is being asked about CD players, I think it's fair to suggest that this question applies to all digital players while reproducing 16/44.1 music which of course accounts for the majority of the digital music we're consuming today and for the foreseeable future.

The opinion expressed above is not unreasonable. We can easily come up with a few explanations as to why devices could perceptibly differ:
1. Different quality analogue outputs. Some will be noisier. Some will have different frequency response. Some (especially those with tube output stages) may also have higher measurable distortion (but distortion could be subjectively "euphonic"). 
2. Different DAC chip/circuits will also inherently vary as per (1). But from the measurements I've shown over the years, these differences tend to be small when looking at 16/44.1 test signals from good quality modern devices. The ability to resolve bit-depth will depend on whether the device is truly hi-res capable or not.

3. Depending on the digital input used and the circuitry, jitter will vary.  For example, back in the day I demonstrated the different jitter levels using SPDIF vs. USB interfaces. HDMI also will typically have higher jitter levels than asynchronous interfaces (check out the Oppo BDP-105 jitter with HDMI input for example though these days modern updates like the Oppo UDP-205 perform even better). I refer you again to the jitter demo I put up last year to consider for yourself whether jitter is a significant issue.
4. Different digital filters affect the analogue output. This is perhaps best demonstrated by results from filterless designs like the old "non-oversampling" (NOS) DACs and newer devices using weak filters. Examples over the years include the old Philips TDA154X NOS chips and the PonoPlayer with its Ayre-designed slow roll-off, minimum phase filter. Both of these will affect frequency response and a minimum phase setting will result in relative phase/time deviation across the spectrum which may or may not be of audible significance.
While we can say and know the above to be true, how many of us have had an opportunity to actually evaluate the impression that digital players have significant "sonic signatures" in any "controlled" fashion with our own ears? In other words, how many of us have actually evaluated the magnitude of these differences when volume is controlled, using the same amp and speakers, have the ability to quickly A/B switch so that we can compare within the limits of echoic memory? Just as important, how many of us have tried this evaluation in a blinded fashion, free from preconceived biases (price, brand, reputation, appearance)? I bet very few of us could say that we have taken the time to try listening with "controls" in place.

Here's your chance to put your ears to the test and listen in the comfort of your own homes! Time for one of our "Internet Blind Tests". Here is your mission, should you wish to accept it...

Ready for the test? Here's the procedure...

1. Download this big file. Make no mistake, this is a 470MB ZIP file. Within it contains 4 song samples captured using the professional grade RME ADI-2 Pro FS ADC at 24/96 from four different digital audio devices. Care was taken to ensure there was no clipping or other anomalies. I will not reveal the identity of these 4 devices yet, but they could be CD players, standalone DACs, mobile devices, network audio players, etc. What I will say is that when I reveal the identity of these devices, they will be recognizable as representative of their "class" of device and will not be esoteric audio players unfamiliar to music lovers and audiophiles. In fact, you might be a little surprised by the devices chosen.

The 4 songs all originated in 16/44.1 as verified "bit-perfect" CD rips. They were played as a playlist through the respective test devices, and then captured using the ADC in 24/96 quality at approximately the same volume. Afterwards I tweaked the files to make sure that the average loudness was almost identical:

I recorded the music in sequence using a playlist, totaling 6 minutes 32 seconds of audio for each device. With fine tuning of loudness in Adobe Audition (ITU-R BS.1770-2, amplitude processing with 32-bit resolution) afterwards, notice there is only +/-0.05dB difference between each recording's average RMS volume. At most, I had to boost the lowest amplitude device by +1.16dB, and lower the "loudest" device by -1.25dB. After this, I separated the recordings into individual songs. Notice the peak volumes have slightly larger range with 0.5dB between highest and lowest.
Obviously, I cannot ensure that you are hearing exactly what the devices sound like since we all have different playback set-ups and your DAC will have to convert the data back to analogue. What is more important I believe is that the direct recording of the analogue output derived from 16/44.1 playback into the same high resolution ADC at 24/96 is capable of capturing all the significant audible differences between the devices; or at least hopefully enough for your playback system to reproduce with high fidelity.

2. Listen to the samples. You will see that I have the music arranged in 4 directories corresponding to the device letters - Device A to D. To ensure that I did not consciously or subconsciously "rank" them by price or reputation, I had my son roll a dice to determine the order presented.

Within each Device's directory are the 4 sample tracks. The 4 musical samples I picked:
Maxi Priest - "Wild World"
- Well recorded pop song from the days when I was coming of age :-). Good DR13 recording without much dynamic compression, from the album The Best of Me (1991). 
Stephen Layton & Britten Sinfonia - "Chorus: For Unto Us A Child Is Born"
- Excellent classical / vocal recording from Handel Messiah. DR15 sample. Some strings, various vocal parts, delicately recorded. I know, we're past Christmas... Just work with me, alright? :-) 
C├ęcile McLorin Salvant - "Le Mal de Vivre"
- If you have not heard of Ms. Salvant, I think you're in for a treat. Let's not kid ourselves dear audiophiles, we love solo female vocals (eg. Diana Krall :-) and piano attacks and decays for sound evaluation. Here we have the combination of both in a nice DR14 recording from one of the best young jazz vocalists, IMO. This track can be found on For One to Love.
Joe Satriani - "Crowd Chant"
- For those who want to pump up the volume, here's a modern production weighing in at DR9. Rock with Joe as he hands out some distorted guitar licks while the crowd... er... chants through the "wall of sound". From Super Colossal.
Each track is a 1:30 to 2:00 excerpt. I don't expect you to have to listen to all 4 tracks with every one of the digital devices. Perhaps concentrate on the tracks that might appeal to you the most and that you feel would be the easiest to hear a difference with.

Can you hear the difference between the 4 devices? Is there a device that seems to sound best to you? Perhaps one device comparatively sounds poor? Grab a pen and paper and see if you can rate the devices from best sounding to worst and subjectively try to describe why.

Feel free to take the time to listen over a few sessions. Grab a friend / your kids / significant other. Maybe get a group of audiophiles together at the audio club meeting to listen and discuss.

As usual, I am using only a portion of the songs/tracks on the basis of fair use for the purpose of study / evaluation / research. I am not doing this for financial gain. Please delete these sample tracks once you are done with the evaluation. If you like the music, please purchase the albums as identified above.

3. Done listening? Submit your preference online! 
Before you click on the response link, remember that I'm not just interested in a simple "yes/no" answer since responses should be more nuanced. I'm interested in a little bit of demographics about you and why you respond the way you do... Here's what I want to collect:
- Your demographics - what part of the world, male/female, what decade of life? 
- Your sound system - briefly describe the system and approximate US$ MSRP cost.
- Did you use speakers or headphones (or both)? 
- Try to rank as best you can the 4 devices (Device A to D) in order from best to worst. 
- Think about how much difference you heard between BEST and WORST devices. 
- Think about how much difference you heard between BEST and SECOND BEST devices. (To get a sense of the difference between adjacent-ranked devices.)
- Did any specific track stand out as sounding different between devices?
- Let me know if you write / publish audio equipment reviews. This can be a blog, YouTube videos, all the way to having published magazine articles. Also, if you're a trained musician or have formal experience/training in sound evaluation, I want to know!
- Feel free to comment on subjective differences heard. Sense of "presence", "air", "frequency extension", "sound stage", "dynamics", "pacing"... All fair game for subjective descriptions.
If you spent some time listening to these tracks, then make sure to let your perceptions be counted. Whether you heard a big difference or very little difference or even no difference, I have provided the options for you to choose. Remember, all data is gathered anonymously for the results database although the poll site will prevent multiple entries from the same computer.

Ready? Submit your responses here.

By all means send this test out to all the audiophiles, head-fi folks, and "golden ears" you know who have the ability to play hi-res 24/96 music with good gear. Spread the link to this test around the audiophile forums. The more responses the better to get a good sense of the tested "real world" answer to the question being posed.

As usual, remember this is a listening test with the aim of gauging subjective preference. Please try not to load these tracks up into an audio editor for a peek until at least after submitting your responses. To be honest, I don't think looking at the files in an audio editor will help. Also, please let's not "share" answers which might bias others.

Unless something unforeseen comes up, let's run this test for 3 months - I'll close off the survey on April 30, 2019. Plenty of time to listen, consider, and submit results. As usual, afterwards I will compile the data and present them here. Let's see what people thought and if there appears to be any measurable subjective preference trends.

Thank you in advance for trying this out and getting me your survey results. Have fun and I hope you're enjoying the music!

February 1, 2019 - Fixed the slight pop in one of the tracks...

May 1, 2019 - Survey closed! Thanks for everyone who submitted their result... Will talk about results on the blog shortly.

19 comments:

  1. Fun!

    If you don't mind, for the first test, I'll see if I (well, my wife) can hear a difference between your bootleg Redbook and the songs streamed from the Fire TV.

    To my mind, if that low bar can't be cleared, what hope is there for splitting hairs amongst devices. But my bias is showing ;) ... So I'll start the test with my wife. I won't be able to perfectly level match though. Darn.

    ReplyDelete
    Replies
    1. Indeed Allan, your bias is showing :-O.

      I don't know at what quality the FireTV is able to convert audio, but I can imagine that some of the devices I used here could objectively be even "worse" than that!

      Anyhow, first things first, let me know what you heard in the survey :-).

      Good to see that some folks starting to send me results... Keep 'em coming!

      Delete
    2. Strictly speaking, the FireTV doesn't convert audio. The Oppo 105 does that. FireTV is just the "transport."

      Delete
    3. Oops. Just remembered how you had your system set up, Allan. In that case, no problem! Your BDP-105 should have no problems resolving this...

      Delete
  2. This is really interesting, Arch
    I'm hearing differences, but I can see how difficult it is to filter out externals.
    You don't say if the devices letters are the same for each of the tracks, but I'm definitely hearing the 'same difference' propagated for some of the devices on different tracks. This could mean that I've biased myself based on my response to a previous track. Good argument for double blinding, I think.
    Will continue to test these from time to time on better phones.
    Keep up the really interesting work
    Phil

    ReplyDelete
    Replies
    1. Hi Phil,
      Yes, the letters are the same for each Device. There's no trickery here. All the music in each "Device X" directory were from the same device.

      While I chose devices that are quite different, it's certainly possible that some are "more similar" than others so if you hear similarities, that's certainly not inconceivable! Let me know your preference. Remember folks, listen, "feel", ask yourself which one(s) you "enjoy" - don't over-think this.

      The fun part will be to correlate preferences as a group with things like objective results, price, "reputation" of devices, "class" of devices and seeing how this jives with "received knowledge" as music lovers and audiophiles.

      Ultimately, an honest audiophile who truly simply values sound quality should also be able to dissociate him/herself from other variables and just be open to "call it" as he/she hears it. Recognizing that there is "joyful equanimity" in whatever the results may be for oneself. Certainly no shame in honest reflection...

      Delete
    2. Hello Phil and everyone,

      You know, there's an easy way to confirm your audio perception abilities. Have a friend or family member randomly rename the files from A/B/C/D to 1/2/3/4, writing down on a piece of paper which they've rename to which. Then see if you can determine the mapping.

      Should be easy peasy, right? OK, maybe not all of them, but surely 1 or 2 with certainty. Right? ;)


      Arch,

      This also gives me an idea for next time, 2 of the 4 tracks are available on Amazon Music. You could do the same thing with streaming... either a fifth group that was streamed instead of from CD; or across streaming transports eg. Chromecast Audio vs. FireTV; or all the same streamer but across different transport layers: USB vs. SPDIF vs. TOSLINK vs. HDMI.

      Should be easy to tell them all apart, right? Of course, if one can't it can only mean one's kit must need upgraded. /sarc :)

      This idea came to me this evening when I happened to read a certain Lumin U1 review. My eyes started rolling with the Sound Quality section. I almost laughed out loud when I saw you had already commented. I don't know why I go there. And that new name... all I can think of is Baggie Style (as in bagholder, which is so apt in the ultra high-end audio market).

      BR,
      Allan

      Delete
  3. Hi
    I will certainly take the test, but I am wondering what I will be testing really.

    In my audiophile setup, I am making use of Roon + HQPlayer. Upsampling everything to DSD256, I do hear (experience) differences between Roon Filters & HQPlayer filters (and certainly between apodizing and non-apodizing filters). For now I am settled on an apodizing filter.
    I am quite 'concerned' that listening results will be quite different with this setup, compare e.g. using Jriver without upsampling.

    If that would be the case (have not tested yet), what am I testing? Not the perceived SQ of the source files, but the impactof the different audio players.

    Furthermore, is it not possible that different DACs with different filter implementations/ settings would again give different results.

    So, to summarize, I am not sure if my test results will in fact have any meaning at all.

    Some guidance on how to test this in the most 'valid' way (alos for myself) would be appreciated

    Dirk

    ReplyDelete
    Replies
    1. You will be testing yourself. And I use DSD256 as a sound effect so it should sound different.

      Delete
    2. Hi Dirk.

      Given that what we're trying to determine is whether these 4 very different DACs "sound different", the way I see it is that so long as you listen with exactly the same playback settings, the use of high quality filters with DSD256 would not be an issue.

      In fact, as a "naturalistic" study of audiophile and how they enjoy music, your input with Roon/HQPlayer/DSD256/apodizing filter would be just fine since objectively I can tell you that when I looked at HQPlayer with my Oppo, better-than-16-bit performance was easily achieved.

      Bottom line... Ensure your settings are high quality. Use the same settings in your listening trials. How much difference can you hear since these test tracks "only" have 16/44.1 resolution to begin with?

      Delete
  4. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. LOL. Thanks stalepie. I'll have to check that out. Hopefully no other issues and will see if that needs fixing.

      Appreciate the detailed listening!

      Addendum: Fixed the <1 second pop in the current file link... I trust this little anomaly won't affect results in any significant way. Comment removed in order to reduce bias.

      Delete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. Thanks Unknown,
      Yup, found your response and thanks for the details! I've added the comment... :-)

      Delete
  6. Thanks for taking the time to set this up. I’ve started listening to the tracks today and will report back this week.
    Great selection of music as well.

    ReplyDelete
  7. Would you mind making the original 16/44 extracts available? They will help me determine which device A-D is more accurate. I'd consider the more accurate one to be the best one.

    It's quite possible that the one that sounds best in my system (say A) is inaccurate in some way that compensates for a problem in my system, making A seem superior.

    You didn't ask for ranking from best sounding to worst sounding after all, but ideally, that best sounding one would also be the most accurate one.

    Thanks,
    - Eric


    ReplyDelete
    Replies
    1. Hi Eric,
      Yes, no worries, I will make the original 16/44's available after the blind test is complete!

      Accuracy and "best sounding" could be quite different things. As a naturalistic blind test through various systems, my interest is primarily to see if there are any patterns out there; any trends when it comes to the various devices used here. Perhaps more important are the questions about relative difference in sound quality heard between devices.

      We'll discuss more once the results are out and of course I release the original 16/44 files...

      Delete
    2. This comment has been removed by the author.

      Delete
  8. I'm a fool , i had not heard of Ms. Salvant ... thank you thank you !

    ReplyDelete