Saturday, 23 February 2019

MUSINGS / MEASUREMENTS: On why 24/96 sampling of the Blind Test of 16/44.1 devices is good enough. [On idealized filtering tests and "real life" music playback.]



As promised, I want to address a comment raised about the idea that the 24/96 recording using the RME ADI-2 Pro FS ADC was somehow inadequate for the 16/44.1 samples in the "Do digital audio players sound different?" blind test; that the 24/96 captures are somehow missing significant amounts of the output from the devices.

For reference, I've included the original comment here from Miska:
"It [the samples] just barely catches the first image band between 22.05 - 44.1 kHz and a little bit of next between 44.1 kHz and 48 kHz. One of the devices is clean though. The ADC anti-alias filter fixes rest by improving the reconstruction by removing further image frequencies. Strongest image for most current DACs is around 352.8 kHz.

Of course playback system then defines how much overlay of these come back again at different frequencies from the DAC, since it is now running at different rate family than the original source.

This doesn't really replicate real device playback performance, but at least it captures some apparent differences. So it gives kind of hint or shadow of how the device actually did."
Although the blind test is ongoing, I figure it's still good to talk about this while not revealing the devices used of course.

As a start, yes, of course poor filters can result in terrible looking spectra with all kinds of ultrasonic imaging beyond the first and second image band. Most infamously, we have seen these issues when we examined MQA in the past.

Remember that when we run objective measurements, we're often using extreme test signals to see the limits of the machine and whether some property of the device functions close to an ideal state. For example, when I run the "digital filter composite", I'm including the 19 and 20kHz -6dBFS tones to check for intermodulation distortion. There's a wideband noise with 0dBFS peaks allowing us to see if the filter's calculations overload. DAC/amplifier/speaker testing might also include the use of square waves running through them.

But these are not "normal" music signals by any means! For a device to pass with flying colors through these "torture tests", we are basically showing that the machine can achieve a certain level of idealistic performance which more than likely will not be tested with typical music. Or if it is tested, only rarely with certain types of recordings. When I evaluate devices, I certainly look for ideal performance since it tells us something about whether a device is capable of very high fidelity (which for me is what the "audiophile" pursuit is about).

Which brings us to the "real world" blind listening test. As you know, for this test, what I have done is used real music that were of reasonably good quality (notice I didn't pick nasty DR4 mastered pop music even though there are plenty around!). While the Joe Satriani track "Crowd Chant" did contain some clipped peaks in the digital signal itself, the other 3 tracks all were properly bandwidth limited. As such, the demands on the digital filter's performance are not extreme.

Let me show you what I mean using what I would consider are the 2 devices in the blind test with the poorest performing filters - let's call them Device X and Device Y so we don't make any associations with Device A/B/C/D in the actual blind test.

Let's start with simply playing a 16/44.1 1.0000kHz 0dBFS test tone (with TPDF dither) through these devices and I'll capture back the analogue output using the RME ADI-2 Pro FS at 24/384 for 192kHz of bandwidth:


That doesn't look so good at all. Device X is clipping the strong 0dBFS signal. As for Device Y, we can see imaging all the way out to almost 100kHz which a 24/96 capture indeed will not record.

Okay, let's get even more challenging... How about the -6dBFS 19 & 20kHz intermodulation test played back through these devices?


Ugly indeed especially for Device X where we have noise content going all the way to ~190kHz!

But remember, these are synthetic test signals... Do we listen to 1kHz 0dBFS tones? What music actually contains high amounts of 19 and 20kHz peaks? (Even if present, how many of us can honestly claim that we can hear such high frequencies?)

No friends, we listen to real music. And here is what real music "looks like" when reproduced with Device X and Device Y despite how awful the synthetic test tones appear in the graphs above:

Notice I added the noise floor graph in the "Crowd Chant" capture. The music signal was captured during one of the louder passages where there was some clipping showing excess signal above Nyquist. This is the only track of the 4 samples that does this.

What I've done is monitor the playback of the actual songs in realtime using WaveSpectra (ASIO ADC input at 24/384) and captured places during the playback where the ultrasonic defects looked the worst.

Quite a different "look" from the 0dBFS 1kHz and 19/20kHz synthetic signals above, right?

Knowing this, consider that when I created the test files, I captured the music at 96kHz using a sharp linear phase filter on the ADC. This means I've preserved almost all of the content up to 48kHz. For Device X, this means I have captured essentially all of that ultrasonic distortion clustered around 40-45kHz. For Device Y, the 96kHz capture would have missed out on the last bit of distortion closer to 50+kHz, but certainly kept a good amount of what's there.

Suppose you downloaded the Blind Test, and are about to listen to the sample tracks in the comfort of home through your laptop with an inexpensive hi-res USB DAC like the SMSL iDEA (previously measured) through your headphones.

Here's a comparison of what that SMSL iDEA DAC's analogue playback of the 24/96 test sample looks like compared to a direct feed from Device X/Y for the same 5-second segments (again, captured back at 24/384 through my RME ADI-2 Pro FS ADC):



Notice that I matched up the amplitude of the audible spectrum quite closely. As expected, since Device Y had a bit more ultrasonic content stretching out beyond 50kHz, the limits of a 96kHz samplerate with Nyquist at 48kHz would miss out on some of that higher ultrasonic stuff. As you can see, apart from that bit of "missing" ultrasonic data, a listener using something inexpensive like the little iDEA can still replicate the 16/44.1 playback quite well including most of the subtle, low level filter artifacts. Obviously each DAC will be a little different but I trust that most hi-res players can perform in a similar fashion to the <US$100 iDEA. I certainly do not believe that a substantial reason why players could/should/might sound different is because of some ultrasonic content out around 50kHz!

While I did not show measurements with the other 2 devices in the blind test, suffice to say that I have looked into this and I have even less concerns about "missing something" in the test samples with those other two devices.

In my opinion:

1. Sometimes I agree with the "subjectivists" :-). We don't listen to test tones and not all measurements will correlate with the sound of real music playback. While we can see marked imperfections when playing back test tones with Device X and Y, actual music playback shows much less ultrasonic artifact. Notice that both devices have relatively steep filtering that attenuate frequencies quite nicely by 25kHz, and the imaging artifacts out at 40-50kHz are at worst down at around -100dBFS.

2. There's a comment about the fact that since I'm recording at 96kHz, somehow using this different family of sample rates (48kHz base rate) is an issue. ("Of course playback system then defines how much overlay of these come back again at different frequencies from the DAC, since it is now running at different rate family than the original source".)

Remember that while the original data is 44.1kHz, this is hi-res 96kHz conversion of a continuous analogue signal... Not the same as non-integer conversion of 44.1kHz samples to 96kHz digital where perhaps some might take issue. In fact, it is because I used 96kHz that I am able to capture more of the artifacts than something like 88.2kHz. Sure, I could have captured the signal at 24/176.4 or 24/192, but as you can see, if I did that, most of the data would have been wasted bits and would have made the test files even larger than the already substantial 470MB package! I just didn't feel this was worth it for practical reasons.

If the only difference between something like a cheap $50 CD player and a $1000 device (not necessarily saying these are the devices I used, but similar :-) is because of some fragile low-level ultrasonic noise >48kHz, should we not automatically doubt the value of expensive devices?

3. There's an implication in the comment that speaks of the belief that filters are somehow causing a major difference between devices. Even to the point of suggesting the "Strongest image for most current DACs is around 352.8 kHz". Is ultrasonic content at 352.8kHz that important as to audibly differentiate expensive vs. cheap playback devices?! While I certainly hope not to see high amounts of noise at 352.8kHz or in the MHz range, I have seen no evidence that such high frequencies (at most around -50dBFS in measurements) coming out of typical DACs ultimately affect the sound from one's speakers/headphones.

I don't deny that anti-imaging filters are important. I don't like the sound of NOS with high levels of ultrasonic distortion, and I do not prefer weak Ayre/PonoPlayer filters that roll-off the highs. Furthermore, I see no point in promoting a system like MQA where the digital filters used are suboptimal. However, most of our playback devices are "somewhere in the middle" which could very well be "good enough" with real music.

Remember that CD and other digital audio players have deficits within the audible frequency that can be much more important than "good enough" filtering. For example, the noise floor will be clearly different between devices (hum especially tend to be high with cheap gear). Frequency responses vary. Crosstalk between the channels can be quite high at times. Not all devices are created equal when we compare harmonic and intermodulation distortions. Phase distortions may be present. Low level linearity errors diminish DAC resolution with cheap devices. Etc...

I would consider any of these factors above within the audible spectrum significantly more important than a few little ultrasonic imaging distortions with these Blind Test samples! A high quality, high-resolution capture of the 16/44.1 signal at 24/96 would most certainly retain these differences between devices - I can say this without hesitation because I have already measured the devices and can consistently demonstrate relative weaknesses.

In conclusion... What I have captured in these 24/96 test signals I believe is way more than just a "kind of hint or shadow of how the device actually did". While I respect Miska who wrote the comment (and as the author of HQPlayer discussed last week), I must disagree with this assessment.

Since the beginning, this blog is about being "more objective". While objectivity is obviously important especially since these days the mainstream audiophile press is strongly biased towards subjective reviews and clearly at times fanciful nonsensical tales told to audiophiles, there is inherently a balance we must seek. IMO, this balance is to be found when we correlate the objective facts with subjective experiences and adding some common sense into the equation.

Often common sense, knowledge of how things work, and a little systematic testing tells us that certain beliefs are clearly false (eg. the value of thousand-dollar power cables, or the notion of significant differences between otherwise error-free digital cables). In the other extreme, we can say that relying too much on objective findings - like making claims about sound quality based on pre-ringing of the impulse response - is folly as well. Either path alone leads us down a state of neurotic preoccupation as one can "miss the forest for the trees".

This is why I like running these blind tests once awhile. To give you, readers, an opportunity to add your voice to the discussion based on experience while attempting to reduce bias. I believe this is more powerful than the voice of certain "respected" opinions especially in the audiophile press over the years.

So, did different digital devices playing the same 16/44.1 music sound different? And by how much? Remember, you have until the end of April, so take your time and please share the test among audiophiles, music lovers, and those who just like testing things out (!) with time to compare the 24/96 samples and answer some straightforward questions.

-----------------------------------

I watched the Queen biopic Bohemian Rhapsody last night with my wife. Excellent flick and of course up for consideration this weekend at the Oscars; amazing acting by Rami Malek. The 4K UHD Blu-ray looks excellent with good HDR punch, natural colors and deep blacks. Alas, it looks like despite being filmed digitally in 3.4+K, the digital intermediate was still 2K - very common these days (remember I posted some discussions on the benefits of >2K movies awhile back). Nonetheless it looks sharp and clean. Impressive Dolby TrueHD Atmos soundtrack as well!

Hope you're all enjoying the music (and movies!) :-).

13 comments:

  1. Nice work again Archimago. It is very good seeing what realworld playback looks like and comparing with some of those artificial tests.

    As usual, you've also expressed the ideas as a gentleman and scholar!

    ReplyDelete
    Replies
    1. Thanks for the note Jonathan...

      Real-world is important!

      Delete
  2. If you look at Crowd Chant example above, overall level difference from 0 to 20 kHz is about 30 dB. Then, if you look at level difference of 20 kHz and highest correlated error signal in ultrasonic range it is about 40 dB. That translates to 1% distortion... And that error is just still within the 48 kHz bandwidth of the ADC capture. What we don't know now is how high the distortion components outside of ~200 kHz band are...

    Now whoever plays this back on their system, gets their local system's inaccuracies and distortions added, corresponding to the now 96k sampling rate. How this coincides with the same from original system is of course system dependent.

    As I said earlier, this already clearly demonstrates differences even though it doesn't capture entire picture, and what it renders is system dependent. No, I don't have anything against this test, yes it already shows lot of differences as it is, just not all of them. And it does add it's own twist through the ADC and through listeners playback system. So what it doesn't represent, is how the original converter would exactly sound like, when placed in listeners system in place of their current DAC.

    ReplyDelete
    Replies
    1. Hi Miska,
      Yes, some of the devices are "poor" and indeed can add that 1% distortion... However we are looking at ultrasonics. Not sure I'm that worried about this in the real world.

      The bottom line:

      Direct Listening: Crappy vs. good DAC 16/44 --> Amp --> Transducer

      Test: Crappy vs. good DAC 16/44 --> [RME ADI-2 Pro ADC --> Listener's hi-res DAC @ 24/96] --> Amp --> Transducer

      Do I believe that the intervening [RME ADI-2 Pro ADC --> Listener's hi-res DAC @ 24/96] step is likely to be so distorted as to nullify the sonic difference between something like a $50 CD player (or possibly worse!) and a $1000 DAC? No, unless the listener's DAC is of very poor quality. Since I ask the listeners to tell me what DAC device they're using, I can at least gauge this to some extent when I collate the data.

      Delete
  3. If you could transfer sound of a system to another system by just recording playback through DAC/ADC on that system, it would be a very cheap way to deliver high performance sound to what ever mobile phone...

    ReplyDelete
    Replies
    1. "...transfer sound of a system to another system..."

      But that's not what I am claiming!

      All I'm saying is that the sound of 16/44.1 playback off cheap vs. more expensive DACs (not full systems) can be captured directly from the analogue output and reproduced by reasonably transparent 24/96 high-resolution devices!!! And that if this is a significant difference in audibility, then I expect listeners with high-fidelity sound systems can hear the difference.

      Just to play the "Devil's advocate" for a second... I don't think there's anything wrong with believing that in fact to some extent the coloration of sound systems can indeed be recorded and transmitted. I do it all the time when I rip my vinyl records! I can certainly hear the difference between these rips and a CD-sourced FLAC on my mobile phone...

      Delete
  4. You can delete or edit my comment if it is too much a spoiler, but I had to resort to opening up the samples in audition, and then could (only) visually see the differences in the spectrum view.

    However, the blind test pushed me to explore myself in another way: is my hearing good?
    I generated numerous frequency sweeps in audition, and then tested my headphones with a microphone. Suffice to say, true to the AKG spec sheet, the cans can happily reproduce anything up to 22 KHz.

    My ears, on the other hand, are deaf above 17,5 KHz.
    When I was younger, I didn't bother to test how high can I hear, so I can not make assumptions to how my hearing changed through the years (37 y.o. now), but will measure from time to time to monitor it in the future. One has to know and consider his or her own capabilities / limitations.

    Bottom line is, that all the bling and clarity the fine equipment can produce above 18 KHz, means absolutely zero to me. If nothing else, I can save some € now when buying equipment :)

    ReplyDelete
    Replies
    1. Hi Turrican,
      Ahhhh, to be young and 37y.o. again :-). That was a good age...

      Seriously, 17.5kHz is fantastic. Make sure to use hearing protection when attending concerts, man!

      Delete
  5. I'm in the same boat as Turrican, can't hear anything above 17. (I have a lot of rock concerts in my youth combined with Sony Walkmans (tape) when traveling by subway as the culprit). Same here I downloaded the samples and I couldn't hear any difference. I still argue that the biggest difference in sound is the last mile I E speaker and headphones. Unless you have a really badly matched amp or one with really high distortion, you won't hear any difference. Same goes for the DACs we are testing. Unless something is really, really wrong you shouldn't hear anything besides the music. Even the cheap and cheerful ChromeCast Audio is in many cases "good enough" as a headphone amp. But I really appreciate all the test Archimago does here, puts things in perspective. Good sound has never been more affordable. (The mastering is so much more important than 16bit/44.1KHz vs 24bit/96KHz debate.) We have reached the limit of the human hearing with the 16bit / 44.1 CDA format. (24 bit is for mastering in digital systems)

    Stepping of soapbox.

    ReplyDelete
    Replies
    1. Thanks GRCE Admin,
      Hmmmm...

      No difference? Are you sure? :-)

      Let's see what the big picture looks like with everyone at the end of the test!

      I see some folks submitting results with everything from <$500 to $75,000 systems. ODACs to Allo DigiOne streamers to Lyngdorf to Emotiva to Oppo to Roon ROCK to JBL speakers to Rogers LS3/5A to Bluesound to NAIM to Genelec to Focal speakers. Solid state and tube amps as well.

      Nice. Keep 'em coming!

      Delete
  6. Dear Archimago, i am selling raspberry pi 3 B+ kit and want to invite you to review our item. Do you have a contact email?

    ReplyDelete
    Replies
    1. Hello Sally,
      While I typically do not review items beyond those that personally pique my interest, I'm certainly happy to talk about items that might bring features and value to the audiophile.

      Perhaps as a start, contact me through private mail at "Audiophile Style"...

      Best regards...

      Delete