MUSINGS / MEASUREMENTS: About THD(+N) and standardizing testing here... (With a taste of the RME ADI-2 Pro FS as DAC...)

Last week, when I published my look into the RME ADI-2 Pro FS ADC performance, I started using SpectraPLUS for measurements in high resolution for THD and THD+N. I know, it's an old measurement and everyone does it. I realized shortly after publication of the article that while I was using the same technique to measure the RME vs. Focusrite ADCs, I had the ADCs running at 192kHz and so issued an addendum that the results included the rising ultrasonic noise from the ADC and not just a reflection of the DAC. Not unreasonable as a comparison between the two ADCs I think, but this would not be fair to the DAC or other component measurements since much of the noise would be arising from the ADC stretching out to a 96kHz bandwidth. It's worth taking some time to think, going forward, how I could improve the usefulness of this test and in a standardized way here when focusing on whatever device is being tested...

If we look around, we see that the THD+N spec is probably the most used objective "number" for audio equipment as a quick snapshot of fidelity. The THD (page on calculation, how it's done) component tells us whether harmonics are being added to the sine wave at the integer multiples of the fundamental frequency (these are the results of "nonlinearities" in the equipment), and the +N piece adds the noise component found in the signal being tested which of course is also subject to noise limitations of the measurement device and the computational limits of the FFT technique used. In essence, the THD+N ratio is a representation of everything that's being added to a simple single-tone test which of course has its limitations as a test paradigm as well when real music is far from static.

Like most measurements, one does need to know the context in which the result is produced. This is especially important when this single specification is being used for the purpose of making comparisons between devices. To state that I found "DAC X has a THD+N of -110dB" sounds great, but just stating this value itself is not enough obviously unless I explicitly tell you how that measurement was derived and with what general procedure.

A good reference I have seen for how measurements should be expressed is by Rane Audio here. For the THD+N value, it should at least be expressed like: "-110dB THD+N, +4dBu input level, 20-20kHz, with a 22kHz bandwidth".

There are recommended standards out there as well. The AES17 is one that originated back in 1991 (you can have a look here for the 2004 printing, here's a page on implementation). It specifies a 997Hz sine wave to be used. There is however no clear standard of all the variables that then go into the measurement itself. When using FFT techniques, how many FFT points? What FFT windowing function to use? What bandwidth?

Furthermore, when we're thinking about DACs, is the best test level actually a 0dBFS signal or should we be concentrating on lower amplitudes? After all, unless we're always listening to primarily modern pop and rock with crushed dynamic ranges and maximized amplitudes, even if a device throws up beautiful values at 0dBFS, most of the musical content "lives" lower down... Therefore, for the purpose of "real-world" testing, isn't it much more interesting to make sure the THD+N remains low at output levels significantly less than 0dBFS? (Note that the RightMark test suite I also use does include a THD measure with a -3dBFS sine wave.)

Considering those factors, here's a suggested "standardized" scheme I'll probably use going forward as 2 sets of measurements/graphs.

Set 1. The stereo 0dBFS THD+N: Yes, I think it is good to keep a 0dBFS measure. For better or worse, this is what others commonly use and is shown in specifications. Looking around, almost all the published measurements focus on a limited bandwidth close to the 20Hz-20kHz audible spectrum - fair enough.

So let's do this. With SpectraPLUS, let me start off with using these parameters for a 0dBFS measurement with the ADC set to the "Sharp" linear phase filter, running at 48kHz; therefore bandwidth up to 24kHz, slightly higher than the typical 20 or 22kHz but this should not change the result by much:

The test signal will be a 0dBFS 24/48 997Hz (as per AES17) sine wave created in Adobe Audition with spectrally "pure" characteristics. The measurement above will perform the test with an FFT size of around 250k (262144 to be precise), using Hann (Hanning) window function (windowing choice has its own complexities to consider). And I'll set averaging to 10 to get a more stable reading. Using these parameters, this is what the Oppo UDP-205 XLR output at +13dBu sent to the RME ADI-2 Pro FS looks like with the THD(+N) and SINAD results shown for both stereo channel:

Notice that I measured the output at around -3dBFS on the ADC so that there is some overhead but it is the full 0dBFS level from the Oppo XLR being fed in. As you can see, the measured THD is excellent at ~0.000120% or below -118dB and the THD+N result is very good at less than 0.00013% which is close to -118dB (reflected in the SINAD value). Oppo's specifications indicate <0.00018% THD+N, so what I'm seeing here is certainly to be expected.

Set 2. The "THD+N Matrix": Why don't we also try to capture a range of THD(+N) results at different amplitudes to see how the harmonics change and whether THD(+N) fluctuates in any unusual way below 0dBFS? Furthermore, we're in the era of high-resolution music - why not run the test then at 96kHz? This way we will also capture distortions and noise up to a 48kHz bandwidth, ensuring a measurement well beyond human audibility.

I decided to pick 4 amplitude values - -5dBFS, -15dBFS, -30dBFS, and -50dBFS. Unless automated, it would be onerous to run too many measurements so the 4 here captures a reasonable range for musical signals I think.

The SpectraPLUS settings would look like this for the 96kHz measurements:

Notice that I've doubled the FFT size to ~520k which corresponds to the doubling of sample rate so there will be no change to the computed noise floor compared to the 0dBFS reading above. Let's just focus on a single channel for each graph for space, and again we set averaging to 10 for more stable readings.

Here's what this would look like in a 2x2 matrix, again, with the Oppo UDP-205 XLR output as "guinea pig" fed into the RME ADI-2 Pro FS - notice that since I'm starting at -5dBFS, I'll set the RME ADC to reflect the actual levels (as usual, click on image to enlarge):

The "THD+N Matrix"

Interesting to see that the system was able to compute a remarkably low THD+N value of -122dB at -5dBFS off the Oppo. In fact, notice that this is even lower than the 0dBFS measurement above (-118dB). Of course, I'm not suggesting that anyone would be able to hear a difference between THD+N of -118dB vs. -122dB; they're both really low numbers :-). But it does suggest that at least with the Oppo and its ES9038Pro DAC, the signal remains extremely clean across the top levels. If we peek at those graphs, we see that lowering the volume to -5dBFS does reduced the 3rd, 4th, and 5th harmonics plus reduced sideband levels seen around the fundamental (we'll talk more about this later).

As expected, by the time we measure -30dBFS most harmonic peaks should be very low and by the -50dBFS signal almost all should be fully buried within the noise floor of the ADC; to see otherwise would speak poorly of a high fidelity DAC. One interesting thing we see with the Oppo is that with the -30dBFS measurement, notice how the 3rd harmonic increased unexpectedly for some reason though still low.

Here's a comparison showing the 0dBFS FFT at 24kHz bandwidth and the -5dBFS FFT measured across 48kHz with the latter constrained to just the 20Hz-24kHz portion of the spectrum so we can compare the harmonic levels and see why the computed result is lower for the -5dBFS signal:

Nice reduction of the harmonic levels in the -5dBFS signal. I know... Totally geeky pixel peeping to show off the microscopic but impressive THD(+N) that the Oppo DAC + RME ADC system can produce. :-)

Notice also in the "THD+N Matrix" image above, as the fundamental changes amplitude, how we see the appearance of "clusters" of various harmonics from the Oppo DAC (for example, at -15dB, there's a little group around 10kHz-25kHz) which is different from the -5dBFS clustering that tends toward higher up in frequency (>20kHz). I doubt this is audible, but in lesser DAC devices with higher distortion levels, perhaps there is an audible effect that a detailed test like this could inform us of.

Remember last time I also compared the result from the Focusrite Forte and showed that it did not fare as well compared to the RME? Since 0dBFS direct from the Oppo into the Forte causes overloading, here's the -5dBFS signal with a 24kHz bandwidth and then with the 48kHz bandwidth:

At 48kHz bandwidth (ADC sampling at 24/96), we see the obvious noise out around 37kHz which I've documented over the years originating from the Focusrite Forte. It is now being calculated into the measurement with the 48kHz bandwidth. As you can see, this along with other ultrasonic content significantly impact the THD+N score dropping from -110dB to around -100dB without as much of a change to THD itself.

For completeness, here's a look at the RME ADI-2 Pro FS going from 24kHz to 48kHz bandwidth (zoomed into just 20Hz to 24kHz for both for a more comparative look, we already know from the "THD+N Matrix" that the RME noise floor remains low all the way out to 48kHz):

Notice that there is a small difference between the 48kHz and 96kHz samplerates with a few noise/harmonics above 10kHz added when the RME ADC is running at 96kHz. It's tiny, each -140dB or less on the graph, and as you can see, the doubling of sample rate does increase the THD(+N) value slightly.

Summary...
Okay, with that little exploration and experimentation this week, using the better quality RME ADI-2 Pro FS ADC, I'm going to start "standardizing" my procedure for THD(+N). It will include both the stereo measurement of 0dBFS output level using 24kHz bandwidth so the value can be comparable to published results and manufacturer specs elsewhere using typical similar methodology, as well as the 2x2 "THD+N Matrix" captured at 24/96 to show harmonics and noise out to 48kHz for -5 / -15 / -30 / -50dBFS signals. With all the tests, the RME ADC will be set to its "Sharp" linear phase lowpass filter.

FFT parameters are ~260k points in 48kHz, and ~520k in 96kHz so that the calculated noise floor remains the same for both sample rates, and Hann(ing) windowing will be used. I have 16/44 variant of the test signals that I can use in the event that I thought there's something wrong with the way the device handles the 44.1kHz clock base (I expect this to be rather unusual).

Apologies for being a bit pedantic, geeky and OCD in this post :-). But I figure it's worth spending time documenting the exact settings and procedures... As I've said before, my hope as a "more objective" hobbyist is to have others be able to measure and produce comparable results themselves. Sometimes, even if we can't hear a difference, it is nice to be able to understand and know about the quality of our devices. For example, even if we cannot directly hear high noise levels in the ultrasonic spectrum from a component, I think it would be nice to know that it exists since the signal needs to be fed into one's amps and speakers which could have negative consequences.

---------------------------------

I did promise last week that I would show the RME ADI-2 Pro FS's DAC performance. While I obviously am not able to show much this time, let me offer you this consolation teaser from the THD(+N) measurements with the RME as DAC, compared to the Oppo UDP-205:

Isn't that interesting!? The above is the 0dBFS 997Hz pure sine comparing the RME itself (AKM AK4490 DAC chip, output level at +19dBu) with the Oppo (ESS ES9038Pro DAC chip, output level at +13dBu), both measured at -3dBFS on the RME ADC for a bit of overhead, with 24kHz bandwidth and showing both stereo channels to ensure good levels in each and concordant results.

Notice that the RME ADI-2 Pro FS DAC's THD(+N) measured slightly higher than the Oppo due to the harmonics (stronger even order harmonics like 2nd, 4th, 6th). But there is more relative noise in the Oppo's output hence the increase from THD to THD+N is larger than the RME. I see no reason to doubt ESS's claim that the ES9038Pro is currently the "king" of low THD+N DAC chips.

But of course it's never as simple as a single number even though that number does have meaning. Notice those low-level "sidebands" seen with the Oppo and its ESS DAC around the fundamental 997Hz signal; some of which appear stronger than the harmonic distortions. In comparison, the RME's AK4490-based DAC performance has a more consistent pattern with stepwise decline in levels with each higher harmonic even though overall THD is higher. Of course, the Oppo sidebands being close to the fundamental would mask audibility (hard to imagine humans hearing those extremely low levels down at -130dB!). These sidebands have been commented on elsewhere also and likely represent modulation of the DAC's voltage level. I do wonder whether this pattern is a typical finding of ES9038Pro DACs in general or is this just with the Oppo UDP-205? Unfortunately I did not have this level of measurement resolution when I examined the Oppo Sonica DAC (also based on ES9038Pro) back in mid-2017 to see if these were present.

Ultimately, remember that compared to the amount of distortion levels of amplifiers and speakers, or room effects / interactions during playback, the Oppo sidebands are like pimples on an elephant's back. That in fact is the main message audiophiles need to keep in mind! The level of DAC performance these days with well-engineered digital devices have I believe handily surpassed auditory perception such that the vast majority of subjective opinions and claims are a function of the placebo effect and human imagination.

Furthermore, I think it is impressive the amount of resolution even the home hobbyist can have at his/her disposal these days using a device like the RME ADI-2 Pro FS when it comes to seeking and understanding the performance of high fidelity devices. By all means, do try this at home!

---------------------------------

To end off, in the "let's-talk-about-MQA-for-no-other-reason-than-we're-supposed-to-support-it" news bin, I see that Rafe Arnott in AudioStream appears to believe that:

"High-res downloads seemed to be the next big thing, but now streaming has become the undisputed heavyweight of the music industry’s music-delivery method, with MQA seemingly set to rise brightest on the format horizon."

As a subjective review, I think the article is fine, but why embed this IMO baseless statement while posing a question to McIntosh's CEO? MQA and Bob Stuart have been silent for the last year in the presence of significant criticism and evidence pointing to the limitations of the algorithm (instead, preferring to speak through their audiophile magazine mouthpieces). It has been almost 4 years since announcement with realistically only a trickle of encoded music primarily supplied by one streaming service. More recent high-resolution streaming services like Primephonic for classical and Qobuz are IMO wisely using an open, almost universally supported compression format (FLAC), not the lossy, essentially 16-bit, proprietary MQA codec. How is MQA the "brightest on the format horizon"!?

As per the article, apparently there is no comment about MQA from the CEO in his response. Perhaps Mr. Arnott missed the notice a few months back (June 2018) that McIntosh engineers "prefer to take a wait-and-see position, finding the format too lossy, with distortion that doesn’t meet the company’s high standards". Well, good to see that even The Absolute Sound was able to pass this along to the readership. Bizarre how the audiophile press seems to be the only place these days even bothering to mention MQA and out-of-the-blue bringing it up.

Have a great week ahead everyone! I've been listening to Paul McCartney's new one - Egypt Station - in the last week while working on this post. Not bad if you're into "Macca" and enjoy a more traditional sounding pop although I cringe a bit listening to some of the young-adult-oriented lyrics which would have been more appropriate from a much younger Beatle rather than from a man of my father's generation. I'm not sure how memorable this album will be for me. As usual, I will need to spend a little more time in replay.

Hope you're all enjoying the music...

Addendum - September 19, 2018
Hmmm, just a quick note. Looks like there's some issues with the SpectraPLUS measurement for the "THD+N Matrix" especially notable with the lower amplitude signals and abnormally low THD+N results. Will look into this more...