BLIND TEST RESULTS Part I: "Is high Harmonic Distortion in music audible?" Procedures & Settings

As you perhaps know, over the years, there have been a number of blind tests conducted here on the blog - "Does high bitrate MP3 sound different from lossess FLAC?", "Does 24-bit sound different from 16-bit?", "Audible difference between linear vs. minimum phase filter?", "Do digital audio players sound different playing 16/44 music?" - for a sampling over the last while...

The question this time being posed is whether harmonic distortions are audible in music, and if so, perhaps through this test, we might be able to get a sense of the level of audibility. Remember that this is a complex question... It's not just "YES" or "NO" because it can depend on the AMOUNT of distortion in question. Furthermore, we can look at ODD vs. EVEN distortions. There are also questions around HIGH vs. LOW order amounts contributing to audibility.

I mentioned in the BLIND TEST Invitation post that I was in contact with Paul K [aka pkane] who wrote an intriguing piece of software - "Distort" (current version 1.0.19) - which allows us to plug in parameters and "dial in" a transfer function to model nonlinear distortions that would result in anomalies like harmonic distortion. The GUI allows anyone to easily enter how much and what levels of distortion to introduce, identify an audio file to process, and save out a modified file for playback. The DSP is able to process the audio data with high precision (32-bits).

You can imagine that if we are to systematically evaluate the variables listed above (total distortion amount, even vs. odd, low vs. high order), we would actually need not just one test, but a number of parallel experiments in order to tease out these factors. One would need a proper audio test lab for that and for an open blind test like this, complexity is not good! As a "first stab" then into exploring this question, let's just focus mainly on various amounts of harmonic distortion added without preference to even or odd order, and as we generally see with actual audio equipment, higher order harmonics tend to be lower.

First, I selected 4 music passages from albums I believe are well-produced and have a more "natural" sound to them to serve as test samples. Choosing undistorted music is of course essential! There would be no point looking at harmonic distortion if we started with excessively loud, compressed, or clipped music with little dynamic subtlety or already have high distortion "baked into" the sound. For completeness, here are the descriptions I posted initially for each track and I think it's worth repeating here:

I. Clavier: "Prelude No. 19 in A major from the Well-Tempered Clavier Book I, BWV 864" - from Yo-Yo Ma, Chris Thile, and Edgar Meyer's excellent album Bach: Trios (2017). Notice that this is a high-resolution track in 24/96. This is also the only track in this test that started as high resolution. Track length is 1.5 minutes. As the album title suggests, this is a classical string trio recorded intimately. Lots of speed, low noise floor, listen to the instrument placement in the soundstage. Did the timbre of the string instruments change among the samples?

II. Horse: "Ballad of the Runaway Horse" is found on Jennifer Warnes excellent collection of Leonard Cohen songs (well known to audiophiles), Famous Blue Raincoat (1987). It's a great example of well-recorded female vocals with simple instrumental backing. Can you hear the change in the vocals and music introduced by harmonic distortion? Is there any change to the subjective "transparency" between tracks - any apparent "fog" or "dirt" between you the listener and Jennifer's performance?

III. Tootie: "Tootie" from Hootie & The Blowfish's Fairweather Johnson (1996) is another vocal track, but this time it's Darius Rucker's baritone voice taking center stage. Again, this is a good recording with tastefully done light rock instrumentation. Any change to timber of Darius' voice or instruments with the different amounts of harmonic distortion? Any apparent "artificiality" or dysphoric elements noticed?

IV. Rhapsody: "Hungarian Rhapsody No. 6 in D flat major" by Lang Lang from his album Liszt: My Piano Hero (2011). It has been claimed by many audiophiles that piano playback is difficult to reproduce well. Here is an example of a solo piano track with various amounts of harmonics added. Can you perhaps rank the samples from what you believe is "best" sounding to "worst", perhaps correlated to added distortion?

Notice that I posted the Dynamic Range analysis of each sample to show that I am using songs with good dynamic range of at least DR11. Furthermore, I made sure that the RMS volume for each song was controlled to within 0.01dB; unless this is finely controlled, volume differences could be a very significant psychoacoustic confounder (we generally will prefer louder tracks even if very subtle; this is of course the principle behind the "Loudness Wars").

As you can see, each sample was processed to create 4 variants (labeled samples A / B / C / D) depending on how much distortion was added. Randomization was achieved by tossing a dice. Advertising for this test was seeded through usual "audiophile websites" visited recently including Audiophile Style, Steve Hoffman Forums, Audio Asylum, and Audio Science Review. I see that the invitation was also linked on Reddit.

Without further ado, here is the "answer key" showing how much distortion was added for each of the samples:

Click image to expand.

Using Distort's custom option, I entered harmonic distortion amounts up to the 31st harmonic (!) in a gradually decreasing fashion. My THD targets were -30dB (3.2%), -50dB (0.3%), -75dB (0.02%), and the "control" of -175dB (essentially nil) - remember this is all calculated with 32-bits precision in the program. The samples themselves were converted to 24-bits with no dither added. Notice that so as not to imbalance even vs. odd harmonics, I only changed the amount of distortion every 2 orders therefore both 2nd and 3rd harmonics are the same, likewise 4th and 5th are at the same level.

As for higher order harmonics, I tried to emulate what one might see with most electronics which is a reduction of higher harmonic components. You'll notice that there is a -10dB drop every 2 steps up.

Just looking at those graphs above, I trust you'll appreciate how much distortion was added as demonstrated with the 50Hz 0dBFS signal! Typically, we see measurements done with a 1kHz signal and typically the THD is calculated using the 2nd to 9th harmonic. The reason I'm showing the 50Hz signal is so you can see all the harmonics I've added up to the 31st.

Let's look at each of the settings and discuss the rationale for the choice:

-30dB (3.2%) THD "D" sample would be on the extreme end for any modern amplifier whether we're talking solid state or tube based. In fact, this amount of distortion would be more akin to a poor loudspeaker that distorts throughout the audible spectrum! Notice that this amount of nonlinear distortion can be easily visualized in the right-lower transfer function graph. I don't think it would be unreasonable to suggest that if THD is audible, this level of distortion >3% probably would be audible through a good high-fidelity sound system by audiophiles.

-50dB (0.3%) THD "A" sample - clearly less harmonic distortion added. I selected this level of distortion as representative of the absolute bottom end of passable amplifier performance IMO. Remember that while manufacturers might quote an amplifier's power rating up to 1% distortion, it's certainly not recommended to listen at levels approaching that 1% convention! For high-fidelity purposes, many audiophiles (including myself) like to see clean power with distortion <0.1% (-60dB) as a rule of thumb. For example, the Hypex NC252MP is capable of 267W into 4Ω by the time we see 0.1% THD(+N); in practice, I would never listen to music anywhere near that 0.1% threshold for my Hypex. Notice with the transfer function graph how visually linear this looks already.

-75dB (0.02%) THD "C" sample is next with even less added distortions. If this were an amplifier, we're looking at middle-of-the-road hi-fi performance for measured devices I've seen. Both 2nd and 3rd harmonics are set at -69dB with -80dB 4th and 5th harmonics with the fundamental at 0dBFS. A "high fidelity" solid state amplifier (like my Emotiva XPA-1L) should have no trouble achieving better performance than this at normal listening power.

Finally there is the -175dB THD "B" sample which is our "control". The tracks were still run through Distort to allow the software to perform very minimal 32-bit adjustments. As you can see in the graph, there's essentially no additional harmonic content except for some tiny 2nd and 3rd order bumps down at the noise floor well below -150dBFS. At this level, any distortion added to bit-perfect playback will be from one's DAC, pre/amplifier, and speakers.

Based on these settings then, an audiophile with excellent equipment (DAC, pre/amplifier, speakers/headphones), listening with low ambient noise level, and of course with good hearing might be able to identify from BEST to WORST (LEAST to MOST distortion) in order B > C > A > D. Remember that subjectively, what sounds "best" does not necessarily mean lowest distortion which is why in the survey I added a question about this to pick out the individuals who believe their choice did not relate to a strict definition of fidelity as in "lowest distortion". It would be interesting to see among this group whether there was still a correlation between "better sounding" and the distortion amount added.

One more thing, remember that intermodulation and harmonic distortions are both the result of a non-linear transfer function. Intermodulation test signals use combination tones that show sums/differences and multiples which can result in non-harmonic, potentially "more objectionable" frequencies showing up. In Distort, we can have a look at what happens with the 19 & 20kHz CCIF intermodulation signal and the respective distortion amounts:

Click image to expand.

Nasty looking, eh?! Since the CCIF test signal is a popular one done with amplifier measurements, if you compare published results with the graphs above, you'll see that the settings I use are rather high!

A big thank you to the 67 of you brave enough to take this test and send in your results! I don't know how many people downloaded the test files but I do know that >6000 people saw the blind test invitation. As I have seen before, participation with submission of full results when it comes to a more "disciplined" blind test like this isn't generally high among audiophiles compared to something like a simple poll question. No question it's much easier to subjectively "just listen" in a sighted fashion and claim all kinds of "easily heard" differences as claimed among forums and shared through social media. In my mind, this makes those of you who took the time to listen and evaluate the samples a very special group - thanks again :-).

Also, massive appreciation to Paul K for all the work put into creating Distort for audiophiles to explore the audibility of distortions - not just THD but also adding noise, effect from jitter, and bit-depth reduction. (Let's also not forget his work on DeltaWave.)

Next week, I'll take the lid off the survey results and review what the 67 respondents heard!

A side note... With all the activity around us the last few months, this is the first time in all these years of running a blind test that I did not even peek at the cumulative results throughout the 3-month data collection period. As such, at this time, as I'm about to click the "publish" button on this post, I still have no idea what the final results look like! Let's see if there are surprises in store... :-)