Saturday 30 May 2020

BLIND TEST RESULTS Part I: "Is high Harmonic Distortion in music audible?" Procedures & Settings

As you perhaps know, over the years, there have been a number of blind tests conducted here on the blog - "Does high bitrate MP3 sound different from lossess FLAC?", "Does 24-bit sound different from 16-bit?", "Audible difference between linear vs. minimum phase filter?", "Do digital audio players sound different playing 16/44 music?" - for a sampling over the last while...

The question this time being posed is whether harmonic distortions are audible in music, and if so, perhaps through this test, we might be able to get a sense of the level of audibility. Remember that this is a complex question... It's not just "YES" or "NO" because it can depend on the AMOUNT of distortion in question. Furthermore, we can look at ODD vs. EVEN distortions. There are also questions around HIGH vs. LOW order amounts contributing to audibility.

I mentioned in the BLIND TEST Invitation post that I was in contact with Paul K [aka pkane] who wrote an intriguing piece of software - "Distort" (current version 1.0.19) - which allows us to plug in parameters and "dial in" a transfer function to model nonlinear distortions that would result in anomalies like harmonic distortion. The GUI allows anyone to easily enter how much and what levels of distortion to introduce, identify an audio file to process, and save out a modified file for playback. The DSP is able to process the audio data with high precision (32-bits).

You can imagine that if we are to systematically evaluate the variables listed above (total distortion amount, even vs. odd, low vs. high order), we would actually need not just one test, but a number of parallel experiments in order to tease out these factors. One would need a proper audio test lab for that and for an open blind test like this, complexity is not good! As a "first stab" then into exploring this question, let's just focus mainly on various amounts of harmonic distortion added without preference to even or odd order, and as we generally see with actual audio equipment, higher order harmonics tend to be lower.

First, I selected 4 music passages from albums I believe are well-produced and have a more "natural" sound to them to serve as test samples. Choosing undistorted music is of course essential! There would be no point looking at harmonic distortion if we started with excessively loud, compressed, or clipped music with little dynamic subtlety or already have high distortion "baked into" the sound. For completeness, here are the descriptions I posted initially for each track and I think it's worth repeating here:

I. Clavier: "Prelude No. 19 in A major from the Well-Tempered Clavier Book I, BWV 864" - from Yo-Yo Ma, Chris Thile, and Edgar Meyer's excellent album Bach: Trios (2017). Notice that this is a high-resolution track in 24/96. This is also the only track in this test that started as high resolution. Track length is 1.5 minutes. As the album title suggests, this is a classical string trio recorded intimately. Lots of speed, low noise floor, listen to the instrument placement in the soundstage. Did the timbre of the string instruments change among the samples?

II. Horse: "Ballad of the Runaway Horse" is found on Jennifer Warnes excellent collection of Leonard Cohen songs (well known to audiophiles), Famous Blue Raincoat (1987). It's a great example of well-recorded female vocals with simple instrumental backing. Can you hear the change in the vocals and music introduced by harmonic distortion? Is there any change to the subjective "transparency" between tracks - any apparent "fog" or "dirt" between you the listener and Jennifer's performance?

III. Tootie: "Tootie" from Hootie & The Blowfish's Fairweather Johnson (1996) is another vocal track, but this time it's Darius Rucker's baritone voice taking center stage. Again, this is a good recording with tastefully done light rock instrumentation. Any change to timber of Darius' voice or instruments with the different amounts of harmonic distortion? Any apparent "artificiality" or dysphoric elements noticed?

IV. Rhapsody: "Hungarian Rhapsody No. 6 in D flat major" by Lang Lang from his album Liszt: My Piano Hero (2011). It has been claimed by many audiophiles that piano playback is difficult to reproduce well. Here is an example of a solo piano track with various amounts of harmonics added. Can you perhaps rank the samples from what you believe is "best" sounding to "worst", perhaps correlated to added distortion?
Notice that I posted the Dynamic Range analysis of each sample to show that I am using songs with good dynamic range of at least DR11. Furthermore, I made sure that the RMS volume for each song was controlled to within 0.01dB; unless this is finely controlled, volume differences could be a very significant psychoacoustic confounder (we generally will prefer louder tracks even if very subtle; this is of course the principle behind the "Loudness Wars").

As you can see, each sample was processed to create 4 variants (labeled samples A / B / C / D) depending on how much distortion was added. Randomization was achieved by tossing a dice. Advertising for this test was seeded through usual "audiophile websites" visited recently including Audiophile Style, Steve Hoffman Forums, Audio Asylum, and Audio Science Review. I see that the invitation was also linked on Reddit.

Without further ado, here is the "answer key" showing how much distortion was added for each of the samples:

Click image to expand.
Using Distort's custom option, I entered harmonic distortion amounts up to the 31st harmonic (!) in a gradually decreasing fashion. My THD targets were -30dB (3.2%), -50dB (0.3%), -75dB (0.02%), and the "control" of -175dB (essentially nil) - remember this is all calculated with 32-bits precision in the program. The samples themselves were converted to 24-bits with no dither added. Notice that so as not to imbalance even vs. odd harmonics, I only changed the amount of distortion every 2 orders therefore both 2nd and 3rd harmonics are the same, likewise 4th and 5th are at the same level.

As for higher order harmonics, I tried to emulate what one might see with most electronics which is a reduction of higher harmonic components. You'll notice that there is a -10dB drop every 2 steps up.

Just looking at those graphs above, I trust you'll appreciate how much distortion was added as demonstrated with the 50Hz 0dBFS signal! Typically, we see measurements done with a 1kHz signal and typically the THD is calculated using the 2nd to 9th harmonic. The reason I'm showing the 50Hz signal is so you can see all the harmonics I've added up to the 31st.

Let's look at each of the settings and discuss the rationale for the choice:

-30dB (3.2%) THD "D" sample would be on the extreme end for any modern amplifier whether we're talking solid state or tube based. In fact, this amount of distortion would be more akin to a poor loudspeaker that distorts throughout the audible spectrum! Notice that this amount of nonlinear distortion can be easily visualized in the right-lower transfer function graph. I don't think it would be unreasonable to suggest that if THD is audible, this level of distortion >3% probably would be audible through a good high-fidelity sound system by audiophiles.

-50dB (0.3%) THD "A" sample - clearly less harmonic distortion added. I selected this level of distortion as representative of the absolute bottom end of passable amplifier performance IMO. Remember that while manufacturers might quote an amplifier's power rating up to 1% distortion, it's certainly not recommended to listen at levels approaching that 1% convention! For high-fidelity purposes, many audiophiles (including myself) like to see clean power with distortion <0.1% (-60dB) as a rule of thumb. For example, the Hypex NC252MP is capable of 267W into 4Ω by the time we see 0.1% THD(+N); in practice, I would never listen to music anywhere near that 0.1% threshold for my Hypex. Notice with the transfer function graph how visually linear this looks already.

-75dB (0.02%) THD "C" sample is next with even less added distortions. If this were an amplifier, we're looking at middle-of-the-road hi-fi performance for measured devices I've seen. Both 2nd and 3rd harmonics are set at -69dB with -80dB 4th and 5th harmonics with the fundamental at 0dBFS. A "high fidelity" solid state amplifier (like my Emotiva XPA-1L) should have no trouble achieving better performance than this at normal listening power.

Finally there is the -175dB THD "B" sample which is our "control". The tracks were still run through Distort to allow the software to perform very minimal 32-bit adjustments. As you can see in the graph, there's essentially no additional harmonic content except for some tiny 2nd and 3rd order bumps down at the noise floor well below -150dBFS. At this level, any distortion added to bit-perfect playback will be from one's DAC, pre/amplifier, and speakers.

Based on these settings then, an audiophile with excellent equipment (DAC, pre/amplifier, speakers/headphones), listening with low ambient noise level, and of course with good hearing might be able to identify from BEST to WORST (LEAST to MOST distortion) in order B > C > A > D. Remember that subjectively, what sounds "best" does not necessarily mean lowest distortion which is why in the survey I added a question about this to pick out the individuals who believe their choice did not relate to a strict definition of fidelity as in "lowest distortion". It would be interesting to see among this group whether there was still a correlation between "better sounding" and the distortion amount added.

One more thing, remember that intermodulation and harmonic distortions are both the result of a non-linear transfer function. Intermodulation test signals use combination tones that show sums/differences and multiples which can result in non-harmonic, potentially "more objectionable" frequencies showing up. In Distort, we can have a look at what happens with the 19 & 20kHz CCIF intermodulation signal and the respective distortion amounts:

Click image to expand.
Nasty looking, eh?! Since the CCIF test signal is a popular one done with amplifier measurements, if you compare published results with the graphs above, you'll see that the settings I use are rather high!

A big thank you to the 67 of you brave enough to take this test and send in your results! I don't know how many people downloaded the test files but I do know that >6000 people saw the blind test invitation. As I have seen before, participation with submission of full results when it comes to a more "disciplined" blind test like this isn't generally high among audiophiles compared to something like a simple poll question. No question it's much easier to subjectively "just listen" in a sighted fashion and claim all kinds of "easily heard" differences as claimed among forums and shared through social media. In my mind, this makes those of you who took the time to listen and evaluate the samples a very special group - thanks again :-).

Also, massive appreciation to Paul K for all the work put into creating Distort for audiophiles to explore the audibility of distortions - not just THD but also adding noise, effect from jitter, and bit-depth reduction. (Let's also not forget his work on DeltaWave.)

Next week, I'll take the lid off the survey results and review what the 67 respondents heard!

A side note... With all the activity around us the last few months, this is the first time in all these years of running a blind test that I did not even peek at the cumulative results throughout the 3-month data collection period. As such, at this time, as I'm about to click the "publish" button on this post, I still have no idea what the final results look like! Let's see if there are surprises in store... :-)


  1. Only 67 respondents? In the middle of a planet-wide stay-home order? What else did the other 5000 people have to do that they couldn't make time for this?

    That's even more disappointing than finding out I bombed the test. ;)

    1. Greetings Allan,
      Remember, it's not easy doing a blind test while anxious! Baking to do, spring cleaning, learn a new language, listen to music, and of course watching news networks...

      But that's OK because the data set is still interesting. Will hopefully be ready with the data this weekend!

  2. I didn't see it or I would have taken part. Regardless, as one who has owned both tube integrated amps, SS integrated amps, and high power separates, I can say that it was easy to pick out the tube vs. solid state presentations. I enjoyed my time with both which is no surprise since not all CDs or LPs sound the same and the sound of microphones and microphone preamps vary just as much as the gear. The recorded medium is all over the place quality wise. I just tried a couple of streaming services for their trial period and cancelled both. They may be fine for casual listening while doing something else, but for critical listening they are not for me. Many of the high resolution files did not sound that great. Most of my downloaded 2496 files I burn to DVDs for playback anyway. I don't mind making my own physical media.

    1. If your 2496 file sounds different from the same file burned to DVD, it means you are hearing difference in DACs. If 2496 from your computer/streaming device sounds worse - just get a better DAC for it. The digital data is the same.

    2. That's too bad you didn't get a chance to participate Jim.

      Even with tube amps, up to 3% THD is quite high distortion and for most people, it's more likely than not that the ultimate limiting factor would be the speakers one is using.

      While I have not tested lossless streaming myself (I only have Spotify lossy, and here in Canada the only option is TIDAL), remember that provenance might be different depending on the service. I believe often the streaming sites use newer remasters that are louder and more compressed, sometimes marketed as "hi-res" 24/48+ when IMO the more dynamic original 16/44 CD might actually sound better for audiophiles who care about quality rather than just being loud.

      I agree with Zepplock... Assuming otherwise bit-perfect copying/DVD burning and data retrieval, differences heard should simply be a result of different DACs.

    3. I agree as I love your tests. As for the difference I am hearing between Spotify (320kbps) and Qobuz that has high bitrate files, I am playing the same music as Qobuz is offering in the same bitrate (downloads), through the same Focusrite Scarlett 2i2, comparing a download or the ripped disc and I do hear a difference. Now we do only have AT&T copper internet in our subdivision, so if that accounts for anything I have no knowledge of internet networks, but I do know that the packets can come through anywhere and are reassembled in-order for playback. Pretty complicated and a miracle it works at all. I have had some tube integrated amps here along with integrateds with tube preamps with mosfet output stages that I really liked, but they do have a sound all their own and can be enjoyable. They are not spec machines if that is what one is looking for. At 73 I am less into convenience and more into hearing the best quality I can afford. It is one of the reasons that I do like NCH WavPad as it has an FFT display so I can readily see any frequency abnormalities that can account for sound differences. I just downloaded JRiver Media Center as they had a $20 off coupon code and I think it sounds very nice, and surprisingly, I think on some high rez material I prefer it over just playback through Sony SoundForge which I use for mastering recordings. I haven't even begun to think about why that might be, but is not just different, but clearer and deeper bass extension. Now it is not that little things matter, it is everything matters. My new computer with a new MSI Tomohawk mother board, Ryzen 3200 processor, 32 gigs of ram, and separate Gigabyte GEForce GTX 1650 video card and 1.8TB of HD space on 3 drives and one 250g SSD for just boot up. The IFi Purifier 3 is into my Focusrite Scarlett, and MIT power cord from a Furman power conditioning rack mount (I have 4 for all of my audio gear) which is the input to my 500 watt power supply. I have a -80db noise floor in my DAW before I open my mics. I doubt that I can get much better for not a lot of money. Being a former broadcast engineer when idler drive TTS with the swell Shure phono Preamp?, and either a Shure M44E or Stanton 500E into our Gates consoles and our R2R machines plus the cart machines for commercials, I have much to learn, but glad those days our over. It has been my experience that some of the best sounding FM radio stations I've ever listened to were NPR stations driven by the classical music formats and great engineering. And that quality was always a function of the FM tuner and other gear. The McIntosh and Marantz tuners of that day were the best.

    4. You might check out this Steinway streaming site as this is what quality streaming is to me.!/steinway/streamHome.jsp

    5. Great sounding Steinway piano streams! Wonder if anyone has captured the output to have a look at the quality. I assume they're not lossless streams?

    6. I don't remember my choice: please, can you tell me how can I find it?

  3. Can't wait to finally see the results! I think I was one of the first to take the test back then, and it's been a long wait.
    My prediction is that some people might be able to spot the -30 db THD samples as the most distorted, but it will not be easy (maybe barely getting past significance threshold). I think it is almost impossible to detect -50 db THD or less, so it and the -70 and -175 will get similar scores on average.

    1. I suspect you're right.
      I did the test with PBS headphones and it was quite difficult to hear a difference.
      I think most transducers will have higher distortion than the virtual amps and will mask the distortion artifacts.
      I intended to try again with (possibly) more revealing phones but unfortunately didn't get around to it. I probably would have been prejudiced by my first listen anyway.

    2. All will be revealed soon, friends :-).

      Still a bit of analysis, graphs, and write-up to be done... Certainly interesting looking at some of the subgroups like "younger"/"older", "headphone"/"speaker"/"both" listeners, etc. Even if not exactly statistically significant with p<0.05, I can say that there are patterns here that "trend" in ways that might or might not be unexpected :-).

  4. Good point about the newer remasters. At one time i assumed remastered versions would always sound better than the originals. Some do, but when i listened to some steve hoffman remasters i found the sound disappointing. I then set up my A/B switch and compared different versions of the same CD. An eye opener. I also compared some albums in their CD LP and Tape versions. My conclusion was a good recording will sound good...Spend more time finding good recordings and less time fretting over DSD/MQA/hires, interconnects, speaker wire, and power cords.

    1. Yeah, definitely do not make the assumption that remasters better.

      The other night I was listening to some AC/DC in Roon and realized it just sounded terribly compressed and un-dynamic... Had a look and indeed Roon had selected a newer remastered version I had on the drive as default (bought a Japanese remaster years ago of The Razors Edge when there a few years back thinking might be good - a mistake).

  5. I am surely interrestd to see the analysis if people could identify correctly.
    As well if distortion is recognized as better :)
    I analyzed my rating and it coinceeded with what I experienced earlier. I will not comment now, will wait for the results of archimago.

  6. Indeed a very interesting test and the subsequent reading too.
    I don't use computers for audio, so I gave up on participating.