RESULTS: Internet Blind Test of 24-Bit vs. Dithered 16-Bit Part Deux - Daft Punk Edition

Well ladies and gentlemen, the time has come to reveal the results of the 2023 24-bit vs. 16-bit Internet Blind Test as laid out in the post: "Internet Blind Test: 24-Bit vs. Dithered 16-Bit Part Deux - Daft Punk Edition!".

This test was launched on March 4th and remained open to May 5, 2023; plenty of time I trust for folks who wanted to perform the test to listen and give me their results. As indicated in that invitation post, this test was created as a response to discussions on a message forum where it was said that 24-bit audio could be audibly differentiated from 16-bit files. It was offered in the discussions that the Daft Punk track "Giorgio by Moroder" from Random Access Memories, an album well-known to audiophiles, is an example of a well-produced, modern recording that benefits from 24-bits "high resolution". For years, it has been available for download at places like HDtracks, and Qobuz.

In this post, let's discuss how the test samples were created, reveal which sample is the high resolution version, and results from the respondents. Grab a drink, let's have a peek... ;-)

0. Let's talk about how I created the two samples for testing.

I started with the 24-bit official hi-res download of Random Access Memories off HDtracks at 24/88.2, and selected a portion about 2:30 in length which should be long enough for adequate A/B testing consisting of a mixture of human voice, background incidental sounds, and music. It was very convenient that this track had such a diversity!

I then used iZotope RX 10 Advanced, well respected and mature audio editing/restoration software, to downsample 88.2 into 44.1kHz and 16-bit dithering with settings I use regularly.

Here's a look at the iZotope "Module Chain" along with specific settings applied in this test:

iZotope module chain consisted of the "Resample" and "Dither" functions.

As you can see, I applied steep, linear phase resampling (200 steepness, 1.00 pre-ringing which is linear phase), and the dithering applied was a "Normal" amount with a gentle "Lightest" level of MBIT+ noise shaping which provides subtle improvement to the noise floor at the expense of slightly higher noise level <1kHz and >15kHz; a very reasonable compromise given human auditory sensitivity. Noise shaping is commonly used in commercial CD releases like Sony's Super Bit Mapping which I believe has stronger amount of noise shaping than this.

You might be wondering why I downsampled to 44.1kHz. First, this is to reduce file size for download by 50%. Furthermore, I was able to hide the fact that a small amount of noise shaping was added to the dithering process. Random Access Memories is basically a 48kHz recording as it has little musical content >24kHz and the noise shaped dithering would have shown up quite obviously in the 16-bit version if I didn't downsample.

A peek at what the original 24/88.2 FFT looks like during quieter portions of the music.

This is a bit-depth resolution blind test; therefore, as long as we keep the samplerate constant, whatever difference heard would be a result of the change from 24-bits to dithered 16-bits.

One more thing. In order to further keep the test "blinded", I converted the 16-bit dithered file back to 24-bits so both files would be feeding the DAC with 24-bit data and the 16-bit version would be padded with "0"s in the lowest bits. Plus, I added a very small amount of white noise in the 24th lowest bit in the 16-bit version. The reason I did this was to prevent someone from easily cheating by loading up the file into an audio editor like Adobe Audition and looking at the amplitude statistics:

In an open, Internet-distributed, blind test like this, it's important to close off potentially easy ways to "cheat" without affecting audibility. As you know, state-of-the-art DACs these days can achieve dynamic range up to 21-bits so the 24th-bit noise would be well below the noise floor for any consumer/audiophile device.

One other good "side effect" of adding that very low level 24th bit noise to the 16-bit file is that the FLAC algorithm has a harder time with lossless compression, resulting in larger file size. It would be very easy to just look at the file size of the 2 samples and say the 16-bit one takes up less space!

As far as I can tell, I don't think there was "cheating" going on that would grossly impact the results although I know some folks analyzed the files with various tools over the last couple months.

Whenever we do blind/controlled testing, it's essential that the amplitude be matched. No problem with this test:

Everything is an exact match with 0dBFS peaks and average -11dB RMS amplitude. This also allows the user to try something like the ABX Comparator Foobar plug-in for testing. While the average DR is measured at 8, as I mentioned in the test invitation, the first part of the track where Moroder is speaking and you hear some incidental noises in the background (restaurant noise and chatter) actually averages DR11.

Here's the unblind: Track A = 24-bits, Track B = 16-bits.

To keep this as "blind" as possible, I did not monitor the results during the course of the survey (heck, I was busy going away for a few weeks anyways ;-) just in case I might say anything in the forums to sway respondents. I trusted that there were no major issues...

So, with that preamble, let's get to the listening test results!

I. Who responded?

In total, I received 121 responses for this blind listening test. This is pretty normal based on previous experience. While I can obtain >1000 responses for a detailed survey like last week, once it gets more complicated, the numbers drop off given the "work" that's needed to load up the files and take the time to listen. I appreciate the work that everyone put into this; thanks to those who also sent me their ABX logs and detailed descriptions of multi-sitting testing regimens!

As you can see in the map above, most respondents were from North America or Europe. Top countries were USA (26), Canada (13), UK (11), Germany (7), Netherlands (6), France (5), Switzerland (4), Russia (4), Italy (4), and Czech Republic (4) in the top 10.

I suggested that respondents could tell me about their systems and they were very open with sharing this information. Let's have a look at the kinds of hardware listeners used:

Headphones: Earsonics Corsa, Sennheiser HD650, Etymotic ER4XR, Dunu Vulkan, AKG K371, Beyerdynamic T1, Dan Clark AEON Noire, Westone W80 V2, TRUTHEAR x Crinacle Zero, HiFiMan Arya Stealth, Dan Clark Ether CX, Sennheiser HD600, Sennheiser HD580, AKG K702, AKG K240, Sennheiser HD800S, HiFiMan HE400i, Sony MDR-7506, Sennheiser IE 600, Meze Liric, Sennheiser HD 6XX, Philips SHP9500, Denon AH-D7000

Speakers: Canton Reference 3.2.DC, ScanSpeak Revelator custom build, Dynaudio LYD 48 + Dynaudio 18 sub, Magnapan MG 1.7, Q Acoustics 3050, KRK RP-5 G2, Dynavoice Definition DF-5, Martin Logan Aerius, KEF Reference 207/2, Kii THREE, B&W DM603 S2, Linkwitz LX521.4, Elipson Facet 6BT, Wharfedale Diamond 12.4, ELAC FS 409, Genelec 8010A, ELAC BS41, ProAc DT8, KEF LS50 Meta

Amps (headphone and speakers): Hidizs S9 Pro, Burson DAC/headphone amp, Audiolab 6000A, Electrocompaniet ECI 5, Denon PMA-707, Meier Audio Corda Classic, NAD C316BEE, BVAudio PA300SSE, Topping L30 II, Anthem STR, Levinson No.23.5, Bryston 4B, Hypex NC252MP, iBasso DX200, JDS Labs Element III, Luxman L-505, Sound Blaster X7 LE, Cambridge CXA 80, SMSL SP400, NuPrime IDA-8, Hypex NC400, Purifi Eigentakt, Topping LA90, Schiit Jotunheim, Schiit Heresy, NAD C298, Graham Slee Solo Ultra-Linear DE

Streamers/DACs: Bluesound Node2, Auralic Vega DAC, Innuos Zenith Mk2, Topping E50, Cambridge CXN v2, Mojo 2 + Poly, FiiO X3 GenII DAP, Audiolab M-DAC, Topping D10s, Topping D70, Topping D30, SMSL DO100, Bluesound Node2i, Linn Numerik, Schiit Bifrost 2/64, Cambridge CXN v1, Soncoz SGD-1, MOTU M2, miniDSP 4x10 HD, Logitech Transporter, SOtM sMS-200, SMSL M300 Mk2, Rockna Wavelight, Fosi Audio Q5, Dragonfly Red, Mola Mola Tambaqui, SMSL Sanskrit 10th MkII, RME ADI-2 DAC FS, E1DA 9038D6K, Oppo BD-105D

I think that captures most of the reported gear without repeating the same items. Topping and RME DACs, Dan Clark headphones, and Sennheiser HD*** products were quite popular. There were some DIY speakers and amplifiers as well that did not have enough of a description for me to list. I think it's fair to say that the respondents are hi-fi enthusiasts running this kind of hardware!

I didn't see anyone testing with Apple EarPods/AirPods or Beats headphones this time around. ;-)

II. Overall, which of the two tracks did listeners prefer? And how many had no preference?

Wow. Look at that. Almost evenly divided into 1/3's! 38 preferred Sample A (24-bits), 40 preferred Sample B (16-bits), and 43 had no preference of one being "better" than the other.

Obviously with numbers like that, we're not going to achieve statistical significance between those who chose Sample A and Sample B. Already we can surmise that even if there is an audible effect between 16-bits and 24-bits, a large number - 36% - felt whatever difference heard, if any, did not warrant a preference.

III. How much difference did the listeners find?

Let's see if we can go deeper by asking listeners to tell us how much difference they thought they heard:

We already know that about 1/3 reported no audible preference above so it's no surprise that on a 0-10 scale where 0 is "no difference" and 10 is "obvious difference", it would be skewed towards 0. Notice that the number who chose "0" was 27%. Since 36% indicated that they had no preference, this means that 9% heard differences but were not significant enough to pick one they thought sounded "better". This is not unexpected. IMO, often even between something like amps, one might have the impression of audible differences, but it could be hard to choose one as a clear "winner".

Of the 64% of listeners who chose either Sample A or Sample B, let's look at the distribution of how much difference they subjectively heard:

Good to see that there's internal consistency as none of the ones who made a choice between A or B chose "0". Notice the skew towards the left though as 66% scored their perception of audible differences at <5/10; and the weighted average below 4/10.

Some reported hearing more of a difference hence the "bump" at 7-8/10, and another group at 10/10. Let's have a look at those who were more confident and chose 6+/10. Given the subjective impression of hearing more difference, which of the samples did these 22 respondents think sounded "better"?

That's interesting! Those who thought they heard more of a difference (6+/10), actually preferred the 16-bit dithered version. I found this surprising.

Statistically, assuming a 50:50 probability for either Sample A or B being chosen if random, the chance of achieving at least 14/22 preferring the 16-bit sample is 14% (p=0.14). Not "statistically significant" (p < 0.05 is typical threshold), but that's certainly an interesting trend which could have been significant if there were more samples and this proportion held.

IV. Between headphone and speaker listeners, any difference?

For this test, I see that slightly more listeners (53%) used wired headphones:

This is reasonable given that I think most audiophiles know by now that 24-bit vs. 16-bit would not be an easy test and headphones can be more intimate and resolving for these kinds of assessments.

One user used wireless headphones, reported that it was "very hard to tell difference", and selected Sample A as "better", which was indeed the 24-bit sample.

So, what were the preferences for headphone vs. speaker listeners?

Again, interesting! The speaker guys were pretty balanced in choosing either the 24-bits or 16-bits sample, but it was the headphone users who pushed that tendency towards picking the 16-bit sample as sounding "better". Also, many more of the headphone users conceded that they had no preference.

I suppose we could speculate on a number of hypotheses as to why this happened. For now, let me just leave you with the numbers to consider.

V. What characteristics of the sound seemed to change between the two samples?

Suppose we looked at just the 38 respondents who preferred the hi-res 24-bit sample (Sample A), were there any specific sonic characteristics (eg. noise level, bass quality, soundstage...) the listeners thought changed?

As before, we're using a 0-10 scale going from "no audible difference" to "substantial" audibility with each color representing specific sonic characteristics for the listener to consider: noise level, vocal quality, bass quality, treble quality, soundstage, and perceived resolution. As you can see, differences were not identified as large given the skew towards the left.

Of the characteristics surveyed, notice that few thought "NOISE level" changed much between their preferred 24-bit file and the 16-bit sample. Note the very low weighted average of only 0.47/10 for this characteristic. Since in principle it's the noise floor (and dynamic range) that changes most when we go from 24-bits down to 16-bits, the implication here is that noise level / dynamic range of a 16-bit file was clearly adequate when listening to the Daft Punk track for those who still preferred the 24-bit sample.

Of the other characteristics, the sense of "perceived resolution" variation (weighted average 2.89) was rated slightly higher than the others.

On the other hand, how about the 40 respondents who preferred the 16-bit sample, what characteristic differences did they describe hearing?

Again, the listeners did not indicate hearing that the noise floor changed much between 24 and 16-bits. On average, these respondents thought there was more of a change in "perceived resolution" even though the preference was for the dithered down 16-bit sample.

As an exercise in consistency, let's look at the profile of folks who had no preference for which sample sounded better (n=43):

As expected, these respondents generally did not hear a difference between the two samples (lots of "0"). However, there were "blips" suggesting maybe they still thought they could hear audible differences here and there, but ostensibly not enough to declare either the 24-bit or 16-bit file "better" sounding.

VI. Were there any golden ears?

As with my other blind tests, I look for evidence of extraordinary individuals. :-)

There were 2 respondents out of the 121 who selected Sample A (24-bits) as preferred and at a level of certainty of 10/10! 

One person submitted the response from Latvia but did not leave any details for me to report on unfortunately. The other person submitted from Slovakia, assessed the sound using headphones, stating that he (presumably male) heard significant differences in vocal, bass, treble, soundstage, and resolution, and left a detailed message (emphasis mine):

Hi,

I am a highly non-standard participant. Since you don't know me, I'll write something about me. My lineup is extremely beyond anything that people can even imagine.

I mean the quality of the reproduction, not the price. My hearing is extremely superior to anything humans can imagine. To eliminate this, I first used commonly available, not expensive, but high-quality headphones. The differences were immediately and easily audible. Then I used the best audio system I have. The differences are huge. It's like comparing a VW versus a Ferrari. Space, reverberation, separation of tones, airiness, impact and so on.

I can bet a million dollars that if someone plays me one of those songs, I'll tell them correctly which one is playing. This implies that you chose an excellent sample, or that the dither was not optimal.

So that you can verify me in the future, my digital ID is HEEKH44A-8W7SUOVFVR3SD7QQDN4KLKF2. It's the equivalent of the public key in the communications app I use.

If you want, you can publish my answer.

P.S. A is absolutely unquestionably sonically superior to B. Period.

What can I say!? Assuming we're not looking at a very sophisticated AI analyzing the resolution of the data (can't be sure these days!), the man's got cojones and he gave me permission to publish the comment. Well, he's right. I therefore award this respondent with the "Golden Ears Award". Great job!

Unfortunately he did not add a full description of what hardware he used (other than "high-quality headphones") for the listening test so maybe he can leave a comment below along with some tips as to whether there were portions in the music he specifically used to differentiate the samples.

Congrats! In recognition of our golden ears! :-)

"I can bet a million dollars that if someone plays me one of those songs, I'll tell them correctly which one is playing." is a pretty bold statement, so someone might want to take up this wager. :-)

Now to be balanced, I looked in the data and found 2 listeners who preferred the 16-bit sample and also rated the audible difference as 10/10. One thought that the 24-bit version had "irritating, rough edges" when listened using high quality IEMs. Another person works as an audio engineer and used very high quality speakers that cost close to US$15k/pair and felt that the subjective sense of "resolution" was better with the 16-bit sample.

Conclusion...

Overall, looking at the results as a whole, I simply could not find evidence that there was a bias among the 121 audiophile respondents towards the 24-bit sample using this Daft Punk Random Access Memories track compared to the iZotope RX dithered 16-bit version.

If anything, there seemed to be an interesting tendency for the headphone listeners to choose the iZotope RX 10 dithered 16-bit file as the "better" sounding one although this was not statistically significant. I can only vaguely speculate as to why this was the case. (Interestingly, in the 24-bit vs. 16-bit test in 2014, we saw one track where listeners also seemed to have a slight preference to the 16-bit version.)

Even when listeners felt they could hear a change, their ratings for magnitude of difference tended to be small, with few listeners rating >6/10 (where 10/10 defined as "obvious difference"). Audible differences therefore were likely better described as subtle than overt.

As usual with previous blind tests, within the data set, we can find the rare "golden ears" as per Section VI. It's good to find traces of these individuals with potentially very sensitive hearing since these are the folks who should be invited for further study and to ascertain the reliability of their hearing abilities.

In this blind test, the music used is quite different (modern electronica / pop), and the procedure has changed (much simpler with 1 track, and an option for "no difference" heard) compared to our classical music 24-bit vs. 16-bit test done in 2014 using 2L demo tracks. Nonetheless, results remain similar in that in the big picture, there was basically no audible preference for the higher resolution 24-bit version.

While intellectually we can still desire the "best" 24-bit resolution file we can get for our favourite albums, this test adds to the body of evidence that suggests the absence of audible benefits with "high-res" audio. Other tests like Mark Waldrep's comparing not just 24-bits but also increased samplerates (2020) have not been able to ascertain meaningful audible superiority for hi-res (also see the meta-analysis from Reis in 2016, not impressive results as discussed previously).

I think over the course of time as we audiophiles have had more experience with hi-res content, the suggestions to be critical of expectations (such as this from 2014), as well as other discussions we've had on the blog, have proven to be true. I remain "post hi-res" these days and keep 24-bit tracks only for the best sounding albums I own. While I don't think most albums need to be >16-bits, high-resolution 24-bit DACs can still be valuable when we do things like perform DSP and apply gain control. Likewise, in audio production 24/32-bits should be standard these days to achieve the best final mix with least unintended distortions.

For those who might have missed it from the original test invite, here's the 8-bit version of the Daft Punk sample for a listen. Notice the obvious difference between 8-bit and the 16/24-bit versions. This is what lower bit-depth sounds like. There is an increase in the audible noise level as the dynamic range drops, with no significant change in frequency response, general musical tonality/timbre, "speed" of the sound, bass content, or soundstage.

We can see in the graphic above the dithered noise floor comparing 24-bits, 16-bits, and 8-bits. The non-flat 16 and 8-bit noise floor is reflective of the small amount of noise shaping added using iZotope RX in this test.

Finally, a little Public Service Announcement for audiophiles and some practical considerations because ultimately, bit-depth is about dynamic range. And when we're listening to music, this correlates to the range of sound pressure levels (SPL) we can hear:

Make sure not to listen too loud! Protect your hearing in order to achieve a lifetime of hi-fi enjoyment.

For my own listening, I don't push the average sound level above 80dB(A) SPL.  In the quietest domestic sound rooms, we'll be very fortunate to achieve 25dB SPL ambient noise. IMO, 100dB peak SPL for transients is more than enough for enjoyment if your speakers are able to hit this level without gross distortions.**

Assuming we listen at an average level of 80dB SPL, 100dB peak would be 20dB above, adequate for some of the most dynamic classical recordings. This implies a dynamic range of ~75dB in basically an ideal setting (100dB peak - 25dB ambient noise); lesser settings like in a car, 50dB range might be all you can achieve comfortably which might be served by the 48dB dynamic range of 8-bit digital.

** Note 1: We can estimate how much power is needed to drive our speakers. If we want to achieve 100dB SPL from a speaker with 90dB 1W/m in-room sensitivity, allowing for 3dB amp headroom so there's no clipping, seated a comfortable 3m/10' away, we'll need around 180W. Since loudness is logarithmic, 93dB/W/m speakers would need 90W to hit 100dB. Here's a calculator to play with. This is just an estimate though, so I would still recommend doing your own measurement.

Note 2: See CDC chart on loudness and risk to hearing. 50 minute exposure at 95dB SPL can lead to damaged hearing. Constant 100dB SPL exposure risks hearing loss after 15 minutes.

Comparatively, the best dynamic range that vinyl LPs achieve would be around 70dB assuming an excellent cartridge and supporting playback gear which is why we can hear the noise floor with vinyl playback. Don't believe the idealistic vinyl lovers who say they "never" hear noise or hiss - they're either exaggerating, listening at below-normal levels, have poor hearing, or listening in a room with high ambient noise.

As a consumer product, 16-bit digital audio data (CD-resolution, download files) already provide 96dB dynamic range for audio engineers and musicians to exploit. As audio consumers, I think we should be very thankful that Sony/Philips chose to standardize on 16-bits when they created the CD!

--------------------

For the record, to be fair, since I ask others to submit their details, here's my Foobar ABX result listening with Sennheiser HD800 headphones, using the Drop+THX AAA 789 amp, fed by Topping D10 Balanced DAC:

foo_abx 2.1 report

foobar2000 v1.6.16

2023-04-05 17:35:44

File A: 01 - Daft Punk - A - Giorgio by Moroder (16-bits or 24-bits).flac

SHA1: 02d9134e168d7946004a9be93e8c4477544e415a

File B: 02 - Daft Punk - B - Giorgio by Moroder (16-bits or 24-bits).flac

SHA1: fc5e0adac56cd504c7fd9c48acfe00fec593680b

Output:

Default : Speakers (2- TOPPING USB DAC)

Crossfading: NO

17:35:44 : Test started.

17:36:27 : 01/01

17:37:07 : 01/02

17:37:44 : 01/03

17:38:15 : 02/04

17:40:11 : 02/05

17:40:40 : 03/06

17:41:36 : 03/07

17:42:17 : 04/08

17:43:04 : 04/09

17:44:03 : 04/10

17:44:47 : 05/11

17:45:52 : 05/12

17:46:38 : 06/13

17:47:47 : 06/14

17:48:40 : 07/15

17:49:13 : 08/16

17:49:13 : Test finished.

 ---------- 

Total: 8/16

p-value: 0.5982 (59.82%)

 -- signature -- 

5c15a09ccd0957c61dd8b97967690986d680751a

Statistically, the result is literally a "flip of the coin" and I would have selected "No preference" for this Daft Punk track as I'm well aware that I was basically guessing.

---------------

IMO, the whole 24-bit vs. 16-bit debate I think is quite clearly resolved. To me, where this still might matter is with audio streaming. While I am a proponent for lossless PCM since there could still be substantial differences compared to lossy even if psychoacoustic techniques minimize audibility, I'm not sure whether 24-bits or >48kHz is worth it given the increased data rate. Another of those "diminishing returns" situations. If streaming lossless 16/44.1 or 16/48 allows a company to lower costs for consumers while investing in more multichannel content, I think that would be a meaningful path. 

[Although we have been waiting quite awhile already, I am curious what direction Spotify HiFi might take when it eventually becomes available, or if TIDAL might make changes as it sheds MQA and evaluates the tier pricing.]

I see that Random Access Memories 10th Anniversary is out now, mainly featuring some bonus and outtake tracks. I've also recently finally heard the lossless Atmos version of Dark Side of the Moon (50th Anniversary) - the spatial effect is fantastic on a multichannel Atmos system, instruments seem to emanate and sound effects like the clocks ("Time") and cash register ("Money") anchored in defined space floating at various height levels.

Also catching up on some Enigma recently with the most recent release The Fall of a Rebel Angel (2016, DR8). Some pretty cool tracks on there like "Mother" (low bass), "The Die Is Cast" (neat sounds and special effects) and "Amen" (pretty cool tune). "Sadeness, Pt. 2" utilizing Bach's "Toccata and Fugue in D minor" sounded a bit silly to me though, and doesn't rekindle how special the first "Sadeness/Find Love" combo sounded on MCMXC A.D. back in 1990.

Hope you're all enjoying the music!

Addendum: May 15, 2023

Just thought I'd show a comparison of the 16-bit noise floor when we use Audacity 3.3's "Shaped" dithering option going from 24-bit to 16-bit as per the comment from Stephan_M:

Notice that Audacity's "Shaped" setting uses stronger noise-shaping than the iZotope RX 10 setting I applied for this test. With noise-shaping, there's a give-and-take. For Audacity, they've prioritized <6kHz to achieve the lowest perceived noise floor at the expense of higher noise further up, especially above 13.5kHz.