.png) |
| BTW: 1.5kHz sine waveform, -65dBFS. 16-bits on the left, 24-bits on the right. |
It's amazing how time flies. Nine years ago - 2014 - at a time when hi-res audio downloads were being promoted as the "next best thing" for audiophiles, we ran a 24-bit vs. 16-bit blind listening test on this blog. I'll leave you to examine the results.
Since then, electronics like DACs have advanced quite significantly. Compared to 2014, these days, high quality asynchronous USB DACs have become ubiquitous. Jitter is basically a non-issue now. Inexpensive audio streamers like Raspberry Pi devices can be easily put together. We can all play music with wide selections of digital filters without great difficulty. Very high quality headphone amps and phenomenal headphones have become commonplace with a growing "head-fi" community.
Recently, I had some fun on this Steve Hoffman Forums thread engaging in the discussion about 16-bit vs. 24-bit; the topic of audibility again came up. As usual, things can get pretty contentious among audiophiles and since it has been awhile and perhaps some of you did not have the opportunity to participate in 2014, let's do a blind test again for those wanting to listen to 24-bit vs. 16-bit audio.
Let's use popular music rather than the more esoteric 24-bit 2L classical recordings last time - let's try some "classic" Daft Punk. ;-)
No need to spend too much time talking about the procedure, it's a very simple listening test.
1. Download this file (there's a download icon top right of the page to grab the ZIP):
Inside you will find 2 samples of "Giorgio by Moroder" from Random Access Memories (Sample A and B). It's a good track I think because of the diversity you'll find over the 2.5-minute segment I selected. There's the sound of the natural spoken voice, some subtle background audio (sort of like the incidental noises on Jazz At The Pawnshop), and in the latter half of the sample, you'll hear heavier beats with good bass.
I think the 2.5 minute sample should be enough to hear significant or even subtle differences. To keep download size small, I downsampled the original 24/88.2 official hi-res download to 24/44.1kHz using an excellent resampler. This was done before dithering the file to 16/44.1 to create the 16-bit sample. Regardless of whether you believe 88.2kHz sounds much different from 44.1kHz, any audible differences between the 2 samples would still be a result of the bit-depth reduction and dithering.
To maintain the blinded condition, both samples are presented as 24-bit FLAC (even though one of them contains 16-bit dithered content) to feed your DAC in case that's a source of sonic difference. As usual with my Internet Blind Tests, various "honesty control" mechanisms were implemented to reduce the chance of bias as much as I could without compromising the 24-bit quality compared to dithered 16-bit resolution. All details will be revealed after the data has been collected. It's a listening test so please try to refrain from running the samples thru analyzers and audio editors.
I'll be going on vacation during the last weeks of April, so I'll keep the listening test open until May 1st - plenty of time to listen and complete the survey.
Feel free to use software or procedures like the ABX Comparator in Foobar2000. As usual, make sure that your system is properly set-up for full 24-bit playback; much easier these days compared to the test back in 2014.
2. After you've listened, send me your impressions, audiophiles!
For convenience, I've embedded the survey below if you want to quickly look at the questions and format being asked.
Not only will I ask which sample sounded "better" (knowing that one of the samples is the hi-res 24-bit source), it's also important to indicate "no difference" if that's what you perceived and have that perception counted.
I'll also ask how much difference you heard - so if you're very confident, rate this as 10/10 ("obvious audible difference"). Optionally, you can also tell me what seemed to have changed if a difference was audible - noise floor, bass quality/amount, soundstage change, perceived resolution, etc...
Tell me what kind of listening system you used primarily - speakers, headphones, and wireless heaphones. That last category is a new one as I have seen an increase in the number of people primarily using mobile, wireless devices these days. Yes, some codecs like aptX HD and LDAC could offer advantages with 24-bit settings.
I've included an optional comment box at the end - feel free to include your anonymous alias there if you want to let me know who you are and we can go over your results another time privately perhaps. I find this is helpful because sometimes a tester will E-mail me because they forgot how they answered and I have no way to be sure I retrieved their data accurately.
There's a README.pdf in the package with full instructions as well if you want to send this ZIP to others, and a Foobar2000 DR report to confirm there are no changes to volume. The test track is DR8 average, typical for many modern recordings, but portions of it are significantly more dynamic. That first part with Moroder being interviewed measures out to around DR11.
Enjoy the music if you haven't heard this track before!
Maybe by doing this, more of us will have the experience of doing a direct comparison between 24-bits and 16-bits. Recommended to try for those of you with good ears and uncompromising high-end sound systems.
In aggregate, let's look at the data for some "facts" once it's all collected after May 1st.
By the way, while the French duo Daft Punk broke up in 2021, expanded editions of their albums are being re-released like Homework (25th Anniversary Edition). And soon Random Access Memories 10th Anniversary Edition on May 12. I'm looking forward to the bonus and demo tracks. Too bad the multichannel Atmos will only be available on streaming services. Ten years already... Time flies!
Make sure to do the poll/survey from a few weeks back if you have not done so on what kind of system and which streaming service you're using! Would be great to capture as many audiophiles as possible willing to disclose some not-too-private details.
Well, work remains very busy here so I'll let you guys and gals do some listening until next time. As always, hope you're enjoying the music!
------------------------
.jpg)
Seeking truth as audiophiles...
One last thing I thought I'd just address. As a "more objective" audiophile, I know that pure "subjectivist" will at times perceive/allege that the "objective" guys come across as too presumptuous or pompous in comments. That objectivists tend to make too many rigid assumptions about others, and in doing so, come across as insulting. (There were such allegations here about "because... science" guys in the 16-bit vs. 24-bit thread.)
Most times, I don't think this is a fair criticism so long as the comment wasn't actually written in a rude fashion. Audio products are based on engineering and various invented technologies. They simply are products of science. As participants in this hobby then, shouldn't we explain perceptions and beliefs ("because...") out of scientific principles ("science")? Indeed, as another poster noted, we really should not be "because... Stereophile says so" or "because... Voodoo" (choose your religious analogy) hobbyists, right?
Regardless of our differences in beliefs and personal philosophies, I hope there is one unifying phrase that can unite all audiophiles - "an honest desire to seek truth". There are two parts to that statement which I believe are needed in order to find balance.
Honesty can be defined from different angles. In particular with audiophilia which deals with the ephemeral experiences of sound (echoic memory is brief), I think we all need to have a certain level of self awareness. A delusional person can be very honest but at the same time, can be very wrong. I know, it's a hobby and it's supposed to be fun, but we still need to be able to make good judgements which is why we discuss and debate in the first place; preferably not just hanging out at echo-chambers where everyone always completely accepts without question. I simply don't see how unchallenged delusional beliefs could be considered appropriate in any hobby regardless of how much "fun" it's supposed to be, especially when there are financial transactions taking place.
I try to gauge whether what I say is based on just opinions or citing "facts" as best can be discerned, contemporaneous with what is believed to be known at the time of the writing. Facts with evidence is always better in any debate. And watching that our ego doesn't intrude on fruitful discussions. ;-)
Recently, a reader suggested that I watch a few of the YouTuber OCD Hi-Fi Guy's videos. Yikes, I think many of his videos are a bit of a mess. On the one hand here's a guy who seems to present himself as an "everyman audiophile" looking to take care of his viewers based on decades of experience. But he sells audio stuff, and profits from it (so you know there's a financial bias). He also owns and sells VeraStarr cables which can cost >$1000/m for interconnects. He's clearly predisposed towards products like Playback Designs - stuff which he sells. I see him trash-talking products he doesn't represent and seems to take pride in announcing that he doesn't carry certain popular brands. He seems to think up to US$10,000 cables can be worth it for sound quality (really? why $10k?). His anti-China bias is rather ignorant since the topic is highly nuanced. Furthermore, he tells his viewers not to go onto audio forums because apparently they're full of shills (and he's not shilling stuff in the first 5 minutes of this video!?). Are the ideas demonstrated in his videos really any more worthwhile or true compared to others? Does having >20 years experience in the industry, which he seems to play up, actually make his advice any better? I find it fascinating watching personalities like this on YouTube as entertaining "character studies" I suppose, but there's nothing there to suggest technical expertise about high fidelity audio that we wouldn't find almost anywhere else nor does he show particularly impressive levels of self-awareness or humility. His "Audio Autopsy" videos looking at what's inside expensive cables and certain products look interesting though.
The second part to that statement then is that we should try to find truth. Going back to the Steve Hoffman Forum about 24-bit vs. 16-bit audio and dithering, with all the differing opinions expressed, eventually a poster (THOMAS STRAIGHT) said (page 9 and 213 messages in):
"What a rabbit hole. I have read most of the posts and watched most of the videos. I have never seen this much conflicting information about anything. People who write like they know what they are writing about cannot agree on the basics let alone the finer points. I'm really only interested in something that makes an improvement in what I can hear. Most of what I've read from this thread says that higher bit and sampling rates are helpful with recording and mastering but 44.1/16 for final playback is as good as it gets. Your milage may vary."
What a beautiful expression of the frustration in that thread when even "basics" become contentious. This was why I thought it would be fun to hopefully enhance that discussion by creating this listening test since Daft Punk was suggested as a good choice.
Well ladies and gentle, empirically, as a group of listeners, tell me whether you can hear a difference with this blind test. We can debate, argue, yell at each other until we pass out of hypoxia. But why bother with just that when we can all have a listen to some nice music and elevate our discussions beyond our individual impressions?
Objectivists - don't assume without listening that there are no differences. Also, there could still be effects with the dithering I used such as possible tonal shifts. If so, I'd like to know!
Subjectivists - why not experience what a blind test between 16-bits vs. 24-bits sounds like. Tell me how much difference you hear and in what ways. Make a note for future reference for how you voted. Once the data collection is over, it'll be fun to know if what sounded "better" to you was indeed the 24-bit higher resolution version.
Finding truth together, and as honestly as we can be to each other, is I think much more fun and meaningful in the long run. Looking at ways to do this is how we truly "up our game" as audiophiles which will be reflected in the quality of our discussions.
Addendum: March 23, 2023
There were discussions in the comments about listening to an 8-bit version of the test. Here it is for those who want to experience what reduced dynamic range down to ~48dB (from 96dB with 16-bits, and 144dB with 24-bits) sounds like. A small amount of noise shaping used.
Notice the sample is presented in a larger 24/44.1 "bit bucket" like the A/B test tracks:
Daft Punk's Giorgio - 8-Bit Dithered Version
-------------------------
Archimago, that's a nice test and I've personally found no significant difference between both tracks. I've made this test with my own tracks and also never found significant differences - the only signficant difference I've found this far is using DSD upsampling and I suspect the improvements are solely due to my DAC processing DSD better than PCM.
ReplyDeleteI would like though to raise question, and I hope it's not a dumb one (and it comes from a musician's perspective).
If I understand correctly, one of the biggest differences between 16bit and 24bit is a significant increase in dynamic range (although I'm aware our own room's noise floor might mitigate that difference a lot). This would be, for me, the main reason to use true 24 bit music (not upsampled from 16bit, I can do that by my own).
If that's the case, wouldn't it make sense the test tracks on such a test to be from a piece that acually contains dynamic range? Nothing against Daft Punk and EDM, but there's abasolutely no dynamics on these tracks (and in music in general these days). Wouldn't a classical piece, which often contains abrupt and sudden dynamic shifts, make more sense? And, in that case, maybe the (hypotetical) differences between 16bit and 24bit would become more obvious?
Again, I hope this is not a dumb question. And, as always, thanks for your posts!
Hi there jorge,
DeleteNo dumb questions friend! As they say "only dumb answers" ;-).
Yes, you're right, in general it is good to use more dynamic music for these kinds of test which was why in the first round in 2014, I specifically use those 2L hi-res classical samples.
Having said this, I was okay with using this Daft Punk track because there are portions in there that are dynamic like at the beginning when Morodor is speaking and you can make out the background details (I think they interviewed him in a restaurant?), along with a layering of the music at lower levels than later on. That portion has a DR value of 11 or so.
Another reason I thought would be interesting is the fact that these days, EDM, synthpop, electronica, R&B are just way more popular as music people actually listen to! (I'm a classical listener as well but in reality, to be honest, maybe 10% of the time when enjoying my music...)
Compared to many other albums, at least Random Access Memories sounds better than the majority out there and has this cool section to test out.
Ok, that all makes sense! Thanks for the clarification, learned a thing or two :)
DeleteI found another online test here: https://www.audiocheck.net/blindtests_16vs8bit.php I got 60% right. Therefore I will give myself the 24bit vs 16bit test.
ReplyDeleteHave fun Stephan!
DeleteYeah, 16 vs 8-bits (8-bits only ~50dB potential dynamic range) should show statistically significant audible differences when testing a broad population of audiophiles.
It might have been good to supply a positive control...say, "Giorgio" downconverted to 8 bits (or less)
DeleteSee addendum Steven. ;-)
DeleteCannot wait for the result. I thought that I could hear the difference. Still no clear "better" just different :) I might have slight preference for one sample (mentioned it in survey comment)
ReplyDeletePatience, friend. ;-)
DeleteAll in good time... Interesting comment regarding "different" but not necessarily "better". I guess it comes down to a forced choice then!
Ah yes, that's the problem. How you define 'better' :)
DeleteSome would say that it depends on your subjective view.
But some would say that better is the one which is closer to the truth - in this case to the original 24bit file :)
But how can you know what was original when you don't have a reference point.
:) as I said better can be tricky word.
ps. maybe next time we should be doing something like three files. File A - 16bits File B - 24bits (original) and File C - original 24bits file marked clearly. Then we can try which file from A/B is original one and if people can hear that.
Anyway as always thanks for the work you are doing.
Hi Milan,
DeleteAs you can imagine, as a blind test distributed over the Internet, I would not be able to properly keep it blind unless the music were wrapped in some kind of player software; which would mean certain computer testing only, and we can't sent the audio to playback devices. If I gave a Sample C which is known to be 24-bits to compare, then some could even inadvertently notice that 2 files are of the exact same size, thereby there's your answer...
I think choosing which one "sounds better" is fine because much of the time these days we're just streaming whatever version is on the service or whatever version we bought as a hi-res download without assurance that a 16-bit CD version was derived from that exact mastering.
Here, we have a choice. While ultimately all perceptions will have to run through our mental filters, which of the two, as a "high fidelity" lover, do we choose? Whether it's perceived lower noise level, better tonality, clearer vocals, broader soundstage, or other characteristics that makes it presumably more enjoyable, then let's pick that one - and see as a group whether we have a preference towards the original 24-bit.
We can imagine 3 potential outcomes:
1. If we do have a preference to the 24-bit version, then maybe hi-res with 24-bit files could have value to audiophiles listening to the music - or my dithering isn't totally transparent.
2. If the 16-bit version is considered "better" then maybe the dithering technique was awesome and there's no need for 24-bit hi-res! ;-)
3. And if it looks like the vast majority said "no difference" with few having any strong opinions, then maybe we just never needed 24-bits to begin with when it comes to modern recordings like this and the dithering settings I used were good enough.
What do you think ladies and gents? Gimme your opinion - which sounds "better".
Yes, my 'idea' about 3 files were more a theoretical one, hard to do it in the digital world and keep it fully blind (actually I'm a software developer myself).
DeleteAs for 24 bits vs 16bit. Personally I really don't see it as being 'the only way' to good sound. If the recording/mixing/mastering is done right and nothing wrong was made during "resampling" down from studio bit depth and studio sampling frequency than it will sound good.
You may be interested in this dithered 8bit v 16bit test and demonstration. It is quite convincing that the only difference between the two is noise, not fidelity. The noise difference between 16bits and 24bits would not be perceptible to humans. https://www.youtube.com/watch?v=Rc_-eavKptY&t=3s
ReplyDeleteJust to add to that, it would have been interesting if Sheppard also compared 16 or 24 bits with increments from 8bits. I'm willing to bet that once we get to differed 10bits very few would pick the noise from 16 or 24bits (after all, 10 to 11bits is considered sufficient to capture the full dynamic range of an LP) and would fully transparent at 14bits (this was the orginal red book spec and under independent blind tests by Phillips, was found sufficient to capture the full dynamic range of analogue reel to reel master tapes). 16bits is already overkill for playback.
DeleteI've also been wondering where the audible limit is, and couldn't find tools to test it other than reducing to 16 bits. So I wrote a command line tool to simply copy audio files replacing the N least significant bits with random bits, for all N, so you can try to hear the effect on any track or excerpt. That probably doesn't dither the way it's supposed to, but it's an indication of how much it may matter. Preserving only 8 bits or 9 bits is easy to pick up, but not in noisy music. The only time I can hear the disadvantage of 12 bit sound is while zooming in on silence in tracks.https://github.com/ssomers/audiobits - if anyone is interested, I can put up Windows or Linux binaries too.
Delete> I've also been wondering where the audible limit is, and couldn't find tools to test it other than reducing to 16 bits.
Deletesox can do reduction to any bitdepth:
sox INPUT OUTPUT dither -p10
It can also apply noise shaping:
sox INPUT OUTPUT dither -p10 -f gesemann
My thinking on audio file quality is that between storage being so cheap and streaming being so easy, there's not much reason to grab anything but the highest-resolution file that works with my DAC. Production techniques and file formats are just as interesting to me as objective measurements and what's on the Hot 100.
ReplyDeleteDoes bit depth "matter?" Not when I'm listening to music, but that's not all I do as an audiophile. I like talking about the math, the file formats, the differences between different masters of the same track. The "Play" button is a black box, and I want to unpack it just because it's there.
Yeah, I agree Allan,
DeleteIn a way it's kind of academic these days since storage is cheap and unless we're going nuts with 32/384 + multichannel recordings that can take gigs, a large hard drive should cover everything.
However, as a "perfectionist" audiophile who desires to find out the "truth" of claims, and also being obsessive about efficiency and not being wasteful, I find this interesting. I bet there are some who claim that 32-bits sounds better than 24-bits, and I presume there must be salesmen somewhere willing to sell us more music we might already have. ;-)
Re: the math. It's quite incredible to see how one can monkey around with the files. Waves. Energy transfer. Sound perception, among humans (and dogs, bats, hummingbirds,..). I wish this material was added to my employer's curricula (my gr10 students are introduced to the electromagnetic spectra, and "light"). Ah well. Maybe the next provincial update will add it. Parents: insist upon it, eh.
Delete(Important advice: English is not my native language, and the topic deserves top lexical ability. I'll do my best).
ReplyDeleteAdmired Archimago:
I haven't done the test yet. There's a BIG flaw in these kind of blind tests; albeit they're quite informative and useful.
The "problem" with these scientific approaches, which are perfect, is that they try to be radically objetivist. And that's good, but when we talk about enjoying music (or other art objects), some subjetivist facts should be considered, inevitably. I miss an important variable when measuring perception. A variable that wouldn't affect the scientific impartiality of measured perception: PRIOR KNOWLEDGE OF THE ARTISTIC OBJECT; TRAINING.
I like Daft Punk, ...I like a lot of music, lot of genres... I'm SURE I'll be "blind", "deaf", perceiving technical differences when listening to random music pieces (even liking them all). There're a few "records" I've listening regularly to since 30+ years, and still loving them (The Dark Side of the Moon, to name one). Is with this music that I'm capable of distinguishing minor and subtle differences! Only with this kind of music! Music that I've mentally studied during a long time.
It's the same when an art (painting) expert distinguishes a copy from the real thing. I ignore if this important fact has been considered when measuring "fidelity"; I have never found a study considering this cognitive variable.
I think it is important, because that prior knowledge of the music piece doesn't interfiere with a blind perception test, and it would measure the subject perception at its best, which is desirable.
I find this important too because, using perfectly scientific data, it helps to affirm dubious facts: Yes, I'm unable to tell the difference between a 320mp3 and a SACD file when listening to RANDOM music. Let me listen to "Wish you were here" song, and I'm sure I could discern! ...(Or perhaps not?)...
Is there any scientific study that considers previous knowledge of the artistic object?
(I'm a philosophy teacher, I'm quite interested in sensation/perception processes... I admire your work with this blog: it's a joy, it teaches me a lot, and it's made with personality and good taste/style. Thank you very much, sir).
Thanks for the note Ludovico,
DeleteI suspect that it is possible that a listener would be heighted to very subtle differences based on intimate awareness of the music. Since this is quite a popular album, I assume there will be many listeners out there already familiar with this music. Note that even if a listener is intimate with this song, I'm not sure how many would have been "trained" on the 24-bit version. I bet most fans would have only heard the 16/44.1 CD version.
From other research, we know that trained listeners and those with musical backgrounds are known to be better when asked to judge subtle timbal differences and timing variations. I think this would be non-specific though and that skill can be applied to any audio comparison. From previous tests, I know that a number of you do have musical training so let's see if there's any evidence that this works in the favor of the 24-bit sample...
I definitely try not to be "radically objective" in this blog... I think it's important to find balance overall. A test like this one I see is more "radically pragmatic". ;-)
If 24-bit audio is worth the expense in cost and storage space, let's see a preference for the 24-bit version! With enough people testing, I trust we might be able to have enough "power" to show a positive bias if there is one to be found.
Cool that you're a philosophy teacher Ludovico! Always good to think about all sorts of questions deeply and appreciate those fundamentals of existence... Cheers!
Submitted! Patiently looking forward to seeing and reading about the overall test results. Cheers,
DeleteThanks Ricochet,
DeleteAll in good time, friend!
Hey everyone,
ReplyDeletePlease find in the "Addendum" above an 8-bit dithered version of the test track to listen to.
As noted, this is what a dynamic range reduced version down to 48dB sounds like. I trust that you don't need "Golden Ears" to obviously hear the difference compared to the blind test samples!
Wow, I really enjoyed the accompanied noise! :-p...
DeleteBack and forth. Back and forth. Nope. My kit was able to smooth out the rough spots. 16-bit. 24-bit. With mid-range (open-back) headphones. My Schiit Audio gear was remarkable at making this EDM pleasant to listen to. I enjoyed this test, Archimago. In less than 2 years, I'll be in a QUIET part of Ontario. That. That will be an interesting time to revisit this test for me.
ReplyDelete