As I discussed with a few people over the last 2 weeks, I wanted to put up an audio test to see if I can capture some data among music lovers and audiophiles regarding MP3. Well... Here it is!
Below are the instructions... You can get the whole test package (music, instructions) from upload.to:
Feb 1, 2013 Update: The following links will be going down soon...
Original package:http://ul.to/ixt7vkgl
or
Feb 1, 2013 Update: The following links will be going down soon...
Original package:http://ul.to/ixt7vkgl
or
NEW tagged files (easier to integrate into music server):
(Thanks Abe!): FAST
(Upload.to): http://ul.to/a5wz1oh0
(FilePost): http://fp.io/588362aa/
or
TORRENT of the tagged files: Get Torrent File
BTW: Feel free to convert to ALAC/WAV/AIFF/etc. or use the Foobar ABX plugin...
(Thanks Abe!): FAST
(Upload.to): http://ul.to/a5wz1oh0
(FilePost): http://fp.io/588362aa/
or
TORRENT of the tagged files: Get Torrent File
BTW: Feel free to convert to ALAC/WAV/AIFF/etc. or use the Foobar ABX plugin...
-------------------------------------------
'Audiophile' High-Bitrate MP3
Audibility test:
Hello folks,
As "everybody" knows, MP3 quality is poor compared
to the original uncompressed source, right? Recently, there was a discussion on
Audio Asylum (hangout for a number of audiophiles:
http://www.audioasylum.com/index.html) about
the merits of MP3 in 2012. As expected, a number of respondents opined that
within seconds, even high bitrate MP3 (~320kbps) could be differentiated from
the original CD source due to the poor sound quality from lossy compression.
Thus was born the idea for this test... My hope is to obtain ANONYMOUS statistics from music lovers to see if:
1. It is true that high bit-rate MP3 can be differentiated
in a naturalistic setting (ie. in the comfort of your own home with your own
equipment).
2. There is any correlation between ability to differentiate
the sound quality with the equipment used (ie. would be great to gather info on
cost of equipment used and list of what was used to listen!).
I know over the years there have been many blind tests and
such, but I hope this isn't a stressful exercise. Have fun with it, maybe have
a little get together with audiophile friends and enjoy the evening while
listening.
-------
PROCEDURE----------
Your 'mission' (should you choose to accept):
In this ZIP are 3
songs in 2 Sets presented as
FLAC's. ONE of these Sets (either A *or* B) was processed extensively through
an MP3 encoder (LAME 3.99.5 – latest stable version) at a high bitrate
~320kbps:
1. "Time" from Pink Floyd off the 2011 re-master
of "Dark Side Of The Moon" - a 2.5 minute excerpt with all the ruckus
of chimes, bells and clocks in wonderful detail and space. A classic audiophile
test track. DR11 - good dynamic range for a remaster in the 21st century.
2. "Church" from Lyle Lovett off his 1992 record
"Joshua Judges Ruth". Even if you're not into country music, this
track has plenty of layered vocals, hand clapping, and a choir to appreciate.
IMO the early 90's resulted in some awesome recording and mastering efforts. DR16 here folks... About as dynamic as you'll ever hear off a
CD!
3. "Keine Zeit" from Megaherz off the recent 2012
album "Götterdämmerung". About as heavy as they come! Recommended by a resident metalhead on Audio
Asylum who feels this would strain MP3 encoding. DR6 is typical of recent
releases.
Have a listen to each
Set, remember ALL three songs were processed the same way for each Set, so if you think "Time_X" is an MP3
encode, then "Church_X" and "KeineZeit_X" (X = either A or B in this case) must also be MP3 processed.
I know the temptation is there for many, but please do NOT open up the
audio files with a file editor until you’ve had a good listen and completed the
survey! We listen with our ears, not with our eyes, right :-)?
Once you have listened, come fill out the ANONYMOUS survey – EVEN/ESPECIALLY IF YOU DO NOT HEAR A
DIFFERENCE! (only 6 questions):
I will report on the results of this survey in about 2
months (February 2013) on my blog including analysis to give everyone time to
listen as well as specifics on the bit rate and how I molested the files
through the MP3 encoder!
Happy listening and happy holidays everyone!
-------------------------
Note that the procedure and testing is purely for research /
educational purposes. The audio files are partial segments borrowed with no
intent or opportunity for any financial gain on my part. Please erase the audio
files once testing is over and if you enjoyed the music, purchase the
respective CD's as described above. Thank you!
Happy holidays to all the respondents from around the world so far!:
ReplyDeleteBrazil, US, UK, Argentina, Slovenia, Switzerland, Germany, Australia, New Zealand, Poland, Netherlands, Finland...
Keep 'em coming! Hey, where's Asia - Japan? China? Singapore? Malaysia? Thailand? India? Surely there must be massive # of audiophiles there!
Happy New Year everyone! A reminder with plenty of time to go before closing the data collection.
ReplyDelete--- Originally posted on AudioAsylum ---
Since opening the high bitrate MP3 vs. CD test on Dec. 11th, I have received 41 responses so far to the detailed survey. I will not analyze the data until the end of January, but just eyeballing the spread of results (Set A, B, "same") is already quite interesting. The respondents have come from 4 continents so far and reading some of the comments, I really appreciate the time people have put into this!
Furthermore, it's great to see a nice spread of equipment used from inexpensive (but good) headphone gear all the way to megabuck $50K+ systems.
Although we "shoot the breeze" around here and have great discussions around the hardware (sometimes inflaming arguments), it is infrequent that we actually do something like this where we stand up and be counted based on the actual experience of listening. I know that this isn't strictly "scientific" and many variables cannot be controlled in an open test like this, but for us "non-pro's", this could be the closest we get to participating in something which I hope is educational and (hopefully) fun as a hobbyist beyond theoretical discussions.
If you haven't given this a try, I encourage you to take some time and give it a shot. Be involved in a simple "blind test" (perhaps the only time in one's life) knowing you've tried something like this and contributed to the data set (whether one believes it's significant or not).
Thanks again AbeC. for hosting the fast link! Much appreciated, bro.
PS: One request - could some of you who participate in audiophile discussions in Asia (India, China, Malaysia, Singapore, Indonesia, Philippines, etc...) spread the test around. Would love to get some data from those folks!
My $0.02:
ReplyDelete1) With only samples A and B but not X, people can claim to hear difference between A and B but there is no way of verifying it.
Let's say your results end up with 2 guys picking the right choice and 2 guys picking the wrong choice. But in reality 3 guys didn't hear the difference, one guy picked the right choice by luck, and 1 guy genuinely heard the difference. How would you do the analysis to avoid discounting the guy who actually heard the difference?
2) For greater credibility, you could put the answer in a password-protected RAR file that is downloadable now and release the password along with the test results.
Good points wwenze.
ReplyDelete1. I wanted to keep the test simple. To have an X sample would add 30% to the download size, plus it would be very easy to compare X with the other files to find the answer without much effort... As for the test of significance, this is why I've been canvassing forums to get as many responses as possible and also asking people if they found the difference easily. Presumably if MP3 degrades sound, then there should be some predilection for the correct Set to be chosen irrespective of the 'no difference' group. Also the ones who thought it was easy to detect should bias towards the correct Set as well if their impression is correct that one Set sounds much better. The more the respondents, the better.
2. Interesting idea! Would have done that if I thought about it at the time the test was started. Nonetheless, when I release the answer I'll also release stuff like spectral analysis and wavdiff results for anyone to do their own comparisons if they mistrust the answer.
BTW, we're up to >80 respondents as of tonight... Let's make it 3 figures soon :-)
Deletewwenze (and Archimago),
ReplyDeleteJust a quick point. I think the mindset that there's a "right" or "wrong" choice, or that one is "degraded", is the wrong way to look at this. I listened to the samples on and off over the past couple weeks, primarily "Time" because I like it and I've listened to it a bazillion times in various formats on lots of different systems over the past 30 years. I found I liked "B" better. Does that mean that "A" is the processed MP3 version? Why on Earth would that follow? Assuming they *DO* sound different for a moment (whole 'nother can of worms) why would the fact that I prefer one over the other mean that the one I like is the unprocessed one? Maybe whatever Archimago did to convert the sample to an MP3 resulted in music that I like better. Maybe the drums sound "boing-ier" after processing, and I *LIKE* boing-ier drums. Maybe the dynamics got a bit squished. Maybe a bit of the high-end tizzel got tizzelated. Whatever. Given that I have absolutely no idea what this music is "supposed" to sound like, all I can do is tell you which one I'd rather listen to. I'm very curious to see whether that turns out to be the MP3 or not, but either way I'd say "processed = different", not "processed = worse".
I do agree that the ABX thing would have added an interesting dimension to the exercise. At the same time, I have a feeling it'd be a *LOT* harder to make up my mind. Booting up Foobar and listening to A, then B, then A, then B, asking myself which one I like better, then going on to other stuff and coming back a few days later and doing it again seems like it'd be a *LOT* easier than listening to A, listening to B, listening to X, trying to figure out whether X is A or B, and repeating the whole process over and over. I know this argument doesn't carry weight with a lot of folks, but I believe it's possible for someone (uh, well, me, I guess) to pick up subtle, really hard to nail down differences that lead me to prefer one thing to another thing over time, but trying to articulate those differences, or even identify them to myself, then pick out those same similarities / differences in a third choice, is really, really hard. I like Coke more than Pepsi. The differences between them are probably pretty easy to spot given the right analysis tools (gas chromatograph?). But I bet if you lined up a bunch of paper cup triads, with Coke, Pepsi and X in each set, and asked me which one X was, I'd be wrong a bunch of times. Doesn't mean I don't like Coke better. I think.