RESULT: Archimago's LP Needle Drop Blind Test

Okay, finally it's time to put up the test results for the "LP Needle Drop Blind Test"!

Of course the TMS system looks great compared to my "man cave" vintage Technics rig. :-)

A big thank-you to those who spent the time to listen to the 3 vinyl rips and then entered your results in the survey site I put up!

As I noted with the test page, this test is more for fun and obviously has less potential significance than the previous high bitrate MP3 listening test, or the more recent 24-bit audio blind test. Nonetheless, I think it does illustrate some points about vinyl sonic reproduction to those who had a listen and provides an opportunity to talk about the relative cost of audio hardware and what that actually buys in terms of sound quality...

Procedure:

As described previously, the purpose of this blind test was to ascertain the preference of respondents listening to a high-resolution rip of Paul Simon's "Diamonds On The Soles Of Her Shoes" from the same Graceland album (2012 25th Anniversary Edition - RTI 180gm remaster - Matrix / Runout (Side A): 88691914721-A RE1 20315.1(3) STERLING RKS). The samples were all ripped with the same ADC (my Creative E-MU 0404USB) and laptop (Acer Aspire 5552-7858) using Audacity 2.0.5 at 24/96 bitdepth/samplerate. The raw audio files were edited with Adobe Audition 3 for equivalent fade-ins and fade-outs, and average volume was normalized across the three samples (alas resulting in a few clipped samples which I do not affect the final "gestalt" of the sound). I felt that a 2 minute sample was adequate for evaluation of sound quality. Instantaneous A/B testing with something like the foobar ABX Comparator was encouraged.

A significantly more expensive turntable system consisting of the Roksan TMS, SME309 tonearm, Ortofon Cadenza Black moving coil cartridge and Whest PhonoStage.20 preamp was compared with my own modestly priced stock Technics SL-1200 M3D turntable with either a Denon DL-110 (high output moving coil) or Shure M97xE (moving magnet) cartridge fed to an Emotiva XSP-1 preamp with phono input. 

In total, I received 39 responses to this test. I know that my FTP site uploaded >150 copies of the audio test file so I suspect many more people must have listened than took the time to respond. There was also an alternate download site so I suspect much more than 150 people downloaded the samples.

Compared to previous tests with survey result of >100 responses, 39 results is relatively small but realize that this is still larger than most informal surveys I have seen done on audiophile forums. Also, compared to previous surveys, this one wasn't advertised to nearly as many places. Vinyl rips are a little odd, "straddling" both computer audio in that they're often ripped at high-resolution (24/96 in this case) so ideally the tester has high-res digital playback gear, but the actual sonic differences are mainly due to the analogue hardware and would likely only interest those who know about turntables and cartridges. Needless to say, despite the growth in LPs, the LP format remains a niche market (8.3 million LPs in all of 2014 for the U.S. isn't bad, but remember, weekly CD sales is on the order of 4-5 million units but dropping).

I announced the test only on the Audio Hardware forum on Steve Hoffman Music Forums, the Squeezebox Audiophile forum, the Vinyl forum on Audio Asylum, and the Vinyl Circle at AudioCircle.

As in previous tests, the responses came from various countries around the world:

Interestingly the majority of survey responses came from Europe - ~54%. Followed by North America with 33%.

Results:

In the survey, I asked respondents to rate which of the 3 files they liked the MOST, the SECOND, and finally LEAST.

As you can see Sample A marginally edged out Sample B as the "best" sounding file by only 1 vote. However, if we include results for what was "second best", clearly Sample A beat out B and C as preferred. Consistently, Sample C was considered least well liked.

A weighted score (3-points for best, 2-points for second best, and 1-point for least preferred divided by total of 39 entries) results in:

Sample A - 2.28
Sample B - 2.03
Sample C - 1.69

So, what analogue gear produced these samples:

Sample A - Technics SL-1200 M3D with Denon DL-110 cartridge
Sample B - Roksan TMS/SME309/Ortofon Cadenza Black cartridge
Sample C - Technics SL-1200 M3D with Shure M97xE cartridge

Overall, it looks like the Technics SL-1200 and Denon combo connected to the Emotiva XSP-1 preamp was the "winner" in this survey! Not by a huge margin of course.

To gauge whether respondents thought the difference was worth an upgrade between the "best" and "worst", these were the results:

In total, about 1/3 of respondents thought the difference was worth the upgrade versus a combined 51% thinking there's probably not enough of a difference to consider a $1000 upgrade (of course the price difference between the TMS system and the Technics is much larger!). Interesting to see that 15% basically found that the difference was significant but would not change their level of musical enjoyment if upgraded.

I was curious - of the 33% who thought an upgrade was warranted, which system they thought sounded the "best", "second best", and "worst" (to get an idea of which system they thought was most worth upgrading to):

Weighted score of the data above (of 13 respondents, maximum 3 "Best"):

Sample A - 2.23
Sample B - 2.0
Sample C - 1.77

Again, we're seeing a similar pattern to the results of the full sample; that overall Sample A (Technics SL-1200 M3D + Denon DL-110 cartridge) was preferred and the same turntable with the Shure M97xE was least preferred. There was no evidence of any special preference with the group that heard a greater difference.

Like with my other tests, I asked the respondents to identify the digital gear used to listen to these samples. Also like before, clearly most audiophiles who took this test used very good equipment to listen with. DACs used included the Schiit Modi, Naim DAC-V1, Meridian Explorer, ATOLL DAC 200, M-Audio Delta 410, iFi Micro iDSD, FiiO X3, Benchmark DAC2, Mackie Onyx Blackjack, Simaudio Moon 100D. Only about 4 respondents reported using the built-in computer motherboard DAC. About 50% of respondents identified themselves as using headphones for the evaluation. Headphones identified included many using Sennheiser models (HD800, HD650, HD600, HD280, HD580, HD428), Sony MDR-7520, HiFiMan HE-500, Audeze LCD-2 rev2, AKG K530, AKG K612 Pro, AKG K701, Denon AH-D2000, AudioTechnica ATH-M50x, Grado SR80. Speaker systems included Monitor Audio RX8, Linn Isobarik, B&W 805, Rega RS5, Klipsch Heresy, Dynaudio Contour S3.4, Devialet 200 + B&W 802D system, Meridian active speakers, Mission 702e. 18% reported using foobar ABX Comparator or equivalent.

Summary - So What?

Well, I did say this is for fun, right? But I think there are a few general facts to keep in mind when talking about vinyl drops in general and specifically analogue gear.

1. I believe a well recorded LP needle drop (especially at high resolution) does represent the actual sound produced by the turntable/cartridge/preamp - essentially 100%. I tested this by recording the track at 24/96 (with the old E-MU 0404USB) and volume matched with the turntable playing back the song in realtime, switching between inputs resulted in indistinguishable sound at least for me (and my wife was willing to lend her ears); as in all subjective experiences YMMV. Of course, it's important to make sure you play the digital files back with an accurate digital front end. As a result, I believe one can evaluate the sonic differences quite easily - audibly obvious compared to say high bitrate MP3 vs. lossless or the difference between 16/44 and high resolution digital. for example, a friend sent me his rip of this same track using the Audio-Technica AT150MLx and the high frequency boost from that cartridge was easily heard.

2. A number of people commented they did not like the sound of vinyl rips in general compared to the CD rip. I can totally understand, especially since so many respondents used headphones for evaluation. Vinyl rips, especially when completely untouched like the samples in this test do contain the imperfections of vinyl playback. Surface noise is heard during quiet segments and the occasional crackle and pop will be heard from dust, static, or vinyl defects. I have seen people post on forums that they've never experienced surface noise and never heard a pop - that's just nonsense. It's not uncommon to find new LPs with noisy background even after thorough cleaning and many of my records from the oil crisis days of the early to mid-1970s clearly were made with noisy, probably recycled vinyl. (Interestingly, most of my 1980's LPs sound great even though they may appear thin on inspection.) Over the years I have checked out some beautifully recorded vinyl rips rivalling the quality of professionally produced remasters; there's a real skill and art in restoring the sound especially for music that was dynamically squashed in the CD/digital release.

3. I think this is a nice demonstration of how important the choice of a good cartridge is! Notice that the "best" overall sample was the Technics turntable with Denon DL-110 cartridge. And the "worst" sample was the Technics turntable with Shure M97xE. The only difference being the different cartridges. Nice to know that the more expensive Denon in this case was felt to be superior on the whole (at about 3x the price). The main complaint about the Shure was that it accentuated the sibilance in Paul Simon's vocals (like in the word "she"). Even so, 20.5% of respondents thought the Shure sample sounded best so there's obviously significant subjective variation.

4. BUT a much more expensive system like the TMS turntable + SME tonearm + Whest phono preamp did not draw higher preference overall from the respondents. It goes without saying that there is always a point of diminishing return with mature technologies and it's no surprise that price itself is not necessarily a determinant for best sound quality. I have already previously posted on wow & flutter measurements for both the Roksan TMS and Technics SL-1200 M3D demonstrating that the Technics is actually more accurate in terms of absolute speed. A few respondents and chatter on the Steve Hoffman Forum noted that Sample B (TMS system) had an audible hum in the silent portion. This is true and can be heard at 57 seconds at the end of the Ladysmith Black Mambazo segment as it transitions to the Paul Simon vocals. This is much more obvious with closed headphones or a quiet sound room with low ambient noise. It's obvious that not everyone heard this anomaly though. I've gone back to my friend's place and confirmed that it's coming from the Whest phono stage (rather than the TMS turntable or phono cable). It serves as a good reminder of the importance of sensitivity to noise of an analogue setup. Remember that the Ortofon Cadenza Black is a low output voltage (0.33mV) moving coil cartridge (vs. 1.6mV Denon vs. 4mV Shure). This means there's greater amplification applied by the phono preamp and a concomitant increased risk of picking up hum in the system. The quality of the phono amplification stage therefore becomes extremely important. I cannot tell if this hum is unique to my friend's Whest unit (which is a number of years old at this point). For the record, vibration isolation isn't really an issue in this case because I was recording the needle drops without the speakers playing the music and the room was basically silent during LP playback.

Well folks... I hope you enjoyed listening to the Paul Simon LP samples. I'm still having fun with some used vinyl collecting. Lots of good stuff out there to collect and at exceptional prices as well. I know a few people have gotten into vinyl and have been disappointed with the sound. I think there's a lot of hype out there around this almost mythical "sound quality" of vinyl that some people will excitedly plunk a wad of cash down expecting audio nirvana to emanate from their speakers. Sure, if everything goes right, the sound can be excellent, but like everything in life, it's important to have realistic expectations. :-)

Hope you're all enjoying the music as we head into the Holiday Season... Thanks again for all the respondents; I obviously wouldn't be able to do any of these blind tests / surveys without your time and participation!

I have a new audio "toy" to play with over the holidays! Stay tuned...

--------------

ADDENDUM:
Last night I went to watch the movie Interstellar at the local megaplex. Wow! Neat movie; reminds me of 2001: A Space Odyssey with all kinds of sci-fi ideas - AI (robots with humor!), evolution of humanity, almost "divine" guidance, centrifugal artificial gravity and cryosleep. Elements of modern science gets thrown in: black holes, wormholes, Grand Unified Theory, space-time relativity, and genetics. Of course we must throw in the human elements: love, family, relationship, fear, loneliness, perhaps even madness. Highly recommended flick and one I'll be looking forward to on Blu-Ray.

But as reported elsewhere (here, here and here for example), the movie soundtrack was remarkably "loud"! Although I didn't have any issues with deciphering the dialogue (seriously, Matthew McConaughey's southern accent and lack of enunciation at times doesn't help), there were parts that were obviously rife with clipping distortion. I have actually never heard these levels of distortion to the point of hearing "pops" and "crackles" at this specific theatre before (with excellent sound setup including Dolby Atmos decoding). As I noted previously, the Dark Knight Rises soundtrack (another Christopher Nolan film with Hans Zimmer score) from 2012 was remarkably compressed for an orchestral score. I really don't know what the point is of doing this other than annoying the moviegoer! Loud is one thing which the theatre can accomplish by turning the knob up, but to the point of pushing the levels to "11" on the mix itself? That's obscene. (In fact, ironically I wouldn't be surprised if this theater turned down the volume if many viewers complain thinking it's their sound system crackling rather than inherent in the soundtrack mix. I believe I've heard louder transients in the theater before on other movies.)