Well, I guess I am flattered that the results of my recent Digital Filters Test got attention on AudioStream (Stereophile affiliate). The other day, Michael Lavorgna posted an entry entitled "The Trouble With Audio Tests" including a few quotes originating from my INVITATION to the test as well as my ANALYSIS posts.
I'm going to start today's entry addressing some of his thoughts on the matter and how I view tests like this.
First, let's just start with the quote from Ernest Rutherford (1871-1937) brought up by Mr. Lavorgna: "If your experiment needs statistics, you ought to have done a better experiment."
Well, it is a romantic notion isn't it that it could be so simple... That we don't need statistics to understand complex phenomena because somehow results end up being "black or white" or the conclusion just jumps out and declares itself. Perhaps in high school classrooms. Now I don't know enough about nuclear physics (to which Rutherford is known for) to provide any direct links, but I'd be surprised if some form of statistical analysis were not used in his famous Geiger-Marsden apparatus experiments to demonstrate alpha particle scattering, making sense of the empirical data, probabilities of deflection angles, and the margin of acceptable experimental error. Or whether modern particle physicists at CERN bother with statistics. But I do know that statistical analysis is essential in the biological and social sciences to make inferences out of observations of natural and social phenomena. It is acknowledgement that the natural world is "noisy" and that there is a "normal curve" to continuous variables out there in the "real" world. In research, we're trying to find answers based on whether a signal can be isolated from the noise. And this of course would include explorations into audibility and evaluation for the presence of "golden ears".
According to the post, Mr. Lavorgna was given the audio test samples in May as he states. He then did "one run-through" and "easily" picked out his preference for 2/3 samples. It now turns out he preferred 2 of 3 minimum phase samples and 1 of 3 for the linear filter.
I must say I am however disappointed! Why didn't you enter the results in my survey if you listened to these samples in May, Mr. Lavorgna? Remember, the test ended in late June (maybe you did but declined to indicate you were an audio reviewer?). Blind tests and claims of audibility are only good if one takes on the test and commits before the results are shown! You could have even added your subjective description as to what you heard so "easily", indicating strong confidence, and this 2/3 preference in the minimum phase filter could have pushed my results further towards an overall minimum phase preference for some of these samples. Alas, the opportunity is missed and what we end up with is a retrospective report - post hoc testimony.
He even goes so far as claim that he didn't even need to listen to the linear phase version before picking out the minimum phase one in the "second sample". ("So much so that I picked out the minimum phase filter in the second sample without having to hear the linear phase filter version.") If by "second sample" he meant the "GrandPiano" piece, recall that most respondents actually preferred the linear phase filter setting even to the point of a statistically significant p < 0.05 with headphone listeners! Impressive I guess that he was able to hear the difference so strongly. But also consider what this means if an audio reviewer's preference seems to be different from what most listeners seem to prefer...
I do agree with him that experimental results of group data may not apply to any one individual. It's not supposed to. But it does give us clues of trends and tendencies, what is important and what isn't. Sure, some unique individuals in a population will have remarkable abilities exceeding standard deviations of the "norm". These individuals have a "gift". Does Lavorgna have such a gift of having "golden ears"? I don't know nor does it necessarily matter. He only has to be true to himself just as I have to be true to myself and try my best to avoid my own preconceptions as best I can. This is just basic insight into human limitations and imperfections of the ear/mind mechanism. [Note that in this case, since the samples are all 24/176.4 and level matched, an honest ABX test with the foobar plugin will show just how "easily" the samples can be differentiated.]
What I can say is that out of 45 people who spent their time evaluating the test samples blinded (whom I wholeheartedly appreciate), I was not able to find a cohort who consistently selected preference of one filter type over the other (signal) beyond what was expected by chance (noise). The only significant result I could find was a preference for linear phase filtering with headphone users for one of the test samples, and that perhaps there is value in using the minimum phase setting with speaker listeners for 2 of the 3 samples based on subjective preference. This is all in the context of filter settings that exaggerate the amount of ringing due to an extremely steep "brick wall"! [That is, the vast majority of digital filters in use do not have close to this amount of ringing and I would expect audibility to be even less than this for most DAC filters.]
I certainly do not think the "answer" is at all "wishy washy" as Lavorgna claims. This is I believe a demonstration of reality, and a limited glimpse of the "forest" rather than individual trees to perhaps generate other hypotheses and further exploration. This is what you find when subjects are put to the test, when audibility between samples exist at the threshold of acuity. Not the remarkable "slam-dunk" subjective descriptions of "obvious", "clear" or "easily heard" comments made by almost every reviewer doing sighted evaluation as if "golden ears" were ubiquitous in the world.
One final comment before I end this segment. I do appreciate Mr. Lavorgna for admitting that he picked the linear filter in 1/3 samples. It's a demonstration that even if one believes there is an obvious difference, there could still be preference for the other filter setting in certain situations. Hence based on this empirical test, I would support DACs having switchable filters with both linear and minimum phase settings for audiophiles to choose their preference and do their own experimentation. Note that I'm not saying this should be a necessity since I personally feel the differences are at best subtle for the majority of listeners using typical filters with less ringing than the ones tested. And I think it would be presumptuous for any manufacturer to claim there is such a thing as an ideal filter setting to listen with. "Wishy washy" or realistic respect and appreciation for the subjectivity of human experience?
Mr. Lavorgna gave me a quote to start off discussions. Let me close with another I had mentioned previously courtesy of Karl Popper (1902-1994):
And who shows greater reverence for mystery, the scientist who devotes himself to discovering it step by step, always ready to submit to facts, and always aware that even his boldest achievements will never be more than a stepping-stone for those who come after him, or the mystic who is free to maintain anything because he need not fear any test?... All mystics, as F. Kafka, the mystic poet wrote in despair, "set out to say... that the incomprehensible is incomprehensible, and that we knew before."
(from The Open Society and Its Enemies, 1945, chapter 24 - Oracular Philosophy and the Revolt Against Reason)
I do not want to end this post with just philosophical debate. Rather, let's be practical and discuss a digital filter setting that I have been using over the last year for all my audio down-conversions.
I suspect many of you like myself may be a bit disappointed by all the hype around 192kHz samplerate material. Sadly, the majority of the 192kHz albums I have bought clearly have no "desirable" content at all in the ultrasonic frequency range (ie. most of the time the last octave is just low-level noise if anything). And they clearly don't sound any better than a lower samplerate version. Since I don't like to waste storage space, I routinely will downsample these to 96kHz if it looks like there is significant natural-looking material above 25kHz, down to 48kHz if it looks like it may have originated as that, and 44kHz if it looks literally like something upsampled from CD resolution. Even though disk space may be cheap, I'm still against wastage.
For a moment, let's not argue if we need anything more than 44/48kHz :-).
Suppose then, I want to down-convert 192kHz music to 44kHz, what filter setting would be good?
I've been using the excellent iZotope RX 4 software package for the last while. It's an excellent toolbox for experimentation and tweaking. I have been using the following setting for the lowpass resampling filter:
Pre-ringing at 1.0 means this is a linear phase setting - no phase shift, which I believe is preferable for the audio file itself (especially since it could be played back with a minimum phase filter DAC that would further distort phase). Filter steepness of 30 is moderate and while there is a little bit of aliasing, it's down at -28dB or so and above 20kHz. Finally the "cutoff shift" is down to 96%, lowering the passband slightly but reducing the amount of aliasing.
I find this setting works really well for me intellectually and it sounds just great :-). The reduced steepness correlates with less severe ringing / energy smearing at the Nyquist frequency. Pushing the cutoff shift down reduces frequency response only above 20kHz and inaudible for adults (and likely for everyone).
Impulse response for such a setting (using a 192kHz impulse downsampled to 44kHz):
And the spectral frequency display:
|Spectral frequency display - Audition 3, Blackman-Harris windowing, 128 bands, 100% window width, 132dB range.|
A similar setting if you're using the free SoX software would look like this:
sox.exe "Input_File" "Output_File" rate -v -a -b 91 44100The -v flag is for "very high" quality accuracy, -a allows a small amount of aliasing, and -b 91 sets the passband to 91% or about 20.1kHz. For 48kHz, this would be pushed up to over 21.8kHz. Anyhow, a suggestion that works well for me, YMMV :-).
|Frequency response of the SoX parameters downsampling 24/192 white noise to 44kHz - frequency response only rolls off over 20kHz.|
I guess we'll see what happened to the Ars Technica / Randi ethernet cable test soon... (AudioQuest Vodka teardown now posted.)
Time for summer holidays and a few weeks off with the family. Until next time... Enjoy the music!
:-/ Mr. Michael Lavorgna wrote this "lovely" thing: http://www.audiostream.com/content/blind-testing-golden-ears-and-envy-oh-my#gEa97M7Gj0D0EiP3.97 for me that was the end of a otherwise reputable website.ReplyDelete
I'm sure he's a really nice person but he needs to level up if he wants to show "the finger" to objectivists, that's why I enjoy M. Fremer so much, he also shows "the finger" to vinyl haters but with a lot of class, I respect that. XD
Anyway Archimago thank you for your blog, for me its a reference just like NwAvGuy blog was. You are up there with Tyll Hertsens in terms of quality, Tyll is the perfect reviewer, THE benchmark for all others...XD...
I'm being subjective now..OH MY! Soon I'm gonna hear differences in CAT cables and use magic stones...XD
Ok now I'm just trolling...but you got an amazing blog Archi it's a true pleasure to read.
Rafael Lino Aka Journeyman
Great to see blogs bring with them the personality of the writers... And how concepts are approached in different ways. I agree, from my perspective, Tyll is the best of the Stereophile crowd!
Well, i don't wanna throw stones at anyone, but i like Mr. Tyll too. He is subjective but tries to find a explanation in measurements about what he is hearing (like the Philips Fidelio X2 case, where he suggested some super cool and difficult books about headphones acoustics).Delete
As for Audiostream, i like the site, but as i said to Mr. Lavorgna, i would like to see more audio reviews and tech articles of how to take the best of our gear, instead of all that attacking position.
In my first post in Audiostream i wrote about my fear of the "angry crowd with pitchforks" that want to "kill" everyone that has na contrary opinion. Sometimes i feel sad because it seems that the "angry crowd" is recruting more members with success...
Anyway, lets play some music and enjoy the happy things of life!
Best regards and have a nice weekend!
Raphael -- you say Fremer has class? Then why did he write this about Randi: "WHAT A LAME LITTLE NAVEL GAZING EGOMANIACAL WEASEL!" Regardless of the reason, that ain't class. The prima facie evidence hints more at it being psychological projection.Delete
You refer to "vinyl haters", but that is poor form to vilify other people. "Vinyl haters" is the sort of language Fremer The Classy (see quote above) would use to denigrate people who disagree with his indefensible and evidence-free statements of 'fact'. The people being so denigrated rarely if ever HATE vinyl, in fact many of them quite happily use vinyl. The reason that they are being denigrated is for not accepting bold-as-brass claims of the innate superiority of analog and vinyl music recording formats over digital, of the magical joy literally claimed to be 'in the grooves', of the asymptotic way the sound of more and more advanced digital 'approaches' the sound of a vinyl record, and slowly becomes 'acceptable', yet it is intrinsic to such claims that digital will *always* never exceed vinyl, any more than an impersonator can exceed the impersonated. These factually erroneous claims are always presented as fact, and defended without end, all factual evidence of the error dismissed without evidence or argued ad infinitum, and on and on and on. As frustrating as this scenario must be for the 'anti-anti-digital' commentators, it is they who are held to saintly standards of behaviour, whereas Fremer The Classy is described as classy after he writes such words as quoted above. If you don't want to be part of the problem, there is no place for simplistic name-calling.
Really!. If Fremer is the exemplar of 'lots of class', the audiophile punditocracy is in even worse shape than I thought.Delete
This comment has been removed by the author.Delete
"Why didn't you enter the results in my survey if you listened to these samples in May, Mr. Lavorgna?"ReplyDelete
Because I saw no value in doing so. Someone I respect asked me to listen to the files so I did. I reported my findings to him on May 28.
"But also consider what this means if an audio reviewer's preference seems to be different from what most listeners seem to prefer..."
This is but one easy example of why I do not take you seriously.
"This is but one easy example of why I do not take you seriously."Delete
And likewise your numerous claims without (IMO) adequate self-reflection, lack of curiosity, inability to demonstrate said claims, intolerance of dissent, and inability to articulate in active civil debate remain astounding!
So we share the love, excellent ;-)Delete
I've been troubled by my response here and realize that this clash of personalities is rather childish. So I'd sincerely like to apologize for my behavior. While we may disagree about many things related to hi-fi, these disagreements are not rooted in anyone acting out of ill will. imo.
Going forward, if I decide to comment on anything you write, I will do so in a respectful and professional manner.
On a personal note, where you first crossed the line with me was in suggesting that my reviews are essentially nothing more than paid adverts (I'm paraphrasing from comments you made on my site so please correct me if you feel I'm misstating what you said). I take my job very seriously.
Appreciate the response Michael.Delete
Likewise, although our philosophical viewpoints are clearly different, I mean no desire to promote any form of personal attack from my end. As in the countless discussions of this sort on-line, I know how easily things get out of hand. I as well aim for calm dialogue rather than heated arguments.
I apologize for any comments to the extent of reviews being paid advertisement.
Best regards. Have a great summer.
Kudos for both of you!Delete
Some point of view in hi-fi can sometimes trigger some flamed discussion!
I'm very happy to see that we have two reasonable persons here, especially since i enjoy very much both sites! I already said that here and in Audiostream!
Best regards and a nice weekend to everyone!
"On a personal note, where you first crossed the line with me was in suggesting that my reviews are essentially nothing more than paid adverts (I'm paraphrasing [...]). I take my job very seriously." -- Michael LavorgnaDelete
Since this fits to An Honest Liar:
So do faith healers, palm readers, psychics, ...
The problem is not a lack of seriousness, but a lack of ability to question what you are doing and how you are doing it.
If you ever wonder "Why the heck did he say that?" feel free to send me an email (email@example.com) or comment directly on AS (unless this becomes an overwhelming task ;-) Some day a phone call might go a long way to clear things up.
Enjoy your summer.
Thanks. Will do!Delete
With the rise of computer-based audio systems it's unfortunate that The Enthusiast Network family of online audio publications has not put more effort into creating high quality content for their AudioStream site. Publishing products from Synergistic Research as "Greatest Bits" - seriously? It does provide a great advertising platform for Audioquest and other high end audio manufacturers though, often doing little more than republishing content from the manufacturer.ReplyDelete
Fortunately there are much better sources of information on computer audio such as Computer Audiophile - and this blog!
Like with anything, I think it's important to adjudicate what is of value and what is not.Delete
I'm thinking about the article on Computer Audiophile on the Windows Server 2012R2 "Audio Optimizer" software for example... That's one review I would read with more than a "pinch of salt"!
I have been reading through your blog, and I would like to see a comparative test of listeners and their reaction to overly audio compressed tracks. Many of our favorite albums have been audio remastered. Often times to the detriment of sound quality, as they have had their loudness levels elevated. I'd be willing to bet that in a test similar to those you have run on digital compression, that the listeners would very clearly be able to tell the difference. There are many versions of tracks that have been remastered over the years, and if you could select a variety of those through their various remasters, it would be interesting.ReplyDelete
Interesting idea. Indeed it could be fun to put together a small sampling and see how people rate.Delete
It's going to be quite clear of course which is which (just by the fact that volume changes are obvious) but just to get a poll going it would be fun to see how many would prefer the more compressed remasters... Certainly, I think for some remasters, more compression that's not too severe would not necessarily be a bad thing especially if they went back to the source tapes and used modern, more accurate ADCs compared to "first pressings" from the 80's.
The question is, what tracks should we select? Any ideas, folks?
1) I am new to reading your blog, and found it in the last week because I was also interested in the Tascam UH-7000 you measured in January, since I would also like to do similar types of tests at home. So, forgive me in advance if I say things that you may have blogged previously, until I have time to read more of your earlier posts.ReplyDelete
2) By way of background, I have enjoyed reading many of your musings, and have enjoyed listening to a number of well-recorded, standard CDs (16 bit, 44.1 kHz sampling) as well as higher resolution recordings.
3) Your test attempts to discover if one or another upsampling method significantly affects the playback quality of initial selections at standard CD quality. I am not surprised by the mixed results that seem to indicate that the upsampling method is not that significant.
4) I think a more interesting test would be to start with well-recorded samples of specifically selected music at 24 bits, 176.4 or 192 kHz, and to test what effective bandwidth, bit depth, noise shaping, etc. significantly degrades the quality of the sound for well-trained listeners in a double blind experiment.
5) I would love to have concrete results from such an experiment. My hypothesis would be that some “simple” music would not have significant degradation from high resolution to standard CD quality, if care is taken. On the other hand, I would expect high dynamic range music like a symphony orchestra with brass and percussion for transients and simultaneous lower level signals that may be masked by poorer effective bandwidth, bit depth and transient response would be easily distinguished from the original recording by symphony conductors and others familiar with the sounds of the instruments.
Thanks for the comments! Many good points there and I think definitely worth studying. A major point you bring up is with individual differences; such as trained listeners and symphony conductors. I wonder what the studies with these populations look like. Specifically, just how "good" can human hearing really be? Truly, the folks with genuine unique "golden ears" - how good is it and can we use these gifted individuals to define the upper boundary of technical parameters that we need...
If you come across some papers addressing this or have some data from your own experiments, please do drop me a note!
Let me be clearer about the training I am talking about. A person that mostly listens to MPs and intentionally distorted music needs to be trained to hear what live acoustic music (that would challenge a recording system) sounds like before determining which recording sounds better in an experiment. If participants do not have a memory of what live music sounds like, they can dilute results, especially as the A/B differences diminish. Training might be attending a few of the right sort of live concerts for anyone that does not have the memory already.Delete
As expected ... no discernable difference in ethernet cables:ReplyDelete
Of course the test + used 'components' and unknown testers (only 7) will never convince the ones claiming differences are 'really' there.
Any audiophile will tell you the used test equipment 2x Grado RS2e directly ? out of a DELL laptop using Win Media player won' t 'resolve' the, to them, so obvious differences on high end gear.
Yeah... This is problematic and I do not believe this will impress / convince anyone.Delete
Ars basically pooched the entire thing soup to nuts. Poorly thought out. The only thing worth mentioning is the Fluke analysis.Delete
The same Rutherford once said "All science is either physics or stamp collecting", then went on to win the Nobel Prize in Stamp Collecting (known in less esteemed quarters as chemistry). Anyway, his statistics quote was made as a pre-quantum physicist, and he was speaking to all scientists, i.e. physicists only (if you get my drift).ReplyDelete
Point being, he was a heavyweight as a scientist and a lightweight as a philosopher, perhaps even a joker.
Mr Lavorgna seems to have been on a crusade lately to discredit the scientific method, scientific testing, and, by association, science itself. He has put himself out there, not only in his own forum, but in the forums of some key people who like to discuss audio (and audibility) rationally and sensibly. Unfortunately, he is doing it as a missionary, not as a co-learner. A teacher and a teller, not a listener and a learner. While presenting as suave and sophisticated, one need not look far to see what lies underneath: disrespect. He let the truth out in his first comment on this article, "I do not take you seriously", and promptly tried to paper it over for reputation's sake, but it's everywhere and easy to find, starting with his blind testing article: the sneer.
Mr Lavorgna's fundamental missionary position on blind testing can be found in his article on that topic: "....listening tests of hi-fi gear at best tell you about the listening capabilities of the people taking the test under those specific testing conditions." Whereas the alternative, sighted tests, what do they tell us? Apparently they tell us all about the hi-fi gear itself! And now we see the sad hypocrisy: perform a cold rational critique of the method you wish to discredit (and that's fair), while maintaining an uncritical golden faith in the method you wish to endorse. Dear Sir: LOL! No sale!
This comment has been removed by the author.ReplyDelete
I think it's important to always try to look for the motivation behind someone's behavior. In this case, it's pretty easy to guess. Just look at the main advertisers on his site and you'll see familiar names like AudioQuest. Lavorgna writes glowing reviews on $1000+ AudioQuest ethernet cables. Coincidence? I don't think so.ReplyDelete
Adhering to the scientific method of testing (both using equipment and proper listening ABX tests) would cause him to have to expose the fact that much of the equipment advertised on his site is snake oil. And Audiostream would be out of business. So he is therefore on a mission to discredit any process that would expose the bogus claims of his advertisers who are paying his salary.