Saturday, 9 September 2017

MUSINGS: Evidence of digital player jitter with asynchronous USB DACs? Melco N1ZS20/2 review in HFN&RR...


The UK magazine Hi-Fi News & Record Review is interesting. Like Stereophile here in N. America, it includes objective measurements. Over the years, I've seen times when they have criticized questionable "hi-res" downloads showing nothing more than upsampled music. But they also seem to like "showing" rather meaningless measurements such as a few years ago with their USB cable roundup.

Recently, I believe in the May 2017 issue, they reviewed the very expensive "cost-no-object" digital server - the Melco N1ZS20/2. This device is the "mk2" of their top-of-the-line music server/player with the capability for data output through ethernet or USB connection to one's DAC.

Let's get the "elephant in the room" out of the way - yes, it costs... a lot. We're looking at a device that's listed as starting at £2099 and goes up to the £7700 top end which is what was reviewed and measured. That top end machine is about US$10,000. That's 5-figures for a low-power computer server (they don't say what CPU but I presume a fanless ARM, nor mention what amount of RAM is inside; BTW you can see the "stripdown" of their lower end first generation N1A here). For the price, there's no built-in DAC, with 2 x 1TB SSD drives.

As I have expressed in the past, non-utilitarian functions cost money. I'm totally OK with any company asking an arm and a leg for a product - fancy cars, nice handbags, beautiful clothes, jewelry, watches... A $10K low-power computer server - why not!? After all, it's got "heavy aluminium extrusions, bead blasted to a beautiful anodised finish", weighs 19 lbs, has "a single press power button", "comfortable front panel buttons with positive click for easy operation", "OLED display to reduce internal noise", "TAOC vibration isolating feet", among other "interesting" features and "desirable" talking points. Also, 1TB SSDs are still pretty expensive although one should be able to get a couple of Samsung 850 EVO 1TB drives for around US$650 these days. Let's just say I've got some doubts about firmware hacks to make SSDs "Audio Grade" - LOL! :-)**

However, what I think bothers many reasonable audiophiles are claims about the utilitarian improvements offered by these kinds of devices - the ability for actual improvement in hi-fi sound quality. There's a desired impression that the cost can be justified by "better" sound, presumably a significant and noticeable difference.

As I mentioned above, HFN&RR is an interesting publication that posts objective results. Have a look at this review of the Melco; freely accessible from the Melco website by PDF. In it, there are claims being made which really deserves some exploration and thought.

First, feel free to explore the subjective review of course... It's the usual golden-eared commentary about how this is the very best. Sure. More "clarity and focus", "greatly improved", "better slam and drive" to the bass frequencies, "drive and pace of the music is enhanced"... OK, so the reviewer thinks it sounds better than his "£200 secondhand Mac mini".

What is much more interesting are the objective results, supposedly proving change in jitter performance and noise level with some DACs compared to a "standard PC" via USB. They show some numbers in the "Lab Report" section, but much more visually interesting are the graphs. For reader convenience and since the PDF is freely downloadable, why don't I reproduce the graphs in the article here for the purpose of criticism and discussion?


This first graph is of the Melco connected to the iFi iDSD BL; a ~US$450-550 DAC that's really meant for desktop headphone listening and RCA outputs only. As labeled, in red is the DAC connected to a "standard PC", and in black the Melco. It shows that there's more noise in the background and noise spikes with the "standard PC" connection and more "skirting" of the base of the J-Test peak at the primary frequency to indicate more low-frequency phase noise/jitter. They claim that the jitter difference showed a "marginal improvement" from 105ps to 140ps in the summary table. Notice that there are some symmetrical, probably jitter-induced sidebands that actually look to be of similar amount between the "standard PC" and Melco. Other than that single red noise peak in the "standard PC" graph at -110dB very close to the primary signal, everything else is below -120dB - including the fact that the "skirting" differentiates only below -120dB... Improvements worth US$10K?

Then there's the second graph:

OMG. That's ugly! Supposedly it's a reflection of the "standard PC" vs. Melco through USB to the Chord Mojo DAC on the 24-bit J-Test. At face value, the first thing this graph tells me is NOT that I should buy the Melco. It's that I should AVOID the Chord Mojo (~US$550)! Seriously, if the difference between the "standard PC" and Melco is so small for the iFi iDSD, what's wrong with the Chord Mojo such that it's so terribly sensitive to noise (not necessarily jitter) that it should give such a result? Especially bad considering this is supposed to be battery powered!

Permit me to make a few (obvious) critiques of these results:

1. We don't know what is a "standard PC". The article says nothing about whether the hardware used is a simple modern i5 laptop, a recent Intel NUC maybe, or some kind of monster i7 CPU with 1000W switching power supply and dual GTX 1080 Ti graphics boards. For all we know this could be an old Mac running Parallels with iTunes in a Windows Vista virtual machine!

2. Related to the above, we know nothing about this "standard PC" playback set-up. Did they use iTunes? foobar? JRiver streaming off the Melco over UPnP? Did they use a toss-away crappy unshielded USB cable (like this one) or something more substantial? Did they even bother using ASIO or WASAPI drivers properly?

3. We also know nothing about the physical layout of the tests... Did they at least take some caution with placing the Chord Mojo a few feet from whatever "standard PC" was used? For all we know, they could have sat the DAC right on top of a beefy switching power supply soaking up the EMI!

Here's my belief... The Chord Mojo is NOT this poor and that 2nd graph either is wrong, mislabeled and/or caught the eye of and put there for effect by an editor who didn't know better. Stereophile's measurements already show us that the Mojo has good low noise and jitter performance; obviously Stereophile did not need a $10K Melco server/player (2012 MacBook Pro). Rather, I think the guys doing the measurements here must have made a mistake. They could have used an unreasonably poor "standard PC" hardware, neglectfully placed the Mojo too close to noisy components, or might have been sending 16-bit data rather than a true 24-bit signal (dithered DirectSound output?), or if streaming to JRiver, they forgot to keep the stream at bit-perfect 24-bits. Another possibility is that there was a ground loop in the system causing noise and hum resulting in that awful noise floor difference. Maybe the dB scale is off between the "standard PC" and Melco? The text descriptions of S/N improvement and reduction in jitter were not this extreme as on the second graph! Something is seriously wrong.

Having said this, IMO, the "evidence" presented in this article is worth considering around the potential of jitter being affected by the computer/server used when connected to an asynchronous DAC. I believe that if HFNRR wanted to seriously convince of the merits of the Melco objectively, they should have been more forthcoming with the hardware used for the PC. Remember that in science, this is important because reliable measurements that can be reproduced independently by others are part of the checks & balances of the process. Unlike subjective commentary, you can't just claim something without clearly informing the reader of the parameters surrounding the test/experiment/measurement. In any event, I'll certainly keep looking when I do measurements to see if I can detect changes as described by this HFNRR article.

The question in my mind though is ultimately this: suppose we accept the findings of the iFi iDSD (like I said, there's something wrong with that Chord Mojo graph so I cannot take that seriously unless verified), do we actually believe the jitter difference is audible with real music? Will a spurious noise spike up to -110dB with the "standard PC" and that "skirting" below -120dB make a difference in the way the subjective reviewer describes what he heard (clarity and focus, bass slam, sense of "drive")? Do we even have many 24-bit recordings with such low noise floors? Let's also not forget that the J-Test artificially stimulates jitter anomalies for testing purposes to show tiny imperfections without making claims as to audibility. (Needless to say, these questions are irrelevant for vinyl lovers due to inherent resolution limitations.)

I know different people will have various perspectives on the questions above... But personally, I'd be very reluctant to believe that human physiology has this type of ability to resolve the difference between a calculated 105ps and 140ps jitter documented for the iFi DAC between their Melco and "standard PC"!

Remember, since the Melco is essentially a low power computer server at its processing heart, IMO they should have shown us what the jitter spectrum looks like with something like an inexpensive Raspberry Pi 3 + external HD enclosure (like this Startech with RAID0 and fan speed control) + dual Samsung 1TB standard SSDs. All that can be built for easily <US$1000. Throw in Volumio or RuneAudio for DLNA/uPnP or piCorePlayer and turn on Logitech Media Server if you prefer. Simply use a decent 6' USB cable. Good luck with measuring the jitter and noise difference between this and the Melco, I think.

The lab report actually stated:
"Ironically, it is the more rudimentary USB hub-powered DAC/ headphone amplifier solutions – as opposed to high-end USB DACs with integral power supplies, etc – that provide us with the best indicator of incoming data integrity and noise (or lack of) on the +5V supply."
What is so ironic about this? If you have to dig into measurements with DACs meant for headphone and battery-powered mobility output in order to show an effect, isn't this good evidence that a device like the Melco makes no difference for actual higher end hi-fi system meant to be experienced in a nice sound room with full-range speaker reproduction? These DACs don't even have balanced output for the lowest noise floor possible (clearly you'd want this with the Melco!).

In summary, any audiophile with $10K to purchase a 2TB low-power computer server should at the very least have the common sense to buy a contemporary Chord DAVE (also about US$10,000) DAC rather than the Chord Mojo :-). They would look fantastic together on the audio rack and together synergize to express the ultimate "non-utilitarian function" of audio hardware as art, furniture, jewelry and of course science. I'm sure we'd all be interested in seeing the jitter differences between a "standard PC", Pi, and Melco feeding the DAVE :-).

** For some more Melco-related audiophilia hysterica, check out their "Software" link. "Purification" of ethernet data with their direct port (!?); they make it sound like accurate data transfer is some kind of difficult thing these days (remember my UDP/TCP test awhile back). Oh yeah, apparently "LAN lights can be disabled for highest possible data integrity" - thanks Melco! No pseudoscience or FUD here. :-) IMO this is shameful considering the respectability and quality of Buffalo's business and consumer NAS products. Perhaps an audiophile hobby where the "mainstream" tolerates this kind of rhetoric deserves to be treated like fools by a computer company selling a souped-up NAS...

--------------------

Anyone see new Sonore micro/ultra/Signature Rendu (SE) measurements? I notice that they have a full line-up of devices now featuring all kinds of stuff including various linear power supplies... So, how noisy are these things and any "evidence" like the Melco here of jitter effects with asynchronous DACs?

HFN&RR / Paul Miller / Andrew Everard / Jim Lesurf: Since you guys have already got some measurements with the Melco and "standard PC", why not compare devices? Melco vs. "standard PC" vs. Mac Mini vs. MacBook vs. Raspberry Pi 3 streamer vs. Sonore *Rendu? For the benefit of audiophile consumers, would it not be good to do some investigative journalism independent of device reviews? Would it not be nice to know whether claims are true and to what extent when it comes to jitter with ubiquitous asynchronous USB DACs these days?

Likewise maybe Stereophile would be interested in verifying the results? Again, an independent article exploring measurable differences between devices would be a return to previous form (like this article from a decade back)... For audiophile magazines to provide honest independent explorations into these issues would certainly go a long way to reassure readers that they are not just the advertising arm of the "high end" hardware industry.

Let's be honest though. In 2017 with the decline of news media in general, and perhaps especially with the diminished size of the hi-fi/audiophile hobby, the "bottom line" more than likely is dictated by retention of advertising revenue. Suppose a respected audiophile journalist goes through the science, measurements, controlled listening tests and writes a magazine article concluding that jitter is minimal these days and inaudible. This would take the hype off a certain segment of cable, computer server, streamer, maybe DAC advertisers still trying to convince audiophiles of the "importance" of their jitter performance. Maybe audiophiles will realize they can safely ignore fluff talk about jitter altogether, or more specifically "gee, I don't need a femtoclock upgrade!" Could the current cohort of audiophile magazines tolerate such an outcome? Would such an article ever have a chance to come out in print through the editorial process? What would happen to the reputation of said respected audiophile writer at least within the inner sanctum of the mainstream audiophile press?

Hhhmmmm...

A big "THANK YOU" to those who submitted their results for the MQA Core Decode vs. Hi-Res Audio Blind Test! As promised, I have closed off the online survey yesterday (September 8). In total, there were 83 submissions from around the world (after taking out 2 that were resubmissions)! I do believe this is the largest listening test you're going to find for MQA with any amount of serious attempt at blinding the "subjects"... I have started the process of initial analysis of the data and will present the results over the next little while. Stay tuned - let's see what the data set tells us about audibility and preference of MQA Core encoding/decoding in the real world tested by actual ears/minds in actual home systems...

Enjoy the music everyone!

19 comments:

  1. Well done Archimago. I guess the proper scientific question would indeed be to see if it sounds better than alternatives like the Raspberry Pi or the Chromecast Audio, compared in a proper double blind set up. I think I can predict the answer. So such research serves no scientific purpose because the answer is so predictable - it only serves a cultural one.

    ReplyDelete
    Replies
    1. True Willem, the proper question is whether it "sounds" better.

      Doesn't help that we're not likely to ever get a proper answer from audiophile magazines because they don't believe in controlled listening tests...

      Yeah, a "cultural" purpose :-). To sustain a "cult"ish subgroup?

      Delete
    2. Hello, they do not "believe" in controlled listening tests, because they cannot trust their ears!
      Simple as that: They don't hear nothing about sound quality.
      It would be too revealing, if in a controlled listening experiment, those self-appointed "trained experienced ears" could not tell no difference between the cheapest consumer grade equipment and the praised expensive "hi-end" gear.

      Delete
  2. $7,700, nice figure.
    That would be like the combined cost of well equipped, acoustically treated room and high grade recording equipment.
    To think one would pay that just for the server...when an Odroid XU4 would do essentially the same task.
    Yes, actually it fits with the priorities of most subjective folk, sacrifice ANYTHING for every minute "improvements" in SQ, regardless whether are those audible or not.
    This reminds me a discussion at head-fi for Fiio's X5 3rd gen.
    For marginally better audio parameters, which are hardly audible, some folks sacrificed battery life, dismissed native Opus and WavPack support, agreed on extra weight and size, pardoned bad mechanical design and defamed all those who shared a bit of skepticism.

    To be honest, I no longer care about whether something is audiophile grade or not, as long as there aren't any obvious problems with the product. Music matters more to me than sound, plus I consider my time better spent on active listening and making music than playing mr. unhappy audiophile and stack my apartment with various audiophool products.

    My 2c ;)

    ReplyDelete
    Replies
    1. Don't forget SUBIT, that's *£7,700*. As I type, that's 7.5 ounces of gold, baby, or about US$10,150!

      Indeed, that's some excellent studio system or maybe an excellent pair of beautiful hi-fi speakers.

      The 8-core Odroid XU4 with 2GB RAM will I'm quite sure run circles around the CPU in the Melco. The fact that they're not touting it as being fast almost certainly implies that there's nothing to write home about inside there.

      Thanks for the anecdote about the Fiio X5iii. But then again, I don't the point of owning a DAP since I have a phone :-). Much less care about "hi-res" audio when walking about!

      Delete
    2. Yes I know, only Google won't let me correct the typo, of course 7,700 quids.

      Personally I use Sansa Clip Zip and Clip+ with latest Rockbox and honestly couldn't be happier. Wanted to use phone as well, but the TRS jack is located on the top of the phone, and cable handling thus causes too much strain on the socket.
      Also I couldn't find one single app for music playback that would tick all checkboxes for me. Most of them lack even basic transport controls such as jump to time or stop button. Also none of them offered basic visualization tools, such as FFT display, oscilloscope, XY graph or phase. Heck, I couldn't find ONE with VU meters.
      Rockbox has FFT, oscope and good log VU (unfortunately no LUFS or RMS) and also allows adjusting volume by 1dB increments. Android smartphone jumps 3-5dB per step.


      Delete
  3. Alas, this is not really a surprise. The rot set in at HFN & RR back in the eighties, around the time that John Atkinson became editor. There was a clear policy of dumping the previously rigorous approach to equipment reviewing and replacing it by subjective judgements based, it seemed, on little else but the price ticket.

    ReplyDelete
    Replies
    1. Thanks for the note Roderick.

      Interesting background. Atkinson or not, I suspect it's ultimately about technological maturity of gear and the crossing of the threshold beyond which human perception can no longer audibly differentiate... This happened with the advent of the CD which would be right in the heart of the 80's.

      Maybe this is why subjectivism (aka "imaginary story telling" projected as generalizable truth) took over. They started realizing that there wasn't much left in the digital world to measure and probably was getting a bit frustrated by the technical superiority of digital over the analogue stuff.

      I'm sure there's a good history/sociological research paper in there somewhere :-).

      Delete
    2. Arch, I received a note from Peter Aczel, RIP :-( that sheds some historical insight from his perspective:

      "As for the longtime war between the objectivists and subjectivists, it was not inevitable, as I have frequently stated. It could have been nipped in the bud in the ’70s if the scientifically qualified technologists had bothered to attack the unqualified extremists instead of dismissing them as unimportant. I remember begging the great Stanley Lipshitz and some of his colleagues to do something, but they just shrugged. A whole multibillion-dollar industry of voodoo audio has been the eventual result. The gullibility of audiophiles has no limits."

      Delete
    3. Thanks Mitch.

      Indeed, RIP Mr. Aczel (June 22, 2017 at a good 91 years). He certainly was one of the few rational voices back in the early 2000's when I was getting into audio. Those Audio Critic articles available online provided great reading!

      Thanks for sharing that E-mail, Mitch! I wonder what the hobby would be like in 2017 if indeed over the decades there were a more concerted effort on the part of science and objectivism.

      Quite possible of course that it would make no difference... After all, once technology had reached beyond the audible threshold, the science guys probably went on to other things and let the advertisers tell stories. Money needed to be made; the essential life blood of any venture.

      Delete
    4. Archimago, your reply from 10 September 2017 at 19:17 nails it.

      With the matured electronic equipment, practically every CD player, every amplifier would re-produce the same sound quality, equal or beyond the abilities of human hearing.
      The only pieces of gear with audible differences were (outdated) analogue gear: Cassette decks, record players, video recorders (with hi-fi sound, remember head chatter?)
      And, of course: Speakers. And, room-correction, which now is integrated in AV receivers.
      So, these magazines could only tell about speakers. Speakers really make a difference. (Room corrections only came later than the 80s.)
      About the other gear (amps, CD players, ...), they would just need to state whether there are "bummers" like hum, noises, or other stupid manufacturing mistakes, like stereo volume potis not in sync.
      Instead, they choose to go into vodoo. What else could they do? They had no choice! I stopped reading these magazines when I read an article about a CD cleaning solution that made the vocals on some bar-jazz CD sound "fresher" and more realistic, or whatever. That was early 90's, and they already reached full bullshit standard.
      A few years before, such magazines provided usefull hints and tricks on about how to align your cassette deck azimuth and playback and recording gain level to "Dolby level".

      Delete
  4. Thanks Archimago for a great post! I totally agree that "random" measurements are no more objective than evaluations "by ear", and actually introduce more questions than they are intending to resolve.

    I noted that you have mentioned "low noise" balanced outputs. After reading Douglas Self's "Small Signal Audio Design" book, I'm not so convinced that balanced connections bring that much improvement for SNR. As it follows from Doug's evaluations, it's actually quite hard to design and implement a low noise balanced input (which is needed if you have a balanced output on the other side). So having balanced connections doesn't actually mean your SNR is the best--it depends.

    ReplyDelete
    Replies
    1. Greetings Mikhail...

      True. Not all "balanced" devices are created equal. However, even a relatively inexpensive PonoPlayer can be shown to provide a bit lower noise floor in balanced mode.

      But if we're talking $10,000 just for a digital player/server in the belief that this is meant to be technically the "best" system we're building, then might as well "pull out all the stops" with a fully balanced DAC paired with some dual differential amps. If every picosecond jitter makes a difference, so much more important every dB in lower noise floor, right :-). (Not to mention XLR cables also more robust and secure compared to RCA.)

      How much this actually adds to the sound in the real world is debatable of course; and I certainly would not argue strongly either way!

      Delete
  5. >the first thing this graph tells me is NOT that I should buy the Melco. It's that I should AVOID the Chord Mojo (~US$550)!

    Exactly.

    http://archimago.blogspot.hk/2017/07/measurements-mqa-filters-on-mytek.html?showComment=1499859290484#c3832567785115182963

    It would be very funny if you send this blog entry to Chord and get a verification about the validity about the test result, then eat some popcorn and see how these two audiophool companies fight against each other.

    ReplyDelete
    Replies
    1. Yeah, toss a penny between them and let them fight to the death :)

      Delete
  6. This comment has been removed by the author.

    ReplyDelete


  7. There are some measures of another DAC that is sensitive to the USB connection here:

    https://audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

    And more of another inexpensive device that isn't so sensitive.

    https://audiosciencereview.com/forum/index.php?threads/budget-dac-review-behringer-umc204hd.1658/

    The results of jitter from an iFi DAC 2 are shown for comparison with both of the above DACs.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Yes... But Behringer has a bad reputation of being "noisy"... And, Behringer is just too inexpensive ....

      Now, in earnest: That's why I would advice to look for musician's devices: Gear that's aimed at the semi-pro project studios.

      Of course, the "noisy Behringer reputation" is borne out of ignorance. They have some inexpensive reliable good products, just like a lot of other brands in that market.

      Delete