Saturday 15 February 2020

MEASUREMENTS & LISTENING: Topping D30 DAC by Greg Dunn. (And briefly, more USB cable nonsense.)


[Ed: Every once awhile, it's a pleasure to have guest posters adding their voice to this blog! There are many folks out there doing measurements, experiments, and I've certainly been in contact with a number of audiophiles with great stories about the gear they have, music they love, and the audiophile pursuit in general!

A few weeks back, Greg got in touch with me around the Topping D30 DAC that he has. This DAC was brought up in comment discussions back in 2018. The D30 measured quite well on Audio Science Review, but has been blasted as subjectively sounding bad - "like ass" as it were - on the Super Best Audio Friends forum. Elsewhere in that SBAF comment thread, the D30 was described as having "poor soundstage and poor timbre".

With this polarization of opinion, it's great to see Greg stepping up with his own opinion based on objective testing and subjective listening... Take it away, Greg!]

I recently had the opportunity to measure and audition several DACs varying from cheap to moderately priced, and learned quite a lot about my measuring system in the process. Not yet being ready to commit to (afford) a laboratory-grade setup, I decided to go with an inexpensive but capable external sound interface and see what I could find out about some of the electronics which I had access to. My choice, based on reviews, was a Focusrite Scarlett 2i2 (3rd gen) which seems to have quite respectable performance for the money. I realize I'm pushing its limits for some measurements, but since my goal was more to get a sense of how the equipment under test measured than to characterize it to ten decimal places, I’m pretty happy with the results. For vintage electronics and most modern stuff, it’s more than adequate.

I. Focusrite Scarlett 2i2 (3rd Gen) ADC Loopback

Unfortunately, even though it works perfectly with 24-bit and high rate audio signals, REW currently only displays down to -96 dB of relative input level which is a limitation in these tests (this will be changed in a future release).  I have done spot checks on the Scarlett and Topping showing good linearity well below these levels, but it’s harder to plot the data without an automated setup. I use 96k/24 bit mode for consistency between measurements and to ensure that there isn’t an artificial roll-off at 20 kHz which might bias the measurements.  Here's a plot, in loopback mode, of the Scarlett's frequency response. This is pretty uninteresting (good as a measurement device):



Next, the Scarlett distortion vs. frequency baseline:

Focusrite Scarlett 2i2 (3rd Gen), THD, noise level in loopback.
Distortion is well below 0.01% or better than -80dB with the input signal just below clipping. Noise is ever further down, by 20 to 30 dB. Not state of the art, but decent for an inexpensive tool and good enough to measure most of the equipment I want to look at (which is mainly speakers and power amplifiers). These numbers aren’t as good as other measurements of the Scarlett I’ve seen; it's likely possible to improve the performance of my bench setup with some effort. Since this is mainly for my own comparisons and to verify that my DUTs are working properly, I'm not too worried about not squeezing out the absolute best results from this setup. Maybe some day I can afford an RME Pro for testing, but not today. :-)

I recognize that most any really good piece of equipment will be limited by the Scarlett's limitations, so I do not claim the following numbers to be fully representative of the devices being tested – all I can say is that the device is at least this good, and probably much better.

For reference, the Scarlett 2i2 (3rd Gen) distortion vs. level:


If I had set this up so that the Scarlett did not clip, the distortion and noise floor measurements would have exactly paralleled each other.  In order to show the clipping behavior, I increased the gain on the Scarlett by about 10 dB for this measurement only. It was useful to have this info, so that we know the Scarlett is very clean right up to the point where its inputs are overdriven. For the Topping D30 measurements, the Scarlett's gain was adjusted so that it did not clip at all; any evidence of clipping would therefore be that of the Topping alone.

II. Topping D30 DAC


Now, on to the Topping D30 measurements. As you can see from the reviews discussing the DAC's features (such as this one), this device is rather flexible with USB input as well as both coaxial and TosLink SPDIF. Outputs are single-ended RCA.

First, let's have a look at the USB input frequency response:


Nothing to see here; ruler flat response to well beyond audibility.

Next, the distortion measurement (THD and noise) vs. frequency:


Much of this is limited by the Scarlett's internal distortion though there does seem to be a little added noise, more than likely below audibility.

Finally, the distortion vs. level at the Topping input:


It's apparent that the distortion (+ noise) decreases steadily as signal increases. Unfortunately, with the current version of REW it’s not possible to overdrive a purely digital input so 0 dBFS is the limit for testing (and neither device clips at this level). Again, behavior is limited mainly by the Scarlett’s input noise + distortion.

The optical TosLink input's measurements are so close to these values that there isn’t much point in showing them. The USB input might be 1-2 dB better at certain frequencies or levels, but for all practical purposes, results were identical.

III. Listening tests

Of course, the real purpose of this DAC is to listen to music through it, right? There have been tests which claim the D30 has an audibly different "sound" compared to other DACs, and even condemning it as sounding "bad". So ultimately, after finding that the device measured quite well and seemed to behave as a DAC should (within the limitations of the test equipment) the only thing left was to listen to it.

Sadly, I don't really have a double-blind test setup for comparison to another DAC, but I can still try my best to A/B compare using a switch box and a high impedance passive attenuator (to match levels). This will easily allow me to compare line-level devices, but there will be no easy way to randomize the selections. I was prepared to take more thorough measures in the event that there was an audible difference, but first it seemed prudent to try and see if such a difference existed.

For the test, which took most of an evening, I compared the D30 to my regular DAC, the SMSL M100. The SMSL actually measures about 10 dB worse in both SNR and distortion, though still quite good in overall sound. I found out when hooking them up that one device had higher output voltage at 0dBFS than the other, so I knew I was going to have to level-match them if there was any hope of achieving an accurate listening test. It turns out that the difference was on the order of 1.0-1.5 dB when connected to the stereo, so that needed to be dealt with using a test tone and a meter.  Unfortunately, unless the overall level was precise (volume control potentiometer linearity is not always perfect) there was still a slight difference in output between the DACs. Since sub-1dB differences are audible, I had to set the overall level with care to ensure essentially identical volume. This was aided by the in-line attenuator actually having detents on its rotary position so it could be precisely and consistently set, independent of the system volume control. Very handy as we'll see.

I tried both the TosLink optical and USB inputs for the DACs. Just to be clear, I did not notice any difference between the digital inputs which would affect the outcome of my listening test. Alas, I don't have a S/PDIF coaxial source to test that input.

I started playing music, comparing the two DACs. As it happened, I was using two identical RCA cables to connect the DACs to my stereo, and I actually forgot which pair was which after connecting them and putting the equipment all back in place on the shelf!  I decided this was fine, because even though I don't have the ability to easily randomize, at least not knowing exactly which wire was which provided a level of uncertainty to reduce bias.

In a nutshell, no matter what music I chose and which DAC was playing, I couldn't hear a difference! I tried to take my time and listen for several minutes at a stretch, as well as switching rapidly between DACs during parts of the test – which is also useful.  There are adherents to both methodologies, so I tried both – which did prolong the test somewhat, because I refused to rush the test just to get it completed. I tried rock, classical, and jazz selections; vocals and instrumentals; samples of the best recordings in my library. It didn’t matter; the two DACs remained, after the evening's work, as nearly identical as I could have expected.  I “thought” maybe I knew which one was which, based on the way I usually wired my inputs, but to be completely honest, it was totally a guess. Very interesting!

IV. Conclusions

At the conclusion of the formal test, I thought it was time to try something I had noticed in a blind test many years ago and which is mentioned but not talked about in many modern reviews. I bumped the stepped attenuator up one notch, making one of the DACs slightly louder (about 0.5 dB according to the meter). Instantly, one of the devices sounded “better” in a vague but consistent way – specifically, it didn’t sound louder, just different. I returned the attenuator to its original position and the difference vanished. Then I turned the attenuator down by the same amount – now the other device sounded better. The result was repeatable, with multiple iterations and multiple music types.  Yes, this was actually sort of a sighted test, but I still wasn't certain which DAC was which – only which input channel I was using.  In fact, when I unplugged them after the evening's tests, I purposely didn't note which cables went to which DAC so any sort of retrospective bias could be avoided. 

After the evening's tests, I came to two conclusions:

1) The Topping D30 (~US$120, Cirrus Logic CS4398 DAC chip) and SMSL M100 (~US$80, Sashi Kasei AKM4452 DAC chip) sounded identical in my system.

2) A very slight level difference of ~0.5 dB between the devices was immediately noticeable and significant in terms of perceived sound quality, at normal listening levels.

In my experience with this and another earlier test, my ears needed at least 1 dB of sound level difference before perceiving it as actual "loudness"; below that value, the louder device exhibited a consistent but vague "improvement" in sound quality, which correlates fairly well with other properly-conducted listening tests I’ve seen. The ear is an awful precision measuring device. Even though one of the DACs had about 10dB worse noise and distortion performance than the other (though still very good), that wasn't audible if levels were matched. Level differences totally biased the listening test results, as I had expected and was glad to confirm. If you can't match levels to the order of 0.1 dB, you’re likely introducing a bias to the test which can't be dismissed.

At the moment I'm writing this (actually before compiling all the test results) I still am not sure which DAC was which with certainty.  When I look at the objective data, of course, I’ll know for sure (because of the output level measurements) and that's why I'm writing this portion first even though it's at the end of the write-up. All I can say is that I'd be happy to have either DAC in my system; in fact, I have left the SMSL connected, in spite of its slightly worse measurements, because I like the digital display of bit rate on the front panel. It's as simple as that. Other than visual presentation, either one is satisfactory in my stereo and as far as I’m concerned, they’re both recommended.

--------------------

Thank you Greg for that excellent write-up which also included the loopback results from the Scarlett 2i2 DAC/ADC. The objective results of the Topping D30 certainly look good and I really like how you took the time to perform the listening tests! The fact that you were unclear about which input cable connected to which DAC is a nice twist on being "blinded" :-). Might be a good way to reduce bias if one can't get a friend or family member to help randomize for listening tests so long as levels were matched before hand.

Also, that qualitative comment about 0.5dB difference is an important one to keep in mind. Often we are told that a 1dB difference is the threshold of audibility for amplitude change. I agree with Greg that it's lower than that. Even if we cannot fully verbalize the experience, small differences like 0.5dB can be perceived as "different" and typically, a slightly louder sound of +0.5dB will be "preferred" as the better-sounder version in comparisons.

Obviously we must not disparage the importance of our ears and mind as the facilities from which we perceive the complex emotional tapestry of music. This is a fact to be celebrated. But as an actual "instrument", I agree with Greg that human hearing is understood to be far from a "precision measuring device".

I hope readers who might have strong beliefs in audible differences of high-fidelity DACs but have never tried a level-matched, preferably blinded, comparisons like this consider Greg's article as encouragement to experiment for yourself. Seems to me like the Topping D30 doesn't "sound like ass". :-)

Thanks again, Greg!

In other news this week... I see Darko is tripling down on the idea that "USB cables make a difference". As you probably know, I discussed this topic last week. Good to see that he's now arguing with some objective data like the 8kHz noise (which we've known about for years) that is not a problem with any decent DAC I've ever come across. Realize though that the argument was never about people hearing an 8kHz tone from their DACs (even that graph he posted showed a -105dB tone!). It was the idea that typically expensive digital cables (like these he reviewed a couple years ago) lowered audible noise, improved frequency effects ("filtering", better bass and treble), and improved jitter. Sure, USB data errors happen (in reference to Gordon Rankin's screenshot) - just get better cables that don't have to be expensive. Juergen Reis' quote in the article about PHY noise and jitter does not suggest that cables are the culprit nor that cables could "fix" a problematic, jittery DAC as far as I can tell.

The funny thing is that Darko is framing this subjective awareness of USB cable differences as some kind of transitioning "period pain". Perhaps insinuating that something like a "paradigm shift" is happening in digital audio thinking!? (Not working so well for MQA.) Supposedly, like the Earth is "spherical" vs. "flat", he reckons that his subjective opinion on USB cables (Industry-supported with interviews and such) leans toward the "spherical" side of the argument and will eventually be proven right? Well, I'll take my chances betting that audiophiles will turn to being "more objective" and discard the vast majority of cable beliefs, the "need" for expensive cables and associated groundless rationalizations fueled mostly by the advertising budgets of the last few decades. That kind of fantastical thinking is IMO regressive, a return to superstitions.

To me it's a bit disappointing that Simaudio would talk about stuff like USB cables (1:40) as I've always viewed them as quite reasonable fellow Canadians aiming to produce rational audio products. Looks like Costa Koulisakis, the "Director of Training" is more on the sales and marketing side than holding a technical portfolio in the company, thus not introduced as an "expert" in the article. As usual, we see more hand-waving around "listening tests" with zero details provided. Much of this is clearly nonsense and not applicable to the majority of audiophile systems.

The recognition that the Earth is spherical (or more accurately "oblate spheroid") belongs in the domain of natural sciences discovered and verified to be true by various objective means. Remember, USB cables (and digital audio in general) are products of human engineering with properties endowed by the USB Implementers Forum (USB-IF). Let's not be so grandiose as to suggest that the belief in fancy USB cables is in any way approaching a fundamental understanding of the natural world! Here's a thought, maybe Darko should interview some engineers in USB-IF companies to hear their thoughts on the "sound" of USB cables or even whether "bits are bits"... After all, they are the Creators of the various versions of the digital interface standard since 1996!

Eventually, knowledge like "the Earth is spherical" becomes self-evident for modern humanity; surely there's no need to call in an "expert" every time someone comes around and screams "The world is flat!", right?  Isn't this where we are with USB cables (and many other esoteric audiophile beliefs)? Experts need only be called if there is evidence to challenge the default belief, to determine the validity of such findings. Where is the evidence that one "needs" to or "should" spend more than a generous US$50 on USB cables to make any sonic difference?

The world is not black & white. Audiophiles are not either "experts" or "sheep". Just because one might not be an "expert" does not mean one is uneducated, cannot do a bit of research, or unable to activate one's critical thinking abilities. Those are I believe also the hallmarks of good, honest journalists. I suggest that Mr. Darko do a bit more research and soul-searching looking outside the audiophile bubble instead of following along just with Industry "experts" eager to give interviews. (I realize that I'm making an assumption here that he sees himself as a journalist. Perhaps in reality his role is as a "salesman" in which case he's probably doing a fine job for the companies.)

As usual, enjoy the music! Recently I've been digging into the works of Jerry Garcia's later years and his collaborations with David Grisman... There's the well known Garcia/Grisman (1991, available as SACD and 24/96, DR14), also 1993's Not For Kids Only, the jazzy Miles Davis cover So What, The Pizza Tapes with guitarist Tony Rice, and the posthumous Shady Grove that came out the year after Garcia's death. Generally, excellent acoustic recordings with good sound to feed the hi-fi. ;-)

And remember to do the blind test. Remember that I'm simply looking for honest results. Responses are collected completely anonymously, and a negative result is just as useful and as important as one who hears clear differences.

17 comments:

  1. In one such interview that John Darko had with Paul McGowan, Paul invited anyone eager to visit psaudio and hear this for themselves.

    This might be worth considering to experience for one's self.

    ReplyDelete
    Replies
    1. Yeah, that's not a bad idea. Maybe next time I'm in Colorado or maybe even RMAF again, it'll be a fun visit.

      That however does not solve the issues raised. The lack of what should be rather easily identified evidence that USB cables resulted in significant changes in sound should not require one to take a flight! :-)

      Delete
  2. My favorite subjective audio experiment is to perform listening test with clearly inferior or unsuitable cables to see what bad cables lack. Crappy, long USB cable confirm that- particularly with asynchronous data - bits are not necessarily bits. As far as subjective tests go, what's crappy has always been easier to identify than the minutia of the bleeding edge, but I've always found it just as eye(ear) opening.

    ReplyDelete
    Replies
    1. I once had a $6 1-meter mini-Toslink optical cable in my system. Worked ok at 16/44.1, but produced annoying intermittent pops and crackles at 24/96. I suspected every component in my system, until I finally looked at the cable's output. The visible light was unmistakably dim compared to another cable from same manufacturer. That was enough to make me invest in a $60 glass fiber, which was most likely overkill, but altogether a worthwhile experience.

      Delete
    2. Right, jcw, this is a good example and reasoning for why I also use those long, crappy, terrible examples of power cords and cables for the testing. If ultimately I can measure and show that down to minuscule distortion and noise levels that something made no difference, no matter how ridiculous, one can with some assurance recognize that it would be false to put too much weight into certain dramatic claims.

      Hey Mark. Yeah, I certainly would not suggest that one should get the cheapest thing one could find. It certainly has to work well and material quality of course comes with a price!

      The job of the consumer is to find good value in what we buy. If my articles are able to help in doing this, then I would have done my job...

      Delete
  3. Something amusing I notice with audiophiles (and some manufacturers) is that we tend to use the word "invest" for things we buy in order to obtain better quality. I find it really hard to justify my purchases as investments as they are a luxury expenditure.

    Probably that explains why I also tend to have a "subjective" bias (I still enjoy measurements, that's why I keep reading your blog). These are "unnecessary" expenditures meant to bring enjoyment. If there's scientific data to back it up, great. If there's none, then ears will be the final arbiter (and if wallet permitting). No need to get indignant over the lack of measurements or science.

    ReplyDelete
    Replies
    1. Hi YW,
      Yeah, and I can certainly appreciate where you're coming from. There's certainly nothing wrong with enjoying the material comforts of life and we all spend money on these things whether it be a clothing, a nice house, fancy cars, etc... I personally would not use the word "invest" either!

      I've tried from the start to see the hobby as more than just "objective" vs. "subjective" because that is oversimplistic. There are in fact multiple dimensions - for example:

      What type of "audiophile" are we?
      "Music lover" (Darko's "Music-First Audiophile" perhaps) vs. "hardware audiophile"

      Philosophically what are we aiming to achieve
      "Euphonophile" for subjective "good sound" vs. "high fidelity" enthusiasts

      Benefits of owning hi-fi/audiophile gear?
      "Utilitarian" (good sound) vs. "Non-utilitarian" (looks great, status symbol)

      I'm sure there are more than the above. We all are free to choose...

      Having said this, I've always seen myself as "more objective", happy to spend time as a "hardware audiophile", seeing "fidelity" as the object of my pursuit, and placing "utilitarian" benefits as more important than "bling" which I consider good but more superficial.

      As a blog, of course I welcome all readers including those with "subjective bias" like yourself. I don't expect folks to agree with me but I will try my best to be honest and be rational if there is a debate about something.

      What I don't like are highly questionable claims especially from folks who position themselves as sources of information (ie. Industry "experts"?) and those who disseminate information as fact (ie. "journalists"?), especially when we know that there could be financial gain in disseminating such questionable "information". This is why I believe pointing out and showing evidence contrary to such beliefs and statements are necessary and sorely missing - one could of course imagine why!

      This is also why I applaud work like Greg's this week and encourage audiophiles to maintain balance between what we "love" and what the "facts" show. Only in this way can we promote this hobby honestly with a sound foundation rather than one built upon mere opinions or worse outright fabrications.

      Delete
  4. also archimago - my favorite Garcia/Grisman album is "so what" they do a couple of miles davis songs in different renditions - very vintage garcia and well recorded

    ReplyDelete
    Replies
    1. Hi Stewart,
      Yup, great album! I think for many of us audiophiles, Davis' Kind Of Blue is simply a staple of our jazz collection. To hear it done by Garcia/Grisman is a treat!

      Delete
  5. Really great write-up Greg. Thanks for sharing. Two points I really appreciated (by which I mean reinforced my biases. ;-) Which is nice.)

    I recently bought a used D30. Nice to see it measured great and sounded indistinguishable from your other DAC, as they all should. I thought it sounded fantastic. I cannot believe anyone would have anything to complain over with it. I think a lot of complaining, when it's not just shilling by market competitors, is status signaling. Folks want to complain as a way of showing they are in the "cool kids" group of refined and critical listeners. Whatever. As I've said many times, zero popular music and maybe 10% of all music is mastered to take full advantage of Red Book, much less actually exceed it. Find a piece of kit that can faithfully reproduce at the Red Book standard and get on with one's life.

    Which leads into my second bias, practically they all do. Instead of SOTA signal reproduction far beyond the abilities of even your dog's hearing, it's the features stupid, with UIX being the elephant in the room. I had to smile when you wrote you'll probably just stick with the SMSL because you liked that it displayed the sample rate. Yep, it's the little things that matter, eh. :)

    Again, great write-up. Thanks for sharing.

    ReplyDelete
  6. I have certainly been taking my hits as of late over liking high rez files when the redbook standard is "all anyone really needs", theory of digital audio. Since I have been recording over the last 30 years and gone through R2R, Cassette, DAT, and now high bit rate up to 24/192 and hear the 1st gen copies of such I am through arguing with those who just read "white papers" . In audible testing, blind or not, most perceive louder as better even if it is +.5db or even less of an increase.

    I do believe that different dacs sound different, but again, the resolution of your complete system must be up to the challenge. What I find sad is that many with high end systems claim to hear no difference in CDs or SACDs, yet "jump our stuff" if we have average gear and can discern better .

    I use 2 of the Focusrite Scarlett 2i2, 2nd gen, and gave my son my Steinberg UR-22 interface. I would like to get a Clarett 3rd gen at some point and would expect to hear more. My recording gear for location recording are 2 Tascam DR-40's and a 6 track Tascam DR-680 MK2 that can do 6 tracks at 2496 and 2 at 24192.

    ReplyDelete
    Replies
    1. Hi Jim,

      Sorry to hear that you are taking hits for expressing what you hear. I hear differences in much equipment as well, despite my deteriorated hearing at 67.

      But I believe you may be forgetting that quite many of the audiophiles seem to have somewhat defective hearing abilities.

      Delete
    2. Since I do a great deal of recording for our local university and public school concerts I started doing better recordings with my MidiMan flying cow and calf and their superdac. Then even tried the The Indigo card in my laptop, but when I lost an entire concert doing that I have never done computer recording again. Now it is just my Tascam SDHC card recorders for 2496; the DR-2D and a pair of DR-40's; and since now a Tascam DR-680 MK2 for 24/192. I can hear the smoothness even at my old age. It seems to be that rather then take the time to do some testing, many doubters just read some White Paper and take it as gospel and let it go. Me, I like trial and error and prove things to myself. Maybe a Tascam DA-3000 by year's end for a run at DSD. Everything I record now starts at 2496, and I can burn them to DVDs to share. The noise in my FFT display is about -100db over 20 khz so I don't worry about it much. I've never heard artifacts while listening. Eve the 24192 files through my Scarlett into my main system sound wonderful, or on my AKG K-701 cans.

      Delete
  7. "The ear is an awful precision measuring device."

    Yeah, but only where it matters, and I'm not talking audio here. We were never meant to be able to distinguish if lion #1 had grass in its mouth, or if lion #2 was fat or skinny. But we were developed to be able to distinguish if lion #3 was moving towards us.

    Our hearing is extremely accurate with volume levels, but not very accurate with other things. Just saying...

    "Nature never over-endows a species beyond the needs of everyday existance"
    Alfred Russell Wallace (1823-1913)

    ReplyDelete
  8. Isn't the properties of different types of cables depending on the amount of Voltage they are carrying?

    I have always thought, that power cables and speaker cables that carry high voltage have to be well shielded, made of quality copper, have a good balance between inductance and capacitance, low impedance etc, whereas e.g. cartridge wiring that carry very tiny voltages can be of lower quality with very small audible implications. Am I wrong?

    ReplyDelete
  9. I just took the AIX audio challenge to find out if I COULD hear the diff in high rez and low rez files (I prefer high rez most of the time and have 5 SACD players which I like and I love the sound of SACDs. I do recording and most do 2496 masters of everything. I have 2 of the 2nd Gen Scarlette 2i2 interfaces and one Steinberg UR-22 my youngest son now owns for his listening and for his Twitch Gaming/Streaming. He admitted a big improvement. I am going to invest in a Topping 30 as I have not done any DSD downloads as the Scarlette is limited to 24192. I am curious as to if you found any difference between the 2nd and 3 gen Focusrite interfaces worth worrying about? I love to test, but in the end if I can't hear it I don't fret it.

    ReplyDelete
  10. Love to find such a science based analysis. When it comes to cables, I've not had the chance to do the sort of blind test that would convince me what I hear. I do always question whether tech readings can tell the whole story, though. By definition, they can't tell us anything about nuance of signal that we don't know. Our knowledge is always growing so there could be things that we don't even know how to measure yet that provide some difference. To me, the knife cuts both ways. It doesn't make sense that someone should claim that their ears know better than actual numbers and testing. By the same token, listening to music has nothing to do with numbers and everything to do with how one hears something. So on the one hand I trust the numbers tests far more for accuracy, on the other side how things sounds to the ear is all that matters. We so rarely have extensive blind testing done on these things so we rarely have anything more than one person or another talking about their experiences. It would be great to see a wider study done with actual scientific controls.

    ReplyDelete