Many moons ago (back in 2013), I discussed DSD audio. While over the years I've talked about DSD from a a number of different angles (eg. JRiver realtime conversion listening, PCM <--> DSD conversion, conversion 2015, equipment DSD playback measurements like PonoPlayer / TEAC UD-501 / Oppo BDP-105 / Oppo Sonica DAC), concerns from that very first discussion about DSD still irks me these days!
The main issue is simply this: the lack of ability for gear and software to support both tagging and data compression when dealing with DSD. This IMO truly has been a ridiculous state of affairs for so many years! Despite years of cheerleading in the industry and even smaller sales outfits like NativeDSD Music already putting files out there for consumers, the relative inelegance of it all is rather silly.
As usual, I find it rather baffling that the typical mainstream audiophile press remains silent on deficiencies like this. To me, it is issues like these, while perhaps not "sexy" nor provides the press something to market to consumers, that when addressed can actually move the hobby forward. Unlike yet another megabuck product that provides a few minutes of eye-candy at best and affects the few, "infrastructural" changes can affect the many. Yet another example of the myopia of the audiophile press and an inability to muster any leadership in changing the industry and hobby in meaningful ways.
For today, let's not worry about the pros and cons of DSD itself or if 1-bit multi-megahertz sampling sounds significantly different to PCM. Also, let's for a moment not worry about the fact that many SACDs are just PCM upsamples. Suffice it to say that there are albums that originated in DSD such as those from Channel Classics. Also, some old analogue recordings such as some of the original Sony DSD's appear to have been converted directly to DSD from tape. As a perfectionist (neurotic) audiophile, since I have purchased a number of these recordings over the years, I'm happier to keep them in the original digital format rather than conversion to PCM if possible. And ideally, it would be nice for these few DSD albums to be played back using mature systems that don't feel like they were "hacked" to support them.
I. Intro
As I laid out in the 2013 DSD article, the problem with the traditional DSD file formats .dsf and .dff is simply how primitive they are! DSD is typically quite an inefficient data format and screams out for compression. And as I noted earlier this year, I have a system of meticulous tagging for my music library and it would be ridiculous to incorporate a bunch of DSD albums without a decent way to keep them tagged within the collection.
Historically, SACDs can utilize DST for lossless compression (Richard Murison of BitPerfect wrote a post about DST a few years back). These days, there's just no support for this. While DST decompression can be found in the wild to convert a ripped SACD .iso, and both foobar (using SACD plugin) and JRiver can play them back, as far as I am aware, there is no free compression software. Years ago, the Philips ProTECH DST Encoder was available for studio projects, but I don't know what has happened to the software these days or if it has been incorporated into some other studio package; last I saw was this announcement that Sonic Studio took over the software in 2005.
With the release of SACD, the world was introduced to what we would call these days "DSD64" (2.8MHz) or 1-bit 64 x 44.1kHz sampling rate. Around 2007, with affordable devices like the Korg MR1000, we started seeing higher-resolution DSD128 (5.6MHz), then DSD256 (11.2MHz), and most recently even DSD512 (22.6MHz) capabilities in modern DACs. These higher sample rate variants suck up even more data and IMO, it would be ridiculously unwise not to implement some form of lossless compression with such files!
It was therefore wonderful to see that in December 2016, with the release of WavPack 5.0.0, we finally have the inclusion of DSD data compression into an open-source file format.
Although these days, for PCM, almost all devices have the capability to play back and convert FLAC (even finally the latest Apple products), not as many can manage WavPack. For the audio historians among us, the development of WavPack in fact predated FLAC. The first version of WavPack was released way back in 1998!
II. Batch conversion to WavPack...
Using WavPack and BatchEncoder in Windows, conversion of .dff and .dsf files is easy... Go download the WavPack software. I'll typically just uncompress the command line package to C:\WavPack (or some place easily accessible). Then grab BatchEncoder.
In BatchEncoder, go into the Options --> Configure Presets menu to add the -d switch for DSD compression like so for each of the fast/normal/best compression options:
Then go into Options --> Configure Formats menu and make sure the WavPack command line path is correct, like so:
With that, you're ready to convert whatever .dsf and .dff files you have to a compressed .wv files using standard drag-and-drop into the BatchEncoder main screen:
As you can see in the "threads" option below, feel free to use all the threads at your disposal. The process is fast and easy. In the image above, I converted my SACD-ripped copy of the excellent Alison Krauss + Union Station album New Favorite.
Since WavPack uses standard mechanisms of tagging such as ID3v1 and APEv2, it's a joy to use my standard tagging software Mp3tag for all the final editing:
Awesome. Everything in place with the tagging system I use with all my DSD albums now :-).
If you're wondering, WavPack compression isn't as efficient as the old DST. For example, here's "Mighty" Sam McClain's album Soul Survivor as uncompressed .dff, "best" lossless compressed WavPack, and DST-compressed .dff:
What we see is that for this album, WavPack reduced the data utilization down to 52%, while DST was able to compress it down even more to only 40.7%! Still pretty good for WavPack even if it's not as good as the old DST algorithm. Each album will vary although the relative superiority of DST is clear in terms of smaller file sizes. However, I believe WavPack can be encoded/decoded much quicker and in realtime with slower devices.
As you would expect, no problem at all decompressing these .wv files back to .dff or .dsf using the aptly named wvunpack.exe command line if you ever wanted to one day (check out the WavPack command line documentation).
As you would expect, no problem at all decompressing these .wv files back to .dff or .dsf using the aptly named wvunpack.exe command line if you ever wanted to one day (check out the WavPack command line documentation).
III. Playback... The road ahead?
Is everything good with DSD now that compression and tagging can be accomplished just like with PCM? Well, not quite but I think it's getting there... If you're a PC computer audio user, look no further than Winamp, foobar and JRiver 23 already capable of handling the DSD decompression and playback. Be aware that WavPack does take a little bit more CPU power to decompress than FLAC, but these days, any reasonably fast computer should have no problems.
Unfortunately compatibility has not reached streaming devices quite yet. For example, MPD as typically used in many Linux devices already has incorporated WavPack-DSD handling, but I have not seen or heard of DLNA streaming to Pi3/Volumio or similar set-ups with WavPack-DSD compatibility.
Likewise, although DSD playback with Squeezebox/LMS systems using DSDPlayer plugin has been available for awhile (including realtime DSD-to-PCM transcoding), even though the .wv files are recognized and tagged in the music library, decoding for DSD playback currently is not implemented when streaming to the latest piCorePlayer 3.22 (and nightly LMS 7.9.1 as of November 16, 2017). There are a few pieces that need to be implemented like scanning to report true track duration, reporting to DSDPlayer that the output could be native DSD for consistent transcoding / DoP streaming like with uncompressed .dff and .dsf.
Looking around, I also do not see compatibility yet for WavPack-DSD with Roon.
Well guys, I think it's pretty obvious that I am hoping that WavPack-DSD support can be expanded in the months and years to come as a common way to deal with these DSD files. Hopefully as an open-source format it won't be hard to incorporate into these streaming systems!
It has been a remarkably long time coming... I must admit that with all the hype DSD has received over the years, it's amazing that it has taken this long to finally see the release of a modern way to handle tagging and compression. As I mentioned in my DSD article from 2013, I believe the file format is extremely important and serves as the foundation from which other support software and mechanisms are built. To me, it was rather silly to have all kinds of hype without serious development of this scaffold upon which libraries of music can grow.
Perhaps this is not surprising at all since DSD was never meant to be a consumer computer-audio "format". Rather, it started life as SACD, basically Sony's attempt at another revenue source with strong copy protection mechanisms as their patent life on CDDA was near the end (sort of like how MQA is being offered by Meridian as MLP's patent life comes to a close... That's another story!). DSD64 really only became interesting to audiophiles once SACDs could be digitally ripped when hackers figured out how to use early "jailbroken" Sony Playstation 3's to get the job done back around early 2011 (some instructions). So from the start, DSD playback was driven really by the "underground" and not officially sanctioned by Sony or any of the big companies although I recall Sony did have DSD software in some of their Vaio laptops back in 2007/2008.
Lastly, I think it's great to see WavPack become the first "universal" audio archiving format! It can of course handle high-resolution PCM, now DSD including higher samplerate variants like DSD128 and DSD256. There are also allowances for lossy PCM encoding if needed.
I'm not sure of the longterm commercial viability of DSD these days. It has been quite awhile since I have bought a SACD or DSD download. But at least WavPack 5 is now available and Måns Rullgård added support in SoX awhile back in 2015 so the tools are finally better... I'm not sure if many of us would shed tears if DSD ends up being a legacy format.
Pssst... Logitech Music Server / piCorePlayer developers... I know more DSD work is being done on the code so hopefully there will be good functionality ahead!
I tried michaelvv's post here to play back the DSD-encoded .wv files. I found it worked off and on with my TEAC UD-501 USB DAC playing the music through DoP on my Linux VM LMS set-up - not sure why and I guess YMMV. Eventually I just used the default WavPack 5 decimation to WAV which plays back as 352.8kHz PCM on my "Pi 3 Touch" (in the Linux server, replaced the /usr/share/squeezeboxserver/Bin/x86_64-linux/wvunpack with a newer version 5 binary). Still far from ideal; the ideal being out-of-the-box no hassles playback, native DSD/DoP streaming, handling DSD128+, and seamless transcoding to PCM-only devices like my Transporter. At least I can now incorporate these compressed DSD albums in my system along side all the standard and hi-res PCM.
Unfortunately compatibility has not reached streaming devices quite yet. For example, MPD as typically used in many Linux devices already has incorporated WavPack-DSD handling, but I have not seen or heard of DLNA streaming to Pi3/Volumio or similar set-ups with WavPack-DSD compatibility.
Likewise, although DSD playback with Squeezebox/LMS systems using DSDPlayer plugin has been available for awhile (including realtime DSD-to-PCM transcoding), even though the .wv files are recognized and tagged in the music library, decoding for DSD playback currently is not implemented when streaming to the latest piCorePlayer 3.22 (and nightly LMS 7.9.1 as of November 16, 2017). There are a few pieces that need to be implemented like scanning to report true track duration, reporting to DSDPlayer that the output could be native DSD for consistent transcoding / DoP streaming like with uncompressed .dff and .dsf.
Looking around, I also do not see compatibility yet for WavPack-DSD with Roon.
Well guys, I think it's pretty obvious that I am hoping that WavPack-DSD support can be expanded in the months and years to come as a common way to deal with these DSD files. Hopefully as an open-source format it won't be hard to incorporate into these streaming systems!
It has been a remarkably long time coming... I must admit that with all the hype DSD has received over the years, it's amazing that it has taken this long to finally see the release of a modern way to handle tagging and compression. As I mentioned in my DSD article from 2013, I believe the file format is extremely important and serves as the foundation from which other support software and mechanisms are built. To me, it was rather silly to have all kinds of hype without serious development of this scaffold upon which libraries of music can grow.
Perhaps this is not surprising at all since DSD was never meant to be a consumer computer-audio "format". Rather, it started life as SACD, basically Sony's attempt at another revenue source with strong copy protection mechanisms as their patent life on CDDA was near the end (sort of like how MQA is being offered by Meridian as MLP's patent life comes to a close... That's another story!). DSD64 really only became interesting to audiophiles once SACDs could be digitally ripped when hackers figured out how to use early "jailbroken" Sony Playstation 3's to get the job done back around early 2011 (some instructions). So from the start, DSD playback was driven really by the "underground" and not officially sanctioned by Sony or any of the big companies although I recall Sony did have DSD software in some of their Vaio laptops back in 2007/2008.
Lastly, I think it's great to see WavPack become the first "universal" audio archiving format! It can of course handle high-resolution PCM, now DSD including higher samplerate variants like DSD128 and DSD256. There are also allowances for lossy PCM encoding if needed.
I'm not sure of the longterm commercial viability of DSD these days. It has been quite awhile since I have bought a SACD or DSD download. But at least WavPack 5 is now available and Måns Rullgård added support in SoX awhile back in 2015 so the tools are finally better... I'm not sure if many of us would shed tears if DSD ends up being a legacy format.
-----------------------
Pssst... Logitech Music Server / piCorePlayer developers... I know more DSD work is being done on the code so hopefully there will be good functionality ahead!
I tried michaelvv's post here to play back the DSD-encoded .wv files. I found it worked off and on with my TEAC UD-501 USB DAC playing the music through DoP on my Linux VM LMS set-up - not sure why and I guess YMMV. Eventually I just used the default WavPack 5 decimation to WAV which plays back as 352.8kHz PCM on my "Pi 3 Touch" (in the Linux server, replaced the /usr/share/squeezeboxserver/Bin/x86_64-linux/wvunpack with a newer version 5 binary). Still far from ideal; the ideal being out-of-the-box no hassles playback, native DSD/DoP streaming, handling DSD128+, and seamless transcoding to PCM-only devices like my Transporter. At least I can now incorporate these compressed DSD albums in my system along side all the standard and hi-res PCM.
----------------------------
Well, glad to be back home after a few weeks overseas. Home, sweet home...
Doesn't look like much new going on in the audio world in the last few weeks. I guess this means that folks are simply enjoying the music :-).
Carry on.
Doesn't look like much new going on in the audio world in the last few weeks. I guess this means that folks are simply enjoying the music :-).
Carry on.
DSD users seem to have much more severe problems to face than just tagging. Here is an excerpt from the RME ADI-2 Pro manual:
ReplyDeleteYou may notice clicks and cracks at the title change quite often with DSD, even when the next title has the exact same sample rate. This is caused by the 1-bit format, which, unlike PCM, requires absolute silence and DC freedom at the beginning and end of a title, so that the transition as required by the 1-bit stream does not represent a random signal, which can sound like a click or crack. Unfortunately, many freely available tracks are not 'clean' at the beginning and the end. If these are played back one after the other by player software, the ADI-2 DAC's level meters show that the noise to be heard does not originate from the ADI-2 DAC, but is presented to the DAC as a signal to be played. The level meters are in the digital domain before the DAC, so proof of a faulty input signal is easy.
And I don't see how that will ever change unless the player software starts to process the DSD signal into PCM and back or so, totally defeating the DSD purpose.
Hi Techland,
DeleteCan't speak about the RME ADI-2 Pro of course and this certainly sounds like a headache for the pros when noise, clicks, and other distortions intrude on the signal.
Speaking from a consumer perspective, ripped SACD DSD64 generally sounds good and plays quite well to the DACs I've had experience with like the TEAC UD-501, Oppo Sonica DAC, iDEA, etc... without issue. Most of the time when I have problems, it's been compatibility of software and the hassles of drivers; whether it's getting native DSD playback or having to use DoP and whatnot. It's thankfully all gotten easier over the years.
Audirvana, a software soon also for PC, "study" the file and mount it in RAM before sending DSD files to the DAC ans avoid clicking.
ReplyDeleteYup... Good buffering will correct a multitude of audio "sins" :-).
DeleteCool. Didn't know Audirvana coming for the PC. I guess it'll also have the software MQA decoder like the Mac version currently.
AFAIK, originally DSD wasn't even meant to be offered in a consumer medium. It was meant as an archiving format for record companies. The idea was that the archival copy could be down-rendered easily and mathematically transparently to any consumer PCM format (so long as the target sample rate was an integer multiple of 44.1 kHz) for production.
ReplyDeleteYeah, I've heard this idea as well about DSD being an archival format; specifically for analogue tapes which was what Sony basically did with some of the early releases - direct transfers of some of the back catalog to DSD64.
DeleteNot sure how many people still would agree to this. IMO, given the limitations of DSD64, archiving should really start at DSD128 to extend the frequency response and keep ultrasonic noise low beyond 30-40kHz if they still insist on using it at all instead of just 24/96+ PCM.
Yup, 176 kHz/24 bit PCM or higher (with a 44kHz base rate, >=192 if it's 48kHz) would have served the same purpose just fine, and ultrasonic noise would not have been a problem, and there's be no need for
Deleteproduction workarounds like DXD. But there were of course other elements in play, like copy protection and tech ownership. Seems like there was no real audio 'need' for DSD....hmm that sounds familiar *cough*MQA*cough*
I use Tag & Rename (http://www.softpointer.com/tr.htm) for tagging my .dsf, but not .dff. It is a paid program, but I do like the functionality and usage. It can also handle up to 11.2MHz DSF files.
ReplyDeleteIt would be possible to add an APE tag to a chunk in the DSDIFF format without breaking it. I'm disappointed at the reluctance from developers to implement such a scheme. They claim it is not a standard, and of course it can't become one unless it emerges defacto, as ID3(v1) once did. Back in the day, developers were free to add APE tags to WAV files even (foobar 0.8.3), and to streamed formats like AAC, AC3, and DTS. But today AIMP developer says tagging cannot be supported for a lack of standard.
ReplyDeleteContent providers would probably prefer we view tags in some proprietary X/HTML browser window, and do no editing of our own.
WavPack is indeed a versatile nearly universal format, for floating-point and unbounded dynamic range during editing, for near-lossless hybrid encoding, which would be a decent open-source alternative to mqa. WavPack also overtakes FLAC in compression ratio with upsampled content using its x-modes.
The speed of WavPack is indeed slow (especially, high bitrate lossy and x-modes). While realtime playback is achievable on any existing computer, the slow speed is still felt when scanning a collection for ReplayGain, DR statistics, or during CUETools verification.
EZ CD Audio Converter (https://www.poikosoft.com/music-converter) handles DSF, DFF, and Wavpack tagging. I have been using it with my DSD music collection for few years now. I also use it for converting files from DSD to FLAC and for packing DSD files to Wavpack DSD.
ReplyDeleteSince 2015 i'm adding dsf to my music library DSD64, DSD128, DSD256 and DSD512. The rest is in FLAC with different resolutions and sample rates. I also have strict rules to catalog and tag all the files and never had problems tagging dsf files. In the past i use MediaMonkey to catalog my files but not support dsd and move to foobar first and next to MusicBee when support for DSD has added. MusicBee has no problems reading dsf tags including image covers, lyrics, ratings and more, but has problems writing tags for dsf files. Because of this i use mp3tag for tagging all of my dsf files. Its free. And basically all of my Windows and Android players read at least the more basic and important tags and others reads all the tags: MusicBee, foobar, JRiver, Samsung Music, Fiio Player, Neutron, etc.
ReplyDeletehow can i convert from DSD to DST
ReplyDelete