I've just transferred almost 8TB of data from an old mechanical external drive to an external SSD drive. Windows Explorer crashed a couple of times and the source mechanical drive dropped connection once or twice, meaning I had to unplug and reconnect the USB cable.

From right clicking and checking properties, the data value is in the ball park of what it should be. However, is there any way to validate and verify the content easily in Windows, be i at a drive-to-drive level, or at a folder-to-folder within the drives level?

As they're mostly Kontakt libraries, I just need to check to ensure full content transferred ok before beginning to solely rely on the SSD and only storing the mechanical as a hardware backup of the data.

 

WinMerge does a comparison like that, down to the byte level. Works on single files and directories.
For larger comparisons (in the TBs) it might take a while but if you wanna be certain a specific directory (like Kontakt Library) is still intact this should do the trick fine.

 

Here is an How To for WinMerge:

https://www.wintips.org/how-to-compare-folders-and-files-in-windows-10/

 

All my backup, file synchronized using this tool:

https://freefilesync.org/

Why do I use this?

I have several drives that need to be updated from the main working drive, so this tool will scan and check the other drives and then copy all the data bit by bit to it only copying the difference.

Each drive will had a database file in each root directory so that FreeFileSynch will see that library when updating the drives. Make sure you have the Source and Destination checked first, the main screen is self explanatory. You do not have to copy anything over, just scan both drives and you will get a report of the differences, before you decide to sync them.

Perfect tool when I moved all my thousands of CDR's and DVD'Rs backups to Master backup HDD drives last year, replication of the data then sync test was easy using this tool.
Sadly said goodbye to the CDR and DVDR media as all were shredded ;( it was time to dump this stuff.

 

Last edited: Dec 26, 2022

I used Beyond Compare to compare two 2TB drives with around 40,000 files. It focuses on filenames and file dates / sizes only. It lets you do a side by side comparison of two folder structures. Was pretty fast to load, but no clue how it might handle 8 TB structures with maybe a million files. Mileage may vary.

Though assuming you mean bit for bit validation... That's outta my scope.

 

Another vote for FreeFileSync.
https://freefilesync.org/
I actually donated ($5, i think ?) to have the benefit of automatic updates and something else I can't remember.
Actually I donated because it's the best I have found.
It is very easy for normal mirroring, and that is what I use continually for mirroring my archives & backups onto external drives.

It also has two-way and custom comparing / updating, and those are great for comparing DAW libraries (presets,expansions,etc), with the ability to show what will be copied / removed from each side with the click of buttons along the bottom, and to just copy from either side to another place instead of doing an actual sync.

I also use it to mirror a few folders to my phone, and I have it on my wife's Win7 computer doing similar stuff there.

 

You could calculate checksums for the files and compare them. I guess there will be programs that would do just that, but it should be possible to write a script that would do it too.

 

You can also use window's built in PS tools for that.

Code:

PS C:\Users> comp /?
Compares the contents of two files or sets of files.
COMP [data1] [data2] [/D] [/A] [/L] [/N=number] [/C] [/OFF[LINE]] [/M]
  data1      Specifies location and name(s) of first file(s) to compare.
  data2      Specifies location and name(s) of second files to compare.
  /D         Displays differences in decimal format.
  /A         Displays differences in ASCII characters.
  /L         Displays line numbers for differences.
  /N=number  Compares only the first specified number of lines in each file.
  /C         Disregards case of ASCII letters when comparing files.
  /OFF[LINE] Do not skip files with offline attribute set.
  /M         Do not prompt for compare more files.
To compare sets of files, use wildcards in data1 and data2 parameters.

If you lave the *NIX subsystem installed you can also use diff.

 

Negative. That is not "easy". One typo and you are screwed.

 

> comp d:/samples/*.* f:/samples/*.* /OFF /M

and walkaway for however many hours that will take.

 

I do this sh*t all the time. Like 12 8TB USB HDDs and 16 4TB 2.5" USB HDDs for 'back-back-up'. Huge #'s of Kontakt and WAV, etc files. This is how I do it :

For super quick, I just use SUPERCOPIER 2.2b, and change options to "overwrite if different". Start up SC, select your content, drag it to destination, it will scan for already existing content and either overwrite the content if it's destination content is different, or skip if it's the same (size). It has an error log if SC runs into any errors (in options, turn on "on error put files to bottom of list". Then check the "error" tab for any errors encountered.

When copy/validation of copy is complete, I check properties of source and destination contents (from same folder levels), and check that :
1.) numbers of folders match
2.) number of files match
3.) "SIZE", in bytes, matches exactly. ("SIZE ON DISK" does NOT have to match.)

Supercopier 2.2b is old, but for some reason works waaaay better that the new versions and better than Ultracopier for me.

SC should blaze through 'checking' what you had already copied 1:1, and perform a copy of what is not already in the destination.
http://greatcomputerworld.blogspot.com/2011/11/super-copier-22bexe.html

 

Last edited: Dec 27, 2022

hey, I use TeraCopy 3.9.2 (free) to do massive backups... much in the way that @bluerover (above) uses SuperCopier... that way drives are matched with Skip All of already copied data... have also successfully used FreeFileSync...

 

I also use freefilesync.
One very useful function, when you have a problem like yours is to put it into "mirror" function and tell which drive is the "master" one, so it will show what is missing in the other one, then you only have to click to transfer missing files.
I also use it for backups, in different ways, mirror or incremental, depending of the material and situation.

 

Me too, though it's not clear to me what "quick test" means to it, nor if to trust it.
Once in a while, I do full binary comparison and it finds errors not targeted by quick.
Lot of options, maybe the most complete backup/comparison sw.

 

Perhaps it uses hashes like MD5 or SHA. They're good but the bigger the size of the data compared the bigger chance of fail.

It's a program I have in my "totest" list but still haven't got my hands on it yet

PS. Could also be checksums like he said, not the same as hashes but perhaps even more suited for this comparison stuff. I think it's what torrent clients use to validate the chunks of data they receive.

 

Last edited: Dec 27, 2022

Since 2005 I use Total Commander... which makes that perfectly along with tons of other things too.

Not free, but not expensive either. And it is the best swiss army knife for the disk folders management and files management. Even on networks, local or not, FTP, SFTP, etc.
Multi-tab and tweakable as deeply as you want, able to automate actions, creation of your own tools and toolbars, internal or external viewers, internal or external archivers, hash files with many protocols, compare files, dup finder, compare dirs recursively, batch renaming tool with or without patterns, use of regex, support of many addons, support of many external tools, etc. yet very easy to use (quite intuitive) for all the common daily actions. Hundreds of features... and with a very comprehensive offline help system updated at each update of the program.
The "Synchronize Directories" feature is one of these extremely comprehensive features with dozens of possible actions on the result.

A tutorial wiki (fed by registered users):
https://www.ghisler.ch/wiki/

 

Last edited: Dec 28, 2022

According to the help file:

So basically if I decide to not override quick tests, there could be differences, but mostly due to timestamps or other neglectable things depending on the file format type that has few influences on the file working.

 

You're right. One simple text file re-saved from ANSI to Unicode8 and "oops".
Very nice, what a shitload of options, including for programmers code files.
As said before I wanna test this bad boy and compare it to UltraEdit's compare.

 

The simple way to handle this issue is by using a third-party tool like

• Beyond Compare
• Gs Richcopy 360
• Teracopy
• SyncThing

all can meet your requirments in this question