Was curious what experiences others have had when backing up/copying their large sample libraries, files, etc.

My 8tb drive is getting close to full and I was considering purchasing a 16tb drive to replace it.

My question is, what's the best/fastest way others have found to go about cloning/copying a whopping 8tb of data onto the new drive? Ideally I want to keep all the attributes of my files (date created, modified, etc) for search purposes.

Is it a pain in the ass? How long did it take? Are there RAID setups to where you can simply add a new drive to expand storage space without having to copy data over to a new drive?

 

Only for making bootable operating systems.

That's not what RAID does.
And you would have had to have formatted the 8 in a RAID format already if you wanted to be running new drives in any type of RAID array.
So if the new drive and the old were both formatted in RAID, you would just link them.
Why would you want to copy the data?

You have to do it in stages because your computer will cook!
I have been transferring about 5TB of libs on and off over the last few weeks, coz I want to actually use my computer too!
I guess you could leave some going over night.
Big files go quickly, lots of small files goes slowly.

As long as you still have 15-20% free on that drive it can be your backup rather than a replacement.
Interesting to think you must be running sample libs off only 5400 RPM?
Or are you extremely very very rich?

 

Buy another 8tb drive.
And span your libraries on both.
You can use "symlink" to "join" both drives.

But remember : BACKUP data first.
And that's the main problem.

Ask yourself what happens if drive die

 

I love how no one is asking questions about why you'd need that much storage because most of us have probably crossed the 8 TB line a long time ago

I noticed you wanted to keep the Dates tags preserved too. If this is important to you, then Windows Explorer won't, it messes up one of the date timestamps, creation date if I remember correctly. To remedy that, you have to use a specialized copy program that will preserve those or at least give you an option or checkbox to do so. Google for it and check for reviews, I haven't used a dedicated file copy program in a while so I don't have recommendations for you.

 

I think
Fastcopy should do the trick
and or
Teracopy

@Smoove Grooves
Fastcopy can be automated to run @ a certain time
and what makes it fatser than Teracopy. you can adjust how many Threads it can/should use!
Does not use "System Memory"

Teracopy has features like "keep date etc."

Both are Free/mium
If crash (rare), they Remember what to do!!!
Check integrity of files
if needed make a list of Files

@BeatsNFreakz
forgot if Fastcopy does things like "keep date etc". But since its open source you can change & Modify+
It has provisions to ad your own scripts.

 

Don't mate. Unless you plan on getting another 16tb to back it up, you will be risking losing it all if the drive fails. Do what @Pipotron3000 said, it's safer and you will get better performance spanning those libs to two drives because we 're talking mechanical drives here. ( For a more detailed explanation read my answer on this thread: https://audiosex.pro/threads/1-or-2-ssds-for-sound-libraries-drive-load-etc.58477/ )

Errr. Nope bro. You can make a 1 to 1 clone of any data disk/partition not just system partitions using any good cloning soft like Acronis for instance. Because it will a 1 to 1 copy though, it won't be any faster than normal transfer. So the user will most likely be better off using a good third party copy softie. I am a fan of TeraCopy ( freeware, here: https://www.codesector.com/teracopy ), which let's you pause and continue simple copy/transfers, making sure you won't miss data or your transfer won't crash with the godamm unreliable Windows copy manager. It also displays current transfer speeds correctly, logs the files transferred and will retain date data (as @dondada said). I use it constantly (my friends and clients too) and is as good for copying small data as it's also great for copying large chunks of data like 800gb-1000gb at a time, it hasn't let me down many years now.
Cheers

 

Doh! Yes, of course. I'll get my coat!
I have never needed to create a clone.
Just a bootable clone!
Thank you for correcting me.

 

UnRAID software does this easily. But unlike typical RAID 5 or 6 redundant partitions where max data transfer speed is multiplied accordingly to how many drives are used for storage/redundancy, UnRAID uses a redundant partition format where the max speed is the speed of one drive, no matter how many drives you use, because it transfers data one drive at a time. So, it's good as a storage solution with user defined backup (you can set how many drives you want for redundancy) but it certainly doesn't add anything to performance and if we 're talking ssds it actually sacrifices some performance as well.
Most important here, is UnRAID is an OS itself, so it needs a dedicated pc to run on, with all advantages and limitations this carries.
https://unraid.net/
Drobo NAS machines which use their own proprietary "Beyond RAID" software, also do this out of the box. You can add any empty drive to the DROBO machine and Drobo's partition(s) at any time and it will be added to your existing DROBO partition "magically". Typical current model should be the Drobo 5C, which takes up to 5 hard drives and costs about 350 dlrs US (with no drives). Still, rebuilding a failed DROBO array can take even 48 straight hrs (lol) and it's not a particular performer either. Just a handy, more friendly solution as external NAS with a USB-C connection (most other models use Ethernet and/or USB 3 for connectivity purposes) : https://www.drobostore.com/products/drobo-5c
Cheers

 

Last edited: Mar 31, 2021

I use it too since long but with from v. 3.x it happens (many times) that the standard windows copy management takes over even if option "Use TeraCopy as default copy handler" is enabled, so I use it mainly from explore context menu, not with direct copy/paste, a bit annoying.
For some reason installing back v. 2.x won't solve the problem.

 

Make sure your Explorer Shell Integration for TeraCopy is in place. Check your profile. You can also, off the top of my head, see if you have admin/elevated rights on TeraCopy and folders you use, and also check if you can set TeraCopy as the default software for transfers in Win10 third party software options (not really sure if there is such an option though).

 

The easiest way if you have a case/chassis that has space is to use something like these offerings:

StarTech 4 Bay Aluminum Trayless Hot Swap Mobile Rack Backplane for 3.5in SAS II/SATA III - 6 Gbps HDD
Connect and hot swap four 3.5in SATA III or SAS II hard drives to your computer system in three 5.25” bays, with support for transfer speeds up to 6 Gbps

ICY DOCK ToughArmor MB872MP-B

ICY DOCK ToughArmor MB996SK-6SB

Another option is a workstation/server case (tower or rack) that has included hot swap with the ability to add more.
An old non-proprietary IBM, Dell, etc... case would fill that need.

Then there are the USB 3.0, USB 3.1 & USB-C external docking stations. Usually 1 to 4 drives.

Hope this helps.

 

Yes, actually here is the only sure way to use Teracopy.

Yes.

Win7 x64 here, but the problem is that is random, somtimes it works, sometimes not, for no apparent reason, like handlers are fighting each other.
In past 2.x was always working, but not anymore, maybe some win update/security patch.

 

Oh, the computer i spend most time at home is an i3 8100 with Win7 x64. Your situation is very strange, i run Teracopy 3.26 (not the latest 3.8x) since i setup this comp. Perhaps check the MaxPathLimit in the app's options.ini which is located at Users/Username/AppData/Roaming/TeraCopy. If that value is 1 it doesn't allow extra long paths. When set to 0 allows the soft to create extra long transfer paths, larger than 255 characters. Although i find it very impossible for this to be interfering with the Shell Integration, the random behavior maybe a hint towards this direction.
If nothing works there must be something screwed in the registry, so perhaps do an uninstall, clean sweep of every Teracopy registry trace and re-install ?
Hope this helps and sorry if you already have done all this
Cheers mate
Ps: If you want an older Teracopy version like mine to check behavior, https://www.getintodesktop.com/utilities/teracopy-3-26-free-download/

 

Last edited: Apr 1, 2021

Before you I could read your suggestions I updated to 3.8 (I had 3.25 before) and crossing fingers it seems to work fine at the moment.
Pity that the new gui is worse (to me), but I'm happy in anyway.

Thank you for your answer I'll report if anything with TC goes wrong.


Edit: I spoke too early. Same problem. I'm going to try as suggested.

 

Last edited: Apr 2, 2021

8TB of what?

 

I tried as suggested, installed 3.26 and all the rest, same behaviour.

So just to have some fun I tried to unregister TC handler leaving TC as default, it works at the very moment, crossing fingers.

IF it will fail again, I'll come back to a very old copy tool that I used before TC 2: KillCopy, more or less the same features of TC.

 

Welcome to the world of the industry-wide used Kontakt sample player!

 

Wonder if someone is going pro with a dedicated NAS/raid 10 ?

Or also having the stuff in a private datacenter?

 

If you find out how this was done, you'd save a lot of space.

 

You mention other ideas that are better so while this will certainly double your space (not quite), you will likely fill it up.

The answer in your case is both no and yes.

Normally you would need to build a RAID array separately and then copy the data across to the array. You can clone anything these days with high-level software without changing the partition size. This is for data drives and system drives. The yes aspect is only applicable to RAID hot-pluggable environments with hardware being previously setup to add more drives to the array.
The best, most stable and fastest RAID array I have had contact with is a RAID 10, but it is not a cheap exercise. There is an even better one not so common called a RAID 50 (Invented by HP) which is a parallel striped RAID 0. Regular people do not need these nor can they afford them.
RAID 0 is fast but if one falls over, it all falls over. If you go for Mirroring RAID1, it has redundancy but at the cost of duplicated drives that are truly only live hot-swappable backup. RAID 5 by the sound of it is what you are looking for and more economical than RAID 1.

If you set up raid 5 and if you have a great RAID board, you can add more drives so that is a yes there too but in your current setup...no without doing several things first. When you add a new drive into an array, it also needs to cross-check all the data with the new drive so it can consider it part of the array - in simple English This is so it can map it properly e.g.



EDIT - So in the above image in plain English - the little 'p' in this diagram stands for parity on each of the disks. You can add a new drive like you asked or many depending on how good your RAID card/facility is, but each time it has to stripe the new drive with parity. This is the safety mechanism if one drive fails it can be rebuilt with the parity and data from the other disks. The array remains functional but should warn you with a message like 'degraded'. In some arrays, you can also open the case and replace the faulty HDD while the machine is running (Hot-swappable).
In this image Disk 0 parity (Dp) has a full snapshot of D, Disk 1 of C, Disk 2 of B and Disk 3 of A. So combined with each drive being evenly spread across all four, it can rebuild any failed drive, in English.

 

Last edited by a moderator: Jun 28, 2021