Admittedly, it's not all for music. I also need to store a huge amount of video footage. But am more concerned with fast access to samples and Kontakt Libraries, which I simply can't store on my HDD. Currently, I have 20 external hard disks, portable ones, that sometimes do the job, sometimes don't. and I'm worried they're going to blow up one day. So I have a need for massive back-up AND fast access (for samples and Kontakt libraries°. In addition, I can only drive a couple of USB external hard disks at a time (sometimes, only one when I used the 4TB disks I have) . Finally, they're kind of sluggish too. Performance varies from disk to disk, but they're not ideal for fast concurrent access to multiple Kontakt libraries. In short, my current storage solution is piecemeal and inadequate.

Would you be able to describe the ideal non-cloud storage solution for this kind of capacity, which is scalable and fast enough? I would also appreciate a gentle and patient explanation of how you access large external storage on your PCs when I am told I can only connect to about 8TB (two 4TB external disks with USB connections). Is that a universal USB restriction? What is it I am not getting here?

Finally, I would also appreciate an explanation of how you allocate resources to SSD, HDD and external storage. Do you reserve the limited SSD capacity to the OS, or the OS and the main software you run (in my case DAW, Photoshop, After Effects Premiere etc) and reserve the HDD for samples and libraries? And when you run out of HDD space, do you keep deinstalling and reinstalling different libraries to the HDD or have you found an efficient way to access large storage and fast access externally?

I know budget obviously plays a great part in the choice of setup. Let's pretend it's not an issue here so I can begin to understand the ideal setup, and I'll work my way down from there.

As always, thank you for your suggestions and comments.

 

Are you taking 300TB because you want to RAID the disks? If not, for that amount of data you will need backup

 

Well, that's the question. In fairness, the music stuff is less than 50TB right now. The video footage and other files I keep account for about 120TB and growing. The rest is estimated scalability requirements. So I guess I only need RAID for the music stuff, but eve then, that's a lot of storage for direct access.

Regardless of the actual size, if we assume that your internal HDD is not enough, how would you set up storage for the "direct access" stuff (largely sample libraries)?

 

300 Terrabyte ? who are you ? Mr Spotify himself ?

 

Maybe better suited for https://old.reddit.com/r/DataHoarder/

 

At this point you need to set up your own data server with back-up drives because as they begin to fail (2-5 years) it will become a total nightmare to trace and recover.

 

I'm gonna try this from Degoo
10 TB for life for just $100 USD Looks Good

https://deals.gdgt.com/sales/degoo-premium-lifetime-backup-plans-10tb

Theres also Thunderdrive
2 TB for $60 for life

https://www.neowin.net/news/save-95-off-a-lifetime-subscription-of-thunderdrive-cloud-storage

 

With this amount of data you need to think about redundancy and resilience. Smaller drives are less error prone than larger capacity drives, so you are talking hundreds of drives here. If you want to ensure your data remains available and free from corruption and drive failures you will need two copies for each drive.

Have a look at Ceph as an object store solution, this takes redundancy into consideration as a solution.

Windows fails when indexing vast amounts of data, have a look at an Elastic cluster to solve this.

tl;dr

You will need 900TB to ensure your data is safe, so you will need to think enterprise solutions rather than home solutions

 

No matter how I look at it, it's too much for hardware designed for users, even for power ones.
Meaning enterprise-level hardware and perhaps setup, like someone told.
If you want fast access, that would mean almost an small datacenter, so to speak.

 

If "only" up to 170TB is enough for your needs, with a DX517 expander, this Synology DS1019+ NAS [network attached storage] could be the solution for you. I have nothing but praise for Synology NAS devices. However, I would definitely build a RAID 3 or 4, or possibly 5 for at least some redundancy [one disk can fail]. RAID 2 is the most secure, though, and the more disks there are, the better the chances one of them will break.

What you are doing now, with all these external HDs, is actually pretty safe. But it is admittedly inconvenient for usage, of course.

 

who are you? what are you doing? 50tb of music stuff you mad? reply me sir

 

You really saved the whole thing with the "sir" ....

 

how bout read his post...

 

UnRAID server for storage. a few small RAID setups with SSD for fast access/current work, esp for video - since video is 5x the amount of data/speed for rendering/editing, over audio. UnRAID is prob cheapest (use any size/brand drives, enterprise not needed) large server solution

 

Lol, thats a shed load of porn you've collected there.

 

With a requirement of 300TB, forget USB. You want a NAS of some sort. Probably the cheapest way to achieve your goal would be to look up some server gear on eBay. They always need cutting edge, so really good stuff that's just 2 years old and way more powerful than a regular desktop PC becomes worthless for them and can be had for cheap. Make sure you have a place to put that stuff though, you don't want to have a server in your room. That stuff is loud (about as loud as a jet taking off) and the room you put it in will heat up *a lot*. The parts such as power supply (there likely will be two, one works as a backup), memory (important for caching) and disks (yes you need special server ones) are proprietary, but easily available and cheap.

Edit: also, don't go 10TB. Reliability goes down bigtime with such tightly packed drives. Look at what's most cost effective for you in terms of $/TB, but keep in mind 4TB is about as big as I'd go. That is unless you want to set up a RAID with redundancy. But then to have effective 300TB you'll need a lot of drives. Remember 50 drives will take a lot of space, no way around it.

 

I had read tnat HD Porn takes a lot of space. This thread confirms it.


As to reliability I would make that decision based on size, manufacturer and model. As explained by @Qrchack 4TB drives are more reliable than a 12TB BUT the 4TB Seagate ST4000DM00, and 3TB Western Digital WD30EFRX should be avoided.




Here is Backblaze's lifetime failure rate data going back to April 2013

 

Last edited: Aug 4, 2019

@Xupito Did you agree with the porn or the failure rate data?

 

I didn't wear my glasses on which I'd need to read those tables.
Does that answer your question?

 

Freaking pervert !