Best storage solution for fast access to about a minimum of 300 terabytes of data?

Discussion in 'PC' started by Remi, Aug 3, 2019.

  1. Remi

    Remi Member

    Joined:
    Jan 9, 2019
    Messages:
    99
    Likes Received:
    16
    Admittedly, it's not all for music. I also need to store a huge amount of video footage. But am more concerned with fast access to samples and Kontakt Libraries, which I simply can't store on my HDD. Currently, I have 20 external hard disks, portable ones, that sometimes do the job, sometimes don't. and I'm worried they're going to blow up one day. So I have a need for massive back-up AND fast access (for samples and Kontakt libraries┬░. In addition, I can only drive a couple of USB external hard disks at a time (sometimes, only one when I used the 4TB disks I have) . Finally, they're kind of sluggish too. Performance varies from disk to disk, but they're not ideal for fast concurrent access to multiple Kontakt libraries. In short, my current storage solution is piecemeal and inadequate.

    Would you be able to describe the ideal non-cloud storage solution for this kind of capacity, which is scalable and fast enough? I would also appreciate a gentle and patient explanation of how you access large external storage on your PCs when I am told I can only connect to about 8TB (two 4TB external disks with USB connections). Is that a universal USB restriction? What is it I am not getting here?

    Finally, I would also appreciate an explanation of how you allocate resources to SSD, HDD and external storage. Do you reserve the limited SSD capacity to the OS, or the OS and the main software you run (in my case DAW, Photoshop, After Effects Premiere etc) and reserve the HDD for samples and libraries? And when you run out of HDD space, do you keep deinstalling and reinstalling different libraries to the HDD or have you found an efficient way to access large storage and fast access externally?

    I know budget obviously plays a great part in the choice of setup. Let's pretend it's not an issue here so I can begin to understand the ideal setup, and I'll work my way down from there.

    As always, thank you for your suggestions and comments.
     
    • Like Like x 2
    • Interesting Interesting x 1
    • List
  2.  
  3. Haliax

    Haliax Kapellmeister

    Joined:
    Feb 2, 2019
    Messages:
    88
    Likes Received:
    74
    Location:
    Inside the Outside
    Are you taking 300TB because you want to RAID the disks? If not, for that amount of data you will need backup
     
    • Like Like x 1
    • Agree Agree x 1
    • List
  4. Remi

    Remi Member

    Joined:
    Jan 9, 2019
    Messages:
    99
    Likes Received:
    16
    Well, that's the question. In fairness, the music stuff is less than 50TB right now. The video footage and other files I keep account for about 120TB and growing. The rest is estimated scalability requirements. So I guess I only need RAID for the music stuff, but eve then, that's a lot of storage for direct access.

    Regardless of the actual size, if we assume that your internal HDD is not enough, how would you set up storage for the "direct access" stuff (largely sample libraries)?
     
  5. korte1975

    korte1975 Rock Star

    Joined:
    Sep 3, 2014
    Messages:
    959
    Likes Received:
    449
    Location:
    Ireland
    300 Terrabyte ? who are you ? Mr Spotify himself ?
     
    • Funny Funny x 2
    • Winner Winner x 1
    • Interesting Interesting x 1
    • List
  6. junh1024

    junh1024 Rock Star

    Joined:
    Jul 28, 2011
    Messages:
    1,225
    Likes Received:
    340
  7. LuckySevens

    LuckySevens Platinum Record

    Joined:
    Sep 23, 2012
    Messages:
    487
    Likes Received:
    221
    Location:
    3rd planet from the fireball...
    At this point you need to set up your own data server with back-up drives because as they begin to fail (2-5 years) it will become a total nightmare to trace and recover.
     
  8. Dimentagon

    Dimentagon Kapellmeister

    Joined:
    Aug 7, 2018
    Messages:
    75
    Likes Received:
    58
    • Interesting Interesting x 1
    • Useful Useful x 1
    • List
  9. Haliax

    Haliax Kapellmeister

    Joined:
    Feb 2, 2019
    Messages:
    88
    Likes Received:
    74
    Location:
    Inside the Outside
    With this amount of data you need to think about redundancy and resilience. Smaller drives are less error prone than larger capacity drives, so you are talking hundreds of drives here. If you want to ensure your data remains available and free from corruption and drive failures you will need two copies for each drive.

    Have a look at Ceph as an object store solution, this takes redundancy into consideration as a solution.

    Windows fails when indexing vast amounts of data, have a look at an Elastic cluster to solve this.

    tl;dr

    You will need 900TB to ensure your data is safe, so you will need to think enterprise solutions rather than home solutions
     
  10. Xupito

    Xupito Audiosexual

    Joined:
    Jan 21, 2012
    Messages:
    2,476
    Likes Received:
    1,348
    Location:
    Europe
    No matter how I look at it, it's too much for hardware designed for users, even for power ones.
    Meaning enterprise-level hardware and perhaps setup, like someone told.
    If you want fast access, that would mean almost an small datacenter, so to speak.
     
    • Agree Agree x 3
    • Like Like x 1
    • List
  11. SineWave

    SineWave Audiosexual

    Joined:
    Sep 4, 2011
    Messages:
    2,974
    Likes Received:
    1,975
    Location:
    The darkest pit of my mind
    If "only" up to 170TB is enough for your needs, with a DX517 expander, this Synology DS1019+ NAS [network attached storage] could be the solution for you. I have nothing but praise for Synology NAS devices. :wink: However, I would definitely build a RAID 3 or 4, or possibly 5 for at least some redundancy [one disk can fail]. RAID 2 is the most secure, though, and the more disks there are, the better the chances one of them will break.

    What you are doing now, with all these external HDs, is actually pretty safe. But it is admittedly inconvenient for usage, of course.
     
    • Like Like x 2
    • Interesting Interesting x 2
    • Winner Winner x 1
    • List
  12. indianwebking

    indianwebking Producer

    Joined:
    Jun 3, 2018
    Messages:
    202
    Likes Received:
    75
    who are you? what are you doing? 50tb of music stuff you mad? reply me sir
     
  13. Xupito

    Xupito Audiosexual

    Joined:
    Jan 21, 2012
    Messages:
    2,476
    Likes Received:
    1,348
    Location:
    Europe
    You really saved the whole thing with the "sir" ....
     
    • Funny Funny x 4
    • Agree Agree x 1
    • List
  14. Gyro Gearloose

    Gyro Gearloose Audiosexual

    Joined:
    Jul 8, 2019
    Messages:
    1,840
    Likes Received:
    736
    Location:
    Germany
    how bout read his post...:hillbilly:
     
  15. mapjay

    mapjay Member

    Joined:
    Sep 3, 2015
    Messages:
    94
    Likes Received:
    13
    UnRAID server for storage. a few small RAID setups with SSD for fast access/current work, esp for video - since video is 5x the amount of data/speed for rendering/editing, over audio. UnRAID is prob cheapest (use any size/brand drives, enterprise not needed) large server solution
     
  16. Willum

    Willum Rock Star

    Joined:
    Jun 13, 2011
    Messages:
    644
    Likes Received:
    324
    Lol, thats a shed load of porn you've collected there. :)
     
  17. Qrchack

    Qrchack Platinum Record

    Joined:
    Dec 15, 2015
    Messages:
    511
    Likes Received:
    199
    With a requirement of 300TB, forget USB. You want a NAS of some sort. Probably the cheapest way to achieve your goal would be to look up some server gear on eBay. They always need cutting edge, so really good stuff that's just 2 years old and way more powerful than a regular desktop PC becomes worthless for them and can be had for cheap. Make sure you have a place to put that stuff though, you don't want to have a server in your room. That stuff is loud (about as loud as a jet taking off) and the room you put it in will heat up *a lot*. The parts such as power supply (there likely will be two, one works as a backup), memory (important for caching) and disks (yes you need special server ones) are proprietary, but easily available and cheap.

    Edit: also, don't go 10TB. Reliability goes down bigtime with such tightly packed drives. Look at what's most cost effective for you in terms of $/TB, but keep in mind 4TB is about as big as I'd go. That is unless you want to set up a RAID with redundancy. But then to have effective 300TB you'll need a lot of drives. Remember 50 drives will take a lot of space, no way around it.
     
    • Agree Agree x 2
    • Interesting Interesting x 1
    • List
  18. The Pirate

    The Pirate Audiosexual

    Joined:
    Dec 20, 2018
    Messages:
    2,373
    Likes Received:
    2,072
    I had read tnat HD Porn takes a lot of space. This thread confirms it. :hillbilly:


    As to reliability I would make that decision based on size, manufacturer and model. As explained by @Qrchack 4TB drives are more reliable than a 12TB BUT the 4TB Seagate ST4000DM00, and 3TB Western Digital WD30EFRX should be avoided.


    2018-05-0116-20-51.jpg

    Here is Backblaze's lifetime failure rate data going back to April 2013

    2018-05-0116-34-34.jpg
     
    Last edited: Aug 4, 2019
  19. The Pirate

    The Pirate Audiosexual

    Joined:
    Dec 20, 2018
    Messages:
    2,373
    Likes Received:
    2,072
    @Xupito Did you agree with the porn or the failure rate data?
     
  20. Xupito

    Xupito Audiosexual

    Joined:
    Jan 21, 2012
    Messages:
    2,476
    Likes Received:
    1,348
    Location:
    Europe
    I didn't wear my glasses on which I'd need to read those tables.
    Does that answer your question? :rofl:
     
  21. The Pirate

    The Pirate Audiosexual

    Joined:
    Dec 20, 2018
    Messages:
    2,373
    Likes Received:
    2,072
    Freaking pervert !
     
Loading...
Similar Threads - Best storage solution Forum Date
The Best of RiFF RAFF aka JODY HiGHROLLER humor Today at 9:08 PM
best oriental workstation Instruments Nov 23, 2019
What's the best library for string runs? Working with Sound Nov 22, 2019
Best online multimedia porfolio portal for presenting your work Lounge Nov 22, 2019
The best lecture explaining ray tracing... Lounge Nov 12, 2019
Loading...