Best storage solution for fast access to about a minimum of 300 terabytes of data?

Discussion in 'PC' started by Remi, Aug 3, 2019.

  1. Infidel

    Infidel Producer

    Joined:
    Jan 18, 2015
    Messages:
    443
    Likes Received:
    147
    One stack of DVD+R's should cover it. $40 for M-DATA quality. nuf said. :shalom:
     
  2. indianwebking

    indianwebking Platinum Record

    Joined:
    Jun 3, 2018
    Messages:
    380
    Likes Received:
    174
    hmm let me just count that hd porn from hdd
     
  3. indianwebking

    indianwebking Platinum Record

    Joined:
    Jun 3, 2018
    Messages:
    380
    Likes Received:
    174
    I don't think he mentioned what he is doing of this much data and how he got this much data in first post. I mean what the hell is he doing? stealing free HDD from akamaihd servers or what ?
     
    • Interesting Interesting x 1
    • List
  4. jhn

    jhn Ultrasonic

    Joined:
    Dec 15, 2016
    Messages:
    56
    Likes Received:
    23
    sounds plausible if all his porn is 4k & he has no "type"
     
  5. rah

    rah Ultrasonic

    Joined:
    Jan 22, 2018
    Messages:
    107
    Likes Received:
    39

    must be all in 4k format :)
     
  6. tzzsmk

    tzzsmk Audiosexual

    Joined:
    Sep 13, 2016
    Messages:
    3,170
    Likes Received:
    1,941
    Location:
    Heart of Europe
    Synology RS2818RP+ with RX1217/RP+ can handle 336TB with 28x 12TB drives - be aware 10Gbit isn't included, but you get PCIex expansion slot for copper or fibre 10GbE adapters you wish,
    more expensive RS4017xs+ will perform better, but costs noticeably more;
    with Synology Hybrid RAID 2 (with 2 disk fail tolerance), that should be a solid solution
     
  7. Qrchack

    Qrchack Rock Star

    Joined:
    Dec 15, 2015
    Messages:
    798
    Likes Received:
    338
    Location:
    Poland
    Q1 2018 is kind of meaningless, really. That's just for 3 months, so lots of drives have not failed yet and show a 0.00% failure rate.
    Now we're talking. One thing you've got to remember though: all disks will fail, that's guaranteed. If OP wants to go with server gear, then setting up a proper RAID with redundancy means a disk or two failing is *not* anything to be scared of, actually it's to be expected in several years. For a long-term strategy, while reliability matters, you also have to consider the cost/availability versus reliability of the drives. If it's cheaper to replace the "worse" drive than to buy "better drives" with 0.5% less chance of failure per year, I'd still go with the cheaper drive. Which is why you see Blackblaze using Seagate ST4000DM000 4TB the most - 30,941 - twice as much as the more reliable HGST. Also a lot of the drives listed have a small sample size - they had just 45 of Toshiba MD04ABA500V and 2 of them failed, or WDC WD30EFRX - 22 of 180 failed. That gives a huge percentage failure rate, but the evidence is not as solid as with the drives they had 30,000 of. Also worth mentioning, the Seagates have been working for far longer - see "Drive Days" and divide by drive count. That gives you an average age of 1230 days ~= 3.5 years. So higher failure rates are to be expected. Meanwhile the WDC WD30EFRX has an average age of just 689 days (less than 2 years) and the failure rate per year is already 6.47%. So the Seagate drives are not as bad as you'd think. There's a reason why they're using Seagate the most.
     
    • Agree Agree x 1
    • Interesting Interesting x 1
    • List
  8. The Pirate

    The Pirate Audiosexual

    Joined:
    Dec 20, 2018
    Messages:
    5,183
    Likes Received:
    4,405
    Location:
    NOYMFB
    Another way of putting it is that the Seagates 4000D are NOT as good as the HGST 4040BLE640.

    Bro you got this all wrong. Your time calculation is incorrect. Drive Days it is not what you think.I was going to write an equation but i will illustrate it to you in simpler terms.

    Let us say that we have 20 Seagates and 10 HGTS. In a 30-day month period the amount of Drive Days will be 600 for the Seagate and 300 for the HGTS. That does not mean that the Seagates operated twice the amount of days. Both are running in parallel. Both run 30 days nor 600 nor 300.

    In fact, the first table shows just a quarter. From 1/1/2018 to 3/31/2018. That is only an actual 90-days period. Not the number reflected by Drive Days.
     
    Last edited: Aug 5, 2019
    • Interesting Interesting x 1
    • Useful Useful x 1
    • List
  9. Gyro Gearloose

    Gyro Gearloose Audiosexual

    Joined:
    Jul 8, 2019
    Messages:
    4,237
    Likes Received:
    1,846
    Location:
    Germany
    he does video/graphics and audio production...lifetime output or so....a busy guy
    graphics stuff is huge amount of data i think..
     
    • Agree Agree x 1
    • Interesting Interesting x 1
    • List
  10. Gyro Gearloose

    Gyro Gearloose Audiosexual

    Joined:
    Jul 8, 2019
    Messages:
    4,237
    Likes Received:
    1,846
    Location:
    Germany
    the hdds of seagate,hgst,toshiba,texas are all good...but its always 50 50....monday devices,bad batch,bad version,bad transport...
    i had no luck with amazon and hdds..i did sent 3 hdds back..

    but yea hgst is maybe best
     
  11. Legotron

    Legotron Audiosexual

    Joined:
    Apr 24, 2017
    Messages:
    1,908
    Likes Received:
    1,841
    Location:
    Hyperborea
    All HGST here, they also come in verbatim model, just check them before you buy

    Edit: Verbatim, and not toshiba
     
    Last edited: Aug 5, 2019
  12. Qrchack

    Qrchack Rock Star

    Joined:
    Dec 15, 2015
    Messages:
    798
    Likes Received:
    338
    Location:
    Poland
    Yes, that's why I divided drive days by number of drives for my calculation. 600 drive days / 20 Seagate drives = 30-day month period, 300 drive days / 10 HGST drives = 30-day month period. Again, Seagate ST4000DM000 came up with 1230 days (per drive using formula just mentioned) and WDC WD30EFRX with 689 days (also per drive). This is an important factor when gauging the credibility of the results.

    If you have any issues with the formula, please demonstrate a better alternative than the average value I'm using by dividing the sum by the number of occurrences. The point I'm making is Seagate ST4000DM000's 2.90% AFR is proven with more confidence than WDC WD30EFRX's 6.47% AFR. For what it's worth, the WDC could actually be better or worse in terms of failures, but the point stands that they're using almost exclusively Seagate and HGST. I did the math for you:
    upload_2019-8-5_22-14-13.png
    As you can see they have over 3 times more Seagate drives than HGST, while Toshiba and WDC drives were just "let's see if those could be useful for us" kind of tests. That's over April 2013 - March 2018. If other brands were more cost effective (in terms of both price and how quickly they fail and need to be replaced), believe me, they would already have switched to using them.

    Which is why I told you it's not useful at all and was wondering why you even included that table. There were newly introduced disks there that have not yet failed so show an AFR of 0.00% which is definitely not correct and shouldn't be even taken into consideration. By using the whole dataset, we're decreasing the probability of error by averaging both years-old and newly installed disks together. So the results don't get skewed into either making us think the drive is worthless and fails all the time (when looking at old disks) or that the drive is perfect (when the disk is a month old).

    So, yeah. Your argument was just invalid. By all means, if you come up with something better, show it.
     
  13. The Pirate

    The Pirate Audiosexual

    Joined:
    Dec 20, 2018
    Messages:
    5,183
    Likes Received:
    4,405
    Location:
    NOYMFB
    @Qrchack I'm not going to turn this into a pissing contest. All of this started when I said that the HGST was more reliable than the Seagate. Visit backblaze and you WILL read exactly what I said. Even though it is corroborated by the data, you continue to contradict it. You even go as far as to say that the Quarterly data is not useful. In the IT world, we use Backblaze as a reference. All of its data serves a purpose. It goes way beyond those 2 tables I posted. Indeed, several days per week I will spend a couple of hours just going over it in order to make the right decisions for my clients. If you visit the Backblaze site you can read its extensive archives, learn how to properly interpret its data, and put it into context.
     
  14. Qrchack

    Qrchack Rock Star

    Joined:
    Dec 15, 2015
    Messages:
    798
    Likes Received:
    338
    Location:
    Poland
    Yes, HGST was more reliable. But you can see them using Seagate a lot more. Do you have any genius ideas why they would use 3/4 of the worse drives and 1/4 of the HGST (more reliable drives)? No drives last forever and numerous factors should be considered, including performance, price, availability and reliability. You have to engineer that stuff to fail gracefully, so you can afford disks failing, because they will and that's for sure.

    The plug with "in the IT world", was that supposed to be a "if you were an IT guy you would have known" sort of reference? Because I'm perfectly aware of who Backblaze are and of their stellar reputation. I have been following their data for the past 2 years. I have also been studying CS for the past 4 years, currently finishing my bachelor's thesis, and am perfectly aware of factors at play with enterprise storage. And yes, quarterly data is not useful. Newly introduced drives are almost certain to not fail in that short of a timeframe, so you get no reliability data in that regard - or worse, with your attitude, you assume they are perfect and have a 0% failure rate.

    I actually use two HGST UltraStar 7K4000 3TB model HUS724030ALE641 drives in my current desktop and they are workhorses. But I am aware they're not the absolute best for every workload. You can get Seagate Barracuda (2016) ST3000DM008 3TB and it's effectively about 30% faster while being pretty much the same price in my area. My point is OP should do his own research and not just go for the "most reliable" drive, because it's just a false sense of security. No matter what you choose, it will fail eventually. You should embrace that and provide proper redundancy for your setup. And it's wiser to pick drives that are cheaper and/or easier to get in the area you live in as opposed to getting that exact model number you found on the Internet that has the lowest failure rate of them all and having it shipped internationally to you just because you think you're now protected for years to come.

    Edit: Also, I wouldn't consider showing you didn't understand the Drive Days metric, explaining the methodology I used in detail, and asking for a better method from you a "pissing contest". But if you think addressing this is not worth your time, hey it's your call.
     
    Last edited: Aug 6, 2019
  15. The Pirate

    The Pirate Audiosexual

    Joined:
    Dec 20, 2018
    Messages:
    5,183
    Likes Received:
    4,405
    Location:
    NOYMFB
    Bro you just dont know when to stop, do you? If you had been following Backblaze as you stated you do, you would have known the answer to your genious question.
    The ONLY reason why they were using more Seagate was financial. According to them, the failure rate differential beteween the Seadate and HGST did not justify spending the extra money on the better drive which is the HGST. They further explain that with the prices of 4TB drives getting lower they decided to purchase some more HGTS. Now, I leave you with the latest Quarterly data. Yeah, I know. According to you it is useless because newly introduced drives are almost certain to not fail in that short of a timeframe, so you get no reliability data in that regard. Really? There is a higher failure rate during the first 2 months of usage than in the 8 to 16 month of a drive life. It is time for me to put an end to this back and forth. I certainly will be wasting my time by continuing to do so.

    Screenshot_2019-08-05 Backblaze Hard Drive Stats Q1 2019 .png

    Screenshot_2019-08-05 Backblaze Hard Drive Stats Q1 2019.png
     
    Last edited: Aug 6, 2019
  16. Qrchack

    Qrchack Rock Star

    Joined:
    Dec 15, 2015
    Messages:
    798
    Likes Received:
    338
    Location:
    Poland
    Which is exactly why I said HGST is better but OP should check which one makes more sense financially, since HGST will eventually fail as well.

    Edit: Also, don't be so quick with your definition of "better", the Seagate drives are one of the fastest money can buy and leave HGST behind.
    The quarterly data includes disks that have been sitting in the pool for years, too. But the failure results are skewed by newly introduced drives, if what you say is true then the only difference is the results are biased the other way around.
     
Loading...
Similar Threads - Best storage solution Forum Date
The best and unlimited free file storage is not what you would expect (Telegram) Internet for Musician Nov 16, 2023
Best Singing Course - Which one? Education Monday at 12:20 PM
The Best Virtual Drummer for Ludwig/Beatle's ERA Kit Software Mar 20, 2024
Best (convenient) ducker plugin? Software Mar 17, 2024
Best Virus soundbanks for industrial, metal, EBM, gothic ... Instruments Mar 11, 2024
Loading...