Best way to store large files

Discussion in 'Software' started by Nimbuss, Mar 23, 2016.

  1. Nimbuss

    Nimbuss Platinum Record

    Joined:
    Nov 21, 2015
    Messages:
    235
    Likes Received:
    162
    Location:
    Africa
    Good day all

    I currently have Omnisphere 1 in 6 parts on my external hard drive which is about 40Gb (6gb a file) compressed. I will be extracting and installing on my main drive but my main question is:

    - What would be the best method of backing up the 6 files? Eg: Winrar, 7 zip etc..
    - Would compressing these files (separately) to fit on 4.7gb disks create problems in future (i've got a lot of 4.7gb disks)?
    - Could I compress all 6 files into a smaller file (I would keep it on my external drive)

    I don't know much about compressing (other than extracting) and i'd love to hear any options you guys may have

    Thanks in advance :shalom:
     
  2.  
  3. Xupito

    Xupito Audiosexual

    Joined:
    Jan 21, 2012
    Messages:
    6,956
    Likes Received:
    3,830
    Location:
    Europe
    First decompress your files until you have the .iso files. Make a compression test, if winrar shows a low compression ratio (>90%) it means the data files are already compressed, so there's no point in compressing them.

    You can easily use Winrar to make DVD sized volumes. If the data is already compressed disable the compression (store mode). It doesn't hurt to add a %5 recovery record which means you can repare until 5% of corrupted data.

    7z is similar to Winrar. In fact it can compress more, but winrar is the king of usability. An alternative would be an imaging soft like Acronis backup / trueimage. Or Aomei, though hasn't been tested as much as acronis.
     
    • Like Like x 1
    • Interesting Interesting x 1
    • List
  4. junh1024

    junh1024 Rock Star

    Joined:
    Jul 28, 2011
    Messages:
    1,396
    Likes Received:
    432
    Dual-layer DVD-Rs & burners exist. (~7,900MB for DL)
    Dual-layer BD-Rs & burners exist. (~23,098MB for SL or ~44GB for DL)

    I have one drive that can do the above.
     
    • Interesting Interesting x 1
    • List
  5. lerkjurk

    lerkjurk Platinum Record

    Joined:
    May 22, 2014
    Messages:
    403
    Likes Received:
    297
    Location:
    Behind you
    i agree with some of what @Xupito say, but even if compression does not help i think its best to multi part rar to about size of 100mb it make copy and move go quicker in future

    i would not be surprise if burning 100mb part goes faster than burning single 4.3gb file to dvd too

    if i am like you and have many dvds, i would compress (even at little compression Fastest setting in winrar) with 100mb part with recovery record, i would burn two sets if it is important because dvds can be scratched, and some of them start to split after time

    does 7z really beat 128mb best compression in winrar5 ????
     
    • Interesting Interesting x 1
    • List
  6. Nimbuss

    Nimbuss Platinum Record

    Joined:
    Nov 21, 2015
    Messages:
    235
    Likes Received:
    162
    Location:
    Africa
    Would I have to set the ' split to volume, size ' to 100mb to achieve this? And would it create the files one usually sees like .r00, r01 etc?

    I could use your above settings and create a custom volume right?

    I remember using one of these towers once. Good times
    [​IMG]
     
    Last edited: May 2, 2016
  7. Xupito

    Xupito Audiosexual

    Joined:
    Jan 21, 2012
    Messages:
    6,956
    Likes Received:
    3,830
    Location:
    Europe
    That's right, only for the multipart and recovery record features is worth using Winrar. What I don't understand is your preference for 100MB parts. This was intended for the extinct Iomega ZIP 100MB drives. For burning it doesn't matter the size of number of files, that's for sure.

    Yes, the 7z format is another compression algorithm, newer and more efficient that the rar (I think that is the same used in Unix .bz2, but not sure). Some people complains that Winrar 5 has few important new features, because the most important is simply allowing a larger disctionary (128 MB).

    When I say more efficient I mean it can compress more, I don't know how it compares to the rar format in velocity for the same compression level. One of the goodies of WinRar is that it's very optimized, it takes the best of all CPU cores.

    Well, the modern Winrar versions use the .part01.rar, .part02.rar, etc nomenclature, but yes, this is the option you've to set.

    I could gladly share my Winrar settings, but there must be hundreds of tutorials on the net. PM me if you want my settings or I can prepare ones for 100MB volumes and other settings of your choice. Winrar has an option to export/import settings to one file.

    Finally, about BD and DVD-DL discs/drives, they're expensive but the BD worth it if you burn a lot of data. I never recommend using DVD-DL discs because they are very slow and very expensive.
     
  8. Nimbuss

    Nimbuss Platinum Record

    Joined:
    Nov 21, 2015
    Messages:
    235
    Likes Received:
    162
    Location:
    Africa
    No need too, you've shown me more than I asked for, Thank you :bow:
     
  9. Diabulus in Musica

    Diabulus in Musica Platinum Record

    Joined:
    Jun 7, 2011
    Messages:
    761
    Likes Received:
    195
    Location:
    AudioSex
    Nimbuss if you have a fast connection think about a cloud storage service...
     
    • Interesting Interesting x 1
    • List
  10. HPF

    HPF Kapellmeister

    Joined:
    Jun 23, 2012
    Messages:
    201
    Likes Received:
    56
    Location:
    Block 4
    rule of thumb:
    7za is good for executables (bad choice in this case)
    RAR is good for multimedia
    RAR5 is not better in multimedia compression than RAR! You might save 2 megs on one file while wasting 2 megs on another, RARv5 isnt as compatible as RARv3 - in the end there wont be any benefit at all

    i tested recompressing my akai library to rar5 and noticed a slight gain on some and a small increase on others. In the end i'd even waste diskspace using RARv5 over RARv3

    small benchmark of a 300m big Steam DB container
    262m 7za.mx5.7z (normal compression)
    262m 7za.mx9.7z (ultra)
    240m rar3.m3.rar (normal)
    240m rar3.m5.rar (best)
    236m rar5.m3.rar (normal)
    235m rar5.m5.rar (best)

    save your time and do not use BEST COMPRESSION but normal (-m3 switch) - avoid SOLID archive creation and consider using the RECOVERY RECORD option (-rr switch), just in case some bits go mad and in the end do always check the archives created (-t switch)

    so the commandline would be like this : rar a -ep1 -m3 -rr -t ArchiveToCreate.rar FolderToCompress

    your approximate savings compressing the files should be 20% using RAR / 21% using RAR5 / 10% using 7zip

    If you really want to backup the data on DVD-R(-DL) you should consider using the terminal commands as with it you can create parts of different sizes to suit your needs.

    rar a -v1024m -v1024m -v1024m -v1024m -v354m -v1024m ... .etc... (afair i used 4450m per single layer ROM)

    resulting
    [DVD1 4450m + filesystem overhead]:
    part1-4 (1g)
    part5 (354m)
    [DVD2]:
    part 6-9 (1g)
    part10 (351m)
    etc ...

    if you wonder why not using 4450m as part size : due to compatibility with filesystems - joliet wont handle it, fat wont handle it, etc...

    as filesystem on dvd-rom you should use UDF and additionally joliet for compatibility

    after burning you have to test the written data or leave the whole idea and backup stuff on another harddrive - its pointless archiving stuff on dvd as you cant tell when the disc becomes corrupted as the quality differs - and real quality media is way more expensive than harddisk space - and count in the time preparing the data + burning the stuff at reasonable speed + testing written data et etc ... and if one disks fails at later point its all wasted time. Who knows when DVD reader and writers become an unaffordable relic on pc

    last thing to consider is: cant you just backup the images as they are without compression? 20% isnt that much considering nowadays harddisk capacities.
     
    Last edited: Mar 25, 2016
    • Like Like x 1
    • Interesting Interesting x 1
    • List
  11. ArticStorm

    ArticStorm Audiosexual

    Joined:
    Jun 7, 2011
    Messages:
    7,132
    Likes Received:
    3,436
    Location:
    AudioSexPro
    buy new 2,5HDD 500GB or more and store there important Audio Stuff, use it when you need it, otherwise dont turn the HDD on.
     
    • Like Like x 2
    • Agree Agree x 1
    • List
  12. lerkjurk

    lerkjurk Platinum Record

    Joined:
    May 22, 2014
    Messages:
    403
    Likes Received:
    297
    Location:
    Behind you
    when i move or copy large amounts of data from one drive to another i find large files (multi gb) go slower than smaller files, but too small, like .mid (4kb) go even slower, i find around 100mb move along steady speed
     
    • Interesting Interesting x 1
    • List
  13. midi-man

    midi-man Audiosexual

    Joined:
    Sep 25, 2013
    Messages:
    1,592
    Likes Received:
    796
    Yes I remember those days also. Thank god for NAS Devices.
     
  14. Ankit

    Ankit Guest

    When many Small files(like 4kb each) makes up 100mb, there will be too many address names to process, takes more cpu than 1 100mb file.
     
  15. Cardamom

    Cardamom Platinum Record

    Joined:
    Jan 9, 2015
    Messages:
    203
    Likes Received:
    227
    I use 3.5 floppy discs for large sample libraries.

    Take something like Kontakt Session Pro Strings, which is about 32Gb, then use Winrar to split everything into 1.44mb files. At the end of the job, which only takes a few days, you'll have to have 22,223 floppy disks ready to go. Load them up one at a time (aaah ... love that loading sound. Ehnk-ehnk-ehnk ...) and in about 7 months, you're all done! Easy as cake! When your new computer dies, it'll only take another 7 months load-time to put all the floppy-disk rar files onto your new Solid State Drive and then you can decompile it back into your fantastic - can't wait to play with it - sample library.

    Can't recommend this more strongly! :rofl::wink:
     
    • Funny Funny x 2
    • Like Like x 1
    • List
  16. Xupito

    Xupito Audiosexual

    Joined:
    Jan 21, 2012
    Messages:
    6,956
    Likes Received:
    3,830
    Location:
    Europe
    Makes sense. Have you tested TeraCopy?
     
  17. Infidel

    Infidel Producer

    Joined:
    Jan 18, 2015
    Messages:
    443
    Likes Received:
    147
    Cardamom, I bet you love your futuristic 3.5" floppy, (No I'm not talking about that you sick bastards) when the rest of us are still on 5 1/4".
    Next thing you'll say is Star Wars will be a big hit and have a sequel or two. :mad:
     
  18. lerkjurk

    lerkjurk Platinum Record

    Joined:
    May 22, 2014
    Messages:
    403
    Likes Received:
    297
    Location:
    Behind you
    next time i must make copy of large data i will see if i can xtract some files to see with tera copy
     
Loading...
Loading...