Would any of you guys know if anything is documented regarding music production with a ReFS system?

It interests me. But I don't know enough about it just yet.

 

For the moment it hasn't been thoroughly tested. I will be building a new music pc for a client around the end of next week based on a Ryzen 9 7900X + 64gb ram and 2 Samsung 990 Pro SSDs, some mechanical drives in there too. I will test ReFS for sure and will tell you what gives. As it is now, there are certain caveats that render ReFS useful only for straight up storage volumes. For example a drive with Kontakt libraries and ReFS integrity streams enabled. Hybrid usage drives like a "work drive" to constantly record onto and store/modify DAW projects etc., for the moment it might be better on NTFS. But i will check this too for sure as it is of great interest to me, mainly due to the robustness of ReFS vs NTFS.
Some tests done by others in the past on mechanical drives, showed ReFS being a little behind NTFS in general performance when the integrity streams feature was enabled, so they would advise to turn it off. In our case and concerning a drive to store libraries and samples, i certainly disagree with this, what you want from a drive that is used mainly for read purposes, is first and foremost to prevent file corruption also known as bitrot. Integrity streams does exactly this preventing your data from corruption and in this particular way it is tons more robust than NTFS.
For those who like technical jargon i will explain what happens. When int.streams is enabled all write operations become allocate-on-write operations. This avoids any read/modify/write bottlenecks so that's an advantage over NTFS. But when used with mechanical drives it can cause frequent fragmentation of file data, actually delaying read ops. SSDs though in contrast to mechanical drives, constantly allocate/move data blocks around to optimize performance and practically it is impossible to cause fragmentation on a healthy ssd, so i think that it will have a negligible impact, if any, on a SSD. We 'll see.
Now, coming from peeps i know and you might very well take this with a grain of salt, MS are working to make ReFS perform better in more scenarios. But it's been at least 5 years with nothing new about it in the performance section. Making it part of a mainstream platform like Win11 though, means there will be more updates for sure. It also implies third party involvement is sooner or later imminent. Now if they get it right, that is something not even the engineers working on it can answer for sure. So stay tuned. Nothing can be more revealing than trying it out hehe.
Cheers
PS: If you guys want photos of the build mentioned i will be doing next week, performance metrics and such, let me know. I'll be happy to provide them.

 

I have just managed to get my PC set up after a house move process and am literally just about to do a fresh format and install of all my internal drives.

My internal drives are:

1. 2TB NVMe OS drive
2. 4TB NVMe drive
3. 4TB NVMe drive
4. 4TB SSD
5. 4TB SSD

I do also have a couple of 36TB external enclosures set up in RAID0, giving me 18TB capacity each, where one enclosure is a clone of the other, and all of my critical and personal data as well as software backups and so on are backed up onto these.

The external enclosures are NTFS, but as I'm doing a fresh format and install of everything on the new drives, would it be worth me considering ReFS for non-Os internal drives?

 

You dont get around much apparently. HARDWARE RAID is the defacto standard by which ALL RAID is measured. Software raid is by definition and comparison slow, tedious and unreliable.

Mickeysoft is notorious for releasing "new" tech wherein that "tech" has flaws (that no one know about until it's too late) that make it not ready for prime time. Personally I stick with what I know works proven over time, and time again. The heavy use environment I speak of is media production be it video, audio, or both combined. I would not be trusting mickeysoft "new" tech on anything I had a vested interest in keeping intact.

 

Not worth it for non-tinkerers / people who just need a long term backup strategy right now.

I'd wait another 3-5 years for ReFS to mature. Presumably there will be (many) more updates to ReFS (booting Windows...). In the past, feature updates to ReFS required reformatting, so that's a hard showstopper until ReFS is "done".

ReFS deduplication and transparent compression is mighty tempting for backups, but is restricted to Windows Server/Enterprise editions I believe.

 

For real. You shouldn't be as aggressive without any real world examples my friend hehehe. I think it's you who needs to get out a bit more. But i won't play this cat and mouse game really.
What you are (probably) speaking of, is leftover clusters from a past era, that the cost of re-deployment is too high for their owners. The majority of NEW servers running Linux ZFS and BTRFS do not rely on hardware raid anymore. And when i say "new" i mean the last 10 years, that's how new. In fact there's people who lost their jobs because they wrongly implemented hard. raid on ZFS servers, which is indeed a recipe for disaster as my man @tzzsmk would say.
Anyway a raid controller consists mainly of a dedicated cpu/memory for the parity calcs etc. With the absurd amount of cores and memory available to modern cpus, it would be highly illogical to depend on a legacy device that at most of the times sits dumb waiting for an error reported by a drive or for a drive to fail completely. So what reliability are you talking about, when the controllers themselves are unreliable for fuck's sake. I would bet my money entirely on any cpu+soft raid rather than a legacy raid controller device.
And speaking about raid controllers, i cannot think of one hardware raid controller that can use S.M.A.R.T. data to predict SSD failure.
A good example of the eclipse of hardware raid is NAS machines. Do you see any dedicated controllers in NAS no matter the scale of the machine? I don't think so. They 're all fine running with Intel, ARM and AMD cpus. Even NAS machines of 100+ drives scale. I dare you to name any manufacturer like Synology, Qnap or whatever that relies upon hardware controllers for their machines.
People who still do hardware raid don't do it for parity. The only do performance (raid 0) and mirroring layers (raid 1) or a combo of these two like raid 10, and this is a whole different story, which i am always game for. And even on this you 'd be hard pressed to find a dedicated controller doing raid 0 or 1 on modern machines. You can see an indication of this decline, in Intel and AMD dropping native cpu support for raid 5 in recent years, something that was standard in the Core2 era.
Be sure i have quite the experience in all types of raid as i was implementing game servers with raid 01 and raid 5 layers some 20+ yrs ago. And i have experimented with all combinations of raid in my machines as well and have experienced first hand the complete death of raid 5 and 6 arrays running on industry standard LSI controllers with top notch SAS drives. Like i said earlier but you didn't get it, can't lecture the lecturer mate. BTDT.
The only reason to do hardware raid in 2024 is because most controllers atm are dirt cheap, as the whole niche of hw raid is on a halt all over the manufacturing industry. You will find them either second hand or as stock products. I will be happily schooled if you can point us to a new hardware controller that does raid 5/6. For example, 10 years ago Avago acquired LSI. Avago is known as Broadcom nowadays of course. The only raid tech i see nowadays from Broadcom is raid 0 & 1 controllers in multi ssd pci-e gen 3 and 4 x8 or x16 cards.
I think it's needless to say get your facts straight again. You seem to ignore the science behind raid tech and file systems in favor of personal beliefs. What you think of as a defacto standard could be true 20-25 yrs ago. But not the norm anymore.
Cheers

 

Last edited: Oct 6, 2024

I do agree. The whole privacy thing is getting more ridiculous with every new Windows version. You can see how these policies have benefited MS since they have recently surpassed even Apple with a 3.21 trillion market cap. Not from sales if you ask me, but ads and data mining. And something that many people ignore, MS is huge on crypto mining too. So why change when they can force their bloatware and even more of it on users, when -as you and tzzsmk and others correctly pointed out- most of them will accept it willingly.
Still, if Win11 is a choice, you can disable both Copilot And Recall. Here's the registry value.
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsAI]
"DisableAIDataAnalysis"=dword:00000001
And really now man, because everybody wants the "latest and greatest", this for me as an independent sys integrator/builder (with very few rich clients), has always given me the hardest time these last years. I am an inch far from becoming a legit certified windows expert with all these modifications we 're supposed to do in order for a Win11 system to run "properly" and privately. Absurd.
Cheers

 

Just for the lols. I remember doing RAID 0 with 2 10000rpm Raptor HDD for the windows partition. The two combined made a noise and vibration totally massive. Not to mention the heat.
It seemed the whole fuckin' PC was gonna take off like a damn helicopter

What could possibly go wrong?

Edited: but hey, Windows loaded fast as fuck

 

Last edited: Oct 6, 2024

Sure do!

 

...and you don't apparently work with big corporate systems, just saying. ZFS and NetApps are all over the place.

 

The thing I hate about NTFS is that the max file size keeps my tracks too short, I can't really express myself to the fullest.
ReFS takes the measly 8 petabytes to 35 petabytes!

 

@taskforce just to be clear so I read all that you've posted correctly.
The resilient file system is not as yet available for a system drive i.e. it won't work on it.
It should work with any audio recording software but nobody can answer that with 100% certainty as yet.
It perpetually file checks for errors and does faster block transfers, allowing for drastically increased file transfers.

A couple of things...
Theoretically, then the increased file transfers in speed should make audio latency from disk to audio improve noticeably, perhaps even seamlessly on really large audio projects. The perpetual checking, more information may be needed. If it uses a miniscule amount of resources to file check, it probably will never be an issue, if not....
From what I can gather, the relatively painless old way of a cmd line convert is unavailable for NTFS to ReFS. This makes it a more lengthy process of backing up all your data, formatting to ReFS and then transferring the data back. I'm assuming that is correct? Not preferable though. I figure they'll eventually work out a more painless way if so.

Good info posts. Thank you.

 

What really rubs me the wrong way about NTFS is how digital it sounds compared to other file systems. You listen to your songs from a FAT32 partition - sounds nice and warm. Copy over to NTFS - sounds cold, brash, digital - hate it. I'm really hoping Microsoft put the analog back in ReFS.

 

Yep exactly like this. And i highly doubt ReFS will ever be able to sustain a desktop WinOS as a c: drive with acceptable performance, due to the basic nature of the file system constantly generating checksums for data integrity. It would have to undergo major modifications+additional new features and admins should have complete control on all (old and new) functionality, turning off and on certain features of the file system at will.

There is penalty on performance. When used with single drive volumes the measured penalty vs NTFS should be in the area of ~10-15% but all measurements i am aware of where done using mechanical drives which behave very differently than a solid state drive. Be sure to check back on this thread as i will post my findings from the testing i will do on a brand new pc, right here in this thread

Yep. Console commands like diskpart can only be used to format to ReFS.
Supposedly 3rd party partition soft EaseUS Partition Manager claim to be able to non-destructively (that is without losing data) convert NTFS to ReFS but i have never tried this plus from the little i saw on their webpage, they say convert but it's actually a format lol. I'm gonna have to investigate to see what gives. Something as transfering a couple of folders with wav and a couple of kontakt libs on a brand new ssd formatted to ntfs and then convert this volume to ReFS and see what happens. But all known guides including MS ones, indicate that format is the only way to transition from NTFS to ReFS.
Lastly again, i would not recommend using ReFS for a recording and active projects data drive. Not atm anyway, as i am not aware of how big the performance penalty is really. Theoretically it should be fine for fast nvme ssds but i really am not sure. Stay tuned
Cheers

 

Fat32 was FATASFUK!

 

I've worked in IT for nearly 40 years now. Usually the people I hear dissing one system over another (except NIS+, that was truly shit) know fuck all about real world computing and build their opinions on synthetic benchmarks and anecdotal evidence from smartarse 12 year olds in chat groups.

 

I want to see the tests to find out if it is as good as you say

 

That refs is a joke
It's.absolitely not how cloud FS are handled ( think GFS, Hadoop in a lesser extent...) Large scale high 99.9999999% avail are block and very simple on purpose, and handled more on client side.
I did some sharded large scale by hand in prehistoric cloud times, using bays and bays of nas.
Raid and those complex architecture would never work at scale, guarantee ( scale= millions of users and petabytes ) a file could be a pointer to a Google of exabyte of data that won't change. It takes some ideas from cloud FS ( like auto scan, auto repair ) and out the fancy word 'petabyte' to look smart. Anyway ntfs is quite old and need to evolve, refs is not all bad and will certainly have some use. competition is harsh ( btrfs ) but misleading that one windows instance could handle exabyte now or anytime is just bullshit.
My own advice is, backup ( three, better five, and one remote ). That's how Google did and yes they're evil now but they have the most invented the distributed cloud, Jeff Dean is a beacon after the keenighan , tompston and Ritchie ( nerdy and not relevant to audio sorry about that )

 

I had a good laugh with that man. Brought back so many memories. My case was almost the same but with a different twist.
True story. Somewhere around 2008, i recall having switched from an aging first gen dual Raptor 74gb/raid 0 to an Intel X-25m SSD 160gb. I was blown away at first by the performance uplift. And yes, finally no noticeable noise! My happiness didn't last more than a couple of months.
To cut a long story short, i had three of these fuckers die on me in 6 months time! Sudden deaths all 3 of them! The frustration was unbelievable. One of my best mates (PHD on computer sciences) was loling at my luck, as his own X-25m was fine and dandy and mine were dying one after the other. Even my supplier would take the piss at me saying "you must be doing something wrong" hahah.
So what to do, i gave up completely on the ssd idea and switched again now to dual VelociRaptor 300gb/raid0. These little drives, although having less vibration than their predecessors as they were essentially a 2.5" drive in a 3.5" enclosure to minimize vibration, they were still noisy as fuck! I remember modding my huge CoolerMaster case's side panels with dampeners and deploying Noctua fans everywhere to cut as much as i could hehe. But the SSD experience was a really bad one for me, initially. I went on to learn as much as i could about ssds, as it seemed like the way to go, but it wasn't until 2012 and the Intel 520 that i tried again to move my own main system to a solid state drive.