I just found that the Ezdrummer midi folder has over 200,000 files. No wonder it takes so long to create midi database every time I open Ezdrummer or Superior drummer.
As far as I understand, it makes slow down the loading speed if there are duplicated files or compressed files in the midi folder.
How do you find them easily&quickly? Is there a specific way to find them?

 

EZ Drummer has its own way to manage the MiDi "database".
Actually its not a real database like mySQl. (those would be much much faster for big collections) but just directories and subdirectories logic. Or at least the actual Midi are not written as a BLOB into a real database. The Midi files are written to a long hierarchy of directories abd subdirectories what makes the rebuilding really s ... l ... o ... w .
(in my case ~17.000 subdirectories and 130.000 Midi- and 20.000 other files)

The actual Midi (infromation) databases (SQLite) are written per collection, e.g. for AVATAR, Metal Machinery etc. into a fille called MIDIdb in those subdirectories.
But there is no general MIDIdb for all of the MIDI files in the collection for quick access and quick rebuild or management.

If you call the MiDi database manager from EZ drummer or Superior drummer they will do a full rebuild and "optimize' the 'database' as much as its possible.

You know the result.

The only way to optimize the MiDi database is to delete defective files and the collections you don't need and run dbManger again.
200.000 files are just too much for that kind of managemant.

 

Last edited: Sep 16, 2022

I compress with winrar and then delete the midi collections I don't use. There's just too much files, it's insane even for many SSDs

 

I'm with Xupito.

To avoid problems with too much small files in every situation, i archive them in the same directory.
And unpack them only if needed.

It helps with my file based backup too.

 

Of course a good idea because you dont have the need to manage so many files. But you need to know wich files you need beforehand.
Did you ever try to burn the whole 'database' as an .ISO image? This may work if EZD or SD don't need R/W access. I haven't tried it, just comes to mind.

Or is there any RAR-on-the-fly-utility that may work as single file an only unpacks when needed?

 

Last edited: Sep 16, 2022

Yeah, I use DISM/WIM for exactly that. Comes with Windows and let's you pack files and folders into a single file that you can mount as a folder. Let's you make changes and commit them back to your file, or you can throw away the changes when you unmount. Or just mount the file read-only to begin with.

- Barebones option: The WIM-tool that comes with Windows is really daunting and command line only.
- 7-zip can open and create WIMs, but it can't compress or mount WIMs.
- PowerISO covers all the functionality of DISM needed for the task.
- GImageX is what I use for mounting and unmounting. It can also create ('capture') and append to WIMs.
- If you need the best compression ratio, then wimlib is 100% the way to go, but it's command line only.

I use XPRESS compression at max level. If there's a choice between solid and non-solid, I'd choose non-solid or a small solid-block size to preserve random access.

 

Last edited: Sep 16, 2022

Nice. I've been looking around for something like this for a while. Precisely PowerIso's propietary .daa image format (compression, password) was the next I was gonna try.
Didn't know you also can do it with WIM files. I knew them only because are used on the Windows installers isos.

 

Ye, that's what I'm curious about. So, how do you sort out the defective files?

 

Oh, I didn't know about that format. Feature set sounds interesting. Would have to test if (de)compression is fast enough!

 

Yep, that's the important part, isn't it. I basically always wanted something easy to mount and fast like a regular iso but with the compression and features of a rar, 7z.

I went down the rabbit hole of the WIM/ESD types of compression and software you posted and wow, even for a mega-geek like me it was huge. Deep deep dive lol.

Apparently It's also used for that less known feature of Windows of compressing certain Windows folders.

 

I have not many MIDI files, but *many* drum samples [it's my obsession] and I make a folder with certain drums an ISO file and mount with Alcohol portable [free soft] to save the space and speed up loading. PowerISO can also make .ISZ [ISO ZIP] files that Alcohol portable can use and in WIndows I use .ISZs. I don't understand how hard is it for these stupid programmers [XLN, Arturia and others with millions of files in installations] to learn how to make proper database files and cut the installation time down to 1/16 [approx. ;)].

I haven't yet found a way to mount .ISZ files in Linux but I use normal ISOs for now. I shall research that...

btw. Alcohol Portable is extremely useful as it can also mount various Akai, Roland, E-MU format CD images to be read by CDXtract. ;) CDXtract often fails to mount these, weird.

 

Last edited: Sep 18, 2022

Here is a DISM GUI :
https://www.vladan.fr/dism-gui-4-0-utility/

 

That's the first candidate I checked out, but couldn't find how to make an ISZ fast. Only found single-core speed. I think .daa is newer but isz is similar.

Well, you need the database, but when a lot of small files is involved the most important is to pack them in some container. Classic samples containers like Kontakt's .nkx, Toontrack's .ocw... oh, wow, they do know how to do it!!

Generally is not a good idea to embed files in the database itself. Though being so small we could make a case with midi files as blob data. But I guess constantly adding official and 3d party midi collections would make the process way more difficult

 

Last edited: Sep 19, 2022

Am I missing something here?
I have 191,000 midi files in that folder, in 25,000 subfolders.
My EZDummer3 loads in seconds, everything is just immediately available.
On an SSD, but it's a really old Win7 system.
After about 10 years of EZDrummer, I never once did any kind of 'optimize'.

 

Yes! Your missing the fact that everyone here seems to find it very important to monitor the loading time of the 'Groove database'? While the program/vst itself loads fast and is instantly usable after the default kit has loaded. I must admit it can take a few minutes to load (from an SSD) on the first start. I have like 185k files in my database.

But I honestly don't see what the issue is here? Unless you find it absolutely necessary you have access to the midi database immediately after you start the program? But lets be honest? How important is it that you need 'instant' access to your groove database? Can't it wait for a minute or two? That's just a small price to pay for all the grooves you choose to load yourself?

Unless your running ez/sd drummer from a conventional sata drive I absolutely don't see any issue here!

 

That's always an excuse for bad programming habits and doing nothing to change oldfashoined file structures. We have always done it that way and we will continue to do it that way in the future; no one complains...
I have literally hundreds of programs installed on different computers, so if each program took a minute or two to start, the workday would be over quickly.

A somewhat more serious answer:
Users today no longer feel like waiting. Even small delays annoy them.
Thus a smoother and more responsive working software is a must today. If you don't pay attention to such things, companies are ruined in the long run. The best/worst example is Avid.

 

I agree that we should not look for any excuse and always try to challenge manufactures for writing better software. But file structures itself have nothing to do with this matter. It has to do with the amount of files (185k+) that need to be scanned.
As of now all these midi files are separate. You may think of a way to incorporate these in a single intelligent database, but....
Most of the midi files are third party. How do you get all these third parties to agree on a single database commitment?
And what about DAW's? We also use midi files to load.

The solution of writing more efficient software for one platform seems obvious. The implementation and making it work on 'all' platforms is not so straightforward.

And yes, you use literally hundreds of programs on your computer. But almost none of them need to load in 185k+ files!

So for now I'll stick to my earlier conclusion. Use EZ/SD drums after it loads and if you really want to use the groove database you'll just have to wait for xx seconds depending on how many midi files you have installed and from what kind of drive you're loading them.

And that users today are very inpatient and want everything done this minute has very little to do with this discussion.

 

Last edited: Oct 8, 2022