Hello folks, I wanted to show you what I've achieved (among other things) with AI cloning. What I did was basically take samples of violin loops and let the AI learn from them. Then, I played an improvised melody on my electric guitar and cloned it using the AI violin model I generated during the cloning process. The result is the audio I'm sharing here. It has some effects and panning to appreciate, and I also made some adjustments in the violin to separate it a bit from the guitar.

The point of sharing this, besides showing you what I've done, is to highlight the creativity that can be unleashed with AI. I don't mean to say that I'm creative myself, but rather that one can use this as a tool to clone instruments and even combine them. I know there are software programs that can achieve this, but they are not always as precise, like MIDI devices, for example. Moreover, you often have to fix the notes afterwards. Another interesting aspect is that some instrument emulation software requires additional accessories to add more realism. Well, with AI, you can bypass all of that because with just a few sound samples and a few minutes of training, you can achieve good results that will enhance the creativity in your music.

 

Sounds dope. E-bows watch your back...

 

Nice result, but wouldn't this be easier?
https://www.jamorigin.com/

 

"Thank you, tiger. Yes, I always use it, but despite being very good, it creates a lot of small notes that I have to keep deleting afterwards. Also, there are things it can't interpret accurately based on the instrument and its attributes, such as legatos, staccato, etc. So, I have to edit or look for a more specific instrument. Moreover, I can't 'clone' it. For example, I don't know if you remember, but many times I've asked here how I can make my bass or guitar sound like another model, and we talked about layering both sounds, but it ends up with undesired results. On the other hand, in cloning, if you clone a bass, your bass, and feed the AI with a specific bass sound, you obtain that bass with the same timbre. Cloning has the ability to capture the timbre of an instrument and then apply it to your instrument. I don't know of any VST that can do that. Anyway, Jam Origin is still a good tool, but when it comes to cloning, AI has surpassed all the VSTs and VSTis I know. But, please note, I'm talking about the cloning capability, not necessarily the sound quality

 

Yep, this is the shit @No Avenger . As far as I understand it, this is like an EQ matching on steroids. Capturing the full timbre of the cloned instrument.

Would love to see more examples @Ryck . How about a Cello?

 

Ah, yes, in that regard AI is superior.

 

@No Avenger @Xupito Yes, it's not a replacement for what already exists, like Jam Origin, because when you play your guitar and run it through the instrument, you're getting the pure and real sound of that library. The AI, on the other hand, "clones" the sound or timbre, I'm not exactly sure how it does it, I'm still researching. It seems to use Python libraries like Librosa, PyTorch, I've also seen Ffmpeg and other dependencies mentioned, but I'm not a programmer, just an enthusiast. I wish I could program, I'd love to, but I'm not good at math. But getting back to the point, yes, in terms of sound quality, traditional sound libraries are still superior, but when it comes to cloning, they fall short. It's worth noting that the samples I've uploaded were obtained from loops and not sound libraries. So, if you have some "monophonic" instrument loops (they must be monophonic), you can clone voices and instruments, although the RVC (which is the method I use) is primarily designed for voices, it works quite well with instruments that fall within that range.
I'm going to try with a cello and upload it right here. It sounds interesting, and the sound is beautiful. I should mention that since the pre-trainers are designed for voices, they don't capture very low notes. So, in the Discord channels I've found so far, they tell me it's more intended for voices, not instruments. But you know me, I'm persistent in this subject, and I'll keep researching. In fact, I would love to open up the pre-trainers and modify them, but I haven't found information yet, and contacting the creators is difficult. Many GitHub projects are actually discontinued. I've cloned the saxophone (it's marvelous), a violin, a trumpet, and now I'm going for an alto flute or piccolo, but I haven't found loops yet. I found something in the London Symphonica; they offer free samples. The issue is that they are individual notes, and I believe the cloning program works better with loops than with individual notes since it can "understand" the source's behavior (originally for voices, but it also applies to instruments). I'll be sharing other sounds with you soon.

Do you remember Spleeter? It was before Izotope and other programs came out with this function. It was similar to these open-source projects, only available in places like GitHub. Later on, it got incorporated into programs. I have a feeling something similar will happen in the near future with some programs, meaning they will include a function for cloning voices. The challenging aspect for developers, who I'm sure are considering incorporating this, is that the cloning process requires a lot of GPU power. So, in order to clone a voice or instrument, you would need a good graphics card. Perhaps releasing a product that not everyone can use due to the lack of a good graphics card wouldn't be convenient. I've seen several people on Discord who installed the code locally and either their graphics card wasn't recognized or they encountered many errors. That's why Colab is currently being used, as it offers a good GPU (15GB). Just imagine, to clone a voice or instrument in 50 epochs (which is a small number, more epochs are needed for proper training) if you have 20 minutes of audio to train it, with a 15GB GPU, it takes about 40 to 90 minutes (quite a bit of time). So, if you wanted something more "precise" like the Paul's voice I uploaded, which was trained for 500 epochs, it would require many hours of training with a 15GB GPU.

What I also wanted to mention with this is that the audios I'm showing you only have 50 epochs, so they still need refinement. The thing is, I'm really curious to see how it can clone each instrument. Once I have the settings defined, I'll train them for 500 epochs

 

@No Avenger @Xupito
Hello guys, how are you? I promised to bring you new results, but from various AI training sessions, I didn't obtain anything really significant in terms of instrument cloning. It's unpredictable because the AI I use to train the samples is designed for vocals. I have tried things like saxophone, violins, and the cello that Xupito recommended, but there are some flaws in the results. I still haven't been able to find good samples with a wide variation of tone and articulation. However, I have something here that personally amazed me, and I don't think any plugin is capable of achieving this level of realism. It's the cloning of a Resonator guitar. There are some details that may sound a bit robotic, which is due to the presence of polyphonic sounds where I played. This AI tends to make polyphonic sounds sound "robotic." Nevertheless, that's a minor detail because if you want to use it for a production with an instrument where you'll play polyphonic notes, you can use Melodyne to separate the notes and transform them with AI. First, you process some notes, then the others, and finally you put them back together. That's what I did with a voice that had harmonies. I separated the harmonies with Melodyne, cloned them separately, and then merged them again. Well, here's the result: my electric guitar playing with a slide and then cloned with a Resonator guitar. Let's see what you think. I don't believe Jam Origin is capable of doing this. At least, I have never achieved something like this

 

Sounds pretty realistic.