Try to compare with a simple IR loader like NadIR.
You will probably feel no difference at all.

 

I am used to playing with 32 samples buffer-size and with very low latency (and not 100% happy, really struggling to find something "immediate"-sounding like an actual amp), so N4's latency really makes a big difference for my hands compared to simple IRs.

As you can see in the posted screenshot, just loading N4 (with no preset loaded) introduces 66 samples of latency. Also changing the AHEADLENGTH value doesn't seem to affect this specific latency.

 

Load NadIR and tell us latency.
You will probably be surprised ...

About latency and "real amp feeling", it is a TOTALLY different thing.
Especially if you play your amp "in the room" and amp sim on headphones

Try something fun :
Mic your real amp in ANOTHER room, and listen to it with headphones in "direct monitoring" mode.
No latency at all ... but it will feel a lot different trust me

Comparing real amp in a room AND sims is like comparing apples and oranges.
And latency have NOTHING to do with it.
This "debate" runs for 20 years at least.

Do you know guys like Dragonforce use digital modeling ? and digital wireless guitar to digital modeler ? and digital console ? and digital in hear ?
Do you really want to know how much latency all this digital path create ?
A lot MORE than your 32 samples.

It is almost the same path used all around the world by ULTRA speedy guitarists.

Now it is them playing LIVE with "all this latency" :

Here is another guitarist will a little less latency :

Latency is mostly a non argument. It is FEEL.
Problem is ... a real amp in a room IS a real amp in a room.

Even FRFR will not solve this.