What period setting are you using when firing up jackd.
A lower setting will reduce this delay (latency you are experiencing) but the system will not be able to process as much (less plugins e.t.c.)
I personally use the following line
jackd -R -d alsa -d ice1712 -r 44100 -p64
This results in a latency of under just over 1.5 ms. which is almost unnoticeable (tested by doing a/b comparison from Mixer (prior to hitting sound card so 0 latency) to Tape Return (software output).