Ardour 2.8 & JAMin Export

32 replies [Last post]
Jamie
User offline. Last seen 4 years 25 weeks ago. Offline
Joined: 2009-10-02
Posts:

Sorry if this has been answered or brought up before (I searched and didn't see exactly what I'm describing here).
I'm using Ardour 2.8 & JAMin on a "current" Arch Linux box, x86_64. I've noticed when I try to export a session with a JAMin insert with the default kernel (non-RT) it exports, but, it hangs up. It doesn't "hang" like this if I use my RT kernel. If I do the same thing with the RT kernel I have, it finishes perfectly. Not really a big problem or anything, just wanted to make sure if no one knew about this, they do now :o). I know a RT kernel is probably prefered and all but recording, mixing, everything works just fine with the normal 2.6.30 kernel I have. I would think exporting would also?
Anyway, if anyone know of something I could be doing wrong let me know. If this could be some kind of bug in the export or something, just wanted to let someone know.

Thanks,
Jamie

seablade
User offline. Last seen 6 hours 31 min ago. Offline
Joined: 2007-01-22
Posts:

Hmm this is the first I have heard of it, or at least you may be the first that has tied it with Jamin inserted, does it work fine without Jamin in the mix?

Seablade

tbonedude
User offline. Last seen 2 years 8 hours ago. Offline
Joined: 2008-11-08
Posts:

It happens to me as well. It happens in all generic and RT Ubuntu kernels I have tried (as well as in several different versions, hardy and jaunty to be specific), and even custom kernel compiles. Not sure why, but its annoying enough to look into...

thorgal
User offline. Last seen 1 year 21 weeks ago. Offline
Joined: 2007-08-03
Posts:

works fine here with vanilla 2.6.31.

Benjamin Scherrer
User offline. Last seen 2 days 19 hours ago. Offline
Joined: 2006-07-27
Posts:

Hi,

my ardour hangs itself up and then crashes including jackd everytime I do an export with or without jamin. That's jackd 0.116.2 and Ardour 2.8.2 (but happens since 2.7 or something). Running custom 2.6.31.1 kernel and debian amd64 testing.

If you find anything out, it would be great if you could post it here.
Benjamin

tbonedude
User offline. Last seen 2 years 8 hours ago. Offline
Joined: 2008-11-08
Posts:

Also, what info do you guys need to see to debug this issue? I also guess it would help if we file a bug report, at least if it hasn't been filed already...

Jamie
User offline. Last seen 4 years 25 weeks ago. Offline
Joined: 2009-10-02
Posts:

Err, yes. Sorry, I thought I put that in my first post. Yes, it will finish just fine without the JAMin insert with either kernel, non-RT or RT. But with the non-RT kernel and the JAMin insert it hangs everytime at the end of the export. I can kill Ardour and JAMin, go to the export folder and listen to the .wav and it exported perfectly it just hangs Ardour. As I said before, I'm just using the RT kernel and it's no problem. I just think there is a little bug in there somewhere and I wanted to bring it to someone's attention in hopes that it could be fixed ;o).

Thanks,
Jamie

lowen
User offline. Last seen 42 weeks 6 days ago. Offline
Joined: 2009-07-30
Posts:

I've had issues with JAMin and Ardour exports for a while, but not with JAMin as an insert (simply because I've not tried JAMin as an insert). I typically route the desired track outputs to JAMin's input, then JAMin's output to Ardour's main for the work I've done (as I have a preprocessed intro and matching lockout, I just process the main portion of the program through JAMin and bypass JAMin for the intro and lockout).

I've had intermittent hangs and 'long tails' (where it 'hung' for five or ten minutes after the end of export) and I've had it export with no hang, all on the same versions of Ardour, JAMin, and JACK (and on several different versions; what I'm getting at is that on any given version combination I've experienced complete and partial hangs and 'things working like they should' at random). Up until a few days ago, the AVLinux machine hadn't experienced a JAMin export hang; and then it 'hung' for ten minutes post-export. My Fedora 8 box, with an older Ardour and JAMin (and JACK) does the full hang consistently. To recover, I kill Ardour (from the command line), then JAMin wakes up and can be terminated normally. JACK has to be restarted in order to be usable again. But even the F8 box 'used' to export without a hang, but then it developed a 'long tail' occasionally, and now it consistently hangs (no updates/upgrades of any packages have been done on that box since initial setup; it's not got an Internet connection (it's in a church)).

[EDIT]
Note that the exported audio is complete and plays perfectly; it's at the very last that the hang occurs.
[EDIT]

I've not had any correlation of RT versus non-RT; all the kernels have done it at some point in time or another. I need to more accurately assess what versions I'm having issues with, including the ancient ones I'm running on F8/PlanetCCRMA.

I asked in a previous thread how one could go about triaging these sorts of issues to determine where the fault lies, but didn't get anything that would help me nail down which of Ardour, JACK, or JAMin is where the issue lies (it was more about a JACK error being thrown after Ardour stops freewheeling and tries to go back to a realtime thread, but I mentioned this issue (I think) as well.

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

Hi,

I have this problem from Ubuntu hardy, Jaunty to Karmic. I installed/compiled debug version of jamin and libjack0, and the following stack trace each time I try to export my work while jamin is an insert on the master bus:

(gdb) i s
#0  0x00cdd422 in __kernel_vsyscall ()
#1  0x00fc74d1 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0x00fca932 in *__GI_abort () at abort.c:92
#3  0x0805b78f in io_queue (nframes=64, nchannels=2, in=0xb62a3298, out=0xb62a3290) at io.c:477
#4  0x0805b82b in io_process (nframes=64, arg=0x0) at io.c:545
#5  0x00501b8e in jack_client_thread_aux (arg=0x8139ca8) at client.c:1726
#6  jack_client_thread (arg=0x8139ca8) at client.c:1767
#7  0x005068a1 in jack_thread_proxy (varg=0x81d6b88) at thread.c:110
#8  0x007e080e in start_thread (arg=0xb62a3b70) at pthread_create.c:300
#9  0x010697ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Looking in io.c of jamin ar line 477, I see abort() at the line:

    /* queue JACK input buffers for DSP thread */
    for (chan = 0; chan < nchannels; chan++) {
        count = jack_ringbuffer_write(in_rb[chan], (void *) in[chan], nbytes);
        if (count != nbytes) {          /* buffer overflow? */

            /* This is a realtime bug.  We have input audio with no
             * place to go.  The DSP thread is not keeping up, and
             * there's nothing we can do about it here. */
            IF_DEBUG(DBG_TERSE,
                     ((chan == 0)?
                      io_trace("input overflow, %ld bytes written.", count):
                      NULL));
            abort();                    /* take a dump */
            rc = ENOSPC;                /* out of space */
        }
    }

It seems to be on purpose then.

This does not happen when not exporting, and seems to relate to the freewheeling ???

Qjackctl reports:

subgraph starting at qjackctl timed out (subgraph_wait_fd=24, status = 0, state = Running, pollret = 0 revents = 0x0)
process cycle within freewheel failed

I'm running the following kernel:
Linux 2.6.31-9-rt #152-Ubuntu SMP PREEMPT RT Thu Oct 15 05:01:14 UTC 2009 i686 GNU/Linux

So, concluding: no problems with jamin as insert normally, only when exporting. It does not seem to matter whether I deactivate jamin before the export (I tried recording the jamin 'session' in a separate track, deactivate jamin and then exporting that specific track, but same thing happens; only when completely removing the insert before the export I can save the track). It's really annoying.

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

I've noticed a few strange problems with ardour when exporting audio - I tried it a while back and it seemed to work fine, but on closer inspection, I find I get xruns or a complete hang sometimes if I use a plugin that takes a lot of DSP (not necessarily one of mine!)
This happens with Ardour 2.8.3 - 2.8.4 and (probably more importantly) JACK 116 -> 118

I'm puzzled as to why there should be x-runs when using jack in freewheel mode (as I believe ardour does to export) since I thought in this mode jack will wait for the processing graph to complete before starting another run of the graph - so I would expect the export process to just slow down to accommodate the slowest audio process (Its been a while since I looked at the JACK code so please correct me if I've not understood it correctly - I know I'm probably over-simplifying it...)

I'm happy to help debug this issue since I'm normally testing Ardour / JACK and various plugins etc most of the time...

paul
paul's picture
User offline. Last seen 3 hours 39 min ago. Offline
Joined: 2006-03-16
Posts:

@linuxdsp: what type of audio interface?

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

I'm using an EMU 0404 PCI card at the moment - latest ALSA drivers - seems to be rock solid in every other respect, I can run very low latency, full duplex - stock kernel (not realtime, but with the audio group and /etc/security/limts.conf mods) and I never get any xruns when playing or recording audio - even with a lot of processing.

joegiampaoli
joegiampaoli's picture
User offline. Last seen 23 weeks 3 days ago. Offline
Joined: 2008-02-27
Posts:

Just to let you all know this used to happen to me also with both generic and RT kernel from ubutu repos when exporting, and even on a few self compiled RT kernels, so Jamie, yes you are not the only one who has experienced this issue.

Here's an old thread I started a little way back about this:

http://ardour.org/node/2680

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

To complete my previous post (and correct me if this is not the right place to post this): the moment Jamin decides to abort when I just ran it was at the end of the export (previously I had the problem at the very start). I had a breakpoint set at 477 abort(). Then 'count' was 255 and 'nbytes' was 256 when I hit the breakpoint. So still one byte to write, it seems.

The documentation of jack_ringbuffer_write() states that the return value is between 0 to cnt, so why does jamin not try to write the remaining bytes, like you would on Unix file system writes. Maybe a dumb question, I'm not familiar with jack or audio programming at all ?

Next, I jumped over the abort to 483: if (jack_ringbuffer_read_space(in_rb[0]) ... which completed the export just fine, Jamin and Ardour continued normally. Not sure if my export is completely ok.

What would be a wise next step ?

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

I was wrong about jamin continuing after the export (and jumping in the debugger to 483). Ardour continued just fine, but Jamin lost all connections.

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

Me again, I've changed the Jamin code around the abort() and have successfully exported my session, numerous times in a row, without Jamin, Ardour or Qjackctl crashing or anything. I'm really puzzled as to why it works now.

I've changed the wretched Jamin abort() code to retry short writes: one thing that puzzles me, is that the number of frames short is always 1. So in all logs I always get 1 write short, so I retry that, but that not always succeeds and the count reported back is zero. If I then do a usleep(100) or something, and retry the write then it seems to succeed.

So I've changed the above code to:

    /* queue JACK input buffers for DSP thread */
    for (chan = 0; chan < nchannels; chan++) {
        count = jack_ringbuffer_write(in_rb[chan], (void *) in[chan], nbytes);
        if (count != nbytes) {          /* buffer overflow? */
                int try;
                for (try=0; try<5 && count < nbytes; try++) {
                        int cnt, nbt = nbytes - count;
                        fprintf(stderr, "count %d nbytes %d\n", count, nbt);
                        fprintf(stderr, "Trying %d from position %d\n", nbt, count);
                        cnt = jack_ringbuffer_write(in_rb[chan], (void *) in[chan]+count, nbt);
                        fprintf(stderr, "Got %d\n", cnt);
                        if (cnt==0) usleep(100);
                        count += cnt;
                }

                if (count != nbytes) {
                        /* This is a realtime bug.  We have input audio with no
                        * place to go.  The DSP thread is not keeping up, and
                        * there's nothing we can do about it here. */
                        IF_DEBUG(DBG_TERSE,
                                     ((chan == 0)?
                                      io_trace("input overflow, %ld bytes written.", count):
                                      NULL));
                        abort();                        /* take a dump */
                        rc = ENOSPC;            /* out of space */
                } else {
                        fprintf(stderr, "ok\n");
                }
        }
    }

When exporting I get a load of stderr messages at the end of the export (sometimes also a few at the beginning, but in the end I always get everything written, and continue normally.

Do any of you have any idea why this works ? Is this the right thing to do ? I think I should do this only when in freewheeling mode, but could not find a Jack api to tell me whether we're in freewheeling mode. Is there such an api ?

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

There is very good documentation all about the JACK API at www.jackaudio.org/documentation

you register a callback with JACK to tell your app when jack has entered freewheel mode.

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

You're right, it was late yesterday evening, must've overlooked it, thanks for the tip.

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

I've patched Jamin to recognize freewheeling mode and relax the abort() condition. I'm not too comforable with the usleep(), and will ask Jamin development members for their comment. I've uploaded the io.c diff and a Ubuntu Karmic deb to http://sites.google.com/site/keesvveen/jamin/ for those interested.

What this solves for me is: Jamin crashing (aborting) when exporting from Ardour while Jamin is an insert on the master bus. Thus far it didn't happen again, while before it happened all the time, mostly at the very end of the export, but sometimes also at the beginning or during the export.

[EDIT]

Best read on to comment 31, this patch is really a hack which does not solve the problem but rather hides the symptoms.

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

Hmmm, I'm not sure this is the right way to go about fixing this..

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

I am posting this here for suggestions. As I said I'm not familiar with audio programming, but am willing to help (and learn when pointed in the right direction).

Funny thing is it only happens when exporting, I have no problems when running 'normally', not even when I record the output of the master bus, with Jamin inserted, into a new track in the same session.

With my fix, Jamin eventually succeeds in writing all of the bytes, and I cannot hear anything funny in the exported file.

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

While it's great that you have such enthusiasm for fixing this issue - my concern came mainly from the comments such as

"I'm really puzzled as to why it works now."

and

"Do any of you have any idea why this works ? Is this the right thing to do ? "

I think it may be better to study the code properly and understand the exact nature of the problem you are trying to solve before jumping straight in with the debug tools and applying patches if you are not really sure why they work.

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

Ok, in general I would agree, but I was hoping for some comments on whether a retry of jack_ringbuffer_write when the return value is less than the number passed is something that would be allowed in freewheeling mode (as the api states that the return value will be between 0 and cnt, what do we do when this is less than cnt?). I guess this is not the right forum to be asking such questions.

I have high regard for both Ardour and Jamin developers, so the original code will not have been written for nothing (although one could argue about the abort() without shutting down properly). But then again, would Jamin development have forseen it would be run in freewheeling mode ?

My enthusiasm comes mainly from various post I've seen around the issue, my annoyance with the problem, and the fact that everywhere Ardour+Jamin seems to be advised as the combination for mastering on Linux. It would be nice if that also ran ok on Ubuntu (as it seems only on Ubuntu problems with export+Jamin arise).

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

@kees: My comments weren't really intended to be too negative - as I said, its great that you have the enthsiasm for fixing the problem, I just think this stuff can be more complex than it first appears and it is really necessary to get an overview of the entire system and how the problems you are seeing fit with that before changing one specific part - so as to avoid unwanted side-effects.
I have to confess I don't really use JAMIN that much - although those that do speak highly of it (I have written my own set of JACK client audio processors which I prefer)

http://www.linuxdsp.co.uk

Last time I tried JAMIN I found it to be a little out-dated and it didn't seem to be being developed much, IIRC it uses some LADSPA plugins to do the processing and I found some of them weren't completely reliable.

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

@linuxdsp: No offense taken,
(
not anymore, I must confess: made me feel like a junior programmer who thought he had found something, but didn't follow the proper procedure, it's been many years since I felt like that, but well ... keeps me humble; makes you think twice next time before you post something, or maybe not share anything at all
)
but you could have commented on the content rather than on the way of working. The topic of this issue still is Ardour & Jamin export, whether or not you use Jamin. And the problem had been around for quite some time and keeps getting stirred up. I tried to debug the issue and find Jamin goes into panic mode by a plain abort(), what can you do ?

I'll tell you why I was puzzled: if a retry of the short writes works (or seems to), why didn't the Jamin developers do that, it can't be that easy, can it? Furthermore, it seems like a timing issue, because with a short sleep it seems to work eventually without errors reported, or maybe I was just lucky several times, and I know from experience that with timing issues you can be lucky for a long time.

Well, too much said anyway, I'll leave it here.

I'll have a look at your plugins.

linuxdsp
linuxdsp's picture
User offline. Last seen 1 day 17 hours ago. Offline
Joined: 2009-02-04
Posts:

@kees: From a brief look at the section of code you posted (I'm afraid I don't really have time to get into the inner workings of the Jamin code - and the Jamin devs are almost certainly better placed to answer this...) then it would appear that your problem is caused because (as the comment in the code says) "The DSP thread is not keeping up - we have input audio with no where to go" so the buffer is overflowing. In other words the JACK audio callback is providing chunks of audio faster than the DSP thread can process them. Your fix would appear to be related to putting a sleep (in the jack audio processing callback?) which allows the DSP thread to catch up. You may get away with this in freewheel mode - but it is really not recommended for normal use. So it would be nicer if a fix could be found that didn't require it at all.
My guess would be that it is actually the - necessary - decoupling between the two threads provided by the ring buffer that causes the problem in freewheel mode, since in this mode JACK will just free-run as fast as it can. Since the only limit for jacks processing speed is then the time taken to fill the ring buffer - not necessarily the time taken to process the audio there is a good chance it will overun. In a more simplistic architecture where the audio can be processed directly in the jack audio callback (without a ring buffer), jack would be inherently limited by the time taken to process a buffer of audio samples when running in freewheel mode. So your fix may achieve the 'right' result by slowing jack down - but by (in my opinion) the wrong means. I hope I have interpreted all this correctly - as I said - this is only based on the code snippet posted - please correct me if I'm wrong. (Although this forum is probably not the best place for a detailed discussion about how the code works)

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

@linuxdsp: now that's a helpful comment (finally ;-) ), thanks !

I know too little about Jack et al. to judge whether you are right, but it makes sense. In the diff I've uploaded I made my fix dependent on Jack running in freewheeling mode, so normal operation still would have the strict condition.

My own Jamin problems only had to do with export, never had crashes in 'normal' mode (as far as I remember), so I may indeed get away with this 'fix'. Still no real explanation why that mostly happens at the end (Ardour busy writing, Jamin keeping up with the graphics, ???).

Looking around in Jamin development, I get the feeling we're 'beating on a dead horse', correct me if I'm wrong. Not sure if it's worth the while to find the 'real' cause, also because many advise to use Ardour plugins instead of Jamin. It's just that Jamin is really handy for quick improvement, clicking on all three 'auto make' buttons.

Thanks for your comment.

seablade
User offline. Last seen 6 hours 31 min ago. Offline
Joined: 2007-01-22
Posts:

Jamin is absolutely still useful and worthwhile in my opinion.

It isn't just the fact it uses the plugins, but also provides an interface suitable for the fine grained control you will need in mastering.

Seablade

kees
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 2010-01-07
Posts:

I've started a new topic on the jamin devel mailing list on the issue.

seablade
User offline. Last seen 6 hours 31 min ago. Offline
Joined: 2007-01-22
Posts:

I saw, but am not really the best person to answer. Give them a few days though as it can take a bit for them to answer sometimes.

Seablade

mendred
mendred's picture
User offline. Last seen 3 days 3 hours ago. Offline
Joined: 2010-01-16
Posts:

Kees, if possible could you put up a 64 bit deb of your fix? Thanks!

EDIT: I also use ardour with jamin as an insert and in the past export used to freeze after creating the file successfully as described in this thread. However recently in karmic, ardour had issues where it would frequently disconnect from jack. Then i found out a repo for jack2 (ppa:frasten) and then on using that everything ran really well and was incredibly stable. Except for jamin crashing on export.

I am currently getting around the problem by connecting the master outs to a stereo track in ardour and recording it and then exporting the bounced track only.One advantage i see is that before export, i can immediately check in ardour if the track has any clipping..if so then it can be normalised directly.

However the wierd thing is irrespective of whether i deactivate the insert on the master track or even if i disable the master track itself, jamin will crash even if i export only the selective track. So i usually need to close jamin before i do this.