MBWF (RF64): File size limit of 2GiB?

Hello,

using Ardour 4.7 (Linux 32bit on openSUSE 13.2, using ext4 file system) I tried to record a 2 hour long session at 96kHz in MBWF format.

Unfortunately, the resulting / saved files (left and right channel) in the interchange folder are only approx. 2GiB in size, only the initial 93 minutes of recording are available.

sndfile-info:

Version : libsndfile-1.0.25-exp

========================================
File : Audio-3%L.rf64
Length : 2147484362
RF64
WAVE
ds64 : 28
Riff size : 2147484354
Data size : 2147483648
Frames : 536870912
Table length : 0
fmt : 40
Format : 0xFFFE => WAVE_FORMAT_EXTENSIBLE
Channels : 1
Sample Rate : 96000
Block Align : 4
Bit Width : 32
Bytes/sec : 384000
Valid Bits : 32
Channel Mask : 0x4 ©
Subformat
esf_field1 : 0x3
esf_field2 : 0x0
esf_field3 : 0x10
esf_field4 : 0x80 0x0 0x0 0xAA 0x0 0x38 0x9B 0x71
format : IEEE float
bext : 602
data : 0xFFFFFFFF
End


Sample Rate : 96000
Frames : 536870912
Channels : 1
Format : 0x00220006
Sections : 1
Seekable : TRUE
Duration : 01:33:12.405

Ardour is opening the project but the audio track does not contain any data (even though it shows the length of 01:33). Audacity can open, display and play the RF64 files contained in the interchange folder.

Questions:

  • MBWF / RF64 should allow for files larger than 2GiB in size. If this is not the case, which other format may I use?
  • Is there a possibility to automatically split files during recording?
  • Are there specific limitations for the 32bit version of Ardour? I haven't found any list of limitations on the Ardour web site or documentation.
  • How to correctly open the project in Ardour to work on the recorded audio?

Thanks in advance!
Arnd

...I tried to record a 2 hour long session at 96kHz...
Do you need 96K? If you record at 44.1kHz you would have at least twice as much recording time, and, you won't notice any audible difference.

Mike,

do I need 96kHz? Maybe not. But: I think it is unfortunate that Ardour (Linux 32bit at least) does not continue recording and just stops recording without any message.

I did some further testing:

  • I can open the saved session using Ardour 64bit on another Linux system, but navigating through the session (ie. move to end of session with , move back with "<-") causes Ardour to abort with
    (ardour-4.7.0:3843): glibmm-ERROR **: unhandled exception (type std::exception) in signal handler: what: std::bad_alloc
  • It seems that Ardour saved 2GiB of samples fine (Audio-3%L.rf64 / Audio-3%R.rf64), but that Ardour then tried to continue in another file (Audio-4%L.rf64 / Audio-4%R.rf64) but stopped writing to this file after just over 1s:
    Version : libsndfile-1.0.25-exp

    ========================================
    File : Audio-4%L.rf64
    Length : 385738
    RF64
    WAVE
    ds64 : 28
    Riff size : 385730
    Data size : 385024
    Frames : 96256
    Table length : 0
    fmt : 40
    Format : 0xFFFE => WAVE_FORMAT_EXTENSIBLE
    Channels : 1
    Sample Rate : 96000
    Block Align : 4
    Bit Width : 32
    Bytes/sec : 384000
    Valid Bits : 32
    Channel Mask : 0x4 ©
    Subformat
    esf_field1 : 0x3
    esf_field2 : 0x0
    esf_field3 : 0x10
    esf_field4 : 0x80 0x0 0x0 0xAA 0x0 0x38 0x9B 0x71
    format : IEEE float
    bext : 602
    data : 0xFFFFFFFF
    End


    Sample Rate : 96000
    Frames : 96256
    Channels : 1
    Format : 0x00220006
    Sections : 1
    Seekable : TRUE
    Duration : 00:00:01.003
    Signal Max : 0.0317844 (-29.96 dB)

  • Deleting this region allows working on the session and does not cause Ardour to abort anymore.

I will conduct some further testing, also trying other file formats.

Arnd

I would ignore the 96kHz question in this context. It is a relevant question but not for this issue, not at all or in anyway. I will not be able to run my own tests on recording until next week, but will do so as soon as I can.

Hello Paul,
thanks for your support!

I did some further testing:

  • I tested both MBWF as well as plain WAV (which should allow for 4GB)
  • Some recordings already hung very early (for example after just 37MB..), ie. the Ardour GUI allows to select menus, but no action is taken
  • The latest test (recording stereo at 96kHz into WAV) broke after exactly 2GiB:
    • After 01:33:13.25 Ardour stopped recording (without any pop up)
    • The written files (Audio-1%L.wav/Audio-2%R.wav) were 2GiB+RIFF header in size
    • After closing Ardour both files were cut to only 56 bytes, sndfile-info output:
      Version : libsndfile-1.0.25-exp

      ========================================
      File : Test WAV 96kHz 2/interchange/Test WAV 96kHz 2/audiofiles/Audio-1%L.wav
      Length : 56
      RIFF : 48
      WAVE
      fmt : 16
      Format : 0x3 => WAVE_FORMAT_IEEE_FLOAT
      Channels : 1
      Sample Rate : 96000
      Block Align : 4
      Bit Width : 32
      Bytes/sec : 384000
      fact : 4
      frames : 536870912
      data : 2147483648 (should be 0)
      End
      Error : psf_fread returned short count.


      Sample Rate : 96000
      Frames : 0
      Channels : 1
      Format : 0x00010006
      Sections : 1
      Seekable : TRUE
      Duration : 00:00:00.000
      Signal Max : 0 (-inf dB)

    • The error log contained the following information:
      [INFO]: Loaded mixer bindings from /opt/Ardour-4.7.0/etc/mixer.bindings [INFO]: Loading bindings from /opt/Ardour-4.7.0/etc/mnemonic-us.bindings Loading menus from /opt/Ardour-4.7.0/etc/ardour.menus [INFO]: Loading 88 MIDI patches from /opt/Ardour-4.7.0/share/patchfiles [INFO]: Loading history from /home/arnd/Music/Audio/Ardour/Test WAV 96kHz 2/Test WAV 96kHz 2.history [INFO]: Test WAV 96kHz 2: no history file "/home/arnd/Music/Audio/Ardour/Test WAV 96kHz 2/Test WAV 96kHz 2.history" for this session. [ERROR]: /home/arnd/Music/Audio/Ardour/Test WAV 96kHz 2/interchange/Test WAV 96kHz 2/audiofiles/Audio-1%L.wav: cannot seek to 536870912 (libsndfile error: No Error.) [ERROR]: AudioDiskstream 154: cannot write to disk [ERROR]: Butler write-behind failure on dstream Audio [ERROR]: /home/arnd/Music/Audio/Ardour/Test WAV 96kHz 2/interchange/Test WAV 96kHz 2/audiofiles/Audio-1%L.wav: cannot seek to 536870912 (libsndfile error: No Error.) [ERROR]: AudioDiskstream 154: cannot write to disk [ERROR]: AudioDiskstream "Audio": cannot flush captured data to disk!
      The interesting stuff seems to be the libsndfile error: No Error. part.

I’ll do some further tests with MBWF.

Regards, Arnd

Hello Paul,

I did some further tests:

  • The problem happens for (plain) WAV as well as for MBWF (withouth 4GiB limit)
  • The problem happens also when recording 44.1 kHz when the 2GiB file size is reached

I took some time and digged into the Ardour and libsndfile code base and build process. I focused on libs/ardour/sndfilesource.cc around line 717 which is the call to sf_seek which fails according to Ardour’s log. As frame_pos is well below the 32bit limit I were suspicious of libsndfile.so.1 included in Ardour 4.7.0I downloaed from ardour.org. I did the following test:

  • I renamed lib/libsndfile.so.1
  • I created a symbolic link from my distributions (openSUSE 13.2) libsndfile.so.1.0.25 to /opt/Ardour-4.7.0/lib/libsndfile.so.1
  • I started Ardour again and checked that my distribution's libsndfile.so.1 was used
  • The test proceeded without any problem beyond the 2GiB file size and is still running as I want to check whether recording passes 4GiB as well

Conclusion: It seems that Ardour’s libsndfile.so.1 is compiled with 32bit integers for the 32bit Linux version. Is it possible to just recompile it from within the Ardour source code?

Best regards, Arnd

Sadly, your conclusion is not relevant. It makes no different whether libsndfile is compiled “32 bit” or not. 32 bit applications can trivially deal with 64 bit integers. If that were not true, then you could never have a filesystem larger than the 32-bit limit on a 32 bit version of Linux, which is clearly not the case.

Hi Paul,

I’m well aware that 32 bit applications can deal with 64 bit integers (or whatever other size required). Prerequisite is that the correct data types are being used as there are some compiler / standard library related differences between 64 bit and 32 bit systems.

As I wrote: It seems that libsndfile.so.1 provided with Ardour 4.7.0 for 32bit Linux was compiled in a way that 32 bit integers were used in a way they shouldn’t be used as. And this has nothing to do with the 32 bit operating system used.

More detailed reasoning:

  • Using the openSUSE 13.2 (32 bit) supplied //usr/lib/libsndfile.so.1 works flawlessly, I tested up to 4.8 GiB
  • Using the Ardour 4.7.0 supplied /opt/Ardour-4.7.0/lib/libsndfile.so.1 causes writing to stop / fail after exactly 2 GiB written
  • The error message in Ardour's log (cannot seek to 536870912 (libsndfile error: No Error.)) originates in libs/ardour/sndfilesource.cc line 717: if (_sndfile == 0 || sf_seek (_sndfile, frame_pos, SEEK_SET|SFM_WRITE) < 0) { It seems that the sf_seek call to libsndfile.so.1 returns a negative return value, but that the internal libsndfile.so.1 error code is not set. As the frame pointer 536870912 passed to libsndfile corresponds to byte pointer 2147483648 (4 bytes per 32 bit float), which is just out of range for a 32 bit signed integer, I would consider one of the following possible causes:
    • The Ardour supplied libsndfile.so.1 was incorrectly compiled so that possibly the sf_count_t data type is not a 64 bit integer
    • The Ardour supplied libsndfile.so.1 lacks some older fix
    As I don't know where the Ardour supplied libsndfile.so.1 is coming from, I can't check any further myself. If you could point me to the source and makefile used for building it I would be more than happy to have a look.

Best regards, Arnd

PS: Ardour is a really great, powerful, well thought out audio workstation. Thanks!

We build all libraries packaged with Ardour ourselves, using scripts that are in a separate git repository. They are all compiled with so-called “large file” support, in which off_t and thus sf_count_t are supposed to be 64 bit units. I will check this tomorrow or the next day (just back from a 4 day trip).

Hello Paul,
sorry for bothering (again): Were you able to check / recreate this problem?
Please let me know if I may support.
Thanks, Arnd

Sorry, I’ve had my head deep in some really fundamental code redesigns. Still trying to get to it. However, just as I wrote this, I realized that the one check I did (verifying that we set the size of off_t to 64 bits) I only did for the Ardour build itself.

I went back and looked at the script we used to build the dependency stack, and I realize that the relevant macros are not in use there. This would explain why the 64 bit version works, and the 32 bit one does not(since the 64 bit version gets off_t as a 64 bit value automatically). We will fix in an upcoming build.

There’s a 32 bit Linux build on http://nightly.ardour.org/ now and the other builds will all follow tonight. The new build(s)now have the relevant 64 bit file size macros used for the entire build stack. If you have time to check the new build, I’m expecting it to be fixed.

Arnd - thanks very much for the confirmation that we fixed the problem. Ordinarily I would like to have been more responsive, but the development in the “vca2” branch has been so conceptually overwhelming that I’m grateful we got a resolution with this little involvement on my part. Thanks again!

Hi Paul,
just checked today with Ardour 4.7.971 Nightly 32 bit and the problem is fixed.
Best regards, Arnd