automated testing

6 replies [Last post]
jfkelliher
User offline. Last seen 40 weeks 3 days ago. Offline
Joined: 2014-05-11
Posts:

I am interested in hearing from the development team exactly what the philosophy of the development group as a whole is with respect to automated unit testing - going forward at least, but I'd be interested in hearing about any history behind the current testing strategy. I am very curious about this, as a musician, recent subscriber, and as a software developer with years of experience who is interested in possibly contributing some of my time

veda_sticks
User offline. Last seen 1 day 6 hours ago. Offline
Joined: 2011-03-11
Posts:

http://webchat.freenode.net/?channels=ardour
http://ardour.org/development.html

alot of your questions could probably be answered in the ardour irc if you can get in at the right time when the dev's are about.

Ive popped in to ardour irc and had a few questions answered when was having problems and ive heard the dev's discuss bugs and issues on there while working on the code

paul
paul's picture
User offline. Last seen 11 hours 37 min ago. Offline
Joined: 2006-03-16
Posts:

We have no automated testing strategy to speak of. The current "tests" in the source tree are very limited in scope.

For reference, a full automated test of ProTool apparently takes nearly 1 week to execute.

There are no plans to add more testing. I am a skeptic of testing strategies. When done well, they are invaluable and fantastic. When done poorly (which happens often), they are simply a source of more bugs to fix and more time wasted. Testing an application like Ardour is a very challenging task. Testing it in a cross-platform fashion simply adds to the fun.

jfkelliher
User offline. Last seen 40 weeks 3 days ago. Offline
Joined: 2014-05-11
Posts:

oh well, at least you're being honest about it. (i'm hoping to find a less public way to let you know what else I think...not that it's all bad, but I'm afraid I can be pretty easily misunderstood, and my own experiences have not been in a development environment such as yours)...anyway, I haven't really used the stuff enough yet to make any final judgement as to whether I can keep trying to use it)

seablade
User offline. Last seen 4 hours 55 min ago. Offline
Joined: 2007-01-22
Posts:

@jfkelliher

See the links veda_sticks posted, IRC is generally the best place for development discussions, though I will warn you this particular one has happened at least twice that I know of, and probably far more that paul knows of:)

Seablade

linuxdsp
linuxdsp's picture
User offline. Last seen 5 hours 16 min ago. Offline
Joined: 2009-02-04
Posts:

I am a skeptic of testing strategies...

@Paul: It might be interesting to note - for amusement if nothing else - that (during some recent work on Mac / Windows) my first VST3 plugin passed all of the 19000 or so automated unit tests in the SDK's validation tool...

and then failed to load in the first (non Steinberg) application I tried it with.

That particular issue wasn't too complex as it turned out, and in no way reflects on the quality of the tests as such, but, I think it illustrates something about automated test generally even in a relatively simple context. It certainly wouldn't have been practical to rely on that as the only test...

jfkelliher
User offline. Last seen 40 weeks 3 days ago. Offline
Joined: 2014-05-11
Posts:

first off before I go away forever, I have to say "thanks" for the effort; I am grateful that what's here is here...that much said:
a. who was it who said anything about automatically testing down to every last feature? it shouldn't be an all or nothing thing, should it?
b. it's obvious that you guys have already made up your minds; so no point in me wasting any more of my time.
c. good luck(you're going to need it)