Conceptual work on MIDI editing

32 replies [Last post]
Anonymous

None of the pretty pictures everyone prefers yet ;)
http://thorwil.wordpress.com/2007/05/27/ardour-midi-editing/

Feedback welcome.

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

i have been reading through your document with great interest. whilst i honestly believe audio-wise ardour is far beyond *anything* else, proprietary or opensource, midi editing is something that i still haven't found an acceptable replacement for my old logic (on windows, before apple bought it and decided that 48% of their market weren't worth supporting). gripes aside, i think this new midi functionality is a brilliant opportunity to claim the new crown. i have a couple of suggestions for midi implementation, and although they are going to be obviously biased towards my experiences in logic, a straight clone of *any* program would of course be bad.

note representation:
velocity represented as small bars inside the notes, left to right. 100% of the width means 100% (255) velocity etc.
in logic this is used as well as note colouring to see at a glimpse an accurate representation.

note input:
whilst i found it difficult to get the hang of at first, the way logic handled mouse inputs i find incredibly efficient. i *never once* needed to choose a tool-mode (pen, eraser, select etc), as the interface used the two (we could use three) mouse buttons effectively.
here is an idea:

left mouse button: used primarily for selecting and modifying existing events.
dragging the end of a note in a selection left and right changes the duration of all selected notes.
dragging around notes selects them.
dragging a note from the middle moves it left/right or up/down.
double clicking a note should bring up a mini editor window (edit start/end, duration, velocity properties directly)

middle mouse button: creates or deletes notes.
when clicking a blank spot in the matrix, a note of the last used duration/velocity is created. this was the single most useful feature for fast editing i found in logic.
clicking the middle button on a current note deletes it. maybe some people wouldn't like this so make it optional, but again, i'm after speed here.

right mouse button: brings up context menu as always.
in logic this was the create events button, but i think since we can generally assume there is a middle button present for most people that it should remain a context-menu button. also useful to list some less common functions such as split note, join notes etc.

mouse wheel: used with modifiers.
some examples are: adjust duration or velocity, horizontal/vertical zoom.

i believe this sort of functionality as opposed to the 'choose a tool, choose where you want to use it' interface is far better for repetitive, serious work. maybe if people don't like the learning curve, you can keep the tool-pallet way and make one 'multi-tool' that functions like this.

i just read the end of your document, and this seems similar to the 'quasimodes' idea. i agree that it is a steeper learning curve, but with the 'multi-tool' feature i think this is overcome.

sorry if this is a bit rambling, i have a headache :)

thorwil (not verified)

Thank you porl, that's some great input.

For whatever reason I didn't think of representing velocity with horizontal bars inside notes, I only thought about doing it vertically, requiring rather high rows to be readable.

The ability to change the duration of selected notes is something I overlooked. Doing it by dragging the end of one of the selected notes should be the most straightforward way.

Double clicking to bring up note (or selection) properties is a good idea.

Regarding the middle mouse button, it will likely be used for dragging the view like in GIMP or Inkscape.

thorwil (not verified)

Lots of pictures this time ;)
http://thorwil.wordpress.com/2007/05/28/ardour-midi-editing-2/

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

very nice. my only problem with the split on/off velocity display is that it could be confused with two note events when you have a very busy midi passage. is it really used all that much? maybe just have the option to see the off velocity when you are displaying the actual values (top row). i'm not that much of a maestro with midi composition to use off velocities all that often though, maybe someone who does will kill me for suggesting that :)

i definately like the idea of the numbers in the background of the first image. if they can be toggled on and off then i think everyone would be happy.

for the appearence of the velocity 'bars', i think what i would like to see would be a combination of the 2nd last and 3rd last rows. the bar from left to right helps to get a more precise setting of the velocity, but the colouring of the velocity helps see the overall change in a busy passage (this part here looks louder (more red) than this one). in logic, the whole note was coloured the same, and the 'bar' (just a line in the middle of the note, but yours looks much nicer) would move inside it. i think your idea would look really nice if the bar was the only part that changed colour, with the underlying colour staying the same.
if you take the bottom left note image in your last picture, and go diagonally up and to the right, the next two notes in that line show what i mean. the note starts off blank and pale (low velocity), a bar comes out getting more and more red until it reaches the end of the note (100%) and is fully red (or whatever colour it goes to).

the last idea i had is more a 'nicety' type idea. if the canvas moves to cairo in time, maybe make the notes that are highlighted not only show their handles (great idea by the way), but the middle go a *tiny* bit transparent to see the barlines through them. obviously this could be overdone, but i think a subtle indication of the underlying bar lines will help in editing durations, especially, once again, in crowded passages.

edit: i just realised a good example of this effect is the progress bar in ubuntu. in fact i think this may look better than using transparency, as it won't lose any of it's colour and 'fade out', and won't need any fancy effects of the canvas.

keep it up!

i haven't been this excited about a project's development in a *long* time :)

mattb
mattb's picture
User offline. Last seen 46 weeks 1 day ago. Offline
Joined: 2007-02-22
Posts:

Hi

First off, Thorwil, your work is great. Your site is in my bookmarks so I can see what you're up to!

The mock-ups look really nice. The numbers and colouring of beats seems really clear.

I like the look of the target areas at the starts and ends of notes, but I wonder what they would look like with lots of notes in a small space. Would there be any use in the handles being hidden when zoomed out? Some software uses a change in mouse icon when hovering over the start and end to show that the length can change; would the handles be as well as, or instead of? I'm not sure we'd need both, and I prefer the handles.

As far as velocity goes, I would really like both coloured notes and a velocity bar. It would be great if you could use a modifier key and click and drag up or down on the centre of a note; when doing so, the colour would change and the relevant bar would move. Stacked notes would be fairly easy to deal with in this way.

I'm with Porl; very exciting stuff!

Matt

thorwil (not verified)

porl: Showing off velocity should be an option (I think I mentioned that ;)
Regarding bar-lines, good suggestion. I just created lines on the note that fade out to the middle to avoid a look too similar to several notes in a row. To be shown with the next set.

harold
User offline. Last seen 4 weeks 2 days ago. Offline
Joined: 2006-03-28
Posts:

Looking good already!

I've got a few questions, though.

* If you embed the velocity in the note view, what is 100% velocity?
* Will Ardour be able to record/display/edit sysex commands? (example: hihat foot controller, pitch bend, etc)

Considering the above 2 questions, I'd reckon that a separate row with a drop-down-selection is the easiest way to be able to all sorts of information. (velocity, pitch bend, sysex, etc)

-H-

thorwil (not verified)

mattb: Why, thank you!
Like I wrote, it's likely better to show the handles on mouse-over only, as it could look confusing with many small notes otherwise.
The job of the handles is to show the target area (hover over note, see where you need to pull to move start/note/end), something a changing mouse cursor can't do. The cursor is good for indicating the possible action. One could use a 4 way arrow here, but that isn't terrible informative :)

Not allowing moving the note vertically via dragging on the start or end handle to instead alter velocity this way might be worth considering.

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

Not allowing moving the note vertically via dragging on the start or end handle to instead alter velocity this way might be worth considering.

not sure about this one. it isn't an intuitive thought firstly, and more importantly it would actually slow the velocity and movement functions down, since if you are trying to change just one you have to be *extremely* accurate in going horizontal or vertical, not diagonal at all.

look forward to your next mockup though! :)

porl

thorwil (not verified)

porl: good points.

Underlays and smart dragging:
http://thorwil.wordpress.com/2007/05/29/ardour-midi-editing-3/

thorwil (not verified)

harold: MIDI velocity goes from 0 to 127. As old hardware might interpret a velocity 0 note on as an off, we should perhaps limit to 1 to 127. 100% = 127, but I wouldn't advocate using percents at all. It just doesn't match well.
Regarding sysex ... I don't know how well Jack-MIDI does or will support it. I wouldn't expect any support in Ardour above record/playback and a very simple visualization that just indicates there's something (and even that would need to be written by someone)

harold
User offline. Last seen 4 weeks 2 days ago. Offline
Joined: 2006-03-28
Posts:

@thorwil: I wasn't referring 100% as an absolute value, more as the percentage of the editor window which will resemble the velocity.

I'd reckon that the piano roll will show more than 1 octave and will be variable in height.

midi_editor_02_d.png shows the start of the velocity bars in the bottom of the window, which by coincidence is also C4. What if I scroll down? Or resize the window? Or what if there are some notes in the C3-B4 range?

-H-

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

regarding sysex, once again i can only say what logic did. all it provided was an 'event list' view of midi data which was literally just a list of events eg

Event time value data
Noteon 0:1:2.345 c4 127
noteoff 1:2:3.45 c4 127

and so on (in this case data is showing velocity). sysex and controller parameters were also listed, and the list was filterable (show only note events, sysex etc).

it was also useful for finetuning things like tempo changes (which i'm not sure on the status of jack for this yet, i think there were problems).

just looked at your new mockups too.
nice idea with the gridlines showing through :P i really like the way you implement the underlay, this was something that used to piss me off in logic. i would prefer the selection logic to be reversed (select all touching; modifier to select only fully enclosed) but thats just me i suppose. i think it would make things easier for people using long notes (for pad sounds or other weird samples) that may be 8 bars long for instance.

your last idea is great. i think i prefer the 2nd option (the slight red tint) as a screen full of the ones with the black lines might look a bit messy. there isn't really a worry about whether or not the red is obvious enough, as it is very rare that you need to check that sort of thing - it is only if you want to use quantised notes usually, and if you are changing a note you are looking directly at it, so there is less need to see at a glance if 50 notes on screen all are exact. this is the job of quantising.

hope this helps
porl

thorwil (not verified)

harold: Ah! Velocity bars in the background wouldn't be tied to the note grid. The range would be mapped to the full height.

thorwil (not verified)

porl: Showing sysex in a list view/editor might be straightforward, but I don't think such a thing is high on Dave's list.

If rubber-banding would select on touch, it would be harder to draw notes between other notes (and impossible to directly draw notes with partial overlap on the same row). If smart dragging is implemented, that is. That might be acceptable, though.

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

true... didn't think of that :) what about using the middle mouse to place notes though? i still like the idea of a single middle click making a note with the same length/velocity as the previous inputted note, while a middle click and drag would make a note of a different length (changing all future notes). i think this is faster than having to constantly drag out the same length over and over again.

Bussman
User offline. Last seen 4 years 27 weeks ago. Offline
Joined: 2006-03-31
Posts:

Thorwill, from one graphic artist to another: great work! Keep it up, Ardour is looking better and better.

Showing sysex in a list view/editor might be straightforward, but I don’t think such a thing is high on Dave’s list.

I've been using sequencers since Notator on the Atari 1040ST (ancestor to Logic) and I must say that a list event editor is crucial to many MIDI editing tasks and not including one in ardour would be a big mistake, I really hope that's not what you meant.

bennyp
bennyp's picture
User offline. Last seen 12 weeks 2 days ago. Offline
Joined: 2006-03-27
Posts:

It's going to be niiiiice :)
I'm looking forward to slaving ZynAddSubFX into Ardour....

Whoa this is going to seriously rule, Ardour is amazing!
~bennyp

thorwil (not verified)

porl: left click on empty space would add notes (last used length or snap value). With middle-drag for scrolling, middle click could work as "page" forward or backward depending on which half it is in. Have to think about making that discoverable. Otherwise middle click could be used for zoom in (zoom out with modifier) ... or unix style pasting :)

thorwil (not verified)

Bussman: thank you!
Sure, a list editor is good to have. But graphical editing takes priority and GSoC time is limited.

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

that would work pretty well actually. i think it will take a bit of getting used to, but should be very efficient. left click to create and mmb+drag to move around would be very fast indeed.

looking forward to these developments even more now... i work in a school (do the network maintenance etc) and have just had a discussion with a music industry 'guru' teacher who is wanting us to order protools gear, since the kids need to learn protools or they will be DOOMED in the industry with their completely useless knowledge on anything but protools protools protools gear.... grrrr i am sick of that belief still hanging on these days... hopefully one day people will wake up :)

sorry, end rant.

porl

robz
User offline. Last seen 6 years 46 weeks ago. Offline
Joined: 2007-06-02
Posts:

controller data

Porl, sysex is useful, but could it be that you were thinking of continuous controller data, which is more commonly used in sequencing?

Bezier-like curves with handles are a nice way to go for creating such data, though they're not much use for editing pre-existing/imported MIDI CC data.

Thorwil,
Your mockups look fantasic! I am donating to ardour asap!

R

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

oops, you are right, sorry :)

bennyp
bennyp's picture
User offline. Last seen 12 weeks 2 days ago. Offline
Joined: 2006-03-27
Posts:

Heya, So I finally got around to reading the first article, and I have two comments.

4.2 Drums vs Piano

Many sequencers have special drum tracks / views.

A drum editor can do away with having rows represent notes in continuation and instead allow reordering and assigning ports and channels per row.

As note length tends to be of no concern for percussive instruments, symbols that denote a location but no duration are used, often a rhomb.

Drum and piano style editing could be combined by always allowing rows to be reordered, having optional per row port and channel settings and using both note bars and rhombs (or similar symbol,
understood as alias for notes of an adjustable equal length).

In most cases, this is true. however, there are times when a note needs to be choked for example when a hi-hat is closed. The Hydrogen team are addressing this with "Mute Groups" where by certain instruments are muted when a paired instrument or instrument type is played.

5.5.2 Quasimodes

Use modifiers and mouse buttons to have one main mode and everything else as quasimodes. One could even differentiate between modifier keys being pressed down before or after a drag is initiated,
as done in GIMP.

Could be very efficient after training, but hard to learn / memorize. Depends on documentation and users reading it . A way out could be status bar messages, but there’s not even a status bar and
all space is precious.

Perhaps use a translucent message next to the cursor that appears when a quasimode is enabled and disappears in normal mode? There's precedent for this, when moving punch loop ranges this very thing happens and is quite useful.

5.5.4 Keys instead of clicks

Use mouse for position and trigger actions with keys. Example: A to add notes.

Hard to learn. Difficult to find enough free and matching keys. Requires a one hand on mouse, one on keyboard approach for everything.

The pro here is that it's VERY fast. I think this option should be considered as well, but not as the only option. So far with audio, Ardour has been very good about giving the user many ways of doing a task, from more basic and guided to more advanced and faster.

bennyp
bennyp's picture
User offline. Last seen 12 weeks 2 days ago. Offline
Joined: 2006-03-27
Posts:

Referencing the second article, Pictures 2, 3 and 4 (stacked note velocities). Is colour reserved for something else (like pitch?) If not, colour could be use for velocity.

Even if colour is reserved for pitch, maybe On/Off Handle colour could be sued for velocity. This has the bonus of letting you see quickly which velocity handle is for which note. This would really only come into play for stacked notes, and ardour would be smart enough to know when that is.

This is really only necessary for big chords, but would be very helpful there (especially with 7 or more notes). In other cases, there would be no need to colour code.

Photo Sharing and Video Hosting at Photobucket

On IRC, thorwil suggests that the notes would lose visual coherence in such a scheme, I agree, but wonder if the benefits of easy visual access to these data outweigh such a loss, especially if it's only used in specialized scenarios

bennyp
bennyp's picture
User offline. Last seen 12 weeks 2 days ago. Offline
Joined: 2006-03-27
Posts:

As for the third article, underlay is just plain helpful. for visual learners like me, with maybe less music theory under our belts, how the track looks means a lot for deciding where to place the next note.

for the snap to note thing, I like the top right example, but maybe with a bit less subtle colour change.

thorwil (not verified)

bennyp: I'm aware of mute or exclusion groups. They whole point is they work with trigger notes of whatever length, where the sound is played in its full length if no other trigger interrupts it. This is a matter for samplers / synthesizers, though.
My proposal to allow both notes and rhombs in one space would makes it a free choice in every single case.

porl
User offline. Last seen 30 weeks 4 days ago. Offline
Joined: 2006-07-20
Posts:

is there any updates on this? i'd be very interested to know how things are progressing :)

porl

thorwil (not verified)

porl: I've been working on other things while waiting for Dave to get to the point where we can talk about what can be implemented. That will likely happen on IRC.

zoops
zoops's picture
User offline. Last seen 5 years 8 weeks ago. Offline
Joined: 2007-05-12
Posts:

Well, it looks great! I really very like the idea of having only one space for all editing tasks with minimal tool selection. Just an idea for editing CC's:

*A drop down above/to the side of the note grid area allows a user to select an active controller number to edit.
*When holding down spacebar, the background of the note grid changes to show/reflect the CC values/panning etc... instead of note velocities (a bit like the small view underneath the note view in cubase but in the note view background).
*Panning and other stereo CC's could have a line drawn in the center of the view so you know where the center is - better still, a grid could be drawn so that some idea of scale could be determined vertically for all CC's (maybe a horizontal line drawn every 10 units up/down with a dark line for the center if necessary for the CC chosen).
*Also, when spacebar is held, the notes themselves are drawn as very transparent so they don't get in the way (but you can still see them) and note editing is disabled.
*Alternatively - the user could press ctrl-spacebar to toggle the mode instead of having to hold it down continually for more involved editing.
*When the mode is toggled on with ctrl-spacebar or even if spacebar is held down, the other keys (such as letters) could be used to change between bezier/line/direct editing of CC values - unless a way of doing it using the mouse could be found that doesn't require any tool selection (this would be ideal).

This way, all editing could be done within one view with no screen splits which I personally find very annoying. Also, there is no need of a list editor if the view takes up a large enough proportion of the screen as CC values in MIDI only range from 0~127 - most people have enough control to select these values precisely, directly using the mouse if the view is taking up a significant proportion of the screen (such as all of it). Additionally - the feature in cubase that shows what value the mouse is currently over in a small box beside the editing window could be stolen as it would be mandoratory if a user wants to be able to select the values exactly.
-zoops. :)