help with music application

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

help with music application

Dennis Raddle
I realize my current implementation of my music application is not pretty and I want to learn to use Haskell more effectively.

My application reads a musical score exported by the typesetter Sibelius. A score is a high-level representation of music that includes structures, like grouping notes for each instrument, and directions that affect how the musuc will be played, like tempo indications, indications to gradually speed up, gradually get louder, etc. My program translates it into low-level time stamped MIDI events, which say things like "turn on this note", "turn off this note", "change the volume setting on this instrument," etc. 

The outline of my program is this:

1. Read the output of Sibeliusl

2. Construct a Score. A Score has several Parts (individual instruments). A part consists of a list of time stamped Chords, which represent a list of Notes that are sounded at the same time (roughly). There are expression marks in the music that apply to Chords as a whole and some apply only to Notes. Notes are the most basic sound-generating unit -- they have a pitch, a duration, some indicators of musical expression like performance techniques, a time offset (a Note may start slightly ahead or behind the time stamp on the Chord), and more. 

3. Modify the Score to account for forms of musical expression and for a few strange things in the way Sibelius exports its data. Right now I process the score through a LOT of passes, maybe two dozen. Also I have given a ton of extra fields to Notes and Chords that essentially are cache or memos of the results of processing that looks for patterns. So now things are quite unwieldy.. I have two dozen extra fields between Chords and Notes, and I need to make two dozen modification passes on the Score. It's bug prone and hard to understand because the order of the passes is important and I could easily put the Score into an invalid state. 

So the data looks like this:

The basic time stamp type is Loc, indicating a measure number and a beat within that measure.

type MeasureNum = Int
type MeasureBeat = Rational
data Loc = Loc MeasureNum MeasureBeat

type PartName = String
data Score = Score ScoreRelatedData (Map PartName Part)

data ScoreRelatedData =  -- this is data that applies to the score as a whole, like a measure-by-measure list of tempos, time signatures, and a time map that allows the translation of Loc to a number of seconds. 

-- The Chords in a Part occur in time at a Loc. There can be more 
-- than one Chord at a Loc
data Part = PartRelatedData (Map Loc [Chord])
data PartRelatedData -- data about musical expression that applies to a Part as a whole

data VoiceNumber Int -- even within a single Part there can be different "voices" which means independent sequential Chords, independent in the sense they may have different volume levels, different durations, etc.
data Chord = ChordRelatedData VoiceNumber [Note]
data ChordRelatedData -- this includes data about the Chord as a whole, such as playback techniqes that applies to the whole Chord. It also contains a list of Notes.

data Note = Note NoteRelatedData Pitch
data NoteRelatedData -- this includes expression markings that apply to a Note only

So there are some patterns within the Score that need to be identified before it can be translated to MIDI events. Here are some examples:

1. Tied notes. Some Notes within a Chord X are tied to a Note in the following Chord Y. That means they represent a single sound unit that should be sustained from the beginning of Chord X to the end of Chord Y. But actually the note in Chord Y can be tied to Chord Z and so on for any number of sequential Chords. When I want to find the true ending Loc of a Note I need to follow the tie chain. Some Notes in the same Chord may be tied, and others may not. 

2. Double tremolos. Sometimes two sequential chords are actually supposed to played together--actually the player will rapidly alternate between the chords. When I first read the score there will be a marking on the first chord X of the double tremolo. I have to look for a Chord Y that immediately follows X, has the same VoiceNumber and the same duration, and I can infer that's the second chord in the double tremolo. Note that the timing and notes of a double tremolo, when translated to MIDI, look hardly anything like the original data -- the original data just has two Chords, but the playback will contains lots of sequential notes that are drawn from both chords.

3. Arpeggios. Sometime the notes in a chord are "rolled" -- played with the lowest note first, followed by a time-staggered playback of the other notes, going up in pitch. There might be an arpeggio marking that spans several Parts, meaning I have to look at all the parts to compute the time offsets for the notes in an arpeggio

4. problems in the data export. Sibelius has some bugs, so I need to find problems in its output and fix them by removing or altering certain Chords.

And that's just the beginning. There are something like two dozen patterns that need to be identified.

What I'm doing now is adding all sorts of fields to Parts, Chords, and Notes to hold memos of the results of this processing. These memo fields have to be initialized to something, like zero, or an empty Map, or whatever. Then I set them via the processing passes. 

This is bug-prone and unwieldy. 

What I am realizing is that I don't necessarily need to store every field. For isntance, consider tie chains. I don't really need to process them all ahead of time. I can follow a tie chain only in the places where I need to know the full duration of a Note. I might end up with some redundant processing, but the advantage is 

(1) I don't need an extra field
(2) I don't have to worry about that extra field becoming invalid or remaining uninitialized.

But there are some processing data that probably should be done once and memoized. For instance, I need a Map of the END LOCATION of each Chord to the Chord itself to look up certain things. That is expensive to construct. 

So what data do I pass to my MIDI-conversion algorithm? I could create something called ContextNote like this

data ContextNote = ContextNote Score Part Chord Note

Then I could write a function that converts a score to a list of context notes, and pass all of them to the conversion routine.

allContextNotes :: Score -> [ContextNote]

This list would have one entry for every note in the Score.

Also I would probably use named fields. So I would have something like

data Note = Note
  { nPitch :: Int
  , nVolume :: Int 
  , ...
  }

When I'm accessing the data on ContextNote, I don't want to have to type several accessor functions to get down to the Note fields. So I could do something like

class HasNoteData a where
  pitch :: a -> Int

Then 

instance HasNoteData ContextNote where
  pitch (ContextNote _ _ note) = nPitch note

I could create a bunch of accessor functions like that for every field in the ContextNote.

For data that should probably be computed once, I could create Memo data. Maybe I would have

data ContextNote = ContextNote Memo Score Part Chord Note

or something like that.

So if you have read this far, thank you very much. Any suggestions welcome.

D














_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: help with music application

Christopher Allen
Could you post the code to a public repository? Ideally with some example input and expected output if you're that far along.

On Tue, Dec 1, 2015 at 5:11 PM, Dennis Raddle <[hidden email]> wrote:
I realize my current implementation of my music application is not pretty and I want to learn to use Haskell more effectively.

My application reads a musical score exported by the typesetter Sibelius. A score is a high-level representation of music that includes structures, like grouping notes for each instrument, and directions that affect how the musuc will be played, like tempo indications, indications to gradually speed up, gradually get louder, etc. My program translates it into low-level time stamped MIDI events, which say things like "turn on this note", "turn off this note", "change the volume setting on this instrument," etc. 

The outline of my program is this:

1. Read the output of Sibeliusl

2. Construct a Score. A Score has several Parts (individual instruments). A part consists of a list of time stamped Chords, which represent a list of Notes that are sounded at the same time (roughly). There are expression marks in the music that apply to Chords as a whole and some apply only to Notes. Notes are the most basic sound-generating unit -- they have a pitch, a duration, some indicators of musical expression like performance techniques, a time offset (a Note may start slightly ahead or behind the time stamp on the Chord), and more. 

3. Modify the Score to account for forms of musical expression and for a few strange things in the way Sibelius exports its data. Right now I process the score through a LOT of passes, maybe two dozen. Also I have given a ton of extra fields to Notes and Chords that essentially are cache or memos of the results of processing that looks for patterns. So now things are quite unwieldy.. I have two dozen extra fields between Chords and Notes, and I need to make two dozen modification passes on the Score. It's bug prone and hard to understand because the order of the passes is important and I could easily put the Score into an invalid state. 

So the data looks like this:

The basic time stamp type is Loc, indicating a measure number and a beat within that measure.

type MeasureNum = Int
type MeasureBeat = Rational
data Loc = Loc MeasureNum MeasureBeat

type PartName = String
data Score = Score ScoreRelatedData (Map PartName Part)

data ScoreRelatedData =  -- this is data that applies to the score as a whole, like a measure-by-measure list of tempos, time signatures, and a time map that allows the translation of Loc to a number of seconds. 

-- The Chords in a Part occur in time at a Loc. There can be more 
-- than one Chord at a Loc
data Part = PartRelatedData (Map Loc [Chord])
data PartRelatedData -- data about musical expression that applies to a Part as a whole

data VoiceNumber Int -- even within a single Part there can be different "voices" which means independent sequential Chords, independent in the sense they may have different volume levels, different durations, etc.
data Chord = ChordRelatedData VoiceNumber [Note]
data ChordRelatedData -- this includes data about the Chord as a whole, such as playback techniqes that applies to the whole Chord. It also contains a list of Notes.

data Note = Note NoteRelatedData Pitch
data NoteRelatedData -- this includes expression markings that apply to a Note only

So there are some patterns within the Score that need to be identified before it can be translated to MIDI events. Here are some examples:

1. Tied notes. Some Notes within a Chord X are tied to a Note in the following Chord Y. That means they represent a single sound unit that should be sustained from the beginning of Chord X to the end of Chord Y. But actually the note in Chord Y can be tied to Chord Z and so on for any number of sequential Chords. When I want to find the true ending Loc of a Note I need to follow the tie chain. Some Notes in the same Chord may be tied, and others may not. 

2. Double tremolos. Sometimes two sequential chords are actually supposed to played together--actually the player will rapidly alternate between the chords. When I first read the score there will be a marking on the first chord X of the double tremolo. I have to look for a Chord Y that immediately follows X, has the same VoiceNumber and the same duration, and I can infer that's the second chord in the double tremolo. Note that the timing and notes of a double tremolo, when translated to MIDI, look hardly anything like the original data -- the original data just has two Chords, but the playback will contains lots of sequential notes that are drawn from both chords.

3. Arpeggios. Sometime the notes in a chord are "rolled" -- played with the lowest note first, followed by a time-staggered playback of the other notes, going up in pitch. There might be an arpeggio marking that spans several Parts, meaning I have to look at all the parts to compute the time offsets for the notes in an arpeggio

4. problems in the data export. Sibelius has some bugs, so I need to find problems in its output and fix them by removing or altering certain Chords.

And that's just the beginning. There are something like two dozen patterns that need to be identified.

What I'm doing now is adding all sorts of fields to Parts, Chords, and Notes to hold memos of the results of this processing. These memo fields have to be initialized to something, like zero, or an empty Map, or whatever. Then I set them via the processing passes. 

This is bug-prone and unwieldy. 

What I am realizing is that I don't necessarily need to store every field. For isntance, consider tie chains. I don't really need to process them all ahead of time. I can follow a tie chain only in the places where I need to know the full duration of a Note. I might end up with some redundant processing, but the advantage is 

(1) I don't need an extra field
(2) I don't have to worry about that extra field becoming invalid or remaining uninitialized.

But there are some processing data that probably should be done once and memoized. For instance, I need a Map of the END LOCATION of each Chord to the Chord itself to look up certain things. That is expensive to construct. 

So what data do I pass to my MIDI-conversion algorithm? I could create something called ContextNote like this

data ContextNote = ContextNote Score Part Chord Note

Then I could write a function that converts a score to a list of context notes, and pass all of them to the conversion routine.

allContextNotes :: Score -> [ContextNote]

This list would have one entry for every note in the Score.

Also I would probably use named fields. So I would have something like

data Note = Note
  { nPitch :: Int
  , nVolume :: Int 
  , ...
  }

When I'm accessing the data on ContextNote, I don't want to have to type several accessor functions to get down to the Note fields. So I could do something like

class HasNoteData a where
  pitch :: a -> Int

Then 

instance HasNoteData ContextNote where
  pitch (ContextNote _ _ note) = nPitch note

I could create a bunch of accessor functions like that for every field in the ContextNote.

For data that should probably be computed once, I could create Memo data. Maybe I would have

data ContextNote = ContextNote Memo Score Part Chord Note

or something like that.

So if you have read this far, thank you very much. Any suggestions welcome.

D














_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners




--
Chris Allen
Currently working on http://haskellbook.com

_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: help with music application

Dennis Raddle


On Tue, Dec 1, 2015 at 3:13 PM, Christopher Allen <[hidden email]> wrote:
Could you post the code to a public repository? Ideally with some example input and expected output if you're that far along.



It's running fine and mostly complete right now, so just as an aside this is a rewrite project to make it cleaner and easily to modify in the future. But about posting it, it's pretty enormous, many files grouped into about eight modules. What do you recommend I do? Create a BitBucket repository?

D




_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: help with music application

Michael Martin

Github

On Dec 1, 2015 5:23 PM, "Dennis Raddle" <[hidden email]> wrote:


On Tue, Dec 1, 2015 at 3:13 PM, Christopher Allen <[hidden email]> wrote:
Could you post the code to a public repository? Ideally with some example input and expected output if you're that far along.



It's running fine and mostly complete right now, so just as an aside this is a rewrite project to make it cleaner and easily to modify in the future. But about posting it, it's pretty enormous, many files grouped into about eight modules. What do you recommend I do? Create a BitBucket repository?

D




_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners


_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: help with music application

Christopher Allen
In reply to this post by Dennis Raddle
Github or Bitbucket are fine, according to your preference. Some people might complain if it's not Github since there's more people there. I personally don't care.

On Tue, Dec 1, 2015 at 5:22 PM, Dennis Raddle <[hidden email]> wrote:


On Tue, Dec 1, 2015 at 3:13 PM, Christopher Allen <[hidden email]> wrote:
Could you post the code to a public repository? Ideally with some example input and expected output if you're that far along.



It's running fine and mostly complete right now, so just as an aside this is a rewrite project to make it cleaner and easily to modify in the future. But about posting it, it's pretty enormous, many files grouped into about eight modules. What do you recommend I do? Create a BitBucket repository?

D




_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners




--
Chris Allen
Currently working on http://haskellbook.com

_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: help with music application

Dennis Raddle
In reply to this post by Dennis Raddle


On Tue, Dec 1, 2015 at 3:24 PM, Andrew Bernard <[hidden email]> wrote:
Hi Dennis,

This is a massively difficult and long task, fun of musical complexity. Are you aware of lilypond, the open source engraving program? It can export to MIDI and has been developed for over twenty years by hundreds of people. It is not written in Haskell, but C++ and Scheme principally.

Not meaning to disparage your work in any way, but it does seem you are reinventing the wheel, and a rather big and complicated wheel. I am sure it is an interesting exercise, and perhaps Sibelius does not play MIDI – I use lilypond so I do not know.

Help you don’t mind me letting you know.


Hi Andrew, I'm copying the list on this message because these are some relevant questions to the task as a whole. I am aware of LilyPond, and I also own two commercial programs, Finale and Sibelius. Here are the reasons I am reinventing the wheel.

1. I want to experiment with algorithms that introduce "human expression" into the playback. Finale, Sibelius, and Lilypond can export to MIDI, but the results sound mechanical and lifeless -- to my picky ears, anyway. (many composers don't mind). Actually, Sibelius claims to use "human expression" but my point is that I want to experiment with my own algorithms and I want to be able to get into the source code to implement and modify things.

2. Trying to get the musical data into my program via exporting from Sibelius or Finale to MIDI won't work. MIDI is poor way to express musical structures, and cannot represent symbols and expressive marks in the score. 

3. Here's one of the most significant factors. I didn't mention anything yet about what happens to the MIDI after I produce it. What happens is that I feed it to a "software synthesizer." This is a program that produces sound, by simulating or playing little bits of prerecorded notes. Here's what is a big deal -- there is hardly any standard for how software synthesizers respond to MIDI, especially all the control events. And every software synthesizer has custom capabilities. Finale and Sibelius claim to be able to do special controlling of the most popular synthesizers, but their capability is limited to one or two at most, and it actually hard to control (and maybe even buggy). I own three very good synthesizers, and two of them have controls that Sibelius and Finale don't understand in the slightest. I need many of these controls for my "human playback" algorithms.

4. Last but not least, I needed a personal project to learn Haskell (I have not and never will use it for work) so this is fun, and when you consider how it lets me experiment and use non-standard synthesizers, really quite practical in the end.

5. Also, I can limit my functionality to what I need. The big programs like LilyPond need to handle all sorts of instruments and notations. Say, guitar. That's a whole project in itself. But I'm not using guitar. I'm only using maybe 20% of the types of symbols and expressive marks that the big programs handle.

D





_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: help with music application

Dennis Raddle


On Tue, Dec 1, 2015 at 4:05 PM, Dennis Raddle <[hidden email]> wrote:


5. Also, I can limit my functionality to what I need. The big programs like LilyPond need to handle all sorts of instruments and notations. Say, guitar. That's a whole project in itself. But I'm not using guitar. I'm only using maybe 20% of the types of symbols and expressive marks that the big programs handle.




Oh, in case it wasn't clear, I don't handle typesetting at all... I'm still using Sibelius or Finale to do that. I can export via MusicXML rather than MIDI, which gets me most of the symbols on the score and organizes the structure of the music. My program is MusicXML -> MIDI, via custom algorithms.

D



_______________________________________________
Beginners mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners