Diving into the Lilypond

Tools and seams

Mandy Brown has written a fine blog post outlining her relation to writing, technology and independent publishing. “Questions about writers’ tools are both fascinating and repulsive”, she says, recognising the danger of fetishizing tools and in so doing procrastinating getting down to the writing itself.

But assuming we recognize that risk and take pains to avoid it, talking about our tools can be instructive. Routines and tools do matter—not only for writers but for any skill or trade.

Writing here was never just about sharing my writing but also about building systems for that writing: systems that could enable a particular kind of writing, which may even be unique to me (or as unique as anything gets, anyway). And so the process for how that writing comes together, and the tools I use to build it, are as important to me as the rhythm of any particular sentence.

Mandy’s essay pretty much sums up my experiences with building my own website (in my case with the help of the Kirby CMS – the details of which I can unfold on another occasion). Getting those systems in place has certainly taken more time than had I used readymade out-of-the-box solutions, but my productivity (and delight in the process) has certainly increased as a result – something others have apparently experienced as well. While so much of the technology we use attempts to cover up seams for the sake of convenience, diving into some of the more geeky aspects opens up for a more solid relationship with the tools we are using, ultimately placing us in a more powerful position.

Which brings us to Lilypond.

Text files and musical notation

As I mentioned in my previous post Lilypond is open source software that makes use of text files to generate musical notation. The text file is human readable even though it doesn’t look anything like regular musical notation. c4 d e c | c d e c | e f g2 | for example, where the 4 indicates a quarter note rhythm, the 2 a half note and the letters the common note names. With a little practice one can ‘read’ the music just by looking at the text file. Take a look at the full Frère Jacques on the wikipedia to get a better idea of how it works. The text file used to generate the notation can be seen below the Lilypond generated example.

What’s different from a WYSIWYG notation program like Finale or Sibelius is the separation of input and output. Lilypond first has to compile your text file input before you can take a look at the results and this can take some getting used to. In my case I’ve been used to this generative step with the work I’ve done on my website over the years and given the advantages of having a plain text basis on which to build from it’s been an easy transition. And as I mentioned in my Surface Tension post the shift from text file to preview can even become a useful part of one’s workflow – providing a fresh view of your work, somewhat equivalent to printing something out in the old days.

Standalone vs. Integrated Developer Environments

Not much is needed to get started. Lilypond can be downloaded from the Lilypond website and using a basic text editor (e.g. TextEdit on a Mac) you can get Lilypond to generate a PDF with music notation.

So far so good. TextEdit does the job just fine but it’s really useful to have a text editor with syntax highlighting as soon as things start to get a little more complicated. Having line numbers in your text editor can be helpful too, especially when fixing errors after compiling.

My first stop in the search for a better editing environment was following up on the suggestions made in Create Digital Music’s article on Lilypond. Frescobaldi, their first suggestion, is a program that integrates the text editor and PDF output into a Lilypond IDE. At the time (September 2013) installing Frescobaldi on a Mac was a somewhat complicated procedure (this has since changed – more on that later) which led me to explore other possibilities first. CDM’s next suggestion was the LilypondTool – a plugin for the Java based text editor jEdit. Peter Kirn warns in the CDM article that jEdit looks somewhat antiquated on first run and makes some suggestions on how to ‘de-uglify’ it. This certainly helped but I was sill left with an environment that simply didn’t appeal to me. It took the joy out of the clean simplicity of working with plain text and just didn’t feel right on my Mac – perhaps I’ve been spoilt by the polish and attention to detail of programs like Coda – the tool I use for my web projects. It also turned out that the developer would no longer be developing the plugin. A dead end.

Elysium was another Lilypond plugin I came across, this time for Eclipse, another Java based IDE, but after sniffing around a bit I decided to move on.

Giving up on my hopes for an integrated environment I turned to simply finding a text editor with Lilypond syntax highlighting. The Lilypond website had some suggestions and on finding that the Emacs editor is included with every Mac¹ I jumped into seeing what could be done there. Emacs turned out to involve a pretty steep learning curve. It’s a world unto itself and even though Lilypond environments have been created specially for it I didn’t manage to get them set-up. Perhaps I should have tried a fresh install of Emacs for OSX but as the Lilypond site points out, Emacs is probably most suitable for those who know and use the editor already. In the end it turned out to be overkill for my purposes and I turned to looking for a lightweight text editor with support for the Lilyond syntax. I’m glad I went through the Emacs detour though. It gave me some insight into an important piece of computer history and I came away with something that I now make use of every day – the Emacs navigation shortcuts² that work in most OSX applications.

Smultron

Update: For my (much improved) current setup using the Atom editor see this note.

After investigating a few possibilities I ended up settling on Smultron, a text editor with built in support for the Lilypond syntax that can be bought for a few crowns on the Mac App Store. ( There are iOS versions as well, should you want to edit your text files on an iPad, for example³. ) It works pretty well straight out of the box – as long as your text file has a .ly extension the syntax will automatically be recognised⁴. Smultron has the smart feature of being able to ‘preview’ lilypond documents in Lilypond itself through a ⌘U keystroke. That takes you to Lilypond in which ⌘R generates a PDF. I.e. the double stroke of ⌘U ⌘R takes you from your Lilypond text file (open in Smultron) to the resulting PDF (in the Preview app).

It is possible to add a small script⁵ to Smultron which will enable you to run Lilypond directly from Smultron and generate the PDF with a single keystroke. However since OSX automatically saves versions of text files this creates a warning that the text file has been changed by another application and asks which version you would like to retain. There might be some way around this, but since I would like to keep my version history and at the same time don’t wish to be constantly confronted with a warning, the double stroke of ⌘U ⌘R has now become an integrated part of my workflow.

The next step, with Smultron, Lilypond and Preview now chained together, was to get the various windows organised on my screen. I started off using a (free) app called Spectacle which has also turned out to be fantastically useful with everything else I use my computer for. With a single keystroke a window can be assigned to either the left or right, top or bottom half of the screen, or simply centred. A further click of the left or right arrows adjusts the window in thirds. That’s enough to get something approaching an reasonable work environment and the fact that each app is independent turns out to have advantages of its own – more on that in a moment.

MIDI and XML

The final, or rather first, part of my workflow involves MIDI files and Ableton Live. I’ve found the combination of Ableton and Max for Live to be both powerful, flexible, and perhaps most importantly – really easy to use. Unlike Finale, MIDI files in Ableton can be manipulated on the fly and should the piece involve sound or video files these can easily be synced and coordinated. This is really valuable, especially in the early ‘sketch’ phase of a composition. It’s incredibly easy to pull things around and listen to the results. Live’s session view is also invaluable in that it doesn’t tie you down to a timeline.

Once the basic material is in place and the MIDI files have been exported there is unfortunately, as things stand at the moment, an extra step before they can be imported into Lilypond. Lilypond does have a built in midi2ly converter – but it unfortunately doesn’t work on 64bit Macs and it’s not clear when the issue will be resolved. Lilypond’s XML to Lilypond converter does however do exactly what it needs to do and so at the moment it’s necessary to first convert the MIDI files to the music XML format (I’m using an old copy of Finale for that). Lilypond’s XML converter is run from the command line which is by no means difficult, but perhaps a little unusual for those used to GUI interfaces.

The commands can be run from the Terminal app which can be found in the Utilities folder of your Mac. Once opened you need to tell it where to find the musicxml2ly conversion file. Simply open your applications folder and option (secondary/right) click on the Lilypond application icon. Select “Show Package Contents” and constituent parts that make up Lilypond will be revealed. Open the ‘Contents’ folder, then the ‘Resources’ folder followed by the ‘bin’ folder to find the musicxml2ly file. Drag that file onto the terminal window and the path displayed should look something like this:
/Applications/LilyPond.app/Contents/Resources/bin/musicxml2ly ⁶.

Once you have that path established it’s simply a question of dropping the XML file into the space next to it, hitting return and voila your XML file has been converted to Lilypond. This greatly simplifies the process of quickly getting large numbers of notes into Lilypond – and they can be further manipulated from there. Typing the long musicxml2ly file path and the additional options that follow it can be tedious if making conversions are a regular part of your workflow and it’s here that TextExpander can be really useful. TextExpander saves bits of text that you regularly use and expands them whenever you type an assigned shortcut. In my case typing ‘mxml’ into the terminal window fills out my full conversion snippet and saves a lot of time when repeatedly converting XML files. TextExpander is incredibly useful in conjunction with Lilypond, not only when making command line conversions. I have snippets set up for Tempo changes, time signatures, special articulations and so on. Once integrated these shortcuts greatly speed up and simplify the process of creating Lilypond files. TextExpander is really useful outside of Lilypond as well and a great utility to have on your computer.

Frescobaldi

As of November 2013 Frescobaldi, the IDE which I at first didn’t consider due to installation complexities, is included in MacPorts which makes installation much more straightforward. Frescobaldi has since also been added to Homebrew providing an alternative method. Simply follow the instructions on Github.

Frescobaldi seems nicely put together and adds the functionality of being able to point and click on notes in the PDF and have the corresponding place in the text file highlighted – a very useful feature, especially when working with complex scores. I have to admit though that I haven’t really spent much time exploring Frescobaldi in any detail.

Screen Space

It turns out I’ve gotten quite comfortable with my Smultron workflow and that the flexibility that comes with not having all the parts integrated – can be an advantage when using it alongside Ableton. Sometimes I find it useful to take a look at a MIDI file in Ableton alongside the Lilypond text file, sometimes I like to have Ableton open next to the PDF score, perhaps for a check through while listening. With Spectacle I can easily shift back and forth with a keystroke depending on what it is I need to have in front if me. This is especially useful with the limited screen space available on my 13″ laptop.

On my 27″ iMac it’s possible to have all the parts of the Ableton - Smultron - Lilypond - Preview stack visible at once but the luxury of all that gorgeous screen space also brings with it an organizational challenge – windows spread out all over the place can quickly descend into chaos and here I use Moom for the extra possibilities that it opens up⁷. Again the advantage of not having all the parts integrated is that windows can be placed, resized, opened as needed and shifted between depending on where I am in the process. It’s quite possible that I’ll look into Frescobaldi in more depth at a later stage, but for now I have something that works for me, seams and all.

While setting up a Lilypond workflow could be better described on the Lilypond website, the learning and notation manuals they provide are quite solid. The steep learning curve encountered in the first weeks and months does eventually tail off and the joy of generating beautiful notation in this way starts to kick in. There’s also Scores of Beauty, a Lilypond blog that’s well worth checking out for tips and inspiration.

From a composer’s point of view

Lilypond has already proved itself as an typesetting tool but as is probably clear from this article I’ve been more interested in how it can be integrated into compositional workflows. During the first weeks of experimenting with it my initial reservations were brushed away when, for example, I realised how easy it was to create ossia measures or a temporary staff on the fly. No staff that’s hidden in the background for the duration of your piece – Lilypond can create what you you need when you need it. ( I immediately thought of Stravinsky’s late pieces. ) It’s also great when you’d like to construct a large single score from a number of smaller sections (as I am in the midst of doing) which can be a bit of a nightmare in Finale. In fact knowing that this was something I could easily do in Lilypond was one of my main motivations for pressing through with the initial learning curve. It has made it possible for me to compose in a specific way knowing that sections can both easily be put together and moved around.

Lilypond is also great at things like independent meters – take a look at this blog post comparing Lilypond and Finale. Brian Ferneyhough is mentioned in that post and I’ve already touched on the ways in which compositional processes can be integrated with the generation of the notation in my previous Surface Tension post. Take a look at project Abjad’s deconstruction of Ferneyhough’s Unsichbare Farben for a further example.

I’ve also noticed that a number of ‘neo-conceptual’ scores are being created using Lilypond. This makes perfect sense when ‘found’ materials are re-arranged into new compositions: Erik Carlson’s alphabetised version of Schubert’s Winterreise, for example. Manipulating or re-arranging blocks of a text file is infinitely easier (and likely to work) than copy pasting in Finale.

Finale has become something equivalent to Microsoft Word for me. It’s the default format in common use by everybody and anybody and there’s no denying all the advantages that come with that, but its no longer my tool of choice. Writing this article using markdown in Writer Pro makes the process a joy and I can’t imagine ever using Pages or Word again for my own writing projects. At the same time the format isn’t tied to a particular app or platform and in this regard each individual can make the choices that suit them. Likewise with Lilypond. One is free to select the editing tools that fit one’s tastes and needs and build workflows suitable for the project at hand. That freedom may bring its own challenges and frustrations but I’m enjoying the process and the results are beautiful. And there are others that apparently feel the same way too.

It seems as if interest in Lilypond is growing and I’m looking forward to seeing how both the application itself and the community around it develop in the time to come. The hurdles to entry are certainly greater than Finale or Sibelius but then so are the rewards. It might not be for everyone but once you’ve developed a taste for it there’s no turning back.