Hannu’s Blog

Once again we need to return back to the history about 15 years ago.

I was very enthusiastic about the Yamaha OPL(2) FM synthesizer chip found on my SB card when I started woring on the Sound Blaster driver for Minix and Linux. I soon realized that it was not possible to maintain precise timing in application level because both Minix and Linux (at that time) had rather bad scheduling latencies. An application that written the FM chip register directly (using some ioctl calls) couldn’t play any MIDI files with acceptable timing precision.

The solution was to move the timing stuff to the kernel space where the (100 Hz) timer interrupts worked much better. So the /dev/sequencer interface was born. In the beginning this interface was only used to drive the FM chip. Then few months ago the Gravis Ultrasound card was released which was even more exciting. I quickly hacked the /dev/sequencer API to support the hardware wave table engine of Ultrasound and added yet another set of ioctl calls for patch management. In addition the API was expanded to support MIDI (serial) interface ports too. Also support for the better OPL3 FM chip (4 operators instead of just two) was added.
From the very beginning the sequencer API had an event queue where the application could write 4 byte event records such as instrument change, note on and note off. The event queue was read by a timer callback that forwarded the events to the actual device and removed them from the queue. Timing was “great” and the result sounded good. I even did a “gmod” program (or whatever it was called) that played module files (.MOD) using Ultrasound’s wave table engine. That was a great improvement because my mighty 386/25 PC was able to play nice music without hiccup while compiling the kernel in another VT. Similar player was soon implemented for MIDI by Greg Lee.
The problem was that now there were 3-4 slightly different variations of the sequencer API (OPL2, OPL3, Gravis and MIDI ports) that all required slightly different handling in the applications. The solution was to create another version of the sequencer API that was called /dev/sequencer2 and later /dev/music). The idea was to expand the event records to 8 bytes which was enough to distribute MIDI messages to several synthesizer devices and MIDI ports at the same time. The MIDI port driver the converted the events back to MIDI while the hardware synth drivers played them directly. I also got added stpport for the Music Quest MQX-32M MIDI adapter that supported “MPU-401 intelligent mode” with features such as SMPTE timing.
The result looked and sounded great and I was very proud about it. Applications still had to support two different “patch loading” methods for different cards but that was solved by adding an interface for external patch management daemons. Finally it was possible that one program can play MIDI files to any device(s) without any device dependent code.

The real problems started few months after that when I tried to document the sequencer/music interface. The concept itself was easy to document. However there were tens of different event types. Each MIDI message (note on, note off, MTC/SMTPE, etc) had one of them. The MIDI message system itself was documented in the official MIDI 1.0 specification published by MMA. However I should have to duplicate all that to explain how to translate MIDI messages to the sequencer events and vice versa. Due to lack of time I gave up but promised to write the documentation later.

Then few years later I was writing another version of the OSS API documentation for OSS 3.8 and again the job stalled at the same point. I got something written down but then I had to give up. Once again I just promised to complete the documentation later. Unfortunately developers coudn’t do anything usefull with the interface without documentation so it didn’t get widely supported.
Finally couple of years ago I was working on the OSS 4.0 version and noticed that no applications were actually using the sequencer API any more. All Linux MIDI applications were already using ALSA so the OSS sequencer API was no longer active. There was no point in keeping the current sequencer stuff in OSS and we decided remove it and rewrite it from scratch. At the same time we could fix the known problems in it.

But what should be actually be done with that stuff? The main problem was apparently the documentation and we were thinking about hiring a technical writer to do that part. But there was something else that should have been redesigned before that. But what should be changed and how?

There were some major problems that were pointed to us by some MIDI developers before they jumped to the ALSA boat:

• Only one application was able to use the sequencer at the same time. That made it impossible to implement applications like software synthesizers.
• The API was said to be “playback only”. It was not suitable for interactive performances because echoing live keyboard events to the output devices caused hanging notes. Btw, the reason for this was actually not the API design but a bug in the parser that translates incoming MIDI messages to OSS sequencer events. It didn’t handle running status correctly. We noticed this problem later when the same code was used somewhere else in OSS.
• Because there is only one timing queue it’s possible that flood of events going to one device may delay events sent to the other (non-flooded) ones.
• Some software based synthesizer engines (such as the SoftOSS virtual wave table engine of OSS) have greater latencies than many hardware based implementations (such as FM and wave table chips). It is necessary to compensate this by sending events to such devices few microseconds before the scheduled time. In this way the notes will play simultaneously on all devices.

These problems should have been easy to fix while rewriting the code. In addition all the supported synthesizer chips were out of production (they were ISA based) so the amount of code to rewrite was not that big. However something looked to be seriously wrong and I couldn’t realize what it was.

Finally on one stormy night I realized it: The problem was not just the documentation. The whole /dev/sequencer concept was seriously bogus. It looked good some ten years earlier when it was designed. However the problems the previous scheme tried to solve were all gone with the old ISA cards. Fist of all the MIDI streams were first converted to the sequecer events which were then converted back to MIDI before sending them to the output device. And there was even a bigger issue.
When somebody is writing a new MIDI application he/she is (at least supposed to be) familiar with the MIDI 1.0 specification. In addition to the official spec MIDI is also documented in many books and on many internet sites. The sequencer concept requires that the developer knows how to map the MIDI messages to the sequencer events and vice versa. This makes the development process rather difficult if the application should support some advanced MIDI concepts such as MMC. In particular the sequencer scheme makes it difficult to port MIDI applications from the other environments that use APIs derived from the plain MIDI. So it didn’t make any sense to rewrite the sequencer API hoping that somebody will manage to document it in the future.

Instead what we are working on now is a different MIDI subsystem based on plain MIDI message streams with some timing headers. The application simply packs all MIDI messages to be sent at given timing tick to a packet. The packet has a header record that contains the transmission time and some other control information. The MIDI core then appends the records to the output queue of the target device and finally sends them to the device at the right moment. Input is handled in the same way.

There is no central /dev/sequencer device file that would be common to all MIDI/synth devices. Instead each target/source devices have their own /dev/midi# device. An application can open as many MIDI device files as it wants and to slave them to the same timer device (it’s also possible that each MIDI devices use different timers with independent tempo/etc settings if that makes any sense).

It is possible to handle playback of live keyboard input by inserting received MIDI bytes to the head of the queues of the output devices. Such live packets will be played as soon as the target device has played all the MIDI bytes from the currently active (incompletely played) packet. In most situations there are no packets currently playing (the device is waiting for it’s scheduled time) so the delays will be minimal. In the future it’s also possible to implement automatic kernel level echo and rerouting capability on top of this simple API.

Running status is handled so that the application “loses” or flushes the status at packet boundaries. Each packet will be started with full status byte even if the previous message had the same status byte. This doesn’t cause any performance problems. Running status is only needed when large number of MIDI events (say pitch bend changes) need to be sent rapidly to the device. A packet boundary normally means that there is a pause in the output until next bytes need to be sent so retiring the running status doesn’t cause any timing problems. Handling running status in this way makes it possible to insert live packets to the head of the queue. The application just needs to make sure that the MIDI channel used for echo messages is not used by the sequence that is currently playing.

Support for software based synthesizers (wave table, modelling, etc) is handled by a special MIDI loopback driver. Each loopback device pair has two ends. The “server” device is used by the synthesizer application. It can change the device name seen by the “client” side to something descriptive such as “ACME super hyper modelling synth”. The user of the client application (say a MIDI player/sequencer) can pick the modelling synth from the device list and then start playing the MIDI sequence using it (input is also possible if that makes any sense). The server side can report it’s delay level (in microsecods) to the MIDI core so that timed events can be sent to it slightly ahead of time. In this way any latencies caused by the audio side can be compensated.

This is the good news. The bad news is that there is no MIDI support in OSS 4.0. There are few bugs in the MIDI code and we decided to ship OSS 4.0 without it (instead of delaying release of the other parts that were ready). MIDI support will be included in OSS 4.1 (hopefully within this year).

What about ALSA which has superior MIDI/sequencer API?

ALSA’s sequencer API is a ‘clone’ of the OSS’ one with steroids. While they have fixed the rest of the flaws of OSS the design is fundamentally the same. Even with the fundamental problem I mentioned above. It will be interesting to see if they ever manage to get it fully documented (or if anybody will ever care to document anything of it).

In addition ALSA has some “advanced” bells and whistles that make me suspicious. However this is just my understanding and I cannot check the details because there is no documentation available.

• Their timing model is rather odd. I have to confess but I have not managed to understand how it works. However they seem to use something else than the traditional tempo/timebase style MIDI timing.
• ALSA has sophisticated MIDI rerouting (alsaconnect) capability that is supposed to be able to route MIDI input from any device/application to the input of any MIDI output device or application. However the practice seems to be that applications bind themselves to some hardcoded devices that prevents this scheme from working.

This entry was posted on Sunday, April 8th, 2007 at 11:09 am and is filed under Open Sound System. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.