docs/random/wingo/pro-audio-with-gstreamer - imx-gstreamer - Git at Google

 -*- outline -*-

 * Pro Audio with GStreamer

 This file attempts to document usage of GStreamer for so-called "pro
 audio"[0]. Two audiences are considered: programmers that are
 considering GStreamer for their pro-audio app, and GStreamer developers
 interested in which parts of GStreamer pro-audio uses.

 [0] I actually don't like this term, because it's elitist. Of course
     other audio applications are not inferior, but they are different.
     I'll stick with the term out of established practice.

 ** What GStreamer Offers the Pro Audio Developer

 Choosing GStreamer for your application gives you lots of things for
 free.

 *** A high penetration into POSIX desktops

 GStreamer is included with Gnome, so you'll find it already installed on
 an increasing number of desktops. It makes it easier for a user to
 install your app. However, you still have to check for individual
 plugins that you depend on.

 *** An extremely flexible signal flow graph

 You have elements, connection points, different kinds of processing
 functions, schedulers, etc. You can subclass just about everything, or
 replace whole subsystems as you need to.

 All of this you would have to implement somehow. The downside is, of
 course, that it's extremely flexible. The graph isn't run by clock-tick
 -- the delays are carried out by the timekeeping element (if any), when
 execution reaches it. It's cooperative, rather than dictator-style like
 Jack. If all problems have been worked out, etc, it runs smoothly, but
 one poorly coded element can stall the graph.

 Restricting graph operation to clock-ticks and using buses instead, like
 SuperCollider 3, would introduce many simplifications to scheduling and
 such, I would think. However, you'd still have to implement your
 signal-flow infrastructure from scratch if you decided to go it alone.

 I might revise the above paragraph, though. I like GStreamer's level of
 flexibility a bit too much :)

 *** A wide variety of existing plugins

 This includes inputs like ALSA, OSS, sndfile, etc, as well as their
 corresponding sinks (outputs). Then there are the network transports.
 And the sound servers (including Jack). LADSPA plugins for free. Some
 DSP things, but admittedly not too much -- this is an area for future
 expansion.

 *** Generic plugin behavior

 Of course you still have to know some specifics about the plugins you
 use (which properties they have, for example), but in general elements
 of a "pipeline" (signal flow graph -- and no, it doesn't have to look
 like a pipe) are replaceable. Your user can choose between ALSA or OSS
 or even ESD (shudder), and it's simple to implement.

 *** Easy threads

 Adding threads to your signal flowgraph does takes some thought, but
 once you've decided how to set things up it's reasonably easy.
 Unfortunately realtime threads aren't implemented yet, but that should
 be an easy project, knock on wood.

 *** Other Stuff

 GStreamer is big these days. I wouldn't say bloated, but there are a lot
 of subsystems relating to "media" that just aren't applicable to
 processing float data. There's a whole system (called "caps") that deals
 with negotiating common formats between elements, when all pro audio has
 to deal with is sample-rate and the number of frames per buffer. There's
 a typefinding and pipeline autoplugging subsystem. There's "tags", like
 from ID3 tags.

 You might find uses for these things, and thankfully these uses blur the
 lines between "pro" and "consumer" audio. To an extent, these features
 complicate GStreamer programming. But mostly they stay out of your way
 -- besides caps, they only bother you when you ask them to :-)

 ** Pro Audio for GStreamer Programmers

 Pro audio is a restricted, almost purely mathematical domain. There's
 not that much to worry about. Each channel is separate from the rest
 (never interleaved). All data is in float format, and native byte order.
 The sample rate is typically the same in the whole system. Same with the
 number of frames in a buffer.

 So it's simple, but it's different from "normal" audio processing (a
 whole mess of variables to synchronise and convert between, interleaved
 data, codecs, etc). But it's sufficiently different that in the past
 we've had discussions every 8 months or so about why things are
 implemented in such-and-such a way, and why don't we change them, and so
 on. So this part of the document is aimed at GStreamer developer's as a
 kind of documentation for the whole float-caps space.

 *** The Format

 Pro audio deals with floats. I'm not really worried about doubles --
 although LADSPA carefully #define's LADSPA_Sample so you can override
 it, everything's in float.

 There are two variables to be concerned about. One is sample rate, which
 is pretty obvious. The not-so-obvious one is buffer-frames, specifying
 the number of frames that will come in a buffer. If a buffer has fewer
 frames, that indicates EOS is coming on the next pull. This property is
 an optimization to allow easy chaining of buffers in multi-pad elements,
 as well as to prevent deadlocks in circular pipelines, and to comply
 with systems like Jack that operate on clock ticks.

 *** Channels

 One variable that is not in pro-audio is the number of channels in a
 stream. Streams are always mono. All DSP algorithms expect to receive
 mono data. Multichannel processing is done via multiple inputs. This is
 the complicated part of pro audio for GStreamer, because it means lots
 of multi-pad elements, and complicated pipelines, which is a pain to
 code for (if you're not coding it in Scheme, of course ;). So yes, it's
 kindof a pain, but it is a flexibility that's necessary.

 *** Stability

 DSP routines written years back still work, because all you need to use
 them is to -lm. GStreamer is a step towards DLL hell. And audio
 developers are a funny bunch. Look at Paul Davis's Ardour CVS, for
 instance. He has a local copy of every library ever coded, ever. No
 joke.

 If our platform is to remain attractive to this group, we need to start
 to stabilize the way GStreamer works. Of course API and ABI change,
 we're young. But outside of media-related work, the core is pretty
 stable. When we move to change things after 0.8, changes should be well
 documented.

 That's all pretty normal, but there is one special consideration. DSP
 involves lots of custom plugins, maintained outside the GStreamer tree.
 So just because you grep the tree and don't find an instance of X
 function or whatever, it doesn't necessarily mean the feature/behaviour
 is unused. This will be increasingly true for other GStreamer users in
 the future, but it's true now for DSP. I'm talking about me now ;)

 OK, enough rambling. Hope this clarifies things a bit.

 Andy Wingo, 24 Jan 2004.
	-- outline --

	* Pro Audio with GStreamer

	This file attempts to document usage of GStreamer for so-called "pro
	audio"[0]. Two audiences are considered: programmers that are
	considering GStreamer for their pro-audio app, and GStreamer developers
	interested in which parts of GStreamer pro-audio uses.

	[0] I actually don't like this term, because it's elitist. Of course
	other audio applications are not inferior, but they are different.
	I'll stick with the term out of established practice.

	** What GStreamer Offers the Pro Audio Developer

	Choosing GStreamer for your application gives you lots of things for
	free.

	*** A high penetration into POSIX desktops

	GStreamer is included with Gnome, so you'll find it already installed on
	an increasing number of desktops. It makes it easier for a user to
	install your app. However, you still have to check for individual
	plugins that you depend on.

	*** An extremely flexible signal flow graph

	You have elements, connection points, different kinds of processing
	functions, schedulers, etc. You can subclass just about everything, or
	replace whole subsystems as you need to.

	All of this you would have to implement somehow. The downside is, of
	course, that it's extremely flexible. The graph isn't run by clock-tick
	-- the delays are carried out by the timekeeping element (if any), when
	execution reaches it. It's cooperative, rather than dictator-style like
	Jack. If all problems have been worked out, etc, it runs smoothly, but
	one poorly coded element can stall the graph.

	Restricting graph operation to clock-ticks and using buses instead, like
	SuperCollider 3, would introduce many simplifications to scheduling and
	such, I would think. However, you'd still have to implement your
	signal-flow infrastructure from scratch if you decided to go it alone.

	I might revise the above paragraph, though. I like GStreamer's level of
	flexibility a bit too much :)

	*** A wide variety of existing plugins

	This includes inputs like ALSA, OSS, sndfile, etc, as well as their
	corresponding sinks (outputs). Then there are the network transports.
	And the sound servers (including Jack). LADSPA plugins for free. Some
	DSP things, but admittedly not too much -- this is an area for future
	expansion.

	*** Generic plugin behavior

	Of course you still have to know some specifics about the plugins you
	use (which properties they have, for example), but in general elements
	of a "pipeline" (signal flow graph -- and no, it doesn't have to look
	like a pipe) are replaceable. Your user can choose between ALSA or OSS
	or even ESD (shudder), and it's simple to implement.

	*** Easy threads

	Adding threads to your signal flowgraph does takes some thought, but
	once you've decided how to set things up it's reasonably easy.
	Unfortunately realtime threads aren't implemented yet, but that should
	be an easy project, knock on wood.

	*** Other Stuff

	GStreamer is big these days. I wouldn't say bloated, but there are a lot
	of subsystems relating to "media" that just aren't applicable to
	processing float data. There's a whole system (called "caps") that deals
	with negotiating common formats between elements, when all pro audio has
	to deal with is sample-rate and the number of frames per buffer. There's
	a typefinding and pipeline autoplugging subsystem. There's "tags", like
	from ID3 tags.

	You might find uses for these things, and thankfully these uses blur the
	lines between "pro" and "consumer" audio. To an extent, these features
	complicate GStreamer programming. But mostly they stay out of your way
	-- besides caps, they only bother you when you ask them to :-)

	** Pro Audio for GStreamer Programmers

	Pro audio is a restricted, almost purely mathematical domain. There's
	not that much to worry about. Each channel is separate from the rest
	(never interleaved). All data is in float format, and native byte order.
	The sample rate is typically the same in the whole system. Same with the
	number of frames in a buffer.

	So it's simple, but it's different from "normal" audio processing (a
	whole mess of variables to synchronise and convert between, interleaved
	data, codecs, etc). But it's sufficiently different that in the past
	we've had discussions every 8 months or so about why things are
	implemented in such-and-such a way, and why don't we change them, and so
	on. So this part of the document is aimed at GStreamer developer's as a
	kind of documentation for the whole float-caps space.

	*** The Format

	Pro audio deals with floats. I'm not really worried about doubles --
	although LADSPA carefully #define's LADSPA_Sample so you can override
	it, everything's in float.

	There are two variables to be concerned about. One is sample rate, which
	is pretty obvious. The not-so-obvious one is buffer-frames, specifying
	the number of frames that will come in a buffer. If a buffer has fewer
	frames, that indicates EOS is coming on the next pull. This property is
	an optimization to allow easy chaining of buffers in multi-pad elements,
	as well as to prevent deadlocks in circular pipelines, and to comply
	with systems like Jack that operate on clock ticks.

	*** Channels

	One variable that is not in pro-audio is the number of channels in a
	stream. Streams are always mono. All DSP algorithms expect to receive
	mono data. Multichannel processing is done via multiple inputs. This is
	the complicated part of pro audio for GStreamer, because it means lots
	of multi-pad elements, and complicated pipelines, which is a pain to
	code for (if you're not coding it in Scheme, of course ;). So yes, it's
	kindof a pain, but it is a flexibility that's necessary.

	*** Stability

	DSP routines written years back still work, because all you need to use
	them is to -lm. GStreamer is a step towards DLL hell. And audio
	developers are a funny bunch. Look at Paul Davis's Ardour CVS, for
	instance. He has a local copy of every library ever coded, ever. No
	joke.

	If our platform is to remain attractive to this group, we need to start
	to stabilize the way GStreamer works. Of course API and ABI change,
	we're young. But outside of media-related work, the core is pretty
	stable. When we move to change things after 0.8, changes should be well
	documented.

	That's all pretty normal, but there is one special consideration. DSP
	involves lots of custom plugins, maintained outside the GStreamer tree.
	So just because you grep the tree and don't find an instance of X
	function or whatever, it doesn't necessarily mean the feature/behaviour
	is unused. This will be increasingly true for other GStreamer users in
	the future, but it's true now for DSP. I'm talking about me now ;)

	OK, enough rambling. Hope this clarifies things a bit.

	Andy Wingo, 24 Jan 2004.