Dynamic audio buffer creation Number:7,386,356 from the United States Patent and Trademark Office (PTO) owispatent An audio generation system includes a buffer manager that creates audio buffers to receive streams of audio data from an audio data source, such as from a synthesizer component for example. The audio buffers include sink-in audio buffers that receive one or more streams of audio data when the streams of audio data are requested by the buffer manager for input to the sink-in audio buffers. The audio buffers also include mix-in audio buffers that receive streams of audio data from one or more of the sink-in audio buffers, and from any number of other mix-in audio buffers. The audio generation system includes an audio component, such as a speaker for example, that receives audio data from the sink-in audio buffers and produces an audio rendition corresponding to the audio data.<H creation, Dynamic, Dynamic, audio, b, 7386356.html, Patent, pto, 7386356

Title: Dynamic audio buffer creation

Abstract: An audio generation system includes a buffer manager that creates audio buffers to receive streams of audio data from an audio data source, such as from a synthesizer component for example. The audio buffers include sink-in audio buffers that receive one or more streams of audio data when the streams of audio data are requested by the buffer manager for input to the sink-in audio buffers. The audio buffers also include mix-in audio buffers that receive streams of audio data from one or more of the sink-in audio buffers, and from any number of other mix-in audio buffers. The audio generation system includes an audio component, such as a speaker for example, that receives audio data from the sink-in audio buffers and produces an audio rendition corresponding to the audio data.

Patent Number: 7,386,356 Issued on 06/10/2008 to Fay, et al.

Inventors: Fay; Todor J. (Bellevue, WA), Schmidt; Brian L. (Bellevue, WA), Porter; Dugan O. (Redmond, WA), Geist, Jr.; James F. (Kirkland, WA)
Assignee: Microsoft Corporation (Redmond, WA)

Appl. No.: 10/093,099

Filed: March 5, 2002

Related U.S. Patent Documents


Application Number	Filing Date	Patent Number	Issue Date
60273660	Mar., 2001

Current U.S. Class: 700/94

Field of Search: 700/94 719/328,321,52,53,56,310

References Cited [Referenced By]

U.S. Patent Documents


5142961	September 1992	Paroutaud
5303218	April 1994	Miyake
5315057	May 1994	Land et al.
5331111	July 1994	O'Connell
5483618	January 1996	Johnson et al.
5511002	April 1996	Milne et al.
5548759	August 1996	Lipe
5565908	October 1996	Ahmad
5596159	January 1997	O'Connell
5717154	February 1998	Gulick
5734119	March 1998	France et al.
5761684	June 1998	Gibson
5768545	June 1998	Solomon et al.
5778187	July 1998	Monteiro et al.
5792971	August 1998	Timis et al.
5842014	November 1998	Brooks et al.
5852251	December 1998	Su et al.
5890017	March 1999	Tulkoff et al.
5902947	May 1999	Burton et al.
5942707	August 1999	Tamura
5977471	November 1999	Rosenzweig
5990879	November 1999	Mince
6044408	March 2000	Engstrom et al.
6100461	August 2000	Hewitt
6152856	November 2000	Studor et al.
6160213	December 2000	Arnold et al.
6169242	January 2001	Fay et al.
6173317	January 2001	Chaddha et al.
6175070	January 2001	Naples et al.
6180863	January 2001	Tamura
6216149	April 2001	Conner et al.
6225546	May 2001	Kraft et al.
6233389	May 2001	Barton et al.
6301603	October 2001	Maher et al.
6357039	March 2002	Kuper
6433266	August 2002	Fay et al.
6541689	April 2003	Fay et al.
6628928	September 2003	Crosby et al.
6640257	October 2003	MacFarlane
6658309	December 2003	Abrams et al.
2001/0053944	December 2001	Marks et al.
2002/0108484	August 2002	Arnold et al.
2002/0144587	October 2002	Naples et al.
2002/0144588	October 2002	Naples et al.

Other References

Vercoe, et al; "Real-Time CSOUND: Software Synthesis with Sensing and Control"; ICMC Glasgow 1990 for the Computer Music Association; pp. 209 through 211. cited by other .
Harris, et al.; "The Application of Embedded Transputers in a Professional Digital Audio Mixing System"; IEEE Colloquium on "Transputer Applications"; Digest No. 129, 2/ 1-3 (uk Nov. 13, 1989). cited by other .
Vercoe, Barry; "New Dimensions in Computer Music"; Trends & Perspectives in Signal Processing; Focus, Apr. 1982; pp. 15 through 23. cited by other .
Moorer, James; "The Lucasfilm Audio Signal Processor"; Computer Music Journal, vol. 6, No. 3, Fall 1982, 0148-9267/82/030022-11; pp. 22 through 32. cited by other .
J. Piche et al., "Cecilia: A Production Interface to Csound", Computer Music Journal vol. 22, No. 2 pp. 52-55 (Summer 1998). cited by other .
A. Camurri et al., "A Software Architecture for Sound and Music Processing", Microprocessing and Microprogramming vol. 35 pp. 625-632 (Sep. 1992). cited by other .
Berry M., "An Introduction to GrainWave" Computer Music Journal Spring 1999 vol. 23 No. 1 pp. 57-61. cited by other .
H. Meeks, "Sound Forge Version 4.0b", Social Science Computer Review vol. 16, No. 2, pp. 205-208 (Summer 1998). cited by other .
M. Cohen et al., "Multidimensional Audio Window Management", Int. J. Man-Machine Studies vol. 34, No. 3 pp. 319-336 (1991). cited by other .
Malham et al., "3-D Sound Spatialization using Ambisonic Techniques" Computer Music Journal Winter 1995 vol. 19 No. 4 pp. 58-70. cited by other .
Meyer D., "Signal Processing Architecture for Loudspeaker Array Directivity Control" ICASSP Mar. 1985 vol. 2 pp. 16.7.1-16.7.4. cited by other .
Miller et al., "Audio-Enhanced Computer Assisted Learning and Computer Controlled Audio-Instruction". Computer Education, Pergamon Press Ltd., 1983, vol. 7 pp. 33-54. cited by other .
R. Dannenberg et al., "Real-time Software Synthesis on Superscalar Architectures", Computer Music Journal vol. 21, No. 3 pp. 83-94 (Fall 1997). cited by other .
R. Nieberle et al., "CAMP: Computer-Aided Music Processing", Computer Music Journal vol. 15, No. 2, pp. 33-40 (Summer 1991). cited by other .
Reilly et al., "Interactive DSP Debugging in the Multi-Processor Huron Environment" ISSPA Aug. 1996 pp. 270-273. cited by other .
Stanojevic et al., "The Total Surround Sound (TSS) Processor" SMPTE Journal Nov. 1994 vol. 3 No. 11 pp. 734-740. cited by other .
V. Ulianich, "Project FORMUS: Sonoric Space-Time and the Artistic Synthesis of Sound", Leonardo vol. 28, No. 1 pp. 63-66 (1995). cited by other .
Waid, Fred; "APL and the Media"; Proceedings of the Tenth APL as a Tool of Thought Conference; held at Stevens Institute of Technology, Hoboken, New Jersey, Jan. 31, 1998; pp. 111 through 122. cited by other .
Wippler, Jean-Claude; "Scripted Documents"; Proceedings of the 7th USENIX Tcl/TKConference; Austin Texas; Feb. 14-18, 2000; The USENIX Association. cited by other .
Bargen, et al., "Inside DirectX", Microsoft Press, 1998, pp. 203-226. cited by other.

Primary Examiner: Tran; Sinh
Assistant Examiner: Flanders; Andrew C
Attorney, Agent or Firm: Lee & Hayes, PLLC

Parent Case Text

RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No. 60/273,660, filed Mar. 5, 2001, entitled "Dynamic Buffer Creation with Embedded Hardware and Software Effects", to Todor Fay et al., which is incorporated by reference herein.

Claims

The invention claimed is:

1. An audio generation system, comprising: an audio rendition manager corresponding to an audio rendition generated from audio data, the audio rendition manager including a synthesizer component, audio buffers, and logical buses that each correspond to one of the audio buffers; a buffer manager configured to create the audio buffers that receive one or more streams of the audio data generated by the synthesizer component, where each of stream of audio data is assigned to one or more of the logical buses that receive the audio data from the synthesizer component such that the audio data steams assigned to a particular logical bus are routed to an audio buffer that corresponds to the particular logical bus; a sink-in audio buffer created by the buffer manager and configured to receive a steam of the audio data, the buffer manager further configured to request the steam of audio data from a corresponding logical bus for input to the sink-in audio buffer; and an audio component configured to receive modified audio data from the sink-in audio buffer and produce the audio rendition corresponding to the modified audio data.

2. An audio generation system as recited in claim 1, further comprising a mix-in audio buffer created by the buffer manager and configured to: receive streams of audio data from one or more sink-in audio buffers; generate a stream of combined audio data that includes the streams of audio data received from the one or more sink-in audio buffers; and communicate the stream of combined audio data to the audio component to produce an audio rendition conesponding to the stream of combined audio data.

3. An audio generation system as recited in claim 1, wherein the sink-in audio buffer is instantiated as a programming object having an interface that is callable by a software component.

4. An audio generation system as recited in claim 1, wherein the sink-in audio buffer is instantiated as a programming object having an interface that is callable by a multimedia application to receive the stream of audio data.

5. An audio generation system as recited in claim 1, wherein the sink-in audio buffer is further configured to receive at least a second stream of the audio data, the buffer manager further configured to request the second stream of audio data from a corresponding logical bus for input to the sink-in audio buffer.

6. An audio generation system as recited in claim 1, further comprising a mix-in audio buffer created by the buffer manager and configured to: receive streams of audio data from one or more sink-in audio buffers which are created by the buffer manager after the mix-in audio buffer is created; generate a stream of combined audio data that includes the streams of audio data received from the one or more sink-in audio buffers; and communicate the stream of combined audio data to the audio component to produce an audio rendition corresponding to the stream of combined audio data.

7. An audio generation system as recited in claim 1, further comprising a mix-in audio buffer created by the buffer manager and configured to: receive streams of audio data from one or more sink-in audio buffers which are created by the buffer manager after the mix-in audio buffer is created; generate a stream of combined audio data that includes the streams of audio data received from the one or more sink-in audio buffers; and wherein the buffer manager deactivates the one or more sink-in audio buffers before the mix-in audio buffer is deactivated.

8. An audio generation system as recited in claim 1, further comprising: at least a second sink-in audio buffer created by the buffer manager and configured to receive a second stream of the audio data, the buffer manager further configured to request the second stream of audio data from a corresponding logical bus for input to the second sink-in audio buffer; and a mix-in audio buffer created by the buffer manager and configured to: generate a stream of combined audio data that includes audio data received from the sink-in audio buffer, and that includes audio data received from the second sink-in audio buffer; and communicate the stream of combined audio data to the audio component to produce an audio rendition corresponding to the stream of combined audio data.

9. An audio generation system as recited in claim 1, wherein the audio rendition manager is a programming object having an interface that is callable by a software component to generate the audio data.

10. An audio generation system as recited in claim 1, wherein the synthesizer component is instantiated as a programming object having an interface that is callable by a multimedia application to generate the audio data.

11. An audio generation system as recited in claim 1, wherein the synthesizer component is instantiated as a programming object having an interface that is callable by a multimedia application to generate the audio data in response to MIDI instructions.

12. An audio generation system as recited in claim 1, wherein the buffer manager is a programming object having an interface that is callable by a software component to create the audio buffers.

13. An audio generation system as recited in claim 1, wherein the buffer manager is a programming object having an interface that is callable by a multimedia application to create the audio buffers.

14. An audio generation system as recited in claim 1, further comprising a software component that defines the one or more logical buses corresponding to the sink-in audio buffer.

15. An audio generation system as recited in claim 1, further comprising a performance manager that includes at least one audio segment having audio content components that are each configured to generate audio instructions; and wherein the audio rendition manager includes the synthesizer component configured to process the audio instructions to generate the audio data.

16. An audio generation system as recited in claim 1, further comprising a performance manager that includes at least one audio segment having audio content components that are each configured to generate MIDI instructions; and wherein the audio rendition manager includes the synthesizer component configured to process the MIDI instructions to generate the audio data.

17. An audio generation system, comprising: an audio rendition manager corresponding to an audio rendition generated from audio data, the audio rendition manager including a synthesizer component, audio buffers, and logical buses that each correspond to one of the audio buffers; sink-in audio buffers each configured to receive one or more streams of audio data generated by the synthesizer component, where each stream of audio data is assigned to one or more of the logical buses that receive the audio data from the synthesizer component such that the audio data streams assigned to a particular logical bus are routed to a sink-in audio buffer that corresponds to the particular logical bus; and a mix-in audio buffer configured to receive the audio data from one or more of the sink-in audio buffers, the mix-in audio buffer further configured to generate a stream of combined audio data that includes the audio data received from the one or more sink-in audio buffers.

18. An audio generation system as recited in claim 17, further comprising an audio component configured to receive the stream of combined audio data from the mix-in audio buffer and produce an audio rendition corresponding to the stream of combined audio data.

19. An audio generation system as recited in claim 17, further comprising at least a second mix-in audio buffer configured to generate a second stream of combined audio data that includes the stream of combined audio data received from the mix-in audio buffer, and that includes one or more streams of audio data received from one or more of the sink-in audio buffers.

20. An audio generation system as recited in claim 17, further comprising a buffer manager configured to create the mix-in audio buffer and the sink-in audio buffers.

21. An audio generation system as recited in claim 17, further comprising a buffer manager configured to create the mix-in audio buffer and the sink-in audio buffers, the buffer manager further configured to request the one or more streams of audio data from the corresponding logical buses for input to each of the one or more sink-in audio buffers.

22. An audio generation system as recited in claim 17, further comprising a buffer manager configured to create the sink-in audio buffers and the mix-in audio buffer, the mix-in audio buffer being created before the one or more sink-in audio buffers are created to input the audio data into the mix-in audio buffer.

23. An audio generation system as recited in claim 17, further comprising a buffer manager configured to: create the sink-in audio buffers and the mix-in audio buffer, the mix-in audio buffer being created before the one or more sink-in audio buffers are created to input the audio data into the mix-in audio buffer; and deactivate the one or more sink-in audio buffers before the mix-in audio buffer is deactivated.

24. An audio generation system as recited in claim 17, further comprising a buffer manager configured to: implement resources to create the sink-in audio buffers and the mix-in audio buffer, the mix-in audio buffer being created before the one or more sink-in audio buffers are created to input the audio data into the mix-in audio buffer; and release the resources implemented to create the one or more sink-in audio buffers before the resources implemented to create the mix-in audio buffer are released.

25. An audio generation system as recited in claim 17, further comprising a buffer manager instantiated as a programming object having an interface that is callable by a software component, the buffer manager configured to: instantiate the mix-in audio buffer as a programming object having an interface that is callable by the software component; and instantiate each of the sink-in audio buffers as a programming object having an interface that is callable by the software component.

26. An audio generation system as recited in claim 17, further comprising a buffer manager instantiated as a programming object having an interface that is callable by a multimedia application, the buffer manager configured to: instantiate the mix-in audio buffer as a programming object in response to a request from the multimedia application to communicate a stream of audio data from the sink-in audio buffer to the mix-in audio buffer; and instantiate a sink-in audio buffer as a programming object in response to a request from the multimedia application to communicate a stream of audio data from a corresponding logical bus to the sink-in audio buffer.

27. An audio generation system as recited in claim 17, further comprising a buffer manager instantiated as a programming object having an interface that is callable by a multimedia application, the buffer manager configured to: instantiate the mix-in audio buffer as a programming object in response to a request from the multimedia application to communicate a stream of audio data from the sink-in audio buffer to the mix-in audio buffer; instantiate a sink-in audio buffer as a programming object in response to a request from the multimedia application to communicate a stream of audio data from a corresponding logical bus to the sink-in audio buffer; and request the stream of audio data from the corresponding logical bus for input to the sink-in audio buffer.

28. An audio generation system as recited in claim 17, further comprising a software component that defines the one or more logical buses corresponding respectively to one or more of the sink-in audio buffers.

29. An audio generation system as recited in claim 17, further comprising a software component that defines the logical buses corresponding to the sink-in audio buffers, each logical bus configured to receive the one or more streams of audio data from the synthesizer component and communicate the one or more streams of audio data to the corresponding sink-in audio buffer.

30. An audio generation system as recited in claim 17, further comprising a performance manager that includes at least one audio segment having one or more audio content components that are each configured to generate audio instructions; and wherein the audio rendition manager includes the synthesizer component configured to process the audio instructions to generate the one or more streams of audio data.

31. An audio generation system as recited in claim 17, further comprising a performance manager that includes at least one audio segment having one or more audio content components that are each configured to generate MIDI instructions; and wherein the audio rendition manager includes the synthesizer component configured to process the MIDI instructions to generate the one or more streams of audio data.

32. A method, comprising: requesting multiple streams of audio data from audio rendition managers that each correspond to an audio rendition, an audio rendition manager including a synthesizer component, audio buffers, and logical buses that each correspond to one of the audio buffers; defining sink-in audio buffers that each receive one or more of the multiple streams of audio data from the synthesizer component, where each stream of audio data is assigned to one or more of the logical buses that receive the audio data from the synthesizer component such that the audio data streams assigned to a particular logical bus are routed to a sink-in audio buffer that corresponds to the particular logical bus; defining a mix-in audio buffer to receive one or more streams of audio data from one or more of the sink-in audio buffers; and generating a stream of combined audio data that includes the one or more streams of audio data received from the one or more sink-in audio buffers.

33. A method as recited in claim 32, further comprising communicating the stream of combined audio data to an audio component that produces an audio rendition corresponding to the stream of combined audio data.

34. A method as recited in claim 32, further comprising generating the multiple streams of audio data when receiving audio instructions.

35. A method as recited in claim 32, further comprising generating the multiple streams of audio data when receiving synthesizer MIDI instructions.

36. A method as recited in claim 32, wherein defining the sink-in audio buffers comprises instantiating each sink-in audio buffer as a programming object having an interface that is callable to route the one or more multiple steams of audio data.

37. A method as recited in claim 32, wherein defining the mix-in audio buffer comprises instantiating the mix-in audio buffer as a programming object having an interface that is callable to route the one or more streams of audio data received from the one or more sink-in audio buffers.

38. A method as recited in claim 32, wherein defining the sink-in audio buffers comprises reserving computing system resources to implement each sink-in audio buffer to receive the one or more multiple streams of audio data.

39. A method as recited in claim 32, wherein defining the mix-in audio buffer comprises reserving computing system resources to implement the mix-in audio buffer to receive the one or more steams of audio data received from the one or more sink-in audio buffers.

40. A method as recited in claim 32, wherein the mix-in audio buffer is defined before the one or more sink-in audio buffers are defined.

41. A method as recited in claim 32, wherein defining the mix-in audio buffer comprises reserving computing system resources to impitment the mix-in audio buffer before defining the sink-in audio buffers which comprises reserving computing system resources to implement each sink in audio buffer.

42. A method as recited in claim 32, wherein defining the mix-in audio buffer comprises reserving computing system resources to implement the mix-in audio buffer before defining the sink-in audio buffers which comprises reserving computing system resources to implement each sink-in audio buffer, and the method further comprising: releasing the computing system resources that implement the sink-in audio buffers before releasing the computing resources that implement the mix-in audio buffer.

43. A method as recited in claim 32, further comprising defining a second mix-in audio buffer to receive the stream of combined audio data from the mix-in audio buffer, and to receive one or more streams of audio data from one or more of the sink-in audio buffers.

44. A method as recited in claim 32, further comprising: defining a second mix-in audio buffer to receive the stream of combined audio data from the mix-in audio buffer, and to receive one or more streams of audio data from one or more of the sink-in audio buffers; generating a second stream of combined audio data that includes the stream of combined audio data, and includes the one or more streams of audio data; and communicating the second stream of combined audio data to an audio component that produces an audio rendition corresponding to the second stream of combined audio data.

45. A method as recited in claim 32, further comprising: defining a performance manager to generate audio instructions; and defining an audio rendition manager to process the audio instructions and generate the multiple streams of audio data.

46. A method as recited in claim 32, further comprising: defining a performance manager having an audio segment with one or more audio content components, the one or more audio content components generating audio instructions; and defining the audio rendition manager having the synthesizer component to process the audio instructions to generate the multiple streams of audio data.

47. One or more computer-readable storage media encoded with computer-executable instructions that, when executed, direct a computing system to perform the method of claim 32.

48. One or more computer-readable storage media encoded with computer executable instructions that, when executed, direct a computing system to: generate audio data with a synthesizer component when receiving audio instructions, an audio rendition manager including the synthesizer component, audio buffers, and logical buses that each correspond to one of the audio buffers; instantiate a sink-in audio buffer as a programming object having an interface that is callable by a software component, the sink-in audio buffer receiving audio data from the synthesizer component where streams of the audio data are assigned to one or more of the logical buses that receive the audio data from the synthesizer component such that the audio data streams assigned to a particular logical bus are routed to the sink-in audio buffer that corresponds to the particular logical bus; stream the audio data to the sink-in audio buffer when the audio data is requested for input by the software component; and communicate the audio data to an audio component that produces an audio rendition corresponding to the audio data.

49. One or more computer-readable storage media as recited in claim 48, further encoded with computer executable instructions that, when executed, direct the computing system to instantiate a mix-in audio buffer as a programming object having an interface that is callable by the software component, the mix-in audio buffer generating a stream of combined audio data that includes audio data received from one or more sink-in audio buffers.

50. One or more computer-readable storage media as recited in claim 48, further encoded with computer executable instructions that, when executed, direct the computing system to: instantiate a mix-in audio buffer as a programming object having an interface that is callable by the software component the mix-in audio buffer generating a stream of combined audio data that includes audio data received from one or more sink-in audio buffers; and communicate the stream of combined audio data to the audio component that produces an audio rendition corresponding to the stream of combined audio data.

51. One or more computer-readable storage media as recited in claim 48, further encoded with computer executable instructions that, when executed, direct the computing system to: instantiate a performance manager having an audio segment with one or more audio content components, the one or more audio content components generating audio instructions; and instantiate the audio rendition manager that includes the synthesizer component to process the audio instructions to generate the audio data.

Description

TECHNICAL FIELD

This invention relates to audio processing with an audio generation system and, in particular, to dynamic audio buffer creation.

BACKGROUND

Multimedia programs present content to a user through both audio and video events while a user interacts with a program via a keyboard, joystick, or other interactive input device. A user associates elements and occurrences of a video presentation with the associated audio representation. A common implementation is to associate audio with movement of characters or objects in a video game. When a new character or object appears, the audio associated with that entity is incorporated into the overall presentation for a more dynamic representation of the video presentation.

Audio representation is an essential component of electronic and multimedia products such as computer based and stand-alone video games, computer-based slide show presentations, computer animation, and other similar products and applications. As a result, audio generating devices and components are integrated into electronic and multimedia products for composing and providing graphically associated audio representations. These audio representations can be dynamically generated and varied in response to various input parameters, real-time events, and conditions. Thus, a user can experience the sensation of live audio or musical accompaniment with a multimedia experience.

Conventionally, computer audio is produced in one of two fundamentally different ways. One way is to reproduce an audio waveform from a digital sample of an audio source which is typically stored in a wave file (i.e., a .wav file). A digital sample can reproduce any sound, and the output is very similar on all sound cards, or similar computer audio rendering devices. However, a file of digital samples consumes a substantial amount of memory and resources when streaming the audio content. As a result, the variety of audio samples that can be provided using this approach is limited. Another disadvantage of this approach is that the stored digital samples cannot be easily varied.

Another way to produce computer audio is to synthesize musical instrument sounds, typically in response to instructions in a Musical Instrument Digital Interface (MIDI) file, to generate audio sound waves. MIDI is a protocol for recording and playing back music and audio on digital synthesizers incorporated with computer sound cards. Rather than representing musical sound directly, MIDI transmits information and instructions about how music is produced. The MIDI command set includes note-on, note-off, key velocity, pitch bend, and other commands to control a synthesizer.

The audio sound waves produced with a synthesizer are those already stored in a wavetable in the receiving instrument or sound card. A wavetable is a table of stored sound waves that are digitized samples of actual recorded sound. A wavetable can be stored in read-only memory (ROM) on a sound card chip, or provided with software. Prestoring sound waveforms in a lookup table improves rendered audio quality and throughput. An advantage of MIDI files is that they are compact and require few audio streaming resources, but the output is limited to the number of instruments available in the designated General MIDI set and in the synthesizer, and may sound very different on different computer systems.

MIDI instructions sent from one device to another indicate actions to be taken by the controlled device, such as identifying a musical instrument (e.g., piano, flute, drums, etc.) for music generation, turning on a note, and/or altering a parameter in order to generate or control a sound. In this way, MIDI instructions control the generation of sound by remote instruments without the MIDI control instructions themselves carrying sound or digitized information. A MIDI sequencer stores, edits, and coordinates the MIDI information and instructions. A synthesizer connected to a sequencer generates audio based on the MIDI information and instructions received from the sequencer. Many sounds and sound effects are a combination of multiple simple sounds generated in response to the MIDI instructions.

A MIDI system allows audio and music to be represented with only a few digital samples rather than converting an analog signal to many digital samples. The MIDI standard supports different channels that can each simultaneously provide an output of audio sound wave data. There are sixteen defined MIDI channels, meaning that no more than sixteen instruments can be playing at one time. Typically, the command input for each MIDI channel represents the notes corresponding to an instrument. However, MIDI instructions can program a channel to be a particular instrument. Once programmed, the note instructions for a channel will be played or recorded as the instrument for which the channel has been programmed. During a particular piece of music, a channel can be dynamically reprogrammed to be a different instrument.

A Downloadable Sounds (DLS) standard published by the MIDI Manufacturers Association allows wavetable synthesis to be based on digital samples of audio content provided at run-time rather than stored in memory. The data describing an instrument can be downloaded to a synthesizer and then played like any other MIDI instrument. Because DLS data can be distributed as part of an application, developers can be assured that the audio content will be delivered uniformly on all computer systems. Moreover, developers are not limited in their choice of instruments.

A DLS instrument is created from one or more digital samples, typically representing single pitches, which are then modified by a synthesizer to create other pitches. Multiple samples are used to make an instrument sound realistic over a wide range of pitches. DLS instruments respond to MIDI instructions and commands just like other MIDI instruments. However, a DLS instrument does not have to belong to the General MIDI set or represent a musical instrument at all. Any sound, such as a fragment of speech or a fully composed measure of music, can be associated with a DLS instrument.

Conventional Audio and Music System

FIG. 1 illustrates a conventional audio and music generation system 100 that includes a synthesizer 102, a sound effects input source 104, and a buffers component 106. Typically, a synthesizer is implemented in computer software, in hardware as part of a computer's internal sound card, or as an external device such as a MIDI keyboard or module. Synthesizer 102 receives MIDI inputs on sixteen channels 108 that conform to the MIDI standard. Synthesizer 102 includes a mixing component 110 that mixes the audio sound wave data output from synthesizer channels 108. An output 112 of mixing component 110 is input to an audio buffer in the buffers component 106.

MIDI inputs to synthesizer 102 are in the form of individual instructions, each of which designates the MIDI channel to which it applies. Within synthesizer 102, instructions associated with different channels 108 are processed in different ways, depending on the programming for the various channels. A MIDI input is typically a serial data stream that is parsed in synthesizer 102 into MIDI instructions and synthesizer control information. A MIDI command or instruction is represented as a data structure containing information about the sound effect or music piece such as the pitch, relative volume, duration, and the like.

A MIDI instruction, such as a "note-on", directs synthesizer 102 to play a particular note, or notes, on a synthesizer channel 108 having a designated instrument. The General MIDI standard defines standard sounds that can be combined and mapped into the sixteen separate instrument and sound channels. A MIDI event on a synthesizer channel 108 corresponds to a particular sound and can represent a keyboard key stroke, for example. The "note-on" MIDI instruction can be generated with a keyboard when a key is pressed and the "note-on" instruction is sent to synthesizer 102. When the key on the keyboard is released, a corresponding "note-off" instruction is sent to stop the generation of the sound corresponding to the keyboard key.

The audio representation for a video game involving a car, from the perspective of a person in the car, can be presented for an interactive video and audio presentation. The sound effects input source 104 has audio data that represents various sounds that a driver in a car might hear. A MIDI formatted music piece 114 represents the audio of the car's stereo. Input source 104 also has digital audio sample inputs that are sound effects representing the car's horn 116, the car's tires 118, and the car's engine 120.

The MIDI formatted input 114 has sound effect instructions 122(1-3) to generate musical instrument sounds. Instruction 122(1) designates that a guitar sound be generated on MIDI channel one (1) in synthesizer 102, instruction 120(2) designates that a bass sound be generated on MIDI channel two (2), and instruction 120(3) designates that drums be generated on MIDI channel ten (10). The MIDI channel assignments are designated when MIDI input 114 is authored, or created.

A conventional software synthesizer that translates MIDI instructions into audio signals does not support distinctly separate sets of MIDI channels. The number of sounds that can be played simultaneously is limited by the number of channels and resources available in the synthesizer. In the event that there are more MIDI inputs than there are available channels and resources, one or more inputs are suppressed by the synthesizer.

The buffers component 106 of audio system 100 includes multiple buffers 124(1-4). Typically, a buffer is an allocated area of memory that temporarily holds sequential samples of audio sound wave data that will be subsequently communicated to a sound card or similar audio rendering device to produce audible sound. The output 112 of synthesizer mixing component 110 is input to buffer 124(1) in buffers component 106. Similarly, each of the other digital sample sources are input to a buffer 124 in buffers component 106. The car horn sound effect 116 is input to buffer 124(2), the tires sound effect 118 is input to buffer 124(3), and the engine sound effect 120 is input to buffer 124(4).

Another problem with conventional audio generation systems is the extent to which system resources have to be allocated to support an audio representation for a video presentation. In the above example, each buffer 124 requires separate hardware channels, such as in a soundcard, to render the audio sound effects from input source 104. Further, in an audio system that supports both music and sound effects, a single stereo output pair that is input to one buffer is a limitation to creating and enhancing the music and sound effects.

Similarly, other three-dimensional (3-D) audio spatialization effects are difficult to create and require an allocation of system resources that may not be available when processing a video game that requires an extensive audio presentation. For example, to represent more than one car from a perspective of standing near a road in a video game, a pre-authored car engine sound effect 120 has to be stored in memory once for each car that will be represented. Additionally, a separate buffer 124 and separate hardware channels will need to be allocated for each representation of a car. If a computer that is processing the video game does not have the resources available to generate the audio representation that accompanies the video presentation, the quality of the presentation will be deficient.

SUMMARY

An audio generation system includes a buffer manager that creates audio buffers to receive streams of audio data from an audio data source, such as from a synthesizer component, for example. The audio buffers include sink-in audio buffers that receive one or more streams of audio data when the streams of audio data are requested by the buffer manager for input to the sink-in audio buffers. The audio buffers also include mix-in audio buffers that receive streams of audio data from one or more of the sink-in audio buffers, and from any number of other mix-in audio buffers. The audio generation system includes an audio component, such as a speaker for example, that receives audio data from the sink-in audio buffers and produces an audio rendition corresponding to the audio data.

In one embodiment, the audio buffers are instantiated as programming objects, each having an interface that is callable by a software component, such as a multimedia application for example. Further, computing system resources are first reserved to implement a mix-in audio buffer before computing system resources are reserved to implement any sink-in audio buffers that input audio data to the mix-in audio buffer. Subsequently, the computing system resources reserved to implement the sink-in audio buffers are released before the computing system resources that implement the mix-in audio buffer are released.

BRIEF DESCRIPTION OF THE DRAWINGS

The same numbers are used throughout the drawings to reference like features and components.

FIG. 1 illustrates a conventional audio generation system.

FIG. 2 illustrates various components of an exemplary audio generation system.

FIG. 3 illustrates various components of the audio generation system shown in FIG. 2.

FIG. 4 illustrates various components of the audio generation system shown in FIG. 3.

FIG. 5 illustrates an exemplary audio buffer system.

FIG. 6 is a flow diagram of a method for dynamic audio buffer creation.

FIG. 7 is a diagram of computing systems, devices, and components in an environment that can be used to implement the systems and methods described herein.

DETAILED DESCRIPTION

The following describes systems and methods to implement dynamic audio buffers in an audio generation system that supports numerous computing systems' audio technologies, including technologies that are designed and implemented after a multimedia application program has been authored. An application program instantiates the components of an audio generation system to produce, or otherwise generate, audio data that can be rendered with an audio rendering device to produce audible sound.

Dynamic audio buffer creation describes implementing audio buffers as needed in an audio generation system. Computing system resource allocation to create the audio buffers in hardware and/or software is dynamic as necessitated by a requesting application program, such as a video game or other multimedia application. An application program can optimally utilize system hardware and software resources by creating and allocating audio buffers only when needed.

An audio generation system includes an audio rendition manager (also referred to herein as an "AudioPath") that is implemented to provide various audio data processing components that process audio data into audible sound. The audio generation system described herein simplifies the process of creating audio representations for interactive applications such as video games and Web sites. The audio rendition manager manages the audio creation process and integrates both digital audio samples and streaming audio.

Additionally, an audio rendition manager provides real-time, interactive control over the audio data processing for audio representations of video presentations. An audio rendition manager also enables 3-D audio spatialization processing for an individual audio representation of an entity's video presentation. Multiple audio renditions representing multiple video entities can be accomplished with multiple audio rendition managers, each representing a video entity, or audio renditions for multiple entities can be combined in a single audio rendition manager.

Real-time control of audio data processing components in an audio generation system is useful, for example, to control an audio representation of a video game presentation when parameters that are influenced by interactivity with the video game change, such as a video entity's 3-D positioning in response to a change in a video game scene. Other examples include adjusting audio environment reverb in response to a change in a video game scene, or adjusting music transpose in response to a change in the emotional intensity of a video game scene.

Exemplary Audio Generation System

FIG. 2 illustrates an audio generation system 200 having components that can be implemented within a computing device, or the components can be distributed within a computing system having more than one computing device. The audio generation system 200 generates audio events that are processed and rendered by separate audio processing components of a computing device or system. See the description of "Exemplary Computing System and Environment" below for specific examples and implementations of network and computing systems, computing devices, and components that can be used to implement the technology described herein.

Audio generation system 200 includes an application program 202, a performance manager component 204, and an audio rendition manager 206. Application program 202 is one of a variety of different types of applications, such as a video game program, some other type of entertainment program, or any other application that incorporates an audio representation with a video presentation.

The performance manager 204 and the audio rendition manager 206 can be instantiated, or provided, as programming objects. The application program 202 interfaces with the performance manager 204, the audio rendition manager 206, and the other components of the audio generation system 200 via application programming interfaces (APIs). For example, application program 202 can interface with the performance manager 204 via API 208 and with the audio rendition manager 206 via API 210.

The various components described herein, such as the performance manager 204 and the audio rendition manager 206, can be implemented using standard programming techniques, including the use of OLE (object linking and embedding) and COM (component object model) interfaces. COM objects are implemented in a system memory of a computing device, each object having one or more interfaces, and each interface having one or more methods. The interfaces and interface methods can be called by application programs and by other objects. The interface methods of the objects are executed by a processing unit of the computing device. Familiarity with object-based programming, and with COM objects in particular, is assumed throughout this disclosure. However, those skilled in the art will recognize that the audio generation systems and the various components described herein are not limited to a COM and/or OLE implementation, or to any other specific programming technique.

The audio generation system 200 includes audio sources 212 that provide digital samples of audio data such as from a wave file (i.e., a .wav file), message-based data such as from a MIDI file or a pre-authored segment file, or an audio sample such as a Downloadable Sound (DLS). Audio sources can be also be stored as a resource component file of an application rather than in a separate file.

Application program 202 can initiate that an audio source 212 provide audio content input to performance manager 204. The performance manager 204 receives the audio content from audio sources 212 and produces audio instructions for input to the audio rendition manager 206. The audio rendition manager 206 receives the audio instructions and generates audio sound wave data. The audio generation system 200 includes audio rendering components 214 which are hardware and/or software components, such as a speaker or soundcard, that renders audio from the audio sound wave data received from the audio rendition manager 206.

FIG. 3 illustrates a performance manager 204 and an audio rendition manager 206 as part of an audio generation system 300. An audio source 302 provides sound effects for an audio representation of various sounds that a driver of a car might hear in a video game, for example. The various sound effects can be presented to enhance the perspective of a person sitting in the car for an interactive video and audio presentation.

The audio source 302 has a MIDI formatted music piece 304 that represents the audio of a car stereo. The MIDI input 304 has sound effect instructions 306(1-3) to generate musical instrument sounds. Instruction 306(1) designates that a guitar sound be generated on MIDI channel one (1) in a synthesizer component, instruction 306(2) designates that a bass sound be generated on MIDI channel two (2), and instruction 306(3) designates that drums be generated on MIDI channel ten (10). Input audio source 302 also has digital audio sample inputs that represent a car horn sound effect 308, a tires sound effect 310, and an engine sound effect 312.

The performance manager 204 can receive audio content from a wave file (i.e., .wav file), a MIDI file, or a segment file authored with an audio production application, such as DirectMusic.RTM. Producer, for example. DirectMusic.RTM. Producer is an authoring tool for creating interactive audio content and is available from Microsoft Corporation of Redmond, Washington. Additionally, performance manager 204 can receive audio content that is composed at run-time from different audio content components.

Performance manager 204 receives audio content input from input audio source 302 and produces audio instructions for input to the audio rendition manager 206. Performance manager 204 includes a segment component 314, an instruction processors component 316, and an output processor 318. The segment component 314 represents the audio content input from audio source 302. Although performance manager 204 is shown having only one segment 314, the performance manager can have a primary segment and any number of secondary segments. Multiple segments can be arranged concurrently and/or sequentially with performance manager 204.

Segment component 314 can be instantiated as a programming object having one or more interfaces 320 and associated interface methods. In the described embodiment, segment object 314 is an instantiation of a COM object class and represents an audio or musical piece. An audio segment represents a linear interval of audio data or a music piece and is derived from the inputs of an audio source which can be digital audio data, such as the engine sound effect 312 in audio source 302, or event-based data, such as the MIDI formatted input 304.

Segment component 314 has track components 322(1) through 322(N), and an instruction processors component 324. Segment 314 can have any number of track components 322 and can combine different types of audio data in the segment with different track components. Each type of audio data corresponding to a particular segment is contained in a track component 322 in the segment, and an audio segment is generated from a combination of the tracks in the segment. Thus, segment 314 has a track 322 for each of the audio inputs from audio source 302.

Each segment object contains references to one or a plurality of track objects. Track components 322(1) through 322(N) can be instantiated as programming objects having one or more interfaces 326 and associated interface methods. The track objects 322 are played together to render the audio and/or musical piece represented by segment object 314 which is part of a larger overall performance. When first instantiated, a track object does not contain actual music or audio performance data, such as a MIDI instruction sequence. However, each track object has a stream input/output (I/O) interface method through which audio data is specified.

The track objects 322(1) through 322(N) generate event instructions for audio and music generation components when performance manager 204 plays the segment 314. Audio data is routed through the components in the performance manager 204 in the form of event instructions which contain information about the timing and routing of the audio data. The event instructions are routed between and through the components in performance manager 204 on designated performance channels. The performance channels are allocated as needed to accommodate any number of audio input sources and to route event instructions.

To play a particular audio or musical piece, performance manager 204 calls segment object 314 and specifies a time interval or duration within the musical segment. The segment object in turn calls the track play methods of each of its track objects 322, specifying the same time interval. The track objects 322 respond by independently rendering event instructions at the specified interval. This is repeated, designating subsequent intervals, until the segment has finished its playback over the specified duration.

The event