In recent years, digitization has been rapidly carried out in the technical field of broadcasting. For instance, a digital broadcast system has been put into practice. In the digital broadcast system, contents data such as video data and audio data, service information, and program information are multiplexed in a digital format for broadcast. Generally, contents data such as video data and audio data is subjected to compression encoding, based on the specifications such as an MPEG (Moving Picture Experts Group) system; fragments of encoded data i.e. compression-encoded contents data are packetized; and the packets are transmitted as a stream of signals.
A receiver, upon receiving the signal stream, is operable to restore the contents data by extracting the encoded data from the signal stream, and decoding the extracted encoded data. In the digital broadcast system, the digital broadcast is performed by using a technique of transmitting a stream of signals i.e. transmitting a signal stream called a transport stream.
In particular, for instance, in the MPEG system, in generating a transport stream obtained by multiplexing multiple encoded data such as video encoded data and audio encoded data, a system target decoder (hereinafter, called as “STD” according to needs) is used as a standard decoder model for performing a decoding operation. Specifically, in the MPEG system, a transport stream is generated based on the STD so that a multiplexing device does not cause breakdown of the STD, thereby enabling the decoder to properly perform a decoding operation. In view of this, for instance, in a multiplexing device for use in the digital broadcast system, a transport stream is generated by virtually and sequentially monitoring storage statuses of various buffers provided in the STD, and controlling the STD in such a manner as to avoid buffer breakdown such as a buffer overflow or a buffer underflow.
In the technical field of consumer video devices, for instance, a technique of increasing the capacity of a recording medium such as a hard disk or an optical disc for recording information in a digital format has rapidly progressed. With use of a recording and reproducing device called a DVD recorder, it is possible to record large-capacity data such as video data in general houses. Miniaturization of a recording medium has also rapidly progressed. For instance, it is possible to record or reproduce contents data such as video data and audio data by mounting a memory card such as an SD (Secure Digital) memory card or an electronic card in a mobile phone device, a digital camera device, or a like device.
In the case where contents data is recorded in a recording medium, basically, a method for recording contents data in conformity with individual formats of recording media in the aforementioned stream format has been generally put into practice. In view of this, a video camera device or a like device utilizing the above method is provided with an encoder for subjecting contents data to compression encoding; packetizing means for packetizing encoded video/audio data, and data for controlling the device; and multiplexing means for multiplexing the packets in order to generate a signal stream. If the encoder or the multiplexing device for use in the digital broadcast system is used in a consumer device such as a video camera device, the cost of the consumer device may be sharply increased. In view of this, there has been proposed a technique relating to a signal stream generating method to be applied to the consumer device.
As an example of the technique, for instance, there is known a multiplexing device for multiplexing multiple encoded data, without the need of virtual control of the STD. Specifically, in the technique of performing a multiplexing operation while virtually and sequentially monitoring storage statuses of the buffers in the STD, a complicated operation is required. Therefore, it is improper to apply the above technique to the consumer device. In view of this, there is proposed a multiplexing device for generating a multiplexed stream by alternately multiplexing packets according to a predetermined order so that the ratio between the number of packets containing encoded video data and the number of packets containing encoded audio data coincides with the ratio of bit rate between the encoded video data and the encoded audio data (see e.g. patent document 1).
FIG. 23 is a block diagram showing an arrangement of a conventional multiplexing device. The conventional multiplexing device 91 shown in FIG. 23 is a device for recording a multiplexed stream in a portable recording medium 92 by subjecting video data and audio data to compression encoding in accordance with the MPEG-1 specifications, respectively, and multiplexing a video stream composed of the encoded video data and an audio stream composed of the encoded audio data, respectively.
In the multiplexing device 91 shown in FIG. 23, a video encoder 911 subjects inputted video data to compression encoding at a fixed bit rate in accordance with a control signal from a CPU 916. Then, the video encoder 911 sequentially transmits a video stream composed of the video encoded data to a video input buffer 912. The video input buffer 912 temporarily stores the video stream, and transmits the temporarily stored video stream to an output buffer 915 in response to a request from the CPU 916.
Similarly to the above, an audio encoder 913 subjects inputted audio data to compression encoding at a fixed bit rate in accordance with a control signal from the CPU 916. Then, the audio encoder 913 sequentially transmits an audio stream composed of the audio encoded data to an audio input buffer 914. The audio input buffer 914 temporarily stores the audio stream, and transmits the temporarily stored audio stream to the output buffer 915 in response to a request from the CPU 916.
The output buffer 915 stores data outputted from the video input buffer 912 and the audio input buffer 914, header information generated in the CPU 916, and the like in a designated address under the control of the CPU 916. In storing the data, the CPU 916 is operable to extract packet data of an identical data length according to a predetermined order from the video stream and the audio stream of a fixed bit rate, which have been temporarily stored in the video input buffer 912 and the audio input buffer 914, respectively; generate header data of a fixed data length with respect to each of the packet data; and control the output buffer 915 to alternately store the header data therein.
As described above, the output buffer 915 is operable to store a multiplexed stream composed of header data, packet data containing partial data of a video stream, and packet data containing partial data of an audio stream alternately according to a predetermined order.
The multiplexing device 91 outputs the multiplexed stream to the portable recording medium 92. In this way, the multiplexing device 91 performs a multiplexing operation to record a multiplexed stream generated by a multiplexing operation into the portable recording medium 92.
As described above, in performing a multiplexing operation, it is necessary to generate a multiplexed stream in such a manner that a decoder buffer does not cause overflow and underflow. In view of this, the multiplexing device 91 performs a multiplexing operation in such a manner that the ratio between the number of packet data of a video stream and the number of packet data of an audio stream coincides with the ratio of bit rate between the video stream and the audio stream. This enables to make the transmission bit rates of a pack and header data constant, and make the data amounts of a video stream and an audio stream to be transmitted per unit time constant, thereby preventing overflow of an STD buffer at the time of decoding a multiplexed stream.
FIG. 24 is a conceptual diagram showing a method for allocating video packets and audio packets in a conventional multiplexing device. A method for allocating video packets and audio packets is described with use of a multiplexed stream 241 shown in FIG. 24, wherein the bit rate (hereinafter, called as a “video rate”) of a video stream is 128 kbps, and the bit rate (hereinafter, called as an “audio rate”) of an audio stream is 32 kbps, in other words, the ratio between the video rate and the audio rate is 4:1. Referring to FIG. 24, a block with the symbol “V” indicates a video packet i.e. a packet containing a segment of a video stream; and a block with the symbol “A” indicates an audio packet i.e. a packet containing a segment of an audio stream.
Specifically, the multiplexing device 91 is operable to generate a multiplexed stream in such a manner that each of the encoded video stream and the encoded audio stream is divided into a number of packet data of a predetermined amount; and packet groups each composed of four video packets and one audio packet are alternately multiplexed, as shown by the multiplexed stream 241, because the ratio between the number of packet data of the video stream and the number of packet data of the audio stream is set to 4:1, which is a ratio of fixed bit rate between the video stream and the audio stream.
A transfer bit rate Rmux of the entirety of the multiplexed stream is equal to the sum of bit rates of the encoded video stream and the encoded audio stream i.e. Rmux=128 kbps+32 kbps=160 kbps. Specifically, video data and audio data are always contained in a multiplexed stream of 160 kbps at a predetermined ratio so that the ratio of the video rate and the audio rate is set to 4:1. In this arrangement, there is no likelihood that data over a predetermined amount may be inputted to an STD buffer in a decoder per unit time, thereby preventing a buffer overflow.
The multiplexing device 91 is also configured to perform a multiplexing operation in such a manner that after a predetermined amount of inputted data is accumulated in the STD buffer, the decoder sequentially reads out the data to start a decoding operation. Specifically, the data amounts of a video stream and an audio stream per access unit are not constant in an actual operation. Accordingly, the amount of data to be inputted to the STD buffer may not be constant, thereby causing a buffer underflow. The access unit corresponds to a picture or a frame in encoding video data, and corresponds to an audio frame in encoding audio data.
In order to prevent a buffer underflow, a certain time for accumulating data before a decoding operation is started is defined in the multiplexing device 91. The time for accumulating data into a decoder buffer in advance is called a startup delay time. A multiplexed stream 242 shown in FIG. 24 has a startup delay time.
By defining the startup delay time, an operation of decoding a multiplexed stream is started after data is accumulated in the STD buffer by a period corresponding to the startup delay time. Further, a decoding start timing is delayed from a point of time when leading data of a multiplexed stream is transferred to the STD buffer, by the startup delay time.
As described above, the conventional multiplexing device 91 prevents an overflow of an STD buffer at a decoding time by performing a multiplexing operation so that the ratio between the number of video packets and the number of audio packets having an identical data length coincides with the ratio of fixed bit rate between the video packets and the audio packets in generating a multiplexed stream.
Further, in the conventional multiplexing device 91, underflow of the STD buffer at a decoding time is prevented by defining a startup delay time for accumulating video packets and audio packets in the STD buffer before a decoding start timing; and including the startup delay time in time stamp data indicating the decoding start timing or a like timing in performing a multiplexing operation.
The conventional multiplexing device 91 having the above configuration enables to multiplex multiple encoded data, without the need of virtual control of an STD.
As described above, the conventional multiplexing device is capable of avoiding breakdown of a decoder buffer in multiplexing multiple encoded data generated at a fixed bit rate. However, the conventional multiplexing device has a drawback that the buffer may cause breakdown in processing encoded data generated at a variable bit rate.
Specifically, for instance, in the MPEG system, the amount of encoded data generated by subjecting video data to compression encoding greatly differs depending on kinds of format e.g. I-picture format, P-picture format, and B-picture format, and also greatly differs depending on the contents of images. Accordingly, in the case where a sequence of flat or motionless images is processed, the amount of video encoded data per access unit is extremely lessened, with the result that a buffer underflow and a buffer overflow may be unavoidable solely by defining the startup delay time.
In the conventional multiplexing device, a multiplexing operation is performed while simulating the storage status of a decoder buffer to avoid breakdown of the decoder buffer. However, the decoder has multiple buffers, and the operations of the buffers are complicated. Accordingly, the processing amount and the processing time concerning the entirety of a multiplexing operation may be increased.
Patent document 1: JP2003-78873A
USPTO Backgrounds
USPTO Backgrounds