Title: Picture and sound decoding apparatus picture and sound encoding apparatus and information transmission system
Abstract: In order to control the coding quantity depending on the processing situation at the terminal when decoding or synthesizing plural pictures or sounds simultaneously, the decoding apparatus of the invention comprises a reception control unit 11 for receiving the information, a separation unit 12 for analyzing and separating the received information, a priority decision unit 14 for determining the priority of processing of the pictures separated in the separation unit 12, a picture expanding unit 18 for expanding the pictures according to the determined priority, s picture synthesizing unit 19 for synthesizing the pictures on the basis of the expanded pictures, a synthesis result accumulating unit 22 for accumulating the synthesized pictures, a reproduction time control unit 23 for controlling the time for starting reproduction, and an output unit 24 for delivering the result of synthesis according to the information of the reproduction time control unit 23.
Patent Number: 7,006,575 Issued on 02/28/2006 to Yamaguchi, et al.
| Inventors:
|
Yamaguchi; Takao (Sakai, JP);
Kamogawa; Akira (Katano, JP);
Nobori; Kunio (Kadoma, JP)
|
| Assignee:
|
Matsushita Electric Industrial Co., Ltd. (Osaka, JP)
|
| Appl. No.:
|
051547 |
| Filed:
|
August 1, 1997 |
| PCT Filed:
|
August 1, 1997
|
| PCT NO:
|
PCT/JP97/02696
|
| 371 Date:
|
April 7, 1998
|
| 102(e) Date:
|
April 7, 1998
|
| PCT PUB.NO.:
|
WO98/06222 |
| PCT PUB. Date:
|
February 12, 1998 |
Foreign Application Priority Data
| Aug 07, 1996[JP] | 8-208147 |
| Aug 08, 1996[JP] | 8-209942 |
| Nov 13, 1996[JP] | 8-301559 |
| Current U.S. Class: |
375/240.26; 375/240; 375/240.01; 375/240.12; 375/240.24; 375/240.25 |
| Current Intern'l Class: |
H04N 7/12 (20060101) |
| Field of Search: |
375/240,290.25,240.12,240.01,240.24,240.26
386/81
382/304
709/236
370/252
|
References Cited [Referenced By]
U.S. Patent Documents
| 4562466 | Dec., 1985 | Clapp et al.
| |
| 5377051 | Dec., 1994 | Lane et al.
| |
| 5394191 | Feb., 1995 | Sirat et al.
| |
| 5418568 | May., 1995 | Keith.
| |
| 5461679 | Oct., 1995 | Normile et al.
| |
| 5485211 | Jan., 1996 | Kuzma.
| |
| 5510844 | Apr., 1996 | Cash et al.
| |
| 5533140 | Jul., 1996 | Sirat et al.
| |
| 5706346 | Jan., 1998 | Katta et al.
| |
| 5822540 | Oct., 1998 | Caldara et al.
| |
| 6266327 | Jul., 2001 | Hernandez-Valencia.
| |
| Foreign Patent Documents |
| 0713338 | May., 1996 | EP.
| |
| 62-1384 | Jan., 1987 | JP.
| |
| 62-231379 | Oct., 1987 | JP.
| |
| 2-195787 | Aug., 1990 | JP.
| |
| 4-3684 | Jan., 1992 | JP.
| |
| 4-145786 | May., 1992 | JP.
| |
| 5-122681 | May., 1993 | JP.
| |
| 6-105226 | Apr., 1994 | JP.
| |
| 6-153198 | May., 1994 | JP.
| |
| 7-15715 | Jan., 1995 | JP.
| |
| 7-111647 | Apr., 1995 | JP.
| |
| 7-261776 | Oct., 1995 | JP.
| |
| 7-298258 | Nov., 1995 | JP.
| |
| 7-336666 | Dec., 1995 | JP.
| |
| 8-149420 | Jun., 1996 | JP.
| |
| 8-154219 | Jun., 1996 | JP.
| |
| 08-172616 | Jul., 1996 | JP.
| |
| 8-289290 | Nov., 1996 | JP.
| |
| 9-23422 | Jan., 1997 | JP.
| |
Other References
Fukuda et al., "Hypermedia Personal Computer Communication System: Fujitsu Habitat,"
Fujitsu Sci. Tech. J., 26, 3, pp. 197-206 (Oct. 1990).
Nakamura, "Collabolation on Networked Virtual Reality," Information Technology
Research Laboratories, NEC, pp. 17-23 (Sep. 17, 1993).
H. Fujiwara, "Latest MPEG Textbook", Aug. 1, 1994, pp. 104-105.
Japanese language search report for Int'l Appln. No. PCT/JP97/02696 dated Dec.
2, 1997.
English translation of Japanese language search report.
"Intel Indeo ® Video 5", Compression Techniques for Great-Looking Indeo
Video, pp. 1-15 (Dec. 1997).
"What makes ASF good?", Advanced Streaming Format, pp. 1-4 (Dec. 1997).
"9.3 Profile of G.721", pp. 1-5.
Notice of Reasons Of Rejection, Patent Application No. 2002-230974, dated Jun.
29, 2004.
T. Nakajima, " Continuous Media Objects With Media Scaling," Lecture Note, Software
Study 11, WOOC '95, Kindal Kagaku, Inc., pp. 97-106, Dec. 10, 1995, including an
English Abstract.
Notice of Reasons Of Rejection, Patent Application No. 10-507809, Feb. 10, 204
(with English Translation).
Takao Nakamura, et al.; "A Note on a Scrambling Scheme with Variable Concealed
Level for MPEG bitstream"; Technical Research Paper of the Institute of Electronics,
Information and Communication Engineers, p. 19-24, vol. 95, No. 474, Jan. 20, 1996.
Tatsuo Nakajima; "Continuous Media Objects with Media Scaling"; Object-Oriented
Computing III, Modern Science Cp., LTD., p. 97-106, Dec. 10, 1995.
Japanese Office Action for JP 2002-230974, dated Apr. 5, 2005.
|
Primary Examiner: Wong; Allen
Attorney, Agent or Firm: RatnerPrestia
Claims
What is claimed is:
1. A picture decoding and coding apparatus comprising:
a picture coding apparatus including a picture coding unit operable to code pictures
and provide a picture identifier for each picture as an I, P or B picture,
a priority providing unit operable to correlate each coded picture with a priority
identifier which is independent of the picture identifier, and
a transmission control unit operable to transmit or store the coded pictures
with the priority identifiers, and
a picture decoding apparatus including a reception control unit operable to
receive or read the coded pictures, and
a picture decoding unit operable to decode the coded pictures with the priority identifiers,
wherein each priority identifier is used by the picture decoding apparatus to
determine whether each picture should be processed or not be processed according
to a processing capacity of the picture decoding apparatus, and each priority identifier
is used independently of the picture identifiers and independently of whether the
picture is an I, P or B picture.
2. A picture decoding and coding apparatus according to claim 1, wherein the
priority identifiers provided for the I pictures include at least two different
levels of priority.
3. A picture decoding and coding apparatus according to claim 1, wherein the
priority identifiers provided for inter-frame coded pictures and intra-frame coded
pictures include a first priority level and a second priority level, respectively,
which are different from each other.
4. A picture coding apparatus comprising:
a picture coding unit operable to code pictures and provide a picture identifier
for each picture as an I, P or B picture,
a priority providing unit operable to correlate each coded picture with a priority
identifier which is independent of the picture identifier; and
a transmission control unit operable to transmit or store the coded pictures
with the priority identifiers to a picture decoding apparatus,
wherein each priority identifier is used by the picture decoding apparatus to
determine whether each picture should be processed or not be processed according
to a processing capacity of the picture decoding apparatus and each priority identifier
is used independently of the picture identifiers and independently of whether the
picture is an I, P or B picture.
5. A picture coding apparatus according to claim 4, wherein the priority identifiers
provided for the I pictures include at least two different levels of priority.
6. A picture coding apparatus according to claim 4, wherein the priority identifiers
provided for inter-frame coded pictures and intra-frame coded pictures include
a first priority level and a second priority level, respectively, which are different
from each other.
7. A picture decoding apparatus comprising:
a reception control unit operable to receive or read picture identifiers identifying
an I, P or B picture for coded pictures with priority identifiers transmitted from
a picture coding apparatus, and each picture; and
a picture decoding unit operable to decode the coded pictures with the priority identifiers,
wherein each priority identifier is independent of the picture identifiers and
is used by the picture decoding apparatus to determine whether each picture should
be processed or not be processed according to a processing capacity of the picture
decoding apparatus and each priority identifier is used independently of the picture
identifiers and independently of whether the picture is an I, P or B picture.
8. A picture decoding apparatus according to claim 7, wherein the priority identifiers
provided for the I pictures include at least two different levels of priority.
9. A picture decoding and coding apparatus according to claim 7, wherein the
priority identifiers provided for inter-frame coded pictures and intra-frame coded
pictures include a first priority level and a second priority level, respectively,
which are different from each other.
10. A picture decoding and coding apparatus comprising:
a picture coding apparatus including;
a picture coding unit operable to code pictures and provide a picture identifier
for each picture as an I, P or B picture,
a priority providing unit operable to correlate each coded picture with a priority
identifier which is independent of the picture identifier, and
a transmission control unit operable to transmit or store the coded pictures
with the priority identifiers, and
a picture decoding apparatus including;
a reception control unit operable to receive or read the coded pictures, and
a picture decoding unit operable to receive the coded pictures with the priority identifiers,
wherein each priority identifier is used by the picture decoding apparatus to
determine whether each picture should be processed or not be processed according
to a processing load of the picture decoding apparatus and each priority identifier
is used independently of the picture identifiers and independently of whether the
picture is a I, P or B picture.
11. A picture decoding and coding apparatus according to claim 10, wherein the
priority identifiers provided for the I pictures include at least two different
levels of priority.
12. A picture decoding and coding apparatus according to claim 10, wherein the
priority identifiers provided for inter-frame coded pictures and intra-frame coded
pictures include a first priority level and a second priority level, respectively,
which are different from each other.
13. A picture coding apparatus comprising:
a picture coding unit operable to code pictures and provide a picture identifier
for each picture as an I, P or B pictures;
a priority providing unit operable to correlate each coded picture with a priority
identifier which is independent of the picture identifier; and
a transmission control unit operable to transmit or store the coded pictures
with the priority identifiers to a picture decoding apparatus,
wherein each priority identifier is used by the picture decoding apparatus to
determine whether each picture should be processed or not be processed according
to a processing load of the picture decoding apparatus, and each priority identifier
is used independently of the picture identifiers and independently of whether the
picture is an I, P or B picture.
14. A picture coding apparatus according to claim 13, wherein the priority identifiers
provided for the I pictures include at least two different levels of priority.
15. A picture coding apparatus according to claim 13, wherein the priority identifiers
provided for inter-frame coded pictures and intra-frame coded pictures include
a first priority level and a second priority level, respectively, which are different
from each other.
16. A picture decoding apparatus comprising:
a reception control unit operable to receive or read picture identifiers identifying
an I, P or B picture for coded pictures with priority identifiers transmitted from
a picture coding apparatus, and each picture; and
a picture decoding unit operable to decode the coded pictures with the priority identifiers,
wherein each priority identifier is independent of the picture identifiers and
is used by the picture decoding apparatus to determine whether each picture should
be processed or not be processed according to a processing load of the picture
decoding apparatus, and each priority identifier is used independently of the picture
identifiers and independently of whether the picture is an I, P or B picture.
17. A picture decoding apparatus according to claim 16, wherein the priority
identifiers provided for the I pictures include at least two different levels of priority.
18. A picture decoding apparatus according to claim 16, wherein the priority
identifiers provided for inter-frame coded pictures and intra-frame coded pictures
include a first priority level and a second priority level, respectively, which
are different from each other.
19. A picture decoding and coding apparatus comprising:
a picture coding apparatus including;
a picture coding unit operable to code pictures and provide a picture identifier
for each picture as an I, P or B picture,
a priority providing unit operable to correlate each coded picture with a priority
identifier which is independent of the picture identifier, and
a transmission control unit operable to transmit or store the coded pictures
with the priority identifiers, and
a picture decoding apparatus including;
a reception control unit operable to receive or read the coded pictures, and
a picture decoding unit operable to decode the coded pictures with the priority identifiers,
wherein each priority identifier is used by the picture decoding apparatus to
determine whether each picture should be processed or not be processed when the
picture decoding apparatus is overloaded, and each priority identifier is used
independently of the picture identifiers and independently of whether the picture
is an I, P or B picture.
20. A picture decoding and coding apparatus according to claim 19, wherein the
priority identifiers provided for the I pictures include at least two different
levels of priority.
21. A picture decoding and coding apparatus according to claim 19, wherein the
priority identifiers provided for inter-frame coded pictures and intra-frame coded
pictures include a first priority level and a second priority level, respectively,
which are different from each other.
22. A picture coding apparatus comprising:
a picture coding unit operable to code pictures and provide a picture identifier
for each picture as an I, P or B picture;
a priority providing unit operable to correlate each coded picture with a priority
identifier which is independent of the picture identifier; and
a transmission control unit operable to transmit or store the coded pictures
with the priority identifiers to a picture decoding apparatus,
wherein each priority identifier is used by the picture decoding apparatus to
determine whether each picture should be processed or not be processed when the
picture decoding apparatus is overloaded, and each priority identifier is used
independently of the picture identifiers and independently of whether the picture
is an I, P or B picture.
23. A picture coding apparatus according to claim 22, wherein the priority identifiers
provided for the I pictures include at least two different levels of priority.
24. A picture coding apparatus according to claim 22, wherein the priority identifiers
provided for inter-frame coded pictures and intra-frame coded pictures include
a first priority level and a second priority level, respectively, which are different
from each other.
25. A picture decoding apparatus comprising:
a reception control unit operable to receive or read picture identifiers identifying
an I, P or B picture for coded pictures with priority identifiers transmitted from
a picture coding apparatus, and each picture; and
a picture decoding unit operable to decode the coded pictures with the priority identifiers,
wherein each priority identifier is independent of the picture identifiers and
is used by the picture decoding apparatus to determine whether each picture should
be processed or not be processed when the picture decoding apparatus is overloaded,
and each priority identifier is used independently of the picture identifiers and
independently of whether the picture is an I, P or B picture.
26. A picture decoding apparatus according to claim 25, wherein the priority
identifiers provided for the I pictures include at least two different levels of priority.
27. A picture decoding apparatus according to claim 25, wherein the priority
identifiers provided for inter-frame coded pictures and intra-frame coded pictures
include a first priority level and a second priority level, respectively, which
are different from each other.
Description
This application is a U.S. National Phase Application of PCT International Application PCT/JP97/02696.
TECHNICAL FIELD
The present invention relates to picture and sound decoding apparatus, picture
and sound coding apparatus, and information transmission system for decoding, coding
and synthesizing plural pictures and sounds simultaneously.
BACKGROUND ART
Hitherto there was a video communication system with a feel of presence
by satisfying the sensation of the existence of a person in front of oneself, by
extracting, for example, a human figure from the image of the space scene of one's
own side, and superposing and displaying this image, and a human image sent from
the opposite side, and a virtual space image preliminarily stored for displaying
commonly with the opposite side (Japanese Patent Publication No. 4-24914, Hypermedia
system personal communication system; Fukuda, K., Tahara, T., Miyoshi, T.: "Hypermedia
Personal Computer Communication System: Fujitsu Habitat," FUJITSU Sci. Tech. J.,
26, 3, pp. 196-206, October 1990; Nakamura: "Distributed cooperative job support
by virtual reality sensation through network," Audio-Visual Compound Information
Processing Research Group, Information Processing Society of Japan, 1993). In the
prior art, in particular, in order to synthesize the images, methods about increasing
the speed and reducing the memory capacity have been invented (for example, Japanese
Patent Publication No. 5-46592, Image synthesizing apparatus; Japanese Laid-open
Patent No. 6-105226, Image synthesizing apparatus).
In the prior art, however, image synthesizing systems for synthesizing two-dimensional
still pictures and three-dimensional CG data have been proposed, but nothing has
been mentioned about a method of realizing a system for synthesizing and displaying
by decoding (expanding) plural motion pictures or sounds simultaneously. In particular,
in the terminal device for simultaneously decoding, synthesizing and displaying
plural pictures and sounds, nothing has been mentioned about the reproducing method
of picture and sound not leading to breakdown due to shortage of terminal capacity
or fluctuation of processing capacity. In addition, nothing has been discussed
about the method of decoding, synthesizing and displaying plural pictures depending
on the accounting situation.
More specifically, nothing has been considered about:
(1) method of controlling plural pictures, sound information, information describing
the relation of plural pictures and sounds, and information of processing result; and
(2) method about determining method of priority of decoding, synthesizing and
displaying of plural pictures and sounds when the terminal processing state is
overloaded, and also reproducing and accounting.
Further, in the environments of decoding, synthesizing and displaying plural
pictures and sounds simultaneously, nothing has been considered about the method
of controlling the quantity of coding by varying the method of compression of image
depending on the state at the receiving terminal side or the priority of decoding,
synthesizing, and displaying at the receiving terminal.
DISCLOSURE OF THE INVENTION
Considering these problems of the prior art, it is an object of the present
invention to present a picture and sound decoding apparatus, a picture and sound
coding apparatus, and an information transmission system capable of controlling
the quantity of coding depending on the processing situation at the terminal when
decoding or synthesizing plural pictures and sounds simultaneously, and capable
of controlling decoding, synthesizing and displaying of plural pictures and sounds
depending on the accounting situation.
The invention is not limited to synthesis of two-dimensional images alone. It
may include an expression format combining two-dimensional image and three-dimensional
image, and an image synthesizing method for synthesizing an image by combining
plural adjacent images such as panoramic images.
The communication format of the invention is not limited to the wired two-way
CATV or B-ISDN alone. For example, it may include radio wave transmission (for
example, VHF band, UHF band) or satellite broadcasting for picture and sound from
the center side terminal to the household side terminal, and analog telephone circuit
or N-ISDN for information transmission from the household side terminal to the
center side terminal (picture, sound and data are not always required to be multiplexed).
It may further include the communication format making use of wireless means, such
as IrDA, PHS (personal handy phone), and radio LAN.
Applicable terminals include portable terminals such as portable information
terminals, and desk-top terminals such as set-top box and personal computer.
One aspect of the present invention is a picture decoding and coding apparatus
comprising a picture coding apparatus including picture coding means for coding
picture information, and transmission control means for transmitting or recording
the coded various information, and a picture decoding apparatus including reception
control means for receiving the coded various information, picture decoding means
for decoding received various information, picture synthesizing means for synthesizing
one or more decoded pictures, and output means for delivering the synthesized picture.
Another aspect of the present invention is a sound decoding and coding apparatus
comprising a sound coding apparatus including sound coding means for coding sound
information, and transmission control means for transmitting or recording the coded
various information, and a sound decoding apparatus including reception control
means for receiving the coded various information, sound decoding means for decoding
received various information, sound synthesizing means for synthesizing one or
more decoded sounds, and output means for delivering the synthesized sound.
Still another aspect of the present invention is a realtime picture coding
apparatus comprising one or more picture input means for feeding pictures, picture
input control means for controlling the control state of the picture input means,
other terminal control request control means for controlling the reception state
of a reception terminal, coding process decision means for determining the coding
method of pictures depending on at least the controlled reception state of the
reception terminal or the control state of said picture input means, picture coding
means for coding said input picture according to the result of decision by the
coding process decision means, and output means for delivering the coded picture.
A further aspect of the present invention is an information transmission system,
being an information transmission system using at least one of the picture decoding
and coding apparatus and the sound decoding and coding apparatus as the reception
terminal, and at least one of the picture decoding and coding apparatus the sound
decoding and coding apparatus and the real time picture coding apparatus as the
transmission terminal, and connecting these terminals through a communication route,
wherein one of the presence or absence of coding of pictures or sounds, priority
of coding, coding method, picture size to be coded, value of quantization step,
number of frames, and priority of processing when the reception terminal is overloaded
is determined at the transmission terminal, by transmitting at least one of the
load of the reception terminal, the information about the priority of coded information
to be processed as being determined in the priority decision means at the reception
terminal, and the frame skip situation at the reception terminal, to the transmission terminal.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic structural diagram of a picture decoding and coding apparatus
in an embodiment of the invention;
FIG. 2 is a schematic structural diagram of a picture and sound decoding and
coding apparatus showing other example in the same embodiment;
FIG. 3 is a diagram for explaining an example of adding information about priority
in a communication and recording format;
FIG. 4 is a diagram for explaining an example of composing the invention by software;
FIG. 5 is a diagram for explaining the composition of information;
FIG. 6 is a diagram for explaining the operation of DEMUX thread.
FIG. 7 is a diagram for explaining the operation of monitor thread.
FIG. 8 is a diagram for explaining the operation of decoding process;
FIG. 9 is a diagram for explaining the operation of picture synthesizing thread;
FIG. 10 is a diagram for explaining the operation of display monitor thread;
FIG. 11 is a diagram for explaining the user interface picture synthesizing apparatus;
FIG. 12 is a diagram for explaining the method of picture transmission depending
on fluctuation of capacity of the receiving side terminal;
FIG. 13 is a diagram for explaining the picture compression apparatus in an
embodiment of the invention;
FIG. 14 is a diagram for explaining the information controlled by operation
control unit;
FIG. 15 is a diagram for explaining the picture compression apparatus for creating
panoramic pictures; and
FIG. 16 is a diagram for explaining the response situation between transmission
terminal and reception terminal.
REFERENCE NUMERALS
11 Reception control unit
12 Separation unit
13 Transmission control unit
14 Priority decision unit
17 Time information control unit
18 Picture expanding unit
19 Picture synthesizing unit
20 Sound expanding unit
21 Sound synthesizing unit
31 DEMUX thread
36 Decoding process
37 Monitor thread
39 Picture synthesizing thread
42 Display monitor thread
1204 Operation control unit
1205 Picture compression unit
1208 Picture processing decision control means
1401 Priority decision control unit
1402 Operation history control unit
1404 Picture synthesizing unit
1407 Input unit
MODE FOR CARRYING OUT THE INVENTION
The invention is described below by referring to drawings showing its preferred
embodiments. The term "picture" used in the invention includes both still picture
and motion picture. The applicable pictures may be a mixture of two-dimensional
picture such as computer graphics (CG) and three-dimensional picture as composed
from a wire-frame model. In such a case, the relation between pictures corresponds
to the wire-frame model. The script language for describing may include JAVA, VRML,
etc, which language is incorporated herein by reference.
FIG. 1 and FIG. 2 are schematic structural diagrams of picture decoding and
coding apparatus in an embodiment of the invention. FIG. 1 shows a constitution
without sound reproducing function, and FIG. 2 shows a constitution having both
picture and sound reproducing function. Of course, it may be constituted similarly
for sound alone.
This apparatus shown in FIG. 1 or FIG. 2 is composed of a coding apparatus and
a decoding apparatus, and the coding apparatus in FIG. 1 comprises a priority providing
unit 101 for determining the priority of processing of coded picture at
the time of overloading, and corresponding between the coded image and the priority,
a picture coding unit 102 for coding the picture, a transmission control
unit 103 for transmitting or recording coded information provided with priority,
and a reception control unit 104 for receiving coded information. The coding
apparatus in FIG. 2 further comprises a sound coding unit 105 for coding
the sound.
On the other hand, in the decoding apparatus, a reception control unit 11
for receiving information and a transmission control unit 13 for transmitting
information are means for transmitting information in coaxial cable, CATV, LAN,
modem, etc. Connection formats of terminal include two-way sending and receiving
format of video information between terminals such as TV telephone and TV conference
system, and broadcasting type (one-way) picture broadcasting format by satellite
broadcasting, CATV or Internet. In the invention, such connection formats of terminals
are taken into consideration.
A separation unit 12 is means for analyzing and separating the coded (compressed)
reception signal (in the case of compression apparatus, it is a reverse operation,
and hence it corresponds to a multiplexing unit). For example, in MPEG1, MPEG2,
and H. 320 terminal (regulation of TV telephone and conference apparatus using
N-ISDN), the H. 221 is the regulation for multiplexing and separating the video/audio/data,
and in the H. 324 terminal (regulation of TV telephone and conference apparatus
using analog telephone circuit), the H. 223 is the corresponding regulation MPEG
1 and MPEG 2 Standards and the International Telecommunications Union (ITU-T) H.-series
standards are all incorporated herein by reference. The invention may be realized
either by the constitution conforming to the regulation or by the constitution
not conforming to the regulation. Or, as done in the H. 323 or Internet, the picture
and sound may be transmitted independently by different streams.
A priority decision unit 14 determines the priority of decoding (hereinafter
referred to as "expanding") the information obtained from the separation unit 12
(for example, picture, sound, control information) when the terminal is overloaded
in the following method, and expanding the picture or expanding the sound (in the
method of determining the priority of processing, it may be preliminarily decided
in the reception terminal device, or the information about the priority determined
in the following method may be added to the recording media or transmission packet
at the transmission side terminal (coding apparatus), and transmitted and added
as recording format; as the method of expressing the priority, the priority may
be expressed by non-numerical method such as large, medium and small, or by numerical
method such as 1, 2, 3).
Using the identifier for handling data in the stream unit composed of plural
picture or sound frames, by processing transmission and reception of data between
the transmission side and reception side, control of buffer at the reception side,
and scheduling of transmission of data at transmission side may be realized. That
is, by noticing the identifier of the stream sent from the transmission side as
required, the receiving situation at the reception side may be investigated, or
the identifier of the stream not requiring may be noticed to the reception terminal,
or the required stream may be demanded from the reception side.
The priority providing means for determining the priority of processing of coded
information at the time of overloading by the standard mentioned above, and corresponding
between the coded information and the determined priority may be provided in the
picture coding apparatus or sound coding apparatus, and according to the priority
of various receiving information at the time of overloading, the picture frame
or sound of the priority to be processed may be determined by the priority decision
means for determining the method of processing, so that coding or synthesizing
process may be done. As for the picture frame, it is necessary to insert periodically
the frame undergoing intraframe coding (I frame) so that the frame can be skipped.
The unit for providing the priority may be each unit of picture or sound (comparison
of priority between frames), or the stream unit composed of plural frames (comparison
of priority between streams).
The method of taking note of the feature of image includes the methods on the
basis of the image compression format (for example, between H. 263 and run length,
the priority is given to the run length), picture size (for example, between CIF
and QCIF, the priority is given to the QCIF), contrast (for example, the priority
is given to the higher contrast), picture synthesizing ratio (for example, the
priority is given to the higher synthesizing ratio), quantization step (for example,
the priority is given to the smaller quantization step), difference between inter-frame
coding and intra-frame coding (for example, the priority is given to the intra-frame
coding), display position (for example, the priority is given to the middle display
position; or in the case of three-dimensional picture, the priority is low when
the picture is set at inner side, and the priority is set high when displayed at
the front side), frame number (the priority is high in first frame and final frame,
or the priority is heightened for the frame of scene change, etc.), the number
of frames (for example, the priority is higher in the picture smaller in the number
of frames to be reproduced; the frame number corresponds to temporary reference
(TR) in the case of H. 263, and it may be judged according to the change of TR
value), the voiced section and silent section, the display time (PTS), and decoding
time (DTS).
In addition, the same priority is given to the P frame and B frame by inter-frame
coding. In the intra-frame coded pictures, by assigning priority in plural stages,
the skipping frequency can be controlled.
Examples taking note of difference in media include the method of expanding
the sound by priority over expanding of picture. As a result, the sound can be
reproduced without interruption.
Furthermore, on the basis of the reproduction permit information controlled
at the reception side terminal, the information to be expanded (picture, sound)
may be determined, or the information to be expanded may be selected on the basis
of the reproduction permit information sent from the transmission side as control
information. The reproduction permit information specifically includes the accounting
information (for example, if not paid duly, processing of expanding, synthesizing
or display is not done; the accounting information can be controlled at the reception
terminal side, or the accounting information may be controlled at the transmission
side), the service content information (for example, in the case of an adult broadcast
program, unless reproduction is permitted at the terminal side, processing of expanding,
synthesizing or display is not done; the reproduction permit may be controlled
at the reception side terminal or at the transmission side terminal), the password
(for example, a specific program is not expanded, synthesized or displayed unless
the password is entered; the password may be controlled at the reception side terminal
or at the transmission side terminal), the user code (for example, unless the user
is permitted, processing of expanding, synthesizing or display is not done; the
user code may be controlled at the reception side terminal or at the transmission
side terminal), and the nation code (for example, the picture and sound to be expanded,
synthesized and displayed, and the reproduction method are changed depending on
the nation; the nation code may be controlled at the transmission side or at the
reception side; scrambling is realized by changing the reproduction method by the
nation code).
The reproduction method by limiting the permission of reproduction of picture
and sound by the accounting information, service content information, password,
or user code includes the method of deviating the position or pixel intentionally
when synthesizing and displaying the picture, changing the magnification or contraction
of picture or sampling of picture (for example, by low pass), inverting the pixels,
changing the contrast, changing the color pallets, and skipping the frames. In
these picture reproducing methods (expanding, synthesizing and display pictures),
each frame may be restricted. Or, in the unit of GOB (group of block) which is
an independent processing unit smaller than one frame, as defined by H. 263 which
is one of picture compression techniques, the picture expanding, synthesizing and
display method may be restricted, so that more flexible control is realized as
compared with the conventional technique of disturbing the entire picture. That
is, by processing in the GOB unit, only a part of the picture can be scrambled,
and the interactive software such as the software using the image synthesis can
be evaluated.
Similarly, sound reproducing methods include the methods of changing the
sound level, changing the sound direction, changing the sound frequency, changing
the sound sampling, and inserting different picture or sound (in any method, it
is divided into the method of processing preliminarily at the transmission side,
and the method of processing at the reception side).
The reproducing method of picture and sound includes a method of canceling the
synchronism of picture and sound. The priority of picture and sound to be expanded,
or the presence or absence may be determined by the information indicating the
rank of synthesizing and displaying (the display sequence is preliminarily determined
at the reception side terminal, for example, the priority is given to the CIF or
still picture, or the display sequence is added preliminarily to the transmission
information at the transmission side as the information about priority), the information
indicating the rank of expanding (the expanding sequence is preliminarily determined
at the reception side terminal, for example, the priority is given to the QCIF
or intra-frame coded picture data, or the speech sound is expanded by priority
over BGM; similarly, the display sequence is added to the transmission information
at the transmission side), the user's instruction (for example, the picture or
sound information to be expanded, synthesized or displayed is selected by the user's
instruction, or the picture or sound information to be expanded, synthesized or
displayed is determined on the basis the information selected according to the
request), the terminal processing capacity (for example, by measuring the occupation
time of CPU processing in a specific period at the present or in the past, expanding,
synthesizing or displaying of the picture or sound supposed to take time is restricted;
as the estimating method of processing time, the time required for local decoding
when compressing, or the time required for compression is controlled in relation
to the compressed picture information, so that the presence or absence of expanding,
synthesizing or displaying, or the priority can be determined), the reproduction
time (for example, expanding, synthesizing or displaying of the picture or sound
information passing the reproduction time is stopped), or the decoding time.
In addition, as the method for preventing specific picture or sound from being
expanded or displayed by priority, on the basis of the information relating to
the execution rate for processing of expanding, synthesizing or displaying of picture
or sound, the sequence or presence or absence of the picture to be expanded, synthesized
or displayed can be determined. For example, once out of ten times of expanding,
it is set at the reception terminal side that the picture of CIF size is expanded,
or by specifying the execution rate of expanding, synthesizing or displaying the
picture or sound at the transmission side, the picture information or sound information
can be transmitted according to this. The execution rate is specifically defined
by the insertion interval of I frame (intra-frame coded frame). As a result, expanding,
synthesizing or displaying of specific picture or sound object only is prevented.
Such information about the priority for controlling expanding, synthesizing
or displaying may be added not only at the transmission side apparatus, but may
be also added and controlled at a relaying apparatus. Moreover, by transmitting
the information about the priority determined in the priority decision unit 14
of the decoding apparatus at the reception terminal, to the transmission destination
through the transmission control unit 13, the picture or sound can be transmitted
depending on the situation of determination in the priority decision unit 14
(by sending the ID of the picture object that is hardly selected to the transmission
side, wasteful transmission is prevented). Incidentally, the information showing
the priority of processing when the reception terminal is overloaded may be determined
at the reception terminal apparatus, or may be transmitted as transmission format,
or the transport stream of MPEG2 may be expanded as the format for recording in
the recording media such as CD-ROM or hard disk, or the transmission and recording
format without consideration of standardization may be also employed. Or, using
different streams in individual media (picture, sound, information describing the
relation of picture and sound), data may be transmitted and recorded without multiplexing.
The picture expanding unit 18 as picture decoding means is means for expanding
the picture (hereinafter coding means in the case of the coding apparatus), and
the picture formats handled in the picture expanding unit 18 include MPEG1,
MPEG2, H. 261, H. 263, and others. The picture may be expanded in the unit of one
frame, or it may be processed in the unit of GOB specified in H. 263. When processing
in the unit of one frame, in the case of inter-frame coding, the expanded state
of the previous frame must be stored in the picture expanding unit 18. When
the picture is expanded in the GOB unit, the sequence relation of expanding the
pictures does not matter. Therefore, when expanding in the GOB unit, plural picture
expanding units 18 are not required in the receiving apparatus, and plural
pictures can be expanded by one picture expanding unit 18. To the contrary,
the results of expanding must be accumulated.
The sound expanding unit 20 as the sound decoding means in FIG. 2 is the
means for expanding the sound, and the sound format handled in the sound expanding
unit 20 includes G. 721 and G. 723. The method for processing includes the
software processing by DSP or general-purpose CPU, and processing by special-purpose hardware.
When realizing by the software, expanding process of picture and sound is individually
controlled in the unit of one process or thread, and when there are simultaneously
plural pictures or sounds to be expanded, they are processed by dividing the time
by the number of processes or threads in the range to be processed.
The picture expanding control unit 15 is means for controlling the expanded
state of picture. The sound expanding control unit 16 is means for controlling
the expanded state of sound. For example, when realizing these control units by
the software, the compressed information obtained from the separation unit 12
is transferred to the picture expanding unit 18 and sound expanding unit
20 in the predetermined sequence (for example, executing first in the sound
expanding unit 20 and then executing in the picture expanding unit 18),
and the expanded state is controlled. When all expanding is over, the expanded
information is transferred to the picture synthesizing unit 19 or sound
synthesizing unit 21. In the software, by using shared memory and semaphore,
the transferred information is limited, or end of expanding process is known (detail
is described later).
A time information control unit 17 is means for controlling the information
about the time. For example, when realizing the system by a personal computer,
the time information can be realized by utilizing the timer of the personal computer.
The picture synthesizing unit 19 synthesizes a picture on the basis of
the expanded image data. When synthesizing plural pictures, the pictures are synthesized
on the basis of the synthesis ratio (α value) of each picture. For example,
when synthesizing two pictures, if the synthesis ratio of the foreground picture
is α, the RGB value of the background picture is mixed at a rate of 1-α,
and the foreground picture, at α. Incidentally, the picture to be expanded
is controlled in the processing in the unit of one frame, and therefore when synthesizing
plural pictures by using the display time, the system configuration and mounting
can be simplified. Until discarding of the expanding result is instructed from
the transmission side in the picture synthesizing unit 19 or sound synthesizing
unit 21, by holding, controlling and utilizing the expanding result, it
is not necessary to send the information of same pattern repeatedly from the transmission side.
On the basis of the information describing the relation of mutual pictures or
mutual sounds, when combining the picture or sound, by presenting that the pictures
or sounds that cannot be synthesized are present as necessary decoded pictures
or sounds are not ready, the user can know the state of synthesis. Accordingly,
the user instructs to select the necessary picture quality or select the picture
desired to be synthesized beforehand, so that necessary information can be synthesized
without dropout. Incidentally, the method of accumulating and controlling the decoded
picture and sound data in the buffer includes the methods of erasing from the older
data in the sequence of arrival, and erasing by observing the script describing
the relation of mutual pictures or mutual sounds, and also observing the state
of use of the decoded picture or sound data on the whole.
The sound expanding control unit 16 controls the expanding state of at
least one sound expanding unit 20 for expanding the sound.
The sound synthesizing unit 21 is means for synthesizing the sound on
the basis of the expanded information, and a synthesized result accumulating unit
22 is means for accumulating the picture synthesized by the picture synthesizing
unit 19 and the sound synthesized by the sound synthesizing unit 21.
A reproduction time control unit 23 is means for reproducing the synthesized
picture or sound at the time for starting reproduction.
An output unit 24 is means for issuing the synthesized result (for example,
display and printer), and an input unit 25 is means for entering information
(for example, keyboard, mouse, camera, video). A terminal control unit 26
is means for controlling these units.
FIG. 3 is a diagram for explaining an example of a case of adding information
about priority in communication and recording format.
FIG. 3(
a) shows an example of multiplexing all media completely (picture,
sound, control information). The control information includes the priority for
determining the processing when overloaded (the priority as indicated in the invention),
and the priority showing the sequence of display. The control information may also
include the information about the relation between pictures, between sounds, or
between picture and sound (in terms of time or position). The example in FIG. 3
(
a) is suited, for example, to application in multiplexing of MPEG1/2, or
packet multiplexing mixing control information and data (picture, sound) such as
H. 223. The priority of processing in overload is added in the frame unit or stream unit.
FIG. 3(
b) shows an example of multiplexing information in every media.
In this example, the control information, picture information, and sound information
are transmitted from individual communication ports. The information about the
relation between pictures, between sounds, and between picture and sound may be
transmitted as control information from different communication port from that
of picture or sound. It is suited to application where plural communication ports
can be established simultaneously, such as in H. 323 and Internet, and as compared
with FIG. 3(
a), multiplexing process can be simplified, so that the terminal
load can be alleviated.
As the method for describing mutual pictures or mutual sounds, it may be considered
to be applicable by descriptive language such as JAVA and VRML, but the specification
of descriptive language of the script may not be determined automatically. Accordingly,
by adding an identifier for identifying the descriptive method of the information
describing the relation between pictures or between sounds (for example, information
in terms of position or time, such as display period), it is applicable to plural
methods of description. To add the identifier for identifying the information descriptive
method, for example, in MPEG2, it is provided in the program map table for controlling
the stream of MPEG2-TS, or in the stream describing the script. The priority of
processing in overload is added together with the information describing the corresponding
relation of picture and sound (control information). In the MPEG2, in order to
control by the program map table for relating the video stream and audio stream
of MPEG2-TS (transport stream), the structural information and stream for relating
between picture and sound may be defined and controlled, so that it can be transmitted
independently of the data also in the MPEG2.
FIG. 4 is a diagram explaining an example of composing the invention by the
software. When the invention is realized on the operating system capable of operating
in multitask, each process explained in FIG. 1 and FIG. 2 is divided into execution
module units of software such as process and thread, and between processes or between
threads, information is exchanged by shared memory, and the shared information
is controlled exclusively by the semaphore (in the example in FIG. 4, the area
indicated by solid line corresponds to the semaphore). The function of each process
and thread is described below.
A DEMUX thread 31 reads the information (picture, sound, control information)
multiplexed from the network or disk, and separates into a monitor table (detail
given below) describing the corresponding relation of sounds, pictures, and sound
and picture and the information about reproduction time. The DEMUX thread 31
corresponds to the separation unit 12. The information separated by the
DEMUX thread 31 is sent out into a sound ring buffer 32, a picture
ring buffer 33, and a monitor ring buffer 34. In the case of sound
information, the information sent out into the ring buffer 32 is expanded
in a sound decoding thread 35 (corresponding to the sound expanding unit
20). In the case of picture information, the information sent out to the
ring buffer 33 is expanded in a decoding process 36.
The monitor table is sent out to the ring buffer 34, and is utilized in
a monitor thread 37 for determining the sequence for expanding the picture
(corresponding to the control terminal unit 26, picture expanding control
unit 15, and sound expanding control unit 16 above). The same monitor
table is utilized in a picture synthesizing thread 39 for synthesizing a
picture. The monitor table utilized in the monitor thread 37 reads out the
next table from the ring buffer 34 when expanding of all sound and pictures
is over. The picture information expanded in the decoding process 36 (corresponding
to the picture expanding unit 18) is sent out to a picture single buffer
38. When the transmitted pictures are all ready, the picture is synthesized
by using the ratio of the image synthesis controlled in the monitor table, by a
picture synthesis thread 39 (corresponding to the picture synthesizing unit
19). The result of synthesis is accumulated in a synthesis buffer 41
(corresponding to the synthesis result accumulation unit 22), thereby waiting
in display wait state until reaching the display time in a display monitor thread
42 (corresponding to the reproduction time control unit 23).
FIG. 5 is a diagram for explaining the composition of information used in the
constitution in FIG. 4. In the example in FIG. 5, the information received from
the disk or network has a fixed length of 188 bytes (B). The composition of the
sound information separated in the DEMUX thread 31 is composed of packet
synchronous code, reproduction time, frame length showing the length of sound to
be reproduced, and sound data (C). The composition of picture information is composed
of packet synchronous code, frame number for identifying the picture, frame length
for showing the size of picture information, and picture data (D). The invention
does not require processing in the unit of one frame, but it is allowed to process
in a small block unit such as macro block unit.
The composition of monitor table is composed of picture display time, number
of pictures to be displayed (synthesized) in one frame, ID of each picture, frame
number, priority for expanding or displaying, identifier showing the frame type
(I picture, P picture, B picture), display horizontal position, display vertical
position, and each information of layer showing the ratio of synthesis (E). Incidentally,
the synthesis ratio of picture and synthesis ratio of sound may be changed by correspondence.
For example, when two pictures correspond to two sounds respectively, if the picture
synthesis ratio is α:1-α, the corresponding sound synthesis ratio may
be also α:1-α. Not limited to the relation between pictures, the relation
between sounds may be also described (for example, direction, kind such as BGM
or speech sound).
FIG. 6 is a diagram for explaining the operation of DEMUX thread. From a file
or network, data of fixed length of 188 bytes is read in (5-1). The
read data is analyzed, and set in the type of the structure of sound, picture or
monitor table as mentioned above (5-2). If possible to write into
the ring buffer, the sound, picture and monitor tables are written into individual
ring buffers. The corresponding relation of picture object ID and plural picture
expanding means is obtained. In this example, fr
