Neural networks with learning and expression capability Number:7,412,426 from the United States Patent and Trademark Office (PTO) owispatent A neural network comprising a plurality of neurons in which any one of the plurality of neurons is able to associate with itself or another neuron in the plurality of neurons via active connections to a further neuron in the plurality of neurons.<H capability, expression, Neural, networks, 7412426.html, Patent, pto, 7412426

Title: Neural networks with learning and expression capability

Abstract: A neural network comprising a plurality of neurons in which any one of the plurality of neurons is able to associate with itself or another neuron in the plurality of neurons via active connections to a further neuron in the plurality of neurons.

Patent Number: 7,412,426 Issued on 08/12/2008 to Hercus

Inventors: Hercus; Robert George (Kuala Lumpur, MY)
Assignee: Neuramatix SDN. BHD. (Kuala Lumpur, MY)

Appl. No.: 10/560,666

Filed: June 21, 2004

PCT Filed: June 21, 2004

PCT No.: PCT/IB2004/002119

371(c)(1),(2),(4) Date: December 12, 2005

PCT Pub. No.: WO2004/114145

PCT Pub. Date: December 29, 2004

Foreign Application Priority Data


Jun 26, 2003 [MY]			PI20032400

Current U.S. Class: 706/15

Current International Class: G06E 1/00 (20060101); G06E 3/00 (20060101); G06F 15/18 (20060101); G06G 7/00 (20060101)

Field of Search: 706/15

References Cited [Referenced By]

U.S. Patent Documents


5091864	February 1992	Baji et al.
5166539	November 1992	Uchimura et al.
5182794	January 1993	Gasperi et al.
5353383	October 1994	Uchimura et al.
5467429	November 1995	Uchimura et al.
5671335	September 1997	Davis et al.
5712953	January 1998	Langs
5852815	December 1998	Thaler
5937432	August 1999	Yamaguchi et al.
6052679	April 2000	Aparicio et al.
6195622	February 2001	Altschuler et al.
6456993	September 2002	Freund
6468069	October 2002	Lemelson et al.
6581049	June 2003	Aparicio et al.
6707935	March 2004	Kramer
6751343	June 2004	Ferrell et al.
6801655	October 2004	Woodall
7016886	March 2006	Cabana et al.
7089217	August 2006	Kasabov
2005/0063565	March 2005	Nagaoka et al.
2006/0184462	August 2006	Hawkins
2007/0005531	January 2007	George et al.

Foreign Patent Documents


0245508	Nov., 1987	EP
0223486	Aug., 1992	EP
0602717	Jun., 1994	EP
0276068	Apr., 1995	EP
0360674	Nov., 1995	EP
0453939	Jul., 1996	EP
0450522	Jan., 1997	EP
0450521	Mar., 1997	EP
0599347	Jul., 1998	EP
0506730	Nov., 1999	EP
0612033	Nov., 1999	EP
0583217	May., 2000	EP
0574936	May., 2002	EP
1197913	Sep., 2003	EP
1197914	Oct., 2003	EP
1351188	Oct., 2003	EP
WO 95/00920	Jan., 1995	WO
WO 97/29437	Aug., 1997	WO
WO00/36524	Jun., 2000	WO
WO02/15125	Feb., 2002	WO

Other References

Dynamic model of dual layer neural network for vertebrate retina Yagi, T.; Funahashi, Y.; Ariki, F.; Neural Networks, 1989. IJCNN., International Joint Conference on Jun. 18-22, 1989 pp. 787-789 vol. 1 Digital Object Identifier 10.1109/IJCNN.1989.118668. cited by examiner .
Memory based processor array for artificial neural networks Youngsik Kim; Mi-Jung Noh; Tack-Don Han; Shin-Dug Kim; Sung-Bong Yang; Neural Networks,1997., International Conference on vol. 2, Jun. 9-12, 1997 pp. 969-974 vol. 2 Digital Object Identifier 10.1109/ICNN.1997.616157. cited by examiner .
Implementation of ANN on RISC processor array Hiraiwa, A.; Fujita, M.; Kurosu, S.; Arisawa, S.; Inoue, M.; Application Specific Array Processors, 1990. Proceedings of the International Conference on Sep. 5-7, 1990 pp. 677-688 Digital Object Identifier 10.1109/ASAP.1990.145502. cited by examiner .
Neural Network Based Memory Access Prediction Support for SoC Dynamic Reconfiguration Chtourou, S.; Chtourou, M.; Hammami, O.; Neural Networks, 2006. IJCNN '06. International Joint Conference on 0-0 0 pp. 2823-2829 Digital Object Identifier 10.1109/IJCNN.2006.247210. cited by examiner .
RAP: a ring array processor for multilayer perceptron applications Morgan, N.; Beck, J.; Allman, E.; Beer, J.; Acoustics, Speech, and Signal Processing, 1990. ICASSP-90., 1990 International Conference on Apr. 3-6, 1990 pp. 1005-1008 vol. 2 Digital Object Identifier 10.1109/ICASSP.1990.116058. cited by examiner .
Out-of-core backpropagation Diegert, C.; Neural Networks, 1990., 1990 IJCNN International Joint Conference on Jun. 17-21, 1990 pp. 97-103 vol. 2 Digital Object Identifier 10.1109/IJCNN.1990.137701. cited by examiner .
Learning algorithms for suppressing motion clutter in airborne array radar Johnson, J.D.; Li, H.; Culpepper, E.B.; Blasch, E.P.; Klopf, A.H.; Aerospace and Electronics Conference, 1997. NAECON 1997., Proceedings of the IEEE 1997 National vol. 2, Jul. 14-17, 1997 pp. 840-845 vol. 2 Digital Object Identifier 10.1109/NAECON.1997.622738. cited by examiner .
Hopfield associative memory on mesh Ayoubi, R.A.; Ziade, H.A.; Bayoumi, M.A.; Circuits and Systems, 2004. ISCAS '04. Proceedings of the 2004 International Symposium on vol. 5, May 23-26, 2004 pp. V-800-V-803 vol. 5. cited by examiner .
C-NNAP: a dedicated platform for binary neural networks Kennedy, J.V.; Austin, J.; Pack, R.; Cass, B.; Artificial Neural Networks, Fifth International Conference on (Conf. Publ. No. 440) Jul. 7-9, 1997 pp. 161-166. cited by examiner .
Minimisation of torque ripple in a switched reluctance motor using a neural network Reay, D.S.; Green, T.C.; Williams, B.W.; Artificial Neural Networks, 1993., Third International Conference on May 25-27, 1993 pp. 224-228. cited by examiner .
The chaos in the synchrony of abnormal oscillations in a pair of neurons coupled via gap junctionGe Manling; Guo Hongyong; Dong Guoya; Jia Wenyan; Li Ying; Sun Minggui; Wang Baozhu; Yan Weili; Signal Processing, 2004. Proceedings. ICSP '04. 2004 7th International Conference on vol. 3, Aug. 31-Sep. 4, 2004 pp. 2210-2213 vol. 3. cited by examiner .
Hardware realization of novel pulsed neural networks based on delta-sigma modulation with GHA learning rule Murahashi, Y.; Doki, S.; Okuma, S.; Circuits and Systems, 2002. APCCAS '02. 2002 Asia-Pacific Conference on vol. 2, Oct. 28-31, 2002 pp. 157-162 vol. 2 Digital Object Identifier 10.1109/APCCAS.2002.1115144. cited by examiner .
Curvature and orientation estimation by neuronal structures Tanaka, J.S.; Manoel, E.T.M.; da Fontoura Costa, L.; Computer Graphics and Image Processing, 2000. Proceedings XIII Brazilian Symposium on Oct. 17-20, 2000 pp. 44-51 Digital Object Identifier 10.1109/SIBGRA.2000.883893. cited by examiner .
Basic characteristics of hardware neuron model based on CMOS negative resistance: realization of post-inhibitory rebound firing and its application Kanoh, S.; Kumagai, H.; Futami, R.; Hoshimiya, N.; Neural Information Processing, 1999. Proceedings. ICONIP '99. 6th International Conference on vol. 2, Nov. 16-20, 1999 pp. 579-584 vol. 2. cited by examiner .
International Search Report for International Application No. PCT/IB2004/002119, completed on Sep. 24, 2004, mailed Sep. 29, 2004. cited by other .
International Preliminary Report on Patentability for International Application No. PCT/IB2004/002119, completed Feb. 25, 2005. cited by other .
Humpert, B., "Bidirectional Associative Memory With Several Patterns", IJCNN 1990, Jun. 17-21, 1990, pp. I-741-I-750 and IEEE Xplore 1-page printout (11 pages), printed Sep. 22, 2004. cited by other .
Wang, L., "Multi-associated neural networks and their applications to learning and retrieving complex spatio-temporal sequences", IEEE Transactions on Systems, Man and Cybernetics, Feb. 1999, printed Sep. 21, 2004. cited by other .
Carpenter et al., "ART 2: Self-Organization of Stable Category Recognition Codes for Analog Input Patterns", Applied Optics, Dec. 1, 1987, vol. 26, No. 23, pp. 4919-4930. cited by other .
Carpenter et al., "Adaptive Resonance Theory", The Handbook of Brain Theory and Neural Networks, Second Edition; Sep. 1998, pp. 1-12. cited by other .
Fahlman et al., "The Cascade-Correlation Learning Architecture", source unknown, Feb. 14, 1990, pp. 1-13, title page. cited by other.

Primary Examiner: Holmes; Michael B
Attorney, Agent or Firm: Kauth, Pomeroy, Peck & Bailey LLP

Claims

The invention claimed is:

1. An artificial neural network comprising: (a) a plurality of neurons, (b) each of the plurality of neurons being a processor with memory and being in an array; (c) the plurality of neurons comprising a plurality of elemental neurons and a plurality of structural neurons; (d) all elemental and structural neurons being configured to be associated with others of the elemental and structural neurons via active connections; (e) each elemental neuron being configured to: (i) represent a unique input value into the artificial neural network, the unique value being at least one selected from the group consisting of: a stimulus, an event, events, a sequence in a pattern, a sequence of events, an elemental stimulus, a defined elemental pattern, a defined elemental data element, a basic input stimulus, and an output stimulus of information being processed; and (ii) express that unique value as an output when activated by a structural neuron; (f) each structural neuron being configured to: (i) receive input from a pair of neurons of the plurality of neurons and with which it is an associating neuron; and (ii) express that input as an output to that pair of neurons to activate the pair of neurons for expression.

2. An artificial neural network as claimed in claim 1, wherein any one of the plurality of neurons is able to associate with a neuron in the plurality of neurons via the active connections to a further neuron in the plurality of neurons, the further neuron being one of the plurality of structural neurons.

3. An artificial neural network as claimed in claim 1, wherein each structural neuron represents the combined information of patterns by the pair of neurons with which it associates, the structural neuron receiving input from the pair of neurons.

4. An artificial neural network as claimed in claim 1, wherein the plurality of elemental neurons is in a root level of the neural network.

5. An artificial neural network as claimed in claim 1, wherein the pair of neurons comprises at least one selected from the group consisting of: an elemental neuron and an elemental neuron, an elemental neuron and a structural neuron, a structural neuron and an elemental neuron, and a structural neuron and a structural neuron.

6. An artificial neural network as claimed in claim 1, wherein each of the plurality of neurons is one or more selected from the group consisting of: initiating neuron, associated neuron, and associating neuron; an initiating neuron being associated with an associated neuron via connections to the associating neuron.

7. An artificial neural network as claimed in claim 6, wherein the initiating neuron, the associated neuron and the associating neuron are connected based on proximal characteristics, the proximal characteristics being at least one of: temporal, spatial, intensity, magnitude and relative position of the input being processed.

8. An artificial neural network as claimed in claim 6, wherein each initiating neuron is able to associate with a plurality of associated neurons to form a plurality of pairs of neurons.

9. An artificial neural, network as claimed in claim 6, wherein each associated neuron is able to associate with a plurality of initiating neurons to form a plurality of pairs of neurons.

10. An artificial neural network as claimed in claim 6, wherein when an initiating neuron receives input and an associated neuron receives input, the inputs are transmitted to all associating neurons of the initiating neuron and the associated neuron, the associating neuron of both the initiating neuron and the associated neuron then being activated and being able to produce output.

11. An artificial neural network as claimed in claim 10, wherein the associated neuron is activated and able to produce output in a manner selected from the group consisting of: at the same time as the initiating neuron, and after the initiating neuron.

12. An artificial neural network as claimed in claim 10, wherein the activation or production of output of the initiating neuron and the associated neuron is also based on proximal characteristics.

13. An artificial neural network as claimed in claim 12, wherein the proximal activation of or production of output from the initiating neuron and the associated neuron causes at least one selected from the group consisting of: the creation of a new associating neuron if none exists together with new connections between the initiating neuron and the new associating neuron and between the associated neuron and the new associating neuron, the strengthening of existing connections between the initiating neuron and the associating neuron and between the associated neuron and the associating neuron, and strengthening of the associating neuron.

14. An artificial neural network as claimed in claim 13, wherein the strengthening is by maintaining a frequency count of how often the associating neuron receives input from the initiating neuron and the associated neuron.

15. An artificial neural network as claimed in claim 6, wherein the associating neuron represents the sum of what is represented by the initiating neuron and the associated neuron.

16. An artificial, neural network as claimed in claim 6, wherein once the associating neuron represents a result, the result need not be created in another neuron.

17. An artificial neural network as claimed in claim 1, wherein the plurality of elemental neurons is configured to: receive all input to the artificial neural network, and provide all output from the artificial neural network.

18. An artificial neural network as claimed in claim 1, wherein all neurons represent at least one of: value, information and pattern; and processing is at least one of: associating neurons, expressing the pair of neurons with which a structural neuron associates, and expressing the value information or pattern represented by elemental neurons.

19. An artificial neural network as claimed in claim 1, wherein a level of the neural network is a deeper level within the neural network structure if, during expression, more steps are required to express the elemental neurons that it represents.

20. An artificial neural network as claimed in claim 17, wherein associating a pair of neurons is learning, and expressing a pair of neurons is expression.

21. An artificial neural network as claimed in claim 19, wherein the artificial neural network is bi-directional with a forward mode being learning, and a reverse mode being expression.

22. An artificial neural network as claimed in claim 1, wherein the artificial neural network stores associations and not input data and represents patterns within patterns of associations.

23. An artificial neural network as claimed in claim 1, wherein each elemental neuron is selected from the group consisting of: a sensor neuron and a motor neuron.

24. An artificial neural network as claimed in claim 1, wherein each structural neuron represents a plurality of elemental neurons.

25. An artificial neural network as claimed in claim 1, wherein each of the plurality of neurons is able to be expressed.

26. An artificial neural network as claimed in claim 1, wherein the number of elemental neurons and structural neurons required for the memory is determined by the nature of the input to be processed.

27. An artificial neural network as claimed in claim 1, wherein the memory is to store a frequency of received inputs.

28. An artificial neural network as claimed in claim 6, wherein each neuron is a node in the array, each node having a plurality of pointers.

29. An artificial neural network as claimed in claim 28, wherein the plurality of pointers comprises two pointers for providing expression and further pointers to represent associations.

30. An artificial neural network as claimed in claim 28, wherein each pointer in each node contains at least one selected from the group consisting of: an address of another neuron, an elemental value for an elemental neuron, and a frequency count.

31. An artificial neural network as claimed in claim 28, wherein the number of pointers depends on a function being performed by the artificial neural network, the number of pointers for each neuron being at least two.

32. An artificial neural network as claimed in claim 28, wherein a function of each pointer to a neuron is selected from the group consisting of: initiating, associating, successor, next successor of the initiating neuron, precessor, and next precessor of the associating neuron.

33. An artificial neural network as claimed in claim 28, wherein at least one pointer for an elemental neuron represents elemental values.

34. An artificial neural network as claimed in claim 28, wherein all neurons are a fixed length addressable node in the array.

35. An artificial neural network as claimed in claim 1, wherein the artificial neural network is used for at least one selected from the group consisting of: monitoring and predicting stock price movements, Internet surveillance, Internet security, computer virus detection, computer spam detection, phrases in speech and text, clauses in speech and text, plagiarism detection, bioinformatics, vision recognition, semantic analysis, representation of data ontologies, robotics, and data compression.

36. An artificial neural network comprising: (a) a plurality of neurons, (b) each of the plurality of neurons being a processor with memory and being in an array; (c) the plurality of neurons comprising a plurality of elemental neurons and a plurality of structural neurons; (d) all elemental and structural neurons being configured to be associated with others of the elemental and structural neurons via active connections; (e) each elemental neuron being configured to: (i) represent a unique value able to be input into the artificial neural network system, the unique value being one of: a stimulus, an event, events, a sequence in a pattern or sequence of events; and (ii) express that unique value as an output; and (f) each structural neuron being configured to receive input from a pair of neurons with which it is associating, the pair of neurons being selected from the group consisting of: both elemental neurons, both structural neurons, one structural and one elemental neuron, and one elemental neuron and one structural neuron.

37. An artificial neural network as claimed in claim 36, wherein the plurality of elemental neurons are in a root level of the neural network, and each elemental neuron represents a unique value, the unique value being at least one selected from the group consisting of: a stimulus, an event, events, a sequence in a pattern, a sequence of events, an elemental stimulus, a defined elemental pattern, a defined elemental data element, a basic input stimulus, and an output stimulus of information being processed.

38. An artificial neural network as claimed in claim 36, wherein each elemental neuron is selected from the group consisting of: a sensor neuron and a motor neuron.

39. An artificial neural network as claimed in claim 36, wherein all neurons represent at least one of: value, information and pattern; and processing is at least one of: associating neurons, expressing the pair of neurons with which a structural neuron associates, and expressing the value information or pattern represented by elemental neurons.

40. An artificial neural network as claimed in claim 36, wherein neuron associations are represented in a plurality of deeper neural levels; the number of levels in the plurality of deeper levels being determined by the extent of the pattern to be processed or expressed, where a structural neuron represents a plurality of elemental neurons.

41. An artificial neural network as claimed in claim 40, wherein the number of elemental neurons and structural neurons required for the memory is determined by the nature of the input to be processed.

42. An artificial neural network as claimed in claim 36, wherein any one of the plurality of neurons is able to associate with a neuron in the plurality of neurons via the active connections to a further neuron in the plurality of neurons, the further neuron being one of the plurality of structural neurons.

43. An artificial neural network as claimed in claim 36, wherein the artificial neural network is bi-directional with a forward mode being learning, and a reverse mode being expression.

44. An artificial neural network as claimed in claim 36, wherein the artificial neural network stores associations and not input data and represents patterns within patterns of associations.

45. An artificial neural network as claimed in claim 36, wherein each of the plurality of neurons is able to be expressed.

46. An artificial neural network comprising: (a) a plurality of neurons, (b) each of the plurality of neurons being a processor with memory and being in an array; (c) the plurality of neurons comprising a plurality of elemental neurons and a plurality of structural neurons; (d) all elemental and structural neurons being configured to be associated with others of the elemental and structural neurons via active connections; (e) each elemental neuron being configured to: (i) represent a unique value able to be input into the artificial neural network system, the unique value being at least one selected from the group consisting of: a stimulus, an event, events, a sequence in a pattern, a sequence of events, an elemental stimulus, a defined elemental pattern, a defined elemental data element, a basic input stimulus, and an output stimulus of information being processed; and (ii) express that unique value as an output; (f) all of the plurality of structural neurons being able to be expressed in terms of the elemental neurons from which they were derived or represent.

47. An artificial neural network as claimed in claim 46, wherein the artificial neural network is bi-directional with a forward mode being learning, and a reverse mode being expression.

48. An artificial neural network as claimed in claim 46, wherein the artificial neural network stores associations and not input data and recognizes patterns within patterns of associations.

49. An artificial neural network as claimed in claim 46, wherein the neural network is bi-directional with all elemental neurons being able to express their elemental values, and all structural neurons being able to express a pair of neurons with which they associate.

50. An artificial neural network comprising: (a) a plurality of neurons, (b) each of the plurality of neurons being a processor with memory and being a node in an array; (c) the plurality of neurons comprising a plurality of elemental neurons and a plurality of structural neurons; (d) all elemental and structural neurons being configured to be associated with others of the elemental and structural neurons via connections; (e) the artificial neural network being bi-directional and being able to operate in a forward mode where structural neurons are created from input events from the elemental neurons, and in a reverse mode where input events are expressed by the elemental neurons.

51. An artificial neural network as claimed in claim 50, wherein the forward mode is learning, and the reverse direction is expression.

52. An artificial neural network as claimed in claim 50, wherein the neural network stores associations and not input data.

53. An artificial neural network as claimed in claim 50, wherein the neural network represents and recognizes patterns within patterns of associations.

54. A neuronal assembly of an artificial neural network, the neuronal assembly comprising an initiating neuron, an associated neuron, and an associating neuron operatively connected with the initiating neuron and the associated neuron; the associating neuron representing the sum of what is represented by the initiating neuron and the associated neuron, and once the associating neuron represents a result, the result need not be created in another neuron.

55. A neuronal assembly as claimed in claim 54, wherein the artificial neural network is as claimed in claim 1.

56. A neuronal assembly as claimed in claim 54, wherein when an initiating neuron receives input and an associated neuron receives input, the inputs are transmitted to all associating neurons of the initiating neuron and the associated neuron, the associating neuron of both the initiating neuron and the associated neuron then being activated and being able to produce output.

57. A neuronal assembly as claimed in claim 56, wherein the associated neuron is able to produce output in a manner selected from the group consisting of: at the same time as the initiating neuron, and after the initiating neuron.

58. A neuronal assembly as claimed in claim 56, wherein the activation of or producing of output from of the initiating neuron and the associated neuron is based on proximal characteristics.

59. A neuronal assembly as claimed in claim 58, wherein the proximal activation of or producing of output from the initiating neuron and the associated neuron causes at least one selected from the group consisting of: the creation of a new associating neuron if none exists together with new connections between the initiating neuron and the new associating neuron and between the associated neuron and the new associating neuron, the strengthening of existing connections between the initiating neuron and the associating neuron and between the associated neuron and the associating neuron, and strengthening of the associating neuron.

60. A method for creating an association of neurons in an artificial neural network having a plurality of neurons, one of the plurality of neurons being an initiating neuron, another of the plurality of neurons being an associated neuron operatively connected with the initiating neuron, and a further neuron of the plurality of neurons being an associating neuron operatively connected with the initiating neuron and the associated neuron; the method comprising: activating or producing an output from the initiating neuron to potentiate the associating neuron; and activating or producing an output from the associated neuron to potentiate and activate the associating neuron, the associating neuron then being activated and able to produce an output; the associating neuron representing the sum of what is represented by the initiating neuron and the associated neuron, and once the associating neuron represents a result, the result need not be created in another neuron.

61. A method as claimed in claim 60, wherein the associating neuron is expressed by the associating neuron expressing the initiating neuron and the associated neuron.

62. A method as claimed in claim 60, wherein the associated neuron is activated or produces an output in a manner selected from the group consisting of: at the same time as the initiating neuron, and after the initiating neuron.

63. A method as claimed in claim 60, wherein the activation of or producing an output by the initiating neuron and the activation of or producing an output by the associated neuron is based on proximal characteristics.

64. A method as claimed in claim 63, wherein the proximal activation of or producing output from the initiating neuron and the associated neuron causes at least one selected from the group consisting of: the creation of a new associating neuron if none exists together with new connections between the initiating neuron and the new associating neuron and between the associated neuron and the new associating neuron, the strengthening of existing connections between the initiating neuron and the associating neuron and between the associated neuron and the associating neuron, and strengthening of the associating neuron.

65. A method as claimed in claim 60, wherein the associating neuron represents the sum of what is learnt from the initiating neuron and the associated neuron.

66. A method as claimed in claim 64, wherein once the new associating neuron is created to represent a result, the result need not be created in another neuron.

67. A method as claimed in claim 60, wherein once the associating neuron represents a result, the result need not be created in another neuron.

68. A method of constructing an artificial neural network comprising a plurality of neurons, the plurality of neurons comprising a plurality of elemental neurons and a plurality of structural neurons, the method comprising: defining unique events the elemental neurons will represent; creating a required number of elemental neurons for the total number of unique values to be represented for all defined events; the unique value being at least one selected from the group consisting of: a stimulus, an event, events, a sequence in a pattern, a sequence of events, an elemental stimulus, a defined elemental pattern, a defined elemental data element, a basic input stimulus, and an output stimulus of information being processed; the plurality of elemental neurons receiving all input to the artificial neural network, all output from the artificial neural network being from the plurality of elemental neurons; creating the plurality of structural neurons, each of the structural neurons being created by the association of a pair of the plurality of neurons; each of the plurality of structural neurons being configured to produce an output on activation by the pair of neurons, the pair of neurons comprising an initiating neuron and an associated neuron; the association of the plurality of neurons being based on proximal characteristics; and each of the plurality of structural neurons being configured to express the pair of neurons.

69. A method as claimed in claim 68, wherein any one of the plurality of neurons is able to associate with a neuron in the plurality of neurons via active connections to a further neuron in the plurality of neurons, the further neuron being one of the plurality of structural neurons.

70. A method as claimed in claim 68 wherein all elemental neurons are able to express their elemental values, and all structural neurons are able to express a the pair of neurons with which they associate.

71. A method as claimed in claim 70, wherein the pair of neurons is selected from the group consisting of: an elemental neuron with an elemental neuron, an elemental neuron with a structural neuron, a structural neuron with an elemental neuron, a structural neuron with a structural neuron.

72. A method as claimed in claim 68, wherein each of the plurality of neurons is one or more selected from the group consisting of: initiating neuron, associating neuron, and associating neuron; an initiating neuron being associated with an associated neuron via connections to the associating neuron.

73. A method as claimed in claim 72, wherein the initiating neuron, the associated neuron and the associating neuron are connected based on proximal characteristics selected from the group consisting of: temporal, spatial, intensity, magnitude and relative position of the input being processed.

74. A method as claimed in claims 68, wherein a level of the neural network is a deeper level within the artificial neural network if, during recollection, more steps are required to express the elemental neurons.

75. A method as claimed in claim 72, wherein when an initiating neuron receives input and an associated neuron receives input, the inputs are transmitted to all associating neurons of the initiating neuron and the associated neuron respectively, the associating neuron of both the initiating neuron and the associated neuron then being activated and being able to produce output.

76. A method as claimed in claim 75, wherein the associated neuron is activated or an output produced in a manner selected from the group consisting of: at the same time as the initiating neuron, and after the initiating neuron.

77. A method as claimed in claim 75, wherein the activation of or producing an output from the initiating neuron and the activation of or producing an output from the associated neuron is based on proximal characteristics.

78. A method as claimed in claim 77, wherein the proximal activation of or producing an output from the initiating neuron and the associated neuron causes at least one selected from the group consisting of: the creation of a new associating neuron including new synaptic connections between the initiating neuron and the new associating neuron and between the associated neuron and the new associating neuron, the strengthening of existing synaptic connections between the initiating neuron and the associating neuron and between the associated neuron and the associating neuron, and the strengthening of the associating neuron.

79. A method as claimed in claim 78, wherein the strengthening is by maintaining a frequency count of how often the associating neuron receives input from the initiating neuron and the associated neuron.

80. A method as claimed in claim 72, wherein the associating neuron represents the sum of what is represented by the initiating neuron and the associated neuron.

81. A method as claimed in claim 68, wherein a memory represents a plurality of elemental stimuli, and each elemental stimulus is represented directly by an elemental neuron.

82. A method as claimed in claim 68, wherein the number of elemental neurons required to represent the memory is determined by the nature of the input being processed.

83. A method as claimed in claim 68, wherein each neuron is represented by an addressable node in an array, each node having a plurality of pointers.

84. A method as claimed in claim 68, wherein the plurality of elemental neurons is in a root level of the neural network.

85. A method as claimed in claim 72, wherein each initiating neuron is able to associate with a plurality of associated neurons to form a plurality of pairs of neurons.

86. A method as claimed in claim 72, wherein each associated neuron is able to associate with a plurality of initiating neurons to form a plurality of pairs of neurons.

87. A method as claimed in claim 72, wherein once the associating neuron represents a result, the result need not be created in another neuron.

88. A method as claimed in claim 68, wherein the plurality of elemental neurons is configured to: receive all, input to the artificial neural network, and provide all output from the artificial neural network.

89. A method as claimed in claim 68, wherein all neurons represent at least one of: value, information and pattern; and processing is at least one of: associating neurons, expressing the pair of neurons with which a structural neuron associates, and expressing the value information or pattern represented by elemental neurons.

90. A method as claimed in claim 68, wherein associating a pair of neurons is learning, and expressing a pair of neurons is expression.

91. A method as claimed in claim 90, wherein the artificial neural network is bi-directional with a forward mode being learning, and a reverse mode being expression.

92. A method as claimed in claim 68, wherein the artificial neural network stores associations and not input data and represents patterns within patterns of associations.

93. A method as claimed in claim 68, wherein each elemental neuron is selected from the group consisting of: a sensor neuron and a motor neuron.

94. A method as claimed in claim 68, wherein each structural neuron represents a plurality of elemental neurons.

95. A method as claimed in claim 68, wherein each of the plurality of neurons is able to be expressed.

96. A method as claimed in claim 83, wherein the plurality of pointers comprises two pointers for providing expression and further pointers to represent associations.

97. A method as claimed in claim 96, wherein each pointer in each node contains at least one selected from the group consisting of: an address of another neuron, an elemental value for an elemental neuron, and a frequency count.

98. A method as claimed in claim 96, wherein the number of pointers depends on a function being performed by the artificial neural network, the number of pointers for each neuron being at least two.

99. A method as claimed in claim 96, wherein a function of each pointer to a neuron is selected from the group consisting of: initiating, associating, successor, next successor of the initiating neuron, precessor, and next precessor of the associating neuron.

100. A method as claimed in claim 96, wherein at least one pointer for an elemental neuron represents elemental values.

101. A method as claimed in claim 96, wherein all neurons are a fixed length addressable node in the array.

102. A method as claimed in claim 68, wherein the artificial neural network is used for at least one selected from the group consisting of: monitoring and predicting stock price movements, Internet surveillance, Internet security, computer virus detection, computer spam detection, phrases in speech and text, clauses in speech and text, plagiarism detection, bioinformatics, vision recognition, semantic analysis, representation of data ontologies, robotics, and data compression.

103. A computer usable medium comprising a computer program code configured to cause one or more processors and/or memory to execute one or more functions to perform the method claimed in claim 60.

104. A computer usable medium comprising a computer program code configured to cause one or more processors and/or memory to execute one or more functions to perform the method claimed in claim 68.

Description

FIELD OF INVENTION

This invention relates to neural networks and particularly, though not exclusively, to neural networks based on one or more characteristics including temporal, spatial, intensity, magnitude, and relative position; and may be used for one or more of: learning, knowledge acquisition, discovery, data mining and expression.

BACKGROUND OF THE INVENTION

Existing neural networks are typically based on a single interpretation of Hebbian learning. This basic, Hebbian concept is often stated as "Neurons that fire together wire together". The defacto interpretation is that wiring together is effected via the synapse that connects the two neurons together. The strength of the connecting synapse is modified or weighted to reflect the importance/probability of the presynaptic neuron firing concurrently with the postsynaptic neuron, or vice versa.

Using the concept, neural networks have been developed that associate a number of input neurons to a number of output neurons via synapses. The input neurons define the input states; and the output neurons define the desired output states.

Thus nearly all existing neural networks are based on the concept of three layers: an input neuron layer, a hidden neuron layer, and an output neuron layer. FIG. 1 and FIG. 2 are illustrations of existing neural networks.

Training of such neural networks is accomplished, in its most basic form, by applying a specific input state to all the input neurons, selecting a specific output neuron to represent that input state, and adjusting the synaptic strengths or weights in the hidden layer. That is, training is conducted assuming knowledge of the desired output. After training has been completed, the application of different input states will result in different output neurons being activated with different levels of confidence. Thus recognition of an input event depends on how close the original training states match the current input state.

Such neural networks typically require extensive, repetitive training with hundreds or thousands of different input states, depending on the number of desired output neurons and the accuracy of the desired result. This results in practical networks of the order of only 10,000 input and output neurons with as many as 10 million interconnecting synapses or weights representing synapses (current existing neural networks are very small in size as compared to the capacity of the human brain which has 10.sup.12 neurons, and 10.sup.16 synaptic connections).

Furthermore, existing networks are trained on the basis of generating predefined output neurons, and can subsequently recognise inputs that closely resemble the training sets used for input. Existing neural networks are not capable of independent learning as they are trained using prior assumptions--the desired goals are represented by the output neurons. Existing neural networks are not capable of expressing or recollecting an input state based on the stimulus of any output neuron in the output layer.

Existing neural networks are trained on the basis of applying independent input states, to the network, in which the order of training is typically insignificant. On completion of extensive, repetitive training, the output neurons are not significantly dependent on the order in which input states are applied to the network. Existing neural networks provide outputs that are based entirely on the current input state. The order in which input states are applied has no bearing on the network's ability to recognise them.

Existing neural networks may have some or all of the following shortcomings: 1. they require prior training, based on predetermined or desired output goals--they do not learn; 2. they can only recognise input states (objects) similar to the input states for which they have been trained; 3. they are highly computational, and therefore slow; 4. they are computationally restricted to represent only a relatively small number of neurons; 6. they need retraining if they are to recognise different objects; 7. they cannot express or recall an input object by applying a stimulus to the output neurons; 8. they are based on concurrent stimuli of all input neurons; 9. they are not creative and they cannot express or recollect events; they can only identify/recognise events for which they have been trained; 10. they assume neurons that fire concurrently or in quick succession, are linked synaptically but do not distinguish one from the other or the order of neuron firing; and 11. each hidden layer neuron can receive inputs from multiple input neurons concurrently.

SUMMARY OF THE INVENTION

In accordance with one aspect, there is provided a neural network comprising a plurality of neurons in which any one of the plurality of neurons is able to associate or associate with itself or any other neuron in the plurality of neurons via active connections to a further neuron in the plurality of neurons. This process is referred to as learning.

In accordance with a second aspect there is provided a neural network comprising a plurality of elemental neurons, and a plurality of structural neurons for representing associations between any pair of neurons, the pair of neurons being selected from the group consisting of: both elemental neurons, both structural neurons, one structural and one elemental neuron, and one elemental neuron and one structural neuron.

Each structural neuron may represent the combined information or memory represented by a pair of neurons. The process of recalling the pair of neurons that were combined to form a structural neuron is referred to as expression. Each structural neuron may receive input from only two neurons.

The plurality of elemental neurons may be represented in the root level of the neural network structure; and each elemental neuron may represent at least one of: an elemental stimulus, a defined elemental pattern, and a defined elemental data element. Each elemental neuron may represent one or both of: basic input stimuli and output stimuli of information being processed. Each elemental neuron may be an equivalent of a neuron in a brain, the neuron in the brain being selected from the group consisting of a sensor neuron, a motor neuron, an intracortical neuron and an intercortical neuron. The information represented by a neuron may be memory, and the processing may be learning or expression.

The plurality of neuron associations may be represented in a plurality of deeper neural levels. The number of levels in the plurality of deeper levels may be determined by the extent of the memory or pattern to be processed or expressed, where a memory represents a plurality of elemental neurons. The number of elemental neurons and structural neurons required to represent the memory may be determined by the nature of the memory to be processed.

In accordance with a third aspect there is provided a neural network comprising a plurality of neurons linked by associations, all associations of neurons in a level of the neural network that is the same or deeper being able to be expressed.

A fourth aspect provides a neural network comprising a plurality of neurons, each neuron being represented by a unique addressable node in an array.

A fifth aspect provides a neural network comprising a plurality of neurons, each neuron being represented in its entirety by a single node in an array.

A sixth aspect is a neural network comprising a plurality of nodes in an array, each node in the array comprising pointers. Each pointer is a data element of the node that represents a unique address of a specific node in the array, each address representing a neuron of a plurality of neurons. Each pointer represents a synaptic connection.

A seventh aspect there is provided a neural network comprising a plurality of neurons in an array, there being pointers in each node of the array for providing expression and for learning of memories.

A penultimate aspect provides a neural network comprising a plurality of neurons, each neuron being represented by a node in an array, each node having a plurality of pointers, each pointer in each node having a specific and unique function. Except where a pointer may represent the value of an elemental stimulus in the elemental or root level neurons, each pointer contains an address of another neuron. The number of pointers required may depend on the functions being performed by the neural network. For a neural network performing learning and expression functions, the number of pointers needed will be at least four.

In this manner each neuron in the plurality of neurons may be represented by a node of the same size in the array representing the plurality of neurons, each node containing a fixed number of pointers.

Nodes in an array used to represent neurons may also maintain additional data elements other than pointers pertaining to the characteristics of each neuron. Data elements may be defined to represent the frequency of each neuron's activation, the strength of its associations, and so forth.

The present invention also extends to a computer usable medium comprising a computer program code configured to cause one or more processors to execute one or more functions to perform the methods described above.

In a final aspect there is provided a neural network wherein the neural network is bi-directional and is enabled to operate in a forward direction where nodes are derived or created from input, and in a reverse direction where Input is derived from nodes. The forward direction is learning and the reverse direction is expression.

The neural network may be used for one or more of: monitoring and predicting stock price movements, Internet surveillance, Internet security, computer virus and spam detection, data compression, phrase recognition in speech and text, clauses in speech and text, plagiarism detection, bioinformatics, vision recognition, semantic analysis and representation of ontologies, and robotics.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the invention may be readily understood and put into practical effect there shall now be described by way of non-limitative example only preferred embodiments of the present invention, the description being with reference to the accompanying illustrative drawings in which:

FIG. 1 is an illustration of an existing neural network structure;

FIG. 2 is a further illustration of an existing neural network structure;

FIG. 3 is an illustration of an existing relationship between two neurons and a synapse;

FIG. 4 is an illustration of the relationship between three neurons according to the present invention;

FIG. 5 is a flow chart of the process flow of the present invention;

FIG. 6 is an illustration of the relationship between neurons and pointers/links;

FIG. 7 is an illustration of a sequence of events to illustrate association;

FIG. 8 is a flow chart for the learning process;

FIG. 9 is a flow chart for the matching process;

FIG. 10 is a flow chart for creating neurons during the learning process; and

FIG. 11 is a flow chart of the process for expressing of neurons.

DESCRIPTION OF PREFERRED EMBODIMENTS

According to a preferred aspect the present invention provides neural networks, and a method for constructing such neural networks via neuron associations, that are based on characteristics that Include at least one of temporal, spatial, intensity, magnitude, and relative position, for the formation of memories, that consist of one or both of either input stimuli (represented by elemental neurons) or output actions (represented by elemental neurons) in a natural manner.

It also provides for either or both of memory recollection and memory expression of one or more of the memories represented by structural neurons, which represent multiple elemental stimuli. The neural network allows for the potential expression of new actions or ideas other than what It has learnt and in such a manner may exhibit creativity. Input stimuli may include one or more of: audio, visual, tactile, and so forth. Output stimuli may include one or more of: movement, motion, speech, and so forth, each defined by appropriate elemental neurons.

Existing neural networks are based on the assumption that concurrently activating two neurons (neurons B and C) creates an active synaptic connection between them, or strengthens existing synaptic connections. This is illustrated in FIG. 3 where there are two neurons and one synapse.

Accordingly to one aspect of the present invention, stimulating or activating two neurons creates an association between them via another third neuron; the associating neuron. This is illustrated in FIG. 4 where there are three neurons 41, 42 and 43 and two synapses 44 and 45. For convenience, this basic neural structure will be called a "neuronal assembly" throughout this specification. Neurons 41, 42 and 43 may be associated together based on proximal characteristics, Including at least one of temporal, spatial, intensity, magnitude and relative position. Neuron 43 will be at a deeper level within the neural structure than both of neurons 41, 42. Neurons 41,42 may be in the same level, or may be in different levels of the neural structure. The depth or level of a neuron in a neural network structure is based on the number of steps required to express the elemental neurons that it represents.

The neural structure comprises neurons, where each neuron represents a memory of data, events, objects, concepts or actions. The type of information represented by each neuron can vary, and is dependent on the elemental neurons (representing sensor and/or motor neuron stimuli) from which the neural network is constructed. Elemental stimuli are only represented in the elemental neurons maintained at the root levels of every neural network structure. Deeper or subsequent level neurons (structural neurons) only represent the association of other neurons and do not in themselves store sensor, motor or elemental stimulus values.

Each neuron in the neural structure may represent the association of only two neurons, one an initiating neuron and the other an associated neuron, although each neuron may participate as an initiating neuron and/or as an associated neuron in an unlimited number of associations, via associating neurons. An initiating neuron 41 can have any number of successor neurons such as neuron 43, where a successor neuron to neuron 41 is a associating neuron (43) that has neuron 41 as its initiating neuron. Another neuron 42 can have any number of precessor neurons, where a precessor neuron to neuron 42 is a associating neuron (43) that has neuron 42 as its associated neuron. Thus, neuron 43 can be referred to as an associating neuron, or a successor neuron to neuron 41, or as a precessor neuron to neuron 42.

The association is by one of the elemental neurons 41, 42 being an initiating neuron and one is an associated neuron. Assuming neuron 41 is the initiating neuron and thus neuron 42 is the associated neuron, when neuron 41 Is activated or fires associating neuron 43 is potentiated. At the same time as, or subsequent to neuron 41 firing neuron 42 is activated or fires and also potentiates associating neuron 43. Neuron 43 is then considered activated. If the associating neuron 43 was non existent (that is there existed no neuron associating the initiating neuron 41 and the associated neuron 42) then it is created and may be then activated, otherwise it is only activated. The proximal activation or firing of neurons 41 and 42 causes the activation of associating neuron 43, and the creation of active connections, or the strengthening of existing synaptic connections, between neurons 41 and 43 and neurons 42 and 43. The associating neuron 43 represents the sum of what is learnt from the other two neurons 41, 42. This sum may include one or more of a memory trace, a combination of the experience of the two, a sequence of events, a proximity of events and so forth. Once a associating neuron is activated or created to represent a desired memory or events, the desired mem