|
 |
[an error occurred while processing this directive]
Information flow across the real neural complex, in general, often faces an axonal bottleneck. That is, the input arriving to a neuronal cell is not as information poor as the output. On the input-end neurons are more ajar than on the output-end. Often they receive information at a rate three orders of magnitude higher than they give it off. Also, the neurons always fie together to form a group and, hence, permit an anisotropic proliferation of information flow across the neural complex. These groups were termed as compacta by Lengendy [108]. It is the property of compacta (due to massive interconnections) that, whenever a compactum fires, the knowledge imparted by the firing stimulus is acquired by every neuron of the compactum. Thus, neuronal knowledge proliferates from compactum-to-compactum with a self-augmentation of information associated in the process. That is, each successive structure is able to refine the information available to it to a maximum extent that the information can possibly be refined. Due to this self-augmentation of information, McCulloch and Pitts called the neural nets as networks with circles. Further, routing of information across the interconnected compacta is a goal-pursuit problem. That is, the self-organizational structure of the neural complex permits a goal-directed neural activity and the goal-seeking is again monitored adaptively by the associated feedback (self)-control protocols.
Pertinent to conventional memory, information storage is rather an explicit quantity. For example, in a Random Access Memory (RAM) with M address lines and 1 data line (2M memory locations, each storing 1 bit of information), the storage capacity is 2M bits. That is, the RAM as an entity distinguishes cases (in respect of setting 2M bits independently as 0 or 1) and thereby stores a string of 2M bits. This definition enables the entire contents of the RAM as one entity that encodes a message the logarithm of such different messages measures the information content of the message.
Along similar lines, how can the capacity of a neural network be defined? The information content in a neural net refers to the state-transitions, the associated weights, and thresholds. How many different sets of weights and thresholds can then be distinguished by observing the state-transitions of the network? Abu Mostafa and St. Jacques [98] enumerated such threshold functions involved and showed that there are distinguishable networks of N neurons, where α is asymptotically a constant. In logarithmic measure, the capacity of feedback network is, therefore, proportional to N3 bits. However, the above definition of capacity of a neural net is rather heuristic and not apparent. In order to elucidate this capacity explicitly, one has to encode information in the state-transitions across the interconnected network; and decoding of the message warrants observations concerning which states go to which state. Such a format of depicting information storage is not, however, practical or trivial.
Alternatively, the information storage in feedback networks can be specified vis-a-vis the stable states. For this purpose, an energy function can be defined to indicate the state-transitions leading to the stable state.* The stable states are vectors of bits (that correspond to words in a regular memory), and convergence to such stable states is the inherency of a feedback network functioning as an associative memory. Now, how many such stable states can be stored in a feedback network? The number of stable states can be deduced as , each consisting of N bits (depicting the individual states of N neurons), and is an asymptotic constant. Hence, stable state capacity of a feedback network of N neurons is proportional to N2 bits. The reduction from N3 to N2 arises from the loss of information due to the restricted (selective) observations depicting only those transitions leading to stable states. The selection of stable states used in the computation of the memory storage is a rule-based algorithmic strategy that considers a set of vectors and produces a network in which these vectors are stable states. For example, the Hebbian rule chooses the matrix of weights to be the sum of outer products of the vectors to be stored. The stable-state storage capacity of the neural network is dependent on the type of algorithmic rule chosen. Relevant to Hebbian rule, for example, only (gN/log N) randomly chosen stable states can be stored (where g is an asymptotic constant). The corresponding capacity is then proportional to N2/ log N bits.
*Stable state : It refers to a state where the neuronal state remains unchanged when the update threshold rule is applied.
For a feed-forward network, there is no definition of memory capacity that corresponds to stable-state capacity inasmuch as such networks do not have stable states, but rather input-output relations only.
As discussed in Chapter 4, the neural complex has an architecture with layers of cellular units which are fully interconnected and each unit can be regarded as an information processing element. In reference to this layered architecture, Ackley et al. [57] used the concepts of reverse cross-entropy (RCE) and defined a distance function to depict the deviation of neural statistics in the presence of environmental inputs from that in the absence of such inputs in the entropy plane. Minimization of this distance parameter refers to the process of attaining the objective function. Both the reverse cross-entropy approach due to Ackley et al. and an alternative method based on cross-entropy concepts as proposed by Liou and Lin [58], describe implicitly an entropy-based, information-theoretic approach to analyze a layered framework of neural units.
|