Neural Networks How the brain works Neural Networks Mathematical Representation Structure of Neural Networks Examples: Boolean functions Perceptrons CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 2
Neural Networks How the Brain Works Axonal arborization Axon from another cell Synapse Dendrite Axon Nucleus Synapses Cell body or Soma CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 3
Neural Networks How the Brain Works (2) neuron: fundamental functional unit of all nervous system tissue, soma: cell body, contains cell nucleus Axonal arborization Axon from another cell dendrites: a number of fibers, inputs Synapse Dendrite Axon axon: single long fiber, many branches, Nucleus output Synapses Cell body or Soma synapse: junction of axon and den- drites, each neuron forms synapses with 10 to 100,000 other neurons. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 4
Neural Networks How the Brain Works (3) Signals are propagated from neuron to neuron be a electrochemical reaction: 1. chemical substances are released from the synapses and enter the dendrites, raising or lowering the electrical potential of the cell body; 2. when a the potential reaches a threshold, an electrical pulse or action potential is sent down the axon; 3. the pulse spreads out along the branches of the axon, eventually reaching synapes and releasing transmitters into the bodies of other cells; excitory synapses: increase potential, inhibitory synapses: decrease potential. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 5
Neural Networks How the Brain Works (4) Synaptic connections exhibit plasticity – long term changes in the strength of connections in response to the pattern of stimulation. Neuron also form new connections with other neurons, and sometimes entire collections of neurons migrate. These mechanisms are thought to form the basis of learning. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 6
Neural Networks How the Brain Works (5) Comparing brains with digital computers Computer Human Brain Computational units 1 CPU, 105 gates 1011 neurons Storage units 109 bits RAM, 1010 1011 neurons, 1014 bits disk synapses Cycle time 10 8 sec 10 3 sec Bandwidth 109 bits/sec 1014 bits/sec Neuron updates/sec 105 1014 (Computer resources at the state of 1994) Brain is highly parallel: a huge number of neurons can be updated simultaneously (as opposed to a single CPU computer). Brains are more fault tolerant than computers. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 7
Neural Networks Neural Networks A neural network (NN) is composed of a number of nodes, or units, connected by links. Each link has a numeric weight associated with it. Weights are the primary means of long-term storage in neural networks, and learning usually takes place by updating the weights. Some of the units are connected to the external environment input/output units. Each unit has: 1. a set of input links from other units, 2. a set of output links to other units, 3. a current activation level, 4. and a means of computing the activation level at the next time step, given its input weights. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 8
Neural Networks Mathematical Representation input function: linear, ini ∑ Wjiaj Wi ai j activation function: nonlinear, ai g ini , g can be a step, sign, or sigmoid function. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 9
Neural Networks Mathematical Representation (2) a a a i i i +1 +1 +1 t in ini in i i −1 (a) Step function (b) Sign function (c) Sigmoid function usually, all units use the same function, threshold parameter t can be replaced by static input neuron with a certain weight. in most cases the weights change during the learning process, thresholds and functions remain the same. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 10
Neural Networks Mathematical Notation Notation Meaning ai Activation value of unit i (also the output of the unit) ai Vector of activation values for the inputs to unit i g Activation function g Derivative of the activation function Erri Error (difference between output and target) for unit i Erre Error for example e Ii Activation of a unit i in the input layer I Vector of activations of all input units Ie Vector of inputs for example e ini Weighted sum of inputs to unit i N Total number of units in the network O Activation of the single output unit of a perceptron Oi Activation of a unit i in the output layer O Vector of activations of all units in the output layer t Threshold for a step function T Target (desired) output for a perceptron T Target vector when there are several output units Te Target vector for example e Wj,i Weight on the link from unit j to unit i Wi Weight from unit i to the output in a perceptron Wi Vector of weights leading into unit i W Vector of all weights in the network CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 11
Neural Networks Structure of Neural Networks w I 13 H 1 3 w w 35 14 O 5 w w 23 I H 45 2 w 4 24 multilayer networks: input, hidden, output feed forward: directed links from one layer to the next. a5 g W35 a3 W45 a4 g W35 g W13 a1 W23 a2 W45 g W14 a1 W24 a2 CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 12
Neural Networks Examples: Boolean functions W = 1 W = 1 W = −1 t = 1.5 t = 0.5 t = −0.5 W = 1 W = 1 AND OR NOT 0.5 − 2 +1 +1 1.5 +1 +1 XOR CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 13
Neural Networks Perceptrons Term originally used for multi- layered feed-forward networks of any topology(1950), today synonym for a single-layer, feed-forward network. Output units are independent from each other. Ij Wj,i Oi Ij Wj O Input Output Input Output Units Units Units Unit O Step0 ∑WjIj Perceptron Network Single Perceptron j CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 14
Neural Networks Perceptron (2) What can perceptrons represent? Linear separable functions. E.g. boolean majority, AND, OR, NOT. But not XOR. There is a perceptron algorithm that will learn any linearly separable function, given enough training Training update weights Err T O Wj Wj α Ij Err with learning rate α. Algorithm: repeat update of weights for every example until error is minimized gradient descent. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 15
Neural Networks Perceptron: Learning Method function NEURAL-NETWORK-LEARNING(examples) returns network network a network with randomly assigned weights repeat for each e in examples do O NEURAL-NETWORK-OUTPUT(network,e) T the observed output values from e update the weights in network based on e, O, and T end until all examples correctly predicted or stopping criterion is reached return network CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 16
Neural Networks Limitations of Perceptrons 1 1 0.9 0.9 0.8 0.8 Perceptron Decision tree 0.7 0.7 0.6 Perceptron 0.6 % correct on test set Decision tree % correct on test set 0.5 0.5 0.4 0.4 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 Training set size Training set size Comparing the performance of perceptron and decision trees Left: Perceptrons are better in learning the majority function (of 11 inputs). Right: Decision trees are better at learning the WillWait predicate for the restaurant example. CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 17
Neural Networks Multilayer Feed-Forward Networks Output units O i Wj,i Hidden units aj Wk,j Input units I k CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 18
Neural Networks Multilayer Feed-Forward Networks (2) Multilayer networks are able to learn functions that are not linear separable. Output units Oi In contrast to singlelayer networks, Wj,i multilayer learning algorithms are nei- Hidden units aj ther efficient nor guaranteed to con- Wk,j Input units I verge to a global optimum. k Most popular: Back-Propagation Learning CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 19
Neural Networks Back-Propagation Learning Learning: modify the weights of the network in order to minimize the difference between output and training examples change weights to what amount? Update of weights by layers: weight changes depend on the activation of the units in the layer below. Erri Ti Oi Wji Wji α aj Erri g ini CIS 675 P´eter Moln´ar, 2001 (all rights reserved) Slide 20
显示前 20 页。下载资料查看完整内容。
暂无评论。快来抢沙发吧！
评论 (0)
暂无评论。快来抢沙发吧！