Page 1
Lecture 7Lecture 7 Artificial neural networks:Artificial neural networks: Supervised learningSupervised learning Introduction, or how the brain worksIntroduction, or how the brain works The neuron as a simple computing elementThe neuron as a simple computing element The perceptronThe perceptron Multilayer neural networksMultilayer neural networks Accelerated learning in multilayer neural networksAccelerated learning in multilayer neural networks The Hopfield networkThe Hopfield network Bidirectional associative memories (BAM)Bidirectional associative memories (BAM) SummarySummary Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 1
Page 2
Introduction, or howIntroduction, or how thethe brainbrain worksworks Machine learning involves adaptive mechanismsMachine learning involves adaptive mechanisms that enable computers to learn from experience,that enable computers to learn from experience, learn by example and learn by analogy. Learninglearn by example and learn by analogy. Learning capabilities can improve the performance of ancapabilities can improve the performance of an intelligent system over time. The most popularintelligent system over time. The most popular approaches to machine learning areapproaches to machine learning are artificialartificial neural networksneural networks andand genetic algorithmsgenetic algorithms. This. This lecture is dedicated to neural networks.lecture is dedicated to neural networks. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 2
Page 3
AA neural networkneural network can be defined as a model ofcan be defined as a model of reasoning based on the human brain. The brainreasoning based on the human brain. The brain consists of a densely interconnected set of nerveconsists of a densely interconnected set of nerve cells, or basic information-processing units, calledcells, or basic information-processing units, called neuronsneurons.. The human brain incorporates nearly 10 billionThe human brain incorporates nearly 10 billion neurons and 60 trillion connections,neurons and 60 trillion connections, synapsessynapses,, between them. By using multiple neuronsbetween them. By using multiple neurons simultaneously, the brain can perform its functionssimultaneously, the brain can perform its functions much faster than the fastest computers in existencemuch faster than the fastest computers in existence today.today. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 3
Page 4
Each neuronEach neuron has a very simple structure, but anhas a very simple structure, but an army of such elements constitutes a tremendousarmy of such elements constitutes a tremendous processing power.processing power. A neuronA neuron consists of a cell body,consists of a cell body, somasoma, a number of, a number of fibers calledfibers called dendritesdendrites, and a single long fiber, and a single long fiber called thecalled the axonaxon.. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 4
Page 5
Biological neural networkBiological neural network Synapse Synapse Dendrites Axon Axon Soma Soma Dendrites Synapse Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 5
Page 6
Our brain can be considered as a highly complex,Our brain can be considered as a highly complex, non-linear and parallel information-processingnon-linear and parallel information-processing system.system. Information is stored and processed in a neuralInformation is stored and processed in a neural network simultaneously throughout the wholenetwork simultaneously throughout the whole network, rather than at specific locations. In othernetwork, rather than at specific locations. In other words, in neural networks, both data and itswords, in neural networks, both data and its processing areprocessing are globalglobal rather than local.rather than local. Learning is a fundamental and essentialLearning is a fundamental and essential characteristic of biological neural networks. Thecharacteristic of biological neural networks. The ease with which they can learn led to attempts toease with which they can learn led to attempts to emulate a biological neural network in a computer.emulate a biological neural network in a computer. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 6
Page 7
An artificial neural network consists of a number ofAn artificial neural network consists of a number of very simple processors, also calledvery simple processors, also called neuronsneurons, which, which are analogous to the biological neurons in the brain.are analogous to the biological neurons in the brain. The neurons are connected by weighted linksThe neurons are connected by weighted links passing signals from onepassing signals from one neuronneuron to another.to another. The output signal is transmitted through theThe output signal is transmitted through the neuron’s outgoing connection. The outgoingneuron’s outgoing connection. The outgoing connection splits into a number of branches thatconnection splits into a number of branches that transmit the same signal. The outgoing branchestransmit the same signal. The outgoing branches terminate at the incoming connections of otherterminate at the incoming connections of other neurons in the network.neurons in the network. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 7
Page 8
Architecture of a typical artificial neural networkArchitecture of a typical artificial neural network S i g n a l s S i g n a l s I n p u t O u t p u t Middle Layer Input Layer Output Layer Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 8
Page 9
Analogy between biological andAnalogy between biological and artificial neuralartificial neural networksnetworks Biological Neural Network Artificial Neural Network Soma Neuron Dendrite Input Axon Output Synapse Weight Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 9
Page 10
The neuronThe neuron as aas a simplesimple computingcomputing elementelement Diagram of a neuronDiagram of a neuron Input Signals Weights Output Signals x1 Y w1 x2 w2 Neuron Y Y wn Y xn Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 10
Page 11
The neuronThe neuron computes the weighted sum of the inputcomputes the weighted sum of the input signals and compares the result with asignals and compares the result with a thresholdthreshold valuevalue,, θθ. If the net input is less than the threshold,. If the net input is less than the threshold, the neuronthe neuron output is –1. But if the net input is greateroutput is –1. But if the net input is greater than or equal to the threshold, the neuronthan or equal to the threshold, the neuron becomesbecomes activated and its output attains a value +1.activated and its output attains a value +1. The neuron uses the following transfer orThe neuron uses the following transfer or activationactivation functionfunction:: ∑n +1, if X ≥ θ X = xiwi Y = i=1 −1, if X < θ This type of activation function is called aThis type of activation function is called a signsign functionfunction.. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 11
Page 12
Activation functions of a neuronActivation functions of a neuron Step function Sign function Sigmoid function Linear function Y Y Y Y +1 +1 +1 +1 0 X 0 X 0 X 0 X -1 -1 -1 -1 step 1, if X ≥0 sign +1,if X ≥0 sigmoid 1 linear Y = Y = Y = Y = X 0,if X <0 −1,if X <0 1+e−X Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 12
Page 13
Can a single neuronCan a single neuron learn a task?learn a task? In 1958,In 1958, Frank RosenblattFrank Rosenblatt introduced a trainingintroduced a training algorithm that provided the first procedure foralgorithm that provided the first procedure for training a simple ANN: atraining a simple ANN: a perceptronperceptron.. The perceptronThe perceptron is the simplest form of a neuralis the simplest form of a neural network. It consists of a single neuronnetwork. It consists of a single neuron withwith adjustableadjustable synaptic weights and asynaptic weights and a hard limiterhard limiter.. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 13
Page 14
Single-layer two-input perceptronSingle-layer two-input perceptron Inputs x1 Linear Hard w Combiner Limiter 1 Output ∑∑∑∑ Y w2 θ x2 Threshold Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 14
Page 15
The PerceptronThe Perceptron The operation of Rosenblatt’s perceptronThe operation of Rosenblatt’s perceptron is basedis based on theon the McCulloch andMcCulloch and Pitts neuronPitts neuron modelmodel. The. The model consists of a linear combinermodel consists of a linear combiner followed by afollowed by a hard limiterhard limiter.. The weighted sum of the inputs is applied to theThe weighted sum of the inputs is applied to the hard limiterhard limiter, which produces an output equal to +1, which produces an output equal to +1 if its input is positive andif its input is positive and −−1 if it is negative.1 if it is negative. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 15
Page 16
The aim of the perceptron is to classify inputs,The aim of the perceptron is to classify inputs, xx ,, xx , . . .,, . . ., xx , into one of two classes, say, into one of two classes, say 11 22 nn AA andand AA .. 11 22 In the case of an elementary perceptron, the n-In the case of an elementary perceptron, the n- dimensional space is divided by adimensional space is divided by a hyperplanehyperplane intointo two decision regions. The hyperplane is defined bytwo decision regions. The hyperplane is defined by thethe linearly separablelinearly separable functionfunction:: n ∑xiwi −θ = 0 i=1 Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 16
Page 17
Linear separability in theLinear separability in the perceptronsperceptrons x2 x2 Class A1 1 2 1 x1 Class A2 x1 2 x1w1 + x2w2 − θ = 0 x x1w1 + x2w2 + x3w3 − θ = 0 3 (a) Two-input perceptron. (b) Three-input perceptron. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 17
Page 18
How does the perceptron learn its classificationHow does the perceptron learn its classification tasks?tasks? This is done by making small adjustments in theThis is done by making small adjustments in the weights to reduce the difference between the actualweights to reduce the difference between the actual and desired outputs of the perceptronand desired outputs of the perceptron. The initial. The initial weights are randomly assigned, usually in the rangeweights are randomly assigned, usually in the range [[−−0.5, 0.5], and then updated to obtain the output0.5, 0.5], and then updated to obtain the output consistent with the training examples.consistent with the training examples. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 18
Page 19
If at iterationIf at iteration pp, the actual output is, the actual output is YY((pp) and the) and the desired output isdesired output is YY ((pp), then the error is given by:), then the error is given by: dd e(p)=Yd(p)−Y(p) wherewhere pp = 1, 2, 3, . . .= 1, 2, 3, . . . IterationIteration pp here refers to thehere refers to the ppth training exampleth training example presented to the perceptron.presented to the perceptron. If the error,If the error, ee((pp), is positive, we need to increase), is positive, we need to increase perceptron outputperceptron output YY((pp), but if it is negative, we), but if it is negative, we need to decreaseneed to decrease YY((pp).). Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 19
Page 20
The perceptron learning ruleThe perceptron learning rule wi(p +1) = wi(p)+α ⋅xi(p)⋅e(p) wherewhere pp = 1, 2, 3, . . .= 1, 2, 3, . . . αα is theis the learning ratelearning rate, a positive constant less than, a positive constant less than unity.unity. The perceptron learning rule was first proposed byThe perceptron learning rule was first proposed by RosenblattRosenblatt in 1960. Using this rule we can derivein 1960. Using this rule we can derive the perceptron training algorithm for classificationthe perceptron training algorithm for classification tasks.tasks. Negnevitsky, Pearson Education, 2002Negnevitsky, Pearson Education, 2002 20
Affichage des 20 premières pages. Téléchargez le document pour le voir en entier.
Commentaires (0)
Aucun commentaire pour le moment. Soyez le premier !