In response to the exponentially increasing need to analyze vast amounts of data, Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition provides scientists with a simple but systematic introduction to neural networks. Beginning with an introductory diOTHER AUERBACH PUBLICATIONSAgent-Based Manufacturing and control Mobile Computing HandbookSystems: New Agile ManufacturingImad Mahgoub and Mohammad IlyasSolutions for Achieving Peak Perfomance ISBN: 0-8493-1971-4Massimo Paolucci and roberto sacileSBN:1-5744-4336-4MPLS for Metropolitan Area NetworksNam-Kee TanCuring the Patch Management Headache ISBN: 0-8493-2212-Xelicia m. nicastroSBN:0-8493-2854-3Multimedia Security HandbookBorko furht and darko KirovskiCyber Crime Investigators Field Guide,SBN:0-8493-27733Second editionBruce MiddletonNetwork Design: Management andsBN:0-8493-2768-7Technical Perspectives, Second EditionTeresa C. PiliourasDisassembly Modeling for Assembly,sBN:0-8493-1608-1Maintenance, Reuse and RecyclingA.J.D. Lambert and Surendra M. GuptaNetwork Security Technologies,SBN:1-5744-4334-8Second editionKwok T FungThe ethical hack:a framework forSBN:0-8493-3027-0Business Value Penetration TestingJames s. tillerOutsourcing Software DevelopmentSBN:0-8493-1609-XOffshore: Making It WorkTandy GoldFundamentals of DsL TechnologylsBN:0-8493-1943-9Philip Golden, Herve Dedieuand Krista JacobsenQuality Management SystemssBN:0-8493-1913-7A Handbook for productDevelopment OrganizationsThe HiPAA Program Reference Handbook Vivek NandaRoss LeolSBN:1-5744-43526SBN:0-84932211-1A Practical Guide to SecurityAligning IT with Corporate Strategy ardImplementing the IT Balanced Scorecard:AssessmentsSudhanshu KairaJessica KeyessBN:0-8493-1706-1SBN:0-8493-2621-4The Real-Time EnterpriseInformation Security FundamentalsDimitris N. ChorafasThomas R. Peltier. Justin PeltierSBN:0-84932777-6and John A BlackleySBN:0-8493-1957-9Software Testing and continuousQuality Improvement, Second EditionInformation Security ManagementWilliam e lewisHandbook fifth edition volume 2lSBN:0-8493-2524-2Harold F. Tipton and micki KrauseSBN:0-8493-32109Supply Chain Architecture:A Blueprint for Networking the FlowIntroduction to Managementof material information and cashof Reverse Logistics and closedWilliam T. WalkerLoop Supply Chain ProcessessBN:1-5744-4357-7Donald F blumbergsBN:15744-4360-7The windows serial portProgramming HandbookMaximizing ROl on Software Development Ying BaiVijay sikkalSBN:0-8493-2213-8SBN:0-8493-2312-6AUERBACH PUBLICATIONSwww.auerbach-publications.corTo Order cal:1-800-272-7737·Fax:1-800-374-3401E-mail:orders@crcpress.comC 2006 by Taylor Francis Giroup. LL(Neural Networks forApplied Sciencesand EngineeringFrom fundamentals toComplex Pattern RecognitionSandhya samarasinghe△Auerbach PublicationsTaylor francis gBoca raton New yorkAuerbach Publications is an imprint of theTaylor francis group an informa businessO 2006 by Taylor Francis Group, LL.(MATLAB is a trademark of The Math Works, Inc. and is used with permission. The Math Worksdoes not warrant the accuracy of the text or exercises in this book. this book's use or discussionof MaTLAB software or related products does not constitute endorsement or sponsorship by ThMathWorks of a particular pedagogical approach or particular use of the MATLAB"softwareAuerbach publicationsTaylor Francis Group6000 Broken Sound Parkway nw, Suite 300Boca raton Fl 33487-2742o 2007 by Taylor Francis Group, LLCAuerbach is an imprint of Taylor Francis Group, an Informa businessNo claim to original U.s. govcrnment worksPrinted in the United States of America on acid-free paper10987654321International Standard Book Number-10: 0-84193-3375-X (Hardcover)International Standard book Number-13: 978-0-8493-3375-0(Hardcover)This book contains information obtained from authentic and highly regarded sources. Reprintedmaterial is quoted with permission, and sources are indicated. a wide variety of references arelisted. Reasonable efforts have been made to publish reliable data and information, but the authorand the publisher cannot assume responsibility for the validity of all materials or for the conse-quences of their useNo part of this book may be reprinted, reproduced, transmitted, or utilized in any form by anyelectronic, mechanical, or other means, now known or hereafter invented, including photocopying,microfilming, and recording, or in any information storage or retrieval system, without writtenpermission from the publishersFor permission to photocopy or use material electronically from this worlease access wwwcopyrightcom(http://www.copyright.com/)orcontacttheCopyrightClearanceCenter,Inc.(ccc)222 Rosewood Drive, Danvers, MA,978-750-8400. CCC is a not-for-profit organization thatprovides licenses and registration for a variety of users, For organizations that have been granted aphotocopy license by the CCC, a separate system of payment has been arrangedTrademark Notice: Product or corporate names may be trademarks or registered trademarks, andare used only for identification and explanation without intent to infringeLibrary of Congress Cataloging-in-Publication DataSamarasinghe, Sandhya.Neural networks for applied sciences and engineering: from fundamentals toomplex pattern recognition/Sandhya Samarasinghe.pIncludes bibliographical references and indexSBN-13:978-0-8493-3375-0ak. Paper)ISBN-10: 0-8493-3375-X(alk. paper)1. Neural networks(Computer science)2. Pattern recognition systems. I TitleQA7687S2552006006.32-dc222006007265Visit the Taylor francis Web site athttp://www.taylorandfrancis.comd the auerbach web site athttp://www.auerbach-publications.comC 200 hy taylor IGiroup. LLCDedicationTO DoMy husbanFor your constant love, support, and encouragementTo do the best i can do in allMy endeavors as aWoman and aScholarO 2006 by Taylor Francis Group, LL.(C 2006 by Taylor Francis Giroup LLcContentsPreface.×iiAcknowledgmentsAbout the author····:·4··;···:····+··*·:····.····:·············.·.;···········XX1 From Data to Models: Complexity and Challengesin Understanding Biological, Ecological, andNatural Systems1.1: Introduction 11.2: Layout of the Book 4References 72 Fundamentals of Neural Networks and Modelsfor Linear Data Analysis中······2.1: Introduction and overview 1 12.2: Neural Networks and Their Capabilities 122.3: Inspirations from Biology 162.4: Modeling Information Processing in Neurons 182.5: Neuron Models and Learning Strategies 192.5.1: Threshold Neuron as a Simple classifier 202.5.2: Learning Models for Neurons and Neural Assemblies 232.5.2.1: Hebbian Learning 232.5.2.2: Unsupervised or Competitive Learning 262.5.2.3: Supervised Learning 262.5.3: Perceptron with Supervised Learning as a Classifier 272.5.3.1: Perceptron Learning Algorithm 282.5.3.2: A Practical Example of Perceptron on a largerRealistic Data Set: Identifying the Originof Fish from the growth-Ring Diameter of Scales 352.5.3.3: Comparison of Perceptron with LinearDiscriminant Function Analysis in Statistics 38O 2006 by Taylor Francis Giroup, LL.(VIl2.5.3.4: Multi-Output Perceptron for MulticategoryClassification 402.5.3.5: Higher-Dimensional Classification Using Perceptron 452.5.3.6: Perceptron Summary 452.5.4: Linear Neuron for Linear Classification and Prediction 462.5.4.1: Learning with the Delta Rule 472.5.4.2: Linear Neuron as a Classifier 512.5.4.3: Classification Properties of a Linear Neuronas a Subset of Predictive Capabilities 532.5.4.4: Example: Linear Neuron as a Predictor 542.5.4.5: A Practical Example of Linear PredictionPredicting the Heat Influx in a Home 612.5.4.6: Comparison of Linear Neuron Model withLinear Regression 622.5.4.7: Example: Multiple Input Linear NeuronModel--Improving the Prediction Accuracof Heat Influx in a Home 632.5.4.8: Comparison of a Multiple-Input Linear Neuronwith Multiple Linear Regression 632.5.4.9: Multiple Linear Neuron Models 642.5.4.10: Comparison of a Multiple Linear neuronNetwork with Canonical Correlation Analysis 652.5.4.11: Linear Neuron and Linear Network Summary 652.6: Summary 66Problems 66References 673 Neural Networks for Nonlinear Pattern Recognition.....693.1: Overview and Introduction 693.1.1: Multilayer Perceptron 713.2: Nonlinear Neurons 723.2.1: Neuron Activation Functions 733.2.1.1: Sigmoid Functions 743.2.1.2: Gaussian Functions 763.2.2: Example: Population Growth Modeling Usinga Nonlinear Neuron 773.2.3: Comparison of Nonlinear Neuron with NonlinearRegression Analysis 803.3: One-Input Multilayer Nonlinear Networks 803.3.1: Processing with a Single nonlinear Hidden Neuron 803.3.2: Examples: Modeling Cyclical Phenomena withMultiple Nonlinear Neurons 863.3.2.1: Example 1: Approximating a Square Wave 863.3.2.2: Example 2: Modeling Seasonal Species Migration 943.4: Two-Input Multilayer Perceptron Network 983.4.1: Processing of Two-Dimensional Inputs byNonlinear neurons 983.4.2: Network Output 102C 2006 by Taylor Francis Giroup. LLcIX3.4.3: Examples: Two-Dimensional Predictionand Classification 1033.4.3.1: Example 1: Two-Dimensional NonlinearFunction Approximation 1033.4.3.2: Example 2: Two-Dimensional NonlinearClassification Model 1053.5: Multidimensional Data Modeling with NonlinearMultilayer Perceptron Networks 1093.6: Summary 110Problems 110References 1124 Learning of Nonlinear Patterns by Neural Networks4.1: Introduction and Overview 1134.2: Supervised Training of Networks for NonlinearPattern Recognition 1144.3 Gradient Descent and Error Minimization 1154.4: Backpropagation Learning 1164.4.1: Example: Backpropagation Training-A Hand Computation 1174.4.1.1: Error Gradient with Respect to OutputNeuron Weights 1204.4.1.2: The Error Gradient with Respect to theHidden-Neuron Weights 1234.4.1.3: Application of Gradient Descent. inBackpropagation Learning 1274.4.1.4 Batch Learning 1284. 4.1.5: Learning Rate and Weight Update 1304.4.1.6: Example-by-Example(Online)Learning 1344.4.1.7: Momentum 1341.4.2: Example: Backpropagation LearningComputer Experiment 1384.4.3: Single-Input Single-Output Network withMultiple hidden Neurons 1414.4.4: Multiple-Input, Multiple-Hidden Neuron, andSingle-Output Network 14124.4.5: Multiple-Input, Multiple-Hidden NeuronMultiple-Output Network 1434.4.6: Example: Backpropagation Learning CaseStudy--Solving a Complex Classification Problen 1154.5: Delta-Bar-Delta Learning(Adaptive Learning Rate) Method 1524.5.1: Example: Network Training with Delta-Bar-Delta-A Hand computation 1544.5.2: Example: Delta-Bar-Delta with Monentum-A Hand computation 1574.5.3: Network Training with Delta-Bar DeltaA Computer Experiment 1584.5.4: Comparison of Delta-Bar-Delta Method withBackpropagation 159O 2006 by Taylor Francis Giroup, LL.(