InthispaperItrytodescribeboththeroleofmathematicsinshapingourunderstandingofhowneuralnetworksoperate,andthecuriousnewmathematicalconceptsgeneratedbyourattemptstocaptureneuralnetworksinequations.Mytargetreaderbeingthenon-expert,Iwillpresentabiasedselectionofrelativelysimpleexamples