You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
350 lines
34 KiB
350 lines
34 KiB
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# A simple neural network with one hidden layer in pure Python\n",
|
|
"\n",
|
|
"## Introduction\n",
|
|
"We consider a simple feed-forward neural network with one hidden layer:"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {
|
|
"48b1ed6e-8e2b-4883-82ac-a2bbed6e2885.png": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAASwAAAEsCAYAAAB5fY51AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEKklEQVR4nO2ddVxU6ffHD62EgYWBhd2K2F2767prrmLHioqBrWv3rh1rdxeIioGN3WAuusbaIqlISc39/P7gN8/XkYY7DHc879drXvta5855zjzc+5mnzjkGAEAMwzAKwFDXDjAMw6QVFiyGYRQDCxbDMIqBBYthGMXAgsUwjGJgwWIYRjGwYDEMoxhYsBiGUQwsWAzDKAYWLIZhFAMLFsMwioEFi2EYxcCCxTCMYmDBYhhGMbBgMQyjGFiwGIZRDCxYDMMoBhYshmEUAwsWwzCKgQWLYRjFwILFMIxiYMFiGEYxsGAxDKMYWLAYhlEMLFgMwygGFiyGYRQDCxbDMIqBBYthGMXAgsUwjGJgwWIYRjGwYDEMoxhYsBiGUQwsWAzDKAYWLIZhFAMLFsMwioEFi2EYxcCCxTCMYmDBYhhGMbBgMQyjGFiwGIZRDCxYDMMoBhYshmEUAwsWwzCKgQWLYRjFwILFMIxiMNa1A0zqxMTEkL+/P8XHx5OVlRUVKFCADAwMdO2WXsN9nj3hEVY25dGjRzRq1CiqWbMmWVpaUsmSJalMmTJUqFAhKly4MLVr14527NhB0dHRunZVb+A+z/4YAICunWD+x7Nnz2jYsGF05swZKliwILVr147s7e2pTJkyZGxsTB8/fqR79+7RlStX6OLFi2RtbU3Tp0+nESNGkKEh//5kBO5zBQEm27B69WrkzJkTpUuXxp49exATE5Pi9U+fPsWgQYNARGjUqBHevn2bRZ7qD9znyoIFK5swbdo0EBGGDh2KiIiIdH32woULsLW1RfHixfHixQsteah/cJ8rDxasbMD69etBRFiwYEGGbbx9+xZ2dnYoW7YswsPDZfROP+E+VyYsWDrmv//+g7m5OQYPHpxpW8+ePYO5uTmGDh0qg2f6C/e5cmHB0jHt2rVDyZIlZfuFXrlyJYgIt2/flsWePsJ9rlxYsHTIs2fPQETYsmWLbDbj4+NRokQJ9OnTRzab+gT3ubLhPVkdsmXLFsqbNy85OjrKZtPIyIicnZ1p//79FBYWJptdfYH7XNmwYOmQq1evUqtWrShnzpyy2m3Xrh3FxMSQj4+PrHb1Ae5zZcOCpSMkSaI7d+5Q7dq1ZbddoUIFMjc354fnG7jPlQ8Llo6IiIigiIgIKlGihOy2jYyMyNbWlj58+CC7bSXDfa58WLB0BP4/IkpbAbWGhoaiDSYB7nPlw4KlIywsLMjU1JQCAgJktw2AAgICyNraWnbbSob7XPmwYOkIY2Njqlq1qlbWPF69ekUfP36kmjVrym5byXCfKx8WLB1St25dOn/+PKlUKlntenl5kYGBgVYWl5UO97myYcHSIX369KE3b97QiRMnZLMJgP7++28qV66c7Fv3+oC2+nzNmjX0448/UqFChWSzyySG82HpEADk4OBAxsbGdPXqVTIyMsq0zePHj1O7du2IiChv3rw0ZswYcnFxoVy5cmXatj6gzT4/duwY/fzzzzJ4ySSLbg7YM2quXLkCAwMDLFq0KNO2Pn36hCJFiqB69eooV64ciAhEhDx58mDWrFkIDQ2VwWPlo40+//HHHyFJkgzeMSnBgpUNGDt2LExMTHD8+PEM24iKikLLli2RO3duvH37FvHx8dizZw8qVKigIVwzZ87Ep0+f5HNeoWijzxntw4KVDYiNjUX79u1hYmKCzZs3p/uX+t27d2jcuDFy5syJixcvarwXHx+PvXv3olKlSkK4cufOjenTp+Pjx49yfg1FIUefN2nSJMk+Z7QHC1Y2ITY2FgMGDAARoW3btvj3339T/Ux0dDTWr18PS0tL5MiRAxcuXEj2WpVKhf3796Ny5cpCuHLlyoVp06YhJCREzq+iGDLT57lz54aNjQ2uXLmSBZ4yaliwshlHjhxBkSJFQERo2bIlVq1ahevXryM4OBifP3/Gixcv4O7ujjFjxiBPnjwgIpiZmYGIMHfu3FTtq1QquLm5oUqVKkK4rKysMGXKFAQHB8v+fVQqFaKiohAbGyu7bblIT59bW1uDiFCqVKnvVuh1CQtWNuTLly/YuXMnGjduDBMTEyEsX7+MjY3FKGnz5s0gIpiYmOD+/ftpakOlUuHAgQOoVq2asGlpaYlJkyYhKCgow76rVCqcOXMGTk5OqFmzpob/NjY2aNu2LRYsWICAgIAMt6EN0tLnNjY26NGjB4gIBgYG8PX11bXb3x18rCGbExMTQw8fPqQ3b96QSqUiKysrCg0Npe7du5ORkRGpVCratm0bHTp0iDw8PKhmzZp08+ZNMjExSZN9SZLIw8ODZs2aRffv3yciIktLSxo+fDiNHTuW8ufPn2Zf3dzcaMqUKfTs2TMqV64cNWrUiGrUqEHW1tYUHx9Pz549Ix8fH7pw4QKpVCrq0aMHLVy4kAoWLJihvtEWSfV5tWrVqEiRIkRE1KlTJzp06BA5OjrS3r17deztd4auFZNJP9HR0TA3Nxe//A0aNMCHDx/EdGXWrFnptqlSqXDo0CHUrFlT2LWwsMCECRMQGBiY4mdDQkLQpUsXEBF++eUXXLp0KcVF7JCQECxZsgT58uVD/vz54e7unm5/dcn9+/fFKOuff/7RtTvfFSxYCuWXX34RDw0R4eHDh9i7d6+YLt69ezdDdiVJgoeHB2rVqiWEy9zcHOPGjUtyGhcQEIAqVarA2toarq6u6WorICAAHTt2BBFh9erVGfJXV3Tu3BlEhK5du+rale8KFiyFsmbNGhAR8uXLByLC8OHDIUmSeJCqVauWalHQlJAkCUePHoW9vb2GcI0dOxb+/v4AEkZ69vb2KFSoEB49epThdkaOHAkigpubW4b9zWoePHggfjAePnyoa3e+G1iwFMrLly9BRDA0NBRnqyIjIxEQEID8+fODiDBt2rRMtyNJEo4dOwYHBwchXDlz5sTo0aPh4uICExMT3LlzJ9NtdOnSBdbW1vjw4UOmfc4qfvvtNxARunTpomtXvhtYsBSM+jBowYIFNSrBuLq6gohgZGQEb29vWdqSJAmenp6oW7euEC4DAwPMmTNHFvtBQUEoWLAgunfvLou9rODhw4diSp7W3Vkmc7BgKZixY8eCiMR6U926dcV7Xbt2BRGhSpUqiI6Olq1NSZJw8uRJFCxYEAUKFMjUtPNbVq5cCSMjI7x79042m9pG3c+dOnXStSvfBSxYCubcuXMgIuTPnx9GRkYgIty7dw/A/0YsRITJkyfL2m5ERAQsLCwwY8YMWe1+/vwZFhYWmD17tqx2tYmvr68YZan7ntEenA9LwTRq1IgsLS0pODiYmjdvTkRE69evJyKi/Pnz09q1a4mIaP78+XT79m3Z2r179y5FRkZShw4dZLNJRJQrVy5q1aoVXbp0SVa72qRSpUrUrVs3IiKaNWuWjr3Rf1iwFIypqSm1atWKiIiKFStGRES7du2iiIgIIko44Ni9e3eSJIn69u1L0dHRsrTr7e1NZmZmVLlyZVnsfY29vT15e3srqpjD9OnTycDAgA4dOkT37t3TtTt6DQuWwmnbti0RET158oTKlClD4eHhtG/fPvH+ypUrqVChQvT48WOaOXOmLG2+ffuWSpQokebT9OmhTJkyFBoaSpGRkbLb1hYVK1ak7t27ExHJ1sdM0rBgKZyffvqJiIhu3LhBPXv2JKL/TQuJiPLlyyf+f9GiRXTjxo1Mt6lSqcjY2DjTdpJCnQFU7pzr2mbatGlkaGhIHh4edOfOHV27o7ewYCmcYsWKUdWqVQkA2djYkKmpKXl7e2s8NO3bt6fevXuTJEnUr18/+vLlS6bazJMnDwUHB2fW9SQJCQkhY2NjMjc314p9bVGhQgUeZWUBLFh6gHpaePXqVerUqRMRaY6yiIhWrFhBhQsXpidPntC0adMy1V716tUpMDCQ/Pz8MmUnKe7cuUOVK1fWynRT20yfPp0MDQ3p6NGjXLJeW+h6m5LJPBcuXBDHG86ePStSxYSFhWlcd/ToUXHgMzOJ596+fQsiwu7duzPrugaSJKFs2bL4/fffZbWblfTu3RtEhHbt2unaFb2EBUsPiI2NRa5cuUBEuH79OsqXLw8iwtq1axNd269fPxARypYti8jIyAy3V758eTRs2DCzrmugPlfm5eUlq92s5OnTp+JM3K1bt3Ttjt7BgqUnqIOeZ8yYgSVLloCIUKNGjURpXj59+oSiRYuCiDBq1Kh0tREbG4vNmzejdOnSIjzn8uXLsvgvSRJatWqFihUrKr76TN++fUXaZUZeWLD0BHXWUQcHBwQHB4u0yTdv3kx0raenp5gapqWAQkxMDDZu3IhSpUoJoSpQoABKliyZqZHa12zcuBFEhI0bN2balq559uyZGGXduHFD1+7oFSxYesL79++FCAUEBKBXr14gIgwYMCDJ63///XcQEUqXLo2IiIgkr4mJicH69etRokQJIVSFChXCkiVLEBkZiX///RdmZmbo0qUL4uPjM+z79evXkSNHDrH2tmfPngzbyi6op94//vijrl3RK1iw9IgaNWqAiLBjxw5cvnxZ5LBKqoBqaGgobG1tQUQYMWKExnsxMTFYt24dihcvrpHPfNmyZRqjqQcPHiBXrlwwMDBAhw4d8Pnz53T77OnpCUtLS9StWxcNGzYU7Tk5OSEqKir9nZBNeP78uRhlXb9+Xdfu6A0sWHrE5MmTQURwdHSEJEki/cyqVauSvP706dNCIM6fP4/o6GisWbNGCBkRoXDhwli+fHki8bh//77Iu2VnZwdLS0vY2tri6NGjaVqDCgoKEqO8tm3bIjw8HHFxcZg6daoIJq5atSoeP34sS9/oAnUJsR9++EHXrugNLFh6xJUrV0BEyJs3L+Lj47FixQrx4CcnIoMHDxaZS9WlrogIRYoUwd9//53kKOfevXsi02nt2rXx8eNHvHr1Cq1atQIRoWLFili8eDFu3LghPi9JEl6/fo2DBw+iV69eMDMzg5WVFTZs2JDItzNnzohME+bm5ti+fbv8nZUFvHjxQlQ3unbtmq7d0QtYsPSIuLg4Uavw6tWr+Pjxo1gbSuqB+fLlCxYtWiSmLkSEokWLYuXKlfjy5UuSbdy9e1eIlYODg0bZe0mS4OXlhd9++02UyjIwMIC5ublG6Sw7OzssXLgwxXJifn5+aNGihfhMv379kl1ry84MHDgQRITWrVvr2hW9gAVLz+jWrRuICFOmTAHwvy32vn37imuioqKwYsUKjRGV+uXp6Zms7Tt37ojKPHXq1NEQq2+Jjo6Gt7c3tmzZguXLl2PVqlU4cuQI3r9/n+bvEh8fj1mzZok00JUqVVJclZqXL1+KURZXic48LFh6xrZt20QWUgC4du0aiAg5cuTA+/fvsWzZMtjY2AiBsrW1xdq1a8XUsHjx4kkunvv4+CBv3rwis2lSC/na4vz58yhcuLDIJ79582ZFndVycnICUUJVaSZzsGDpGf7+/kKMPnz4AEmSULlyZRAllKRXv1e8eHGsW7dOpE8ODw8X56wGDRqkYdPb21uIVb169bJUrNQEBASgTZs2wv+ePXsmCj3Krrx69UpMiS9duqRrdxQNC5YeUrt2bRAR1q1bh8WLF2sIVYkSJbBhw4Ykc7GfP39eXHfq1CkAwO3bt8W6WP369TN0dEEuVCoV/vzzT7HmVq5cOcWkJVaPYFu0aKFrVxQNC5YeMnHiRBCROO2uXvxWH19IiREjRoCIUKxYMXh5eQmxatCgQbYZ0Vy+fFmEF5mZmWHdunXZfor4+vVrMcpKS3QBkzQsWHpEeHg4FixYIESGiFCqVCls3rxZnLzu1atXijYiIiJgZ2cHIhIPWMOGDbONWKkJCgrCzz//LL5n165ddTr6SwvOzs4gIjRr1kzXrigWFiw9ICwsDH/99Zc4yEn0vwKr6swHN2/eFCOS4ODgFO2tW7dO2KlcuXK2Eys1KpUKixYtErtwdnZ28PHx0bVbyfLmzRuYmpqmaaTLJA0LloL5/Pkz5s2bJ44aEBHKlCmD7du3w9HREUSEP/74A0DCGSl16M6SJUuStXn9+nWRqkZ90v3jx49Z9ZUyxPXr10UYkampKVauXJltp4hDhw4FEaFJkybZ1sfsDAuWAvn8+TPmzp2rIVTlypXDjh07EBcXBwDYtWsXiAjVq1cXn1u7di2ICOXLl0/yYbl27ZpYoG/UqJGYGn59hiu7EhISgvbt24v+6NSpU4rnxHTF27dvxShLyXm/dAULloIIDQ3F7NmzxREDtfjs2rUrUbaEoKAgsdCurqSsLlSa1JTk6tWrsLS0FGssERERuHr1qrBx5MiRrPqaGUaSJCxfvlysvZUsWTLJ9Dq6Zvjw4SAiNG7cmEdZ6YQFSwF8+vQJM2fO1FhMr1ChAvbs2ZNiWpe6deuCSDPHlPoQo6Ojo/i3K1euCLFq3ry5RgjMuHHjxNQwJCREO19QZm7duiXOlJmYmGDp0qXZShjev38vdnDPnj2ra3cUBQtWNubTp0+YMWMGcufOLYSqUqVK2Lt3b5ryT82aNUtMj9R4e3uLBzkwMBCXLl0So64WLVokSsYXFRWFChUqpGmHMTvx6dMnkYWViPDLL79kK8F1cXERO7DZSUyzOyxY2ZCQkBBMmzZNY/G7cuXK2L9/P1QqVZrt3Lp1S5xw//qgqPpg6ZAhQ4RYtWrVKtnMoTdu3BC7jocOHcrs18syJEnC6tWrxZqRra0trl69qmu3ACSMstSB6adPn9a1O4qBBSsbERISgilTpmicTK9SpQrc3NzSJVRqVCoVChQokGiBV52OWL0+1bp161ST5akPoxYqVCjVYxHZjTt37qBMmTIgIhgZGWHBggUZ6k+5GTlypIgg4FFW2mDBygYEBwdj8uTJGkJVrVo1HDhwINMPlrrs1Pjx48W/qXO6ExHs7e3TlNnzy5cvIiFg9+7dM+WTLggLC0P37t3F9/7pp58QGBioU5/8/PzEKEsdCsWkDAuWDgkKCsIff/whFrzVxxAOHjwo2whg7969YkoJAF5eXjA3N9fY/k8rt27dEnF8Bw4ckMW/rESSJGzYsEGIRJEiRXQejDx69GgRVM6jrNRhwdIBgYGBmDBhglg/IiLUrFkThw4dkn2qEhISItaf9uzZg5w5c4rYQCKCsbEx/P3902xvypQpIEqomqPrEUpGuX//vqjdaGhoiLlz5+psiujv7y/+JidOnNCJD0qCBSsLCQgIwPjx4zVGOLVq1YKHh4dWf12/FieihBzqX758Ecce/vrrrzTbio6ORpUqVUBE+O2337Tms7YJDw8X02X1Ol56hFtOxo4dC6KEpIg8ykoZFqwswN/fH2PHjtUQKnt7+zQXbMgs6sBnIsLPP/8scmBt2bIFRAmlvtIzwvDx8RFTw/3792vL7Sxh69atYoRjY2ODc+fOZbkPX4+yjh8/nuXtKwkWLC3y4cMHjB49WtyMRAl50I8dO5Zlv6SnTp0S2/pGRkYaGQ0iIyPFGa/0LvpOnz4dRAnFK3Q1MpELX19fkeTQwMAAM2bMyFSdxYygPqBbu3ZtHmWlAAuWFvDz88OoUaPE4i5RQlphT0/PLL0ZT548KU5Uq/975swZjWvUYSLpWXwHEmoXVq9eHUSEjh07Kv4hi4yMFGW51OFJfn5+WdZ+QECAGIEfPXo0y9pVGixYMvL+/Xu4uLhoCFW9evVw8uTJLH+gT5w4IUSqffv2ohjFmDFjNK57+PChGH2l9wG9e/euWBfTh2rNALBz506xGVKgQIEsPW4wYcIEsVyg9B8AbaH3guXv749169Zh4MCBqFOnDipWrIiqVauibdu2mDZtGs6ePZvpHaJ3795h+PDhGhk+GzRogFOnTunkxjt+/LiYBnbo0AExMTFwdXUVMYjfol6Unzt3brrbUof/5M2bVwheVvS5Nvn3339RrVo1MUWcPHmyyIKhTYKCgoRYKiHYXBforWD5+vqiW7duMDExgZGREapVq4a+ffti9OjRGD58OH788UdxCrxMmTL4+++/031Tvn37FsOGDRPioE7LcubMGZ39Qh47dkz406lTJ8TGxgJIiK1TL5S/ePFC4zPbt28HUUJhivSu3cTGxqJmzZoicFrbfZ5VREVFiTzs6swKb9++1Xq7f/zxh9g95lFWYvROsOLi4vDXX3/B1NQUpUuXxrJly5JNQCdJEi5fvozu3bvD0NAQDg4O8PX1TbWNN2/ewNnZWUOoGjdujHPnzun0Jjt69KjwqXPnzkKs1DRp0gREhNWrV2v8e1RUlEhZk5Fdqjt37sDQ0BCGhoYoVaqUVvpcV+zbt09EIOTLl0/ru3hBQUHiIPHhw4e12pYS0SvBiomJQceOHWFoaIgJEyakKeREzfXr11GhQgVYWlomm1jt9evXGDJkiEYV46ZNm8LLy0vnv4ZHjhwRfnXp0iWRWAHAX3/9JY42fIs6ru3XX39NV7va7vPswLNnz1CrVi3xNx8/fnyS/SsXkyZNAhGhRo0aOr+vsht6I1iSJKFbt24wNTXN8C5LREQEWrduDXNzc9y+fVv8+6tXrzBo0CANoWrWrFm2yct9+PBh4VvXrl2TfZju378PooRipN+Won/06JE4+Z3WqY82+zy7ER0dLSoKqQOWX79+rZW2goODxaju4MGDWmlDqeiNYG3evFmWGLfIyEg4ODigXLlyePToEQYOHCh2wogSckZlpzJNhw4dEmLVrVu3FNeEJEkS5bFOnjyZ6H31lHHmzJlpalsbfZ6eEZoucHd3F2fX8ubNCw8PD620ow6Bql69erbeoMhq9EKw/Pz8YGVlhf79+8ti7/HjxzA1NRXpV4gS8kVdvnxZFvtycfDgQSGmjo6OaVrAHjhwIIgILi4uid7bvXs3iBJqEqZmSxt9bmZmppFVIrvy4sULODg4iHtj1KhRSRamzQwhISEiH5q7u7ustpWMXgjWtGnTYGlpKWt1lylTpsDIyAjNmzfHlStXZLMrFwcOHBBi1aNHjzTvth08eBBEhLJlyyZ6Lzo6Gvny5UvTtrq2+tzS0jLb1xcEEtbu1JkW1BEM3+6+ZpZp06aBiFC1alUeZf0/ihes2NhY2NjYwNnZWVa77969g5GREVatWiWrXTlwc3MTRxR69uyZrqMInz9/FkL39OnTRO+rA3GTWphX8z32eXJ4eHiIHdbcuXPLOhr6+PGjGGW5ubnJZlfJKF6wfHx8QERaGQW1bNky3btm2sbV1VWIVe/evTMU89a8eXMQEVasWJHovSdPnojF9+QWlb+3Pk+N169fo379+mK0NXz48ESbGhlFHbNZpUoVHmUBMCSF4+PjQ4aGhlSzZk3ZbTs4OJC3tzdFRkZmi9f27dupe/fupFKpqEePHrRq1SqKjo5Ot52WLVsSEdGxY8cSvVe0aFFq0qQJSZJEa9asSfLz165d02qf+/j4yG5XmxQvXpwuXrxIEyZMICKiVatWUYMGDej58+eZtj169GjKnTs3/fPPP3TgwIFM21M8ulbMzDJ+/HjY2dlpxfaOHTvErya/NF+2trZa7fPkCmJkdzw9PZE/f34QJRT/2LdvX6Ztzpw5E0QJFZO+91GW4kdYsbGxZGZmphXbOXLk0IpdfUDbfR4bG6sV+9rmp59+onv37lHjxo0pPDycHB0dafDgwfTly5cM2xw1ahTlyZOHHj16RG5ubjJ6qzwUL1jm5uYUHh6uFdtquyEhIRQREaGT1+bNm8nQMOHP1KdPHwoLC5PF7oABA4iIaNCgQUm+P3r0aCIiat26daL3xo0bRzExMVrt85w5c2rFflZQtGhR8vLyoqlTp5KBgQFt2LCB6tWrR0+ePMmQvdy5c9OYMWOIiGjWrFmkUqnkdFdZ6HqIl1l27doFIpJ1e12Ni4tLktv/WcXOnTtFPvaBAwfKOh3w8PAAEaFUqVJJhn88e/YMRAnZCl6+fKnxnrb73MrKSmR1UOrUUM3p06dRsGBBEBEsLCywc+fODNkJDQ0Vlb/1JZVPRlC8YD1+/BhEBE9PT9ltOzg4aJR0z0q2b98uDq46OTnJvnYRHh4uAqX//fffJK9p1aoViAiTJ0/W+Hdt9rk684P6ZWJigoYNG2LKlCk4ffo0IiIiZG9T2/j5+YmdWSJC//79MyTEc+bMAVFCiqCszoiaXVC8YEmShIoVK6JLly6y2n3w4AGIKMO/iJlh27ZtQqwGDx6stYVWtSAtXbo0yffd3NxAlFA89ev4REmSULp0aXTu3FlWf9R9PmjQIPTs2VOEEX39MjY2RoMGDTBp0iScOnUK4eHhsvqgLeLj4zFz5kzxd61UqRL++eefdNn4/PkzrK2tQUTYtWuXljzN3ihesABg5cqVMDIywvPnz2Wz2a9fPxgbG8PGxgbLly/Pshi3rVu3ipt6yJAhWt0VWrp0KYgSwo6SIjY2FoUKFdI4uHjnzh106NBBnNWSs8/79++PwoULC3GUJAnPnz/Hpk2b0Lt3b9ja2iYpYPXq1cMff/yBEydOICwsTDZ/tIGXlxdsbGxAlBCEvmXLlnRlZJg3bx6ICOXKlfsuR1l6IVjh4eEoUaIEmjVrJssDfvbsWRCRWDMgSqiosmzZMq0K1+bNm4VYDR06VOupRdRTO1NT02RHKupUJ3Xr1kX79u01xMLKygpNmzaVtc9TOuUuSRJevHiBLVu2oG/fvihRokQiATMyMkKdOnUwYcIEeHp6Zsswn4CAALRu3Vr43Lt37zSPFMPCwkT4lC5G/7pGLwQLAM6dOwciwqxZszJl5+3btyhatCiaN2+OqKgorFu3DsWLFxc3V6FChbBkyRLZF4M3bdok2hg2bFiW5EGSJAmlSpUCESWbdUC9OK9+GRgYoHv37vD19ZW1z4sVK4bmzZunW/xevnyJbdu2oV+/fuK7fP1SJwkcN24cjh07htDQ0Ez5KhcqlQrz5s0Tmyrly5fH/fv30/RZdV6zsmXLZtuMrdpCbwQL+N+i5OzZszP0wD9//hwlSpSAsbExFixYIP49JiYGGzZsQMmSJcWDULBgQSxatEiWReANGzYIuyNGjMjSpG3Dhg0T08+vuXXrFn7++WeNh79SpUp4/PixxnVy9LmdnR1KlCghSwriV69eYfv27RgwYABKly6dpIDZ29tjzJgxOHLkCD59+pTpNjPDpUuXxFpdjhw5sH79+lT78etR1vbt27PI0+yBXgmWJEmYPXs2iAgtW7ZMtB2fHCqVCmvXroWlpaUIZCUirF+/XuO62NhYbNq0SUO4ChQogIULF2ZYuNavXy9sjRw5MsszTB47dgxECfncJUnCzZs30bZtW40HvGnTpuK7fptGRY4+t7Ozkz3TgZo3b95g586dGDhwIMqUKZNIwAwMDFCzZk2MHj0aHh4eWjmqkRpBQUH46aefhE+Ojo6pTmXnz58PooTc+N/TKEuvBEvNqVOnYGtrC1NTU/Ts2RPnz59PtPYkSRJev36NZcuWoXz58mJ3KjQ0VKQLJiKsW7cukf3Y2Fhs3rxZ4xc8f/78mD9/frp2rdauXSs+P2rUKJ2kw42MjBTVfho2bKghVH369MGTJ08QFxeHIkWKgIiSDTXJTJ9n5TrTu3fvsHv3bjg5OaFcuXJJCliNGjUwcuRIHDp0CCEhIVnil0qlwsKFC0UmjTJlyuDOnTvJXh8eHi5CgLZu3ZolPmYH9FKwgIQt4EWLFsHOzk4sxlauXBmNGzdGvXr1xB/b2NgY3bp1w9WrV8VnJUnCqFGjxE28du3aJNuIjY3F1q1bRRtECYUK/vzzz1R3q9asWSM+M3r0aJ3l7r569aqYXqj7qV+/fnj27JnGdercTM2bN0/WVmb6XFe8f/8ee/bsweDBg4WIfvuqVq0aXFxc4O7ujqCgIK36c+3aNbFmampqilWrViV7byxcuBBEBDs7O3z+/BlHjhzB9OnT0aFDB7Ro0QJt2rTBwIEDsXbtWjx69EirfmcVeitYalQqFW7fvo1169Zh6NCh6N27N/r164fp06fDw8MDgYGBSX5OkiSMGTNG3LTfVpr5mri4OGzfvl1jymFtbY158+YlOXpYtWqVuG7s2LE6EasrV65o7FQREQoXLpzsMYXXr1+LBeInT56kaDujfZ4d+PDhA/bt2wdnZ2dUrFgxSQGrUqUKhg8fjgMHDmjlu4SEhODXX38V7XXp0iXJtbaIiAjxY6OutFOgQAG0bt0a3bp1Q+fOnVGtWjWRjqhx48Zwd3dXdGELvReszCBJkkhol9qWO5AgXDt27NCYauTNmxdz5swRwrVy5Urx3vjx47P85rl06RJatmwpfDA2NsZvv/0GooRT5SlNz9SL8GPHjs1Cj3WLv78/XF1dMXToUFSuXDlJAatcuTKGDh0KV1dXBAQEyNKuJElYtmyZyNdfqlQp3Lp1S+P9bdu2wdzcHLly5cL48ePx5MmTJO+nqKgouLq6olmzZiAitGvXDu/fv5fFz6yGBSsVJEnC+PHjxc35999/p/qZ+Ph47Nq1S2OKkSdPHo2F1QkTJmSpWF24cEEjPMTY2BhOTk5isVs9OkypSsuRI0fEtFeuBHVKIyAgAG5ubhg+fDiqVKmSpIBVrFgRzs7O2LdvHz58+JCp9m7duiWOa5iYmGDZsmWIj4+Hs7MziBLOcKVnne3w4cOwsbFBwYIFcffu3Uz5pgtYsNKAJEmYOHGiuCGXL1+eps/Fx8djz549qFChgsYN3ahRoyzbjTp//rz4ZVXf9IMHD8arV680rnNxcQFRQpB1csTFxaFYsWIgIuzevVvbriuCoKAguLu7w8XFRZS3//ZVvnx5DB48GHv37oWfn1+62/j06RM6d+4s7JUqVQoGBgbYtGlThnwODAyEvb098uXLl2wcaXaFBSuNSJIkyogTEZYtW5bmzy5evDjRTZw7d25Mnz5dK8IlSRK8vLxE2S61UA0ZMiTZtMcnT54EEaFo0aIpjvzUyeSaNGkiu9/6QHBwMA4dOoSRI0eiRo0aGpWX1K9y5crByckJu3fvxrt379JkV5IkrFq1SuwiJrcRlFZCQkJQoUIF2Nvba7UorNywYKUDSZIwefJkceMlFzT8NUuWLBHXT548Gfv27dNYC8mVKxemTZsmy/a5JEk4e/YsGjduLOybmppi6NChePPmTYqf/fLlC3LmzAkiSvHE9du3b8Xiu77sPGmTjx8/wsPDA6NHj0bNmjWTFLAyZcpg4MCB2LlzZ4p/p5CQEFhbW+PXX3+VZTnh1q1bMDIywrx58zJtK6tgwUonkiRh6tSp4mZbvHhxstcuWrRIXDdt2jRxk6lUKri5uWmsgVhZWWHKlCkIDg7OkE+nT5/WOEdlamqKYcOGpev0uHpR/a+//krxOvUO1qhRo9Lt6/fOx48fceTIEYwZMwb29vZC/L9+lS5dGgMGDMCOHTs0RsTz58+HmZlZhqaVyeHi4gJra+tsX8BWDQtWBpAkSZxLIiIsXLgw0TULFiwQ70+fPj3JX0SVSoUDBw5orH1YWlpi0qRJaTrvI0kSTp06pVGxxczMDCNGjEjzVONr1MctUpvuHT9+XGwkKOVGz66Ehobi2LFjGDduHBwcHJIUsFKlSqFPnz4oUKAA+vTpI2v76kSNW7ZskdWutmDBygQzZswQN9XXsYfqsAmitJV9V6lUOHjwIGrUqKEhXH/88UeSwiVJEk6cOIF69eqJ63PkyIGRI0dmarv6v//+A1HCgc+UYuzi4+PF4cbvLZZN23z+/BnHjx/HhAkTUKdOHXGGSv06d+6c7G02atRI9txm2oIFK5OoF6HVU6k///xT/H96sxhIkoTDhw9rZN20sLDAhAkTEBgYCEmScPz4cdSpU0dDqEaNGiXbNEG9o5la4U510HODBg1kaZdJmrCwMJw4cQJt27aFgYGBVsKYxo4di5IlS8puVxuwYMnArFmzEg3jZ8+enWF7kiTBw8MDtWrV0pjqqRO/ESUkfxszZkymz/l8i7r8ev/+/VO87v379+LX/+HDh7L6wCRm4sSJWhOVbdu2gYgUcbaOBUsmvj49nlwGz/SiUqkwbdo0mJuba4hh7dq1tSYSZ86cAVFCwsLUclN17NgRRAmVjhntMnLkSFSuXFkrtl1dXUFEOk+1kxYUX+YrOzB79mw6d+6c+P+zZ8/SnDlzMmwPAHl4eJCDgwPNmTOHoqKiyMzMjGxsbIiIyNvbm+rUqUNjxowhf3//TPv/NY0bNyYLCwvy9/en+/fvp3jt4MGDiYho586dFBUVJasfjCY5cuSgyMhIrdhW/+0UUYdT14qpdL5eeP92DSstC+5fI0kSDh06pLH4bmFhgYkTJ4o1LE9PT9StWzfRYrucW93qYwtz585N8TqVSiXCRpSyy6Q0Pn/+DE9PT/z000+8hgWeEmYYSZIwffr0VHcJZ8yYkaotlUoFd3d3VK9ePU3HGyRJwsmTJzWOM+TIkQMuLi6yBLWuW7cuzQvq6nS9devWzXS7zP+OOYwfPx4ODg68S/gNLFgZ4NtzWIsWLUp0jTpXEaVyDsvNzQ1Vq1bVOEA6efLkNB0gVR8YbdCggcbi/PDhwzN0DkvN69evQZSQxC+1E/j+/v4iXOTevXsZbvN75dOnTzh69CjGjh2L2rVrJ3uQtG/fvihYsCCfw9K1A0pDkiRMmTIl3Sfdp06dqnHS3dXVNdFJ96lTp2YoREeSJJw5cwaNGjVKd0hOcqjDh/bu3Zvqter0NM7Ozhlq63vi61CdWrVqJRmqY2dnh99//z1RqA6fdGfBSheZjSWcNGkS9u7dmyiWUK4gaEmScO7cOY2gZ1NTUzg7Oycb9Jwc6pQ6aflFV5fosrKyUkxh06wiJCQEhw8fxqhRo5INhi5btiycnJywa9euFEOpQkJCkC9fPvzyyy8cS8ikjDayNcyYMUNraWbOnz8vikcQJZ9WJjm8vLxAlJDBMrXjDSqVSuTT2rhxoxzuK5bg4GAcPHgQLi4uqF69epICVb58eQwaNAh79uxJ85qjJElYuXIlZ2vQtQNKQJIkTJgwQdxwK1asSNPn4uPjsXv3bp3mw/o2cZ+JiQkGDRqUanWbmJgYWFlZgYg0Ml0mh3rNrnbt2jJ5rgwCAwNx4MABjBgxQmMt8utXhQoVMGTIEOzbty/D+bA6deok7HE+LCZZvs04unLlylQ/ExcXh507d2pkHM2bN69G+ayszjh68eJFtGjRQrRvbGyMgQMHplheS30wNC0hRoGBgTA1NQURwcfHR07XsxUBAQFwdXXFsGHDkk2ZXKlSJQwdOhT79+/PdCTCzZs3RVk5ExMTLF++XCPjaK9evTjjKJNARnK6b9++HWXLlhWfsba2xty5c5PM6T5u3Lgsz+l++fJltGrVSkO4BgwYgP/++y/RtRs3bkzXkQVHR0cQJZTu0hf8/f2xf/9+ODs7o1KlSkkKVJUqVTBs2DC4ubnJmtN96dKlKeZ03759O/LkyQNra2uMGzcO//77b7I53ffv3y8yz/7yyy+c0z27oq7gsnbtWjg7O6NXr17o168fpk2bhsOHD6dYNUcdV0eUetWcbdu2aVTNSancV3aomnP16lW0adNG+GFkZIT+/ftrVM159+4diBJq9aUl3c358+dBRDA3N8f58+fT3efZAT8/P+zduxdDhgxJNJVXv6pWrYoRI0bA3d09y6rmhIaGJuvvmDFjRAHg/Pnzi6o5nTp10qia06RJE66ak135/PkzFi5cmGSNvPr166NAgQJihNG1a9cM1yXcsmVLorqEf/31l2LqEl67dg0//PCDhnD17dsXT58+BQCRq2vXrl2p2goNDUWBAgXEwnB6+lxXqAurDho0KNnCqtWrV8fIkSNx8ODBLKlLaGtrK3Z4V69enaZ7IyoqCkeOHMGMGTPQsWNHtGzZEj/88AOcnJywbt06vckOq5eC9XUV4l69euHChQuJItElScKbN2+wfPlysdbk5OSE0NBQUZCBKHG5euB/JevVYSnqX7YFCxaka1tffaKcSDdl6r/mxo0bGlV9DA0N0bt3bwwaNAhEhB49eqT4eXWfm5iYoGfPnunq86ys/Pz27Vvs2rULAwcO1Ji6fy1QNWvWxKhRo3D48OEsrfy8YMECMRpKrfLz94peCZYkSZg9ezaICC1btkx1J0yNSqXCunXrYGlpKYbWRIQNGzZoXBcTE4ONGzeKRVAiQsGCBbFo0SJERERkyOf169cLWy4uLjofrt+8eVOkSlY/wEQJ2UXj4+MTXS9Hn9vZ2aW4+J8Z3rx5gx07duD333/XGAl/Lcy1atXCmDFjcOTIkSzbvf2awMBAjR+L7t27pzpC/17RK8FSJ5WbPXt2hh7858+fo2TJkjA2NtZIexwTE4P169ejRIkSGkK1ePHiDAvV16gXt4kSUrXoWrSAhEOF7dq103i427Rpk2hqIUeflylTBsWLF09X/vnkePXqFbZt24b+/ftrjIC/FqjatWtj7NixOHr0qM5Tqly6dAlFihQBUUI86IYNG7LF3z+7ojeCde7cuTRvwafE27dvUbRoUTRv3hxRUVFYu3atSAdMlJAnaunSpYiMjJTJ8wQ2bdokRjPDhg3LNjett7e3eKDUIy5HR0f4+vrK2ufFihVD8+bNUz2k+i0vX77E1q1b0bdvX42R79drcnXq1MH48eNx/PjxZBevsxqVSoW5c+eK2MHy5cunWK2ISUAvBCs8PBwlSpRAs2bN0n3DJ4U61CRPnjzixi9cuDCWL1+u1ZirzZs3C9FydnaW5bvIwZYtWxL1B1FCKE7Tpk1l7fOUjo5IkoT//vsPmzdvRp8+fTR+SL4WqLp162LixIk4ceJEtpxa+fv7o3Xr1sLn3r17c0hTGtELwVq5ciWMjIw0tuQzS79+/WBsbIzChQtjxYoVWRYcunXrViFaQ4YMyRai5efnJx6us2fPigOlhoaGsvZ5//79UbhwYREqIkkSnj17ho0bN6JXr16i6vTXL2NjY9SvXx+TJk3CyZMns6VAfc25c+dEquucOXNi69atunZJUShesCRJQsWKFdGlSxdZ7T58+BBEhJ07d8pqNy1s27ZNiNagQYOyhWip88tv374dkiShdOnSsudQUvf5oEGD0KNHDxQtWjSRQJmYmKBhw4aYPHkyTp8+LcsaYlYQHx+PGTNmiL9r5cqV4evrq2u3FIfiBevx48cgInh6espu28HBAY6OjrLbTQs7duwQN7eTk5PORUudUqdbt25a7fOvKwapBapRo0aYOnUqzpw5oxiB+ho/Pz+NeM4BAwbIvgb6vaB4wdq1axeISCvb0S4uLihbtqzsdtPKzp07xaLs77//rlPRunr1qljH2r59u1b73MrKCtOnT8e5c+cU/2CfPn0aBQsWBFFCumtdjNj1CWNSOL6+vmRra0t58+aV3Xb16tXp77//po8fP5KZmZns9lOjY8eOFBsbSwMHDqTNmzdTbGwsrV69mgwNs752SJUqVShv3rz06dMnOnHihFb7PDw8nCZPnqyTPpeL+Ph4mjlzJv35558EgKpVq0aurq5Uvnx5XbumaBQvWFFRUWRlZaUV22q7+fLl04r99LJz507auXOnrt2gffv2UZkyZbRiW93nX758UaxgvXv3jnr06EGXL18mooTqQsuWLaOcOXPq2DPlo3jBMjU1pZiYGK3Yjo6O1opdfUDbfW5qaqoV+9rG09OT+vTpQyEhIWRlZUUbN26kbt266dotvUHxglW2bFl6+fIlRUVFkbm5uay2Hz16REWKFKGnT5/KajejuLu7U//+/UmSJHJ0dKT169eTkZFRlrUfGBhIpUuXJiKi9+/fa63PixYtKrtdbRMXF0dTpkyhRYsWERFRrVq1aP/+/VobiX636HoRLbP4+PiAiHDlyhXZbbds2RK//vqr7HYzg6urqwiQ7dWrV5LxfdrEwcFB7HZ9L32eGq9fv9YouTZixAhER0fr2i29RPGCFRsbCxsbG9krtrx9+xZGRkapJu3TBQcOHBApXHr27JmloqWuxWhubv5d9XlyeHh4iID53Llzw93dXdcu6TWKFywAmDZtGiwsLGTdZp8yZQqMjIzQrFkzXL58WTa7cuHu7i5Eq0ePHoiLi8uSdm/cuCFyNWmjzy0tLbM03UxGiYmJ0ciZ5uDgoLWME8z/0AvB8vPzQ65cudC/f39Z7D1+/BimpqYaFU9atmyJS5cuyWJfLg4ePChEy9HRMUtEKz4+Hvnz5wdRQnVqOfvczMwM48ePl8WeNnnx4oXG1Hj06NGIiYnRtVvfBXohWEBC4DAR4cCBA5myExkZCQcHB5QrVw6PHz+Gk5OTEAUiQvPmzXHhwgWZvM48hw8fFnm/u3XrliWi1bNnTxCRyOEkZ59n94Ke7u7uyJ07N4gSCot4eHjo2qXvCr0RLEmS0K1bN5iamuLo0aMZshEREYHWrVvDwsICt2/fFv/+6tUrDB48WAgDEaFp06Y4f/68TN5nDg8PD+Hbb7/9pvU6c7t37wZRQvEFbfV5duPLly8YPny4+PvXr18/3cVpmcyjN4IFJKwrdOzYEYaGhhg/fny6fq2vXbuGChUqwNLSMlkhev36NZydnTWEq0mTJjh37pzO81cdOXJE+NWlSxetilZwcLCYLv/3339a7fPswLNnzzRiHCdMmKCo4qP6hF4JFpBQwWb+/PkwNTVFqVKlsHTp0mQXhiVJwqVLl9C9e3cYGhqiTp06aYqgf/PmDYYOHSrq8BElFEc9e/asToXr6NGjwqfOnTtr9aGqV68eiBLSSGdFn+uKffv2iYKy+fLlw/Hjx3Xt0neN3gmWGl9fXzg6OsLExARGRkaoWrUq+vTpg9GjR2P48OH44YcfxOJx2bJlsXLlynSv/7x9+xbDhw+HmZmZEK6GDRvi9OnTOhOu48ePC9Hq2LGj1haD1XncO3ToIP4tK/o8q4iKihIFOIgIjRs3liWFM5M5DABA5rOo2YqAgAA6fPgw+fj40IMHDyg8PJyMjIyoePHiZG9vT02bNqVmzZplKqD4/fv3tGDBAtqwYYMIWalfvz7NmDGD2rRpQwYGBnJ9nTRx4sQJ6tixI8XExFCHDh1o//79soe6eHt7k4ODA1laWlJISIiG/azoc23y5MkT6tq1Kz148IAMDAxo8uTJNHPmTDI2VnxgiPLRtWLqE+/fv8fIkSORI0cO8ctct25dnDhxIstHXCdOnBAjv/bt28s+0lKpVCJtyrlz52S1rUt27twJCwsLUWjk9OnTunaJ+QoWLC3g5+eH0aNHI2fOnEK46tSpg+PHj2epcJ08eVKI1i+//CJ7uEjfvn1BRBg3bpysdnVBZGQk+vfvr3F8xc/PT9duMd/AgqVFPnz4gDFjxmgIl4ODA44ePZplwnXq1Ckx4mvXrp2sorVv3z4QESpVqiSbTV3wzz//oFKlSqIq0MyZM7M8RpNJGyxYWYC/vz/GjRsHc3NzIVz29vY4cuRIlgjXmTNnhGj9/PPPsonWx48fRUbUV69eyWIzK5EkCVu2bBE/KDY2NvDy8tK1W0wKsGBlIQEBAZgwYYJYIyEi1KpVC4cPH9a6cJ09e1Y8mG3btk1URj6jNGzYEESEtWvXymIvqwgPD0fv3r3F36F169bw9/fXtVtMKrBg6YDAwEBMnDhRQ7hq1KiBgwcPajVv+7lz54Ro/fjjj7KI1rx588QamVK4f/8+ypcvL0qVzZs3T+dFPpi0wYKlQ4KCgjBp0iRYWloK4apWrRrc3d219gCdP39eTE1/+OGHTIvW3bt3RboZuUZt2kKSJKxfv15Mj4sWLZrtAtqZlGHBygYEBwdjypQp4kQ1EaFq1apwc3PTinBduHBBiFabNm0yFXAsSRIKFy4MIsrWRwA+f/4MR0dH0b8//fQTgoKCdO0Wk05YsLIRISEhmDp1KnLlyiUerCpVqsDV1VV24bp48aKYkrZq1SpT5bQGDBgAIsKoUaNk9FA+7ty5gzJlyoAooZT9woULeQqoUFiwsiEfP37E9OnTRRoT+v9Kwfv27ZN1u/3SpUtCtFq2bJlh0Tpw4ACICOXLl5fNNzmQJAmrVq0SoUrFixfHtWvXdO0WkwlYsLIxnz59wsyZMzWEq2LFitizZ49swnX58mWxhtaiRYsMiVZoaKjIGfb8+XNZ/Mosnz59QufOnUW//frrrwgJCdG1W0wmYcFSAJ8+fcKsWbOQJ08e8QBWqFABu3fvlkW4rly5IkSrefPmGSoH37RpUxARVq5cmWl/MsutW7dQqlQpUep+2bJlOk//w8gDC5aCCA0NxZw5c0TRA/U0bOfOnZnOenD16lWx6N+sWbN0i9b8+fPFGS9dIUkSli1bJvKClSxZErdu3dKZP4z8sGApkM+fP2PevHmwtrYWwlW2bFls3749U8J17do1IVpNmjRBeHh4mj/74MEDEBFy5MihkzTHISEh+PXXX0V/dOrUCZ8+fcpyPxjtwoKlYMLCwvDnn38iX7584kEtU6YMtm3blmHhun79utilbNy4cZpFS5IkFCtWDESEEydOZKjtjHLt2jUUL15cVPNZtWoVTwH1FBYsPSAsLAzz588XyfGICHZ2dtiyZUuGso7euHFDiFajRo0QFhaWps85OTmBKKGQaFagUqmwcOFCseBvZ2cHHx+fLGmb0Q0sWHpEeHg4Fi5ciAIFCgjhKl26NDZt2pRu4bp586bYnWzYsGGaROvQoUNCOLRNUFAQ2rZtK75nt27dFFHPkMkcLFh6SEREBBYtWiQS7KkXoDdu3JiuRH63b98WO5MNGjRIVRDCwsLEgvfTp08z+zWS5dKlSyhatCiICGZmZli/fj1PAb8TWLD0mMjISCxZsgSFChUSwlWiRAmsX78+zcLl7e0tRKt+/fqpilaLFi1ARFi+fLkcX0EDlUqFefPmwcjICESEcuXK4f79+7K3w2RfWLC+AyIjI7Fs2TLY2NgI4SpevDjWrl2bptxYPj4+4ihFvXr1EBoamuy1ixcvFjGKchIQEIDWrVsL/3v16pWuXUxGP2DB+o6IiorC8uXLRbAyEcHW1hZr1qxJVbju3LkjjlHUrVs3WdHy9fUVU7WMHEBNCi8vLyG2OXPmxJYtW3gK+J2i91VzmMR8+fKFNm3aRPPnzyc/Pz8iIipWrBj98ccf9Pvvv1OOHDmS/Ny9e/eoZcuW9PHjR6pTpw6dOnWK8uTJo3ENACpVqhS9fv2ali5dSrlz56awsDAyNjYWVXOKFCmSpkpCKpWK5s6dS7NnzyZJkqhSpUrk6upKlStXznQfMApFx4LJ6JAvX75g1apVYgGb/j9H1N9//51sbqt79+6Jc1+1a9fWKJgqSRLOnTsHOzs7URnawMAA5ubmGkVnS5cujQULFqSY3sXPz0+shxER+vfvL9uIjVEuLFgMoqOjsXr1anHwk4hQpEgRrFixIslT6/fv3xdnvuzt7fHx40e8evUKrVq1EnGOS5Yswc2bN8XnJUnCmzdvcOjQIfTu3RtmZmawsrJKcofv9OnTYofTwsICO3bsyJJ+YLI/LFiMIDo6GmvXroWtra0QLhsbGyxbtiyRcD148ECIVunSpWFpaQlbW1scO3YsTetLQUFBGDhwoIg/DA8PR1xcHKZMmSJGZ1WrVsXjx4+19XUZBcKCxSQiJiYG69evF+EuauFaunSpRvqZhw8fIleuXDAwMECHDh0ydHDT09MTVlZWqFOnDho0aCDaGzRokE5iEpnsDQsWkywxMTHYsGEDSpYsKYSkUKFCWLx4MSIiIvDvv//CzMwMv/32W6bS3Fy/fl0Ux7C0tMTevXtl/BaMPsG7hEyqxMXF0Y4dO2jevHn08uVLIiLKnz8/WVpakomJCd27d4/Mzc0z1camTZvIycmJNm7cSAMHDpTDbUYPYcFi0kxcXBzt2rWL5s6dSy9evCAiosuXL1OjRo0ybRsAtWnTht6/f0++vr5pOvbAfH+wYDHpJi4ujqpWrUr58+enK1euyGbXy8uLWrZsSV5eXtS8eXPZ7DL6g6GuHWCUh7+/Pz158oSGDh0qq93mzZtT2bJlaffu3bLaZfQHFiwm3Xh7exMRUbNmzWS1a2BgQE2bNhX2GeZbWLCYdHP//n0qWLAgFSlSRHbbtWrVIl9fX4qLi5PdNqN8WLCYdBMaGkr58+fXiu18+fJRfHw8RUVFacU+o2xYsJh0Y2RkRPHx8VqxrVKpRBsM8y0sWEy6sbW1pdevX2tl2vb8+XPKmzcvWVhYyG6bUT4sWEy6qV27NsXExJCvr6/stn18fMje3p7PYTFJwoLFpJuaNWuShYUFHT58WFa7YWFhdPbsWWrSpImsdhn9gQWLSTcWFhbUu3dv2rBhA8XGxspmd8eOHRQdHU0DBgyQzSajX7BgMRlixIgRFBgYSAsXLpTFXnBwMM2ZM4e6du1KRYsWlcUmo3+wYDEZolKlSjRx4kSaPXs23b17N1O2AJCzszPFx8fT0qVLZfKQ0Uc4lpDJMDExMdSwYUN69+4dnT9/nipWrJhuGwBo9OjRtGLFCnJzc6MuXbpowVNGX+ARFpNhzMzMyNPTkwoUKECNGjUiV1dXSs/vX0BAAHXu3JlWrFhBa9asYbFiUoUFi8kUBQsWpIsXL1KLFi2oW7du1L59e7p06VKKwhUSEkJLliyhypUr0+XLl+ngwYPk7OychV4zSoWnhIxsuLm50dSpU+np06dUrlw5atiwIdWsWZOsra0pLi6Onj9/Tt7e3nTx4kWSJIl69OhBCxcupAIFCujadUYhsGAxsgKAvLy8aP/+/eTj40MPHz4UJ+JtbGzI3t6emjZtSn379qWCBQvq2FtGabBgMVpFkiSKjY0lIyMjMjEx0bU7jMJhwWIYRjHwojvDMIqBBYthGMXAgsUwjGJgwWIYRjGwYDEMoxhYsBiGUQwsWAzDKAYWLIZhFAMLFsMwioEFi2EYxcCCxTCMYmDBYhhGMbBgMQyjGFiwGIZRDCxYDMMoBhYshmEUAwsWwzCKgQWLYRjFwILFMIxiYMFiGEYxsGAxDKMYWLAYhlEMLFgMwygGFiyGYRQDCxbDMIqBBYthGMXAgsUwjGJgwWIYRjGwYDEMoxhYsBiGUQwsWAzDKAYWLIZhFAMLFsMwioEFi2EYxcCCxTCMYmDBYhhGMbBgMQyjGFiwGIZRDCxYDMMoBhYshmEUAwsWwzCKgQWLYRjFwILFMIxiYMFiGEYxsGAxDKMYWLAYhlEMLFgMwygGFiyGYRQDCxbDMIqBBYthGMXAgsUwjGJgwWIYRjGwYDEMoxhYsBiGUQwsWAzDKAYWLIZhFAMLFsMwioEFi2EYxcCCxTCMYmDBYhhGMbBgMQyjGFiwGIZRDCxYDMMoBhYshmEUAwsWwzCK4f8ANVcbl94/DP4AAAAASUVORK5CYII="
|
|
}
|
|
},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"In this example the input vector of the neural network has two features, i.e., the input is a two-dimensional vector:\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\mathbf x = (x_0, x_1).\n",
|
|
"$$\n",
|
|
"\n",
|
|
"We consider a set of $m$ vectors as training data. The training data can therefore be written as a $m \\times 2$ matrix where each row represents a feature vector:\n",
|
|
"\n",
|
|
"$$ \n",
|
|
"X = \n",
|
|
"\\begin{pmatrix}\n",
|
|
"x_{00} & x_{01} \\\\\n",
|
|
"x_{10} & x_{11} \\\\\n",
|
|
"\\vdots & \\vdots \\\\\n",
|
|
"x_{m-1\\,0} & x_{m-1\\,1} \n",
|
|
"\\end{pmatrix} $$\n",
|
|
"\n",
|
|
"The known labels (1 = 'signal', 0 = 'background') are stored in a $m$-dimensional column vector $\\mathbf y$.\n",
|
|
"\n",
|
|
"In the following, $n_1$ denotes the number of neurons in the hidden layer. The weights for the connections from the input layer (layer 0) to the hidden layer (layer 0) are given by the following matrix:\n",
|
|
"\n",
|
|
"$$\n",
|
|
"W^{(1)} = \n",
|
|
"\\begin{pmatrix}\n",
|
|
"w_{00}^{(1)} \\dots w_{0 \\, n_1-1}^{(1)} \\\\\n",
|
|
"w_{10}^{(1)} \\dots w_{1 \\, n_1-1}^{(1)} \n",
|
|
"\\end{pmatrix}\n",
|
|
"$$\n",
|
|
"\n",
|
|
"Each neuron in the hidden layer is assigned a bias $\\mathbf b^{(1)} = (b^{(1)}_0, \\ldots, b^{(1)}_{n_1-1})$. The neuron in the output layer has the bias $\\mathbf b^{(2)}$. With that, the output values of the network for the matrix $X$ of input feature vectors is given by\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{align}\n",
|
|
"Z^{(1)} &= X W^{(1)} + \\mathbf b^{(1)} \\\\\n",
|
|
"A^{(1)} &= \\sigma(Z^{(1)}) \\\\\n",
|
|
"Z^{(2)} &= A^{(1)} W^{(2)} + \\mathbf b^{(2)} \\\\\n",
|
|
"A^{(2)} &= \\sigma(Z^{(2)})\n",
|
|
"\\end{align}\n",
|
|
"$$\n",
|
|
"\n",
|
|
"The loss function for a given set of weights is given by\n",
|
|
"\n",
|
|
"$$ L = \\sum_{k=0}^{m-1} (y_{\\mathrm{pred},k} - y_{\\mathrm{true},k})^2 $$\n",
|
|
"\n",
|
|
"We can know calculate the gradient of the loss function w.r.t. the wights. With the definition $\\hat L = (y_\\mathrm{pred} - y_\\mathrm{true})^2$, the gradients for the weights from the output layer to the hidden layer are given by: \n",
|
|
"\n",
|
|
"$$ \\frac{\\partial \\tilde L}{\\partial w_i^{(2)}} = \\frac{\\partial \\tilde L}{\\partial a_k^{(2)}} \\frac{a_k^{(2)}}{\\partial w_i^{(2)}} = \\frac{\\partial \\tilde L}{ \\partial a_k^{(2)}} \\frac{\\partial a_k^{(2)}}{ \\partial z_k^{(2)}} \\frac{\\partial z_k^{(2)}}{\\partial w_i^{(2)}} = 2 (a_k^{(2)} - y_k) a_k^{(2)} (1 - a_k^{(2)}) a_{k,i}^{(1)} \\equiv \\delta^{(2)}_k a_{k,i}^{(1)}$$\n",
|
|
"\n",
|
|
"Note, that it is assumed that the activation function is a sigmoid with the derivative\n",
|
|
"\n",
|
|
"$$ \\sigma(x) \\cdot (1 - \\sigma(x)) $$\n",
|
|
"\n",
|
|
"Applying the chain rule further, we obtain the gradient for the weights from the input layer to the hidden layer: \n",
|
|
"\n",
|
|
"$$ \\frac{\\partial \\tilde L}{\\partial w_{ij}^{(1)}} = \\frac{\\partial \\tilde L}{\\partial a_k^{(2)}} \\frac{\\partial a_k^{(2)}}{\\partial z_k^{(2)}} \\frac{\\partial z_k^{(2)}}{\\partial a_{k,j}^{(1)}} \\frac{\\partial a_{k,j}^{(1)}}{\\partial z_{k,j}^{(1)}} \\frac{\\partial z_{k,j}^{(1)}}{\\partial w_{ij}^{(1)}} $$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## A simple neural network class"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# A simple feed-forward neutral network with on hidden layer\n",
|
|
"# see also https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6\n",
|
|
"\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"class NeuralNetwork:\n",
|
|
" def __init__(self, x, y):\n",
|
|
" n1 = 3 # number of neurons in the hidden layer\n",
|
|
" self.input = x\n",
|
|
" self.weights1 = np.random.rand(self.input.shape[1],n1)\n",
|
|
" self.bias1 = np.random.rand(n1)\n",
|
|
" self.weights2 = np.random.rand(n1,1)\n",
|
|
" self.bias2 = np.random.rand(1) \n",
|
|
" self.y = y\n",
|
|
" self.output = np.zeros(y.shape)\n",
|
|
" self.learning_rate = 0.01\n",
|
|
" self.n_train = 0\n",
|
|
" self.loss_history = []\n",
|
|
"\n",
|
|
" def sigmoid(self, x):\n",
|
|
" return 1/(1+np.exp(-x))\n",
|
|
"\n",
|
|
" def sigmoid_derivative(self, x):\n",
|
|
" return x * (1 - x)\n",
|
|
"\n",
|
|
" def feedforward(self):\n",
|
|
" self.layer1 = self.sigmoid(self.input @ self.weights1 + self.bias1)\n",
|
|
" self.output = self.sigmoid(self.layer1 @ self.weights2 + self.bias2)\n",
|
|
"\n",
|
|
" def backprop(self):\n",
|
|
"\n",
|
|
" # delta2: [m, 1], m = number of training data\n",
|
|
" delta2 = 2 * (self.y - self.output) * self.sigmoid_derivative(self.output)\n",
|
|
"\n",
|
|
" # Gradient w.r.t. weights from hidden to output layer: [n1, 1] matrix, n1 = # neurons in hidden layer\n",
|
|
" # self.layer1.T: m x n1 matrix\n",
|
|
" d_weights2 = self.layer1.T @ delta2\n",
|
|
" d_bias2 = np.sum(delta2)\n",
|
|
"\n",
|
|
" print(self.layer1.shape) \n",
|
|
" \n",
|
|
" # shape of delta1: [m, n1], m = number of training data, n1 = # neurons in hidden layer\n",
|
|
" delta1 = (delta2 @ self.weights2.T) * self.sigmoid_derivative(self.layer1)\n",
|
|
" d_weights1 = self.input.T @ delta1\n",
|
|
" d_bias1 = np.ones(delta1.shape[0]) @ delta1 \n",
|
|
" \n",
|
|
" # update weights and biases\n",
|
|
" self.weights1 += self.learning_rate * d_weights1\n",
|
|
" self.weights2 += self.learning_rate * d_weights2\n",
|
|
"\n",
|
|
" self.bias1 += self.learning_rate * d_bias1\n",
|
|
" self.bias2 += self.learning_rate * d_bias2\n",
|
|
"\n",
|
|
" def train(self, X, y):\n",
|
|
" self.output = np.zeros(y.shape)\n",
|
|
" self.input = X\n",
|
|
" self.y = y\n",
|
|
" self.feedforward()\n",
|
|
" self.backprop()\n",
|
|
" self.n_train += 1\n",
|
|
" if (self.n_train %1000 == 0):\n",
|
|
" loss = np.sum((self.y - self.output)**2)\n",
|
|
" print(\"loss: \", loss)\n",
|
|
" self.loss_history.append(loss)\n",
|
|
" \n",
|
|
" def predict(self, X):\n",
|
|
" self.output = np.zeros(y.shape)\n",
|
|
" self.input = X\n",
|
|
" self.feedforward()\n",
|
|
" return self.output\n",
|
|
" \n",
|
|
" def loss_history(self):\n",
|
|
" return self.loss_history\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Create toy data\n",
|
|
"We create three toy data sets\n",
|
|
"1. two moon-like distributions\n",
|
|
"2. circles\n",
|
|
"3. linearly separable data sets"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html#sphx-glr-auto-examples-classification-plot-classifier-comparison-py\n",
|
|
"import numpy as np\n",
|
|
"from sklearn.datasets import make_moons, make_circles, make_classification\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"\n",
|
|
"X, y = make_classification(\n",
|
|
" n_features=2, n_redundant=0, n_informative=2, random_state=1, n_clusters_per_class=1\n",
|
|
")\n",
|
|
"rng = np.random.RandomState(2)\n",
|
|
"X += 2 * rng.uniform(size=X.shape)\n",
|
|
"linearly_separable = (X, y)\n",
|
|
"\n",
|
|
"datasets = [\n",
|
|
" make_moons(n_samples=200, noise=0.1, random_state=0),\n",
|
|
" make_circles(n_samples=200, noise=0.1, factor=0.5, random_state=1),\n",
|
|
" linearly_separable,\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Create training and test data set"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# datasets: 0 = moons, 1 = circles, 2 = linearly separable\n",
|
|
"X, y = datasets[1]\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
" X, y, test_size=0.4, random_state=42\n",
|
|
")\n",
|
|
"\n",
|
|
"x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n",
|
|
"y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Train the model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"y_train = y_train.reshape(-1, 1)\n",
|
|
"\n",
|
|
"nn = NeuralNetwork(X_train, y_train)\n",
|
|
"\n",
|
|
"for i in range(100000):\n",
|
|
" nn.train(X_train, y_train)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Plot the loss vs. the number of epochs"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"plt.plot(nn.loss_history)\n",
|
|
"plt.xlabel(\"# epochs / 1000\")\n",
|
|
"plt.ylabel(\"loss\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from matplotlib.colors import ListedColormap\n",
|
|
"\n",
|
|
"cm = plt.cm.RdBu\n",
|
|
"cm_bright = ListedColormap([\"#FF0000\", \"#0000FF\"])\n",
|
|
"\n",
|
|
"xv = np.linspace(x_min, x_max, 10)\n",
|
|
"yv = np.linspace(y_min, y_max, 10)\n",
|
|
"Xv, Yv = np.meshgrid(xv, yv)\n",
|
|
"XYpairs = np.vstack([ Xv.reshape(-1), Yv.reshape(-1)])\n",
|
|
"zv = nn.predict(XYpairs.T)\n",
|
|
"Zv = zv.reshape(Xv.shape)\n",
|
|
"\n",
|
|
"fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(9, 7))\n",
|
|
"ax.set_aspect(1)\n",
|
|
"cn = ax.contourf(Xv, Yv, Zv, cmap=\"coolwarm_r\", alpha=0.4)\n",
|
|
"\n",
|
|
"ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright, edgecolors=\"k\")\n",
|
|
"\n",
|
|
"# Plot the testing points\n",
|
|
"ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, alpha=0.4, edgecolors=\"k\")\n",
|
|
"\n",
|
|
"ax.set_xlim(x_min, x_max)\n",
|
|
"ax.set_ylim(y_min, y_max)\n",
|
|
"# ax.set_xticks(())\n",
|
|
"# ax.set_yticks(())\n",
|
|
"\n",
|
|
"fig.colorbar(cn)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.16"
|
|
},
|
|
"vscode": {
|
|
"interpreter": {
|
|
"hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
|
|
}
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|