{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise: Classification of air showers measured with the MAGIC telescope\n",
    "\n",
    "The [MAGIC telescope](https://en.wikipedia.org/wiki/MAGIC_(telescope)) is a Cherenkov telescope situated on La Palma, one of the Canary Islands. The [MAGIC machine learning dataset](https://archive.ics.uci.edu/ml/datasets/magic+gamma+telescope) can be obtained from [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php).\n",
    "\n",
    "The task is to separate signal events (gamma showers) and background events (hadron showers) based on the features of a measured Cherenkov shower.\n",
    "\n",
    "The features of a shower are:\n",
    "\n",
    "    1.  fLength:  continuous  # major axis of ellipse [mm]\n",
    "    2.  fWidth:   continuous  # minor axis of ellipse [mm] \n",
    "    3.  fSize:    continuous  # 10-log of sum of content of all pixels [in #phot]\n",
    "    4.  fConc:    continuous  # ratio of sum of two highest pixels over fSize  [ratio]\n",
    "    5.  fConc1:   continuous  # ratio of highest pixel over fSize  [ratio]\n",
    "    6.  fAsym:    continuous  # distance from highest pixel to center, projected onto major axis [mm]\n",
    "    7.  fM3Long:  continuous  # 3rd root of third moment along major axis  [mm] \n",
    "    8.  fM3Trans: continuous  # 3rd root of third moment along minor axis  [mm]\n",
    "    9.  fAlpha:   continuous  # angle of major axis with vector to origin [deg]\n",
    "    10. fDist:    continuous  # distance from origin to center of ellipse [mm]\n",
    "    11. class:    g,h         # gamma (signal), hadron (background)\n",
    "\n",
    "g = gamma (signal):     12332\n",
    "h = hadron (background): 6688\n",
    "\n",
    "For technical reasons, the number of h events is underestimated.\n",
    "In the real data, the h class represents the majority of the events.\n",
    "\n",
    "You can find further information about the MAGIC telescope and the data discrimination studies in the following [paper](https://reader.elsevier.com/reader/sd/pii/S0168900203025051?token=8A02764E2448BDC5E4DD0ED53A301295162A6E9C8F223378E8CF80B187DBFD98BD3B642AB83886944002206EB1688FF4)  (R. K. Bock et al., \"Methods for multidimensional event classification: a case studyusing images from a Cherenkov gamma-ray telescope\" NIM A  516 (2004) 511-528) (You need to be within the university network to get free access.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"https://www.physi.uni-heidelberg.de/~reygers/lectures/2021/ml/data/magic04_data.txt\"\n",
    "df = pd.read_csv(filename, engine='python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use categories 1 and 0 insted of \"g\" and \"h\"\n",
    "df['class'] = df['class'].map({'g': 1, 'h': 0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fLength</th>\n",
       "      <th>fWidth</th>\n",
       "      <th>fSize</th>\n",
       "      <th>fConc</th>\n",
       "      <th>fConc1</th>\n",
       "      <th>fAsym</th>\n",
       "      <th>fM3Long</th>\n",
       "      <th>fM3Trans</th>\n",
       "      <th>fAlpha</th>\n",
       "      <th>fDist</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28.7967</td>\n",
       "      <td>16.0021</td>\n",
       "      <td>2.6449</td>\n",
       "      <td>0.3918</td>\n",
       "      <td>0.1982</td>\n",
       "      <td>27.7004</td>\n",
       "      <td>22.0110</td>\n",
       "      <td>-8.2027</td>\n",
       "      <td>40.0920</td>\n",
       "      <td>81.8828</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>31.6036</td>\n",
       "      <td>11.7235</td>\n",
       "      <td>2.5185</td>\n",
       "      <td>0.5303</td>\n",
       "      <td>0.3773</td>\n",
       "      <td>26.2722</td>\n",
       "      <td>23.8238</td>\n",
       "      <td>-9.9574</td>\n",
       "      <td>6.3609</td>\n",
       "      <td>205.2610</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>162.0520</td>\n",
       "      <td>136.0310</td>\n",
       "      <td>4.0612</td>\n",
       "      <td>0.0374</td>\n",
       "      <td>0.0187</td>\n",
       "      <td>116.7410</td>\n",
       "      <td>-64.8580</td>\n",
       "      <td>-45.2160</td>\n",
       "      <td>76.9600</td>\n",
       "      <td>256.7880</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.8172</td>\n",
       "      <td>9.5728</td>\n",
       "      <td>2.3385</td>\n",
       "      <td>0.6147</td>\n",
       "      <td>0.3922</td>\n",
       "      <td>27.2107</td>\n",
       "      <td>-6.4633</td>\n",
       "      <td>-7.1513</td>\n",
       "      <td>10.4490</td>\n",
       "      <td>116.7370</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75.1362</td>\n",
       "      <td>30.9205</td>\n",
       "      <td>3.1611</td>\n",
       "      <td>0.3168</td>\n",
       "      <td>0.1832</td>\n",
       "      <td>-5.5277</td>\n",
       "      <td>28.5525</td>\n",
       "      <td>21.8393</td>\n",
       "      <td>4.6480</td>\n",
       "      <td>356.4620</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    fLength    fWidth   fSize   fConc  fConc1     fAsym  fM3Long  fM3Trans  \\\n",
       "0   28.7967   16.0021  2.6449  0.3918  0.1982   27.7004  22.0110   -8.2027   \n",
       "1   31.6036   11.7235  2.5185  0.5303  0.3773   26.2722  23.8238   -9.9574   \n",
       "2  162.0520  136.0310  4.0612  0.0374  0.0187  116.7410 -64.8580  -45.2160   \n",
       "3   23.8172    9.5728  2.3385  0.6147  0.3922   27.2107  -6.4633   -7.1513   \n",
       "4   75.1362   30.9205  3.1611  0.3168  0.1832   -5.5277  28.5525   21.8393   \n",
       "\n",
       "    fAlpha     fDist  class  \n",
       "0  40.0920   81.8828      1  \n",
       "1   6.3609  205.2610      1  \n",
       "2  76.9600  256.7880      1  \n",
       "3  10.4490  116.7370      1  \n",
       "4   4.6480  356.4620      1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Create for each variable a figure with a plot for gammas and hadrons overlayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "df0 = df[df['class'] == 0] # hadron data set\n",
    "df1 = df[df['class'] == 1] # gamma data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x122031130>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 30536 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(nrows=2, ncols=5, figsize=(15,5))\n",
    "plt.subplots_adjust(hspace = 0.3, wspace=0.3)\n",
    "\n",
    "for i in range(10):\n",
    "    kx = i // 5\n",
    "    ky = i % 5\n",
    "    axs[kx, ky].set_xlabel(df0.columns[i])\n",
    "    df0.iloc[:,i].hist(ax = axs[kx, ky], bins = 50, alpha=0.5, density=True, label='h')\n",
    "    df1.iloc[:,i].hist(ax = axs[kx, ky], bins = 50, alpha=0.5, density=True, label='g')\n",
    "\n",
    "axs[0, 0].legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b) Create training and test data set. The tast data should amount to 50\\% of the total data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['class'].values\n",
    "X = df[[col for col in df.columns if col!=\"class\"]]\n",
    "\n",
    "### YOUR CODE ### \n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### c) Define the logistic regressor and fit the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "# define logistic regressor\n",
    "\n",
    "### YOUR CODE ###\n",
    "\n",
    "logreg=linear_model.LogisticRegression(fit_intercept=True,\n",
    "                                       penalty='none',\n",
    "                                       max_iter=1000,\n",
    "                                       tol=1E-5)\n",
    "\n",
    "# fit training data\n",
    "\n",
    "### YOUR CODE ###\n",
    "\n",
    "import time\n",
    "start_time = time.time()\n",
    "logreg.fit(X_train, y_train)\n",
    "run_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### d) Determine the Model Accuracy, the AUC score and the Run time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Accuracy: 78.99%\n",
      "AUC score: 0.74\n",
      "Run time: 0.22 sec\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "### YOUR CODE ###\n",
    "\n",
    "print(\"Model Accuracy: {:.2f}%\".format(100*logreg.score(X_test, y_test)))\n",
    "print(\"AUC score: {:.2f}\".format(roc_auc_score(y_test,y_pred)))\n",
    "print(\"Run time: {:.2f} sec\\n\\n\".format(run_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### e) Plot the ROC curve (Backgropund Rejection vs signal efficiency)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14108 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve\n",
    "%matplotlib inline\n",
    "\n",
    "y_pred_prob = logreg.predict_proba(X_test) # predicted probabilities\n",
    "\n",
    "### YOUR CODE ###\n",
    "\n",
    "fpr, tpr, _ = roc_curve(y_test, y_pred_prob[:,1])\n",
    "plt.plot(tpr, 1-fpr)\n",
    "plt.xlabel('Signal efficiency')\n",
    "plt.ylabel('Background Rejection');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### f) Plot the Signal efficiency vs. the Background efficiency and compare it to the corresponding plot in the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 13340 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### YOUR CODE ###\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xscale(\"log\")\n",
    "plt.xlabel('Background efficiency')\n",
    "plt.ylabel('Signal efficiency');"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
