{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on: Separation of gamma and hadron showers measured with the MAGIC Cherenkov telescope using a boosted decision tree and a random forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [MAGIC telescope](https://en.wikipedia.org/wiki/MAGIC_(telescope) is a Cherenkov telescope situated on La Palma, one of the Canary Islands. The [MAGIC machine learning dataset](https://archive.ics.uci.edu/ml/datasets/magic+gamma+telescope) can be obtained from [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php).\n",
    "\n",
    "Our task is to separate signal events (gamma showers) and background events (hadron showers) based on the features of a measured Cherenkov shower.\n",
    "\n",
    "The features of a shower are:\n",
    "\n",
    "    1.  fLength:  continuous  # major axis of ellipse [mm]\n",
    "    2.  fWidth:   continuous  # minor axis of ellipse [mm] \n",
    "    3.  fSize:    continuous  # 10-log of sum of content of all pixels [in #phot]\n",
    "    4.  fConc:    continuous  # ratio of sum of two highest pixels over fSize  [ratio]\n",
    "    5.  fConc1:   continuous  # ratio of highest pixel over fSize  [ratio]\n",
    "    6.  fAsym:    continuous  # distance from highest pixel to center, projected onto major axis [mm]\n",
    "    7.  fM3Long:  continuous  # 3rd root of third moment along major axis  [mm] \n",
    "    8.  fM3Trans: continuous  # 3rd root of third moment along minor axis  [mm]\n",
    "    9.  fAlpha:   continuous  # angle of major axis with vector to origin [deg]\n",
    "    10. fDist:    continuous  # distance from origin to center of ellipse [mm]\n",
    "    11. class:    g,h         # gamma (signal), hadron (background)\n",
    "\n",
    "g = gamma (signal):     12332\n",
    "h = hadron (background): 6688\n",
    "\n",
    "For technical reasons, the number of h events is underestimated.\n",
    "In the real data, the h class represents the majority of the events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read data\n",
    "filename = \"https://www.physi.uni-heidelberg.de/~reygers/lectures/2021/ml/data/magic04_data.txt\"\n",
    "df = pd.read_csv(filename, engine='python')\n",
    "\n",
    "# relabel: gamma shower (g) --> 1 (signal), hadron shower (h) --> 0 (background) \n",
    "df['class'] = df['class'].map({'g': 1, 'h': 0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# y = value to predict, X = features\n",
    "y = df['class'].values\n",
    "X = df[[col for col in df.columns if col!=\"class\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate training and test samples\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use [XGBoost](https://xgboost.readthedocs.io/en/latest/). The training will take a few seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train XGBoost boosted decision tree\n",
    "import xgboost as xgb\n",
    "import time\n",
    "XGBclassifier = xgb.sklearn.XGBClassifier(nthread=-1, seed=1, n_estimators=1000)\n",
    "start_time = time.time()\n",
    "XGBclassifier.fit(X_train, y_train)\n",
    "run_time = time.time() - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predect labels for the test sample\n",
    "y_pred_xgb = XGBclassifier.predict(X_test) # 0 or 1\n",
    "y_pred_xgb_prob = XGBclassifier.predict_proba(X_test) # probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Accuracy: 89.54%\n",
      "AUC score: 0.87\n",
      "Run time: 34.36 sec\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# print some performance parameters\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print(\"Model Accuracy: {:.2f}%\".format(100*XGBclassifier.score(X_test, y_test)))\n",
    "print(\"AUC score: {:.2f}\".format(roc_auc_score(y_test,y_pred_xgb)))\n",
    "print(\"Run time: {:.2f} sec\\n\\n\".format(run_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a) Plot predicted probabilities for the test sample for signal and background events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15156 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot signal efficiency vs. background efficiency\n",
    "# we want the signal efficiency to be large and the background efficiency to be small\n",
    "\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "fpr_xgb, tpr_xgb, _ = roc_curve(y_test, y_pred_xgb_prob[:,1])\n",
    "plt.plot(fpr_xgb, tpr_xgb)\n",
    "plt.xscale(\"log\")\n",
    "plt.xlabel('Background efficiency', fontsize=18)\n",
    "plt.ylabel('Signal efficiency', fontsize=18);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b) Which is the most important feature for discriminating signal and background according to XGBoost? Hint: use plot_impartance from XGBoost (see [XGBoost plotting API](https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.plotting)). Do you get the same answer for all three performance measures provided by XGBoost (“weight”, “gain”, or “cover”)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here (one line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c) Visualize one decision tree from the ensemble (let's say tree number 10). For this you need the the graphviz package (`pip3 install graphviz`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here (one line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d) Compare the performance of XGBoost with the [**random forest classifier**](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) from [**scikit learn**](https://scikit-learn.org/stable/index.html). Plot signal and background efficiency for both classifiers in one plot. Which classifier performs better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "RFclassifier = RandomForestClassifier(random_state=0)\n",
    "\n",
    "# your code here\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
