1 changed files with 988 additions and 0 deletions
@ -0,0 +1,988 @@ |
|||
% Introduction to Data Analysis and Machine Learning in Physics: \ 1. Introduction to python |
|||
% Day 1: 11. April 2023 |
|||
% \underline{Jörg Marks}, Klaus Reygers |
|||
|
|||
## Outline of the $1^{st}$ day |
|||
|
|||
* Technical instructions for your interactions with the CIP pool, like |
|||
* using the jupyter hub |
|||
* using python locally in your own linux environment (anaconda) |
|||
* access the CIP pool from your own windows or linux system |
|||
* transfer data from and to the CIP pool |
|||
|
|||
Can be found in [\textcolor{violet}{CIPpoolAccess.PDF}](https://www.physi.uni-heidelberg.de/~marks/root_einfuehrung/Folien/CIPpoolAccess.PDF)\normalsize |
|||
|
|||
* Summary of NumPy |
|||
|
|||
* Plotting with matplotlib |
|||
|
|||
* Input / output of data |
|||
|
|||
* Summary of pandas |
|||
|
|||
* Fitting with iminuit and PyROOT |
|||
|
|||
|
|||
## A glimpse into python classes |
|||
|
|||
The following python classes are important to \textcolor{red}{data analysis and machine |
|||
learning} and will be useful during the course |
|||
|
|||
* [\textcolor{violet}{NumPy}](https://numpy.org/doc/stable/user/basics.html) - python library adding support for large, |
|||
multi-dimensional arrays and matrices, along with high-level |
|||
mathematical functions to operate on these arrays |
|||
|
|||
* [\textcolor{violet}{matplotlib}](https://matplotlib.org/stable/tutorials/index.html) - a python plotting library |
|||
|
|||
* [\textcolor{violet}{SciPy}](https://docs.scipy.org/doc/scipy/reference/tutorial/index.html) - extension of NumPy by a collection of |
|||
mathematical algorithms for minimization, regression, |
|||
fourier transformation, linear algebra and image processing |
|||
|
|||
* [\textcolor{violet}{iminuit}](https://iminuit.readthedocs.io/en/stable/) - |
|||
python wrapper to the data fitting toolkit |
|||
[\textcolor{violet}{Minuit2}](https://root.cern.ch/doc/master/Minuit2Page.html) |
|||
developed at CERN by F. James in the 1970ies |
|||
|
|||
* [\textcolor{violet}{PyROOT}](https://root.cern/manual/python/) - python wrapper to the C++ data analysis toolkit |
|||
ROOT [\textcolor{violet}{(lecture WS 2021 / 22)}](https://www.physi.uni-heidelberg.de/~marks/root_einfuehrung/) used at the LHC |
|||
|
|||
* [\textcolor{violet}{scikit-learn}](https://scikit-learn.org/stable/) - machine learning library written in |
|||
python, which makes use extensively of NumPy for high-performance |
|||
linear algebra algorithms |
|||
|
|||
## NumPy |
|||
|
|||
\textcolor{blue}{NumPy} (Numerical Python) is an open source python library, |
|||
which contains multidimensional array and matrix data structures and methods |
|||
to efficiently operate on these. The core object is |
|||
a homogeneous n-dimensional array object, \textcolor{blue}{ndarray}, which |
|||
allows for a wide variety of \textcolor{blue}{fast operations and mathematical calculations |
|||
with arrays and matrices} due to the extensive usage of compiled code. |
|||
|
|||
* It is heavily used in numerous scientific python packages |
|||
|
|||
* `ndarray` 's have a fixed size at creation $\rightarrow$ changing size |
|||
leads to recreation |
|||
|
|||
* Array elements are all required to be of the same data type |
|||
|
|||
* Facilitates advanced mathematical operations on large datasets |
|||
|
|||
* See for a summary, e.g. |
|||
\small |
|||
[\textcolor{violet}{https://cs231n.github.io/python-numpy-tutorial/\#numpy}](https://cs231n.github.io/python-numpy-tutorial/#numpy) \normalsize |
|||
|
|||
\vfill |
|||
|
|||
::: columns |
|||
:::: {.column width=30%} |
|||
|
|||
:::: |
|||
::: |
|||
|
|||
::: columns |
|||
:::: {.column width=35%} |
|||
|
|||
`c = []` |
|||
|
|||
`for i in range(len(a)):` |
|||
|
|||
`c.append(a[i]*b[i])` |
|||
|
|||
:::: |
|||
|
|||
:::: {.column width=35%} |
|||
|
|||
with NumPy |
|||
|
|||
`c = a * b` |
|||
|
|||
:::: |
|||
::: |
|||
|
|||
<!--- |
|||
It seem we need to indent by hand. |
|||
I don't manage to align under the bullet text |
|||
If we do it with column the vertical space is with code sections not good |
|||
If we do it without code section the vertical space is ok, but there is no |
|||
code high lightning. |
|||
See the different versions of the same page in the following |
|||
--> |
|||
|
|||
## NumPy - array basics (1) |
|||
|
|||
* numpy arrays build a grid of \textcolor{blue}{same type} values, which are indexed. |
|||
The *rank* is the dimension of the array. |
|||
There are methods to create and preset arrays. |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
myA = np.array([12, 5 , 11]) # create rank 1 array (vector like) |
|||
type(myA) # <class ‘numpy.ndarray’> |
|||
myA.shape # (3,) |
|||
print(myA[2]) # 11 access 3. element |
|||
myA[0] = 12 # set 1. element to 12 |
|||
myB = np.array([[1,5],[7,9]]) # create rank 2 array |
|||
myB.shape # (2,2) |
|||
print(myB[0,0],myB[0,1],myB[1,1]) # 1 5 9 |
|||
myC = np.arange(6) # create rank 1 set to 0 - 5 |
|||
myC.reshape(2,3) # change rank to (2,3) |
|||
|
|||
zero = np.zeros((2,5)) # 2 rows, 5 columns, set to 0 |
|||
one = np.ones((2,2)) # 2 rows, 2 columns, set to 1 |
|||
five = np.full((2,2), 5) # 2 rows, 2 columns, set to 5 |
|||
e = np.eye(2) # create 2x2 identity matrix |
|||
``` |
|||
\normalsize |
|||
|
|||
## NumPy - array basics (2) |
|||
|
|||
* Similar to a coordinate system numpy arrays also have \textcolor{blue}{axes}. numpy operations |
|||
can be performed along these axes. |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
```python |
|||
# 2D arrays |
|||
five = np.full((2,3), 5) # 2 rows, 3 columns, set to 5 |
|||
seven = np.full((2,3), 7) # 2 rows, 3 columns, set to 7 |
|||
np.concatenate((five,seven), axis = 0) # results in a 3 x 4 array |
|||
np.concatenate((five,seven), axis = 1]) # results in a 6 x 2 array |
|||
# 1D array |
|||
one = np.array([1, 1 , 1]) # results in a 1 x 3 array, set to 1 |
|||
four = np.array([4, 4 , 4]) # results in a 1 x 3 array, set to 4 |
|||
np.concatenate((one,four), axis = 0) # concat. arrays horizontally! |
|||
``` |
|||
:::: |
|||
:::: {.column width=50%} |
|||
\vspace{3cm} |
|||
 |
|||
:::: |
|||
::: |
|||
\normalsize |
|||
|
|||
|
|||
## NumPy - array indexing (1) |
|||
|
|||
* select slices of a numpy array |
|||
|
|||
\footnotesize |
|||
```python |
|||
a = np.array([[1,2,3,4], |
|||
[5,6,7,8], # 3 rows 4 columns array |
|||
[9,10,11,12]]) |
|||
b = a[:2, 1:3] # subarray of 2 rows and |
|||
array([[2, 3], # column 1 and 2 |
|||
[6, 7]]) |
|||
``` |
|||
\normalsize |
|||
|
|||
* a slice of an array points into the same data, *modifying* changes the original array! |
|||
|
|||
\footnotesize |
|||
```python |
|||
b[0, 0] = 77 # b[0,0] and a[0,1] are 77 |
|||
|
|||
r1_row = a[1, :] # get 2nd row -> rank 1 |
|||
r1_row.shape # (4,) |
|||
r2_row = a[1:2, :] # get 2nd row -> rank 2 |
|||
r2_row.shape # (1,4) |
|||
a=np.array([[1,2],[3,4],[5,6]]) # set a , 3 rows 2 cols |
|||
d=a[[0, 1, 2], [0, 1, 1]] # d contains [1 4 6] |
|||
e=a[[1, 2], [1, 1]] # e contains [4 6] |
|||
np.array([a[0,0],a[1,1],a[2,0]]) # address elements explicitly |
|||
``` |
|||
\normalsize |
|||
|
|||
|
|||
## NumPy - array indexing (2) |
|||
|
|||
|
|||
* integer array indexing by setting an array of indices $\rightarrow$ selecting/changing elements |
|||
|
|||
\footnotesize |
|||
```python |
|||
a = np.array([[1,2,3,4], |
|||
[5,6,7,8], # 3 rows 4 columns array |
|||
[9,10,11,12]]) |
|||
p_a = np.array([0,2,0]) # Create an array of indices |
|||
s = a[np.arange(3), p_a] # number the rows, p_a points to cols |
|||
print (s) # s contains [1 7 9] |
|||
a[np.arange(3),p_a] += 10 # add 10 to corresponding elements |
|||
x=np.array([[8,2],[7,4]]) # create 2x2 array |
|||
bool = (x > 5) # bool : array of boolians |
|||
# [[True False] |
|||
# [True False]] |
|||
print(x[x>5]) # select elements, prints [8 7] |
|||
``` |
|||
\normalsize |
|||
|
|||
* data type in numpy - create according to input numbers or set explicitly |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
x = np.array([1.1, 2.1]) # create float array |
|||
print(x.dtype) # print float64 |
|||
y=np.array([1.1,2.9],dtype=np.int64) # create float array [1 2] |
|||
``` |
|||
\normalsize |
|||
|
|||
|
|||
## NumPy - functions |
|||
|
|||
* math functions operate elementwise either as operator overload or as methods |
|||
|
|||
\footnotesize |
|||
```python |
|||
x=np.array([[1,2],[3,4]],dtype=np.float64) # define 2x2 float array |
|||
y=np.array([[3,1],[5,1]],dtype=np.float64) # define 2x2 float array |
|||
s = x + y # elementwise sum |
|||
s = np.add(x,y) |
|||
s = np.subtract(x,y) |
|||
s = np.multiply(x,y) # no matrix multiplication! |
|||
s = np.divide(x,y) |
|||
s = np.sqrt(x), np.exp(x), ... |
|||
x @ y , or np.dot(x, y) # matrix product |
|||
np.sum(x, axis=0) # sum of each column |
|||
np.sum(x, axis=1) # sum of each row |
|||
xT = x.T # transpose of x |
|||
x = np.linspace(0,2*pi,100) # get equal spaced points in x |
|||
|
|||
r = np.random.default_rng(seed=42) # constructor random number class |
|||
b = r.random((2,3)) # random 2x3 matrix |
|||
``` |
|||
\normalsize |
|||
|
|||
|
|||
|
|||
## |
|||
|
|||
* broadcasting in numpy |
|||
\vspace{0.4cm} |
|||
|
|||
The term \textcolor{blue}{broadcasting} describes how numpy treats arrays |
|||
with different shapes during arithmetic operations |
|||
|
|||
* add a scalar $b$ to a 1D array $a = [a_1,a_2,a_3]$ $\rightarrow$ expand $b$ to |
|||
$[b,b,b]$ |
|||
\vspace{0.2cm} |
|||
|
|||
* add a scalar $b$ to a 2D [2,3] array $a =[[a_{11},a_{12},a_{13}],[a_{21},a_{22},a_{23}]]$ |
|||
$\rightarrow$ expand $b$ to $b =[[b,b,b],[b,b,b]]$ and add element wise |
|||
\vspace{0.2cm} |
|||
|
|||
* add 1D array $b = [b_1,b_2,b_3]$ to a 2D [2,3] array $a=[[a_{11},a_{12},a_{13}],[a_{21},a_{22},a_{23}]]$ $\rightarrow$ 1D array is broadcast |
|||
across each row of the 2D array $b =[[b_1,b_2,b_3],[b_1,b_2,b_3]]$ and added element wise |
|||
\vspace{0.2cm} |
|||
|
|||
Arithmetic operations can only be performed when the shape of each |
|||
dimension in the arrays are equal or one has the dimension size of 1. Look |
|||
[\textcolor{violet}{here}](https://numpy.org/doc/stable/user/basics.broadcasting.html) for more details |
|||
|
|||
\footnotesize |
|||
```python |
|||
# Add a vector to each row of a matrix |
|||
x = np.array([[1,2,3], [4,5,6]]) # x has shape (2, 3) |
|||
v = np.array([1,2,3]) # v has shape (3,) |
|||
x + v # [[2 4 6] |
|||
# [5 7 9]] |
|||
``` |
|||
\normalsize |
|||
|
|||
## Plot data |
|||
|
|||
A popular library to present data is the `pyplot` module of `matplotlib`. |
|||
|
|||
* Drawing a function in one plot |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
```python |
|||
import numpy as np |
|||
import matplotlib.pyplot as plt |
|||
# generate 100 points from 0 to 2 pi |
|||
x = np.linspace( 0, 10*np.pi, 100 ) |
|||
f = np.sin(x)**2 |
|||
# plot function |
|||
plt.plot(x,f,'blueviolet',label='sine') |
|||
plt.xlabel('x [radian]') |
|||
plt.ylabel('f(x)') |
|||
plt.title('Plot sin^2') |
|||
plt.legend(loc='upper right') |
|||
plt.axis([0,30,-0.1,1.2]) # limit the plot range |
|||
|
|||
# show the plot |
|||
plt.show() |
|||
``` |
|||
:::: |
|||
:::: {.column width=40%} |
|||
 |
|||
:::: |
|||
::: |
|||
|
|||
\normalsize |
|||
|
|||
## |
|||
* Drawing a scatter plot of data |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
```python |
|||
... |
|||
|
|||
# create x,y data points |
|||
num = 75 |
|||
x = range(num) |
|||
y = range(num) + np.random.randint(0,num/1.5,num) |
|||
z = - (range(num) + np.random.randint(0,num/3,num)) + num |
|||
# create colored scatter plot, sample 1 |
|||
plt.scatter(x, y, color = 'green', |
|||
label='Sample 1') |
|||
# create colored scatter plot, sample 2 |
|||
plt.scatter(x, z, color = 'orange', |
|||
label='Sample 2') |
|||
plt.title('scatter plot') |
|||
plt.xlabel('x') |
|||
plt.ylabel('y') |
|||
# description and plot |
|||
plt.legend() |
|||
plt.show() |
|||
``` |
|||
:::: |
|||
:::: {.column width=35%} |
|||
\vspace{3cm} |
|||
 |
|||
:::: |
|||
::: |
|||
\normalsize |
|||
|
|||
## |
|||
* Drawing a histogram of data |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
```python |
|||
... |
|||
|
|||
# create normalized gaussian Distribution |
|||
g = np.random.normal(size=10000) |
|||
# histogram the data |
|||
plt.hist(g,bins=40) |
|||
# plot rotated histogram |
|||
plt.hist(g,bins=40,orientation='horizontal') |
|||
# normalize area to 1 |
|||
plt.hist(g,bins=40,density=True) |
|||
# change color |
|||
plt.hist(g,bins=40,density=True, |
|||
edgecolor='lightgreen',color='orange') |
|||
plt.title('Gaussian Histogram') |
|||
plt.xlabel('bin') |
|||
plt.ylabel('entries') |
|||
# description and plot |
|||
plt.legend(['Normalized distribution']) |
|||
plt.show() |
|||
``` |
|||
:::: |
|||
:::: {.column width=35%} |
|||
\vspace{3.5cm} |
|||
 |
|||
:::: |
|||
::: |
|||
\normalsize |
|||
|
|||
## |
|||
* Drawing subplots in one canvas |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
```python |
|||
... |
|||
g = np.exp(-0.2*x) |
|||
# create figure |
|||
plt.figure(num=2,figsize=(10.0,7.5),dpi=150,facecolor='lightgrey') |
|||
plt.suptitle('1 x 2 Plot') |
|||
# create subplot and plot first one |
|||
plt.subplot(1,2,1) |
|||
# plot first one |
|||
plt.title('exp(x)') |
|||
plt.xlabel('x') |
|||
plt.ylabel('g(x)') |
|||
plt.plot(x,g,'blueviolet') |
|||
# create subplot and plot second one |
|||
plt.subplot(1,2,2) |
|||
plt.plot(x,f,'orange') |
|||
plt.plot(x,f*g,'red') |
|||
plt.legend(['sine^2','exp*sine']) |
|||
# show the plot |
|||
plt.show() |
|||
``` |
|||
:::: |
|||
:::: {.column width=40%} |
|||
\vspace{3cm} |
|||
 |
|||
:::: |
|||
::: |
|||
\normalsize |
|||
|
|||
## Image data |
|||
|
|||
The `image` class of the `matplotlib` library can be used to load the image |
|||
to numpy arrays and to render the image. |
|||
|
|||
* There are 3 common formats for the numpy array |
|||
|
|||
* (M, N) scalar data used for greyscale images |
|||
|
|||
* (M, N, 3) for RGB images (each pixel has an array with RGB color attached) |
|||
|
|||
* (M, N, 4) for RGBA images (each pixel has an array with RGB color |
|||
and transparency attached) |
|||
|
|||
|
|||
The method `imread` loads the image into an `ndarray`, which can be |
|||
manipulated. |
|||
|
|||
The method `imshow` renders the image data |
|||
|
|||
\vspace {2cm} |
|||
|
|||
## |
|||
* Drawing pixel data and images |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=50%} |
|||
|
|||
```python |
|||
.... |
|||
# create data array with pixel postion and RGB color code |
|||
width, height = 200, 200 |
|||
data = np.zeros((height, width, 3), dtype=np.uint8) |
|||
# red patch in the center |
|||
data[75:125, 75:125] = [255, 0, 0] |
|||
x = np.random.randint(0,width-1,100) |
|||
y = np.random.randint(0,height-1,100) |
|||
data[x,y]= [0,255,0] # 100 random green pixel |
|||
plt.imshow(data) |
|||
plt.show() |
|||
.... |
|||
import matplotlib.image as mpimg |
|||
#read image into numpy array |
|||
pic = mpimg.imread('picture.jpg') |
|||
mod_pic = pic[:,:,0] # grab slice 0 of the colors |
|||
plt.imshow(mod_pic) # use default color code also |
|||
plt.colorbar() # try cmap='hot' |
|||
plt.show() |
|||
``` |
|||
:::: |
|||
:::: {.column width=25%} |
|||
 |
|||
\vspace{1cm} |
|||
 |
|||
:::: |
|||
::: |
|||
\normalsize |
|||
|
|||
|
|||
## Input / output |
|||
|
|||
For the analysis of measured data efficient input \/ output plays an |
|||
important role. In numpy, `ndarrays` can be saved and read in from files. |
|||
`load()` and `save()` functions handle numpy binary files (.npy extension) |
|||
which contain data, shape, dtype and other information required to |
|||
reconstruct the `ndarray` of the disk file. |
|||
|
|||
\footnotesize |
|||
```python |
|||
r = np.random.default_rng() # instanciate random number generator |
|||
a = r.random((4,3)) # random 4x3 array |
|||
np.save('myBinary.npy', a) # write array a to binary file myBinary.npy |
|||
b = np.arange(12) |
|||
np.savez('myComp.npz', a=a, b=b) # write a and b in compressed binary file |
|||
...... |
|||
b = np.load('myBinary.npy') # read content of myBinary.npy into b |
|||
``` |
|||
\normalsize |
|||
|
|||
The storage and retrieval of array data in text file format is done |
|||
with `savetxt()` and `loadtxt()` methods. Parameter controlling delimiter, |
|||
line separators, file header and footer can be specified. |
|||
|
|||
\footnotesize |
|||
```python |
|||
x = np.array([1,2,3,4,5,6,7]) # create ndarray |
|||
np.savetxt('myText.txt',x,fmt='%d', delimiter=',') # write array x to file myText.txt |
|||
# with comma separation |
|||
``` |
|||
\normalsize |
|||
|
|||
|
|||
## Input / output |
|||
|
|||
Import tabular data from table processing programs in office packages. |
|||
|
|||
\vspace{0.4cm} |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
`Excel data` can be exported as text file (myData_01.csv) with a comma as |
|||
delimiter. |
|||
:::: |
|||
:::: {.column width=35%} |
|||
 |
|||
:::: |
|||
::: |
|||
|
|||
\footnotesize |
|||
```python |
|||
..... |
|||
# read content of all files myData_*.csv into data |
|||
data = np.loadtxt('myData_01.csv',dtype=int,delimiter=',') |
|||
|
|||
print (data.shape) # (12, 9) |
|||
print (data) # [[1 1 1 1 0 0 0 0 0] |
|||
# [0 0 1 1 0 0 1 1 0] |
|||
# ..... |
|||
# [0 0 0 0 1 1 1 1 1]] |
|||
|
|||
``` |
|||
\normalsize |
|||
|
|||
## Input / output |
|||
|
|||
Import tabular data from table processing programs in office packages. |
|||
|
|||
\vspace{0.4cm} |
|||
|
|||
\footnotesize |
|||
::: columns |
|||
:::: {.column width=35%} |
|||
`Excel data` can be exported as text file (myData_01.csv) with a comma as |
|||
delimiter. \newline |
|||
$\color{blue}{Often~many~files~are~available~(myData\_*.csv)}$ |
|||
:::: |
|||
:::: {.column width=35%} |
|||
 |
|||
:::: |
|||
::: |
|||
|
|||
\footnotesize |
|||
```python |
|||
..... |
|||
# find files and directories with names matching a pattern |
|||
import glob |
|||
# read content of all files myData_*.csv into data |
|||
file_list = sorted(glob.glob('myData_*.csv')) # generate a sorted file list |
|||
for filename in file_list: |
|||
data = np.loadtxt(fname=filename, dtype=int, delimiter=',') |
|||
print(data[:,3]) # print column 3 of each file |
|||
# [1 1 1 1 1 1 1 1 1 1 1 0] |
|||
# ...... |
|||
# [0 1 0 1 0 1 0 1 0 1 0 1] |
|||
|
|||
``` |
|||
\normalsize |
|||
|
|||
## Exercise 1 |
|||
|
|||
i) Display a numpy array as figure of a blue cross. The size should be 200 |
|||
by 200 pixel. Use as array format (M, N, 3), where the first 2 specify |
|||
the pixel positions and the last 3 the rbg color from 0:255. |
|||
- Draw in addition a red square of arbitrary position into the figure. |
|||
- Draw a circle in the center of the figure. Try to create a mask which |
|||
selects the inner part of the circle using the indexing. |
|||
|
|||
\small |
|||
[Solution: 01_intro_ex_1a_sol.ipynb](https://www.physi.uni-heidelberg.de/~reygers/lectures/2022/ml/solutions/01_intro_ex_1a_sol.ipynb) \normalsize |
|||
|
|||
ii) Read data which contains pixels from the binary file horse.py into a |
|||
numpy array. Display the data and the following transformations in 4 |
|||
subplots: scaling and translation, compression in x and y, rotation |
|||
and mirroring. |
|||
|
|||
\small |
|||
[Solution: 01_intro_ex_1b_sol.ipynb](https://www.physi.uni-heidelberg.de/~reygers/lectures/2022/ml/solutions/01_intro_ex_1b_sol.ipynb) \normalsize |
|||
|
|||
|
|||
## Pandas |
|||
|
|||
[\textcolor{violet}{pandas}](https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html) is a software library written in python for |
|||
\textcolor{blue}{data manipulation and analysis}. |
|||
|
|||
\vspace{0.4cm} |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* Offers data structures and operations for manipulating numerical tables with |
|||
integrated indexing |
|||
|
|||
* Imports data from various file formats, e.g. comma-separated values, JSON, |
|||
SQL or Excel |
|||
|
|||
* Tools for reading and writing data structures, allows analyzing, filtering, |
|||
spliting, grouping and aggregating, merging and joining and plotting |
|||
|
|||
* Built on top of `NumPy` |
|||
|
|||
* Visualize the data with `matplotlib` |
|||
|
|||
* Most machine learning tools support `pandas` $\rightarrow$ |
|||
it is widely used to preprocess data sets for analysis and machine learning |
|||
in various scientific fields |
|||
|
|||
## Pandas micro introduction |
|||
|
|||
Goal: Exploring, cleaning, transforming, and visualization of data. |
|||
The basic indexable objects are |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* `Series` -> vector (list) of data elements of arbitrary type |
|||
|
|||
* `DataFrame` -> tabular arangement of data elements of column wise |
|||
arbitrary type |
|||
|
|||
Both allow cleaning data by removing of `empty` or `nan` data entries |
|||
|
|||
\footnotesize |
|||
```python |
|||
import numpy as np |
|||
import pandas as pd # use together with numpy |
|||
s = pd.Series([1, 3, 5, np.nan, 6, 8]) # create a Series of int64 |
|||
r = pd.Series(np.random.randn(4)) # Series of random numbers float64 |
|||
dates = pd.date_range("20130101", periods=3) # index according to dates |
|||
df = pd.DataFrame(np.random.randn(3,4),index=dates,columns=list("ABCD")) |
|||
print (df) # print the DataFrame |
|||
A B C D |
|||
2013-01-01 1.618395 1.210263 -1.276586 -0.775545 |
|||
2013-01-02 0.676783 -0.754161 -1.148029 -0.244821 |
|||
2013-01-03 -0.359081 0.296019 1.541571 0.235337 |
|||
|
|||
new_s = s.dropna() # return a new Data Frame without the column that has NaN cells |
|||
``` |
|||
\normalsize |
|||
|
|||
## |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* pandas data can be saved in different file formats (CSV, JASON, html, XML, |
|||
Excel, OpenDocument, HDF5 format, .....). `NaN` entries are kept |
|||
in the output file, except if they are removed with `dataframe.dropna()` |
|||
|
|||
* csv file |
|||
\footnotesize |
|||
```python |
|||
df.to_csv("myFile.csv") # Write the DataFrame df to a csv file |
|||
``` |
|||
\normalsize |
|||
|
|||
* HDF5 output |
|||
|
|||
\footnotesize |
|||
```python |
|||
df.to_hdf("myFile.h5",key='df',mode='w') # Write the DataFrame df to HDF5 |
|||
s.to_hdf("myFile.h5", key='s',mode='a') |
|||
``` |
|||
\normalsize |
|||
|
|||
* Writing to an excel file |
|||
|
|||
\footnotesize |
|||
```python |
|||
df.to_excel("myFile.xlsx", sheet_name="Sheet1") |
|||
``` |
|||
\normalsize |
|||
|
|||
* Deleting file with data in python |
|||
|
|||
\footnotesize |
|||
```python |
|||
import os |
|||
os.remove('myFile.h5') |
|||
``` |
|||
\normalsize |
|||
|
|||
## |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* read in data from various formats |
|||
|
|||
* csv file |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
....... |
|||
df = pd.read_csv('heart.csv') # read csv data table |
|||
print(df.info()) |
|||
<class 'pandas.core.frame.DataFrame'> |
|||
RangeIndex: 303 entries, 0 to 302 |
|||
Data columns (total 14 columns): |
|||
# Column Non-Null Count Dtype |
|||
--- ------ -------------- ----- |
|||
0 age 303 non-null int64 |
|||
1 sex 303 non-null int64 |
|||
2 cp 303 non-null int64 |
|||
print(df.head(5)) # prints the first 5 rows of the data table |
|||
print(df.describe()) # shows a quick statistic summary of your data |
|||
``` |
|||
\normalsize |
|||
|
|||
* Reading an excel file |
|||
|
|||
\footnotesize |
|||
```python |
|||
df = pd.read_excel("myFile.xlsx","Sheet1", na_values=["NA"]) |
|||
``` |
|||
\normalsize |
|||
|
|||
\textcolor{olive}{There are many options specifying details for IO.} |
|||
|
|||
## |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* Various functions exist to select and view data from pandas objects |
|||
|
|||
* Display column and index |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df.index # show datetime index of df |
|||
DatetimeIndex(['2013-01-01','2013-01-02','2013-01-03'], |
|||
dtype='datetime64[ns]',freq='D') |
|||
df.column # show columns info |
|||
Index(['A', 'B', 'C', 'D'], dtype='object') |
|||
``` |
|||
\normalsize |
|||
|
|||
* `DataFrame.to_numpy()` gives a `NumPy` representation of the underlying data |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df.to_numpy() # one dtype for the entire array, not per column! |
|||
[[-0.62660101 -0.67330526 0.23269168 -0.67403546] |
|||
[-0.53033339 0.32872063 -0.09893568 0.44814084] |
|||
[-0.60289996 -0.22352548 -0.43393248 0.47531456]] |
|||
``` |
|||
\normalsize |
|||
|
|||
Does not include the index or column labels in the output |
|||
|
|||
* more on viewing |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df.T # transpose the DataFrame df |
|||
df.sort_values(by="B") # Sorting by values of column B of df |
|||
df.sort_index(axis=0) # Sorting by index ascending values |
|||
df.sort_index(axis=0,ascending=False) # Display columns in inverse order |
|||
|
|||
``` |
|||
\normalsize |
|||
|
|||
## |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* Selecting data of pandas objects $\rightarrow$ keep or reduce dimensions |
|||
|
|||
* get a named column as a Series |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df["A"] # selects a column A from df, simular to df.A |
|||
df.iloc[:, 1:2] # slices column A explicitly from df, df.loc[:, ["A"]] |
|||
``` |
|||
\normalsize |
|||
|
|||
* select rows of a DataFrame |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df[0:2] # selects row 0 and 1 from df, |
|||
df["20130102":"20130103"] # use indices, endpoints are included! |
|||
df.iloc[3] # select with the position of the passed integers |
|||
df.iloc[1:3, :] # selects row 1 and 2 from df |
|||
``` |
|||
\normalsize |
|||
|
|||
* select by label |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df.loc["20130102":"20130103",["C","D"]] # selects row 1 and 2 and only C and D |
|||
df.loc[dates[0], "A"] # selects a single value (scalar) |
|||
``` |
|||
\normalsize |
|||
|
|||
* select by lists of integer position (as in `NumPy`) |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df.iloc[[0, 2], [1, 3]] # select row 1 and 3 and col B and D (data only) |
|||
df.iloc[1, 1] # get a value explicitly (data only, no index lines) |
|||
|
|||
``` |
|||
\normalsize |
|||
|
|||
* select according to expressions |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df.query('B<C') # select rows where B < C |
|||
df1=df[(df["B"]==0)&(df["D"]==0)] # conditions on rows |
|||
``` |
|||
\normalsize |
|||
|
|||
## |
|||
|
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}\tiny$\blacksquare$} |
|||
|
|||
* Selecting data of pandas objects continued |
|||
|
|||
* Boolean indexing |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df[df["A"] > 0] # select df where all values of column A are >0 |
|||
df[df > 0] # select values >0 from the entire DataFrame |
|||
``` |
|||
\normalsize |
|||
|
|||
more complex example |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df2 = df.copy() # copy df |
|||
df2["E"] = ["eight","one","four"] # add column E |
|||
df2[df2["E"].isin(["two", "four"])] # test if elements "two" and "four" are |
|||
# contained in Series column E |
|||
``` |
|||
\normalsize |
|||
|
|||
* Operations (in general exclude missing data) |
|||
|
|||
\footnotesize |
|||
|
|||
```python |
|||
df2[df2 > 0] = -df2 # All elements > 0 change sign |
|||
df.mean(0) # get column wise mean (numbers=axis) |
|||
df.mean(1) # get row wise mean |
|||
df.std(0) # standard deviation according to axis |
|||
df.cumsum() # cumulative sum of each column |
|||
df.apply(np.sin) # apply function to each element of df |
|||
df.apply(lambda x: x.max() - x.min()) # apply lambda function column wise |
|||
df + 10 # add scalar 10 |
|||
df - [1, 2, 10 , 100] # subtract values of each column |
|||
df.corr() # Compute pairwise correlation of columns |
|||
``` |
|||
\normalsize |
|||
|
|||
|
|||
## Pandas - plotting data |
|||
|
|||
[\textcolor{violet}{Visualization}](https://pandas.pydata.org/pandas-docs/stable/user_guide/visualization.html) is integrated in pandas using matplotlib. Here are only 2 examples |
|||
|
|||
* Plot random data in histogramm and scatter plot |
|||
|
|||
\footnotesize |
|||
```python |
|||
# create DataFrame with random normal distributed data |
|||
df = pd.DataFrame(np.random.randn(1000,4),columns=["a","b","c","d"]) |
|||
df = df + [1, 3, 8 , 10] # shift column wise mean by 1, 3, 8 , 10 |
|||
df.plot.hist(bins=20) # histogram all 4 columns |
|||
g1 = df.plot.scatter(x="a",y="c",color="DarkBlue",label="Group 1") |
|||
df.plot.scatter(x="b",y="d",color="DarkGreen",label="Group 2",ax=g1) |
|||
``` |
|||
\normalsize |
|||
|
|||
::: columns |
|||
:::: {.column width=35%} |
|||
 |
|||
:::: |
|||
:::: {.column width=35%} |
|||
 |
|||
:::: |
|||
::: |
|||
|
|||
## Pandas - plotting data |
|||
|
|||
The function crosstab() takes one or more array-like objects as indexes or |
|||
columns and constructs a new DataFrame of variable counts on the inputs |
|||
|
|||
\footnotesize |
|||
```python |
|||
df = pd.DataFrame( # create DataFrame of 2 categories |
|||
{"sex": np.array([0,0,0,0,1,1,1,1,0,0,0]), |
|||
"heart": np.array([1,1,1,0,1,1,1,0,0,0,1]) |
|||
} ) # closing bracket goes on next line |
|||
pd.crosstab(df2.sex,df2.heart) # create cross table of possibilities |
|||
pd.crosstab(df2.sex,df2.heart).plot(kind="bar",color=['red','blue']) # plot counts |
|||
``` |
|||
\normalsize |
|||
::: columns |
|||
:::: {.column width=38%} |
|||
 |
|||
:::: |
|||
::: |
|||
|
|||
## Exercise 2 |
|||
|
|||
Read the file [\textcolor{violet}{heart.csv}](https://www.physi.uni-heidelberg.de/~reygers/lectures/2023/ml/exercises/heart.csv) into a DataFrame. |
|||
[\textcolor{violet}{Information on the dataset}](https://archive.ics.uci.edu/ml/datasets/heart+Disease) |
|||
|
|||
\setbeamertemplate{itemize item}{\color{red}$\square$} |
|||
|
|||
* Which columns do we have |
|||
|
|||
* Print the first 3 rows |
|||
|
|||
* Print the statistics summary and the correlations |
|||
|
|||
* Print mean values for each column with and without disease (target) |
|||
|
|||
* Select the data according to `sex` and `target` (heart disease 0=no 1=yes). |
|||
|
|||
* Plot the `age` distribution of male and female in one histogram |
|||
|
|||
* Plot the heart disease distribution according to chest pain type `cp` |
|||
|
|||
* Plot `thalach` according to `target` in one histogramm |
|||
|
|||
* Plot `sex` and `target` in a histogramm figure |
|||
|
|||
* Correlate `age` and `max heart rate` according to `target` |
|||
|
|||
* Correlate `age` and `colesterol` according to `target` |
|||
|
|||
\small |
|||
[Solution: 01_intro_ex_2_sol.ipynb](https://www.physi.uni-heidelberg.de/~reygers/lectures/2023/ml/solutions/01_intro_ex_2_sol.ipynb) \normalsize |
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
Write
Preview
Loading…
Cancel
Save
Reference in new issue