{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to index, slice, modify, and delete data points" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Very usually, we need to select, modify and even delete the data of certain shots. Here there is a short introduction on how to do that.\n", "\n", "Let us start with importing the supporting packages and loading the data. In order to keep things ealier, here we will import all the packages, but not all of them will be us" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load some example data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import supporting packages" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Set the system path for importing packages\n", "# This is just because I put all example scripts in another folder\n", "# You DO NOT need to do this \n", "# -------------- You do NOT need following part --------------\n", "import sys\n", "import os\n", "sys.path.insert(0, os.path.abspath('..'))\n", "# -------------- You do NOT need above part --------------\n", "\n", "import copy\n", "import glob\n", "from datetime import datetime\n", "\n", "# The package for data structure\n", "import xarray as xr\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# The packages for working with uncertainties\n", "from uncertainties import ufloat\n", "from uncertainties import unumpy as unp\n", "from uncertainties import umath\n", "\n", "# The package for plotting\n", "import matplotlib.pyplot as plt\n", "plt.rcParams['font.size'] = 18 # Set the global font size\n", "\n", "# -------------- The modules written by us --------------\n", "\n", "# The packages for read data\n", "from DataContainer.ReadData import read_hdf5_file, read_hdf5_global, read_hdf5_run_time, read_csv_file\n", "\n", "# The packages for data analysis\n", "from Analyser.ImagingAnalyser import ImageAnalyser\n", "from Analyser.FitAnalyser import FitAnalyser\n", "from Analyser.FitAnalyser import ThomasFermi2dModel, DensityProfileBEC2dModel, Polylog22dModel\n", "from Analyser.FFTAnalyser import fft, ifft, fft_nutou\n", "from ToolFunction.ToolFunction import *\n", "\n", "# Add errorbar plot to xarray package\n", "from ToolFunction.HomeMadeXarrayFunction import errorbar, dataarray_plot_errorbar\n", "xr.plot.dataarray_plot.errorbar = errorbar\n", "xr.plot.accessor.DataArrayPlotAccessor.errorbar = dataarray_plot_errorbar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Start a client for parallel computing" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-b0e6c9ce-2097-11ee-9d7c-80e82ce2fa8e

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Cluster objectCluster type: distributed.LocalCluster
\n", " Dashboard: http://127.0.0.1:8787/status\n", "
\n", "\n", " \n", "\n", " \n", "
\n", "

Cluster Info

\n", "
\n", "
\n", "
\n", "
\n", "

LocalCluster

\n", "

2c85baf6

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Workers: 6\n", "
\n", " Total threads: 60\n", " \n", " Total memory: 55.88 GiB\n", "
Status: runningUsing processes: True
\n", "\n", "
\n", " \n", "

Scheduler Info

\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-3edf5152-70c9-41bc-b2e3-dd1cacfc5ef2

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: tcp://127.0.0.1:62125\n", " \n", " Workers: 6\n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Total threads: 60\n", "
\n", " Started: Just now\n", " \n", " Total memory: 55.88 GiB\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", "
\n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 0

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:62161\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:62163/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:62128\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-doscq09y\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 1

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:62167\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:62168/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:62129\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-7b8nhgmg\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 2

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:62156\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:62159/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:62130\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-y1tkfpfy\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 3

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:62162\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:62165/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:62131\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-613gqycm\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 4

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:62150\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:62153/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:62132\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-nf2w0fki\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 5

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:62155\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:62157/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:62133\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-xm8jsrrg\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from dask.distributed import Client\n", "client = Client(n_workers=6, threads_per_worker=10, processes=True, memory_limit='10GB')\n", "client" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set the path for different cameras" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "groupList = [\n", " \"images/MOT_3D_Camera/in_situ_absorption\",\n", " \"images/ODT_1_Axis_Camera/in_situ_absorption\",\n", " \"images/ODT_2_Axis_Camera/in_situ_absorption\",\n", "]\n", "\n", "# give a short name to each path (or let's say each camera)\n", "dskey = {\n", " \"images/MOT_3D_Camera/in_situ_absorption\": \"camera_0\",\n", " \"images/ODT_1_Axis_Camera/in_situ_absorption\": \"camera_1\",\n", " \"images/ODT_2_Axis_Camera/in_situ_absorption\": \"camera_2\",\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set global path for experiment" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "img_dir = '//DyLabNAS/Data/'\n", "SequenceName = \"Evaporative_Cooling\" + \"/\"\n", "folderPath = img_dir + SequenceName + '2023/04/17'# get_date()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load shot 0058" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n",
       "    background        (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n",
       "    dark              (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n",
       "    shotNum           (runs, truncation_value) <U2 '00' '01' '02' ... '53' '54'\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n", " background (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n", " dark (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n", " shotNum (runs, truncation_value) \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray 'shotNum' (runs: 5, truncation_value: 11)>\n",
       "array([['-1', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10'],\n",
       "       ['11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21'],\n",
       "       ['22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32'],\n",
       "       ['33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43'],\n",
       "       ['44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54']],\n",
       "      dtype='<U2')\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0
" ], "text/plain": [ "\n", "array([['-1', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10'],\n", " ['11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21'],\n", " ['22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32'],\n", " ['33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43'],\n", " ['44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54']],\n", " dtype='\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray 'shotNum' (runs: 5, truncation_value: 11)>\n",
       "array([['-1', '-2', '02', '03', '04', '05', '06', '07', '08', '09', '10'],\n",
       "       ['11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21'],\n",
       "       ['22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32'],\n",
       "       ['33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43'],\n",
       "       ['44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54']],\n",
       "      dtype='<U2')\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0
" ], "text/plain": [ "\n", "array([['-1', '-2', '02', '03', '04', '05', '06', '07', '08', '09', '10'],\n", " ['11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21'],\n", " ['22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32'],\n", " ['33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43'],\n", " ['44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54']],\n", " dtype='\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray 'atoms' (runs: 5, truncation_value: 11, y: 1200, x: 1920)>\n",
       "array([[[[ nan,  nan,  nan, ...,  nan,  nan,  nan],\n",
       "         [ nan,  nan,  nan, ...,  nan,  nan,  nan],\n",
       "         [ nan,  nan,  nan, ...,  nan,  nan,  nan],\n",
       "         ...,\n",
       "         [ nan,  nan,  nan, ...,  nan,  nan,  nan],\n",
       "         [ nan,  nan,  nan, ...,  nan,  nan,  nan],\n",
       "         [ nan,  nan,  nan, ...,  nan,  nan,  nan]],\n",
       "\n",
       "        [[102., 103., 100., ..., 155., 154., 142.],\n",
       "         [ 97., 100., 101., ..., 146., 146., 156.],\n",
       "         [ 91.,  93.,  97., ..., 142., 136., 151.],\n",
       "         ...,\n",
       "         [105., 103., 103., ..., 144., 134., 142.],\n",
       "         [102., 104., 112., ..., 133., 134., 135.],\n",
       "         [114., 115., 113., ..., 119., 126., 122.]],\n",
       "\n",
       "        [[101., 105., 103., ..., 148., 152., 146.],\n",
       "         [101., 102.,  98., ..., 143., 146., 149.],\n",
       "         [ 97.,  98.,  98., ..., 139., 137., 146.],\n",
       "         ...,\n",
       "...\n",
       "         ...,\n",
       "         [100., 102., 109., ..., 135., 141., 140.],\n",
       "         [106., 103., 105., ..., 128., 140., 132.],\n",
       "         [114., 107., 108., ..., 121., 123., 126.]],\n",
       "\n",
       "        [[101., 102., 101., ..., 153., 153., 146.],\n",
       "         [101., 103., 100., ..., 148., 143., 149.],\n",
       "         [ 96.,  95.,  98., ..., 145., 138., 144.],\n",
       "         ...,\n",
       "         [ 98., 107., 103., ..., 140., 142., 145.],\n",
       "         [106., 100., 103., ..., 133., 133., 136.],\n",
       "         [115., 103., 102., ..., 123., 125., 128.]],\n",
       "\n",
       "        [[103., 100., 102., ..., 154., 147., 153.],\n",
       "         [ 96., 103.,  97., ..., 150., 150., 152.],\n",
       "         [ 96., 100.,  96., ..., 141., 142., 144.],\n",
       "         ...,\n",
       "         [101., 102., 101., ..., 139., 142., 140.],\n",
       "         [105., 105.,  98., ..., 132., 135., 131.],\n",
       "         [109., 111., 107., ..., 125., 126., 129.]]]])\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Attributes:\n",
       "    IMAGE_SUBCLASS:       IMAGE_GRAYSCALE\n",
       "    IMAGE_VERSION:        1.2\n",
       "    IMAGE_WHITE_IS_ZERO:  0
" ], "text/plain": [ "\n", "array([[[[ nan, nan, nan, ..., nan, nan, nan],\n", " [ nan, nan, nan, ..., nan, nan, nan],\n", " [ nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", " [ nan, nan, nan, ..., nan, nan, nan],\n", " [ nan, nan, nan, ..., nan, nan, nan],\n", " [ nan, nan, nan, ..., nan, nan, nan]],\n", "\n", " [[102., 103., 100., ..., 155., 154., 142.],\n", " [ 97., 100., 101., ..., 146., 146., 156.],\n", " [ 91., 93., 97., ..., 142., 136., 151.],\n", " ...,\n", " [105., 103., 103., ..., 144., 134., 142.],\n", " [102., 104., 112., ..., 133., 134., 135.],\n", " [114., 115., 113., ..., 119., 126., 122.]],\n", "\n", " [[101., 105., 103., ..., 148., 152., 146.],\n", " [101., 102., 98., ..., 143., 146., 149.],\n", " [ 97., 98., 98., ..., 139., 137., 146.],\n", " ...,\n", "...\n", " ...,\n", " [100., 102., 109., ..., 135., 141., 140.],\n", " [106., 103., 105., ..., 128., 140., 132.],\n", " [114., 107., 108., ..., 121., 123., 126.]],\n", "\n", " [[101., 102., 101., ..., 153., 153., 146.],\n", " [101., 103., 100., ..., 148., 143., 149.],\n", " [ 96., 95., 98., ..., 145., 138., 144.],\n", " ...,\n", " [ 98., 107., 103., ..., 140., 142., 145.],\n", " [106., 100., 103., ..., 133., 133., 136.],\n", " [115., 103., 102., ..., 123., 125., 128.]],\n", "\n", " [[103., 100., 102., ..., 154., 147., 153.],\n", " [ 96., 103., 97., ..., 150., 150., 152.],\n", " [ 96., 100., 96., ..., 141., 142., 144.],\n", " ...,\n", " [101., 102., 101., ..., 139., 142., 140.],\n", " [105., 105., 98., ..., 132., 135., 131.],\n", " [109., 111., 107., ..., 125., 126., 129.]]]])\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Attributes:\n", " IMAGE_SUBCLASS: IMAGE_GRAYSCALE\n", " IMAGE_VERSION: 1.2\n", " IMAGE_WHITE_IS_ZERO: 0" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = copy.deepcopy(dataSet.atoms)\n", "# we first need to change the dtype from unsign 16-bit integer (uint16) to float, \n", "# since the uint16 doesn't support to store nan value.\n", "data = data.astype(float)\n", "data = remove_bad_shots(data, runs=0, truncation_value=0.8)\n", "data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }