{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# How to read data from HFD5 files" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "This is an example to read the data from those HDF5 files generated by LabScript.\n", "\n", "The idea is that one just need to give\n", "\n", "+ The path where data is stored\n", "+ The date of the data\n", "+ The shot number\n", "+ The path of data in HDF5 file (i.e. the name of group)\n", "\n", "It can automatically find the scan axes, and package all of them into an object of DataSet() in xarray package.\n", "\n", "Let us start with importing the supporting packages. In order to keep things ealier, here we will import all the packages, but not all of them will be used." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Import supporting packages" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Set the system path for importing packages\n", "# This is just because I put all example scripts in another folder\n", "# You DO NOT need to do this \n", "# -------------- You do NOT need following part --------------\n", "import sys\n", "import os\n", "sys.path.insert(0, os.path.abspath('..'))\n", "# -------------- You do NOT need above part --------------\n", "\n", "import copy\n", "import glob\n", "from datetime import datetime\n", "\n", "# The package for data structure\n", "import xarray as xr\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# The packages for working with uncertainties\n", "from uncertainties import ufloat\n", "from uncertainties import unumpy as unp\n", "from uncertainties import umath\n", "\n", "# The package for plotting\n", "import matplotlib.pyplot as plt\n", "plt.rcParams['font.size'] = 18 # Set the global font size\n", "\n", "# -------------- The modules written by us --------------\n", "\n", "# The packages for read data\n", "from DataContainer.ReadData import read_hdf5_file, read_hdf5_global, read_hdf5_run_time, read_csv_file\n", "\n", "# The packages for data analysis\n", "from Analyser.ImagingAnalyser import ImageAnalyser\n", "from Analyser.FitAnalyser import FitAnalyser\n", "from Analyser.FitAnalyser import ThomasFermi2dModel, DensityProfileBEC2dModel, Polylog22dModel\n", "from Analyser.FFTAnalyser import fft, ifft, fft_nutou\n", "from ToolFunction.ToolFunction import *\n", "\n", "# Add errorbar plot to xarray package\n", "from ToolFunction.HomeMadeXarrayFunction import errorbar, dataarray_plot_errorbar\n", "xr.plot.dataarray_plot.errorbar = errorbar\n", "xr.plot.accessor.DataArrayPlotAccessor.errorbar = dataarray_plot_errorbar" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Start a client for parallel computing" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "If one wants to read tens or few hundreds files, a normal script can finish in several minutes. However, very likely, we have load and analyse thousands or even more shots, which is going to take much longer time. Therefore, it is necessary to use some other technics to accelerate the script. One solution is to do parallel loading and computing by using the dask package." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-836bc314-1f0a-11ee-9fac-80e82ce2fa8e

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Cluster objectCluster type: distributed.LocalCluster
\n", " Dashboard: http://127.0.0.1:8787/status\n", "
\n", "\n", " \n", "\n", " \n", "
\n", "

Cluster Info

\n", "
\n", "
\n", "
\n", "
\n", "

LocalCluster

\n", "

78c63c01

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Workers: 6\n", "
\n", " Total threads: 60\n", " \n", " Total memory: 55.88 GiB\n", "
Status: runningUsing processes: True
\n", "\n", "
\n", " \n", "

Scheduler Info

\n", " \n", "\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-7b91ae15-611f-4967-a585-347a12dbf884

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: tcp://127.0.0.1:60497\n", " \n", " Workers: 6\n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Total threads: 60\n", "
\n", " Started: Just now\n", " \n", " Total memory: 55.88 GiB\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", " \n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 0

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:60537\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:60539/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:60500\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-hgh81x58\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 1

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:60536\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:60538/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:60501\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-l6g2y451\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 2

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:60527\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:60528/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:60502\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-2ecklsip\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 3

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:60524\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:60525/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:60503\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-_bx_rrx9\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 4

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:60531\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:60534/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:60504\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-gi8gzg2l\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 5

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:60530\n", " \n", " Total threads: 10\n", "
\n", " Dashboard: http://127.0.0.1:60532/status\n", " \n", " Memory: 9.31 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:60505\n", "
\n", " Local directory: C:\\Users\\data\\AppData\\Local\\Temp\\dask-worker-space\\worker-oy_0wm2p\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from dask.distributed import Client\n", "client = Client(n_workers=6, threads_per_worker=10, processes=True, memory_limit='10GB')\n", "client" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Set the path for different cameras" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "In our current experiment configuration (06.07.2023), the images of absorption imaging from different camera are stored in the following structure:\n", "\n", " | - images\n", " | | - The name of the camera #1\n", " | | | - in_situ_absorption\n", " | | | | - atoms\n", " | | | | - background\n", " | | | | - dard\n", " | | | \n", " | | - The name of the camera #2\n", " | | | - in_situ_absorption\n", " | | | | - atoms\n", " | | | | - background\n", " | | | | - dard\n", "\n", "Here we do not need to specify the name of the data we want to load. The programe only needs to know the group where the data are stored. Due to the way HDF5 file storing the data, going through and loading all the data under the same group won't significantly increase the time consumption.\n", "\n", "Therefore, in this example, the path of our absorption imaging data taken by the side-imaging camera is\n", " \n", " \"images/MOT_3D_Camera/in_situ_absorption\"\n", "\n", "Then we can create a dictionary for all three cameras " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "groupList = [\n", " \"images/MOT_3D_Camera/in_situ_absorption\",\n", " \"images/ODT_1_Axis_Camera/in_situ_absorption\",\n", " \"images/ODT_2_Axis_Camera/in_situ_absorption\",\n", "]\n", "\n", "# give a short name to each path (or let's say each camera)\n", "dskey = {\n", " \"images/MOT_3D_Camera/in_situ_absorption\": \"camera_0\",\n", " \"images/ODT_1_Axis_Camera/in_situ_absorption\": \"camera_1\",\n", " \"images/ODT_2_Axis_Camera/in_situ_absorption\": \"camera_2\",\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Set global path for experiment" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "img_dir = '//DyLabNAS/Data/'\n", "SequenceName = \"Evaporative_Cooling\" + \"/\"\n", "folderPath = img_dir + SequenceName + '2023/04/17'# get_date()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Here we will load the data from our NAS servers, taken by sequence 'Evaporative_Cooling', on 17.04.2023.\n", "\n", "The get_date() funciton in ToolFunction.ToolFunction module can return the date of today in a format of 'YYYY/MM/DD'." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Load shot 0058" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 dask.array<chunksize=(1, 1, 1200, 1920), meta=np.ndarray>\n",
       "    background        (runs, truncation_value, y, x) uint16 dask.array<chunksize=(1, 1, 1200, 1920), meta=np.ndarray>\n",
       "    dark              (runs, truncation_value, y, x) uint16 dask.array<chunksize=(1, 1, 1200, 1920), meta=np.ndarray>\n",
       "    shotNum           (runs, truncation_value) <U2 '00' '01' '02' ... '53' '54'\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 dask.array\n", " background (runs, truncation_value, y, x) uint16 dask.array\n", " dark (runs, truncation_value, y, x) uint16 dask.array\n", " shotNum (runs, truncation_value) \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n",
       "    background        (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n",
       "    dark              (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n",
       "    shotNum           (runs, truncation_value) <U2 '00' '01' '02' ... '53' '54'\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n", " background (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n", " dark (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n", " shotNum (runs, truncation_value) \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 dask.array<chunksize=(1, 1, 1200, 1920), meta=np.ndarray>\n",
       "    background        (runs, truncation_value, y, x) uint16 dask.array<chunksize=(1, 1, 1200, 1920), meta=np.ndarray>\n",
       "    dark              (runs, truncation_value, y, x) uint16 dask.array<chunksize=(1, 1, 1200, 1920), meta=np.ndarray>\n",
       "    shotNum           (runs, truncation_value) <U2 '00' '01' '02' ... '53' '54'\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 dask.array\n", " background (runs, truncation_value, y, x) uint16 dask.array\n", " dark (runs, truncation_value, y, x) uint16 dask.array\n", " shotNum (runs, truncation_value) \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n",
       "    background        (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n",
       "    dark              (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n",
       "    shotNum           (runs, truncation_value) <U2 '00' '01' '02' ... '53' '54'\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n", " background (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n", " dark (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n", " shotNum (runs, truncation_value) \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n",
       "    background        (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n",
       "    dark              (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n",
       "    shotNum           (runs, truncation_value) <U2 '00' '01' '02' ... '53' '54'\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 99 100 ... 126 129\n", " background (runs, truncation_value, y, x) uint16 103 100 ... 122 124\n", " dark (runs, truncation_value, y, x) uint16 51 52 52 ... 49 50\n", " shotNum (runs, truncation_value) \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray 'atoms' (runs: 5, truncation_value: 11, y: 1200, x: 1920)>\n",
       "dask.array<xarray-<this-array>, shape=(5, 11, 1200, 1920), dtype=uint16, chunksize=(5, 1, 1200, 1920), chunktype=numpy.ndarray>\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Attributes:\n",
       "    IMAGE_SUBCLASS:       IMAGE_GRAYSCALE\n",
       "    IMAGE_VERSION:        1.2\n",
       "    IMAGE_WHITE_IS_ZERO:  0
" ], "text/plain": [ "\n", "dask.array, shape=(5, 11, 1200, 1920), dtype=uint16, chunksize=(5, 1, 1200, 1920), chunktype=numpy.ndarray>\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Attributes:\n", " IMAGE_SUBCLASS: IMAGE_GRAYSCALE\n", " IMAGE_VERSION: 1.2\n", " IMAGE_WHITE_IS_ZERO: 0" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataSet.atoms.chunk((5,1,1200,1920))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Use our home-made auto_rechunk function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We wrote a auto rechunk function to auto rechunk a xarrry DataSet. It works in the most time, but please check." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n",
       "Coordinates:\n",
       "  * runs              (runs) float64 0.0 1.0 2.0 3.0 4.0\n",
       "  * truncation_value  (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n",
       "Dimensions without coordinates: y, x\n",
       "Data variables:\n",
       "    atoms             (runs, truncation_value, y, x) uint16 dask.array<chunksize=(5, 11, 1104, 1104), meta=np.ndarray>\n",
       "    background        (runs, truncation_value, y, x) uint16 dask.array<chunksize=(5, 11, 1104, 1104), meta=np.ndarray>\n",
       "    dark              (runs, truncation_value, y, x) uint16 dask.array<chunksize=(5, 11, 1104, 1104), meta=np.ndarray>\n",
       "    shotNum           (runs, truncation_value) <U2 dask.array<chunksize=(5, 11), meta=np.ndarray>\n",
       "Attributes: (12/100)\n",
       "    TOF_free:                          0.02\n",
       "    abs_img_freq:                      110.866\n",
       "    absorption_imaging_flag:           True\n",
       "    backup_data:                       True\n",
       "    blink_off_time:                    nan\n",
       "    blink_on_time:                     nan\n",
       "    ...                                ...\n",
       "    z_offset:                          0.195\n",
       "    z_offset_img:                      0.195\n",
       "    truncation_value:                  [0.8  0.83 0.85 0.87 0.89 0.91 0.93 0....\n",
       "    runs:                              [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n",
       "    scanAxis:                          ['runs' 'truncation_value']\n",
       "    scanAxisLength:                    [55. 55.]
" ], "text/plain": [ "\n", "Dimensions: (runs: 5, truncation_value: 11, y: 1200, x: 1920)\n", "Coordinates:\n", " * runs (runs) float64 0.0 1.0 2.0 3.0 4.0\n", " * truncation_value (truncation_value) float64 0.8 0.83 0.85 ... 0.97 0.99 1.0\n", "Dimensions without coordinates: y, x\n", "Data variables:\n", " atoms (runs, truncation_value, y, x) uint16 dask.array\n", " background (runs, truncation_value, y, x) uint16 dask.array\n", " dark (runs, truncation_value, y, x) uint16 dask.array\n", " shotNum (runs, truncation_value) \n", "Attributes: (12/100)\n", " TOF_free: 0.02\n", " abs_img_freq: 110.866\n", " absorption_imaging_flag: True\n", " backup_data: True\n", " blink_off_time: nan\n", " blink_on_time: nan\n", " ... ...\n", " z_offset: 0.195\n", " z_offset_img: 0.195\n", " truncation_value: [0.8 0.83 0.85 0.87 0.89 0.91 0.93 0....\n", " runs: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1...\n", " scanAxis: ['runs' 'truncation_value']\n", " scanAxisLength: [55. 55.]" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataSet = auto_rechunk(dataSet)\n", "dataSet" ] } ], "metadata": { "kernelspec": { "display_name": "env", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "c05913ad4f24fdc6b2418069394dc5835b1981849b107c9ba6df693aafd66650" } } }, "nbformat": 4, "nbformat_minor": 2 }