{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Abstract\n",
    "\n",
    "[**hdf5plugin**](https://github.com/silx-kit/hdf5plugin) is a *Python* package (1) providing a set of [**HDF5**](https://portal.hdfgroup.org/display/HDF5/) compression filters (namely: blosc, bitshuffle, lz4, FCIDECOMP, ZFP, Zstandard) and (2) enabling their use from the *Python* programming language with [**h5py**](https://docs.h5py.org/) a thin, pythonic wrapper around `libHDF5`.\n",
    "\n",
    "This presentation illustrates how to use **hdf5plugin** for reading and writing compressed datasets from *Python* and gives an overview of the different HDF5 compression filters it provides.\n",
    "\n",
    "License: [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Notebook requirements\n",
    "# A recent version of Pillow is required!\n",
    "#%pip install numpy h5py hdf5plugin h5glance rise jupyterlab matplotlib ipympl Pillow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# Creates data.h5 used for demos\n",
    "from matplotlib import pyplot as plt\n",
    "import h5py\n",
    "import hdf5plugin\n",
    "import numpy\n",
    "from PIL import Image\n",
    "import urllib.request\n",
    "\n",
    "url = \"https://www.iter.org/doc/www/content/com/Lists/WebText_2014/Attachments/236/platform_view_north_1.jpg\"\n",
    "filename = urllib.request.urlretrieve(url)[0]\n",
    "image = numpy.array(Image.open(filename).convert(\"L\"))\n",
    "plt.imshow(image)\n",
    "\n",
    "h5file = h5py.File(\"data.h5\", mode=\"w\")\n",
    "h5file[\"copyright\"] = \"ITER\"\n",
    "h5file.attrs[\"url\"] = url\n",
    "h5file.create_dataset(\"/data\", data=image)\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data\",\n",
    "    data=image,\n",
    "    **hdf5plugin.Blosc('lz4', shuffle=hdf5plugin.Blosc.BITSHUFFLE)\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Restart kernel once the file is created!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os._exit(0)  # Makes the kernel restart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "def imshow(image):\n",
    "    plt.imshow(image)\n",
    "    plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![ESRF logo](data:image/png;base64,[encoded data, 462192 characters] \"ESRF logo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# hdf5plugin\n",
    "\n",
    "[**hdf5plugin**](https://github.com/silx-kit/hdf5plugin) packages a set of [HDF5](https://portal.hdfgroup.org/display/HDF5/) compression filters (namely: blosc, bitshuffle, lz4, FCIDECOMP, ZFP, Zstandard) and makes them usable from the Python programming language through [h5py](https://docs.h5py.org/).\n",
    "\n",
    "[**h5py**](https://docs.h5py.org/) is a thin, pythonic wrapper around [HDF5](https://portal.hdfgroup.org/display/HDF5/).\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "Presenter: Thomas VINCENT\n",
    "\n",
    "European HDF5 User Group Meeting 2022, May 31, 2022"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>/* Based on https://css-tricks.com/snippets/css/nested-expandable-folders/\n",
       "   Reused with permission from https://css-tricks.com/license/\n",
       "*/\n",
       "\n",
       ".h5glance-css-treeview ul,\n",
       ".h5glance-css-treeview li\n",
       "{\n",
       "\tpadding: 0;\n",
       "\tmargin: 0;\n",
       "    margin-left: 12px;\n",
       "\tlist-style: none;\n",
       "}\n",
       "\n",
       "/* Override padding from Jupyter CSS */\n",
       ".rendered_html .h5glance-css-treeview ul {\n",
       "    padding: 0;\n",
       "}\n",
       "\n",
       "/* Make the checkbox itself invisible */\n",
       ".h5glance-css-treeview input\n",
       "{\n",
       "\tposition: absolute;\n",
       "\topacity: 0;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview\n",
       "{\n",
       "    font-family: sans-serif;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview code {\n",
       "    font-size: 12pt;\n",
       "    font-family: \"courier\", monospace;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview a.h5glance-dataset-copylink\n",
       "{\n",
       "    text-decoration: none;\n",
       "}\n",
       "\n",
       "/* These next two bits drive the expanding/collapsing behaviour */\n",
       ".h5glance-css-treeview input + label + ul\n",
       "{\n",
       "\tdisplay: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked:not(:disabled) + label + ul\n",
       "{\n",
       "\tdisplay: block;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label,\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tdisplay: inline-block;\n",
       "\theight: 16px;\n",
       "\tline-height: 16px;,\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label\n",
       "{\n",
       "    margin-left: -1.5em;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tcontent: \"\\229e\"; /* squared plus */\n",
       "\twidth: 1.5em;\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked + label::before\n",
       "{\n",
       "    content: \"\\229f\";  /* squared minus */\n",
       "}\n",
       "\n",
       "/* webkit adjacent element selector bugfix */\n",
       "@media screen and (-webkit-min-device-pixel-ratio:0)\n",
       "{\n",
       "\t.h5glance-css-treeview\n",
       "\t{\n",
       "\t\t-webkit-animation: webkit-adjacent-element-selector-bugfix infinite 1s;\n",
       "\t}\n",
       "\n",
       "\t@-webkit-keyframes webkit-adjacent-element-selector-bugfix\n",
       "\t{\n",
       "\t\tfrom\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t\tto\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t}\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview li.h5glance-dataset {\n",
       "    margin-bottom: 3px;\n",
       "}\n",
       "\n",
       ".h5glance-dataset-name {\n",
       "    font-weight: bold;\n",
       "}\n",
       "</style><div id=\"h5glance-container-0\" class=\"h5glance-css-treeview\"><ul><li><input checked=\"checked\" id=\"h5glance-expand-switch-0\" type=\"checkbox\"/><label for=\"h5glance-expand-switch-0\">data.h5</label><ul><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">compressed_data</span> <a data-hdf5-path=\"/compressed_data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 1542 × 2500 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">copyright</span> <a data-hdf5-path=\"/copyright\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: scalar entries, dtype: UTF-8 string</li><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">data</span> <a data-hdf5-path=\"/data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 1542 × 2500 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li></ul></li></ul></div><script>// Based on code by Stackoverflow user Dean Taylor\n",
       "// https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript/30810322\n",
       "// Used under Stackoverflow's CC-BY-SA 3.0 license\n",
       "\n",
       "(function() {\n",
       "    function copyTextToClipboard(text) {\n",
       "        let textArea = document.createElement(\"textarea\");\n",
       "\n",
       "        //\n",
       "        // *** This styling is an extra step which is likely not required. ***\n",
       "        //\n",
       "        // Why is it here? To ensure:\n",
       "        // 1. the element is able to have focus and selection.\n",
       "        // 2. if element was to flash render it has minimal visual impact.\n",
       "        // 3. less flakyness with selection and copying which **might** occur if\n",
       "        //    the textarea element is not visible.\n",
       "        //\n",
       "        // The likelihood is the element won't even render, not even a flash,\n",
       "        // so some of these are just precautions. However in IE the element\n",
       "        // is visible whilst the popup box asking the user for permission for\n",
       "        // the web page to copy to the clipboard.\n",
       "        //\n",
       "\n",
       "        // Place in top-left corner of screen regardless of scroll position.\n",
       "        textArea.style.position = 'fixed';\n",
       "        textArea.style.top = 0;\n",
       "        textArea.style.left = 0;\n",
       "\n",
       "        // Ensure it has a small width and height. Setting to 1px / 1em\n",
       "        // doesn't work as this gives a negative w/h on some browsers.\n",
       "        textArea.style.width = '2em';\n",
       "        textArea.style.height = '2em';\n",
       "\n",
       "        // We don't need padding, reducing the size if it does flash render.\n",
       "        textArea.style.padding = 0;\n",
       "\n",
       "        // Clean up any borders.\n",
       "        textArea.style.border = 'none';\n",
       "        textArea.style.outline = 'none';\n",
       "        textArea.style.boxShadow = 'none';\n",
       "\n",
       "        // Avoid flash of white box if rendered for any reason.\n",
       "        textArea.style.background = 'transparent';\n",
       "\n",
       "\n",
       "        textArea.value = text;\n",
       "\n",
       "        document.body.appendChild(textArea);\n",
       "        textArea.focus();\n",
       "        textArea.select();\n",
       "\n",
       "        try {\n",
       "            if (!document.execCommand('copy')) {\n",
       "                console.log(\"Unable to copy text with document.execCommand()\");\n",
       "            }\n",
       "        } finally {\n",
       "            document.body.removeChild(textArea);\n",
       "        }\n",
       "    }\n",
       "\n",
       "    function copy_event_handler(event) {\n",
       "        copyTextToClipboard(event.target.dataset.hdf5Path);\n",
       "        event.preventDefault();\n",
       "    }\n",
       "\n",
       "    function enable_copylinks(parent) {\n",
       "        let links = parent.querySelectorAll(\".h5glance-dataset-copylink\");\n",
       "        links.forEach(function (link) {\n",
       "            link.addEventListener(\"click\", copy_event_handler);\n",
       "        });\n",
       "    }\n",
       "\n",
       "    // The code to actually trigger this is substituted below.\n",
       "    \n",
       "enable_copylinks(document.getElementById(\"h5glance-container-0\"));\n",
       "\n",
       "})();\n",
       "</script></div>"
      ],
      "text/plain": [
       "data.h5/ (1 attributes)\n",
       "├compressed_data\t[uint8: 1542 × 2500]\n",
       "├copyright\t[UTF-8 string: scalar]\n",
       "└data\t[uint8: 1542 × 2500]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from h5glance import H5Glance  # Browsing HDF5 files\n",
    "H5Glance(\"data.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 121284 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import h5py  # Pythonic HDF5 wrapper: https://docs.h5py.org/\n",
    "\n",
    "h5file = h5py.File(\"data.h5\", mode=\"r\")  # Open HDF5 file in read mode\n",
    "data = h5file[\"/data\"][()]               # Access HDF5 dataset \"/data\"\n",
    "imshow(data)                             # Display data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "Input \u001b[0;32mIn [4]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m data \u001b[38;5;241m=\u001b[39m \u001b[43mh5file\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m/compressed_data\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m[\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\n",
      "File \u001b[0;32mh5py/_objects.pyx:54\u001b[0m, in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32mh5py/_objects.pyx:55\u001b[0m, in \u001b[0;36mh5py._objects.with_phil.wrapper\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m~/venv/ub20.04/lib/python3.8/site-packages/h5py/_hl/dataset.py:741\u001b[0m, in \u001b[0;36mDataset.__getitem__\u001b[0;34m(self, args, new_dtype)\u001b[0m\n\u001b[1;32m    739\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_fast_read_ok \u001b[38;5;129;01mand\u001b[39;00m (new_dtype \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[1;32m    740\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 741\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_fast_reader\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    742\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m:\n\u001b[1;32m    743\u001b[0m         \u001b[38;5;28;01mpass\u001b[39;00m  \u001b[38;5;66;03m# Fall back to Python read pathway below\u001b[39;00m\n",
      "File \u001b[0;32mh5py/_selector.pyx:370\u001b[0m, in \u001b[0;36mh5py._selector.Reader.read\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)"
     ]
    }
   ],
   "source": [
    "data = h5file[\"/compressed_data\"][()]  # Access compressed dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32001, b'blosc')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check dataset's filters\n",
    "plist = h5file[\"/compressed_data\"].id.get_create_plist()\n",
    "plist.get_filter(0)[0::3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `hdf5plugin` usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading compressed datasets\n",
    "\n",
    "To enable reading compressed datasets not supported by `libHDF5` and `h5py`:\n",
    "Install **hdf5plugin** & import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip3 install hdf5plugin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Or: `conda install -c conda-forge hdf5plugin`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import hdf5plugin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 121284 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = h5file[\"/compressed_data\"][()]  # Access datset\n",
    "imshow(data)                           # Display data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file.close()  # Close the HDF5 file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Writing compressed datasets\n",
    "\n",
    "When writing datasets with `h5py`, compression can be specified with: [h5py.Group.create_dataset](https://docs.h5py.org/en/stable/high/group.html?highlight=create_dataset#h5py.Group.create_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create a dataset with h5py without compression\n",
    "h5file = h5py.File(\"new_file_uncompressed.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\"/data\", data=data)\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Create a compressed dataset\n",
    "h5file = h5py.File(\"new_file_blosc_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data\",\n",
    "    data=data,\n",
    "    compression=32001,  # blosc HDF5 filter identifier\n",
    "    # options: 0, 0, 0, 0, level, shuffle, compression\n",
    "    compression_opts=(0, 0, 0, 0, 5, 2, 1)  \n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`hdf5plugin` provides some [helpers](http://www.silx.org/doc/hdf5plugin/latest/usage.html#write-compressed-datasets) to ease dealing with compression filter and options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_blosc_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data\",\n",
    "    data=data,\n",
    "    **hdf5plugin.Blosc(\n",
    "        cname='lz4',\n",
    "        clevel=5,\n",
    "        shuffle=hdf5plugin.Blosc.BITSHUFFLE),\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class Blosc in module hdf5plugin:\n",
      "\n",
      "class Blosc(h5py._hl.filters.FilterRefBase)\n",
      " |  Blosc(cname='lz4', clevel=5, shuffle=1)\n",
      " |  \n",
      " |  ``h5py.Group.create_dataset``'s compression arguments for using blosc filter.\n",
      " |  \n",
      " |  It can be passed as keyword arguments:\n",
      " |  \n",
      " |  .. code-block:: python\n",
      " |  \n",
      " |      f = h5py.File('test.h5', 'w')\n",
      " |      f.create_dataset(\n",
      " |          'blosc_byte_shuffle_blosclz',\n",
      " |          data=numpy.arange(100),\n",
      " |          **hdf5plugin.Blosc(cname='blosclz', clevel=9, shuffle=hdf5plugin.Blosc.SHUFFLE))\n",
      " |      f.close()\n",
      " |  \n",
      " |  :param str cname:\n",
      " |      `blosclz`, `lz4` (default), `lz4hc`, `zlib`, `zstd`\n",
      " |      Optional: `snappy`, depending on compilation (requires C++11).\n",
      " |  :param int clevel:\n",
      " |      Compression level from 0 (no compression) to 9 (maximum compression).\n",
      " |      Default: 5.\n",
      " |  :param int shuffle: One of:\n",
      " |      - Blosc.NOSHUFFLE (0): No shuffle\n",
      " |      - Blosc.SHUFFLE (1): byte-wise shuffle (default)\n",
      " |      - Blosc.BITSHUFFLE (2): bit-wise shuffle\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      Blosc\n",
      " |      h5py._hl.filters.FilterRefBase\n",
      " |      collections.abc.Mapping\n",
      " |      collections.abc.Collection\n",
      " |      collections.abc.Sized\n",
      " |      collections.abc.Iterable\n",
      " |      collections.abc.Container\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, cname='lz4', clevel=5, shuffle=1)\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |  \n",
      " |  BITSHUFFLE = 2\n",
      " |  \n",
      " |  NOSHUFFLE = 0\n",
      " |  \n",
      " |  SHUFFLE = 1\n",
      " |  \n",
      " |  __abstractmethods__ = frozenset()\n",
      " |  \n",
      " |  filter_id = 32001\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from h5py._hl.filters.FilterRefBase:\n",
      " |  \n",
      " |  __eq__(self, other)\n",
      " |      Return self==value.\n",
      " |  \n",
      " |  __getitem__(self, item)\n",
      " |  \n",
      " |  __hash__(self)\n",
      " |      Return hash(self).\n",
      " |  \n",
      " |  __iter__(self)\n",
      " |  \n",
      " |  __len__(self)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from h5py._hl.filters.FilterRefBase:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from h5py._hl.filters.FilterRefBase:\n",
      " |  \n",
      " |  filter_options = ()\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from collections.abc.Mapping:\n",
      " |  \n",
      " |  __contains__(self, key)\n",
      " |  \n",
      " |  get(self, key, default=None)\n",
      " |      D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.\n",
      " |  \n",
      " |  items(self)\n",
      " |      D.items() -> a set-like object providing a view on D's items\n",
      " |  \n",
      " |  keys(self)\n",
      " |      D.keys() -> a set-like object providing a view on D's keys\n",
      " |  \n",
      " |  values(self)\n",
      " |      D.values() -> an object providing a view on D's values\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from collections.abc.Mapping:\n",
      " |  \n",
      " |  __reversed__ = None\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from collections.abc.Collection:\n",
      " |  \n",
      " |  __subclasshook__(C) from abc.ABCMeta\n",
      " |      Abstract classes can override this to customize issubclass().\n",
      " |      \n",
      " |      This is invoked early on by abc.ABCMeta.__subclasscheck__().\n",
      " |      It should return True, False or NotImplemented.  If it returns\n",
      " |      NotImplemented, the normal algorithm is used.  Otherwise, it\n",
      " |      overrides the normal algorithm (and the outcome is cached).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(hdf5plugin.Blosc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>/* Based on https://css-tricks.com/snippets/css/nested-expandable-folders/\n",
       "   Reused with permission from https://css-tricks.com/license/\n",
       "*/\n",
       "\n",
       ".h5glance-css-treeview ul,\n",
       ".h5glance-css-treeview li\n",
       "{\n",
       "\tpadding: 0;\n",
       "\tmargin: 0;\n",
       "    margin-left: 12px;\n",
       "\tlist-style: none;\n",
       "}\n",
       "\n",
       "/* Override padding from Jupyter CSS */\n",
       ".rendered_html .h5glance-css-treeview ul {\n",
       "    padding: 0;\n",
       "}\n",
       "\n",
       "/* Make the checkbox itself invisible */\n",
       ".h5glance-css-treeview input\n",
       "{\n",
       "\tposition: absolute;\n",
       "\topacity: 0;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview\n",
       "{\n",
       "    font-family: sans-serif;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview code {\n",
       "    font-size: 12pt;\n",
       "    font-family: \"courier\", monospace;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview a.h5glance-dataset-copylink\n",
       "{\n",
       "    text-decoration: none;\n",
       "}\n",
       "\n",
       "/* These next two bits drive the expanding/collapsing behaviour */\n",
       ".h5glance-css-treeview input + label + ul\n",
       "{\n",
       "\tdisplay: none;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked:not(:disabled) + label + ul\n",
       "{\n",
       "\tdisplay: block;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label,\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tdisplay: inline-block;\n",
       "\theight: 16px;\n",
       "\tline-height: 16px;,\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label\n",
       "{\n",
       "    margin-left: -1.5em;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview label::before\n",
       "{\n",
       "\tcontent: \"\\229e\"; /* squared plus */\n",
       "\twidth: 1.5em;\n",
       "\tvertical-align: middle;\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview input:checked + label::before\n",
       "{\n",
       "    content: \"\\229f\";  /* squared minus */\n",
       "}\n",
       "\n",
       "/* webkit adjacent element selector bugfix */\n",
       "@media screen and (-webkit-min-device-pixel-ratio:0)\n",
       "{\n",
       "\t.h5glance-css-treeview\n",
       "\t{\n",
       "\t\t-webkit-animation: webkit-adjacent-element-selector-bugfix infinite 1s;\n",
       "\t}\n",
       "\n",
       "\t@-webkit-keyframes webkit-adjacent-element-selector-bugfix\n",
       "\t{\n",
       "\t\tfrom\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t\tto\n",
       "\t\t{\n",
       "\t\t\tpadding: 0;\n",
       "\t\t}\n",
       "\t}\n",
       "}\n",
       "\n",
       ".h5glance-css-treeview li.h5glance-dataset {\n",
       "    margin-bottom: 3px;\n",
       "}\n",
       "\n",
       ".h5glance-dataset-name {\n",
       "    font-weight: bold;\n",
       "}\n",
       "</style><div id=\"h5glance-container-1\" class=\"h5glance-css-treeview\"><ul><li><input checked=\"checked\" id=\"h5glance-expand-switch-1\" type=\"checkbox\"/><label for=\"h5glance-expand-switch-1\">new_file_blosc_bitshuffle_lz4.h5</label><ul><li class=\"h5glance-dataset\"><span class=\"h5glance-dataset-name\">compressed_data</span> <a data-hdf5-path=\"/compressed_data\" href=\"#\" class=\"h5glance-dataset-copylink\">[📋]</a>: 1542 × 2500 entries, dtype: <abbr title=\"8-bit unsigned integer\">uint8</abbr></li></ul></li></ul></div><script>// Based on code by Stackoverflow user Dean Taylor\n",
       "// https://stackoverflow.com/questions/400212/how-do-i-copy-to-the-clipboard-in-javascript/30810322\n",
       "// Used under Stackoverflow's CC-BY-SA 3.0 license\n",
       "\n",
       "(function() {\n",
       "    function copyTextToClipboard(text) {\n",
       "        let textArea = document.createElement(\"textarea\");\n",
       "\n",
       "        //\n",
       "        // *** This styling is an extra step which is likely not required. ***\n",
       "        //\n",
       "        // Why is it here? To ensure:\n",
       "        // 1. the element is able to have focus and selection.\n",
       "        // 2. if element was to flash render it has minimal visual impact.\n",
       "        // 3. less flakyness with selection and copying which **might** occur if\n",
       "        //    the textarea element is not visible.\n",
       "        //\n",
       "        // The likelihood is the element won't even render, not even a flash,\n",
       "        // so some of these are just precautions. However in IE the element\n",
       "        // is visible whilst the popup box asking the user for permission for\n",
       "        // the web page to copy to the clipboard.\n",
       "        //\n",
       "\n",
       "        // Place in top-left corner of screen regardless of scroll position.\n",
       "        textArea.style.position = 'fixed';\n",
       "        textArea.style.top = 0;\n",
       "        textArea.style.left = 0;\n",
       "\n",
       "        // Ensure it has a small width and height. Setting to 1px / 1em\n",
       "        // doesn't work as this gives a negative w/h on some browsers.\n",
       "        textArea.style.width = '2em';\n",
       "        textArea.style.height = '2em';\n",
       "\n",
       "        // We don't need padding, reducing the size if it does flash render.\n",
       "        textArea.style.padding = 0;\n",
       "\n",
       "        // Clean up any borders.\n",
       "        textArea.style.border = 'none';\n",
       "        textArea.style.outline = 'none';\n",
       "        textArea.style.boxShadow = 'none';\n",
       "\n",
       "        // Avoid flash of white box if rendered for any reason.\n",
       "        textArea.style.background = 'transparent';\n",
       "\n",
       "\n",
       "        textArea.value = text;\n",
       "\n",
       "        document.body.appendChild(textArea);\n",
       "        textArea.focus();\n",
       "        textArea.select();\n",
       "\n",
       "        try {\n",
       "            if (!document.execCommand('copy')) {\n",
       "                console.log(\"Unable to copy text with document.execCommand()\");\n",
       "            }\n",
       "        } finally {\n",
       "            document.body.removeChild(textArea);\n",
       "        }\n",
       "    }\n",
       "\n",
       "    function copy_event_handler(event) {\n",
       "        copyTextToClipboard(event.target.dataset.hdf5Path);\n",
       "        event.preventDefault();\n",
       "    }\n",
       "\n",
       "    function enable_copylinks(parent) {\n",
       "        let links = parent.querySelectorAll(\".h5glance-dataset-copylink\");\n",
       "        links.forEach(function (link) {\n",
       "            link.addEventListener(\"click\", copy_event_handler);\n",
       "        });\n",
       "    }\n",
       "\n",
       "    // The code to actually trigger this is substituted below.\n",
       "    \n",
       "enable_copylinks(document.getElementById(\"h5glance-container-1\"));\n",
       "\n",
       "})();\n",
       "</script></div>"
      ],
      "text/plain": [
       "new_file_blosc_bitshuffle_lz4.h5/\n",
       "└compressed_data\t[uint8: 1542 × 2500]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H5Glance(\"new_file_blosc_bitshuffle_lz4.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 121284 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "h5file = h5py.File(\"new_file_blosc_bitshuffle_lz4.h5\", mode=\"r\")\n",
    "imshow(h5file[\"/compressed_data\"][()])\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.4M new_file_blosc_bitshuffle_lz4.h5  3.7M new_file_uncompressed.h5\n"
     ]
    }
   ],
   "source": [
    "!ls -sh new_file*.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## HDF5 compression filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Available through `h5py`\n",
    "\n",
    "[Compression filters provided by h5py](https://docs.h5py.org/en/stable/high/dataset.html#lossless-compression-filters):\n",
    "\n",
    "- Provided by `libhdf5`: **\"gzip\"** and eventually **\"szip\"** (optional)\n",
    "- Bundled with `h5py`: [**\"lzf\"**](https://confluence.hdfgroup.org/display/support/Filters#Filters-32000)\n",
    "\n",
    "Pre-compression filter: [Byte-Shuffle](https://docs.h5py.org/en/stable/high/dataset.html#shuffle-filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_shuffle_gzip.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_shuffle_gzip\", data=data, shuffle=True, compression=\"gzip\")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Provided by `hdf5plugin`\n",
    "\n",
    "Additional compression filters provided by `hdf5plugin`: Bitshuffle, Blosc, FciDecomp, LZ4, ZFP, Zstandard.\n",
    "\n",
    "6 out of the 28 [HDF5 registered filter plugins](https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugins) as of May 2022."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "h5file = h5py.File(\"new_file_bitshuffle_lz4.h5\", mode=\"w\")\n",
    "h5file.create_dataset(\n",
    "    \"/compressed_data_bitshuffle_lz4\",\n",
    "    data=data,\n",
    "    **hdf5plugin.Bitshuffle()\n",
    ")\n",
    "h5file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### General purpose lossless compression\n",
    "\n",
    "- [Bitshuffle(nelems=0, lz4=True)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#bitshuffle) (Filter ID [32008](https://confluence.hdfgroup.org/display/support/Filters#Filters-32008)): **Bit**-Shuffle + LZ4\n",
    "- [LZ4(nbytes=0)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#lz4) (Filter ID [32004](https://confluence.hdfgroup.org/display/support/Filters#Filters-32004))\n",
    "- [Zstd(clevel=3)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#zstd) (Filter ID [32015](https://confluence.hdfgroup.org/display/support/Filters#Filters-32015))\n",
    "- [Blosc(cname='lz4', clevel=5, shuffle=1)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#blosc) (Filter ID [32001](https://confluence.hdfgroup.org/display/support/Filters#Filters-32001)):\n",
    "  Based on c-blosc: A blocking, shuffling and lossless compression library.\n",
    "  - Pre-compression shuffle: None, **Byte**-Shuffle, **Bit**-Shuffle\n",
    "  - Compression: `blosclz`, `lz4`, `lz4hc`, `snappy` (**optional**, requires C++11), `zlib`, `zstd`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Equivalent filters\n",
    "\n",
    "`Blosc` includes pre-compression filters and algorithms provided by other HDF5 compression filters:\n",
    "\n",
    "- HDF5 shuffle  => `Blosc(..., shuffle=Blosc.SHUFFLE)`\n",
    "- `Bitshuffle()` => `Blosc(\"lz4\", 5, Blosc.BITSHUFFLE)`\n",
    "- `LZ4()` => `Blosc(\"lz4\", 9)`\n",
    "- `Zstd()` => `Blosc(\"zstd\", 2)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Specific compression\n",
    "\n",
    "- [FciDecomp()](http://www.silx.org/doc/hdf5plugin/latest/usage.html#fcidecomp) (Filter ID [32018](https://confluence.hdfgroup.org/display/support/Filters#Filters-32018)): Based on JPEG-LS:\n",
    "  - **Optional**: requires C++11\n",
    "  - Data type: `(u)int8` or `(u)int16`\n",
    "  - Chunk shape: \"Image-like\"; 2 or 3 dimensions with at least 16 pixels and at most 65535 rows and columns and at most 4 planes for 3D datasets.\n",
    "- [ZFP(rate=None, precision=None, accuracy=None, reversible=False, minbits=None, maxbits=None, maxprec=None, minexp=None)](http://www.silx.org/doc/hdf5plugin/latest/usage.html#zfp) (Filter ID [32013](https://confluence.hdfgroup.org/display/support/Filters#Filters-32013)): **Lossy**\n",
    "  - Data type: `float32`, `float64`, `(u)int32`, `(u)int64`\n",
    "  - Chunk shape: must have at most 4 non-unity dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### A look at performances on a single use case\n",
    "\n",
    "- Machine: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (40 cores)\n",
    "- Filesystem: /dev/shm\n",
    "- `hdf5plugin` built from source\n",
    "\n",
    "\n",
    "- Running on a single thread (with `OMP_NUM_THREADS=1`)\n",
    "\n",
    "\n",
    "- Diffraction tomography dataset: 100 frames from http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5\n",
    "- Dataset: 100x2167x2070, uint16, chunk: 2167x2070"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![Benchmark](data:image/png;base64,[encoded data, 190656 characters] \"Benchmark\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Multithreaded filter execution\n",
    "\n",
    "Some filters can use multithreading:\n",
    "\n",
    "- Blosc:\n",
    "  - Using a pool of threads\n",
    "  - Disabled by default\n",
    "  - Configurable with the `BLOSC_NTHREADS` environment variable\n",
    "- Bitshuffle, ZFP:\n",
    "  - Using OpenMP\n",
    "  - Enabled at compilation time\n",
    "  - If enabled, configurable with `OMP_NUM_THREADS` environment variable\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Summary\n",
    "\n",
    "Having different pre-compression filters and compression algorithms at hand offer different read/write speed versus compression rate (and eventually error rate) trade-offs.\n",
    "\n",
    "Also to keep in mind availability/compatibility: `\"gzip\"` as included in `libHDF5` is the most compatible one (and also `\"lzf\"` as included in `h5py`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using `hdf5plugin` filters with other applications\n",
    "\n",
    "Set the `HDF5_PLUGIN_PATH` environment variable to: `hdf5plugin.PLUGINS_PATH`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HDF5_PLUGIN_PATH=/venv/ub20.04/lib/python3.8/site-packages/hdf5plugin/plugins\n",
      "libh5blosc.so\n",
      "libh5bshuf.so\n",
      "libh5fcidecomp.so\n",
      "libh5lz4.so\n",
      "libh5zfp.so\n",
      "libh5zstd.so\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HDF5_PLUGIN_PATH=`python3 -c \"\n",
    "import hdf5plugin; print(hdf5plugin.PLUGINS_PATH)\"`\n",
    "echo \"HDF5_PLUGIN_PATH=${HDF5_PLUGIN_PATH}\"\n",
    "ls ${HDF5_PLUGIN_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "**Note**: Only works for reading compressed datasets, **not for writing!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A word about `hdf5plugin` license\n",
    "\n",
    "The source code of `hdf5plugin` itself is licensed under the [MIT](https://github.com/silx-kit/hdf5plugin/blob/main/LICENSE) license...\n",
    "\n",
    "It also embeds the source code of the provided compression filters and libraries which are licensed under [different open-source licenses](http://www.silx.org/doc/hdf5plugin/latest/information.html#license) (Apache, BSD-2, BSD-3, MIT, Zlib...) and copyrights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Limitations\n",
    "\n",
    "Some limitations of current HDF5 compression filters:\n",
    "- Compressed data accessed by \"chunks\" even if compressor uses smaller blocks\n",
    "- Multi-threaded access\n",
    "- When reading compressed data, some memory copy could be spared\n",
    "- Need to link filters with libhdf5\n",
    "- Only \"gzip\" available by default and no central repository for registered filters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Comments\n",
    "\n",
    "- Direct chunk access offers a way to improve performance/flexibility, at the expense of more code on the user side\n",
    "\n",
    "- `hdf5plugin` relies on a \"hack\" to ease the installation of HDF5 compression for Python environments\n",
    "\n",
    "- Most of the compression filters provided by `hdf5plugin` are included in [blosc](https://github.com/blosc/c-blosc) (or [blosc-2](https://github.com/blosc/c-blosc2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Idea: HDF5+Blosc(2)?\n",
    "\n",
    "- Time for an upgrade of compression support in HDF5?\n",
    "- What about making blosc(2) available by default in libhdf5?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conlusion\n",
    "\n",
    "`hdf5plugin` provides additional HDF5 compression filters (namely: `Bitshuffle`, `Blosc`, `FciDecomp`, `LZ4`, `ZFP`, `Zstandard`) mainly for use with [h5py](https://docs.h5py.org/).\n",
    "\n",
    "- Packaged for [**pip**](https://pypi.org/project/hdf5plugin/) and [**conda**](https://anaconda.org/conda-forge/hdf5plugin)\n",
    "- Documentation: http://www.silx.org/doc/hdf5plugin/latest/\n",
    "- Source code repository: https://github.com/silx-kit/hdf5plugin\n",
    "\n",
    "**Credits to hdf5plugin** [**contributors**](https://github.com/silx-kit/hdf5plugin/graphs/contributors): [Thomas Vincent](https://github.com/t20100), [Armando Sole](https://github.com/vasole), [Mark Kittisopikul](https://github.com/mkitti), [\\@Florian-toll](https://github.com/Florian-toll), [Jerome Kieffer](https://github.com/kif), [\\@fpwg](https://github.com/fpwg), [\\@Anthchirp](https://github.com/Anthchirp), [\\@mobiusklein](https://github.com/mobiusklein), [\\@junyuewang](https://github.com/junyuewang)\n",
    "**and to all contributors of embedded libraries.**\n",
    "\n",
    "Partially funded by the [PaNOSC](https://www.panosc.eu/) EU-project.\n",
    "\n",
    "<img align=\"left\" vertical-align=\"top\" src=\"data:image/svg+xml,%3Csvg%20width%3D%2240.5%22%20height%3D%2227%22%20version%3D%221.1%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20xmlns%3Axlink%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxlink%22%3E%3Cdesc%3EEuropean%20flag%3C%2Fdesc%3E%3Cdefs%3E%3Cg%20id%3D%22s%22%3E%3Cg%20id%3D%22c%22%3E%3Cpath%20id%3D%22t%22%20transform%3D%22rotate%2818%203.1569%20-.5%29%22%20d%3D%22m0%200v1h0.5z%22%2F%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23t%22%2F%3E%3C%2Fg%3E%3Cg%20id%3D%22a%22%3E%3Cuse%20transform%3D%22rotate%2872%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23c%22%2F%3E%3Cuse%20transform%3D%22rotate%28144%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23c%22%2F%3E%3C%2Fg%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23a%22%2F%3E%3C%2Fg%3E%3C%2Fdefs%3E%3Crect%20width%3D%2240.5%22%20height%3D%2227%22%20fill%3D%22%23039%22%20stroke-width%3D%22.05%22%2F%3E%3Cg%20transform%3D%22matrix%281.5%2C0%2C0%2C1.5%2C20.25%2C13.5%29%22%20fill%3D%22%23fc0%22%3E%3Cuse%20y%3D%22-6%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20y%3D%226%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cg%20id%3D%22l%22%3E%3Cuse%20x%3D%22-6%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%28216%20-2.3442%20-2.1107%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%28144%2C-2.1107%2C-2.3442%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%2872%2C-4.6626%2C-2.0759%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3Cuse%20transform%3D%22rotate%2872%20-5.0759%20.5335%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23s%22%2F%3E%3C%2Fg%3E%3Cuse%20transform%3D%22scale%28-1%2C1%29%22%20width%3D%22100%25%22%20height%3D%22100%25%22%20xlink%3Ahref%3D%22%23l%22%2F%3E%3C%2Fg%3E%3C%2Fsvg%3E\"/>&nbsp;This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 823852."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Thanks for your attention! Questions?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "nbsphinx": {
   "orphan": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
