The HDF Group's Call the Doctor: Recent Episodes
The HDF Group
Find the latest information about the HDF file formats and new developments from The HDF Group and the HDF community.
The HDF Group's Dana Robinson talked about the next release of HDF5 and why we're debating whether it will be 1.16 or 1.14.6. He also briefly talked about the upcoming HEP - HDF Enhancement Process and how that will help provide transparency and access to the project.
This session was recorded on October 1, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Scot Breitenfeld hosted Call the Doctor on September 24. Scot's our HPC and Fortran expert and shared some HPC news including some testing on Aurora. There was a question and ensuing discussion on determining chunking parameters for using compression on parallel output.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Aleksandar Jelenak will host Call the Doctor on Tuesday, 9/17. He will take questions and plans to will talk about a Python script he wrote for additional (compared to h5stat) dataset statistics for a given HDF5 file. Aleksandar uses this script to inspect the changes after cloud optimizing a file.
This session was recorded on September 17, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's John Readey used his Call the Doctor session to talk about SWMR (single writer multiple reader) and FancySelections for h5pyd with HSDS.
Example notebooks referenced by John during this session:
- https://github.com/HDFGroup/h5pyd/blob/master/examples/notebooks/swmr_example.ipynb
- https://github.com/HDFGroup/h5pyd/blob/master/examples/notebooks/fancy_selection.ipynb
This session was recorded on September 10, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Director of Engineering Dana Robinson provided some general updates. Dana talked about upcoming releases of HDF5, HDF4, and HDFView. Communications Coordinator Lori Cooper announced the availability of the 2024 HDF5 User Group Meeting video recordings and talked briefly about The HDF Group's new support site (still a work in progress) at https://support.hdfgroup.org.
This session was recorded on September 3, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In addition to answering community questions, we will examine the latest PureHDF release(s). PureHDF v1.0.0 and v2.0.0 were released on June 24 (2024), and there have been subsequent minor releases (see Releases · Apollo3zehn/PureHDF · GitHub). We will examine what has changed since v1.0.0-beta.2 and use this opportunity to update the HDF5 tutorial (GitHub - HDFGroup/hdf5-tutorial: A tutorial for new and intermediate HDF5 users (of all ages)).
This session was recorded on July 30, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Scot Breitenfeld ended up using this session to discuss topics including subfiling, support of multidimensional arrays by various languages, the future of Fortran and more with community members and The HDF Group co-workers.
In this session, we mentioned Aleksandar Jelenak's Call the Doctor session last week where he discussed Hidefix, a Rust package for multithreaded read-only access to local HDF5/netCDF4 files. The link was provided in chat and is being provided here for your convenience.
This session was recorded on August 27, 2024.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group’s Aleksandar Jelenak will talk about a Hidefix, a Rust package for multithreaded read-only access to local HDF5/netCDF4 files. https://github.com/gauteh/hidefix It can also be imported as a Python package and used as an xarray backend engine. Aleksandar will also be available for user questions–he has a background as a scientist working with geospatial and geoscience data, and now works at The HDF Group to make HDF5 and related tools more effective for other scientists, specializing in Cloud-Optimized HDF5.
This session was recorded on August 20, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's senior informatics architect, Aleksandar Jelenak talked about some hdf5lib changes in the pipeline that will bring improvements for Cloud Optimized HDF5 files.
This session was recorded on July 16, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Engineering Director John Readey used his Call the Doctor session to talk about performance enhancements using the h5pyd MultiManager.
For a use case where MultiManager came in clutch, check out this recent forum thread.
This session was recorded on July 9, 2024.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Engineering Director Dana Robinson hosted the first Call the Doctor of the month and used his time to talk about upcoming releases and some activities and thoughts we’ve been working through on versioning. Then he talked about plans for a community proposal or HDF5 improvement system similar to the Python Improvement Process.
This session was recorded on June 25, 2024. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, The HDF Group's Gerd Heber discusses the evolution of HDF documentation, highlighting challenges and past approaches and describes our shift towards treating documentation like code, citing the adoption of tools like Doxygen and plans to streamline documentation under a unified support platform. He previews upcoming updates to enhance readability and functionality, aiming for a rollout by the the HDF Users Group meeting in August. The session underscores the ongoing efforts to improve accessibility and maintainability of HDF documentation amidst evolving technological landscapes.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Aleksandar Jelenak of The HDF Group hosted Call the Doctor and shared his experiences with the s5cmd command-line tool that makes working with S3 object stores much easier and faster. He likes it and thinks you will too!
This session was recorded on June 18, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, John Readey introduced GitHub Codespaces and demonstrates their use with the h5pyd repository. Codespaces allow running a repository in the cloud without local setup. The speaker sets up a Codespace for the h5pyd repository, demonstrating its integration with VS Code, terminal access, and execution of h5pyd commands like hsinfo and python test. They showcase practical examples such as reading and writing HDF5 files, leveraging the S3 interface for data access, and utilizing Jupyter notebooks for data analysis. The video highlights how Codespaces simplify environment setup and enable handling large datasets efficiently through HSDS integration.
This session was recorded on June 11, 2024. You can also watch this video online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, Engineering Director Dana Robinson talks about the potential for a HDF5 1.16 release and discusses some HDF5 roadmap options with community members.
This session was recorded on June 6, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, Scot Breitenfeld went over recent benchmarks comparing all versions of HDF5, and talked through some of the development that caused changes in those benchmarks. Scot was also available for community questions.
This session was recorded on May 28, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's John Readey walked through using Github Codespaces with HSDS.
This session was recorded on May 14, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this "Call the Doctor" session hosted by Aleksander Jelenak, the agenda was opened for questions regarding h5py or Cloud optimization of HDF files. The discussion delveed into understanding the applicability of techniques used in h5py to the VOL connector for HSDS, emphasizing the nuances between Python and C approaches. Additionally, considerations are made regarding read-only implementations, the trade-offs of network latency, and the potential for compressing data to enhance performance in HSDS. Throughout, there's an exploration of user expectations, performance evaluation, and recent developments in Cloud-optimized HDF files, including issues identified by users such as those from the NASA PACE satellite mission.
This session was recorded on May 21, 2024. You can also watch this session on youtube.
Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at https://us06web.zoom.us/j/98286880081
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group’s Director of Engineering Dana Robinson hosted Call the Doctor on Tuesday, May 7. He talked about recent CVEs (more info) and some plans for 1.14.5.
This session was recorded on May 7, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
This Call the Doctor, presented by The HDF Group's executive director, Gerd Heber, presents the Zettelkasten (“slip-box”) is a method of note-taking and knowledge management for writers that was (re-)invented many times and more recently developed by the German sociologist Niklas Luhmann. The Zettelkasten is a collection of notes stored on separate index cards. The notes are linked in a network of references, forming a knowledge graph.
HDF5 has everything necessary to create a nifty (digital) Zettelkasten store. (Spoiler alert: Each note will be stored as a group in an HDF5 file, and the references between notes will be stored as links between the groups.) The only component that needs to be added is an intuitive UI, which we leave to the ingenuity of our resourceful attendees.
You can find notes and resources for this session at https://github.com/HDFGroup/hdf-clinic/tree/main/2024-04-30.
This session happened on April 30, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Scot Breitenfeld hosted Call the Doctor on Tuesday, April 16, 2024. Scot answered some community questions about subfiling in HDF5.
Director of Engineering Dana Robinson talked briefly about the recent HDF5 1.14.4 release.
Communications Coordinator Lori Cooper talked about some upcoming community events:
- 2024 HDF5 User Group Meeting, August 5-7 in Chicago.
- Upcoming webinar: h5rdmtoolbox – A Python Research Data Management Toolbox for HDF5, Presented by Matthias Probst from the Karlsruhe Institute of Technology, Institute of Thermal Turbomachinery. More info or to register.
This session happened on April 23, 2024.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Aleksandar Jelenak hosted Call the Doctor on Tuesday, April 16, 2024. He talked about last week’s h5py 3.11 release and current interesting issues and PRs. Aleksandar has a background as a scientist working with geospatial and geoscience data, and now works at The HDF Group to make HDF5 and related tools more effective for other scientists.
This session happened on April 16, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group’s John Readey talked about using HSDS with Github Codespaces.
This session happened on April 9, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Neil Fortner presented a design for a new feature we’d like to implement to bring crash proofing to HDF5. This feature will allow HDF5 files to be easily recoverable, even if application execution is interrupted due to an application crash, system crash, or hardware failure. The feature is designed to be easy to use and add minimal overhead.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
If you are involved in research involving high-performance computing clusters, this Call the Doctor hosted by The HDF Group's Scot Breitenfeld provided support for using HDF5. Scot presented recent subfiling performance results and answered some community questions on the efforts so far.
This session happened on March 26, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's Aleksander Jelenak hosted this episode of Call the Doctor.
He went through some recent h5py issues and pull requests. He helped a user who was experiencing poor performance while streaming object references. He referenced this poster.
Aleksander also talked about fsspec a Python project which provides a file object-like interface for various remote storage systems.
This session happened on March 19, 2024. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Did you know that HSDS supports SQL-like query operations on datasets? In this session of Call the Doctor, John Readey describes this feature and illustrates with some live examples.
Links mentioned by John in the session:
- H5py + PyTables blog
- query read example
- query update example
This session was recorded on March 12, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, Director of Software Engineering Dana Robinson covers the schedule for new releases for HDF5, HDF4, and HDFView and what those new releases will include. A user question about when The HDF Group would change the name to HDF5 2.0.
This session happened on March 5, 2024. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor," director of software engineering, Dana Robinson covers updates and changes in the HDF4 library. Dana discusses HDF 4.3.0 (released 2/29/24) and version 4.4, with plans to maintain HDF4 indefinitely. The focus is on improving HDF4's compatibility with modern compilers and platforms, addressing issues like the xdr library's compatibility problems and the deployment of internal header files. Significant changes include removing xdr from configuration options, improving compiler support, and cleaning up memory sanitizer issues.
The episode also touches on restructuring the Fortran code, removing outdated Fortran 77 support, and potentially merging libraries into a single HDF4 library. Additionally, plans involve phasing out the old netCDF 2.3.2 API and associated tools in favor of newer alternatives.
The HDF Group will continue to modernizing HDF4 to ensure its long-term maintainability, despite potential disruptions caused by these changes. We do need to hear from users about these plans and how they might work or cause conflict with your code, so please reach out to us to let us know about your concerns on the HDF4 Forum or by emailing help@hdfgroup.org.
This session happened on February 27, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor," Aleksandar Jelenak discusses ways to document and describe the content of HDF5 files in order to generate documentation about the file content, as inspired by this forum post from Mike Jackson of Blue Quartz. The session emphasizes the need for a format that is approachable, shareable, and easily parsable by multiple programming languages. Jelenak discusses various options that have been used, including Excel spreadsheets, JSON, and text formats like YAML. He also presents his own idea of using YAML documents to describe the content of HDF5 files in a hierarchical and straightforward manner. The session concludes with a discussion about the importance of bidirectionality in the toolchain and the potential for future developments in this area.
This session happened on February 20, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, The HDF Group’s Matt Larson will give an introduction to the HDF in the Cloud tutorials:
- REST VOL,
- HSDS,
- ROS3. and
- H5Coro.
If you’re an experienced HDF5 user, come to this Call the Doctor session, take a look at the tutorial content, and feel free to make improvements or let us know of any issues.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
For this week's Call the Doctor, The HDF Group's Jordan Henderson, author of the Adding support for 16-bit floating point and complex number datatypes to HDF5 RFC, briefly talked through that RFC and lead a community discussion for feedback on the concept and proposal.
For more to discussion, and to participate in the discussion, please visit this forum post: https://forum.hdfgroup.org/t/hdf5-rfc-adding-support-for-16-bit-floating-point-and-complex-number-datatypes-to-hdf5/11975
This session happened on February 6, 2024. You can also watch this session online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
For an individual, creating an HDF5 tutorial is hard; making a good HDF5 tutorial is nearly impossible. There are many technical facets to cover; the global community it serves is phenomenally diverse, and so is its rich ecosystem. Wouldn’t it be great to have a tutorial that anyone could enjoy without pesky installation details, and everyone with an idea to improve upon it could contribute easily? Introducing the HDF5 Tutorial developed by the community for the community. Fork it on GitHub at https://github.com/HDFGroup/hdf5-tutorial.
The HDF Group's Executive Director Gerd Heber will uses this session of Call the Doctor to give an overview of the tutorial and introduce the underpinnings so that everyone so inclined can contribute and help create the best possible HDF5 tutorial ever.
Would you like to discuss this tutorial with Gerd and others? Come to the forum and let us know your thoughts.
This session was recorded on January 30, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor," Scot Breitenfeld of The HDF Group hosted an open help session for your HPC HDF5 questions. (Questions are welcome at any session, but we knew we had some community members with specific needs for this session.)
A research engineer working on open-source thermohydraulics code discusses issues with writing results in the CGNS (CFD General Notation System) format. The engineer explores different approaches, including writing the CGNS skeleton on the master processor and distributing data writing on multiple nodes. However, there are concerns about the slowdown when increasing time steps, possibly due to opening and closing the HDF file at each time step. The discussion also touches on options like sub-filing and using the core file driver to eliminate the file system component. Additionally, there's a brief inquiry about compressing large strings in HDF5. See this forum post for details. Overall, the episode addresses technical challenges in parallel file writing and optimization strategies.
This session happened on January 23, 2024. You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor," Aleksandar Jelenak delves into recent work on analyzing Earth science HDF5 files, comparing their original versions with cloud-optimized versions. Using data from the Global Ecosystem Dynamics Investigation Instrument (GEDI) on the International Space Station, he showcases the instrument's laser beams mapping Earth's vegetation. Aleksandar discusses the challenges of optimizing HDF5 files for cloud usage, emphasizing the need for user-friendly data for scientists worldwide. He presents a detailed analysis of chunk data sets, storage settings, and statistics, highlighting the potential benefits of cloud-optimized files. The episode concludes with performance comparisons between original and cloud-optimized files, shedding light on the advantages of efficient data storage and access.
This session was recorded on January 16, 2024. You can also watch it online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, The HDF Group's John Readey discusses upcoming features in the HSDS release. Since John is in China, the session is pre-recorded, with software engineer Matt Larson available to answer questions. The discussed features include shape reduction, broadcasting, UTF-8 fixed strings, quick scan, N-bit and scale offset filters, enhanced array type support, field ops for compound types, support for long attribute names, non-UTF-8 encodable attributes, multi-op attributes, long link names, and hyper chunking. John also introduces the concepts of async tasks for long-running operations and using Parquet for encoding chunks with variable-length types. The design doc for async tasks can be viewed on github. The episode includes a demonstration of the attribute multi-op feature, showcasing its efficiency compared to a serial approach.
You can participate in the discussion of features for HSDS 0.9 on the forum.
This session happened on January 9, 2024. You can also watch this episode.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
While we're off for this week, please enjoy this audio recording from the 2023 HDF5 Users Group meeting held in Ohio.
This session was presented by Aleksandar Jelenak, The HDF Group.
Zarr is a fairly recent format for multidimensional data arrays specifically targeting storage systems with key-value interface. Some scientific communities interested in implementing scalable cloud-native data analysis are considering Zarr as their chosen data format because of its straightforward implementation in cloud object stores. HDF Group had developed its own cloud-native HDF5 format, called HSDS schema, about the same time as Zarr. Only HDF Group’s developed software, HSDS, currently creates data in the HSDS schema. Since both Zarr and HSDS schema share the same design approach, it would be worthwhile to consider whether Zarr could serve as the cloud-native HDF5 format. The currently developed Zarr version 3 specification introduces the concept of extensions as a way to add more storage features. The goal of this session is to discuss pros and cons of using Zarr v3 to formulate a new cloud-native HDF5 format. Some technical information will be provided with aim to open up discussion among all attendees.
If you'd like, you can watch this session online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
This is a bonus episode recorded live in August 2023 at the HDF5 User Group meeting held in Ohio.
Dana Robinson and Neil Fortner talked about some future work The HDF Group is planning, plus took questions, input, and discussion from the audience. Unfortunately not every audience contribution will be easy to here but the majority of them in this recording are fairly audible. Sorry about that!
You can watch this session online or access the slide deck.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
While we're off for the holidays, enjoy this audio recording of the session, "State of HDF5 and New Features" from Dana Robinson (Director of Software Engineering) and Neil Fortner (Chief HDF5 Software Architect), presented at the August 2023 HDF5 User Group meeting held in Ohio.
Dana talked about some changes being made: the HDF5 Working Group meetings, the Sustaining Engineer of the Week, centering development on GitHub and the need for external Codeowners, a new process for Change management, and some of the HDF5 issues and development work we plan to focus on.
Neil talked about new features being added to HDF5: Multi Dataset I/O, Selection and Vector I/O, and the Subfiling VFD.
Note: At the end when we took questions from the audience, you'll find the audio is not that great. We apologize for that. Later sessions recorded during this event were better.
You can watch the recording of this session on youtube, and also access Dana's slide deck and Neil's slide deck if the audio experience isn't doing it for you.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group’s Aleksandar Jelenak will talk about the conda, mamba, and micromamba package managers. These package managers are frequently used in the data science communities. Aleksander talked about the differences between these packages and how to get started using them.
You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor," The HDF Group's John Readey explores the functionality of linked data sets in HSDS (Highly Scalable Data Service). Using a Python notebook running on AWS, he walks through examples using data from the National Renewable Energy Lab, which has substantial HDF5 and HSDS data freely accessible. John covers various aspects, including domain information, data set details, and how to read and analyze chunks. He delves into the specifics of the chunk layout, discussing file URIs, offsets, and sizes. Comparisons between HSDS and direct S3 access using the HDF5 library reveal differences in performance due to the sequential nature of the HDF5 library's requests. John concludes by demonstrating a new feature for querying specific data sets using hsls.
You can also watch this session online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor" hosted by Gerd Heber, the theme revolves around documentation, particularly in the context of large language models. The discussion begins with recent technical difficulties at https://portal.hdfgroup.org due to a cyber attack, resulting in the unavailability of certain documentation. The team is working on recovery, utilizing GitHub Pages and seeking community involvement. Gerd Heber introduces a project using ui.chat, demonstrating its capabilities by creating a custom chatbot trained on the HDF5 GitHub repository and RFC documents. While showcasing the chatbot's potential, the discussion touches on challenges, such as outdated documentation and the need for ongoing updates. The episode highlights the intersection of technology, documentation, and community engagement.
You can watch this video online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this session of Call the Doctor, The HDF Group's Matt Larson introduces the REST VOL (Virtual Object Layer) in the HDF5 library, focusing on its role as a VOL connector that maps API calls to rest requests sent to external servers, particularly HSDS instances. He emphasizes the flexibility of the VOL layer, allowing users to develop their own connectors, and compares REST VOL to alternative options for cloud-based HDF5 work. The talk covers REST VOL's history, recent updates, and performance advantages, showcasing its support for multi-read and multi-write operations. Matt also highlights new features, including support for fill values and enhanced symbolic link traversal, while discussing ongoing developments like automatic numeric type conversion and multi-server connectivity with a single REST VOL instance. The presentation concludes with an opportunity for questions, addressing topics such as dynamically loading multiple distinct REST VOL instances concurrently.
You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this session of Call the Doctor, The HDF Group's Dana Robinson discusses various topics related to the HDF5 library. He mentions an upcoming session at Supercomputing 23, provides information about the release schedule for version 1.14.3, and talks about a Fortran floating point issue. Dana explains the fix for the issue, which involves handling floating-point exceptions during initialization. He also outlines plans for version 1.14.4, including improvements such as Unicode file support on Windows, new data types, better testing for the read-only S3 VFD, addressing CVE issues, and other enhancements.
You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
A large part of the HDF5 usability story is the story of the HDF5 community. If it weren’t for community projects such as PyTables, h5py, rhdf5, HDFql, etc., the HDF5 library would have remained an obscure and highly unusable technical detail. I want to use this HDF clinic to draw your attention to PureHDF , an independent implementation of HDF5 in C#/.NET. By ‘independent,’ I mean independent of the HDF5 library. This is no small achievement and offers a fresh perspective on what’s possible with modern languages and platforms. If you thought that .NET was a Windows-only Microsoft thing, then this is your chance to experience a modern, free, cross-platform, open-source developer platform for building many different types of applications, and HDF5 is a first-class citizen.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
HPC Lead Scot Breitenfeld of The HDF Group discussed the design of Fortran wrappers for those who want to learn more about this aspect of the HDF5 library.
This session happened on October 17, 2023. If you'd like, you can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Aleksandar Jelenak of The HDF Group covers what’s new in the recent release of h5py-3.10 and then talked about the Read-Only S3 virtual file driver logging. Finally Aleksasndar introduced a demo Binder web app he created for anyone to analyze such logs.
This session happened on October 17, 2023. You can also watch this session online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor," John Readey goes over the HSDS (Highly Scalable Data Storage) package. John starts by explaining the origins of HSDS as a Python code designed to run in Docker, and later introduces HSDS Python Package for convenient desktop application usage. He walks through the installation process, demonstrating how to create a new Python environment and set up the HSDS package and the flexibility of HSDS, showcasing data loading, metadata organization, and other essential features. The session aims to provide an accessible entry point for both new and experienced users, making the power of HSDS more accessible.
This session happened on October 10, 2023. You can also watch this episode.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, Dana Robinson provides updates on the HDF5 release schedule and discusses recent improvements in security. He touches on the upcoming releases, including 1.12.3 and 1.14.3, emphasizing the importance of these updates. Dana also highlights the efforts to address security issues and mentions the integration of static code analysis through tools like Coverity. Additionally, he introduces a new working group dedicated to enhancing HDF5 compression filters, aiming to establish clearer standards and documentation for filter implementation. More info on this working group will be posted on the forum (https://forum.hdfgroup.org) and you can reach out to Dana at derobins -at- hdfgroup.org to get a meeting invite or more info.
You can also watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, Neil Fortner discusses an architectural documentation project for HDF5 at The HDF Group. The document aims to provide an overview of HDF5's architecture, targeting various audiences: community contributors, new library developers, extension developers, and power users. The documentation includes guided tours (higher-level explanations) and a reference section (more in-depth discussions). Topics covered range from initialization processes, file objects, and object headers to parallel hdf5 and performance considerations. The document also outlines hdf5's package dependencies and introduces features and compatibility matrices.
Neil welcomes feedback from anyone interested in the project. Email Neil to get involved.
This session happened on September 26, 2023.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
This podcast episode is an audio only recording of a presentation by Mark C. Miller at the 2023 HDF5 User Group meeting. You can watch this presentation or check out the youtube playlist for the entire event.
Bonus Episode Description:Some of the earliest conversations about the development of HDF5 took place as part of the Data Models and Formats (DMF) effort of the Accelerated Strategic Computing Initiative (ASCI). ASCI-DMF was a tri-lab (LLNL, SNL and LANL) effort to develop a common and flexible scientific data modeling API and file format for the emerging generation of scalable parallel, high performance computing applications starting in the late 1990s. The NCSA HDF team participated in ASCI-DMF efforts beginning with the initial development of HDF5. Early designs of HDF5 benefited from lessons learned and knowledge gained from not only its predecessor, HDF4, but also from several other tri-lab data management technologies such as XFiles, PDB, Aio, Silo, ExodusII, Nemesis, and SAF. We’ll take a walk through this early history and some of the influence it had on the early development of HDF5.For more information on this conference including all sessions and slide decks, visit https://www.hdfgroup.org/hug/hug23/agenda
To learn more about upcoming HUG events, please visit https://www.hdfgroup.org/hug
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
This podcast episode is an audio only recording of a presentation by Mike Folk at the 2023 HDF5 User Group meeting. You can watch this presentation or check out the youtube playlist for the entire event.
Bonus Episode Description:HDF5 has endured and grown for a quarter century thanks to countless contributions from people, organizations, and applications. We will share our perspective on those 25 years. We will visit some drivers that led to creating HDF5, highlight some early adopters and applications that helped HDF5 to gain acceptance, look at some interesting and fun uses of HDF5, call out the people and organizations that have kept HDF5 alive and healthy for a generation through their guidance and support, and describe the evolution of HDF5 over time.
For more information on this conference including all sessions and slide decks, visit https://www.hdfgroup.org/hug/hug23/agenda
To learn more about upcoming HUG events, please visit https://www.hdfgroup.org/hug
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of Call the Doctor, Scot Breitenfeld introduces the h5fuse.sh script, an updated utility for consolidating sub-files into a single hdf5 file. Using Cabana as an example application, he showcases the script's practicality. Breitenfeld presents performance metrics, comparing writing with no local storage to writing as a single shared file on the Leicester file system, with the latter demonstrating notable improvement. He also explores the effectiveness of Progressive File Layout on Frontier, which surprisingly outperforms manual configuration of stripe size and count. However, execution times vary, ranging from 5 to 35 seconds per node.
Throughout the discussion, potential enhancements are mentioned, such as employing more sub-filing threads and multiple processes per node. Breitenfeld also suggests the possibility of a workshop feedback survey to gather user input. He reports AMR failures on AArch64 architecture and highlights upcoming releases (1.13.3 and 1.14) addressing bug fixes and enhancements. The presentation concludes with addressing ongoing challenges related to NFS compatibility and outlining plans for further development in this domain
This session happened on August 22, 2023. You can also watch this video online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this "Call the Doctor" session, John Readey addresses various topics related to HSDS (Highly Scalable Data Service and H5 REST VOL (HDF5 REST Virtual Object Layer).
In this "Call the Doctor" session hosted by John Readey, discussions revolve around various topics related to HSDS (HDF Scalable Data Service) and HDF5 (Hierarchical Data Format). The session addresses challenges with accessing NREL data using the HSDS API and offers solutions for setting up HSDS locally on different platforms. A 503 error when connecting to hsds.atrip.org is linked to HDF Lab, prompting considerations for alternative setups. Adding data from Azure storage to the HSDS server is discussed, as well as authentication methods and plans to enhance the Rest VOL connector.
Forum posts discussed:
https://forum.hdfgroup.org/t/how-much-is-too-much-for-the-nrel-api/11400/
https://forum.hdfgroup.org/t/http-hsdshdflab-hdfgroup-org-gives-503-service-unavailable/11376/
https://forum.hdfgroup.org/t/how-to-add-data-in-azure-storage-container-to-hsds-server/11365/
https://forum.hdfgroup.org/t/alternative-bearer-token-generation/11387/
https://forum.hdfgroup.org/t/vol-rest-h5is-accessible-and-h5dset-extent/11375/
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode, Dana Robinson begins the session with an update to the release schedule for the HDF5 library. They are about to release version 1.14.2, followed by the final release of version 1.10 at the end of September. Additionally, there will be another release of HDF5 1.14 for the ECP project at the end of October. HDF5 1.12.3 is expected to release at the end of November.
They discuss some pending fixes related to CVE (Common Vulnerabilities and Exposures) issues in HDF5 1.14.2 and how they plan to address them before the release. The conversation also touches on cross-compiling issues with the NAG Fortran compiler and potential changes to the HDF5 API.
Later in the episode, Lori Cooper talks about the upcoming HDF Group User Group (HUG) event scheduled for August 16th to 18th. She invites both in-person and virtual attendees to participate in the event. The HUG will feature sessions from community members and HDF Group staff, as well as tutorials on HDF5 and related topics.
Towards the end, Aleksandar Jelenak briefly discusses the relationship between HDF5 JSON (HDF5js) and the HDF Server (HSDS) and how HSDS inherited some components from HDF5 JSON. Dana also mentions the ongoing discussion about HDF5 version 2.0 and the plans to make certain changes and improvements in the library. The attendees are encouraged to share their feedback and ideas for the future of HDF5.
This session happened on August 1, 2023. You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In the July 25th episode of "Call the Doctor" hosted by Gerd Heber, the focus was on using AI tools, specifically Chat GPT, to perform practical tasks related to HDF5 and image processing.
The episode started with a brief introduction, and then Gerd mentioned that he received a directory filled with TIFF image files and wanted to explore ways to combine them into a single HDF5 file. He emphasized that the goal was to let Chat GPT write the code for this task, avoiding manual coding.
Gerd interacted with Chat GPT, presenting different prompts related to the task. Initially, he asked for a Python program to create an HDF5 file, and Chat GPT provided the code using the Python Imaging Library (PIL) module.
Next, Gerd requested the code to compress the images while writing them into the HDF5 file. Chat GPT complied and provided code for compressing the images using GZIP compression.
The third prompt asked for the images to be combined into a 3D dataset instead of individual 2D datasets. Chat GPT produced code to create a 3D dataset and efficiently stack the images into it.
In the subsequent prompt (version 2.1), Gerd suggested that Chat GPT should infer the dimensions of the images from the provided TIFF files instead of explicitly specifying them. Chat GPT accurately determined the image dimensions and generated the code accordingly.
Finally, Gerd introduced version 2.2 and 2.2.1. He explained that when using version 2.2, HDF5's default chunking strategy resulted in poor performance when dealing with compressed 3D datasets. To fix this, he asked Chat GPT to modify the code to manually set the chunk size for each image to achieve better performance. Chat GPT provided the code accordingly.
Throughout the episode, Gerd showcased how Chat GPT effectively wrote Python code to handle various aspects of the task, making the process efficient and practical. He also mentioned that the integration of GitHub Copilot with Visual Studio Code had been helpful in assisting with the code generation process.
In conclusion, the episode demonstrated the power of AI tools like Chat GPT and showcased how they could assist in practical tasks related to HDF5 and image processing, making the development process more efficient and less manual.
(This description was created by ChatGPT, naturally. :) )
This session happened on July 25, 2023. You can also watch the video for this session online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this episode of "Call the Doctor," Scot Breitenfeld discusses the use of the h5fuse script and subfiling VFD in HDF5. The discussion begins with questions about the typical users of subfiling VFD, which is commonly used by scientific and engineering applications that deal with large HDF5 files in parallel. The subfiling VFD is designed to address issues with writing to a single file when dealing with large-scale problems involving thousands of ranks.
Scot then explains how to use the h5fuse script within an application. He demonstrates an example using the ExaMPM application, which performs time steps and outputs HDF5 files. The h5fuse script is called after each time step to fuse the subfiles into a valid HDF5 file. Scot provides code snippets and explains the process of determining the location of the configuration file, setting up the h5fuse utility, and checking the completion of the process.
He also shares performance results of fusing files on the Frontier system, where he ran the application on 1024 nodes. The average time taken to h5fuse a nearly 1 TB file was around 40 seconds, while the simulation time remained consistent with or without the h5fuse operation.
During the Q&A session, Scot clarifies that the h5fuse script is executed on all nodes but only fuses the sub-files accessible to each node. He also explains that modifying NetCDF or CGNS libraries for subfiling involves replacing the MPIO driver with the sub-filing driver and configuring the application to use sub-filing through environment variables or hardcoded options.
This session happened on July 18, 2023. You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
The HDF Group's John Readey and Matt Larson discuss the multi-dataset functionalities in HDF5 and the potential for parallel processing in HPC applications. They explain how the HDF5 library now allows users to write or read from multiple datasets with multiple selections in a single call. They highlight the implementation of this feature in the HSDS (Highly Scalable Data Service) and its advantages for improving performance in client-server-based applications using cloud storage. Matt shares his benchmark results, demonstrating the significant speedup achieved by rearranging the requests using the H5D read/write multi functionality. John also presented a Python analog called "multi-manager," which provides an easy way to manage and perform actions on multiple groups, attributes, and datasets. They discuss the limitations and potential benefits of the H5D read/write multi functionality and its integration with HSDS. Overall, the video explores the possibilities of parallel processing and the potential for improving performance in HDF5 applications.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Registration is now open!!! ➡️ https://www.hdfgroup.org/hug/hug23
Join us in Columbus, Ohio August 16-18, 2023 in Scott Laboratory on The Ohio State University campus. This year is extra special, as we are celebrating 25 years of HDF5! HDF5 was developed November 6, 1998 at the National Center for Supercomputing Applications, on the campus of the University of Illinois. We’re celebrating a little early but hope you will join us. We’ve lined up great keynote speakers, fun lunch talks, a group dinner out, and a full day of tutorials on day 3.
Complete information, including information on the conference’s hotel room block, daily schedules, and more can be found on the conference website.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
In this session of Call the Doctor, Scot Breitenfeld discusses a program called hdf5-iotest available on GitHub. The program helps explore data layout and performance optimization in HDF5. It allows users to experiment with parameters such as data dimensionality, chunking, metadata size, and MPIO operation. The program generates results and timings for different combinations of these parameters. Scot also mentions additional research additions, including support for collective metadata apis, compression filters, split I/O driver, and mpio driver. He demonstrates the effectiveness of the log-based VOL and the impact of tuning parameters with the DAOS VOL. Overall, the program aids in optimizing HDF5 performance and data organization.
You can also watch this episode.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Aleksandar Jelenak covered the latest release of h5py and walked through some of its recent issues.
This session happened on June 13, 2023. You can watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Dana Robinson, Director of Software Engineering, subbed in for Call the Doctor. Dana talked about some changes he has made to make cross compilation should be a lot easier--these changes will go to HDF5 1.14. Dana went through the code modifications, including the creation of template files and library initialization, and covered addressing issues related to cross-compilation and their plans to improve MinGW build support. The resulting changes in the library's behavior and compilation information remain the same.
This session happened on June 13, 2023. You can watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Dana Robinson, Director of Software Engineering at The HDF Group talked about the new CVE repository and test script, plus the new change policy we'll be using to work with the community.
Also, every Thursday Dana hosts an HDF5 working group at 10 am central time. Email derobins at hdfgroup.org for the Teams meeting invite. During this session Dana will discuss development and go over PRs and issues, and community developers can also bring up any technical issues they'd like to discuss.
This session happened on June 6, 2023. You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Gerd Heber hosted Call the Doctor and tested ChatGPT on its skills with HDF5. He evaluated ChatGPT's response on several coding questions around HDF5 and talked about how newbies might use ChatGPT to learn how to use HDF5.
This session happened on May 30, 2023. You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Neil Fortner, Chief HDF5 Software Architect, hosted Call the Doctor to talk about Selection I/O, a new feature in HDF5 1.14.1. This feature allows the library to pass a non-contiguous I/O operation to the file driver in a single function call, and can also be used to enable collective I/O with type conversion. Neil talk about:
- What selection I/O is,
- Benefits of and requirements for selection I/O,
- Benefits of and requirements for collective type conversion,
- Additions to the VFD layer, and
- Other API additions.
This session happened on May 23, 2023. You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Aleksandar Jelenak presents a proposal for HDF5 columnar table data convention. This is continuation of what Aleksandar talked about during his last Call the Doctor session.
This session was recorded on May 16, 2023. You can watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
John Readey of The HDF Group and lead developer for HSDS (Highly Scalable Data Service) went over some recent HSDS related tickets/forum posts.
You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Dana Robinson, Director of Software Engineering, announced some changes he’s making to the way The HDF Group interacts with our community members.
If you would like to communicate directly with Dana, you can email him at derobins -at- hdfgroup.org.
HDF5 WORKING GROUP
There will be a 1-hour weekly meeting on Thursday at 10 am CDT where we will go over more complicated pull requests, new issues, and discuss library development. This meeting is open to the public. For right now, this will require a Teams invite, which I will be happy to hand out to anyone who wants to join.
We should probably emphasize that this is NOT a replacement for the Call the Doctor session. It’s not for people to ask technical questions. It’s only for developers or people who care about a PR or issue.
HDF5 INTERNALS SEMINARS
These will be informal seminars about library internals on the 4th Friday of every month at 10 am CDT. These will be open to the public, but will also require a Teams invite for now. These will be VERY low level and targeted at library engineers.
BETTER GITHUB ENGAGEMENT
We will be dedicating a rotating engineer to keep an eye on GitHub, so issues should be tagged and triaged within a day of being created. This person will also be responsible for helping to get simpler PRs merged (more complicated PRs will have to wait for the working group meeting, above). The tagging system we have adopted in HDF5 will be exported to HDF4, HDFView, and hsds and I’ll demonstrate that. Product-specific project management on GitHub is being investigated, but is still a work in progress.
HDF5 1.10 RELEASE CHANGES
HDF5 1.10.11 will be released in the fall. This will be the last release of the 1.10 line.
HARDENED FILE I/O AND THE CVE REPOSITORY
We’ve made a pass over the library to harden the layer that reads HDF5 metadata from the disk. It’s now more careful about checking for invalid files and cleaning up resources when mis-parsed files are encountered. This was a frequent source of CVE and oss-fuzz issues. We are also putting together a repository that will consolidate CVE proof-of-concept files and a test script that can be run against any version of the library to see which CVE issues are unfixed (and to ensure we don’t regress).
https://github.com/HDFGroup/cve_hdf5
CHANGE COMMUNICATION
I’ll introduce our scheme for communicating changes to our products via the forums and other modes of communication.
You can also watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Scot Breitenfeld provides an update on CGNS results with subfiling on Summit.
This episode was recorded on April 25, 2023. You can watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Aleksandar Jelenak talked about the Anndata Python package in the context of HDF5 file content interoperability. This session included a discussion with attendees about columnar tables.
This episode was recorded April 18, 2023. You can watch this episode online.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info at https://www.hdfgroup.org/hug/hug23/
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
We started off this episode of Call the Doctor with John Readey working through some of the issues raised on this article "What Comes After HDF5? Seeking a Data Storage Format for Deep Learning." We will probably pick up this discussion with John's session on the second Tuesday in May.
Next, Gerd talked about his experience playing with GitHub CoPilot and made plans to talk more during his next Call the Doctor session on May 30, 2023. Then we had a mostly fun discussion about the use of ChatGPT and other AI in documentation and coding help.
Join us at our 2023 HDF5 User Group (HUG) meeting in Columbus, Ohio
August 16-18, 2023! More info: https://www.hdfgroup.org/hug/hug23/
This session happened on April 11, 2023. You can watch this episode.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
Join us every Tuesday at 12:20 p.m. central (US/Canada.) on Zoom!
Dana Robinson discussed upcoming release dates and will also talk about VFD SWMR (latest RFC here), a new implementation of the single-writer/multiple-readers I/O pattern that works with most HDF5 API calls. This feature was intended for release in HDF5 1.14.0, but has been delayed. Dana talked about the current status of the feature, what it will take to bring it up to speed, and our future plans for it.
This session happened on April 4, 2023. You can watch this episode.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
Scot Breitenfeld hosted Call the Doctor Tuesday, 3/28 and reviewed the introduction of HDF5 in Cabana, a portable library for particle-based simulations. Topics include interfacing HDF5 with Kokkos and GPU memory, HDF5 design, and a comparison with HDF5 and, alternatively, Silo I/O performance for ExaMPM.
This session happened on March 28, 2023. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
In Call the Doctor on Tuesday, March 21st, Chief HDF5 Software Architect Neil Fortner subbed in to talk about The HDF Group’s work on the HDF5 multiple dataset APIs and highlight the performance benefits when using them.
The HDF5 library allows a data access operation to access one dataset at a time. However, accessing multiple datasets requires the user to issue an I/O call for each dataset. Hence, the new multiple dataset APIs allow users to access multiple datasets with a single I/O call. In addition, the new routines can improve performance, especially when data is accessed across several datasets from all processes.
This session happened on March 21, 2023. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
Dana Robinson, Director of Software Engineering, talked about upcoming HDF5 and HDFView releases, plus his plans to bring back the Technical Advisory Board and adding people as Codeowners in Github.
This session happened on March 14, 2023. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
John Readey hosted Call the Doctor on March 7. John is part of our HDF5 cloud team, and for this episode, John talked about HDF5 object and region references.
This session happened on March 7, 2023. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
Scot Breitenfeld discussed and provided results for using node-local storage with subfiling on Summit. Scot is one of The HDF Group's resident HPC experts.
This session happened on February 28, 2023. You can watch this episode online.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
Senior informatics architect Aleksandar Jelenak hosted Call the Doctor on Tuesday, February 21. Aleksandar has a background as a scientist working with geospatial and geoscience data, and now works at The HDF Group to make HDF5 and related tools more effective for other scientists. This week, Aleksandar gave a run down on some current developments in h5py,
This session happened on February 21, 2023. You can watch this episode online.
Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
Dana Robinson, Director of Software Engineering at The HDF Group, hosted Call the Doctor on Tuesday 2/14 at 12:20 p.m. Central time. Dana will be talking about the HDF4 update that we are finishing up right now, plus some discussion of future plans and usage of HDF4.
This session happened on February 14, 2023. You can also watch this episode.
Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at https://us06web.zoom.us/s/9828688008
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
John Readey provides a preview of a revamped set of command line tools in version 0.13.0 of h5pyd https://github.com/HDFGroup/h5pyd which will be released this week.
In the beginning, John referenced this link (also displayed on screen):
https://portal.hdfgroup.org/display/KITA/HDF+Lab+Frequently+Asked+Questions
This session happened on February 7, 2023.
Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at https://us06web.zoom.us/s/9828688008
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)
This Call the Doctor session is about what we hope will turn into a community project. It’s called HDF5 Poker. It’s everything everyone ever wanted to know about the HDF5 file format (but didn’t dare ask). Raise your game here: https://github.com/HDFGroup/hdf5-poker ! Presented by Gerd Heber.
This session happened on January 31, 2023. You can watch this session on Youtube.
Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
For future events please submit your questions in the google doc and join us on zoom Tuesdays at 12:20 p.m. central (US/Canada.)