nf-cast - the bioinformatics podcast: Recent Episodes
Seqera Labs
Bioinformatics is advancing at a rapid pace and plays a vital role in the research, as well as development of new targeted medications, personalized healthcare, and delivering better clinical outcomes. In this podcast, Rob Lalonde of Seqera discusses advancements at the intersection of biology and informatics with some of the world’s leading bioinformaticians and data scientists.
A small but intrepid team of Seqerans recently attended the ISMB 2024 conference in Montreal, a great mix of computational biology and bioinformatics with a largely academic / research-focused audience. Join Geraldine Van der Auwera, Rob Syme and Florian Wuennemann for a lively discussion about the scientific themes of the conference, their experience running the booth, meeting with the ISMB and BOSC community, and resources for job seekers.
Join us in exploring the latest Nextflow release, 24.04. Phil Ewels and Ben Sherman dive deep into the new updates in Nextflow’s most recent version. We discuss advanced retry strategies, job arrays, resourceLimits, Singularity’s OCI mode, and the game-changing Workflow Output Definition.
We’re actively seeking feedback from the community on these new features. We would love for you to experiment with the new syntax and let us know your thoughts.
Please let us know what you think either via Nextflow GitHub issues or via the community forum.
In this episode we refer to a the recent blog post about the release, which you can find here: Nextflow 24.04 - Release highlights.
Intro - tweaks, fixes and new support in 24.04
Performance and Stability Improvements
Ben kicks off the discussion with key performance and stability improvements. The latest stable release, 24.04, includes numerous bug fixes and performance enhancements. There’s a significant focus on closing gaps and adding retry strategies for better stability, especially concerning cloud providers. Notably, if Nextflow’s API calls against cloud providers fail for any server-side reason, it will retry automatically, avoiding pipeline failures caused by temporary server issues.
Publishing Changes
Previously, if a file failed to publish, Nextflow would only issue a warning. It was possible for a pipeline to complete successfully without noticing missing output files. Now, by default, Nextflow will fail the pipeline if the publishing fails, though there’s an option to revert to the old behavior. Additionally, retry strategies for publishing have been introduced to ensure retry attempts if an issue arises.
Job Arrays
One highly requested feature is the introduction of job arrays. This allows users to submit many jobs as a single submission, alleviating strain on schedulers. The submission happens in a batch, and then the scheduler can process and plan effectively. Once a job is submitted as part of a job array, the jobs run independently. If any child job fails, it is resubmitted without affecting the rest of the array.
Singularity OCI mode and GA4GH TES
In the past, Singularity supported Docker images by converting them into SIF files, consuming storage and time. Now, both Singularity and Apptainer can run OCI images directly, saving valuable resources.
Additionally, the TES executor has seen significant improvements, now supporting TES 1.1, which brings broader compatibility and integration with existing workflows.
Major new syntax features
Topic Channels
A new feature called topic channels offers a more straightforward approach to collecting channel outputs across pipelines. Channels can emit data to a named topic, simplifying the collection and use of version information from various processes within Nextflow pipelines.
Eval Outputs
Eval outputs simplify the addition of shell commands to tasks. With eval outputs, necessary post-task commands can be defined neatly, avoiding repetitive code within process definitions.
Workflow Output Definition
The concept of workflow output definitions has been introduced. This new syntax streamlines the publication of files by defining publish targets within workflows. Instead of defining publication behaviors within process definitions, users can now manage them at the workflow level, ensuring better clarity and fewer repetitions in the code.
Join us to hear Phil Ewels chat with Maxime Garciaand Franziska Bonath about how nf-core started at theNational Genomics Infrastructure (NGI) at SciLifeLab in Sweden.We chat about how the nf-core/sarek pipeline grew alongside and eventually joined nf-core,as well as the early days of nf-core bytesize talks and outreach organisation.Links:- SciLifeLab- National Genomics Infrastructure- Maxime's Nextflow Summit Boston 2023 presentation about Sarek- nf-core bytesize talks
Tune in to hear all about Pixelgen Technologies - creators of a novel technologyfor doing single-cell spatial proteomics.
We chat about how this fascinating technology works, what you can do with the dataand why they chose to develop their analysis suite as an open-source nf-core pipeline.
Links:
- Software:
- nf-core/pixelator pipeline
- Pixelator Python package
- Science:
- Nature Methods publication: “Molecular pixelation: spatial proteomics of single cells by sequencing”
- Presentations:
- Nextflow Summit 2022 talk: “Pixelgen Technologies ❤︎ Nextflow”
- Nextflow Summit 2024 Boston talk: “Nextflow allows Pixelgen innovation engine to reach the market quicker”
- Nextflow Summit 2024 Barcelona
- Pixelgen:
- Pixelgen Technologies homepage
- @PixelgenTech on twitter / X and LinkedIn
- BioLizard:
- BioLizard homepage
- @BioLizard_nv on twitter / X and LinkedIn
Chapters:
- Start (00:00)
- Welcome (00:09)
- Introductions (00:50)
- Intro to Pixelgen Technologies (01:41)
- Lab methods (03:08)
- Nature Methods publication (06:15)
- Data visualisation (09:35)
- Working with single cell data (12:29)
- Cell interactions (15:06)
- Why Pixelgen chose open source (16:05)
- Pixelator Python package (20:10)
- Building within nf-core (21:40)
- Florian - Biolizard (23:01)
- Pipeline overview (25:19)
- Using nf-core from a commercial setting (26:50)
- Coming out of stealth (28:53)
- Extending Pixelator (32:24)
- Dealing with customers (34:58)
- Looking to the future (39:38)
- Interactive analysis (41:32)
- Advice for others (43:12)
- Pixelgen at the Nextflow Summit (45:16)
Evan Floden (CEO and co-founder of Seqera) joins Phil Ewels (Product Manager for OSS at Seqera) to talk about two of the big announcements at the recent Nextflow Summit Boston 2024 - Seqera Pipelines and Seqera Containers.We dive into what these two new community offerings are, why we built them and how they work. Links:
- https://seqera.io/pipelines/
- https://seqera.io/containers/
- Announcement blog post
- Nextflow Summit Boston 2024 agenda
- Summit YouTube playlist
- Nextflow 24.04 blog post
Timeline:
00:00 - Introduction
00:37 - Nextflow Summit Boston 2024
01:14 - Summit talks available online
01:57 - Summit announcements
04:32 - Seqera Pipelines - motivations
05:59 - Seqera Pipelines - first look
06:48 - Testing requirements for community pipelines
10:05 - How we tested pipelines
11:53 - SRA-Explorer tangent
12:34 - Seqera Pipelines - live demo
16:14 - Getting Nextflow launch commands
19:12 - Seqera Containers - intro
24:03 - Seqera Containers - live demo
27:13 - Containers - ARM cpu arch builds
28:33 - Singularity containers
29:09 - Wave CLI
31:19 - Nextflow with Seqera Containers
33:22 - Seqera Containers - future usage
35:09 - Where to read more
36:53 - Thank you to the community
Join us as we discuss the brand new @nf-core / EuroFaang pre-print! 🔥 We chat to some of the key authors, covering the background to the paper, the main messages and how EuroFAANG and nf-core have collaborated. 🧬 👩🏻🔬
Join us in this episode of the Channels podcast for a closer look at the Nextflow Ambassadors Program. Marcel Ribeiro-Dantas and Geraldine Van der Auwera discuss the origins of the program, recent highlights and next steps, including the open call for new Ambassadors, which will close June 14.
Relevant links:
- Call for applications: https://www.nextflow.io/ambassadors.html
- Original announcement: https://www.nextflow.io/blog/2023/introducing-nextflow-ambassador-program.html
- Marcel’s favorite Ambassador-contributed Nextflow blog post: https://www.nextflow.io/blog/2024/reflections-on-nextflow-mentorship.html
This podcast episode is quite different to normal. Rather than an interview or a discussion, you'll get to watch as Ben Sherman guides Phil Ewels through the process of creating a new Nextflow Plugin from scratch. Right from the first line of code to creating a release and publishing the plugin.Nextflow plugins have been getting more attention lately and we've been getting a lot of questions in the community. There is better documentation and resources planned, but in the mean time we hope that this can be a useful guide for anyone curious in how they work, and interested in getting started.After this we will be back to our usual routine in the podcast, but we'd love to hear what you thought of this taster!
- 00:00:00 Welcome
- 00:03:27 Nextflow docs
- 00:08:34 Starting by forking nf-hello
- 00:10:32 Overview of project files
- 00:17:13 Trying a first compile
- 00:18:12 Different publishing methods
- 00:19:54 nf-boost local publish method
- 00:23:05 Trying the new compile
- 00:24:00 Running locally with the plugin
- 00:27:32 Looking at the nf-hello plugin code
- 00:39:29 Deleting files we don't need
- 00:42:34 Finding event names in the Nextflow source
- 00:47:46 Writing some custom functions
- 00:49:41 Testing our new code
- 00:51:51 Modifying the test pipeline to create files
- 00:54:14 Coding up the JSON output
- 00:59:20 Looking at nf-prov code for BCO files
- 01:05:30 Testing JSON output
- 01:09:39 Automatic work dir cleanup / nf-boost
- 01:12:17 Publishing a release on GitHub
- 01:19:07 Custom plugin repositories
- 01:26:47 Publishing for all Nextflow users
- 01:29:52 Conclusion and end
In this episode of Channels, we talk to Jakob Zeitler - Head of R&D at Matterhorn Studios.We dive into how they use Nextflow and Seqera Platform for material science research, paving the way for cheaper and more eco-friendly products for the future ⚗️👩🏻🔬🔬🌎It may not be bioinformatics, but there's a lot that's in common!
In this episode of the podcast, Geraldine and Ben talk about the Global Alliance for Genomics and Health (GA4GH) and its efforts to develop interoperability standards for pipelining infrastructure.If the words TES, WES, TRS and DRS are just as much alphabet soup to you, tune in to learn all about how organizations around the world are collaborating to streamline the process of sharing and analyzing data at scale.
In this episode of Channels, we talk to Josh Chorlton - CTO & cofounder of BugSeq Bioinformatics Inc.We talk about how BugSeq got into using Nextflow and MultiQC and the tips and tricks that they've employed to push scale and performace of their tools to the limit.We cover topics like testing and CI/CD, passing around structured data objects between Groovy and Python using Protobufs, building Docker images at scale and the ins and outs of using MultiQC for custom clinical data reports.
A recap of our current training formulas, how we got here and where we’re heading to next.
Join us in this episode of the Channels podcast as Geraldine Van der Auwera and Chris Hakkaart discuss options for learning Nextflow in 2024. A recap of our current training formulas, how we got here and where we’re heading to next.
Upcoming training opportunities
- Community Foundational Nextflow Training — Online, 5-6 March 2024
- Training at Seqera Sessions — London, 21 March 2024
- Training at Nextflow Summit — Boston, 21-22 May 2024
Other upcoming events of interest
- nf-core Hackathon — Online + Local Hubs, 18-20 March 2024
In this episode of Channels, we talk to Paolo Di Tommaso (creator of Nextflow, Seqera CTO & cofounder) and Jordi Deu Pons (software engineer @ Seqera) about Fusion - a file system written specifically for Nextflow.
Join us in this episode of the Channels podcast to get a recap of some of the biggest features to be added to Nextflow in 2023 and take a look at some of the things coming in 2024.
We tried to do this in Episode 27 but ended up spending nearly all the time discussing community and nf-core, so this episode is dedicated to just Nextflow features.
We cover Phil's top hits:
- 2023
- Fusion support on Azure Batch, Google Batch, SLURM, LSF
- Spack integration
- Markdown docs, developer docs
- New
nextflow inspectcommand - Channel "topics"
- AWS Fargate for compute tasks
- 2024
- Job arrays
- Garbage collection (aka work directory cleanup)
- Command line interface v2
- Improvements to Nextflow packaging
- Workflow inputs and outputs schema
- Module configuration / config v2
Rob Lalonde of Seqera interviews Kelsey Florek, Senior Genomics and Data Scientist at the Wisconsin State Laboratory of Hygiene. In this interview, Rob and Kelsey discuss how genomic surveillance has changed since the COVID pandemic, and how Seqera Platform and Nextflow pipelines have substantially lowered the barrier to entry for bioinformaticians running workflows in public health labs and in the field.The viralrecon pipeline Kelsey mentioned in the interview can be found at https://nf-co.re/viralrecon
Today in the Channels podcast, we're talking about what happened this year in the Nextflow world, with particular emphasis on the many exciting developments in nf-core.We cover:- Community growth (with stats and graphs!)- The evolution of nf-core governance- The latest crop of nf-core pipelines that saw their first release this year and the expansion into domains beyond genomics (even beyond biology!)- Infrastructure updates including subworkflows and testing with nf-test- Community updates including mentorships and training- Upcoming events in 2024: nf-core online hackathon in March, Boston Summit in May and Barcelona Summit in OctoberAfter today’s episode, we’ll be going on break for the winter holiday and will return in January 2024 with a fun mix of topics. Stay tuned!
In a special edition of the Channels Podcast, Geraldine Van der Auwera interviews four members of the Nextflow community attending the 2023 nf-core hackathon: Raquel Manzano, Edmund Miller, Rieke Hanssen and Francesco Lescai. Recorded in-person at the Nextflow Summit Barcelona, we hear a bit about what they have been working on, how they came to use Nextflow and what motivated them to join the nf-core hackathon.
In a special edition of the Channels Podcast, Phil Ewels chats to Geraldine Van der Auwera! Recorded in-person at the Nextflow Summit Barcelona, we hear a bit about Geraldine’s background - her work at the Broad with GATK and WDL, and the path that brought her to Seqera and the Nextflow community.Oh, and the WDL runner whose name got away is miniWDL, by Mike Lin!
Today in the Channels podcast, we're talking about computing resources... Ben Sherman takes us through new and upcoming runtime improvements in the Nextflow, leading to better performance fine tuning and simpler syntax for pipelines.
We cover:
- Fractional CPU requests
resourceLimitsdirective (to replacecheck_max)- Virtual threads in Groovy
- WebAssembly runtime (a lightweight alternative to containers)
Software containers using Docker / Singularity are a core part of modern data science, found at the heart of Nextflow pipelines. On today's show, we bring on the creator of Nextflow – Paolo Di Tommaso – to tell us the good, the bad and the ugly of containers.
He talks about "Wave", one of the newest tools to come out of Seqera Labs and how this can make Nextflow pipelines scale on the cloud, work behind closed doors in protected environments and how it can build & augment images with additional requirements on the fly. And all this whilst reducing the amount of boilerplate code required within your pipeline.
As if that wasn't enough, we get a live demo of Wavelit - the open source command-line interface tool, that allows you to leverage the full power of Wave as a stand-alone tool, outside of Nextflow.
In this episode of the Channels podcast, it’s all about developer experience! Ben Sherman takes us through new and upcoming language improvements in the Nextflow language which will help people writing Nextflow pipelines write code faster, with fewer errors, and of course with beautifully formatted indentation 🤩
We cover:
- What exactly was happening in the last episode, when Marcel, Chris and Phil ran wild with a beta version of breakpoint debugging in IntelliJ IDEA.
- Why informative error messages are difficult in Nextflow, and how we’re making things better (coming to your console in the next stable release!)
- How we’re building a Nextflow language server, for linting, formatting and code completion in your favourite IDE.
Ever wanted to peer inside Nextflow as it runs your pipeline? We play with an interactive debugger to do just that. Also, a first peek at a new Nextflow Ambassador program, plus a revamped hands-on Nextflow training course.
In this episode of the Channels podcast we have Evan Floden (Seqera Labs CEO) and Harshil Patel (head of scientific development at Seqera Labs) on the show! We talk about:
0:00 - Introduction 1:42 - History of automated testing in nf-core 4:10 - Using Tower CLI with GitHub actions in nf-core 5:09 - Coverage of full-size tests in nf-core 8:00 - Nextflow Tower "pipeline showcase" automation 14:18 - Python library for Tower automation (tw-pywrap) 18:23 - Seqera roadmap: pipeline detail view and run linking 22:06 - Nextflow Tower - Data Explorer 28:01 - Nextflow Tower - Datasets 31:55 - Upcoming events 35:15 - Wrap up
In today's episode of the Channels Podcast, we talk to Seqera software engineer Ben Sherman about what's going on "under the hood" with Nextflow. We touch on task provenance, cloud cache databases and task / sub-workflow grouping. Gotta cache 'em all!
0:00 - Introduction 2:21 - Task provenance 14:32 - Difference to the nf-prov plugin 17:13 - Quilt packages 22:30 - Cloud cache database 32:16 - Task grouping (horizontal) 45:18 - Workflow grouping (vertical) 1:00:22 - Upcoming events 1:03:31 - Close
In this News and Views episode, Chris Hakkaart and Marcel Ribeiro-Dantas talk to Rob Syme about the exciting new Nextflow training material and ways to learn Nextflow in 2023.
In this "News & Views" episode of Channels we talk to Nextflow developer Ben Sherman about some exciting new language features (hopefully) coming soon to Nextflow:
0:00 - Introduction
2:42 - Default values for process inputs
11:59 - Arity, nullable options for input and output paths
28:17 - Named arguments for processes and workflows
36:16 - Map inputs/outputs for processes
41:14 - Custom record inputs/outputs for processes
51:37 - New community content and upcoming events
Tune to to Episode 16 of the Channels podcast to hear from the nf-core community infrastructure team: Matthias Hörtenhuber and Júlia Mir Pedrol.
In a bumper episode, we discuss the new "nf-validation" Nextflow plugin (https://nextflow-io.github.io/nf-validation/) and talk nf-core components: modules and subworkflows.
In this News and Views episode, Phil Ewels talks to Ben Sherman about upcoming features in core Nextflow development: a new CLI v2 interface; submission of job arrays and intermediate file cleanup during runs.
In this "News & Views" episode of Channels we kick off the Summer 2023 series of Channels podcasts and go through some of the highlights from the latest Nextflow releases.
A special edition of Channels, Phil Ewels talks to Paolo Di Tommaso and Evan Floden about the past, present and future of Nextflow as the project celebrates its 10th anniversary.
In this "News & Views" episode of Channels we talk about improvements to handling S3 file names, support for the Spack package manager, and local hackathon sites.
In this "News & Views" episode of Channels we talk about the new Sarus container engine, staging outputs using 'rclone' and 'fcp', and using GITHUB_TOKEN to access private repositories.
In this "News & Views" episode of Channels we talk about the new Flux executor for Nextflow, the initiative to add transcripts to all nf-core bytesize videos, and some improvements for error messaging when using Nextflow.
In this "News & Views" episode of Channels we talk about how support for Nextflow DSL1 is going to be dropped and what that means for you. We also explore the wondrous world of AI and ML with a few examples of recent pipelines we've written.
The second "News & Views" episode of Channels - join us as we chat about the nf-test framework for testing Nextflow pipelines, the second round of the Nextflow & nf-core mentorship program and a new Nextflow blog post about how to figure out exactly why your pipeline won't resume!
Rob Lalonde of Seqera Labs interviews Robert Petit, a senior bioinformatics scientist at Wyoming Public Health Laboratory. In this interview, Rob and Rob discuss public health challenges unique to Wyoming, genomic surveillance, the Bactopia pipeline, and the latest fashion trends among leading bioinformaticians in the state.
The first episode of a new format - meet Channels: the nextflow podcast. We'll be joining Rob Lalonde in bringing podcast episodes to your feeds with short News & Views episodes where you can hear us discuss hot topics in the Nextflow community.
In this Podcast, Rob Lalonde of Seqera Labs speaks with Dr. Evan Floden, CEO and co-founder of Seqera Labs. Rob and Evan talk about the history of Nextflow, breaking bad, and the future of bioinformatics. Evan also provides a live demo of Nextflow Tower.
In this Podcast, Rob Lalonde of Seqera Labs speaks with Cedric Notredame, head of the Notredame Lab at the Centre for Genomic Regulation in Barcelona, Spain. In this far-ranging discussion, Rob and Cedric talk about cloud, pipelines, and trace the evolution of Nextflow from its early days as a research project to its current role in the global COVID effort.
In this Podcast, Rob Lalonde of Seqera Labs speaks with Lorena Pantano, head of Computational Biology at NextRNA, a firm leading the next revolution in RNA-directed therapeutics located in Cambridge, Massachusetts. In this fascinating discussion, Rob and Lorena talk Bioinformatics, life in a startup, and women in technology.
In this Podcast, Rob Lalonde speaks with Nils Homer, founding partner at Fulcrum Genomics. Nils shares his unique perspectives on breaking into the bioinformatics industry, advances in technology and tools, and what to expect in the next three to five years.
In this Podcast, Rob Lalonde speaks with Phil Ewels of SciLifeLab Sweden, part of NGI Stockholm. In this fascinating discussion, they cover everything from nf-core to future bioinformatics challenges.