The AutoML Podcast: Recent Episodes
AutoML Media
There are so many great foundation models in many different domains - but how do you choose one for your specific problem? And how can you best finetune it? Sebastian Pineda has an answer: Quicktune can help select the best model and tune it for specific use cases. Listen to find out when this will be a Huggingface feature and if hyperparameter optimization is even important in finetuning models (spoiler: very much so)!
Designing algorithms by hand is hard, so Chris Lu and Matthew Jackson talk about how to meta-learn them for reinforcement learning. Many of the concepts in this episode are interesting to meta-learning approaches as a whole, though: "how expressive can we be and still perform well?", "how can we get the necessary data to generalize?" and "how do we make the resulting algorithm easy to apply in practice?" are problems that come up for any learning-based approach to AutoML and some of the topics we dive into.
AutoML can be a tool for good, but there are pitfalls along the way. Rahul Sharma and David Selby tell us about how AutoML systems can be used to give us false impressions about explainability metrics of ML systems - maliciously, but also on accident. While this episode isn't talking about a new exciting AutoML method, it can tell us a lot about what can go wrong in applying AutoML and what we should think about when we build tools for ML novices to use.
In today's episode, we're introducing the very special Theresa Eimer to the show.
Theresa will be taking over the hosting of many of the future episodes. Theresa has already recorded multiple episodes and we are stoked to air those shortly.
We also spend a few moments explaining my relative absence in the last few months (since the war in the middle east erupted) and what I'm up to now.
Theresa, we are all so excited to be doing this together!
To learn more about Theresa,
Follow her on Twitter here: https://twitter.com/The_Eimer
Connect with her on LinkedIn here: https://www.linkedin.com/in/theresa-eimer-a724b5b0/
As you'll hear in the episode, she's also one of the co-organizers of COSEAL, which you can learn more about here: https://www.coseal.net/
Today we're talking with Nick Erickson from AutoGluon.
We discuss AutoGluon's fascinating origin story, its unique point of view, the science and engineering behind some of its unique contributions, Amazon's Machine Learning University, AutoGluon's multi-layer stack ensembler in all its detail, their feature preprocessing pipeline, their feature type inference, their adaptive approach to early stopping, controlling for inference speeds, the different multi-modal architectures, the ML culture at Amazon, the unique challenges of time series, the role of competitions, the decision to reject hyperparameter optimization, benchmarking in AutoML, what the research community can do to help industry along, AutoGluon's relationship with pre-trained tabular models like Tab-PFN, whether the rise of LLMs is likely to affect AutoGluon, what's stopping more people from adopting AutoML solutions, AutoGluon Cloud, the dream and reality of an auto-benchmarking tool, how to contribute to their project, and many, many other topics.
This was one of my favorite episodes. Nick, thank you for joining!
You can follow Nick on Twitter here: @innixma.
And you can follow AutoGluon on GitHub here: https://github.com/autogluon.
Some more resources on AutoGluon:
- The original AutoGluon Paper: "AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data": https://arxiv.org/abs/2003.06505
- AutoML Fall School 2022 AutoGluon presentation, a good way to understand the philosophy behind AutoGluon: https://www.youtube.com/watch?v=VAAITEds-28
- AutoGluon multi-modal paper: https://dl.acm.org/doi/abs/10.1145/3534678.3542616
Today we're talking with Joseph Giovanelli about his work on integrating logic and argumentation into AutoML systems.
Joseph is a PhD student at the University of Bologna. He was more recently in Hannover working on ethics and fairness with Marius’ team.
The paper he published presents his framework, HAMLET, which stands for Human-centric AutoML via Logic and Argumentation. It allows a user to iteratively specify constraints in a formal manner and, once defined, those constraints become logical premises. Those premises, when combined together, can produce conflicts with one another, thereby reducing the search space and providing deeper intuition back to the user.
To learn more about HAMLET, see the paper here: https://ceur-ws.org/Vol-3135/dataplat_short2.pdf and the repo here: https://github.com/QueueInc/HAMLET
To follow Joseph on LinkedIn, see his profile here: https://www.linkedin.com/in/joseph-giovanelli/
Today we're talking with Caitlin Owen, a post-doc at the University of Otago about her work on error decomposition.
She recently published a paper titled "Towards Explainable AutoML Using Error Decomposition" about how a more granular view of the components of error can lead the construction of better AutoML systems.
Read her paper here: https://link.springer.com/chapter/10.1007/978-3-031-22695-3_13
Follow her on Twitter here: @CaitAshfordOwen
Connect with her on LinkedIn here: https://www.linkedin.com/in/caitlin-owen-5b9b08193/
Today we're talking with Gaurav Rao, the EVP & GM of Machine Learning and AI at AtScale, a company centered around the semantic layer.
For some time now, I've been feeling that there is a deep connection between a formal articulation of business context and the realization of the dream of AutoML, so I searched for people in the space who can help shine light on this direction.
Gaurav is one of the few who can speak about this. As you'll hear, he's extremely pedagogic and he's walking us through the origins of the concept, how it addresses some of the challenges that businesses face when trying to operationalize their ML, what it takes to build a universal semantic layer, how downstream ML applications are affected by the presence or absence of a semantic layer, and how the space of AutoML factors into this.
Connect his Gaurav and learn more about the semantic layer through his LinkedIn: https://www.linkedin.com/in/gauravraotechenthusiast/
Today Ankush Garg is speaking with Rishi Bommasani, PhD student at Stanford and one of the originator of the term Foundation Models.
They’re talking about the origins of the term Foundation Model, which he and his group advanced, in the paper "On the Opportunities and Risks of Foundation Models".
They’ll talk about self-supervision, issues of scale, the motivation behind the terminology, the origins of the Research for Foundation Models Institute at Stanford, outcome homogenization, emergence and phase transitions, and some of the social consequences to look out for.
Thank you both for this conversation. As the world is coming to terms with GPT-4, this will be increasingly relevant.
Paper: On the Opportunities and Risks of Foundation Models.
Rishi's twitter: @RishiBommasani
Center for Research on Foundation Models (CRFM)
Today we're talking with Raymond Peck, a senior engineer and director in the AutoML space. He spent time at H2O, dotData, Alteryx and many other places.
This is a fascinating conversation about the business, engineering, and science of machine learning automation in production. Learning about his experience is crucial for understanding the biography of the space.
We discuss the early motivations behind AutoML, the initial value propositions that propelled the first movers in the market, the market dynamics that operated in the early days, the evolution of the relevant engineering and science, how customers evaluate AutoML tools, the role of feature engineering and relational tables, the crucial role that explainability plays in AutoML, and many more topics.
Raymond is a prolific writer on LinkedIn. You should follow him here: https://www.linkedin.com/in/raymondpeck/.
Today we’re talking to Noah Hollmann and Samuel Muller about their paper on TabPFN - which is an incredible spin on AutoML based on Bayesian inference and transformers.
[Quick note on audio quality]: Some of the tracks have not recorded perfectly but I felt that the content there was too important not to release. Sorry for any ear-strain!
In the episode, we spend some time discussing posterior predictive probabilities before discussing how exactly they’ve pre-fitted their network, how they got their training data, what the network looks like, and how the system is performing.
To give you a taste of it, on datasets up to 1,000 training instances and 100 features, it takes less than a second to train and predict a classifier!
Read their paper here: https://arxiv.org/pdf/2207.01848.pdf
Follow Samuel on Twitter, here: https://twitter.com/SamuelMullr
Follow Noah on Twitter, here: https://twitter.com/noahholl
Today we’re talking to Sean Sexton, the Director of Modeling and Analytics Consulting at KPMG, about the role of models in financial institutions and how the risks associated with them is managed.
This turned out to be an incredibly deep and interesting topic, and we really only scratched the surface of it.
Sean has a unique ability to summarize developments in an entire space. If you're interested to learn more about modeling in financial institutions and about the history of how we got here, you should definitely study his dissertation, on managing model risk, here: https://macsphere.mcmaster.ca/handle/11375/28049
Today we’re talking to Matt Levine. Matt is a PhD student in computing and mathematical sciences at Caltech, and he focuses on improving the prediction and inference of physical systems by blending together both mechanistic modeling and machine learning.
This episode is one of my favorites: we go pretty deep into dynamical systems, and into Matt's new framework for solving them by blending traditional, mechanistic, approaches with machine learning. This is a fascinating use of machine learning, and hopefully gets us one step closer to the automation of science, in general.
A Framework for Machine Learning of Model Error in Dynamical Systems - https://arxiv.org/abs/2107.06658
Related works
Autodifferentiable Ensemble Kalman Filters - https://epubs.siam.org/doi/abs/10.1137/21M1434477
Universal Differential Equations for Scientific Machine Learning - https://arxiv.org/abs/2001.04385
Continuous-time nonlinear signal processing: a neural network based approach for gray box identification - https://ieeexplore.ieee.org/document/366006
A generalised approach to process state estimation using hybrid artificial neural network/mechanistic models - https://www.sciencedirect.com/science/article/abs/pii/S0098135496003365
Today we’re chatting with Junhong Shen, a PhD student at Carnegie Mellon.
Junhong and her team are working on the generalizability of NAS algorithms across a diverse set of tasks.
Today we'll be talking about DASH, a NAS algorithm that takes diversity of tasks at its center. In order to implement DASH, Junhong and her team implemented three clever ideas that she'll share with us.
Efficient Architecture Search for Diverse Tasks - https://arxiv.org/pdf/2204.07554.pdf
Tackling Diverse Tasks with Neural Architecture Search - https://blog.ml.cmu.edu/2022/10/14/tackling-diverse-tasks-with-neural-architecture-search/
Does AutoML work for diverse tasks? - https://blog.ml.cmu.edu/2022/07/07/automl-for-diverse-tasks
Follow Junhong @JunhongShen1
Today we're speaking with Marius Lindauer and it is certainly one of my favorite episodes!
As you’ll hear, Marius is full of ideas for where AutoML systems can and should go. These ideas are crystallized in a blog-post, published here: https://www.automl.org/rethinking-automl-advancing-from-a-machine-centered-to-human-centered-paradigm/
If you’re searching for research directions, this conversation left me with dozens of ideas. Marius and his team are doing phenomenal work to make AutoML systems more trustworthy and more human-centric.
We will be reviewing content from the following papers:
- Bayesian Optimization with a Prior for the Optimum - https://arxiv.org/abs/2006.14608
- Explaining Hyperparameter Optimization via Partial Dependence Plots - https://arxiv.org/abs/2111.04820
- Enhancing Explainability of Hyperparameter Optimization via Bayesian Algorithm Execution - https://arxiv.org/abs/2206.05447
- πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization - https://arxiv.org/abs/2204.11051
Follow Marius on Twitter here - https://twitter.com/LindauerMarius, and AutoML.org on Twitter here - https://twitter.com/AutoML_org.
To help Marius complete the survey he mentioned, please visit the link here - https://www.soscisurvey.de/hpo-method-validation/.
To learn more about AutoML, visit AutoML.org, here - https://www.automl.org/
Today Ankush Garg is talking to Mehdi Bahrami about his recent project: BERT-Sort.
BERT-Sort is an example of how large language models can add useful context to tabular datasets, and to AutoML systems.
Mehdi is a Research Staffer at Fujitsu and, as he describes, he began using AutoML systems for his research, yet he came across some crucial limitations of existing solutions. The modifications he made highlight a promising future for the relationship between language models and AutoML. This is a direction we're going to continue to explore on the show.
References:
BERT-Sort: A Zero-shot MLM Semantic Encoder on Ordinal Features for AutoML - https://proceedings.mlr.press/v188/bahrami22a.html
PyTorrent: A Python Library Corpus for Large-scale Language Models: https://arxiv.org/abs/2110.01710
AugmentedCode: Examining the Effects of Natural Language Resources in Code Retrieval Models: https://arxiv.org/abs/2110.08512
In today's episode, we’re talking to Lars Kothoff about the fascinating origin story of AutoML (as he sees it), and how it emerged from the SAT community.
While talking to many of you, it became clear that this origin story is one that a lot of people have some vague sense about, but not a very concrete knowledge of so hopefully this episode can help to flesh out the narrative with greater clarity.
We'll be discussing his survey paper from 2016, "Algorithm selection for combinatorial search problems: A survey", to be found here:
https://scholar.google.com/citations?view_op=view_citation&hl=en&user=P37OkUUAAAAJ&citation_for_view=P37OkUUAAAAJ:W7OEmFMy1HYC
As well as the original SAT-zilla paper, to be found here:
https://www.aaai.org/Papers/JAIR/Vol32/JAIR-3214.pdf
Follow Lars' work on Twitter, @larskotthoff.
Today we’re talking to Cedric Kulbach about online learning, the challenges of doing it properly, why it is so promising, how it’s connected to evolutionary strategies, and recent advances in the field that can help to unlock these promises.
We then discuss the close connection between online learning and AutoML systems, and we explore a recent framework that he recently published, called EvoAutoML.
Cedric is a PhD student at Karlsruhe Institute of Technology, in Germany, where was defending his thesis 2 days after we spoke!
You can find his paper here: https://hal.inria.fr/hal-03667231/document
This short episode is a narration of Richard Sutton's The Bitter Lesson.
Richard Sutton is a distinguished research scientist at DeepMind and a professor of computing science at the University of Alberta. He is considered one of the founders of modern computational reinforcement learning, having several significant contributions to the field, including temporal difference learning and policy gradient methods.
If you haven't yet read his piece, it is definitely worth reading. This episode is a narration of it.
I would like to invite more discussion about these thoughts, so if you know of anybody who would like to discuss them, please send me to me at hello@automlpodcast.com.
You can read the article here: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
More drama in the contest between traditional machine learning models and deep learning models when it comes to tabular data.
We have on the show Vadim Borisov, a research fellow at the University of Tubingen as well as Kathrin Sessler, a PhD student from the same university.
This episode will be led by Ankush Garg, exploring Vadim and Kathrin's recent paper “Deep Neural Networks and Tabular Data: A Survey”.
Paper: https://arxiv.org/pdf/2110.01889.pdf
SAINT paper: https://arxiv.org/abs/2106.01342
Deep learning architectures and the hippocampus paper: https://arxiv.org/abs/2112.04035
We will also do a brief form of house keeping about the show and some of the upcoming episodes.
As always, if you have ideas, questions, feedback, or would like to be interviewed on the show: please reach out at hello@automlpodcast.com.
Today we’re speaking with Jeff Clune about a new path towards general artificial intelligence, that he calls AI Generating Algorithms (AI-GAs).
Jeff is a Professor of Computer Science at the University of British Columbia. He was previously a Senior Research Scientist at Uber AI, and more recently a Research Team Leader at OpenAI. He's currently a Faculty Member at the Vector Institute.
We’re going to be talking about whether it is even possible that humans never create AGI, the enormous potential upside of AGI, the risks of AGI (including existential risks), Jeff’s fascinating transition into the field of AI, whether academia remains receptive to similar academic transitions, how long it might take to get to AGI and whether we have the right playbook for it, the prevailing status quo track (he mentions a couple): The first being a large language model approach, that he calls “standing on the shoulders of giant human datasets”, some of the limitations of this approach, the main track, that he calls “the manual path to AGI”, his new contribution, the AI Generating algorithms, which is essentially how we get out of the way and let the AI do much more, whether we already have an existence proof for how this can work, the three pillars of the AI Generating approach, AI Generating environments and capability oriented evaluation, the production of interesting or useful behaviors, the hidden Turing award to the person who would figure out AI Generating environment, learning pathologies, the circuitous paths of innovation, the seeming miracle of Darwinian evolution and the challenges of replicating it, the increasingly complex environments Darwinian evolution produces, the moral dimension to toying around with increasingly intelligent agents, AI and vegetarianism, and the extent to which we should be dismissive of the subjective claims of increasingly intelligent models.
His paper can be found here: https://arxiv.org/abs/1905.10985
Follow him on Twitter here: https://twitter.com/jeffclune
Learn more on his website here: http://jeffclune.com/
Today I'm speaking with Jan N. van Rijn about metalearning.
Jan is an assistant professor at Leiden University, where he also did his PhD. He is one of the founders of the OpenML Foundation, he previously did a post-doc at Freiburg in the Frank Hutter lab, and he is one of the authors of the metalearning book, which we'll be discussing.
We’ll be primarily examining the contents of their chapter titled “Evaluating Recommendations of Metalearning/AutoML Systems”.
We'll be covering such topics as OpenML and metalearning, how the space of AutoML has changed in the last few years, benchmarks in the space of AutoML, how benchmarks measure progress of a field, some of the challenges benchmarks present, the need to build better tooling around benchmarks, the rise of NAS within the AutoML umbrella, the scope of AutoML (algorithm selection, hyperparameter optimization, the CASH problem, pipeline optimization, etc.), how meta learning helps traverse through the various scopes of AutoML problem types, metalearning in the context of hyperparameter optimizations, the importance of properly designing meta-datasets, approaches to the inputs and outputs of meta-models and their advantages and disadvantages, surrogate models, how AutoML systems interact with meta-models, how to think about metalearning across dataset difficulties, diagnosing meta-models and meta-datasets, how to compare different metadata systems, loss-time curves, meta-features and the various approaches to creating them (including dataset 2 vec), contemporary meta-learning in a deep-learning context, and other topics.
Link to book - https://link.springer.com/book/10.1007/978-3-030-67024-5
Link to OpenML - https://www.openml.org/
Today we’re speaking with three researchers: Karan Grewal, Abhi Iyer and Akash Velu, about multi-task learning and how their new brain-inspired approach can help tackle it.
We’ll be discussing what a task is, what exactly we mean by multi-task systems, distances between tasks, the difference between continual learning and multi-task learning, catastrophic forgetting, catastrophic interference and their causes, various approaches out there like context-dependent gating and synaptic intelligence, the role of scale, and sparsity, we’ll cover some basics of the brain like dendrites, proximal and distal dendrites, apical and basal dendrites, how active dendrites can help us solve the challenges of gradient interference in multi-task learning, how they attach dendrites to each of their neurons in their deep learning models, their various approaches to representing context vectors, the challenges of brain-inspired approaches to machine learning, and some speculation about the future of this line of research.
Some references:
Their paper can be found here: https://arxiv.org/abs/2201.00042
All three of them had a great appearance on the Yannic Kilcher YouTube channel here: https://youtu.be/smxwT82o40Y
They mentioned Hierarchical Temporal Memory as the foundation stone for a lot of their research: https://numenta.com/blog/2019/10/24/machine-learning-guide-to-htm
Today I’m speaking with Mayukh Das about using neural architecture search for resource-constraint devices and about a new multi-objective reinforcement-learning based framework that he recently published called AUTOCOMET.
We’ll be covering such topics as how NAS research is done both at Samsung and at Microsoft, the relationship between NAS and product teams, devices and the various types of constraints they expose, how to featurize hardware contexts, layer-wise latency calculations, surrogate models and the kinds of hardware-aware data they require, the current limitations of NAS, reinforcement learning and NAS, multi-objective optimization in the context of reinforcement learning, reward sparsity, reward shaping and shaping functions, primary and secondary rewards, their concept of co-regulated shaping, Q functions and the effects of potentials, AUTOCOMET, the future of NAS and other topics.
Thank you for tuning in!
To learn more about AUTOCOMET, find the paper here: https://arxiv.org/pdf/2203.15408.pdf
Today we’re speaking with José Hernández-Orallo. José is a Professor at the Polytechnic University of València in Spain and a Senior Research Fellow at the Leverhulme Centre for the Future of Intelligence, at Cambridge.
We'll be covering an enormous amount of ground surrounding intelligence and its evaluation.
We’ll touch on topics such as operating conditions in ML, agent characteristic curves,, the challenge with average performance scores, task-oriented evaluation, capability-oriented evaluation - in humans and other animals, animal-AI Olympics, meta-data annotation, performance robustness as a function of latent attributes, the limitations of aggregating behavioral performance into a handful of metrics, whether intelligence is meaningless or all-explanatory, the concept of generality in intelligence, AGI and the problem of distributions of tasks, calibration of task difficulties, whether capability metrics are incommensurable, complexity and compressibility, compression-based algorithmic information theory, the limits of intelligence, the intelligence of hybrid systems, whether we’re ready to talk about consciousness given our limited understanding of intelligence, whether rights should be tethered to pain or to cognition, animal rights, and what the AutoML community can do to help with this line of work.
If you're interested in this line of work, here is a recent paper by José and his team: https://ryanburnell.com/wp-content/uploads/Burnell-et-al-2022-Not-a-Number.pdf
His book, The Measure of All Minds, can be found here: https://www.amazon.com/Measure-All-Minds-Evaluating-Intelligence/dp/1107153018
Today I’m talking with Wout Schellaert about assessor models. Wout is a PhD student at the Polytechnic University of Valencia..
We’ll be covering a lot of different topics, such as the distributional hypothesis in machine learning, evaluation criteria, the reductive nature of current evaluation methods, task systems, the desiderata of assessor models, how to build assessor models, when to use them, what happens when our models become increasingly complex, how assessor models can help with AI explainability, whether they can help against adversarial scenarios, the challenges of scoring, bias in human evaluators, how assessor models relate to AI alignment, and other topics.
If you'd like to learn more about assessor models, visit the following pages.
José Hernández-Orallo (Wout's advisor): https://scholar.google.com/citations?user=n9AWbcAAAAAJ&hl=en
Training on the Test Set: Mapping the System-Problem Space in AI paper: https://www.aaai.org/AAAI22Papers/SMT-00432-Hernandez-OralloJ.pdf
Today we’re talking to Karthi Ramamurthy about a novel approach to similarity learning explainability.
Karthi is a research staff member in IBM Research at the Watson Research Center.
He studies the relationship between humans, machines, data and the societal implications of machine learning.
He was involved in the initial development of the open source AI Fairness 360 toolkit, where he’s still an active contributor.
His papers won various best paper awards like the 2015 IEEE International Conference on Data Science and Advanced Analytics.
He is an associate editor of Digital Signal Processing and a member of the IEEE and he holds a PhD in electrical engineering from Arizona State University.
We will be discussing a recent paper that he published about explainability methods for similarity learners. So we’ll go into detail about what similarity learning, or metric learning, is.
You can find the paper here: https://arxiv.org/pdf/2202.01153v1.pdf
To learn more about metric learning, please see this survey paper: https://people.bu.edu/bkulis/pubs/ftml_metric_learning.pdf.
Today I’m speaking with Vasco Lopes, about the state of Neural Architecture Search, NAS, and about a new method that he published that takes a very creative look at how to do NAS.
We’ll be discussing the motivation behind NAS, the current state of its deployment, the biggest use-cases today, the three components that make up NAS, the drawbacks to the current NAS paradigm, search spaces and how to design them, search strategies and how to choose them, graph representations of neural architectures, evaluation strategies and zero-cost approximations, bias in search space design, risks, the future of hand-designed architectures, and other topics.
Vasco is a PhD student at NOVA School of Science and Technology in Portugal and a co-founder of a Computer Vision startup called DeepNeuronic.
Find his paper, Towards Less Constrained Macro-Neural Architecture Search, here: https://arxiv.org/pdf/2203.05508.pdf
Today I’m speaking with Yury Gorishniy about the state of the competition between Deep Learning and Gradient Boosted Decision Trees when it comes to tabular datasets, and about a recent paper he published that seems to take a stab at improving the state of deep learning on tabular datasets.
We discuss whether or not there exists a gap between deep learning and gradient boosted decision trees, what the future of a gap might look like, and the extent to which the embedding of numerical features can give deep learning architectures a necessary boost in performance.
Two of his recent papers are useful in this discussion:
- On Embeddings for Numerical Features in Tabular Deep Learning - https://arxiv.org/abs/2203.05556
- Revisiting Deep Learning Models for Tabular Data - https://arxiv.org/abs/2106.11959
You can find Yury in the following places:
- GitHub - https://github.com/Yura52
- Twitter - https://twitter.com/YuraFiftyTwo
Enjoy!
Today we’ll be speaking with Julian Morimoto about missing data, its impact on the reliability of statistical inference, and two theorems that he recently discovered using concepts from real analysis about what guarantees we can expect, at the limit of arbitrarily large data sets.
Julian has a background in math, and studied law at Harvard Law School and he speaks about the unique challenges of adopting machine learning in the legal world due to the various mechanisms of missingness in confidential documents.
If you're interested in learning more about missing data and statistical inference, here are several resources to help you get started:
- GOV2001 - Harvard: https://projects.iq.harvard.edu/gov2001/home
- One hour course by Dr. Gary Keng on missing data on YouTube: https://www.youtube.com/watch?v=qlPs8Ioa56Y
- Julian’s paper: https://arxiv.org/abs/2112.09275v4
- Statistical Analysis with Missing Data book: https://www.amazon.com/Statistical-Analysis-Missing-Roderick-Little/dp/0471183865?
In this episode, Adam introduces the show, the motivations for it, and why and how you should participate.
Today we're speaking with Luigi Quaranta about the state of reproducibility in machine learning.
Luigi published a taxonomy of support for reproducibility by various tools in the space and together we’re exploring the need for reproducibility, challenges and limitations, how to evaluate opportunities for improving your current systems, and what the future might hold.
A few papers would be relevant here.
The first is A Taxonomy of Tools for Reproducible Machine
Learning Experiments: http://ceur-ws.org/Vol-3078/paper-81.pdf
and the second is a study of how to make Jupyter notebook code of high quality, Eliciting Best Practices for Collaboration with Computational Notebooks: https://dl.acm.org/doi/abs/10.1145/3512934.
In this episode, Adam speaks with Doctor Torsten Ensslin about simulating reputation networks and their manipulation using Information Theory.
Torsten is an Astrophysicist and cosmologist at the Max Plank institute, where he’s held many titles and positions. His current scientific work investigates theoretical cosmology and information field theory.
As he discusses in this episode, Torsten co-created a simulation of how reputation propagates through a network of agents and how those agents can manipulate their communication strategies, through lies, targeted deceit, propaganda, and many other tactics, in an effort to achieve domination over their networks.
One of the theses of Torsten’s work is that in order to understand the impact of our AI on our existing systems, it’s important to have a deeper understanding of those systems to begin with. His simulation is a first step in that direction: an attempt to model the interaction of various agents according to information theoretic principles and study the consequent emergent phenomena.
There are two versions of the paper.
The first is a shorter summary of the work, light on the math, and can be found here: https://psyarxiv.com/wqcmb/.
The longer version of the paper can be found here: https://arxiv.org/pdf/2106.05414.pdf.
Enjoy!
In this episode, Adam discusses Multi-objective optimization with Laurent Parmentier.
Laurent works at OVHCloud, most recently as a data scientist but previously in various software engineering roles. He published his thesis on AutoML at OVHCloud, and had previously released a paper titled TPOT-SH: A Faster Optimization Algorithm to Solve the AutoML Problem on Large Datasets.
The conversation centers around two classic papers in AutoML:
- Multi-Objective Automatic Machine Learning with AutoxgboostMC by Pfisterer et al: https://arxiv.org/abs/1908.10796
- An ADMM Based Framework for AutoML Pipeline Configuration by Sijia Liu, Parikshit Yam et al: https://arxiv.org/abs/1905.00424
Enjoy!
This episode launches us into the deep waters of ML interpretability with Jessica Schrouff.
Jessica is a Senior Research Scientist at Google Research working on machine learning for healthcare. Before joining Google in 2019, she was a postdoctoral fellow at University College London (UK) and Stanford University (USA), developing machine learning techniques for neuroscience discovery and clinical predictions.
Throughout her career, her interests have lied not only in the technical advancement of machine learning methods, but also in critical aspects of their deployment such as their credibility, fairness, robustness or interpretability.
As you’ll hear, we are orbiting Jessica’s latest paper titled “Best of both worlds: local and global explanations with human-understandable concepts”. You will find lots more reference material in the show notes on the website.
Jessica, thank you for joining us!
This is a conversation between data scientist Ankush Garg, from Telepath, and fourth-year Ph.D. student Iman Mirzadeh and they’ll be talking about Continual Learning and about Iman’s paper titled “Architecture Matters in Continual Learning”.
Iman is interested in Artificial General Intelligence and so he’s researching systems that can over time develop increasingly more complex skills and a richer body of knowledge.
Iman received his Bachelor’s degree in Computer Engineering from the University of Tehran and his Master’s degree from Washington State University and that’s where he’s currently a research assistant.
Today we’re talking about MLOps - with our guide Georgios Symeonidis and we’ll be orienting around a recent paper he published titled “MLOps - Definitions, Tools and Challenges”.
Georgios studied electrical and computer engineering at Democritus University of Thrace at Xanthi, in Greece. He specialized in information and electronics.
He’s also worked as a research engineer at Athena Research and Innovation Center - studying ML systems in production. For his PhD, he’s researching Reliable Machine Learning pipelines for mature MLOps systems at International Hellenic University in Greece.
This episode covers the relationship between Curriculum Learning and AutoML with Lucas Nildaimon dos Santos Silva.
Lucas is a data scientist at americanas s.a., in Brazil, and is currently pursuing a Ph.D. in computer science from the Federal University of São Carlos where he also did his master’s. He’s researching machine learning and NLP and has recently published a paper on Curriculum Learning in AutoML systems, found below.
https://ieeexplore.ieee.org/document/9680164
In this episode, we explore the relationship between Machine Learning and Statistical Mechanics with the guidance of Alia Abbara. This conversation centers around her PhD dissertation and we cover such topics as the relationship between statistics and physics, the long legacy of physics on machine learning, and the role of physical intuition in the future of machine learning.
Find her original paper here: https://tel.archives-ouvertes.fr/tel-03497407/document