[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’ve always wanted to have a minimal unified interface to XGBoost, CatBoost, LightGBM and sklearn's GradientBoosting, without worrying about the different parameters names aliases. So, I had a lot of fun creating unifiedbooster (which is not part of Techtonique, but is a personal swiss knife tool, under the MIT License).
In unifiedbooster, there are 5 main common parameters for each algorithm:
n_estimators: maximum number of trees that can be builtlearning_rate: shrinkage rate; used for reducing the gradient stepmax_depth: maximum tree depthrowsample: subsample ratio of the training instancescolsample: percentage of features to use at each node splitIn many situations, these are enough for obtaining robust “baselines” (and the whole documentation can be found here). Additional parameters can be provided thanks to the **kwargs (even though that’s not the main philosophy of the tool).
I present a Python version and an R version.
Python version
!pip install unifiedbooster
There are many ways to calibrate the boosters, which all rely on GPopt. I’ll present only one today (the other ones in a few weeks): Bayesian optimization.
import unifiedbooster as ubfrom sklearn.datasets import load\_iris, load\_breast\_cancer, load\_winefrom sklearn.model\_selection import train\_test\_splitfrom sklearn.linear\_model import ElasticNetCVfrom sklearn.kernel\_ridge import KernelRidgefrom sklearn.metrics import f1\_score, accuracy\_score, precision\_score, recall\_scorefrom time import timedataset = load\_breast\_cancer()X, y = dataset.data, dataset.target # data setX\_train, X\_test, y\_train, y\_test = train\_test\_split( X, y, test\_size=0.2, random\_state=42) # split data into training set and test set# Find 'good' hyperparameters for LightGBM# Obtain 'best' model's performance on test setres = ub.cross\_val\_optim(X\_train=X\_train, y\_train=y\_train, X\_test=X\_test, y\_test=y\_test, model\_type="lightgbm", # or 'lightgbm', 'gradientboosting', 'catboost' type\_fit="classification", scoring="accuracy", n\_estimators=250, cv=5, # numbers of folds in cross-validation verbose=1, seed=123)print(res) Creating initial design... ...Done. Optimization loop... 190/190 [██████████████████████████████] - 45s 237ms/stepresult(best\_params={'learning\_rate': 0.9611431739764045, 'max\_depth': 1, 'rowsample': 0.597564697265625, 'colsample': 0.508392333984375, 'model\_type': 'lightgbm', 'n\_estimators': 250}, best\_score=-0.9780219780219781, test\_accuracy=0.9736842105263158)
How do we verify what we’ve just did?
```
``` Classification report
from sklearn.metrics import classification\_reportprint(classification\_report(y\_test, y\_pred)) precision recall f1-score support 0 0.98 0.95 0.96 43 1 0.97 0.99 0.98 71 accuracy 0.97 114 macro avg 0.97 0.97 0.97 114weighted avg 0.97 0.97 0.97 114
Confusion matrix
import seaborn as snsimport matplotlib.pyplot as pltfrom sklearn.metrics import confusion\_matrixconf\_matrix = confusion\_matrix(y\_test, y\_pred)sns.heatmap(conf\_matrix, annot=True, fmt='g', xticklabels=clf.classes\_, yticklabels=clf.classes\_, )plt.ylabel('Prediction',fontsize=13)plt.xlabel('Actual',fontsize=13)plt.title('Confusion Matrix',fontsize=17)plt.show()
R versionIn the same environment as the Python environment:
utils::install.packages("reticulate")library("reticulate")unifiedbooster <- import("unifiedbooster")
Get data:
utils::install.packages("palmerpenguins")library("palmerpenguins")penguins\_ <- as.data.frame(palmerpenguins::penguins)# replacing NA's by the medianreplacement <- median(palmerpenguins::penguins$bill\_length\_mm, na.rm = TRUE)penguins\_$bill\_length\_mm[is.na(palmerpenguins::penguins$bill\_length\_mm)] <- replacementreplacement <- median(palmerpenguins::penguins$bill\_depth\_mm, na.rm = TRUE)penguins\_$bill\_depth\_mm[is.na(palmerpenguins::penguins$bill\_depth\_mm)] <- replacementreplacement <- median(palmerpenguins::penguins$flipper\_length\_mm, na.rm = TRUE)penguins\_$flipper\_length\_mm[is.na(palmerpenguins::penguins$flipper\_length\_mm)] <- replacementreplacement <- median(palmerpenguins::penguins$body\_mass\_g, na.rm = TRUE)penguins\_$body\_mass\_g[is.na(palmerpenguins::penguins$body\_mass\_g)] <- replacement# replacing NA's by the most frequent occurencepenguins\_$sex[is.na(palmerpenguins::penguins$sex)] <- "male" # most frequent# one-hot encodingpenguins\_mat <- model.matrix(species ~., data=penguins\_)[,-1]penguins\_mat <- cbind(penguins$species, penguins\_mat)penguins\_mat <- as.data.frame(penguins\_mat)colnames(penguins\_mat)[1] <- "species"y <- as.integer(penguins\_mat$species) - 1LX <- as.matrix(penguins\_mat[,2:ncol(penguins\_mat)])n <- nrow(X)p <- ncol(X)set.seed(123)index\_train <- sample(1:n, size=floor(0.8*n))X\_train <- X[index\_train, c("islandDream", "islandTorgersen", "flipper\_length\_mm")]y\_train <- y[index\_train]X\_test <- X[-index\_train, c("islandDream", "islandTorgersen", "flipper\_length\_mm") ]y\_test <- y[-index\_train]
Find hyperparameters:
res <- unifiedbooster$cross\_val\_optim(X\_train=X\_train, y\_train=y\_train, X\_test=X\_test, y\_test=y\_test, model\_type="xgboost", type\_fit="classification", scoring="accuracy", n\_estimators=100L, cv=5L, # numbers of folds in cross-validation verbose=1L, seed=123L)print(res)
check
```
``` To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Auto XGBoost, Auto LighGBM, Auto CatBoost, Auto GradientBoosting