[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionYesterday I posted on performing a benchmark on reading in a compressed .csv.gz file of a 2,000 by 2,000 data.frame. It was brought to my attention by someone on Mastadon (@mariviere@fediscience.org – https://fediscience.org/@mariviere) that I should also use {duckdb} and {arrow} so I will perform the same analysis as yesterday but I will also add in the two aforementioned packages.
FunctionThe functions that we will be using this time around are as follows:
fread() – data.tableduckdb_read_csv() – duckdbread_csv() – readrread.csv() – baseread.table() – basevroom() with altrep = FALSE – vroomvroom() with altrep = TRUE – vroomoprn_csv_dataset() – arrowExampleMake the DataLet’s make that dataset again:
library(R.utils)# create a 1000 x 1000 matrix of random numbersdf <- matrix(rnorm(2000000), nrow = 2000, ncol = 2000) |> as.data.frame()# Make and save gzipped filewrite.csv(df, "df.csv")gzip( filename = "df.csv", destname = "df.csv.gz", overwrite = FALSE, remove = TRUE)
BenchmarkingTime to benchmark
library(rbenchmark)library(data.table)library(readr)library(duckdb)library(arrow)library(vroom)library(dplyr)library(DBI)n <- 30benchmark( # Base R "read.table" = { a <- read.table( "df.csv.gz", sep = ",", colClasses = list(numeric = 1:2000) ) }, "read.csv" = { b <- read.csv( "df.csv.gz", sep = ",", colClasses = list(numeric = 1:2000) ) }, # data.table "fread" = { c <- fread( "df.csv.gz", sep = ",", colClasses = list(numeric = 1:2000) ) }, # vroom "vroom alltrep false" = { d <- vroom("df.csv.gz", delim = ",", col\_types = "d") }, "vroom alltrep true" = { e <- vroom("df.csv.gz", delim = ",", altrep = TRUE, col\_types = "d") }, # readr "readr" = { f <- read\_csv("df.csv.gz", col\_types = "d") }, # Arrow "arrow" = { g <- open\_csv\_dataset("df.csv.gz") }, # DuckDB "duckdb" = { con <- dbConnect(duckdb()) h <- duckdb\_read\_csv( conn = con, name = "df", files = "C:\\Users\\ssanders\\Documents\\GitHub\\steveondata\\posts\\rtip-2023-03-28\\df.csv.gz" ) dbDisconnect(con) }, # Replications replications = n, # Columns columns = c( "test","replications","elapsed","relative","user.self","sys.self")) |> arrange(relative)
test replications elapsed relative user.self sys.self1 arrow 30 3.01 1.000 5.04 0.252 fread 30 28.28 9.395 19.56 4.303 vroom alltrep false 30 31.89 10.595 26.25 10.754 vroom alltrep true 30 33.72 11.203 25.75 10.675 duckdb 30 94.09 31.259 90.70 2.776 readr 30 98.28 32.651 113.05 45.127 read.table 30 109.97 36.535 107.78 1.248 read.csv 30 153.79 51.093 152.44 0.56
Important note is the session info on the pc I am using to write this:
sessionInfo()
R version 4.2.3 (2023-03-15 ucrt)Platform: x86\_64-w64-mingw32/x64 (64-bit)Running under: Windows 10 x64 (build 19045)Matrix products: defaultlocale:[1] LC\_COLLATE=English\_United States.utf8 [2] LC\_CTYPE=English\_United States.utf8 [3] LC\_MONETARY=English\_United States.utf8[4] LC\_NUMERIC=C [5] LC\_TIME=English\_United States.utf8 attached base packages:[1] stats graphics grDevices utils datasets methods base other attached packages: [1] dplyr\_1.1.1 vroom\_1.6.1 arrow\_11.0.0.3 duckdb\_0.7.1-1 [5] DBI\_1.1.3 readr\_2.1.4 data.table\_1.14.8 rbenchmark\_1.0.0 [9] R.utils\_2.12.2 R.oo\_1.25.0 R.methodsS3\_1.8.2loaded via a namespace (and not attached): [1] pillar\_1.9.0 compiler\_4.2.3 tools\_4.2.3 digest\_0.6.31 [5] bit\_4.0.5 jsonlite\_1.8.4 evaluate\_0.20 lifecycle\_1.0.3 [9] tibble\_3.2.1 pkgconfig\_2.0.3 rlang\_1.1.0 cli\_3.6.1 [13] rstudioapi\_0.14 parallel\_4.2.3 yaml\_2.3.7 xfun\_0.38 [17] fastmap\_1.1.1 knitr\_1.42 generics\_0.1.3 vctrs\_0.6.1 [21] htmlwidgets\_1.6.2 hms\_1.1.3 bit64\_4.0.5 tidyselect\_1.2.0 [25] glue\_1.6.2 R6\_2.5.1 fansi\_1.0.4 rmarkdown\_2.21 [29] tzdb\_0.3.0 purrr\_1.0.1 magrittr\_2.0.3 htmltools\_0.5.5 [33] assertthat\_0.2.1 utf8\_1.2.3 crayon\_1.5.2
Sys.info() |> as.data.frame() |> tibble::rownames\_to\_column() |> as\_tibble() |> slice(1,2,3,5)
```
Sys.info() ```
memory.profile() |> as.data.frame()
memory.profile()NULL 1symbol 24303pairlist 642504closure 11189environment 4009promise 22963language 189766special 47builtin 701char 2039073logical 18866integer 108132double 20060complex 5character 160381... 21any 0list 58500expression 5bytecode 41555externalptr 12382weakref 13860raw 10113S4 1362
gc()
used (Mb) gc trigger (Mb) max used (Mb)Ncells 3363479 179.7 5830931 311.5 5830931 311.5Vcells 32950395 251.4 81254422 620.0 81254324 620.0
To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How fast does a compressed file in Part 2