[This article was first published on Data Analytics and AI Archives - Giles, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.TLDR: This is the third post in a series examining relationships between the rule of law and other institutions as measured by the World Bank’s Worldwide Governance Indicators (WGI). Earlier posts demonstrated that the World Justice Project’s (WJP) measure can be proxied by the WGI Rule of Law (RoL) index, to take advantage of its wider coverage over a longer time-period.

This post builds on that analysis by asking how institutions should be conceptualized and analyzed: can the rule of law (and other institutions) sensibly be examined one at a time, or do they need to be treated as parts of an interdependent system? Using the six dimensions of the WGI as a proxy for institutional strength, results suggest dimensions are indeed bound together. Countries that score well on the rule of law also frequently score well on government effectiveness and corruption controls, year-to-year movements are positively linked across most dimensions and a common measure (or cause) accounts for the bulk of the variation between countries.

None of this will surprise anyone familiar with the literature and the analysis isn’t intended to prove institutions matter or that the WGI measure them. The point is narrower: to show why the rule of law can’t be examined in isolation and the risks of coming to the wrong conclusions when we try. It also sets up later posts in the series which will look at how the rule of law is connected with other national characteristics, such as economic growth.

BackgroundInstitutions describe the constraints that structure political, economic and social interaction, such as how power is gained and exercised, how order is kept, and the way public resources are sourced, divided and used. Institutions shape incentives and opportunities: good institutions lead to more of the things we care about, like economic growth, peace and prosperity; while bad institutions lead to more of the thing we want less of, like poverty, inequality and conflict.

A sizable share of my work sits in economic development, so institutions are never far from mind. This series, though, started with a specific job: a client hired me to look at the links between the rule of law and economic growth, and I went looking for an accessible introduction to the topic I could share. What I found was this analysis from the Atlantic Council on why the rule of law is the key to prosperity.

Having noted how important institutions are and how difficult they are to define and measure, the authors move briskly to their conclusion that the rule of law is the single most influential factor behind long-term economic growth and societal wellbeing:

Source: Annie (Yu-Lin) Lee, Joseph Lemoine, 20 August 2025, Why the rule of law is the key to prosperity: Lessons from thirty years of data, Atlantic Council (link) [Accessed 5 August 2026]

That’s quite the claim and one of the inspirations for this post as from the looks of it, much of their evidence seems to come from the strength of a collection of pairwise correlations between the rule of law and their chosen proxies for well being (as measured by the Atlantic Council’s Freedom and Prosperity Index).

Now, I’m no big city lawyer, but it seems to me that pairwise correlations are a poor piece of evidence to provide for such a strong claim. Particularly when we’re wading into the murky waters of prosperity, institutions and the causal connection between the two. And the characteristics being analyzed are plausibly all part of the same interdependent system.

But, I’m not here to judge. Firstly, as the authors do mention that prosperity depends on the interplay of multiple institutional pillars, not just the rule of law. They’ve also tackled a complicated topic in an accessible way, which is an achievement in itself. And I’ve been there before too: as cross-country correlations are fun to explore and can provide a seemingly endless array of plausible policy interventions for making the world a better place. Also, the point of this post isn’t to criticize their analysis, but to help fill the gap I noticed when searching for resources on the topic.

Instead, this post attempts to fill a gap by presenting analysis demonstrating why the rule of law, and institutions more generally, are best conceptualized as interdependent pillars within a mutually reinforcing system. That interdependence is what makes the literature so conceptually interesting and so frustrating to analyze, since the little data available on the topic arrives bundled with multicollinearity, endogeneity, collider bias, omitted variables and measurement error.

The rule of law and institutions: a primerTo obnoxiously paraphrase research1, the rule of law is thought to influence economic development and wider prosperity through a variety of avenues, such as enabling the enforcement of property rights, easing trade between unrelated parties and providing a peaceful means for resolving disputes. However, because the rule of law and its outcomes also depend on a wider set of institutions and a tangled coalition different interests, analyzing it is no simple task.2

On one level this is because institutions and their outcomes are interdependent. For instance, maybe the rule of law does drive prosperity, but perhaps prosperous countries are better able to invest in their legal systems too. Similarly, perhaps how the rule of law drives economic outcomes changes depending on the political system, geography and urbanization.

Defining and measuring amorphous concepts like the rule of law also happens to be hard, which means researchers lean heavily on surveys asking respondents for their perception of corruption, law and order and government effectiveness (etc). However, because a person’s opinion of a country’s performance in one area is likely to be influenced by their impression of it in others, these measures are likely to agree with one another for reasons that have little to do with the institutions themselves. A respondent who has watched a corruption scandal unfold is unlikely to rate the courts generously that same year.

Finally, it’s generally accepted that institutions are slow moving (or ‘sticky’), which means they don’t change much from one year to the next and will often exert their influence on wider outcomes indirectly. As a result, some of the most influential studies exploring the connection between institutions and prosperity analyse periods of a hundred years or more, such as Acemoglu, Johnson and Robinson who compared current income with a century old proxy for institutional quality (settler mortality).

Figure: Income vs. Settler mortality

Countries with higher GDP per capita now tended to be those with stronger colonial institutions (as proxied by settler mortality)

Source: Acemoglu, Daron, Simon Johnson, and James A. Robinson. 2001. “The Colonial Origins of Comparative Development: An Empirical Investigation.” American Economic Review 91 (5): 1369–1401. DOI: 10.1257/aer.91.5.1369

The Worldwide Governance IndicatorsAlthough there are good reasons to question whether the WGI provides a reliable and holistic measure of institutions, (or even that it measure what it claims to) it is arguably one of the more reliable and well-tested attempts at measuring the cross-country quality of institutions. I also have a personal preference for the WGI as somebody who is hired to design, build and evaluate composite indicators, because it gets a core part of index design right: transparently sharing their methodology and data, and being open to making revisions to reflect feedback.

The WGI also has the practical advantage of being intuitive enough for outsiders to understand what it’s trying to measure. With governance reflecting the traditions and institutions by which authority in a country is exercised across six dimensions:

Voice and Accountability: perceptions of the extent to which citizens can participate in selecting their government including electoral integrity, and of accountability mechanisms for citizens—reflected in the ability to access information, governmental oversight bodies, and a robust traditional/digital media landscape; and
Political Stability: perceptions of the extent to which political power and governance are secure from destabilization, and of the likelihood that authority will be challenged or altered through violent, coercive, or unconstitutional means. The second aspect—government capacity to formulate and implement sound policies—includes:
Government Effectiveness: perceptions of the quality of public services, the civil service, policy formulation and implementation, and the credibility of a government’s decisions; and
Regulatory Quality: perceptions of the government’s ability to design and implement policies and regulations that promote private sector development. The third aspect—respect for institutions that govern economic and social interactions— comprises:
Rule of Law: perceptions of the extent to which agents respect and follow the rules of society, including contract enforcement, property rights, the police, courts, and the likelihood of crime and violence; and
Control of Corruption: perceptions of the extent to which public power is used for private gain, including both petty and grand corruption, as well as capture of the state by elites and private interests.

Source: World Bank, 2025, “The Worldwide Governance Indicators: Revised Methodology for Measuring Governance Using Perception Data December 2025”, link.

Note: For brevity, this post uses institutions, governance and the WGI interchangeably, in full knowledge that they aren’t the same thing. The WGI is the World Bank’s conceptualization of a particular set of institutions it considers useful, interesting and/or relevant. The WGI is not meant to be an all-encompassing measure suited to every use case, it’s just the most suitable measure for this analysis. The scores are also proxies for institutional quality rather than measures of it. Proxies come with the territory when trying to measure hard-to-define things like institutions and/or governance, as direct measurement is unavailable or unsuitable, particularly for cross-country comparisons.

Figure: Worldwide Governance Indicator Dimensions

The WGI attempts to measure governance across six dimensions

Source: my exceptional design skills

FYI: Daniel Kaufmann has also publicly responded to criticism of the WGI, which has resulted in several iterations of the methodology over time.

Project Setup and Data CleaningOnce again, the WGI data can be downloaded here. Because the code analyzes all WGI indicators, fnc_read_wgi_sheets() load data from each sheet and combine it into a single dataframe.

Code: Plotting functions

```

load the packages we'll probably needlibrary(tidyverse)library(readxl)library(janitor)library(qgraph)library(rstatix) library(broom) #Note: ppcor is also required, but called directly to avoid conflicts#Install with install.packages("ppcor") if needed.#define pathref_wgi_path <- "./Data/wgidataset_with_sourcedata-2025.xlsx"# sheet code -> output column nameref_wgi_indices <- c( va = "wgi_voice", # Voice and Accountability pv = "wgi_polstab", # Political Stability & Absence of Violence ge = "wgi_goveff", # Government Effectiveness rq = "wgi_regqual", # Regulatory Quality rl = "wgi_rol", # Rule of Law cc = "wgi_corrupt" # Control of Corruption)# Read one WGI sheet and return iso3c, year, and the single renamed estimate.# Renaming per sheet avoids the shared 'governance_estimate...' column colliding# on join. Fails loudly if a sheet's estimate column is named differently.fnc_read_wgi_sheet <- function(sheet_code, col_name, path) { read_excel(path, sheet = sheet_code) |> clean_names() |> rename(iso3c = economy_code, !!col_name := governance_estimate_approx_2_5_to_2_5) |> select(iso3c, year, all_of(col_name))}# read all six, then join on the common keysdta_wgi_2025 <- ref_wgi_indices |> imap((col_name, sheet_code) fnc_read_wgi_sheet(sheet_code, col_name, ref_wgi_path)) |> reduce(full_join, by = c("iso3c", "year"))#Add metadata (country + year-specific income bracket) taken once, from any sheettmp_dta_wgi_meta <- read_excel(ref_wgi_path, sheet = "rl") |> clean_names() |> rename(iso3c = economy_code, country = economy_name) |> select(iso3c, year, country, income_classification)#add meta datadta_wgi_2025 <- dta_wgi_2025 |> left_join(tmp_dta_wgi_meta, by = c("iso3c", "year")) |> relocate(iso3c, country, year, income_classification)#drop temporary objects (ref_ objects are kept so the chunk can be re-run)rm(tmp_dta_wgi_meta)#define index column names and labels:ref_wgi_cols <- c("wgi_voice", "wgi_polstab", "wgi_goveff", "wgi_regqual", "wgi_rol", "wgi_corrupt")ref_wgi_labels <- c(wgi_voice = "Voice & accountability", wgi_polstab = "Political stability", wgi_goveff = "Govt effectiveness", wgi_regqual = "Regulatory quality", wgi_rol = "Rule of law", wgi_corrupt = "Control of corruption")#shortened labels ref_wgi_short <- c(wgi_voice = "Voice & Acc.", wgi_polstab = "Pol. stability", wgi_goveff = "Gov. effect.", wgi_regqual = "Reg. quality", wgi_rol = "Rule of law", wgi_corrupt = "Corruption")# assumptionsref_min_years <- 20 # minimum years of data before a country average is usedref_pcor_cut <- 0.05 # partial correlations below this aren't drawn in the network

``` The rule of law as part of an institutional portfolioTo demonstrate what a statistical minefield analyzing institutions is, we’ll start by applying the same technique that I criticized earlier: pairwise correlations. With the idea being to demonstrate that different measures of institutional strength tend to agree with one another. In addition to this being what you might expect if institutions were endogenously determined, interdependent and/or WGI dimensions measured something similar. It also illustrates why you can’t rely on associations alone to cleanly establish the causal connection between a particular institution and economic and social outcomes.

For comparing the shared strength of institutions we’ll use a country’s average score for each WGI dimension from 1996 to 2024.3 Although this reduces our sample size and ignores a variety of country-specific factors that might influence institutions, it provides a simple way to reduce year-to-year noise and focus on between-country effects. It’s also unlikely we’ll lose too much information, given institutions move slowly from year-to-year. But, averaging cuts both ways too: as stripping out year-to-year noise also strips out measurement error, making correlations mechanically higher within any single year.

Reflecting that WGI dimensions might just be unrelated series that are trending together, we’ll also examine pairwise correlations for annual changes in WGI scores across dimensions. Aside from checking whether the level associations are just shared trends, dimensions moving together year-to-year would be consistent with institutions being connected to one another. For instance, if the rule of law supports efforts to fight corruption, we might expect the quality of these two institutions to move in the same direction.

However, the use of averages for comparing levels and raw values for first-differences also warrants caution when making comparisons: As while the two describe the same countries, the different units of analysis and number of observations aren’t directly comparable.

Code: calculate country averages and first differences

```

Country averages (levels).#Countries are only kept where every dimension has at least ref_min_years of#datadta_wgi_country <- dta_wgi_2025 |> group_by(iso3c) |> filter(if_all(all_of(ref_wgi_cols), (x) sum(!is.na(x)) >= ref_min_years)) |> summarise(across(all_of(ref_wgi_cols), (x) mean(x, na.rm = TRUE)), nmb_years = n(), .groups = "drop")#how many countries survive the coverage filterchk_wgi_country_nmb <- nrow(dta_wgi_country)chk_wgi_country_nmb#first difference column namesref_wgi_diff_cols <- paste0("d_", ref_wgi_cols)#work out which rows sit next to a consecutive year#(the WGI was published every other year before 2002, so we can't just lag blindly)dta_wgi_gaps <- dta_wgi_2025 |> arrange(iso3c, year) |> group_by(iso3c) |> mutate(yr_gap = year - lag(year)) |> ungroup()#then take the year-on-year change, but only where the gap is a single yeardta_wgi_diff <- dta_wgi_gaps |> group_by(iso3c) |> mutate(across(all_of(ref_wgi_cols), (x) if_else(yr_gap == 1, x - lag(x), NA_real_), .names = "d_{.col}")) |> ungroup() |> select(iso3c, year, all_of(c(ref_wgi_cols, ref_wgi_diff_cols)))#Complete-case frames used for BOTH the zero-order and partial correlations.dta_wgi_lvl_cc <- dta_wgi_country |> select(all_of(ref_wgi_cols)) |> drop_na()dta_wgi_dif_cc <- dta_wgi_diff |> select(iso3c, all_of(ref_wgi_diff_cols)) |> drop_na()#sample sizes chk_sample_sizes <- tibble( series = c("WGI Levels", "WGI First Differences"), nmb_obs = c(nrow(dta_wgi_lvl_cc), nrow(dta_wgi_dif_cc)), nmb_country = c(nrow(dta_wgi_lvl_cc), n_distinct(dta_wgi_dif_cc$iso3c)), unit = c("country", "country-year"))

``` Note: The WGI methodology has been updated in 2025 to improve comparability of governance scores over time.

Code: Correlation analysis

{r}#create a function for calculating correlations fnc\_cor\_p <- function(dta, cols) { dta |> select(all\_of(cols)) |> cor\_test(vars = cols) |> rename(x = var1, y = var2) |> filter(match(x, cols) < match(y, cols)) |> mutate(across(c(x, y), \(v) sub("^d\_", "", v))) |> select(x, y, cor, p)}#calculate association(s) on the shared complete-case framessum\_wgi\_cor\_level <- fnc\_cor\_p(dta\_wgi\_lvl\_cc, ref\_wgi\_cols)sum\_wgi\_cor\_diff <- fnc\_cor\_p(dta\_wgi\_dif\_cc, ref\_wgi\_diff\_cols)#Holm adjustment for testing 15 pairs at once. Note the p-values for the#differences assume ~5,000 independent observations when they are really ~200#countries observed repeatedly - so significance there is close to guaranteed#and the effect sizes are what matter.sum\_wgi\_cor\_level <- sum\_wgi\_cor\_level |> mutate(sig = p.adjust(p, "holm") < 0.05, r2 = cor^2)sum\_wgi\_cor\_diff <- sum\_wgi\_cor\_diff |> mutate(sig = p.adjust(p, "holm") < 0.05, r2 = cor^2) This code chunk just creates a basic bubble plot for visualizing the correlation matrix. This is perhaps more verbose than it needs to be, but it felt fair to give Claude the satisfaction of trying to apply my style guidelines.

Code: Plotting functions

```

Bubble plot for a correlation matrix. value_col lets the same function handle#zero-order (cor) and partial (pcor) tables.fnc_plot_bubbles <- function(dta, value_col = "cor", wrap = 12) { ref_lbl <- setNames(str_wrap(ref_wgi_labels, wrap), ref_wgi_cols) dta <- dta |> rename(value = all_of(value_col)) |> mutate(x = factor(x, ref_wgi_cols), y = factor(y, ref_wgi_cols), lbl = ifelse(sig, sprintf("%.2f", value), sprintf("(%.2f)", value))) ggplot(dta, aes(x, y)) + geom_point(aes(size = abs(value), fill = value), shape = 21, colour = "white") + geom_text(aes(label = lbl, colour = abs(value) > 0.55), size = 3, fontface = "bold") + scale_fill_gradient2(low = "#922C40", mid = "#FFFFFF", high = "#16AF8E", midpoint = 0, limits = c(-1, 1), name = "Correlation") + scale_colour_manual(values = c(`TRUE` = "white", `FALSE` = "#121212"), guide = "none") + scale_size_area(max_size = 16, limits = c(0, 1), guide = "none") + scale_x_discrete(labels = ref_lbl) + scale_y_discrete(labels = ref_lbl, limits = rev) + coord_fixed() + labs(x = NULL, y = NULL) + theme_minimal()}# Convert the long pairwise tibble back into a symmetric matrix for qgraphfnc_pcor_matrix <- function(dta_pcor, col) { tmp_mat <- matrix(0, length(ref_wgi_cols), length(ref_wgi_cols), dimnames = list(ref_wgi_cols, ref_wgi_cols)) tmp_mat[cbind(dta_pcor$x, dta_pcor$y)] <- dta_pcor[[col]] tmp_mat[cbind(dta_pcor$y, dta_pcor$x)] <- dta_pcor[[col]] tmp_mat}#Network of direct links. Edges below `cut` in absolute size are NOT DRAWN, so a#missing line means "smaller than the threshold", not "zero".fnc_plot_network <- function(dta_pcor, cut = ref_pcor_cut, layout = "spring", seed = 123) { set.seed(seed) mat_pcor <- fnc_pcor_matrix(dta_pcor, "pcor") chk_sig <- fnc_pcor_matrix(dta_pcor, "sig") == 1 qgraph(mat_pcor, layout = layout, minimum = cut, maximum = 1, esize = 9, fade = TRUE, posCol = "#16AF8E", negCol = "#922C40", color = "white", border.color = "#E5E7EB", border.width = 2, labels = ref_wgi_short[ref_wgi_cols], label.cex = 0.8, label.color = "#121212", vsize = 11, shape = "circle", lty = ifelse(chk_sig, 1, 2), edge.labels = ifelse(chk_sig, sprintf("%.2f", mat_pcor), sprintf("(%.2f)", mat_pcor)), edge.label.cex = 0.75, edge.label.bg = "white", edge.label.color = "#121212", mar = rep(5, 4))}

``` Strong institutions coincide with one anotherIn this section, pairwise correlations are used to test whether the strength of individual institutions tends to occur together.

It’ll come as no surprise that all of the WGI’s dimensions are strongly associated with each other, which suggests that on average a country scoring highly on one dimension probably scores highly on the others too. For the WGI’s Rule of Law measure, the pairwise associations are strong across the board, with the implied R squared statistic suggesting it accounts for somewhere between 65 and 90 percent of the variation in the other dimensions. Political Stability has the weakest associations, accounting for somewhere between 50 and 75 percent.

Figure: WGI pairwise correlations (levels)

The strength of institutions are strongly associated with one another across WGI dimensions

The network plot presents the same pairwise associations once the influence of the other dimensions is accounted for. Where partial correlations fall well below the zero-order correlations shown above, it points to dimensions carrying overlapping information. For the Rule of Law, the network diagram presents much weaker associations with other dimensions to the figure above, with moderate associations remaining for Voice and Accountability, Control of Corruption and Political Stability. Although it’s best not to take the implications of this analysis too far, one interpretation of these results that makes intuitive sense is that different institutions relate to one another differently. And sometimes this relationship might be indirect, such as government effectiveness indirectly influencing the rule of law via corruption controls.

Figure: WGI pairwise partial correlations (levels)

Pairwise correlations are lower once the influence of other WGI dimensions are accounted for

Code: Plotting pairwise correlations at levels

```

produce and display the correlation matrixplt_wgi_cor_level <- fnc_plot_bubbles(sum_wgi_cor_level)plt_wgi_cor_level#network plot for partial correlations plt_wgi_net_level <- fnc_plot_network(sum_wgi_pcor_level)

``` Institutions are connected to one another over time (albeit, loosely)This section explores pairwise correlations between year-to-year movements of WGI dimension to explore evidence for institutions being interdependent.

Focusing on year-to-year movements in WGI dimensions rather than levels results in smaller correlation statistics across the board, which is to be expected given taking the first difference naturally inflates the influence of noise and measurement errors. Still, the associations point to a similar picture to associations at the levels: a positive and statistically significant association between all dimensions, with Political Stability being the weakest.

Having said this, statistical significance just suggests that shared movement between series isn’t zero, not that it’s particularly interesting. Added to this, given the first differences pool values so they are considered independent (despite being from the same countries), significance is close to guaranteed, which is yet another reason they should be interpreted with caution.

The implied explanatory power is also probably what matters more and it’s generally modest. For instance, the R squared statistic for Political stability suggests it can explain between 1 to 4 percent of the year-to-year variation in other dimensions. Whereas the Rule of Law, which holds the strongest pairwise explanatory power across dimensions (setting political stability aside), only explains somewhere between 9 to 17 percent of the variation, which while being nothing to sneeze at, still leaves a lot of unexplained movement.

Figure: WGI pairwise correlations (year-to-year changes)

Year-to-year movements in institutional quality are statistically associated with one another

Note: Associations between first differences are expected to be weaker for several reasons: institutions are slow moving; interdependence between institutions may operate with lags or in a non-linear fashion; and as differencing strips out persistent components of each series while retaining measurement errors intact, the signal to noise ratio is likely to be lower. The authors of the WGI also warn against taking too much stock of year-to-year changes in scores.

The second plot once again presents how year-to-year movements are associated with each other, but after accounting for influences outside the examined pair. Once again, the Rule of Law holds the strongest association with other dimensions, but its association with Control of Corruption is much lower, suggesting that much of the pairwise association between year-to-year changes above relate to other dimension outside the pair.

Figure: WGI pairwise partial correlations (year-to-year changes)

Pairwise correlations are reduced, but in most cases remain statistically significant, once the influence of other dimensions is accounted for

Code: plot first difference associations

```

correlaton plot for correlations @ first differenceplt_wgi_cor_diff <- fnc_plot_bubbles(sum_wgi_cor_diff)plt_wgi_cor_diff#network plot for partial correlatons @ first differenceplt_wgi_net_diff <- fnc_plot_network(sum_wgi_pcor_diff, layout = plt_wgi_net_level$layout)

``` Six dimensions, one signal(?)This section uses Principal Components Analysis (PCA) to explore whether all six dimensions of the WGI measure, or are caused by, the same thing.

A strong critique of the WGI is that dimensions more or less measure the same thing. From a conceptual standpoint this might have some weight, due to the perceived quality of one dimension influencing perceptions of another, the use of overlapping data sources and the fact that the definitions are not precisely defined. But, as somebody that works a lot with composite indices, I’d say this comes with the territory and I’m happy to leave it to the experts to argue among themselves.

However, from a statistical standpoint this might matter a lot, as it determines whether a measured link between the Rule of Law and outcomes like prosperity can be meaningfully interpreted as telling us anything about the Rule of Law.

This is explored in the code below by applying PCA to the WGI levels and first differences. PCA attempts to collapse collinear variables into a set of principal components (PC) that explain as much variability as possible. If the WGI’s dimensions are measuring or being driven by something common, we might expect a small number of PCs will explain a disproportional share of the variance.

Code: apply PCA to levels and first differences

```

Fit a scaled PCA and pull out variance shares and PC1 loadings.#Scaling so each index counts equally regardless of how spread out it is.#Note the levels PCA runs on country averages (one row per country) while the#differences PCA runs on pooled country-years - the shares are not measured on#the same unit of analysis.fnc_pca_tidy <- function(dta, cols, series) { dta_input <- dta |> select(all_of(cols)) |> drop_na() mod <- prcomp(dta_input, scale. = TRUE) #share of variance picked up by each component sum_var <- mod |> tidy(matrix = "eigenvalues") |> transmute(series, pc = paste0("PC", PC), var_pct = percent, nmb_obs = nrow(dta_input)) #how strongly each index marks the first component sum_load <- mod |> tidy(matrix = "rotation") |> filter(PC == 1) |> transmute(series, index = sub("^d_", "", column), loading = abs(value)) list(var = sum_var, load = sum_load)}ref_pca_series <- c("WGI Levels", "WGI First Differences")tmp_pca <- list(fnc_pca_tidy(dta_wgi_lvl_cc, ref_wgi_cols, ref_pca_series[1]), fnc_pca_tidy(dta_wgi_dif_cc, ref_wgi_diff_cols, ref_pca_series[2]))sum_wgi_pca_var <- tmp_pca |> map("var") |> list_rbind() |> mutate(series = factor(series, ref_pca_series))sum_wgi_pca_load <- tmp_pca |> map("load") |> list_rbind() |> mutate(series = factor(series, ref_pca_series))#drop temporary objectsrm(tmp_pca)

``` The first plot presents the share of variance explained by each principal component. In the case of the the WGI’s levels, the PCA indicates that almost 90 percent of the measured variance in governance can be explained by a single principal component. Indicating that the majority of information presented by the six WGI dimensions could be efficiently described by a single measure. A result that supports the idea that either the WGI is measuring a similar thing and/or that a common factor is driving all six dimensions.

Both PCAs point in a similar direction, although the first differences are much less dramatic: with the first component picks up around 40 percent of the variance, against nearly 90 percent for the levels. Bear in mind these two numbers aren’t measured on the same unit of analysis, making them not directly comparable (i.e. approximately 200 country averages vs 5,000+ first differences). It’s therefore best not to read too much into comparisons, particularly given the authors of the WGI explicitly warn against analyzing year-to-year score movements.

Figure: Variance explained by principal component

The majority of variance can be explained by a single PC for WGI levels, while a larger number of PCs are required to provide a sufficient explanation of variability for first differences

Code: how concentrated is the common dimension?

{r}plt\_wgi\_pca\_scree <- sum\_wgi\_pca\_var |> ggplot(aes(pc, var\_pct)) + geom\_col(fill = "#1E298D", width = 0.7) + geom\_text(aes(label = scales::percent(var\_pct, accuracy = 1)), vjust = -0.4, size = 3, colour = "grey40") + scale\_y\_continuous(labels = scales::percent, limits = c(0, 1)) + facet\_wrap(~ series) + labs(x = NULL, y = "Variance explained (%)") + theme\_minimal(base\_size = 10) + theme(panel.grid.major.x = element\_blank(), panel.grid.minor = element\_blank(), strip.text = element\_text(face = "bold", hjust = 0))plt\_wgi\_pca\_scree The plot below presents PCA loadings, which measure how strongly each dimension contributes to PC1. Higher loadings mean a dimension is more closely tied to the component, and so shares more with the others. That the dimensions carry comparable loadings at both the levels and the first differences is again consistent with them measuring, or being influenced by, something common.

Political Stability sits lower than the rest, but I’d be reluctant to take my interpretation of that too far given how much variance remains unexplained. It’s also roughly what you might expect of a dimension intended to capture shocks rather than gradual shifts, or one behaving non-linearly (questions better answered in a separate post).

Figure: PCA Loadings

PCA loadings for both the levels and first differences are generally evenly spread

Code: PCA loadings

plt\_wgi\_pca\_load <- sum\_wgi\_pca\_load |> mutate(index = factor(index, ref\_wgi\_cols, ref\_wgi\_labels) |> fct\_reorder(loading)) |> ggplot(aes(loading, index)) + geom\_col(fill = "#1E298D", width = 0.65) + geom\_text(aes(label = sprintf("%.2f", loading)), hjust = -0.25, size = 3, colour = "grey40") + scale\_x\_continuous(limits = c(0, 0.7), expand = expansion(c(0, 0.05))) + facet\_wrap(~ series) + labs(x = "Absolute loading on PC1", y = NULL) + theme\_minimal(base\_size = 10) + theme(panel.grid.major.y = element\_blank(), panel.grid.minor = element\_blank(), strip.text = element\_text(face = "bold", hjust = 0))plt\_wgi\_pca\_load Summing upAcross three views of the same data, the six WGI dimensions behave like parts of one system rather than six separable things. Country averages correlate strongly across every pair; most associations remain even when accounting for the influence of other dimensions; and a single principal component accounts for a significant share of variation in governance scores between countries. Dimensions seem to move together too, albeit weakly, which is while not a slam dunk, does point to WGI scores having some interdependence.

But, it’s worth being clear about what this analysis doesn’t establish: Collinearity isn’t endogeneity and while a system of interdependent institutions might produce this pattern, so would WGI dimensions measuring the same thing.

But, the point of this post isn’t to support the legitimacy of the WGI or any other measure. And the distinction matters little to the point of this post. As whether dimensions move together because they co-determine each other, share a common driver, or share source data, the implications for anyone trying to understand or analyze institutions is the same: they have to be examined as a set as a correlation between one institution and an outcome may strong regardless of whether that pillar is doing the actual work or not.

How AI was used for this post: Claude was used heavily to refine the code and lightly leaned on to improve the accuracy and readability of the text. The former was mainly in an attempt to address Claude’s almost endless array of suggestions for adding more code, while the latter was mainly a result of having stared too long at my own writing.

No doubt errors remain as it’s quite the topic, which means I’ve made quite a number of edits to both the text and code. Feel free to contact me here.

Additional Note:

The motivation for writing this was the absence of descriptive analysis on the links between law and order and economic growth aimed at a general audience. Keeping the code in the post has probably cost it some readability, but my reasoning for sharing the analysis so openly was to make it easier for others to build on this post to fill the many gaps out there (including me in my future posts).

The inspiration for this series came from work I completed in 2025 for the Bingham Centre for the Rule of Law and the Law Society of England and Wales. A paper based on the work summarizing research on the topic is available here.

For a great summary of the research, see Dr Lopez-Gomez, L. (2026, The Rule of Law and the Institutional Roots of Economic Performance. The British Institute of International and Comparative Law, (link). Also see: Haggard, S., MacIntyre, A. and Tiede, L., 2008. The rule of law and economic development. Annu. Rev. Polit. Sci., 11(1), pp.205-234; and Besley, T., Bogart, D., Chapman, J. and Nuno, P., 2025. Justices of the peace: Legal foundations of the industrial revolution (No. 20214). Centre for Economic Policy Research. ︎
One of the better attempts I’ve seen to empirically test a causal connection between the two: Besley, T., Bogart, D., Chapman, J. and Nuno, P., 2025. Justices of the peace: Legal foundations of the industrial revolution (No. 20214). Centre for Economic Policy Research. ︎

The post Institutions Are a Package Deal: What a Correlation Can’t Tell You About the Rule of Law appeared first on Giles.

To leave a comment for the author, please follow the link and comment on their blog: Data Analytics and AI Archives - Giles.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Institutions Are a Package Deal: What a Correlation Can’t Tell You About the Rule of Law

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This is another short, simple blog post building a chart I am going to be updating regularly as part of monitoring the impact of the slow-burn fuel crisis caused by the USA war with Iran.

I have a page on this website with a few charts on the fuel crisis.

Petrol (or gasoline) and diesel pricesHere’s today’s new chart.

For those interested, here are some of the key features I thought through and are deliberately part of the polish here

Both New Zealand and US prices on the same basis for direct comparability (and to shock USians with how much the rest of the world pays for gasoline).
Mapping colour to country and using the facets for fuel type, rather than vice versa; doing it this way makes it easier to compare country-to-country (rather than fuel-to-fuel, which I found less interesting).
Using colour in the title (thanks to ggtext by Claus O. Wilke and Brenton M. Wiernik) rather than a legend or direct labelling of the lines—very strong clutter-reduction technique in this context, I think.
Annotations, in carefully chosen grey italics, to indicate the key events/periods of interest and answer the obvious questions that anyone looking at the chart (including me, this morning) would have eg “what happened in XXX?”
Colours for countries chosen to suggest their flags.
Make sure the audience understands the peculiar tax situation for diesel in New Zealand (diesel and electric vehicles pay a road user charge per distance travelled, not fuel consumed, which isn’t included in these prices).
I considered, and eventually decided against, forcing the vertical axis scale to go down to zero. Still a bit unsure about this one, I see pros and cons of both options.

I’ve also got a zoomed in version of the chart looking at just 2026. This is actually quite a bit less interesting. I like the big sweep of the past twenty years shown in the main chart, and the way we can link price changes to major events.

Data sources and codeSo there’s nothing particularly complex in the code. I needed three data sources:

New Zealand fuel prices, provided by the Ministry of Business, Innovation and Employment
USA fuel prices provided by the Energy Information Administration
NZD / USD exchange rate, which I have taken from FRED, the St Louis Federal Reserve data service.

There were choices to make about exactly which series to match up and show, but not too difficult and I think I chose right.

There’s also a bit of fiddling around to not download the data files every time the script is run, but only when the data is to some degree stale. The prices series are weekly so there’s no point hitting the provider’s server for yet another copy of the file when the last published observation was six days or less ago.

library(tidyverse)library(janitor)library(readxl)library(patchwork)library(ggtext)#---------------New Zealand----------------------# Download petrol prices from MBIE. Not sure how to determine if it is 'stale'# or not, seems to get 10 days out of date at least.download.file( "https://www.mbie.govt.nz/assets/Data-Files/Energy/Weekly-fuel-price-monitoring/weekly-table.csv", destfile = "nz-petrol-prices.csv")# For some reason this crashes R# nz <- read\_csv("nz-petrol-prices.csv")# so need to use read.csv instead# The exchange rate file is a bit slow to donwload so only want to download it# if necessary ie latest value is more than 10 days oldstale\_fx <- TRUEif (file.exists("nzd\_usd.csv")) { nzd\_usd <- read\_csv("nzd\_usd.csv") if (as.numeric(Sys.Date() - max(nzd\_usd$observation\_date)) < 10) { stale\_fx <- FALSE }}if (stale\_fx) { download.file( "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DEXUSNZ", destfile = "nzd\_usd.csv" ) nzd\_usd <- read\_csv("nzd\_usd.csv")}nz <- read.csv("nz-petrol-prices.csv") |> as\_tibble() |> clean\_names() |> mutate(date = as.Date(date)) |> filter(variable == "Adjusted retail price") |> left\_join(nzd\_usd, by = c("date" = "observation\_date")) |> arrange(date) |> fill(DEXUSNZ, .direction = "down") |> mutate(value\_usd\_gallon = value * DEXUSNZ * 3.78541 / 100) |> mutate(fuel = ifelse(fuel == "Premium Petrol 95R", "Premium Petrol", fuel)) |> select(date, fuel, value\_usd\_gallon) |> mutate(country = "New Zealand")# Adjusted retail price is# "The national average price paid by consumers for a given fuel for the week. "# note, different from "Board price" which is the advertised rate Decided the# Adjusted retail price (i.e. what actually paid) was most comparable to the USA# series in the next section.#------------------USA---------------------# See https://www.eia.gov/dnav/pet/pet\_pri\_gnd\_dcus\_nus\_w.htmstale\_usa <- TRUEif (file.exists("usa-petrol-prices.xls")) { tmp <- read\_excel("usa-petrol-prices.xls", sheet = "Data 1", skip = 2) if (as.numeric(Sys.Date() - max(as.Date(tmp$Date))) < 7) { stale\_usa <- FALSE }}if (stale\_usa) { download.file( "https://www.eia.gov/dnav/pet/xls/PET\_PRI\_GND\_DCUS\_NUS\_W.xls", destfile = "usa-petrol-prices.xls", mode = "wb" )}usa <- read\_excel("usa-petrol-prices.xls", sheet = "Data 1", skip = 2) |> mutate(Date = as.Date(Date)) |> select( date = Date, `Regular Petrol` = `Weekly U.S. Regular All Formulations Retail Gasoline Prices (Dollars per Gallon)`, `Premium Petrol` = `Weekly U.S. Premium All Formulations Retail Gasoline Prices (Dollars per Gallon)`, Diesel = `Weekly U.S. No 2 Diesel Ultra Low Sulfur (0-15 ppm) Retail Prices (Dollars per Gallon)` ) |> gather(fuel, value\_usd\_gallon, -date) |> mutate(country = "USA")#------------combine the two----------------combined\_petrol <- usa |> rbind(nz) |> filter(date >= min(nz$date)) |> filter(fuel != "Premium Petrol") |> mutate(fuel = fct\_relevel(fuel, "Regular Petrol"))#-----------------plot drawing---------------annotations <- tibble( date = as.Date(c( "2008-01-01", "2013-01-01", "2016-10-01", "2022-06-01", "2026-02-01" )), value\_usd\_gallon = 8.5, fuel = "Regular Petrol", country = "USA", label = c( "Buildup to Global\nFinancial Crisis", "'$100 oil plateau'", "US shale comes online", "Russia invades Ukraine", "USA attacks Iran" )) |> mutate(fuel = factor(fuel, levels = levels(combined\_petrol$fuel)))# Base definition of chart, used in both versions:p0 <- combined\_petrol |> ggplot(aes(x = date, y = value\_usd\_gallon, colour = country)) + facet\_wrap(~fuel, ncol = 1) + geom\_line(linewidth = 0.7) + scale\_y\_continuous(label = dollar) + scale\_colour\_manual(values = c("New Zealand" = "blue", "USA" = "red")) + labs( x = "", colour = "", y = "Price (USD per gallon)", title = "Retail petrol and diesel prices 2004-2026, <span style='color:#0000FF;'>**New Zealand**</span> vs <span style='color:red;'>**USA**</span>, (USD/gallon).", subtitle = "New Zealand prices include petrol excise, GST and other taxes but exclude diesel fuel excise.", caption = "Source: New Zealand MBIE, USA EIA" ) + theme(legend.position = "none", plot.title = element\_markdown())# Main chart:p1 <- p0 + geom\_text( data = annotations, aes(label = label), colour = "grey40", vjust = 1, size = 2.9, fontface = "italic" )# Zoomed in on 2026:p2 <- p0 + filter(combined\_petrol, date >= "2026-01-01") + geom\_point() + labs( title = "Retail petrol and diesel prices 2026, <span style='color:#0000FF;'>**New Zealand**</span> vs <span style='color:red;'>**USA**</span>, (USD/gallon)." ) + scale\_x\_date( date\_breaks = "1 month", date\_labels = "%B" )print(p1)print(p2) That’s all for today.

To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: NZ and US petrol (‘gasoline’) and diesel prices by @ellis2013nz

View Details

[This article was first published on Seascapemodels, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. I realized this week that I won’t be hiring a GIS focused research assistant the same way as I used to.

Claude Code is just too good at GIS in R now. I guess this happened around the release of Opus 4.7. I just hadn’t really pushed the coding agent until now.

This week I downloaded a global dataset of a coastal habitat of several gigabytes, expecting to spend my free time this week poking around getting a feel for how to do my intended analysis.

30 minutes later the analysis was done.

Draft map where Claude intersected multiple coastal wetland distributions, complete with cool looking hex shaped spatial unitsClaude solved what used to be significant and time consuming GIS issues with no help from me. All I gave it was the end goal and reference to a previous workflow for a similar project.

It smoothly addressed complex challenges like geometry self intersections and spatial intersections of raster to vector data. It even ran small-scale tests to identify processing bottlenecks and find the fastest solution for intersecting two large layers.

HeadstartsNow to be fair I should also acknowledge a couple of headstarts Claude had on this project.

The first was my previous workflow for a similar problem (building a predictive model from spatial data). That gave it a rough set of steps to follow. But, my previous workflow was built entirely on vector (polygon) data, whereas the new analysis required working with rasters. So it successfully solved raster to vector issues with no help from me.

I suspect a second big change in the R GIS ecosystem in last 5-10 years has enabled this coding agent success. Credit is due to the developers of core R spatial packages, especially sf, tmap and terra.

These have significantly streamlined spatial processing and mapping in R, making it easier to deal with complex area and distance calculations on the earth’s curved surface and the ubiquitous self-intersection errors that used to cause major traffic jams in GIS work.

Back around 2020 we were compiling global data on coastal wetlands and aligning that for an analysis of coastal wetland health. It took a GIS research assistant and part of the time of a supervising post-doc months just to do the spatial processing.

Do I still need a (human) GIS assistant for spatial analyses?One of my research students has taught me that perhaps the answer is yes, but the requirements of the role are different.

You see, my student enthusiastically launched his spatial analysis project with a heap of vibe coding in R. I quickly realized he understood very little of what his coding agent had done, and there was no way we could be sure the results were accurate.

Even in my 30 minute success story told above there were AI mistakes I had to check and fix. A big one was a silly duplicate correction for fraction -> percentage that Claude made. So it had overestimated everything by a factor of 100.

My student turned his story around as well. His enthusiasm for the result and for learning, meant he started again. This time, working (slightly) more slowly. He did the reading, learnt the GIS theory and then used the AI agent to redevelop his analysis step-by-step. That way his understanding could keep pace with the code.

Now in a few short months he’s a competent GIS user and map maker. Credit to his attention to detail too, he won’t trust any exciting result without first running thorough checks.

So what I would want now in an RA is someone who is enthusiastic to learn and who excels at rapid self-paced learning.

In the past meetings with the assistant would often be about how to solve the next processing challenge. Meetings with my GIS, R and AI enabled student are very different. They come with 5+ graphs and maps every week and we discuss patterns. He’ll show me a beautifully prepared graph and ask me ‘what do you notice here…’ waiting to see if I spot the anomalies he’s noticed. Then we discuss.

Attention to detail is more important than ever. We can’t rely on failures in the code or workflow as a red flag for errors anymore. The agents are too good at ‘making it work’. We need analysts who don’t cross the finish line of declaring success when they get the first good result, but who are happy to go back and thoroughly explore every alternative interpretation.

So yes, I’ll still hire a GIS research assistant. But the position description has changed. Less “must know sf and terra”, more “must be sceptical of a beautiful map.” The coding is the easy part now. The judgement was always the hard part – it’s just that the coding used to take so much time that good judgement was required less often.

To leave a comment for the author, please follow the link and comment on their blog: Seascapemodels.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The GIS Research Assistant Job Has Transformed

View Details

[This article was first published on Bioconductor community blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Announcing the Bioconductor Student-ECR CouncilFrom an idea that hatched during the EuroBioC2026 Birds of a Feather session, we are super excited to announce the formation of the Student-ECR Council!

As a newly established Council, we aim to support students and early-career researchers (ECRs) in the Bioconductor community by providing a platform that brings together developers and data scientists through mentorship, networking, and professional development opportunities.

Meet the Council membersThe first virtual Student-ECR Council meeting on 21st July, 2026. Carissa Chen* (Postdoctoral Researcher at University of Padova, Italy)

I am a computational biologist with a background in molecular biology and bioinformatics. My current research combines deep learning and statistical methods to jointly analyse sub-cellular spatial transcriptomics and histology images. I have been a long-time user of Bioconductor packages and I am now taking my first steps into computational method development. By joining the Student-ECR Council, I hope to support interdisciplinary researchers like myself to navigate the Bioconductor community.

Guillaume Deflandre (PhD student at UCLouvain in Brussels, Belgium)

I am a computational biologist with a background in bio-engineering. Initially out of touch with the biomedical field, I took a leap of faith during my Master’s thesis and absolutely loved it! Since then, I have been doing research on how to optimise peptidoform identification in single-cell proteomics. As soon as I delved into this world of bioinformatics, I was introduced to Bioconductor, its packages and its community. They have helped me so much so far, and by joining the Student-ECR Council, I hope to help other researchers in the same way.

Jasper Spitzer (Postdoctoral Researcher at the University of Bonn, Germany)

I am a computational biologist with a background in immunology. Throughout my PhD, I’ve focused increasingly on computational work and am now 100% on the computational side. Currently, I am working on large-scale CRISPR screen data as well as single-cell projects and am generally interested in how structure in data can be translated into biology. I’ve been a long-time user of Bioconductor packages and have recently tried to be a more active member of the community. With the positive experience I’ve had in the community, I want to share that fun and excitement by encouraging others to be a part of the community as well.

Sophie-Marie Wind (PhD student at University of Münster, Germany)

I am a bioinformatician with a focus on high-throughput data analysis and method development. Currently, I am developing an analysis framework for 4C-seq data. My first contact with Bioconductor was during my master’s studies, when I used Bioconductor packages for data analyses. During my PhD, I became more involved in package development and came to appreciate the supportive and inspiring Bioconductor community. This experience motivated me to become more actively involved. As a member of the Student-ECR Council, I hope to share my enthusiasm for bioinformatics, contribute to the community, and help support other early-career researchers.

Kateřina Matějková (PhD student at Charles University in Prague, Czech Republic)

I am a computational biologist with a background in molecular biology and genetics. My research focuses on splicing analysis in Mendelian diseases. I was first introduced to Bioconductor packages during my Master’s, and during my PhD, I joined EuroBioC. I deeply appreciate the supportive and collaborative nature of this community. Currently, we are working to establish a Czech BioC community and create educational initiatives to support early career researchers working at the intersection of biology and informatics.

Nyasita Ondari (Research Associate at International Institute of Tropical Agriculture (IITA), Nairobi, Kenya)

I am a bioinformatician focused on building pipelines and workflows for the analysis of large-scale biological data. I have a background in bioinformatics and medical biochemistry. My first introduction to Bioconductor came during my MSc, where I used its packages to analyse my data in R. Since then, my involvement has grown. In March 2025, I instructed the first Bioconductor workshop ever held in Africa, in Nairobi. Today, I lead the Bioconductor Africa Seminar Series and support community engagement and social media across the Bioconductor project.

Our visionWe aim to create an inclusive and diverse environment for students and early-career researchers in the Bioconductor community, from those discovering Bioconductor for the first time to long-time users and developers of Bioconductor packages.

Using this platform, our goal is to create opportunities for early-career researchers (ECRs) to connect with one another, cultivating a positive and supportive community that embraces open science and reproducible research practices. We also aim to empower researchers to contribute to Bioconductor projects, strengthening transparency and encouraging community-driven software development. At the same time, we hope to strengthen the feedback loop between users and developers, encouraging collaboration and helping shape and continuously improve the Bioconductor ecosystem.

A few of our proposed initiatives include:

Mentorship programme providing peer support alongside technical guidance on areas such as package development, coding best practices, and contributing to Bioconductor
Targeted seminars covering dissemination of the latest research and career panels
Workshops such as community-contributed package demonstrations
“Good First Issue” hackathons to help new contributors gain experience by working on beginner-friendly issues

…and more!

As we’re just getting started, we would also love to hear your ideas for future events and initiatives! If you have any suggestions, please take a few minutes to fill in our poll.

To be notified about upcoming activities, please subscribe to our Zulip channel!

Bioconductor Mentorship ProgrammeWe are launching the Bioconductor Mentorship Programme and are looking for both mentees and mentors for the upcoming intake (tentatively October/November). The proposed focus areas may include:

Package development: developing R scripts to a fully fledged Bioconductor package
Technical development: guidance on coding best practices, contributing to existing packages, and navigating the Bioconductor development community.
Peer mentoring: managing a PhD, career advice and more. Other areas TBC depending on interest, so please let us know if there are any specific areas you would benefit from!

If you are interested in participating in the Bioconductor Mentorship Programme, please express your interest here.

Join us!We welcome anyone from all career stages and scientific backgrounds to join the Council. Specifically, Council members are people who choose to take an active role, for example by attending meetings when they can and helping to plan or deliver activities according to their interests and availability.

Feel free to pop by our regular Council meetings on the third Tuesday of every month at 9am CET to learn more. As we are welcoming new members who may be from different timezones, the meeting time is amenable to change so please don’t hesitate to contact us.

If you are interested in joining the Student-ECR Council, please connect with us on our dedicated Zulip channel, or reach out to us via email.

Or just say hi and introduce yourself on Zulip! (We need more friends!)

BioC2026 in SeattleWe will be holding a live Q&A and casual Student-ECR gathering during BioC2026 on Tuesday, August 11, from 8:30 to 9:00 am PT. Members of the Council will join by Zoom to meet attendees and answer questions following their pre-recorded lightning talk on Monday. Come along to meet other ECRs in the Bioconductor community and learn more about how to get involved with the Student-ECR Council!

To leave a comment for the author, please follow the link and comment on their blog: Bioconductor community blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing the Bioconductor Student-ECR Council

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.FADE INNine students in the M.S. in Psychology Research program at Brooklyn College, New York City (NYC), USA are taking the course Reproducible Psychological Research, where they are learning how to code for the first time. All of them have their cameras off. They are all learning R for the first time, and as an introduction to R, are being taught using the mtcars dataset.

Nine NYC graduate students who

don’t own a car,
don’t drive a car,
and have no interest in cars.

Even with their screens off, with every additional mtcars column we talked about, they fell deeper and deeper asleep during the lectures.

If I wanted my students to learn anything in my class—not just R—I needed to make an immediate pivot.

Change needed to happen, and happen fast.

Relevance drives rigorI didn’t take my students’ interests into consideration at the start of the course, and I was paying the price. How do I make sure that no matter what each individual student is interested in, they can be accommodated? Probably a big ask, but part of me felt like I was starting the semester in debt and needed to really pay up.

What do all my students have in common? They all live in NYC. Even if they’re not from NYC, they currently call it home.

Then came the idea:

What if I incorporate NYC Open Data into the course?

For anyone who doesn’t know, NYC does an incredible job of making public data available through the NYC Open Data Portal. Thousands of datasets, ranging from 311 calls to transportation data to slashings and stabbings (a favorite of one of my students), are available for free, for anyone, anywhere.

Immediately, there was a switch in my (virtual) classroom. Before, engagement was about as close to zero as you can get. Now, it skyrocketed. Students were asking more questions, which was great.

There were really only two options when it came to using NYC Open Data. The first, which I began using, was downloading static datasets as Excel or CSV files. While this helped with engagement, it also created a new set of problems: using outdated datasets (311, for instance, is updated daily), making sure students downloaded the correct files, and making sure they put them in the correct folders.

Luckily, the portal offers a fantastic solution to all of the problems associated with accessing static files: APIs!

However, this came with an even bigger problem:

In this class, I had to teach R, statistics, reproducible workflows, data analysis, data cleaning, data visualization, writing, interpretation, and really an innumerable number of additional skills. Did I really want to add APIs to this list?

Cue an idea.

nycOpenData: The beginningAt this point, I knew a few things:

Students need relevant data in order to stay engaged.
Students really enjoyed using NYC Open Data.
My students’ heads may explode if I teach them about APIs.

So I started thinking:

How do I incorporate NYC Open Data into the classroom moving forward without using APIs or static files?

What if I make a package?

Students are learning R, which means we’re also learning about packages—a crucial, yet simple, concept. What if there was a package dedicated to pulling NYC Open Data? That would fit right into the students’ learning path.

When I looked, there wasn’t one.

So I thought:

Why not make it myself?

Maybe a crazy idea. I had never developed an R package, never mind one that relied on APIs. What does the process look like? Do I have the skills? Will CRAN accept something that most likely only my nine students and I would ever use?

As a big believer in the mantra, “I’ll do it myself,” I already had the code to pull in the individual datasets in R, so… why not?

So, I went ahead and did it myself.

nycOpenData: The middleOriginally, I thought the easiest thing to do was build the package with one function: nyc_311(), which would simply pull in the 311 dataset. The perk was that it was one of the largest and most data-diverse datasets, so it could kill many birds with one stone.

But then I thought,

“No way will CRAN accept a package with just one function.”

So, I literally copied and pasted the same exact code 30 different times and adapted it for the 30 most popular datasets on the NYC Open Data Portal.

Through a lot of blood, sweat, and documentation, the first iteration of nycOpenData was officially created and accepted onto CRAN! I, a person who only installed packages, was now the creator of one? It was amazing to actually give back to the marvelous R community that provided me with so much for so many years.

This not only made teaching with NYC Open Data easier, but more fun too. The package did exactly what it needed to do: make working with NYC Open Data datasets incredibly easy in R.

With this version of nycOpenData, students were able to complete their final projects with one simple prompt:

Answer any question you have about NYC using open data.

Those projects eventually became the NYC Open Data Student Gallery Book, which was presented at NYC Open Data Week 2026. Each student’s project became its own chapter, and each student had the opportunity to present their work to the world. These same students who were once falling asleep while I basically sang them a lullaby, now were very much awake, displaying what they’ve learned and uncovered to an audience of professionals.

As an educator, my goal is to help my students be as successful as possible. Through nycOpenData, that goal became a reality.

All of the work I put into this package was worth it.

nycOpenData: Student contributionsThe following semester, I had the opportunity to teach a follow-up class to the same cohort. Then came another crazy idea:

What if the same students who used nycOpenData to not only learn, but also conduct research, contributed to the same infrastructure they utilized?

Basically, what if I helped my students contribute to the nycOpenData package?

This would be similar to what a junior software developer might do. Using my code as a template, they could build their own functions for whatever dataset they were interested in. The package already had 30 functions—what’s another nine?

Through light introductions to GitHub, package development, and open-source software, each of my students successfully contributed their own function to the nycOpenData package. Suddenly, they weren’t just users of open-source software—they were contributors.

Good thing I didn’t do this last semester, as that would have guaranteed brain explosions from my students.

nycOpenData: rOpenSciAs time progressed, and as I assessed the student contributions, I became more familiar with package development. Wanting the best product possible, I decided to submit the package to rOpenSci through its public peer review process. I was a first-time package developer and wanted guidance from people who had far more experience than I did. I wanted anyone—whether my students or someone using the package halfway around the world—to have the best experience possible. After all, the goal was to make NYC Open Data more accessible for anyone. To go from helping just my students, to my students helping, to potentially helping the overall R community through rOpenSci—I would have never guessed this trajectory.

This ended up being one of the best experiences I’ve had as a programmer.

The review process didn’t simply determine whether the package was ready—it made the package better. Every suggestion strengthened the documentation, improved the user experience, and taught me something new about developing open-source software.

First of all, my editor and reviewers told me exactly what I needed to hear:

Having 39 functions that basically do the same thing is not only unmanageable, but bad code.

Which was 100% the truth.

And, if I’m being honest, the package itself came about while I was teaching a course on reproducibility. I was literally teaching my students that copy-and-paste was not something to do, while I was doing the exact same thing. Sure, I brushed it off because I needed to put something together quickly, but it was time to face the facts and fix it.

With that came good news and bad news. As for every action, there is an equal and opposite reaction:

The good news: I was able to turn the 39 functions into just 3. Now, all 2,000+ datasets can be pulled from the portal. Easier to maintain, easier for users to understand, and overall a much better product.
The bad news: All of my students’ work was now gone. (Sad face.)

Thanks to the thoughtful feedback and hard work of the editor, reviewers, and everyone involved in the review process, nycOpenData was successfully accepted into rOpenSci—an indescribable privilege. I honestly can’t believe the journey from where the package started to where it is today.

I am especially grateful to my editor, Ronny Hernandez Mora, and reviewers, Haolin Dong and Michael Pascale, for the time, care, and thoughtful feedback they invested throughout the review. As a first-time package developer, I couldn’t have asked for a better introduction to open-source software development.With the foundation finally built, another thought entered my mind:

Can we use this same infrastructure for other open data portals?

Turns out, you can!

The OpenData EcosystemWith the infrastructure in place, another realization hit me: I never had to build it from scratch again.

The same foundation that powered nycOpenData could be adapted to other open data portals, making it possible to create an entire ecosystem of OpenData packages.

If students in NYC were motivated by using data regarding their own city, why wouldn’t the same hold true for students in other cities?

Since then, four additional packages have been built, turning one package into an ecosystem:

nysOpenData — New York State Open Data
mtaOpenData — Metropolitan Transportation Authority Open Data
laOpenData — Los Angeles Open Data
chiOpenData — Chicago Open Data
ausOpenData — Austin Open Data

The ecosystem continues to grow. A former student and I recently collaborated on the ausOpenData package (which was just recently accepted to CRAN), and a current intern at the NYC Open Data Lab is building sfOpenData and SeattleOpenData``. This not only gives others an opportunity to contribute, but also gives me a chance to continue involving students in the ecosystem after the original nycOpenData functions were “refactored.”

Hopefully, there are many more to come. The hard part is no longer building packages—it is deciding which open data portal comes next.

nycOpenData: The Final ProductWhat started as a single function has turned into six OpenData packages, one of which has been successfully peer-reviewed by rOpenSci, with the others built on the same foundation. Tens of billions of rows of public data, from across the country, are now easily accessible in R for anyone—from researchers and journalists to students and educators.

Looking back, I think this journey embodies the spirit of R.

One of the best things about R is that it is open source. Anyone with an idea can contribute, and those ideas have a way of growing into something much bigger than originally imagined. This entire ecosystem started because nine graduate students didn’t care about the mtcars dataset.

On top of that, the tools used to build this ecosystem were free and open source, and the packages themselves are freely available to anyone. In my family we say, “If it’s for free, it’s for me.”

What started in one virtual classroom ended up bringing together students, educators, reviewers, professionals, interns, and thousands of people who have since installed the packages.

Through this entire process, the biggest lesson learned was:

Relevance isn’t a reward for learning—it is what enables learning.

To my studentsWhile this story has my name attached to it, it wouldn’t exist without students who were willing to try something new, contribute code, ask questions, and occasionally tolerate my crazy ideas. When we started the course, I asked you all to keep an open mind and try something new-and I never would ask you to do something that I wouldn’t.

Thank you for trusting me when I asked you to do something different.

FADE OUTTo leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Journey of {nycOpenData}: From Classroom to Community

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Attentive package authors already noticed something new in their r-universe page over the past few weeks: R-universe has quietly started building and checking packages for Windows ARM64.

https://ropensci.r-universe.dev/pdftools#checktable
https://cran.r-universe.dev/dplyr#checktable
https://bioc-release.r-universe.dev/S4Vectors#checktable

R-universe has supported ARM64 on Linux and macOS for some time already, but not Windows. With this addition, packages are now built and checked on both x86_64 and ARM64 across all major operating systems, making the R ecosystem fully ARM ready.

Porting the R ecosystem to ARM WindowsThe R-universe service is part of a broader community effort to bring the R ecosystem to Windows ARM64. This project builds on the work from Tomáš Kalibera, whose efforts laid much of the groundwork for modernizing R on Windows to support UCRT and ARM.

Among other things, the project provides daily builds of base R installers for Windows ARM64 (R-release, R-patched, and R-devel), built on GitHub Actions with the rtools45 toolchain. Because CRAN repositories do not have Windows ARM64 binaries yet, these installers are patched to install binary packages from R-universe instead, so install.packages() just works. There is nothing users need to configure.

What we had to fix along the wayRolling this out on R-universe turned out to be a great way to find out which parts of the ecosystem were not quite ready for Windows on ARM. Some things that needed patching:

The base R installers themselves, to pick up ARM64 binaries from the right repository path.
External software such as JAGS, which needed some changes to compile rjags and runjags for arm.
Rust integration: many R packages using Rust were hard-coding the x86_64-pc-windows-gnu target. The extendr project shipped a fix in extendr-api 0.8.2, and rustup now supports the aarch64-pc-windows-gnullvm target we need.
Big compiled packages like arrow, plus a few dozen other packages with issues such as unconditionally passing -msse compiler flags (ARM chips do not do SSE) or autoconf scripts misdetecting the toolchain.

For broken CRAN and Bioconductor packages we maintain a set of install-time patches, and we are in the process of upstreaming these fixes to the package maintainers. With these patches in place, the entire R ecosystem now builds on Windows ARM64.

Testing in the real worldThe fact that everything builds does not yet guarantee that everything behaves exactly as expected. We would love help from users to find any loose ends. Hence if you are running Windows on ARM hardware, or you are using a Windows VM on MacOS, we would love for you to take this for a spin.

For package developers there several ways to test packages with compiled code on Windows ARM64. As mentioned, if your package is already on CRAN or BioConductor, you can find binaries and check results on R-universe at https://cran.r-universe.dev/{pkg} or https://bioc.r-universe.dev/{pkg}.

In addition, GitHub Actions provides free windows-11-arm runners, and the standard workflows from r-lib/actions support them out of the box.

Helping the ecosystem evolveThis project is exactly one of the reasons for building the R-universe system. Besides providing an open space for developers to publish new software, it also serves as a versatile infrastructure that lets us experiment with new tooling and platforms to help move the ecosystem forward as a whole. Having an automated platform that continuously builds thousands of packages made it possible to find those missing pieces, fix them, and iterate quickly.

By identifying and fixing the main hurdles now, we want to help package authors address the problems before they hit end-users, and pave the way for a smooth introduction of Windows ARM64 support on mainstream repositories such as CRAN and Posit Package Manager when the time is ready.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Windows ARM64 comes to R-universe

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.SAS is almost 50 years old. It predates the web, and it predates most of the open source tools data teams reach for today. While old, it is not obsolete. SAS remains central to clinical research, banking, insurance and other regulated industries, and those organisations do not keep it because it is new. They keep it because it is trusted, governed and built into critical processes.

So its age is beside the point. What matters is whether your organisation would benefit from open source for critical processes and whether it can be trusted. Some teams are already moving off it, and the useful question today is: Can regulated industries trust open source, and which of your workloads should adopt it?

Why SAS lastedSoftware survives when organisations trust it, and that matters most where mistakes are expensive.

Clinical research is a clear example. The US FDA defines the formats it accepts for electronic study data, and those standards include SAS transport files alongside CDISC data models. The FDA does not require sponsors to run their analysis in SAS; the requirement applies to the format used to submit data. But submission standards shape working practices. Over time, organisations have built SAS-based processes around these requirements, including standard outputs, controlled datasets, audit trails and reproducible analysis. The FDA also asks sponsors to provide the software programs used for certain efficacy analyses, so reviewers can reproduce the results. In that setting, reproducibility sits at the heart of the process.

Banking and insurance follow the same pattern. Credit risk models and regulatory reports must be explained, tested and defended, and SAS became established because it supported reliable, governed workflows. That is why SAS lasted: it works in environments where trust matters more than novelty.

Whether you want to start from scratch, or improve your skills, Jumping Rivers has a training course for you.

Why organisations keep SAS too longSome organisations have strong reasons to keep SAS. Others keep it because replacing it looks harder than maintaining it.

Years of logic sit inside the codeA mature SAS system contains more than programs. It contains macros, formats, reporting rules and business decisions built up over decades. Some of those rules only exist in the code, the people who wrote them may have left, and the documentation may be incomplete. This creates grown logic: systems built gradually by different people, without one clear design.

Changing the system feels risky because nobody fully understands it. But leaving it unchanged increases the risk over time, as the system becomes more deeply embedded and fewer people retain the knowledge needed to maintain or replace it.

SAS is only a componentSAS is often connected to reporting pipelines, validation processes, internal tools and downstream systems, so removing one part can affect the rest. Banking teams have described cases where legacy systems are closely tied to daily operations, and replacing them requires more than rewriting code. It requires understanding how the whole process works.

Risks – Skills are getting harder to maintainOrganisations need people who understand the existing SAS system, and they also need people who can design and validate its replacement. These are often the same people. That creates a difficult position: the team needs specialist knowledge to maintain SAS and the same knowledge to move away from it. Waiting has been classed as the safest option by some, but waiting increases the knowledge gap and increases the risk.

The vast majority of UK Universities haven’t taught SAS to undergraduate statistics students in over fifteen years!

Migration to open source is a full-stack problem that needs a full-stack solutionR and Python can perform the analysis, modelling and reporting that regulated organisations need. The main risk in a SAS to R migration lies in the process around the language; R itself is rarely the problem.

Regulated teams must prove that the new process works, and similar results are not enough. Outputs may need to match at row, value and format level, and teams must retain evidence showing how each comparison was completed. Undocumented macros create another risk: a macro may hold an important business rule that exists nowhere else, and a direct conversion can reproduce the code while losing the reasoning behind it. Migration means rebuilding the process itself, not simply translating code line by line.

A proven method for introducing open source: phased migrationReplacing the full SAS system at once creates unnecessary risk. A safer migration runs the old and new systems in parallel, so teams can compare outputs, test business rules, automate regression checks, document differences, maintain a rollback route, and validate each stage before release.

This takes longer than converting the code, but it produces a system the organisation can trust. A successful SAS to R migration is phased, governed and validated, with correctness and auditability built in from the start.

Is SAS worth keeping?Sometimes SAS still has a clear role in regulated environments, supporting established processes, submission standards and governance models. But not every SAS workload is essential. Some remain because they are critical, and others remain because nobody has reviewed them.

Occasionally, the answer is both SAS and open source, not a choice between the two. Teams can use R or Python for new analysis, modelling and visualisation while keeping SAS for existing governance, reporting or submission processes, which lets organisations modernise without introducing unnecessary risk.

The first step is assessment. Before moving anything, identify which workloads are regulator-facing, which processes contain undocumented logic, which systems depend on SAS outputs, which workloads can move now, and what evidence will be required to validate the change. That gives the organisation a clear migration path.

How Jumping Rivers helps you achieve Open SourceWe strategise for teams and systems from SAS to full implementation to R, treating migration as a controlled transformation rather than a code conversion exercise. That means understanding the existing system, identifying risk, validating outputs and moving workloads in stages.

If your organisation is deciding which SAS workloads to keep and which to move, get in touch. If you are still assessing whether R is the right choice, our Why Use R? course is designed for teams moving from SAS and SPSS. And if your organisation already uses R, Litmus provides audit-ready risk assessments for open source packages at package and function level.

Migration helps you adopt R. Validation helps you trust it.

Frequently asked questionsIs SAS still used in 2026?

Yes. SAS remains widely used in pharmaceutical research, banking, insurance and other regulated industries. Its value comes from governance, auditability and its place within established workflows.

Does the FDA require SAS?

No. The FDA accepts SAS transport files as a submission format, but this does not require sponsors to perform their analysis in SAS. These standards have, however, influenced the tools and processes used for clinical submissions.

Can R replace SAS in clinical trials?

Yes, in many workflows. The main challenge is validation: teams must show that R outputs are reproducible, correct and suitable for regulatory review. Many organisations use SAS and R together during the transition.

What are the main risks of migrating from SAS to R?

The main risks include undocumented business rules, incomplete knowledge of existing systems, differences between old and new outputs, and replacing too much at once. A phased migration with parallel testing and formal validation reduces these risks.

Can SAS and R be used together?

Yes. R can support new analysis, modelling and visualisation while SAS continues to support existing reporting, governance or submission processes. For many organisations, this is the safest route to modernisation.

References* Techzine, 47 years of SAS: advanced age gives SAS an edge in the current AI landscape * US FDA, Study Data Standards Resources * US FDA, CDER Study Data Standards Research and Development * US FDA, Study Data Technical Conformance Guide * Banking.Vision, The challenge of legacy transformation from a software development perspective * SDK.finance, How to Modernize Banking Legacy Systems via Cloud Migration * pharmaverse blog, Faster Clinical Trial Reporting: Implementing CDISC SDTM and ADaM Standards with Open Source R Packages * pharmaverse blog, Testing Containers and WebAssembly in Submissions to the FDA

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Is SAS Still Used, and Is It Worth Keeping?

View Details

[This article was first published on r on Everyday Is A School Day, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Textbooks say LOOCV has the lowest bias but highest variance compared to 10 and 5-fold. Coded a K-Fold CV from scratch for learning to test that on simulated data — and at n=1000 it holds up. At n=100? Not so much.

The above image was generated via chatGPT. Uploaded all the text of this blog post and asked it to generate a cartoon. Very impressive! It used to be spelling error and gibberish of text in the past, but now cohesive words on image. Just wow.

Motivations Crossvalidation is such a crucial step in Machine Learning (and traditional methods) that nowadays is incorporated in easy to use sklearn or tidymodels without us needing to build one from scratch. As with my other learning experience, the best way to learn the concept (other than learning the concept ) is to code it from the ground up and see how it works! In K-Fold CV, the training data is split into K chunks; the model is trained K times, each time holding out a different chunk. Performance is averaged across all K folds, giving a more stable estimate. A special case is Leave-One-Out CV (LOOCV), where each individual observation serves as its own validation set. It’s thorough but computationally expensive. I was told that, bias LOOCV < 10-fold < 5-fold; whereas variance LOOCV > 10-fold > 5-fold. Is that true? Also, what’s with the repeats, does that really reduce variance? Let’s check them out.

Objectives: * Simulate data with a known data-generating process * Implement K-Fold cross-validation from scratch * Assessing RMSE * Compare candidate models using CV RMSE * Verify the best model on a held-out test set * Opportunities For Improvement * Lessons Learnt

Simulate Data library(tidyverse)set.seed(1)n <- 1000x <- rnorm(n)w <- rnorm(n)y <- 0.5*x^2 + -0.5*w + 0.3*w*x + rnorm(n)df <- tibble(x,y,w)idx <- sample(1:n, size=0.8*n)train <- df[idx, ]test <- df[-idx, ] The above code simulates a dataset with 1000 observations, where the response variable y is generated based on a known data-generating process involving predictors x and w. The dataset is then split into a training set (80%) and a test set (20%). Let’s visualize.

df |> mutate(w\_cut = cut\_interval(w, n=5)) |> ggplot(aes(x=x, y=y, color=w\_cut, group=w\_cut)) + geom\_point(alpha=0.5) + theme\_bw() + geom\_smooth(method = "gam", se=F) Wow, very interesting visualization where the relationships are definitely not linear here. It’s some form of interaction between x and w. Let’s see if we can recover the underlying data-generating process using K-Fold Cross-Validation.

K-Fold Cross-Validation From Scratch folds <- 5segment\_portion <- nrow(train)/foldsformula\_list <- list(as.formula("y~x"),as.formula("y~I(x^2)"),as.formula("y~I(x^2)+w+w:x"),as.formula("y~I(x^3)+w+w:x"), as.formula("y~w:x"),as.formula("y~w"),as.formula("y~x+w+x:w"),as.formula("y~I(x^2)+w:x"), as.formula("y~I(x^2)+w"))cv\_log <- tibble()for (formula in formula\_list) {print(formula)predict\_log <- y\_log <- vector(mode="numeric",length=segment\_portion*folds)start <- 1end <- segment\_portion for (fold in 1:folds) { val\_i <- train[start:end,] train\_i <- train[-c(start:end),] model\_i <- lm(formula,train\_i) predict\_i <- predict(model\_i, val\_i) predict\_log[start:end] <- predict\_i y\_log[start:end] <- val\_i$y start <- end + 1 end <- start + segment\_portion - 1}val\_df <- tibble(predict=predict\_log,y=y\_log) |> mutate(formula=deparse(formula))cv\_log <- cv\_log |> bind\_rows(val\_df)}## y ~ x## y ~ I(x^2)## y ~ I(x^2) + w + w:x## y ~ I(x^3) + w + w:x## y ~ w:x## y ~ w## y ~ x + w + x:w## y ~ I(x^2) + w:x## y ~ I(x^2) + w Alright, what we’ve done above is a manual implementation of K-Fold Cross-Validation. We loop through each formula in our list, and for each formula, we split the training data into 5 folds. For each fold, we train the model on the other 4 folds and validate it on the current fold. We store the predictions and actual values for later evaluation.

We basically want to see which formula has the lowest RMSE across the folds. Let’s calculate that next. From the DGP formula, we know that the best model should be y~I(x^2)+w+w:x. Let’s see if we can recover that using K-Fold CV.

Assessing RMSE cv\_log |> group\_by(formula) |> summarize(rmse = sqrt(mean((y-predict)^2))) |> arrange(rmse) |> mutate(rmse = format(rmse, digits = 8))## # A tibble: 9 × 2## formula rmse ## <chr> <chr> ## 1 y ~ I(x^2) + w + w:x 1.0397338## 2 y ~ I(x^2) + w 1.1008693## 3 y ~ I(x^2) + w:x 1.1607225## 4 y ~ I(x^2) 1.2144857## 5 y ~ x + w + x:w 1.2826723## 6 y ~ I(x^3) + w + w:x 1.2912046## 7 y ~ w 1.3578304## 8 y ~ w:x 1.3732478## 9 y ~ x 1.4451807 Here our loss function is RMSE since y is a continuous data and we’re trying to predict that. The formula with the lowest RMSE is indeed y~I(x^2)+w+w:x, which matches the underlying data-generating process. OK at least, right now we are able to recover the underlying DGP using 5-Fold Cross-Validation. But is there a difference between 5 fold, 10 fold, or even LOOCV? If there is a difference, how do we even assess that? In the past we were able to assess bias and variance based on a true ATE, but what on earth is a true RMSE !?!

To check whether the textbook claim (bias LOOCV < 10-fold < 5-fold; variance LOOCV > 10-fold > 5-fold) holds up, we ran a small simulation with help from Claude Sonnet 5. Since we control the data-generating process, we can compare the “correct formula” (assuming the correct formula has the lowest RMSE as above) with 500 different simulated dataset against a “true” RMSE estimated from a large test set (n=10000) — large enough, by the law of large numbers, to treat as ground truth. Averaging across simulations gives bias (how far off CV runs from the true error) and variance (how much CV’s estimate swings from sample to sample) for each method. Is this legit? If the textbook claim is correct, we should be able to observe bias LOOCV < 10-fold < 5-fold; variance LOOCV > 10-fold > 5-fold. Let’s see if we can observe that in the simulation below.

Compare Candidate Models ```

set.seed(1)# k-fold CV RMSE for a given formula and dataset (k = n gives LOOCV)cv_rmse <- function(data, formula, k) { n <- nrow(data) folds <- sample(rep(1:k, length.out = n)) preds <- numeric(n) for (i in 1:k) { train_i <- data[folds != i, ] val_i <- data[folds == i, ] model_i <- lm(formula, train_i) preds[folds == i] <- predict(model_i, val_i) } sqrt(mean((data$y - preds)^2))}# "true" RMSE: fit on train, evaluate on a large fresh draw from the DGPtrue_rmse <- function(train, formula, n_test = 10000) { x <- rnorm(n_test); w <- rnorm(n_test) y <- 0.5x^2 - 0.5w + 0.3wx + rnorm(n_test) test <- tibble(x, y, w) model <- lm(formula, train) sqrt(mean((test$y - predict(model, test))^2))}formula_true <- as.formula("y ~ I(x^2) + w + w:x")n_sim <- 500n_train <- 100results <- vector("list", n_sim)for (s in 1:n_sim) { x <- rnorm(n_train); w <- rnorm(n_train) y <- 0.5x^2 - 0.5w + 0.3wx + rnorm(n_train) train_s <- tibble(x, y, w) results[[s]] <- tibble( sim = s, true_err = true_rmse(train_s, formula_true), loocv = cv_rmse(train_s, formula_true, k = n_train), cv5 = cv_rmse(train_s, formula_true, k = 5), cv10 = cv_rmse(train_s, formula_true, k = 10) )}sim_df <- bind_rows(results)sim_long <- sim_df |> pivot_longer(cols = c(loocv, cv5, cv10), names_to = "method", values_to = "cv_estimate")sim_long |> group_by(method) |> summarize( mean_cv_estimate = mean(cv_estimate), mean_true_error = mean(true_err), bias = mean(cv_estimate - true_err), variance = var(cv_estimate), .groups = "drop" ) |> arrange(bias) |> mutate(variance = format(variance, digit = 8))## # A tibble: 3 × 5## method mean_cv_estimate mean_true_error bias variance ## ## 1 loocv 1.00 1.00 0.000703 0.00053227867## 2 cv10 1.00 1.00 0.000945 0.00053192050## 3 cv5 1.00 1.00 0.00124 0.00053176441

``` Wow, looking at the results, we can see that the textbook claim holds up. LOOCV has the lowest bias, but the highest variance. 10-fold CV is in between, and 5-fold CV has the highest bias but lowest variance. But, noticed that we had to increase our digit to 8 to see the difference in variance, it’s really miniscule. Mainly because our n=1000 is already quite large, so the variance is already quite small. If we reduce n to 100, how would that look?

```

# A tibble: 3 × 5## method mean_cv_estimate mean_true_error bias variance ## ## 1 loocv 1.02 1.02 -0.00400 0.0063905148## 2 cv10 1.02 1.02 -0.00138 0.0065753049## 3 cv5 1.03 1.02 0.00244 0.0068098389

``` !?!?!?!?! with n=100, the bias and variance order heuristics no longer hold up? Why is this? I don’t know. If you do, please let me know. I even increased the n_sim to 1000, but still same patern. Intersting how I had to push the n up to 850 in order to observe the textbook variance order again, though again it’s quite miniscule.

Verify On Test Set test |> mutate(predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), train), test)) |> mutate(residual = y-predict) |> mutate(res\_square = residual^2) |> pull(res\_square) |> mean() |> sqrt()## [1] 1.018281 Alright! The RMSE on test set is quite similar to our average validation sets! Let’s visualize the predicted and actual y of the model on test set.

test |> mutate(predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), train), test)) |> ggplot(aes(x=predict, y=y)) + geom\_point(alpha=0.5) + theme\_bw() + geom\_smooth(method = "lm") + labs(title="Predicted vs Actual y on Test Set", x="Predicted y", y="Actual y")

summary(lm(y ~ predict, data=test |> mutate(predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), train), test))))## ## Call:## lm(formula = y ~ predict, data = mutate(test, predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), ## train), test)))## ## Residuals:## Min 1Q Median 3Q Max ## -2.98185 -0.61410 -0.05514 0.62883 2.28837 ## ## Coefficients:## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 0.08581 0.08390 1.023 0.308 ## predict 0.80885 0.07562 10.696 <2e-16 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## Residual standard error: 1.007 on 198 degrees of freedom## Multiple R-squared: 0.3662,Adjusted R-squared: 0.363 ## F-statistic: 114.4 on 1 and 198 DF, p-value: < 2.2e-16 Slope here is 0.81, not 1 — even with the correct formula, coefficients are still estimated from a finite, noisy sample, so predict is really “true signal + estimation error.” Regressing y on a noisy version of itself pulls the slope below 1 — a known effect called attenuation (same idea as errors-in-variables bias). More training data shrinks that estimation error, so the slope should creep closer to 1. Intercept stays near 0 because the estimation error isn’t systematically biased in one direction — just noisy — so predictions still center correctly on average. If this is true, then if we increase our n to 10000, we should see slope is closer to 1 and intercept closer to 0. Let’s test this theory out.

set.seed(1)n <- 10000x <- rnorm(n)w <- rnorm(n)y <- 0.5*x^2 + -0.5*w + 0.3*w*x + rnorm(n)df <- tibble(x,y,w)idx <- sample(1:n, size=0.8*n)train <- df[idx, ]test <- df[-idx, ]formula <- as.formula("y ~ I(x^2) + w + w:x")folds <- 10segment\_portion <- nrow(train)/foldspredict\_log <- y\_log <- vector(mode="numeric",length=segment\_portion*folds)start <- 1end <- segment\_portion for (fold in 1:folds) { val\_i <- train[start:end,] train\_i <- train[-c(start:end),] model\_i <- lm(formula,train\_i) predict\_i <- predict(model\_i, val\_i) predict\_log[start:end] <- predict\_i y\_log[start:end] <- val\_i$y start <- end + 1 end <- start + segment\_portion - 1}val\_df <- tibble(predict=predict\_log,y=y\_log) val\_df |> summarize(rmse = sqrt(mean((y-predict)^2))) |> arrange(rmse) |> mutate(rmse = format(rmse, digits = 8))## # A tibble: 1 × 1## rmse ## <chr> ## 1 1.0126176test |> mutate(predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), train), test)) |> mutate(residual = y-predict) |> mutate(res\_square = residual^2) |> pull(res\_square) |> mean() |> sqrt()## [1] 0.9884315test |> mutate(predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), train), test)) |> ggplot(aes(x=predict, y=y)) + geom\_point(alpha=0.5) + theme\_bw() + geom\_smooth(method = "lm") + labs(title="Predicted vs Actual y on Test Set", x="Predicted y", y="Actual y")

summary(lm(y ~ predict, data=test |> mutate(predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), train), test))))## ## Call:## lm(formula = y ~ predict, data = mutate(test, predict = predict(lm(as.formula("y ~ I(x^2) + w + w:x"), ## train), test)))## ## Residuals:## Min 1Q Median 3Q Max ## -3.08711 -0.67088 0.01239 0.69015 2.86620 ## ## Coefficients:## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 0.02778 0.02527 1.099 0.272 ## predict 0.96577 0.02369 40.762 <2e-16 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## Residual standard error: 0.9884 on 1998 degrees of freedom## Multiple R-squared: 0.454,Adjusted R-squared: 0.4538 ## F-statistic: 1662 on 1 and 1998 DF, p-value: < 2.2e-16 There you have it! The RMSE on test set is quite similar to our average validation sets! The slope is now 0.97, much closer to 1, and the intercept is 0.02, much closer to 0. This confirms our theory that with more training data, the estimation error decreases, leading to better predictions.

Opportunities For Improvement * apply crossvalidation from scratch on hyperparameter tuning such as xgboost hyperparam, lasso/ridge/elasticnet/glmnet * learn about nested resampling that uses an additional layer of resampling that separates the tuning activities from the process used to estimate the efficacy of the model. see here * learn to code grouped and stratified cv from scratch * learn a bit more on time-series CV * need to try n_sim of 10000 and see if the bias and variance order holds up for n=100, since the simulation is quite slow, I didn’t run it to completion. Update: I did complete an n_sim of 10000 of n=100 and LOOCV still had the lowest variance.

Lessons learnt * Learnt that we have to use I to indicate that we want to include polynomial terms in the formula, otherwise R will interpret them as interaction terms. * Learnt deparse is useful to convert a formula object into a character string for logging purposes.

If you like this article:

please feel free to send me a comment or visit my other blogs
please feel free to follow me on BlueSky, twitter, GitHub or Mastodon
if you would like collaborate please feel free to contact me

To leave a comment for the author, please follow the link and comment on their blog: r on Everyday Is A School Day.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cross-Validation From Scratch and a Surprise at n=100

View Details

[This article was first published on R | r4stats.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Graphical user interfaces for the R language are easy to use and getting more powerful all the time. Here is my updated comparison of jamovi, JASP, BlueSky Statistics (free & Pro), Rattle, RKWard, R-Instat, R AnalyticFlow, and R Commander.

With so many detailed reviews of Graphical User Interfaces (GUIs) for R available, which should you choose? It’s not too difficult to rate them based on the number of features they offer, so I’ll start there. Then, I’ll follow with a brief overview of each.

I’m basing the counts on the number of dialog boxes in each category of the following categories:

Ease of Use
General Usability
Graphics
Analytics
Reproducibility

This data is trickier to collect than you might think. Some software has fewer menu choices, depending instead on more detailed dialog boxes. Studying every menu and dialog box is very time-consuming, but that is what I’ve tried to do to keep this comparison trustworthy.

Perhaps the biggest flaw in this methodology is that every feature adds only one point to each GUI’s total score. I encourage you to download the full dataset and consider which features are most important to you. If you decide to make your own graphs with a different weighting system, I’d love to hear from you in the comments below. Thomas Langkamp did an excellent comparison focused on teaching statistics. You can read it here.

Ease of UseFor ease of use, I’ve defined it primarily by how well each GUI meets its primary goal: avoiding code. They get one point for each of the following abilities, which include being able to install, start, and use the GUI to its maximum effect, including publication-quality output, without knowing anything about the R language itself. Figure 1 shows the result. R Commander is abbreviated Rcmdr, and R AnalyticFlow is abbreviated RAF. The commercial BlueSky Pro comes out on top by a slim margin, followed closely by JASP and RKWard. None of the GUIs achieved the highest possible score of 15, so there is room for improvement.

The ease of use features are:

Installs without the use of R
Starts without the use of R
Remembers recent files
Hides R code by default
Uses its full capability without needing R code
Data editor included
Pub-quality tables w/out R code steps
Simple menus that grow as needed
Table of Contents to ease navigation
Variable labels ease identification in the output
Easy to move blocks of output
Ease reading columns by freezing headers of long tables
Accepts data pasted from the clipboard
Easy to move header row of pasted data into the variable name field
Includes AI interface to ease interpretation of output

Figure 1. The number of ease-of-use features offered by each R GUI.General UsabilityThis category is dominated by data-wrangling capabilities, which data scientists and statisticians spend most of their time on. It also includes various types of data input and output. We see in Figure 2 that both BlueSky versions and R-Instat come out on top, not just due to their excellent selection of data-wrangling features but also for their use of the rio package for importing and exporting files. The rio package combines the import/export capabilities of many other packages, and it is easy to use. I expect the other GUIs will eventually adopt it, raising their scores by around 20 points.

The general usability features are:

Operating systems (how many)
Import data file types (how many)
Import from databases (how many)
Export data file types (how many)
Languages displayable in UI (how many, besides English)
Easy to repeat any step by groups (split-file)
Multiple data files open at once
Multiple output windows
Multiple code windows
Variable metadata view
Variable types (how many)
Variable search/filter in dialogs
Variable sort by name
Variable sort by type
Variable move manually
Model Builder (how many effect types)
Magnify GUI for teaching
R code editor
Comment/uncomment blocks of code
Package management (comes with R and all packages)
Output: word processing features
Output: R Markdown
Output: LaTeX
Data wrangling (how many)
Transform across many variables at once (e.g., row mean)
Transform down many variables at once (e.g., log, sqrt)
Assign factor labels across many variables at once
Project saves/loads data, dialogs, and notes in one file

Figure 2. The number of general usability features in each R GUI.GraphicsThis category consists mainly of the number of plot types each software offers. However, the other items can be very important to completing your work. They should add more than one point to the graphics score, but I scored them one point since some will view them as very important while others might not need them at all. Be sure to see the full reviews or download the Excel file if those features are important to you. Figure 3 shows the total graphics score for each GUI. jamovi has a solid lead in this category. R-Instat’s ability to layer any geometric object, or “geom,” on top of another could substantially increase its count, but I don’t include that since it requires knowledge of R code.

When studying all the graphs in this article, it’s important to consider the difference between the relative and absolute performance. For example, relatively speaking, R Commander is not doing well here, but it does offer over 25 types of plots! That might be fine for your needs.

The graphics features are:

Types of graphs (how many)
Small multiples (i.e., facets)
Large multiples (i.e., by group / split-file)
Export graphics formats (how many)
Interactive Plot Builder
Interactive Plot Editor (e.g., add annotations, arrows, etc.)
Able to combine any graphs into a single image

Continued…

The post R GUI Comparison Update first appeared on r4stats.com.To leave a comment for the author, please follow the link and comment on their blog: R | r4stats.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R GUI Comparison Update

View Details

[This article was first published on modTools, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The PROJ format for defining Coordinate Reference Systems (CRS) is becoming deprecated, as PROJ strings cannot accurately store modern datum transformations, coordinate epoch information, or complex axis order rules. However, that format is much easier and more practical for humans to type and read, and it is still often used as a legacy convenience. I, for one, use it often for centering a map projection on the mapped region, thus avoiding the distortion that occurs away from the projection center. For example, imagine I want to map New Zealand in a Lambert Azimuthal Equal-Area projection in R:

library(terra)library(geodata)nz <- gadm(country = "New Zealand", level = 0, path = tempdir())nz\_prj <- project(nz, "+proj=laea")plot(nz\_prj) The map is heavily distorted, because it is far away from the center of the projection, which is 0,0 by default. With the PROJ format, we can easily set the center wherever we want, e.g.:

nz\_centr <- crds(centroids(aggregate(nz)))nz\_centr # 172.6104 -41.82592nz\_prj <- project(nz, "+proj=laea +lat\_0=-41.8 +lon\_0=172.6")# (this can be further automated to avoid the hard-coding,# but I keep it like this here for simplicity) plot(nz\_prj) Much better! We now have an equal-area map of New Zealand without visibly distorting the shape, and without having to look for specific equal-area CRSs normally used over there (and again every time we need to map another region of the world). This works as long as we use the WGS84 ellipsoid rather than a local datum, as those are currently not supported when using PROJ. So, I’m still using this regularly, and apprehensive about PROJ being deprecated… If anyone knows of a non-PROJ way of easily defining user-centered projections like this, please let me know!

To leave a comment for the author, please follow the link and comment on their blog: modTools.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Centering a map projection on the mapped region

View Details

[This article was first published on R – JAGS News, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.View over Loch Muick, Balmoral Estate, ScotlandAfter an extended beta period, JAGS 5.0.0 is now available. The release is more of a process than a single event, but we have started by uploading the source tarball and the Windows installer to SourceForge. The macOS installer will be along hopefully next week.

Thanks to Jeroen Ooms, Windows binary installers are available from the r-windows project on GitHub. This includes binaries for aarch64/Arm64.

Changes since the beta release* Better multi-core support on macOS (More on this in a future post). * Fixed a delayed rejection bug in the MNormal sampler. * Added a new elliptical slice sampler contributed by Marcel Jonker. * The dic.samples function could throw a logic error (See ticket #66). I am unable to reproduce this but JAGS 5.0.0 includes a workaround that should avoid triggering this bug. * Fixed a reproducibility bug in BaseRNGFactory.

I would not normally add a new sampler at this late stage, but the elliptical slice sampler was ready to go and performs surprisingly well for a non-adaptive sampler. If you experience any trouble with this new sampler then you can turn it of with

set.factory("bugs::EllipticalSlice", type="sampler", state=FALSE) JAGS will then fall back on the adaptive random walk Metropolis-Hastings sampler MNormal.

The rjags package and CRAN migrationThe current rjags package on CRAN (version 4-17) will not work with JAGS 5.0.0. You need version 5-x but for the time being this will only be available from SourceForge.

Source tarball
Windows binary (x86_64)
macOS binary

The reason for not uploading the rjags package to CRAN is that it currently breaks a lot of reverse dependencies. I have been quite proactive in contacting the maintainers of these package and providing patches, but we still need to give them more time.

We are monitoring the reverse dependency situation and will be liaising with the CRAN maintainers.

Preview of upcoming postsIn order to help users and developers with the transition to JAGS version 5, a series of “What’s new in JAGS 5.0.0” blog posts are planned covering the following topics.

Initial values
Reproducibility
Multicore support
ABI compatibility
New diagnostics

These will include guidance for package maintainers to make the transition to JAGS 5.0.0.

To leave a comment for the author, please follow the link and comment on their blog: R – JAGS News.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: JAGS 5.0.0 is released

View Details

[This article was first published on Seascapemodels, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. I’ve put together a short guide: How Do I Build a Model? A starting guide for new modellers. It’s free, online, and aimed squarely at the graduate student who has a research question and is staring at a blank script wondering where to even begin.

You can read it online at How Do I Build a Model?, or download a pdf.

First sketch, one of the steps on the way to buliding a working modelI’ve never settled into one corner of ecological modelling. I’m not the Bayesian guy, or the foodweb model guy, or the machine learning guy, though I’ve dabbled in all three. That haphazard career has given me a broad view across a lot of different modelling traditions, and I keep noticing the same meta-principles showing up regardless of the discipline. This guide is my attempt to write those down.

It’s not a textbook on any particular method. Once you know what type of model you’re building, go read a book by a discipline expert (I’ve listed some favourites in the final chapter). This guide is for the step before that: turning a question into a model in the first place.

What’s in it1. Start with the question. How you actually grow into a research question, through reading, writing, and talking it through with people whose work you admire (and some whose work you don’t). 2. From a question to a type of model. Once you know your question, how to read papers differently, paying attention to the how and the why. 3. Terminology: variables, parameters, and the rest. A clean-up of vocabulary that gets used loosely and causes confusion. 4. Know why you’re building a model. Causal inference, prediction, or something else. Knowing your purpose is what narrows down the type of model you need. 5. Sketching your model. Why you should start at a whiteboard with boxes and arrows before you touch an equation, and how to read those arrows as equations later. 6. The different types of uncertainty. Why “how uncertain am I?” is really several different questions, and why that distinction matters for what you do next. 7. Writing your model’s equations. Turning the diagram into notation, with a nod to Edwards & Auger-Méthé’s guidance for ecologists, which I’d recommend reading in full regardless. 8. Understand your constraints. Computational, data, and time constraints, and how they shape what model you can realistically build. 9. The modelling workflow. Feedback loops of writing code, debugging, running, plotting, interpreting. 10. Tools and software. The endless R-versus-Python debate, and why your own familiarity with a tool is a constraint. 11. Tips for finding novel insights. Some of my favourite tricks, including hunting for analogues in other fields and pushing them to their extreme.

There’s also a short further-reading list at the end, covering the books I’d point you to once you’ve picked your modelling approach.

How it came togetherI dictated the original version on a long drive, Claude Code cleaned up the transcript into a first written draft and set up the Quarto book site, and I edited it from there.

To leave a comment for the author, please follow the link and comment on their blog: Seascapemodels.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How Do I Build a Model? A Starting Guide for New Modellers

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In their paper “A Synchronized Multi‑IMU Wearable System for Tracking of Joint‑Angles in Sports Motion Analysis” (arXiv:2607.26027v1), Samarasekera and colleagues set out to solve a very practical problem: how to reliably measure joint angles in dynamic sports movements using wearable IMUs. Their goal was to design a synchronized pipeline that could filter out noise, correct drift, and normalize ranges so that biomechanical signals could be trusted in real‑world conditions.

Our goal is different but conceptually parallel: to apply the same pipeline logic to financial time‑series data. Instead of knee flexion angles, we analyze the comparative dynamics of Gold and Bitcoin. The challenge is similar — financial returns are noisy, prone to drift, and difficult to compare directly. By adapting their pipeline, we aim to extract latent trends, mitigate drift, normalize ranges, and highlight relative divergence between assets.

The pipeline consists of four sequential stages:

Indirect Kalman Filtering (IKF) – extracting latent trends from noisy log returns.
- State equation:
Observation equation:
IKF separates signal from noise, with and learned via maximum likelihood.
Drift Mitigation via High‑Pass Filtering – removing low‑frequency bias.
- High‑pass operator:
This ensures long‑term drift does not dominate short‑term dynamics.
Range Normalization – mapping filtered values into a comparable scale.
- Normalization function:
This allows Gold and Bitcoin to be compared on the same 0–100 scale.
Relative Notation (Directional Divergence) – highlighting when assets move in opposite directions.
- Divergence indicator:
If true, shaded regions show divergence; if false, alignment.

Why This Matters For the biomechanics researchers, the pipeline meant trustworthy joint‑angle tracking. * For us, the same architecture means trustworthy comparative trend analysis* between financial assets. * In both cases, the pipeline’s strength lies in its modularity: IKF for latent signal extraction, high‑pass filtering for drift control, normalization for comparability, and relative notation for divergence detection.

✦ R Packages in Our PipelineTo make this pipeline reproducible and transparent, we rely on a set of R packages. Each package plays a specific role in the architecture:

tidyverse
- Purpose: Data manipulation and wrangling.
- Functions: mutate, rename, inner_join, drop_na.
- Why: Provides a clean grammar for transforming raw financial data into structured time‑series.
tidyquant
- Purpose: Financial data acquisition and transformation.
- Functions: tq_get (download asset prices), tq_transmute (compute log returns).
- Why: Bridges tidyverse with financial APIs, enabling reproducible market data pipelines.
KFAS
- Purpose: State‑space modeling and Kalman filtering.
- Functions: SSModel (define state‑space model), fitSSM (MLE parameter estimation), KFS (Kalman smoothing).
- Why: Implements the mathematical backbone of IKF, separating latent trends from noise.
zoo
- Purpose: Rolling window operations.
- Functions: rollmean.
- Why: Used for high‑pass filtering by subtracting rolling averages, mitigating drift.
scales
- Purpose: Normalization and rescaling.
- Functions: rescale.
- Why: Maps filtered trends into a 0–100 range, enabling comparability across assets.
ggbraid
- Purpose: Divergence visualization.
- Functions: geom_braid.
- Why: Highlights periods of divergence between Gold and Bitcoin with shaded regions.
ggplot2 (via tidyverse)
- Purpose: Visualization.
- Functions: geom_line, labs, theme_classic.
- Why: Provides the plotting framework for presenting normalized high‑pass Kalman trends.

library(tidyverse)library(tidyquant)library(KFAS)library(ggbraid)# --- Step 1: Log returns ---gold <- tq\_get("GC=F", from = "2024-01-01") %>% tq\_transmute(select = close, mutate\_fun = periodReturn, period = "daily", type = "log")btc <- tq\_get("BTC-USD", from = "2024-01-01") %>% tq\_transmute(select = close, mutate\_fun = periodReturn, period = "daily", type = "log")df <- gold %>% rename(ret\_gold = daily.returns) %>% inner\_join(btc %>% rename(ret\_btc = daily.returns), by="date") %>% drop\_na()# --- Step 2: IKF (Kalman filter) ---model\_gold <- SSModel(df$ret\_gold ~ SSMtrend(1, Q = list(NA)), H = NA)model\_btc <- SSModel(df$ret\_btc ~ SSMtrend(1, Q = list(NA)), H = NA)fit\_gold <- fitSSM(model\_gold, inits = c(log(var(df$ret\_gold)), log(var(df$ret\_gold))))fit\_btc <- fitSSM(model\_btc, inits = c(log(var(df$ret\_btc)), log(var(df$ret\_btc))))kf\_gold <- KFS(fit\_gold$model, smoothing = c("state"))kf\_btc <- KFS(fit\_btc$model, smoothing = c("state"))trend\_gold <- as.numeric(kf\_gold$alphahat)trend\_btc <- as.numeric(kf\_btc$alphahat)# --- Step 3: High-pass filter (remove low-frequency drift) ---hp\_gold <- trend\_gold - zoo::rollmean(trend\_gold, k=30, fill=NA, align="right")hp\_btc <- trend\_btc - zoo::rollmean(trend\_btc, k=30, fill=NA, align="right")# --- Step 4: Range normalization ---gold\_norm <- scales::rescale(hp\_gold, to=c(0,100))btc\_norm <- scales::rescale(hp\_btc, to=c(0,100))# --- Step 5: Relative notation (directional divergence) ---df\_norm <- df %>% mutate( gold\_norm = gold\_norm, btc\_norm = btc\_norm, sign\_gold = sign(ret\_gold), sign\_btc = sign(ret\_btc), divergence = sign\_gold != sign\_btc )# --- Plot ---ggplot(df\_norm, aes(x=date)) + geom\_line(aes(y=gold\_norm, color="Gold"), linewidth=1.2) + geom\_line(aes(y=btc\_norm, color="Bitcoin"), linewidth=1.2) + geom\_braid(aes(ymin=gold\_norm, ymax=btc\_norm, fill=divergence), alpha=0.3) + scale\_color\_manual(values=c("Gold"="#FFD700", "Bitcoin"="#1E3A8A"), name = "") + scale\_fill\_manual(values=c("TRUE"="#FF4C4C", "FALSE"="#87CEFA"), name = "Divergence") + scale\_y\_continuous(limits=c(0,100)) + labs(title="Gold vs Bitcoin: Kalman-Filtered High-Pass Trends", subtitle="Directional divergence highlighted with shaded regions", x="", y="Normalized High-Pass Kalman Trend") + theme\_classic(base\_family = "Roboto Slab") + theme( legend.position="bottom", legend.title=element\_text(size=12, face="bold"), legend.text=element\_text(size=10), plot.title=element\_text(size=16, face="bold"), plot.subtitle=element\_text(size=12), axis.title=element\_text(size=12, face="bold"), axis.text=element\_text(size=10), panel.grid.minor=element\_blank() ) ConclusionBy adapting the IKF pipeline from the sports biomechanics paper, we demonstrate how state‑space filtering, drift mitigation, and normalization can reveal comparative dynamics among financial assets. This modular approach is reproducible and extensible to other domains where latent trends and divergence matter.

To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Analyzing Financial Trends: Kalman Filtering for Gold vs Bitcoin

View Details

[This article was first published on Econometrics and Free Software, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

This is a post based on two other posts I wrote for LinkedIn. It has become quite long.

AI Won’t Save People From Work They Never Thought to AutomateI’m a fairly optimistic person, but when it comes to AI adoption and how it will be generating billions in increased GDP I’m more skeptical. Not that AI is not capable: on the contrary, I think that for all intents and purposes, even the models from 6 months ago are more than capable enough to perform a majority of white collar tasks, including programming. No, I think the issue is the same I’ve observed many times over my years on this God’s green earth, and that is that there are people who see computers as tools for doing tasks and people who see computers as systems that can be taught to do tasks.

Consider two people using Excel. Both have used it for ten years.

Jim receives a monthly report and manually copies data between sheets, formats cells, and updates numbers. He’s learned the workflow and can execute it efficiently. He’s essentially performing the task using a computer as he would with a pen and paper. He’s doing procedural work. That’s his job, and he’s good at it.

Bill, on the other hand, asks himself the question: “Why am I doing this every month? Is my job actually about moving data around? Or is it something else? What is my purpose?” Now this may seem overly philosophical when the topic at hand is moving data around in Excel to increase shareholder value, but nonetheless, these questions are the right ones to ask. So Bill starts seeing his computer not as a tool that simply replaced pen and paper, but as something he can teach to perform tasks.

Bill learns about formulas. Bill learns about pivot tables. Using the free tier of an LLM he makes it write macros. He then even creates styles for Word instead of changing the headers manually each time.

The difference is not technical ability. Both Jim and Bill can operate Excel and Word. But the difference is the mental model. Jim sees the computer as a digital version of pen and paper. It is a place where work happens. Bill sees the computer as a machine that can execute rules. It is a place where work can be transformed into a repeatable system.

Bill has an automation mindset. People with an automation mindset naturally ask different questions. Instead of:

“How do I complete this task?”

they ask:

“This fucking sucks and surely there’s a better way of doing it?”

Instead of:

“How do I make this report?”

they ask:

“How can this report generate itself and fuck off?”

Instead of:

“How do I process these 1000 files?”

they ask:

“How can I write a process that handles any number of files?”

The key shift is recognizing that anything you do is required to follow rules, and these rules are mostly implicit and invisible. It is moving from thinking about individual actions to thinking about the underlying rules that matters.

Computers are powerful because they are not just faster versions of humans. Their real advantage is that they can execute instructions that follow rules repeatedly, consistently, and at scale, but you gotta provide them with the rules.

The Real Barrier to AI Automation: Mindset, Tools, and CultureA common assumption is that AI will democratize automation because people no longer need to know how to program, which is certainly true (to some extent). AI dramatically lowers the technical barrier. Someone can now describe a workflow in natural language and get help creating scripts, automations, or processes.

But there is a more fundamental barrier: people have to imagine that automation is possible in the first place!

Someone who has never thought about programming will not suddenly become an automation architect just because they have access to an AI assistant.

Jim may use AI to write an email faster.
Jim may use AI to summarize a document.
Jim may use AI to answer a question.

Those are useful improvements.

But Bill is playing another game entirely. He’s thinking about implicit rules, and how to make them explicit (which, by the way, can also be done with AI!). So he’s rubber ducking with Claude and asking it:

“Why does this task exist at all?”
“Could this entire workflow disappear?”
“Could you (Claude) write a Python script/Excel Macro to run this every week?”

While certainly AI lowers the cost of building systems, it does not automatically teach people to think in systems.

It is not Jim’s fault though. This difference in mindset can even be seen in operating systems. Historically, Windows has emphasized graphical workflows. The philosophy is often: “There is a tool or button for every task. This Word document just looks like an actual sheet of paper, so treat it as such. This Excel workbook just looks like an actual ledger your grandpa used to use, so treat it as such.” Linux and Unix come from a different tradition: “Everything is a program. Programs can be combined. You pass data from program to program by streams of text. Text is the fundamental unit of information.”

A Linux user solving a repetitive problem often starts with a command, then turns that command into a script, then schedules that script to run automatically.

The progression is natural:

Manual action → command → script → automated system.

Windows has powerful automation capabilities too. PowerShell is an extremely capable scripting environment. But for too long, Windows’ main target audience was (and still is) the white collar office drone. Even though it is absolutely possible to automate tasks on Windows, the culture now is not one of automation. And technology alone does not create culture.

There are many such examples. Consider using a regular text editor like VS Code versus something like Vim. With regular text editors, you essentially use your computer as a fancy pen and paper. With Vim, you manipulate text and think about ranges and how to operate on them using macros. Very different mental models. In fact, they’re so different that trying to explain Vim to someone not familiar with it is practically impossible. People can only truly get it if they try it and keep using it for some time. And it’s the same thing with data analysis: using Excel vs R for example. The surrounding culture matters. If users grow up thinking:

“The computer is a place where I click through tasks,”

they will rarely ask:

“How can the computer do this task for me?”

The tool exists. The mindset does not.

Transforming Organizations: Rewarding Work EliminationMany companies are approaching AI transformation as a training problem: teach employees how to use AI tools. But the deeper challenge is a thinking problem: teach employees how to identify opportunities for automation. The most valuable AI employees will not necessarily be the people who write the cleverest prompts. They will be the people who notice:

“Everyone spends two hours every week doing this repetitive process.”
“This information is copied manually between systems.”
“This decision follows a predictable set of rules. These rules are not written down anywhere, but they exist.”
“This workflow exists only because nobody questioned it.”

Those people are the ones who will transform organizations, and organizations need to create an incentive structure where people are rewarded for eliminating work, not for performing more work. The irony is that many organizations say they want efficiency but reward visible effort. True AI transformation requires rewarding people who make themselves unnecessary.

The biggest advantage of AI will not come from people who use it as a faster assistant. It will come from people who use it as a way to redesign work. The question that separates the old way of working from the new one is simple:

“Why am I doing this manually at all?”

AI is powerful because it makes accessing automation easier than ever before. But the greatest gains will go to those who already understand the fundamental idea behind computers:

A computer is not just a tool for performing tasks!

It is a machine for turning ideas, rules, and processes into systems that run.

Jupyter Notebooks (and Interactive Data Analysis) Are Dead (one can hope)Now how do these observations translate to our work as data scientists?

I’ve spent years watching data science workflows evolve, and I keep coming back to the same conclusion:

Jupyter notebooks are making data science and statistics worse. I’d even go so far as to argue that interactive data analysis may be the wrong mental model altogether.

Let’s start with notebooks. If you’ve been reading my work these past years, you know I’m not fond of them.

They encourage messy, non-reproducible “spaghetti” code. They mix data, business logic, and results into a single file (something most of us would never accept in an Excel spreadsheet, yet somehow tolerate in an .ipynb). They’re also not plain text, which makes version control, code review, and collaboration unnecessarily painful, often requiring dedicated tooling just to work around the format.

Joel Grus said it best years ago in his classic I Don’t Like Notebooks talk (go watch it on youtube, it’s great). Very little has changed since then.

So why did notebooks become so dominant?

Because Python’s scientific computing tooling was simply that bad for a long time. Spyder wasn’t great. Installing Python packages was (and, to some extent, still is) an absolute clown show. uv has dramatically improved the experience, but it can’t solve the deeper structural issues of PyPI.

From the mid-2000s through the early 2010s, spinning up a Jupyter server with a curated set of pre-installed packages was often the only practical way for institutions to provide working Python environments to their users. Unlike R, which had RStudio, Python never had a truly first-class IDE built specifically for scientific computing. So a whole generation of data scientists grew up on Jupyter and now teach on Jupyter and force their juniors to use Jupyter. Big mistake! Even without AI, there’s no reason to keep using notebooks. Positron is excellent, and VS Code can be configured into a capable data science environment. But AI may end up making the entire IDE debate irrelevant (of course, us Emacs or Vim chads have solved this since decades).

AI agents thrive on plain text, which also happens to be the simplest and most enduring interface in computing, especially in the Unix/Linux world.

Jupyter notebooks, by contrast, are JSON documents rather than plain text (and the json format itself would actually not be an issue for agents, to be fair). But notebooks have many longstanding problems (hidden state, side effects, and non-linear execution) which are exactly the kinds of things that make life difficult for AI agents. AI systems struggle with implicit state and unpredictable execution. Tools and interfaces that deliberately minimize these issues (like plain-text scripts, explicit pipelines, immutable workflows) don’t just make life easier for humans; they make it easier for AI as well.

If AI becomes central to software development (which it will), I think we’ll see a resurgence of simple text-based workflows.

A notebook is essentially a recording of a conversation between a human and a computer:

Run cell.Inspect output.Modify.Run again.Oops.Restart kernel.Run all. It’s procedural and interactive, and it’s completely pointless when the coding is being done by an AI agent. Just think about it: imagine we’re back in 2010, and you hire some data science consultants to analyse some data. Do you care to see which tools they used for the analysis? You might be interested in the source code of the analysis itself, but why would you care if they used notebooks or plain text scripts? This is essentially the same here. People might say “well I care because I want to collaborate with AI”, and that may be true, but I’m willing to bet that we’re going to move more towards what makes AI efficient (plain text) and move away from what humans like (notebooks if you have shit taste). We start to see it already in the tooling that is being developed and offered to developers: we’ve started with AI agents embedded in text editors doing auto-complete to “harnesses” that edit the code for you and show you the diffs.

That brings me back to the idea of interactive data analysis. Increasingly, I think the workflow itself may be flawed.

To be clear, there is real value in exploring data: filtering, slicing, visualizing, trying different models, testing hypotheses, tweaking graphs. The problem is that this exploratory process usually produces a chaotic script that eventually has to be rewritten before it can be put into production. We accept this rewrite as part of the job, but ideally we shouldn’t have to.

AI changes that equation.

Humans can continue exploring data in messy, experimental code while AI agents continuously refactor that exploration into clean, modular, production-ready pipelines. Instead of treating exploration and production as two separate phases, we can have AI bridge the gap in real time.

Introducing T: Describing Systems, Constraints, and IntentThose two ideas (that plain text is still the best interface we have, and that human-agent collaboration will fundamentally reshape how we build software) are the foundation of the domain-specific language I’m working on, T.

Also, I’m betting that the history of computing will keep on repeating itself, and this history as always about raising the level of abstraction. In the 1980s and 1990s, everyone wanted programming to become more interactive. We got IDEs, GUIs, notebooks, drag-and-drop workflows, visual programming, WYSIWYG editors. The assumption was that making computers more like humans was the future. We went from describing exactly what the CPU should do to describing data transformations with modern functional programming.

With AI, we are increasingly focusing on describing systems, constraints, and intent.

We’re rediscovering that plain text, explicit representations, and declarative specifications are incredibly powerful: not because humans suddenly prefer them (some nerds do, see NixOS), but because both humans and AI can reason about them. Git can diff them. Compilers can analyze them. LLMs can transform them. They compose naturally.

So the future may look surprisingly like the past: not because we’ve failed to innovate, but because we’ve rediscovered that the oldest Unix philosophy was pointing in the right direction all along:

Everything should have an explicit, textual representation that can be composed into larger systems.

For example, instead of writing:

for ... ... you tell some AI-agent in voice mode:

Here’s my data. These transformations must happen. These invariants must hold. This pipeline should be reproducible.

Let’s go back to the intro: Jim thinks in tasks.

“I need to copy these numbers.”

Bill thinks in rules.

“These numbers are generated according to these rules.”

When using an AI, you need to go one step further and think about systems and design. Once the system or the design of your project is explicit, generating Python, R, SQL, Bash, DuckDB, Polars, Spark, or whatever becomes almost an implementation detail.

This is the main design principle behind T.

T is an experimental, reproducibility-by-design DSL for polyglot data science. It provides a functional, immutable language for constructing composable micropipelines: first-class, introspectable computation graphs that coordinate R, Python, Julia, Quarto, and Shell execution within a unified system. Pipelines in T are not configuration artifacts but executable program structures with explicit dataflow, typed nodes, and content-addressed outputs. I’ve talked about it already here.

The main idea is that you will not write any T code. An AI agent should do it. And to achieve this, everything humans can do with T (running code, running tests, looking at error messages, etc) can also be done by AI agents. Agents can run the code and get feedback in json format. Error messages can also be surfaced using json. There is a whole “agentic” way of using T that is first-class.

Built on Nix, T integrates declarative environment management and deterministic builds at the language level. Every node runs in its own hermetic sandbox. Reproducibility isn’t something you configure; it’s something you can’t opt out of. Data moves via Apache Arrow. The actual data analysis code is still written in R, Python or Julia. T “merely” owns the data boundaries and efficiently coordinates the different nodes written in different languages seamlessly. Models can be trained in R or Python and evaluated natively in T via PMML or ONNX. The language is strictly functional: no mutable state, errors as values, tidyverse-inspired verbs for exploration in the REPL, then promotion into production pipelines.

Its source is 100% AI-generated and thoroughly tested. Starting a T project will ship an AGENTS.md and tiered language references so LLMs get the exact context they need. Intent blocks let you embed the “why.” t check is milliseconds and checks the structural correctness of the pipeline; builds are cached and content-addressed.

If T succeeds, I don’t think it should be framed as “another programming language.” It should be framed as a language for describing data science pipelines. Check it out here or the website.

AI won’t eliminate the need to think computationally; it will make computational thinking more valuable than ever. The syntax is becoming cheaper. The implementation is becoming easier. What remains scarce is the ability to recognize systems where others only see tasks. That’s why I think the future of programming looks so much like its past: explicit representations, declarative descriptions, and composable systems remain the best way to communicate intent.

To leave a comment for the author, please follow the link and comment on their blog: Econometrics and Free Software.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Future of Programming Looks Like Its Past or Why I build T

View Details

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We have all seen colourful signposts with great cities and their distances – how far each is from this current standpoint.

And fundamental question is, how correct these distances are? Or even, one should ask, how honest a particular signpost is, regarding the current standpoint.

So let’s put this utterly most fundamental question to the test And since it is vacation time, you can grab the R code and check it yourself

Image stumbling upon the post and wanted to check if this post is really relevant for this location. It can be a photo from other side of the world, hanging in the lobby of your holiday hotel And this sure ain’t honest signpost

The R code iterates through each city, finds matching points (where all circles meet) and searches for the nearest location. It uses OpenStreetMap (short: OSM) to extract all the geolocations and looks for possible intersections of all points

geocode\_city <- function(city\_name) { url <- modify\_url( "https://nominatim.openstreetmap.org/search", query = list( q = city\_name, format = "json", limit = 1 ) ) resp <- tryCatch( GET(url, user\_agent("PointingSignFinder/1.0 (R script)")), error = function(e) { cat(" Error on endpoint") return(NULL) } ) if (is.null(resp) || http\_error(resp)) { cat(" Error on http\n") return(NULL) } result <- fromJSON(content(resp, as = "text", encoding = "UTF-8")) if (length(result) == 0) { cat("Error on result") return(NULL) } lat <- as.numeric(result$lat[1]) lon <- as.numeric(result$lon[1]) cat(sprintf(" found: %.4f°, %.4f°\n", lat, lon)) Sys.sleep(1.1) list(lat = lat, lon = lon, display\_name = result$display\_name[1])}sign\_location\_finder <- function(cities, distances, tolerance = 50, coarse\_res = 0.2, fine\_res = 0.02, nearby\_radius = 200, nearby\_min\_pop = 100000) { stopifnot(length(cities) == length(distances)) stopifnot(length(cities) >= 2) cat("Step 1: Geocoding cities\n") coords <- lapply(cities, geocode\_city) failed <- which(sapply(coords, is.null)) if (length(failed) > 0) { stop(sprintf("Location not founc: %s", paste(cities[failed], collapse = ", "))) } sign\_data <- data.frame( city = cities, lat = sapply(coords, `[[`, "lat"), lon = sapply(coords, `[[`, "lon"), dist\_km = distances )print(sign\_data[, c("city", "lat", "lon", "dist\_km")]) max\_dist\_deg <- max(sign\_data$dist\_km) / 111lat\_min <- min(sign\_data$lat) - max\_dist\_deg - 5lat\_max <- max(sign\_data$lat) + max\_dist\_deg + 5lon\_min <- min(sign\_data$lon) - max\_dist\_deg * 2 - 5lon\_max <- max(sign\_data$lon) + max\_dist\_deg * 2 + 5 lat\_min <- max(lat\_min, -85)lat\_max <- min(lat\_max, 85)lon\_min <- max(lon\_min, -180)lon\_max <- min(lon\_max, 180) cat(sprintf("\nStep 2: Search bounding box: lat [%.1f, %.1f], lon [%.1f, %.1f]\n", lat\_min, lat\_max, lon\_min, lon\_max)) score\_point <- function(plat, plon) { diffs <- sapply(seq\_len(nrow(sign\_data)), function(i) { d <- distHaversine( c(plon, plat), c(sign\_data$lon[i], sign\_data$lat[i]) ) / 1000 abs(d - sign\_data$dist\_km[i]) }) max(diffs) } cat(sprintf("\nStep 3: Search the grid (%.2f° resolution)...\n", coarse\_res)) grid <- expand.grid( lat = seq(lat\_min, lat\_max, by = coarse\_res), lon = seq(lon\_min, lon\_max, by = coarse\_res) ) cat(sprintf("Checking %d grid points...\n", nrow(grid))) grid$score <- mapply(score\_point, grid$lat, grid$lon) candidates <- grid[grid$score <= tolerance, ] cat(sprintf(" Found %d candidate cells within ±%d km tolerance.\n", nrow(candidates), tolerance)) coarse\_best <- grid[which.min(grid$score), ] fine\_grid <- expand.grid( lat = seq(coarse\_best$lat - 1, coarse\_best$lat + 1, by = fine\_res), lon = seq(coarse\_best$lon - 1, coarse\_best$lon + 1, by = fine\_res) ) fine\_grid$score <- mapply(score\_point, fine\_grid$lat, fine\_grid$lon) best <- fine\_grid[which.min(fine\_grid$score), ] cat(sprintf("\n>>> Estimated locations:\n")) cat(sprintf(" Latitude : %.4f°\n", best$lat)) cat(sprintf(" Longitude : %.4f°\n", best$lon)) cat(sprintf(" Max error : ±%.1f km\n", best$score)) cat(sprintf(" Google Maps: https://www.google.com/maps?q=%.4f,%.4f\n", best$lat, best$lon)) cat(sprintf("\nStep 5: Nearby cities (within %d km, pop > %s)...\n", nearby\_radius, format(nearby\_min\_pop, big.mark = ","))) world\_cities <- world.cities nearby <- world\_cities |> filter(pop > nearby\_min\_pop) |> mutate( dist\_to\_sign = distHaversine( cbind(long, lat), c(best$lon, best$lat) ) / 1000 ) |> filter(dist\_to\_sign <= nearby\_radius) |> arrange(dist\_to\_sign) |> select(name, country.etc, lat, long, pop, dist\_to\_sign) |> head(10) if (nrow(nearby) > 0) { cat("\n Cities found:\n") print(nearby, digits = 4) cat(sprintf("\n Nearest: %s, %s (%.1f km away)\n", nearby$name[1], nearby$country.etc[1], nearby$dist\_to\_sign[1])) } else { cat("No major locations found in vicinity!") } } So now, that we understand the useless problem, we can put this to the test

```

sample 2 - real with correct distances calculater from# https://www.distancefromto.net/ "air distance"# result must be Ljubljana, Sloveniaresult <- sign_location_finder( cities = c("Koper", "Celje", "Maribor", "Kranj"), distances = c(83, 61, 104, 24), tolerance = 20)

``` In this case I have entered the correct air distances and there should be a location present – which is capital of Slovenia – Ljubljana.

I have inserted the real values of air distances between Ljubljana all all four cities: Koper, Celje, Maribor, Kranj using air distance calculator.

And this where the circles should meet

So if you are enjoying your vacation and you stumble upon a sign, check it and let me know, if it corresponds to your location of not

As always, the complete code is available on GitHub in Useless_R_function repository and the file Honest_geographical_location_signposts.R is here.

Check the repository for future updates!

Stay healthy, hydrated and happy R-coding!

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Little useless-useful R functions – Honest geographical location signposts

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The package garchf wrapsthe extensive GARCH modeling machinery inrugarchbehind aforecast-styleinterface, so that GARCH-based forecasts can be produced, plotted, andevaluated with the same conventions used throughout the forecastecosystem (in the spirit of functions like forecast::thetaf()). Theworkhorse function, xgarchf(), fits a GARCH model to a univariate timeseries – with a choice of variance model (sGARCH, eGARCH,gjrGARCH, apARCH, iGARCH, …), ARMA order for the conditional mean,GARCH order, and conditional distribution – and returns a standardforecast object containing point forecasts, prediction intervals, andsimulated paths. Because the output integrates with forecast’s S3methods, plot() and other generics work out of the box, and thesimulated paths (sims) make it straightforward to compute richerprobabilistic forecasting metrics (CRPS, pinball loss, Winkler scores,etc.) via cross-validation. Below, we illustrate xgarchf() on Google’sdaily closing stock price returns, and then run a small cross-validationexercise comparing a few GARCH variants using thecrossvalidationpackage.

library(garchf)Registered S3 method overwritten by 'quantmod': method from as.zoo.data.frame zoo 1 – Graphs y <- diff(log(fpp2::goog200))fit1 <- xgarchf(y, h = 20, model = "eGARCH")fit2 <- xgarchf(y, h = 20, model = "sGARCH")fit3 <- xgarchf(y, h = 20, model = "gjrGARCH")fit4 <- xgarchf(y, h = 20, model = "iGARCH")print(summary(fitted(fit1))) Min. 1st Qu. Median Mean 3rd Qu. Max. 0.001504 0.001504 0.001504 0.001504 0.001504 0.001504 Min. 1st Qu. Median Mean 3rd Qu. Max. 0.002783 0.002783 0.002783 0.002783 0.002783 0.002783 Min. 1st Qu. Median Mean 3rd Qu. Max. 0.001233 0.001233 0.001233 0.001233 0.001233 0.001233 Min. 1st Qu. Median Mean 3rd Qu. Max. 0.001485 0.001485 0.001485 0.001485 0.001485 0.001485

Ljung-Box testdata: ResidualsQ* = 10.341, df = 10, p-value = 0.4111Model df: 0. Total lags used: 10

Ljung-Box testdata: ResidualsQ* = 10.341, df = 10, p-value = 0.4111Model df: 0. Total lags used: 10 2 - Cross-validation with the package crossvalidationFor crossvalidation, seehttps://github.com/Techtonique/crossvalidation.

spl\_m5 <- function(predicted, observed, probs = c(0.005, 0.025, 0.165, 0.25, 0.5, 0.75, 0.835, 0.975, 0.995)){ if (is.null(predicted$sims)) { stop("predicted$sims is required") } # training series from forecast object train <- as.numeric(predicted$x) # M5 scaling denominator scale\_denom <- mean(abs(diff(train)), na.rm = TRUE) if (scale\_denom <= 0 || !is.finite(scale\_denom)) { scale\_denom <- 1 } sims <- predicted$sims # quantile forecasts: rows = horizons, cols = probs qhat <- sapply( probs, function(p) apply(sims, 1, stats::quantile, probs = p, na.rm = TRUE) ) observed <- as.numeric(observed) # pinball loss pinball\_vec <- numeric(length(probs)) for (j in seq\_along(probs)) { u <- probs[j] q <- qhat[, j] pinball\_vec[j] <- mean( ifelse( observed >= q, u * (observed - q), (1 - u) * (q - observed) ), na.rm = TRUE ) } # scaled pinball loss for each quantile spl\_vec <- pinball\_vec / scale\_denom # average scaled pinball loss mean\_spl <- mean(spl\_vec) out <- c( mean\_SPL = mean\_spl ) names(spl\_vec) <- paste0("SPL\_", probs) c(out, spl\_vec)}eval\_metric95 <- function(predicted, observed){ error <- observed - predicted$mean rmse <- sqrt(mean(error^2)) mae <- mean(abs(error)) # Only one interval returned lower <- predicted$lower upper <- predicted$upper coverage <- mean( observed >= lower & observed <= upper ) alpha <- 0.05 winkler <- ifelse( observed < lower, (upper - lower) + (2 / alpha) * (lower - observed), ifelse( observed > upper, (upper - lower) + (2 / alpha) * (observed - upper), (upper - lower) ) ) spl <- spl\_m5(predicted, observed) c( RMSE = rmse, MAE = mae, Coverage95 = coverage, Winkler95 = mean(winkler), CRPS = mean(scoringRules::crps\_sample(observed, predicted$sims)), mean\_SPL = as.numeric(spl["mean\_SPL"]) )}eval\_metric80 <- function(predicted, observed){ error <- observed - predicted$mean me <- mean(error) rmse <- sqrt(mean(error^2)) mae <- mean(abs(error)) # Only one interval returned lower <- predicted$lower upper <- predicted$upper coverage <- mean( observed >= lower & observed <= upper ) alpha <- 0.2 winkler <- ifelse( observed < lower, (upper - lower) + (2 / alpha) * (lower - observed), ifelse( observed > upper, (upper - lower) + (2 / alpha) * (observed - upper), (upper - lower) ) ) spl <- spl\_m5(predicted, observed) c( ME = me, RMSE = rmse, MAE = mae, Coverage80 = coverage, Winkler80 = mean(winkler), CRPS = mean(scoringRules::crps\_sample(observed, predicted$sims)), mean\_SPL = as.numeric(spl["mean\_SPL"]) )}res <- crossvalidation::crossval\_ts( y = y, initial\_window = 150, horizon = 10, fixed\_window = FALSE, fcast\_func = garchf::xgarchf, eval\_metric = eval\_metric95, fit\_params = list(arma\_order=c(0, 0), garch\_order = c(1, 1), model = "eGARCH", level=95), show\_progress = FALSE)print(summary(res))res <- crossvalidation::crossval\_ts( y = y, initial\_window = 150, horizon = 10, fixed\_window = FALSE, fcast\_func = garchf::xgarchf, eval\_metric = eval\_metric95, fit\_params = list(arma\_order=c(0, 0), garch\_order = c(1, 1), model = "sGARCH", level=95), show\_progress = FALSE)print(summary(res))res <- crossvalidation::crossval\_ts( y = y, initial\_window = 150, horizon = 10, fixed\_window = FALSE, fcast\_func = garchf::xgarchf, eval\_metric = eval\_metric95, fit\_params = list(arma\_order=c(0, 0), garch\_order = c(1, 1), model = "iGARCH", level=95), show\_progress = FALSE)print(summary(res)) RMSE MAE Coverage95 Winkler95 Min. :0.006459 Min. :0.004896 Min. :0.7000 Min. :0.03441 1st Qu.:0.008001 1st Qu.:0.006266 1st Qu.:0.9000 1st Qu.:0.06153 Median :0.009135 Median :0.007061 Median :1.0000 Median :0.06466 Mean :0.017189 Mean :0.010613 Mean :0.9475 Mean :0.20090 3rd Qu.:0.022415 3rd Qu.:0.015066 3rd Qu.:1.0000 3rd Qu.:0.47706 Max. :0.042741 Max. :0.023054 Max. :1.0000 Max. :0.96625 CRPS mean\_SPL Min. :0.004154 Min. :0.1170 1st Qu.:0.004922 1st Qu.:0.1362 Median :0.005311 Median :0.1481 Mean :0.010039 Mean :0.3380 3rd Qu.:0.017800 3rd Qu.:0.6763 Max. :0.051416 Max. :1.6607 RMSE MAE Coverage95 Winkler95 Min. :0.006554 Min. :0.005380 Min. :0.7000 Min. :0.03856 1st Qu.:0.008424 1st Qu.:0.006561 1st Qu.:0.9750 1st Qu.:0.06261 Median :0.009314 Median :0.007401 Median :1.0000 Median :0.07277 Mean :0.017155 Mean :0.010741 Mean :0.9525 Mean :0.17716 3rd Qu.:0.020081 3rd Qu.:0.013012 3rd Qu.:1.0000 3rd Qu.:0.34988 Max. :0.042568 Max. :0.022999 Max. :1.0000 Max. :0.49756 CRPS mean\_SPL Min. :0.004634 Min. :0.1349 1st Qu.:0.005383 1st Qu.:0.1516 Median :0.005695 Median :0.1606 Mean :0.008842 Mean :0.2986 3rd Qu.:0.012903 3rd Qu.:0.4725 Max. :0.019442 Max. :0.7436 RMSE MAE Coverage95 Winkler95 Min. :0.006556 Min. :0.005514 Min. :0.7000 Min. :0.03580 1st Qu.:0.008306 1st Qu.:0.006239 1st Qu.:0.9750 1st Qu.:0.06307 Median :0.009119 Median :0.007007 Median :1.0000 Median :0.06478 Mean :0.017029 Mean :0.010484 Mean :0.9525 Mean :0.16725 3rd Qu.:0.018881 3rd Qu.:0.011711 3rd Qu.:1.0000 3rd Qu.:0.16760 Max. :0.042713 Max. :0.023064 Max. :1.0000 Max. :0.51546 CRPS mean\_SPL Min. :0.004851 Min. :0.1363 1st Qu.:0.005317 1st Qu.:0.1510 Median :0.005700 Median :0.1591 Mean :0.008720 Mean :0.2928 3rd Qu.:0.009316 3rd Qu.:0.3126 Max. :0.019649 Max. :0.7586 To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: ‘garchf’: GARCH probabilistic forecasting with package ‘forecast’-style interface (and ‘rugarch’ under the hood)

View Details

[This article was first published on Florian Teschner, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.LLMs can produce a convincing media recommendation in a few seconds. The more useful question is whether the recommendation is correct: are the reach calculations right, are the assumptions visible, and does the plan fit the brief?

I work at Havas Media and I am the author of the Havas AI Media Quality Index (HAI-Q), a benchmark for media planning in the German market. I built it because generic knowledge tests are not enough for professional AI applications. A model can write a plausible paragraph about media strategy and still get a budget allocation, a target group definition, or a GRP calculation wrong.

This post has three parts. First, I will build a small media-planning evaluation in R using the vitals package. Then I will discuss why media-planning questions are unusually difficult to evaluate. Finally, I will focus on what the HAI-Q results tell us about general-purpose LLMs.

A small eval in RAt its simplest, an LLM eval has three components:
A dataset containing questions and targets.
A solver that answers the questions.
A scorer that evaluates the answers.

This is the basic structure used by vitals. The target column does not always need to contain one exact answer. For open-ended questions, it can contain grading guidance: what a good answer must include, which assumptions must be stated, and which errors should lose credit.

This is useful for testing the final answer of an LLM. It is less useful as a complete test of an AI agent. An agent consists of a harness and one or more models: it may retrieve data, call tools, run calculations, retry a failed step, maintain state, and decide when to ask for help. vitals mainly gives us a dataset, a solver, and a scorer. It does not automatically tell us whether the agent used the right tool, passed the right arguments, or recovered safely from an intermediate failure. It can be extended with custom solvers, but then the agent harness and trajectory checks still have to be built separately.

The questionsHere is a deliberately small evaluation set. The questions cover calculation, planning, and interpretation. The targets are written as instructions for a grader rather than as model answers.

library(dplyr)library(tibble)media\_questions <- tribble( ~id, ~input, ~target, ~domain, ~task, "grp-calculation", "A campaign delivers 60% reach at an average frequency of 3. What is the campaign's GRP? State the assumption behind the calculation.", "A full-credit answer calculates 60 * 3 = 180 GRP. It explains that this is the standard simplified relationship GRP = reach (in percent) * average frequency. It must not describe GRP as the number of unique people reached.", "Reach and GRP", "Calculation", "budget-allocation", "A client has EUR 100,000 to reach adults aged 25-49 in Germany. There is no historical channel performance data. Propose a first-pass media allocation and explain how you would improve it after launch.", "There is no single correct allocation. A strong answer makes the uncertainty explicit, explains the role of each channel, avoids inventing CPMs or reach figures, and proposes a measurement and test-and-learn plan. It should distinguish a planning hypothesis from an observed result.", "Media strategy", "Recommendation", "reach-frequency-tradeoff", "Two plans have the same budget. Plan A has higher reach and lower frequency; Plan B has lower reach and higher frequency. How would you decide between them?", "A full-credit answer says that the choice depends on the communication objective, the selected channesls, the overall attention, audience size, purchase cycle, creative strength, and expected response. It should explain the reach-frequency trade-off and ask for missing information instead of declaring one plan universally better.", "Media strategy", "Reasoning")media\_questions The questions are intentionally specific. “Create a good media plan” is difficult to score because almost any fluent answer can sound reasonable. A question about GRP, in contrast, gives us a concrete numerical check. The budget question tests something different: whether the model knows when it does not have enough information to justify a precise allocation.

In a real eval, I would expand this table with questions about channel roles, target-group definitions, reach curves, media costs, campaign objectives, and German market data. I would also add metadata such as difficulty, source, market, and the date on which the underlying data was valid.

The solver and scorerThe vitals introduction uses Task$new() to combine the dataset, a solver, and a scorer. The following example uses Claude as the solver and an LLM-based grader. An API key for the selected provider is required.

library(vitals)library(ellmer)media\_task <- Task$new( dataset = media\_questions, solver = generate( chat\_openrouter(model = "claude-sonnet-4.6") ), scorer = model\_graded\_qa(partial\_credit = TRUE), name = "A small media-planning eval")# Run the solver and scorer.media\_task$eval()# Bind the results into a tibble for analysis.media\_scores <- vitals\_bind(media\_task)media\_scoresmedia\_scores |> count(score) With partial_credit = TRUE, the scorer can return C for correct, P for partially correct, or I for incorrect. This is more useful than a single binary score for media planning. A response may use the right calculation but fail to explain its assumptions. Another answer may be directionally useful but invent a media cost. Those are different failures and should not be hidden in one number.

The result is not the end of the evaluation. I would inspect the individual answers and the grading decisions, especially on the first few runs. If the grader repeatedly gives full credit to a response that I consider unsafe, the problem may be the grading guidance rather than the model.

Comparing modelsOnce the task exists, comparing solvers is straightforward. For example, the same questions can be evaluated with an OpenAI model and then combined with the first run.

media\_task\_openai <- media\_task$clone()media\_task\_openai$eval( solver\_chat = chat\_openrouter(model = "gpt-5.4"))comparison <- vitals\_bind( claude = media\_task, openai = media\_task\_openai) |> mutate(model = recode( task, claude = "Claude", openai = "OpenAI" ))comparison |> count(model, score) The exact model names will change over time. The important part is to keep the questions and scoring rules stable while changing one component at a time. Otherwise, a score difference is difficult to interpret.

This example is an LLM eval, not an agent eval. It compares the text returned at the end of the solver. For an agent, I would additionally log the tool calls, retrieved documents, intermediate calculations, retries, latency, cost, and final answer. The test should then assert properties of the whole trajectory, not only the final prose.

Why media-planning evals are hardMedia planning looks like a natural use case for LLMs. The work involves briefs, audience descriptions, channel recommendations, calculations, and written rationales. But the same combination makes the evaluation difficult.

There is rarely one correct planMany planning questions are underdetermined. A sensible allocation depends on the objective, budget, campaign length, audience size, creative assets, buying conditions, historical performance, and measurement setup. Two experienced planners can recommend different channel mixes and both be right.

This means that a target should often describe required properties rather than prescribe one number. For example, a full-credit answer might need to:

state its assumptions;
stay within the budget;
connect channel choices to the objective;
distinguish facts from hypotheses; and
explain how the plan would be measured and revised.

The evaluator has to reward sound reasoning without turning one planner’s preference into a false ground truth.

Media facts are local and time-dependentA media answer can be correct in one market and wrong in another. Reach, costs, inventory, audience definitions, platform availability, and panel methodology all vary by country and by date. A model that has learned general media vocabulary does not automatically have access to the latest German market data.

This is why a useful eval should record the market and the data vintage. It should also test whether the model provides a source or clearly labels an assumption when a current figure is unavailable. “The average CPM is X” is not a safe answer if the model cannot explain where X came from.

Definitions matter more than fluent proseMedia planning has many terms that are close enough to sound interchangeable but are not interchangeable in a calculation. Reach can mean gross or net reach. Frequency can be average or effective frequency. GRP is not the same as the number of unique people reached. Target-group percentages need a denominator.

These distinctions create easy failure modes. A response can contain the right formula but apply it to the wrong quantity. It can also produce a tidy table whose totals do not add up. Numerical checks and constraint checks therefore need to be part of the eval, not just a general assessment of writing quality.

LLM-as-a-judge is useful but not sufficientThe model_graded_qa() scorer in vitals makes open-ended evaluation scalable. It is also a model judging another model’s answer, which introduces its own errors. A grader may prefer confident language, overlook a subtle numerical mistake, or accept an answer because it resembles the supplied guidance.

For a serious benchmark, I would calibrate the grader against human-labelled examples and keep a sample for manual review. I would also separate different dimensions where possible:

factual accuracy;
numerical accuracy;
constraint satisfaction;
quality of reasoning; and
usefulness of the recommendation.

One overall score is convenient, but it can hide which part of the planning process is failing.

vitals is not an agent harnessThe distinction matters in practice. vitals is a useful way to ask an LLM a fixed set of questions and score the answers. It is not, by itself, a test harness for an AI agent. If an agent has access to a reach database, a calculator, or a planning API, the evaluation needs to check the interaction with those components as well.

For an end-to-end agent eval, I would test at least four layers: whether the harness selects the right tool, whether the tool call is valid, whether the intermediate result is interpreted correctly, and whether the final answer communicates uncertainty. A final response can look reasonable even when one of those earlier steps failed.

Results from the Havas HAI-QThe Havas HAI-Q page describes a benchmark built for media planning in the German market. It contains 35 practical tasks across five areas:
Media strategy
Channel and budget allocation
Target-group evaluation
Reach and GRP
Media and effectiveness metrics

In this post I focus on the general-purpose LLM results:

| Model | Correct answers | Share correct | | --- | --- | --- | | GPT-5 | 16 / 35 | 45.7% | | Claude Sonnet 4.5 | 6 / 35 | 17.1% | | GPT-4o | 4 / 35 | 11.4% |

The headline is uncomfortable: even GPT-5, the strongest general-purpose model in this comparison, is correct on fewer than half of the questions. It answers 16 of 35 questions correctly, leaving 19 answers that are not counted as correct. For practical media-planning work, that is not a safe error rate.

When I look at the individual HAI-Q questions, the errors are not evenly distributed. The models particularly struggle with numeric questions involving reach figures and audience sizes. They can produce a fluent explanation of a planning idea and still make a basic mistake in a denominator, a percentage, or the interpretation of a reach number. This is exactly the type of failure that is easy to miss if evaluation is based on plausibility rather than calculation.

There is also a clear improvement between the older and newer models. GPT-4o answers 4 of 35 questions correctly, compared with 16 of 35 for GPT-5. That is 12 additional correct answers and an improvement from 11.4% to 45.7%, or 34.3 percentage points. GPT-5 gets four times as many questions correct as GPT-4o on this benchmark. The improvement is substantial, even though the newer model is still not reliable enough for unsupervised media decisions.

Claude Sonnet 4.5 scores lower in this test, with 6 of 35 correct answers. This does not mean that the model is generally poor. It means that a general language benchmark and a domain-specific media-planning benchmark answer different questions. The HAI-Q is designed around the German market and a specific definition of correctness, including the numerical details that matter in planning.

The aggregate scores should still be read with care. They do not show the difficulty of every question, the variance across repeated runs, or the exact scoring decision for every answer. But the pattern is clear enough to be useful: current general-purpose LLMs are good at sounding like media planners, while still struggling to answer roughly half of these practical questions correctly.

TakeawaysAn LLM eval for media planning should start with real questions from the workflow, not generic prompts about marketing. In R, a tibble with input and target is enough to get started, and vitals provides a practical way to connect questions, solvers, scorers, and result inspection.

The difficult part is not running the model. It is defining what “correct” means when several plans can be reasonable, data changes by market and date, and a plausible answer can contain a subtle but consequential error. For agents, the evaluation also needs to cover the harness, tools, intermediate steps, and failure handling. A final answer score is not enough.

The HAI-Q results show that the latest general-purpose LLMs still answer more than half of these media-planning questions incorrectly. The largest weaknesses appear in numerical reasoning around reach and audience sizes. At the same time, the jump from GPT-4o to GPT-5 shows that model development is moving in the right direction.

For me, the practical conclusion is simple: use LLMs to accelerate media work, but test every numerical recommendation before relying on it. The next step is to combine model evals such as HAI-Q with end-to-end agent tests that measure tool use and the full planning workflow.

Resources* Getting started with vitals * vitals documentation * HAI-Q: Havas AI Media Quality Index * ellmer documentation

To leave a comment for the author, please follow the link and comment on their blog: Florian Teschner.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Evaluating LLMs/AI for Media Planning in R

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Like many people I have been keeping track of the slow-burn fuel crisis arising from the war in Iran, its possible expansion to the Red Sea, and exacerbation from the ongoing war between Russia and Ukraine. The latter seems to have reached a stage where serious damage is being done to Russia’s refining capacity and petro products’ exports.

I’ve now started a page on this website with a few charts on the fuel crisis that I’m going to be keeping up to date.

Stocks of crude are going down, but “how much is left”?The motivation for today’s post was that I’d been seeing many news and opinion articles about how the US crude oil stocks—in total, and in the Strategic Petroleum Reserve in particular—have declined to modern record lows. But expert commentators are reluctant to show this decline in relation to the absolute amount left, as opposed to comparisons to seasonal or historic lows. This is understandable because no-one really knows (or perhaps tells) what the real “tank bottoms” are in a big complex system like the USA. A certain amount of oil, of uncertain volume but certainly in the tens (and possibly hundreds) of millions of barrels, is needed just to keep various bits of plumbing or storage from seizing up or collapsing.

So it’s hard to get a chart like this one, and I had to make it myself:

I’ve got no reputation as an oil expert to worry about, so I can safely draw those lines at a fairly arbitrary 24 (or 30) days of refinery throughput, and calculate the time it will take to get to those low levels “at the current rate”. It’s simplistic, but it’s an important reality check to see that there might be about 19 weeks left.

Fast as the stocks are being used up, the USA has a lot of oil available to it. But 19 weeks isn’t that many. Markets are betting that good things will happen in the weeks or months before the blue line reaches the red line. Because of that, oil is currently selling for around $90 per barrel instead of the $110+ that would be justified if people thought we had lost the traffic through the Strait of Hormuz indefinitely.

Not that I think 19 weeks is a realistic forecast. Long before US stocks reached that red line, there would be economic and political chaos. Efforts to keep retail prices artificially low would probably be overwhelmed and there would be material demand destruction. There would also be huge incentives for the US to give away anything to restore Persian Gulf and Red Sea transit routes. 19 weeks gives us a time frame within which something will have to change.

The value-add for me from this chart is showing the total oil available for the US in the event this becomes a real emergency for them; the comparison of that to a meaningful if arbitrary threshold in terms of days of refinery cover; and the calculation of weeks left at current draw-down rates to reach that high stress threshold.

Break out the SPR and also look at gasoline, distillatesThe same data source has a lot of other detailed information, most of it far too detailed for me, but I’m interested enough in four key headline variables that are more detailed than “total stocks of crude oil”. In this chart I break down the total stocks into those in the Strategic Petroleum Reserve, and those in more normally accessed stocks. I also show the amounts held of two key refined products:

CodeHere’s the code that grabs the data from the US Energy Information Administration and draws that second plot:

library(tidyverse)library(readxl)library(scales)library(glue)fn <- "us-petrol-status-weekly.xls"download.file( "https://ir.eia.gov/wpsr/psw01.xls", fn, mode = "wb")us\_stocks <- read\_excel(fn, sheet = "Data 1", skip = 2) |> rename( crude = `Weekly U.S. Ending Stocks of Crude Oil (Thousand Barrels)`, crude\_spr = `Weekly U.S. Ending Stocks of Crude Oil in SPR (Thousand Barrels)`, gasoline = `Weekly U.S. Ending Stocks of Total Gasoline (Thousand Barrels)`, diesel = `Weekly U.S. Ending Stocks of Distillate Fuel Oil (Thousand Barrels)`, date = Date )#------------------plotting------------------eia\_caption <- glue("Source: Energy Information Administration (EIA). Accessed {format(Sys.Date(), '%d %B %Y')}")# an annotation rectangle we can use in multiple chartswar\_rect <- annotate( "rect", xmin = as.Date("2026-02-28"), xmax = Inf, ymin = -Inf, ymax = Inf, alpha = 0.5, fill = "grey80")# Facet plot of total crude, SPR crude, diesel and gasoline stocksus\_stocks |> mutate(com\_crude = crude - crude\_spr) |> select( date, `Non-SPR crude oil` = com\_crude, `Crude oil in SPR` = crude\_spr, `Gasoline` = gasoline, `Distillate (mostly diesel)` = diesel ) |> gather(variable, value, -date) |> mutate(variable = fct\_reorder(variable, value)) |> ggplot(aes(x = date, y = value / 1000)) + war\_rect + facet\_wrap(~variable, scales = "free\_y") + geom\_line(colour = "blue") + expand\_limits(y = 0) + scale\_y\_continuous(label = comma) + labs( x = "", y = "Thousands of barrels", title = "US stocks of crude oil, gasoline, and distillate fuel oil (effectively diesel)", subtitle = "Showing both total crude oil stocks (crude) and those in the Strategic Petroleum Reserve (crude\_spr)", caption = eia\_caption ) As I side point, I’ve started using air to format my code. Nearly always it comes out looking better and easier to read than my manual formatting, and it’s certainly more consistent. Thanks to D. Vaughan and L. Henry and the tidyverse.org project.

And here’s the code to draw my highly polished main plot, the one of total crude oil stocks with the annotations. I’ve tried to do this in a way that it will keep working in the months ahead with minimal maintenance, we’ll see how that goes.

refinery\_throughput <- 17.3 # as at 24 July, operating at 97% of US capacity. Daily refinery use.plausible\_stress <- 30 * refinery\_throughputplausible\_high\_stress <- 24 * refinery\_throughputcrude\_growth\_summary <- us\_stocks |> summarise( latest\_crude = crude[date == max(date)], weeks = as.numeric(as.Date(max(date)) - as.Date("2026-04-03")) / 7, difference = (latest\_crude - crude[date == as.Date("2026-04-03")]) / 1000, ratio = latest\_crude / crude[date == as.Date("2026-04-03")], # growth/decline per week in million barrels e.g. 10m barrels per week: difference\_rate = difference / weeks, # growth/declien rate per week: growth\_rate = 1 - exp(log(ratio) / weeks) ) |> mutate( weeks\_at\_this\_rate = (plausible\_stress - latest\_crude / 1000) / difference\_rate )plot\_us\_stocks <- function(min\_date = "2020-01-01", lab\_x\_diff = NULL) { if (!"Date" %in% class(min\_date)) { min\_date <- as.Date(min\_date) } if (is.null(lab\_x\_diff)) { days\_shown <- as.numeric(as.Date(max(us\_stocks$date))) - as.numeric(min\_date) lab\_x\_diff <- days\_shown / 50 } lv <- tail(us\_stocks, 1)$crude / 1000 p2 <- us\_stocks |> filter(date >= min\_date) |> ggplot(aes(x = date, y = crude / 1000)) + war\_rect + geom\_hline(yintercept = plausible\_stress, colour = "darkred") + geom\_hline(yintercept = plausible\_high\_stress, colour = "red") + annotate( "text", x = min\_date + lab\_x\_diff, y = plausible\_stress + 50, label = "Illustrative stress threshold:\n30 days of refinery cover", colour = "darkred", hjust = 0 ) + annotate( "text", x = min\_date + lab\_x\_diff, y = plausible\_high\_stress - 30, label = "Illustrative high stress threshold:\n24 days of refinery cover", colour = "red", hjust = 0, vjust = 1 ) + annotate( "text", x = max(us\_stocks$date) - 5e6, y = lv, label = glue( "{round(lv)} million barrels;\n{round(lv / refinery\_throughput)} days of cover" ), size = 2.9, hjust = 1, vjust = 1, colour = "blue" ) + geom\_line(colour = "blue") + expand\_limits(y = 0) + scale\_y\_continuous( label = comma, sec.axis = sec\_axis( ~ . / refinery\_throughput, name = "Days of refinery throughput" ) ) + labs( x = "", y = "Millions of barrels", title = "U.S. Total Crude Oil Stocks, including Strategic Petroleum Reserve", subtitle = glue( "Comparison of existing inventories with refinery throughput as at July 2026.Decline since peak on 3 April 2026 is at {abs(round(crude\_growth\_summary$difference\_rate, 1))} million barrels per week; {round(crude\_growth\_summary$weeks\_at\_this\_rate)} weeks from stress threshold at this (hypothetical and linear) rate." ), caption = eia\_caption ) + theme(axis.line.y = element\_line(colour = "grey50")) svglite( glue("all-us-crude-from{min\_date}.svg"), width = 10, height = 7 ) print(p2) dev.off()}plot\_us\_stocks("2000-01-01")plot\_us\_stocks("2020-01-01")plot\_us\_stocks("2025-01-01") Because I did this plot in the form of a function, I have a couple of variants of the plot. One starting in 2020:

And the other zoomed right in to just January 2025 and onwards:

On the whole I prefer the plot I started with, showing the situation from 2000 and onwards. I’m trying to get some big picture perspective after all.

That’s all for today. The images on this page will stay the same as at the time of writing; but those on the fuel crisis monitoring page will be updated each week.

To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: US oil stocks by @ellis2013nz

View Details

[This article was first published on Joe Roe, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.era v0.6.0 is now available on CRAN:

install.packages("era")library(era) This minor release adds functions for chronological comparison of year vectors:

x <- yr(c(200, 100, 300), "BCE")yr\_earlier\_than(x, yr(150, "BCE"))#> # BCE years <yr\_lgl[3]>:#> [1] FALSE FALSE TRUE#> # Era: Before Common Era (BCE): Gregorian years (365.2425 days), counted backwards from 1yr\_later\_than(x, yr(150, "BCE"))#> # BCE years <yr\_lgl[3]>:#> [1] TRUE FALSE FALSE#> # Era: Before Common Era (BCE): Gregorian years (365.2425 days), counted backwards from 1 And for calculating chronological differences between year vectors:

yr\_difference(yr(300, "BCE"), yr(100, "BCE"))#> # years <yr[1]>:#> [1] 200#> # Era: Gregorian years (365.2425 days) These functions are aware of era directionality: “BCE” years count backwards, so 300 BCE is earlier than 100 BCE. They support work on tempo, a new package I am working on that represents chronological intervals and the temporal relations between them.

Links* era on CRAN * era package documentation * Introductory vignette * Source code (GitHub)

To leave a comment for the author, please follow the link and comment on their blog: Joe Roe.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: era 0.6.0: chronological comparison and difference

View Details

[This article was first published on https://pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The goal of spuriouscorrelations is to keep alive the amazing examples from Tyler Vigen. Unfortunately, as of 2023-10-09, the website is down as my students noticed. Therefore, I decided to use the snapshot from the Internet Wayback Machine to save the datasets from 2023-06-07.

InstallationYou can install the CRAN version of spuriouscorrelations with:

install.packages("spuriouscorrelations") You can install the development version of spuriouscorrelations with:

remotes::install\_github("pachadotdev/spuriouscorrelations") ExampleThe package covers multiple examples for different spurious (and curious) correlations.

Here is a basic example which shows you how to plot a spurious correlation for the variables

x: Number of people who drowned by falling into a pool
y: Films Nicolas Cage appeared in

library(spuriouscorrelations)library(tinyplot)pool\_drownings year x y1 1999 109 22 2000 102 23 2001 102 24 2002 98 35 2003 85 16 2004 95 17 2005 96 28 2006 98 39 2007 123 410 2008 94 111 2009 102 4cor(pool\_drownings$x, pool\_drownings$y)[1] 0.6660043tinyplot( y ~ x, data = pool\_drownings, main = sprintf("Correlation %s", round(cor(pool\_drownings$x, pool\_drownings$y), 3)), xlab = "Number of people who drowned by falling into a pool", ylab = "Films Nicolas Cage appeared in") Converting the data to long format simplifies plotting both variables per year:

pool\_drownings\_2 <- reshape( pool\_drownings, varying = c("x", "y"), # columns to collapse v.names = "value", # name of the new value column timevar = "variable", # name of the new ID column times = c("x", "y"), # values to populate the ID column direction = "long" # target format)pool\_drownings\_2 year variable value id1.x 1999 x 109 12.x 2000 x 102 23.x 2001 x 102 34.x 2002 x 98 45.x 2003 x 85 56.x 2004 x 95 67.x 2005 x 96 78.x 2006 x 98 89.x 2007 x 123 910.x 2008 x 94 1011.x 2009 x 102 111.y 1999 y 2 12.y 2000 y 2 23.y 2001 y 2 34.y 2002 y 3 45.y 2003 y 1 56.y 2004 y 1 67.y 2005 y 2 78.y 2006 y 3 89.y 2007 y 4 910.y 2008 y 1 1011.y 2009 y 4 11tinyplot( value ~ year | variable, # "|" indicates the groping variable for the legend data = pool\_drownings\_2, main = sprintf("Correlation %s", round(cor(pool\_drownings$x, pool\_drownings$y), 3)), xlab = "Year", ylab = "Pooled observations", pch = 19 # solid dot shape) How about standarzing the variables to avoid the different scale visibility issue?

pool\_drownings\_3 <- pool\_drowningspool\_drownings\_3$x <- (pool\_drownings\_3$x - mean(pool\_drownings\_3$x)) / sd(pool\_drownings\_3$x)pool\_drownings\_3$y <- (pool\_drownings\_3$y - mean(pool\_drownings\_3$y)) / sd(pool\_drownings\_3$y)pool\_drownings\_3 <- reshape( pool\_drownings\_3, varying = c("x", "y"), # columns to collapse v.names = "value", # name of the new value column timevar = "variable", # name of the new ID column times = c("x", "y"), # values to populate the ID column direction = "long" # target format)pool\_drownings\_3 year variable value id1.x 1999 x 0.8952867 12.x 2000 x 0.1696333 23.x 2001 x 0.1696333 34.x 2002 x -0.2450258 45.x 2003 x -1.5926679 56.x 2004 x -0.5560201 67.x 2005 x -0.4523554 78.x 2006 x -0.2450258 89.x 2007 x 2.3465935 910.x 2008 x -0.6596849 1011.x 2009 x 0.1696333 111.y 1999 y -0.2470999 12.y 2000 y -0.2470999 23.y 2001 y -0.2470999 34.y 2002 y 0.6589330 45.y 2003 y -1.1531327 56.y 2004 y -1.1531327 67.y 2005 y -0.2470999 78.y 2006 y 0.6589330 89.y 2007 y 1.5649658 910.y 2008 y -1.1531327 1011.y 2009 y 1.5649658 11tinyplot( value ~ year | variable, # "|" indicates the groping variable for the legend data = pool\_drownings\_3, main = sprintf("Correlation %s", round(cor(pool\_drownings$x, pool\_drownings$y), 3)), xlab = "Year", ylab = "Pooled standardized observations", pch = 19, # solid dot shape type = "b" # add line to see the trend clearly) To leave a comment for the author, please follow the link and comment on their blog: https://pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Spurious correlations 0.2 is on CRAN!

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Introduction to All Things About bpvars, the R package for Forecasting with Bayesian Panel Vector Autoregressions, which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: All Things About bpvars, the R package for Forecasting with Bayesian Panel Vector Autoregressions

Date: Thursday, September 10th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Tomasz Woźniak is a Bayesian econometrician developing new econometric methods for applied macroeconomic research. He has been a specialised R user for eighteen years, and has recently joined The R Journal as an Associate Editor. He is the author of several R packages, available at https://bsvars.org/, that combine blazingly fast algorithms written in C++ with the convenience of data analysis in R. He works as a senior lecturer at the University of Melbourne, where he has an extensive research, teaching, and engagement portfolio. Tomasz supports Ukraine. Slava Ukrainie!

Description: This session is a unique opportunity to learn to use the bpvars package for forecasting with Bayesian panel vector autoregressions in a wide range of applications spanning:

data preparation,
specification of panel vector autoregressions,
estimation of the models,
estimation with missing observations,
forecasting labour market outcomes globally,
plotting and reporting predictions of different qualities, including point and density forecasts, marginal, conditional, or restricted forecasts,
performing recursive expanding window forecasting,
generating forecasting performance reports.

A sequence of hands-on exercises that fully prepares attendees to work with the package supports all of this. You are welcome to browse existing resources, including:

bpvars website: https://bsvars.org/bpvars/
bpvars CRAN profile: https://cran.r-project.org/package=bpvars
package vignette: https://doi.org/10.48550/arXiv.2606.14143
other materials: https://bsvars.org/bpvars/#resources

See you soon!

Preparation:

This session assumes the attendees know the basics of time series analysis. Install and load the bpvars package, then check the documentation by running the code below. Run the example you will find there in R. If it works, you’re ready to go. Thanks!

“`

install.packages(“bpvars”)

library(bpvars)

?bpvars

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

Please note that the registration confirmation is sent 1 day before the workshop to all registered participants rather than immediately after registration

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

ntroduction to All Things About bpvars, the R package for Forecasting with Bayesian Panel Vector Autoregressions workshop was first posted on July 29, 2026 at 12:43 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: ntroduction to All Things About bpvars, the R package for Forecasting with Bayesian Panel Vector Autoregressions workshop

View Details

[This article was first published on Seascapemodels, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. I’m a big fan of Github Copilots ghost text and next edit suggestions. As a data analyst they give you much more control over the pace and direction of analysis code than a full AI agent. They also let you practice the core skill an agent needs from you: writing a clear specification.

I’ll walk through it in R, but the same ideas apply in any language. I’m assuming you’re on VScode with the Copilot extension installed. This should work with the free or paid plans (though I haven’t checked free myself).

Start with ghost textGhost text is the greyed-out completion that appears as you type. Say you have a long-format dataset with timeseries for three species:

library(dplyr)library(ggplot2)dat <- data.frame( time = rep(1:10, 3), species = rep(c("A", "B", "C"), each = 10), value = c(rnorm(10, mean = 5), rnorm(10, mean = 10), rnorm(10, mean = 15))) You want to filter to one species, fit a linear model to its timeseries, pull out the slope, and make a labelled plot. Start typing the code for a single species and ghost text fills in the rest of the line. Here I’ve typed m1 <- lm(value ~ and Copilot offers the completion in transparent text (note change in image after the ~):

Press tab to accept. Working one line at a time, you end up with the code for species A:

datA <- dat |> filter(species == "A")m1 <- lm(value ~ time, data = datA)coef(m1)["time"]ggplot(datA, aes(x = time, y = value)) + geom\_point() + geom\_smooth(method = "lm", se = FALSE) + labs( title = "Species A Time Series", x = "Time", y = "Value" ) Notice we’re not trying to be general yet. We’re developing our ideas for a single case, species A. We know we’ll want to generalise later, but we’re not worrying about it now. This is the same discipline that makes agents work well: get one concrete case right first, then automate it.

Turn on next edit suggestionsNext edit suggestions go a step further than ghost text. Instead of completing the line you’re on, Copilot predicts the next change you’ll want to make elsewhere and points you to it.

Click the octocat icon in the bottom right of the VScode window and turn on Next Edit Suggestions. I recommend leaving this off most of the time — it’s distracting when it suggests edits you don’t want — and switching it on for jobs like this one.

Now click above the species A code and start typing the name of a function to wrap it in. As soon as I write the fit_fun <- function(species_name) header, Copilot spots that the hardcoded "A" needs to change, and flags it with an arrow in the gutter:

Press tab and it walks you through the edits needed to generalise. It replaces species == "A" with species == species_name, and further down it rewrites the plot title from the literal "Species A Time Series" to a paste() call that builds the title from species_name:

Tab through each suggestion and you land on a working, generalised function:

fit\_fun <- function(species\_name) { datA <- dat |> filter(species == species\_name) m1 <- lm(value ~ time, data = datA) coef(m1)["time"] ggplot(datA, aes(x = time, y = value)) + geom\_point() + geom\_smooth(method = "lm", se = FALSE) + labs( title = paste("Species", species\_name, "Time Series"), x = "Time", y = "Value" )}fit\_fun("A") The single-species code was the specification and then Copilot did the mechanical work of generalising it. This is a good scaffold for thinking about agentic programming, where you write a clear spec and the agent does the automation.

The big difference from a true agent is that ghost text and next edit don’t run your R code and iterate to fix errors. They just predict edits. But that makes them a safe place to build the skills.

Drive it with commentsYou can run the same next-edit approach with text only. Write out a recipe for what you want as comments, then start typing under the first step. Ghost text and next edit take over and help you write the rest.

Here’s a simple recipe:

```

Simulate a new dataset of abundance at x-y coordinates# plot a 2D map# fit a model with interaction between x and y

``` With just the comments in place and the cursor after the simulation, Copilot reads the recipe ahead of it:

Then suggests a next edit suggestion after the following comment:

Keep going and it writes the model for the final step, picking up the interaction from the comment # fit a model with interaction between x and y:

The finished block writes itself from the recipe:

A few tips. I set Copilot’s eagerness to High so suggestions come up quickly. Sometimes you need to type the first few characters of a line to kickstart the ghost text.

The clearer and more specific your comments, the closer the suggestions will be to what you actually wanted.

To leave a comment for the author, please follow the link and comment on their blog: Seascapemodels.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Using ghost text and next edit suggestions to learn agentic coding

View Details

[This article was first published on R Works, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Four hundred twelve new packages were submitted to CRAN in June. Here are my Top 40 picks in nineteen categories: Bioarchaeology, Biology, Climate Studies, Computational Methods, Ecology, Epidemiology, Finance, Functional Data Analysis, Machine Learning, Medical Statistics, Networks, Pharmacokinetics, Probability, Programming, Psychometrics, Risk Analysis, Statistics, Time Series, and Utilities.

BioarchaeologybaytaAAR v1.0.3: Provides Bayesian age estimation for bioarchaeological skeletal data using ordinal probit regression models implemented in JAGS and NIMBLE. The package is designed to handle multiple ordinal traits of adult individuals and incorporates a Gompertz prior on age to reflect population-level mortality. It accounts for estimation uncertainties and supports full customization of model parameters and Markov Chain Monte Carlo settings. For more details, see Müller-Scheeßel et al. (2026). There are five vignettes including Introduction and Mathematical background.

Biologypower.nb v0.1.0: Provides functions for estimating statistical power and required sample sizes in differential abundance microbiome studies using negative binomial models and includes tools for simulation-based power analysis and sample size estimation using generalized additive models (GAMs), and visualization utilities for exploring the relationship between power, effect size, abundance, and sample size. The methods are based on Agronah and Bolker (2025). See the vignette.

Climate Studiesclim4health v0.1.0: Provides functions to obtain, transform and export climate data, including reanalyses, seasonal forecasts and hindcasts, and weather stations for their use in epidemiological analyses. Features include downscaling, verification, spatiotemporal aggregation and threshold-based indicators. See Duzenli et al. (2026) for downscaling methods and Manubens et al. (2018) for verification methods. There are six vignettes, including Introduction and Overview.

Computational Methodsmomst v0.1.1: Provides functions to solve the Multi-Criteria Minimum Spanning Tree problem on complete weighted graphs by combining the Non-dominated Sorting Genetic Algorithm II with optional Pareto local search operators. Chromosomes are represented as Prufer sequences so that every random individual decodes to a valid spanning tree (Cayley’s theorem), avoiding repair operators. Four solver variants are provided: NSGA-II, Path Relinking, Pareto Local Search, and Tabu Search. See Parraga-Alava et al. (2017) for background. There are two vignettes: Getting Started and Comparing the Four MO-MST Variants.

nmathopencl v0.8.3: Ships statistical and mathematical routines from the R internal nmath (Mathlib) as OpenCL C sources under directory inst/cl/, with R wrappers. Uses the GPU when OpenCL is available at compile time and falls back to stats equivalents otherwise. Aimed at package developers building custom kernels (for example Bayesian GLMs via suggested package glmbayes) using opencltools kernel loaders and related helpers. There are thirteen vignettes, including Package Overview and Case study.

sparsediff v0.4.0: Implements bindings for the SparseDiffEngine C library, the sparse Jacobian and Hessian differentiation backend used by CVXPY for its Disciplined Nonlinear Programming extension. Provides low-level routines for building nonlinear expression graphs and evaluating sparse derivatives, intended as a backend for higher-level modeling layers such as CVXR. This is the R analog of the sparsediffpy Python package and wraps the same C library. See the vignette.

StochSimR v1.1.0: Implements a modular simulation engine for a wide range of stochastic processes. Provides exact and approximate simulation methods for Poisson processes, Brownian motion, discrete- and continuous-time Markov chains, birth-death processes, the Yule pure-birth process, infinitesimal generator matrix utilities, Markovian queuing systems with exact steady-state statistics, Levy processes, Merton jump-diffusion models, Hawkes self-exciting processes, geometric Brownian motion, and Ornstein-Uhlenbeck mean-reverting diffusions. See Glasserman (2003) and Asmussen & Glynn (2007) for background and the vignette for an introduction.

Uno v2.7.4: Provides bindings to Uno (Unifying Nonlinear Optimization), a C++ solver for smooth nonlinearly constrained optimization that unifies Lagrange-Newton methods, including sequential quadratic programming and interior-point methods, by decomposing them into interacting building blocks (constraint-relaxation, inequality-handling, Hessian, and globalization strategies). The framework is described in Vanaret and Leyffer (2024). See the vignette for an example.

EcologyBayesFR v1.0.1: Enables fitting various functional response models for single- and multi-prey experiments by providing nonlinear prediction functions for brms and provides a framework for easily testing hypotheses on trophic interactions. Models can incorporate covariates such as temperature gradients, experimental treatment variables, or random effects that account for grouping in experimental units. See Rosenbaum and Rall (2018) and Rosenbaum et al. (2024) for background and the vignette to get started.

nicheR v0.1.0: Provides tools to construct and define virtual ecological niches using ellipsoid geometries. It enables the identification and extraction of suitable environmental areas, simulation of species occurrence points with various sampling strategies, and visualization of niche boundaries and simulated occurrences in both environmental and geographic space. See Etherington et al. (2009) and Qiao et al. (2015) for background. There are six vignettes, including Visualizing ellipsoids in environmental space and Virtual community simulation.

spacc v0.8.3: Implements kNN and kNCN sampling methods with a C++ backend to compute spatial species accumulation curves. Supports Hill numbers, beta diversity partitioning, coverage-based rarefaction and extrapolation, phylogenetic diversity (Faith’s PD, mean pairwise distance, mean nearest taxon distance), functional diversity accumulation, diversity-area relationships, endemism-area curves, sampling-effort correction and fragmentation analysis, and species-area relationship models based on extreme value theory. See Chao et al. (2014) and Baselga (2010) for background. There are seven vignettes, including Getting Started and Diversity Accumulation.

TemporalModelR v0.3.0: Provides functions to assist with three major steps for building temporally-explicit ecological niche and species distribution models: (i) preprocessing species and environmental data, (ii) building a niche model and generating temporally-explicit predictions, and (iii) model postprocessing to explore spatiotemporal trends. See Ingenloff and Peterson (2021 and Blonder (2018) for the methodological and theoretical foundations. Modeling with a GLM and Modeling with a Random Forest.

EpidemiologySmokingHistoryGenerator v7.0.0: Implements an interface to the Cancer Intervention and Surveillance Modeling Network (CISNET) Smoking History Generator microsimulation engine, which synthesizes individual smoking histories (initiation, cessation, intensity) and ages at death from calibrated initiation, cessation, cigarettes-per-day, and mortality tables. See Jeon et al. (2012) for background and look here to get started.

FinanceCamelRatiosIndex v1.0.0: Computes a composite year-on-year index for bank performance assessment using the CAMEL framework (Capital Adequacy, Asset Quality, Management Efficiency, Earnings, Liquidity). The multivariate weighting scheme employs factor analysis with robust covariance estimation to derive communality-based weights from the correlation matrix of CAMEL ratios. Provides functions for index computation, visualization, and comparison across banks and time periods. The methodology is described in Ayimah et al. (2023a) and Ayimah et al. (2023b). See the vignette for an introduction.

JumpDiffSim v0.1.0: Implements the Merton (1976) and Kou (2002) jump-diffusion models through a unified S4 object-oriented interface. Provides exact compound-Poisson asset price simulation, maximum likelihood parameter estimation with Hessian-based standard errors, Wald-type confidence intervals, European option pricing via the Merton analytic series expansion, and publication-quality diagnostic plots. All functionality operates entirely offline without market data dependencies. See the vignette.

Functional Data Analysisfda.vi v1.0.0: Implements a variational Expectation-Maximization algorithm for smoothing one or multiple functional observations via basis function selection. The algorithm estimates all model parameters simultaneously and automatically, while accounting for within-curve correlation to provide a flexible and computationally efficient framework for smoothing correlated functional data. See da Cruz et al. (2024) for a description of the algorithm and the vignette for examples.

Machine Learningsvmodt v0.1.0: Implements Support Vector Machine Oblique Decision Trees. Recursively builds classification trees using linear Support Vector Machine hyperplanes at each node instead of axis-parallel splits, creating oblique decision boundaries. Features include multiple feature selection methods, dynamic feature subset strategies, class weight support for imbalanced datasets, pruning, and feature penalization. See the vignette.

yaap v1.0.0: Fits archetypal analysis models, including Euclidean, probabilistic, kernel, and directional variants. Methods include classical archetypal analysis from Cutler and Breiman (1994), PCHA and kernel variants from Mørup and Hansen (2012), probabilistic archetypal analysis from Seth and Eugster (2016, directional archetypal analysis from Olsen et al. (2022), AA++ initialization from Mair and Sjölund (2023), coreset-style initialization from Mair and Brefeld (2019), and adapted AIC from Suleman (2017). There are four vignettes including an Introduction and Tidymodels Workflows.

Medical StatisticsBayesTSM v1.0.1: In screening programs, individuals are usually followed up and tested (screened) for the development of a disease. The target disease often develops progressively in stages; for example, healthy (state 1), pre-state disease (state 2), and the disease state (state 3). When the pre-state disease is found during screening, an intervention may prevent disease progression.BayesTSM functions estimate a progressive three-state model with censoring due to intervention using Bayesian estimation methods, as described in Klausch et al. (2023). See the vignette.

bayprior v0.2.12: Provides a toolkit for constructing, validating, and justifying Bayesian priors in clinical trial settings. Implements expert elicitation via quantile matching, the roulette method, and moment matching, linear and logarithmic expert pooling, and prior-data conflict diagnostics. Includes a fully modular Shiny application for interactive use. See Box (1980) and Oakley and O’Hagan (2010) for background. There are six vignettes, including Introduction and Robust, Sceptical, and Power Priors.

Networksnetify v1.5.3: Provides functions to build, validate, analyze, and visualize network data from dyadic, event, matrix, igraph, and network inputs. Supports cross-sectional, longitudinal, bipartite, and multi-layer networks, with conversion helpers for common modeling workflows and plotting utilities for exploratory analysis. Network methods are described in Wasserman and Faust (1994), Cranmer et al. (2021), and Minhas et al. (2022). There are four vignettes, including Quickstart and Internals.

Pharmacokineticsadmixr2 v0.2.0: Provides functions to fit pharmacokinetic/pharmacodynamic (PK/PD) models to aggregate-level data (mean vector and covariance matrix per study) rather than individual-level data. Integrates with the nlmixr2/rxode2 ecosystem via four estimation methods: a First-Order analytical estimator, a Monte Carlo estimator, a Gauss-Hermite quadrature estimator, and an Iterative Reweighting Monte Carlo estimator. Methods are based on Välitalo (2021) software described in van de Beek et al. (2025). See the vignette to get started.

ProbabilityGLBFP v0.5.2: Implements nonparametric density estimation with Averaged Shifted Histogram, Linear Blend Frequency Polygon, and General Linear Blend Frequency Polygon estimators and provides pointwise and grid-based estimation workflows, sparse-prefix grid-count computation, plotting helpers, and plug-in bandwidth selection. Methodological background follows Scott (1992), Terrell and Scott (1985), and Carbon and Duchesne (2024). There are nine vignettes, including Getting started and Package overview.

Programmingrsgl v0.1.0: Generates plots from a database connection and an SGL statement. SGL is a graphics language designed to look and feel like SQL and is especially useful for those familiar with SQL who want to specify plots in a similar manner. The SGL language is described in Chapman (2025). See the vignettes Get started and Example gallery.

PsychometricseasyRasch2 v1.1.0: Streamlines reproducible Rasch measurement theory analyses for ordinal item-response data, combining estimation routines from eRm, psychotool, mirt, iarm, and lavaan with consistent diagnostic, plotting, and reporting layers. Covers the four basic psychometric criteria summarized by Christensen et al. (2021): unidimensionality, local independence, ordered response category thresholds, and invariance across subgroups, together with item fit, targeting, reliability, category functioning, and descriptive item-response plots. A distinguishing feature is the use of simulation-based critical values to replace rule-of-thumb cutoffs. See the vignette.

Risk Analysisriskutility v0.1.0: Provides comprehensive methods to measure disclosure risk and data utility for anonymized and synthetic data. Implements attribution-based risk metrics including Correct Attribution Probability, Targeted CAP, Within Equivalence Class Attribution Probability, and Risk of Attribute Prediction-Induced Disclosure. Also provides distance-based privacy metrics such as Distance to Closest Record, Nearest Neighbor Distance Ratio, and Identical Match Share. Utility assessment includes propensity score analysis, distribution comparisons, and various statistical tests. Methods are based on Taub et al. (2018). See the vignette.

Statisticsbayesqm v0.1.0: Provides a Bayesian factor-analytic framework for Q methodology. Fits a low-rank factor model to Q-sort data with a Student-t likelihood and a hierarchical normal prior on loadings, samples the posterior with Stan, resolves rotational ambiguity via the MatchAlign post-processing of Poworoznek et al. (2025) and returns posterior summaries including credible intervals for loadings and factor scores, probabilistic dominant-factor membership, distinguishing and consensus statements, and PSIS-LOO-based factor enumeration following Vehtari et al. (2017) with the Sivula et al. (2025) parsimony rule. See the vignette to get started.

depthR v0.1.8: Provides efficient implementations of multivariate statistical depth functions in arbitrary dimension. Implements Mahalanobis depth, Tukey halfspace depth, Liu simplicial depth, projection depth, spatial depth, depth-based medians, central regions, outlier detection, and depth-depth plots. C++ backends via Rcpp and RcppEigen ensure performance at large n and d. See Liu (1990), Serfling and Zuo (2000), and Vardi and Zhang (2000) for background, and the vignette for an introduction.

dppca v0.1.0: Provides tools for differentially private principal component analysis visualization and includes functions for estimating private principal component directions, constructing private scree and proportion of variance explained summaries, and visualizing two-dimensional PCA score summaries using additive and sparse histogram mechanisms. Group-wise score visualizations and an interactive shiny app are also provided. See Kim and Jung (2025), Dwork and Roth (2014) and Ramsay and Spicker (2025) for background. There are four vignettes, including Algorithms and PC Directions in dppca.

ernest v1.2.5: Bayesian evidence estimation and posterior inference with the nested sampling algorithm, described in Skilling (2006) and Buchner (2023), along with S3 methods for simulating uncertainty and creating visualizations. See the vignettes Nested Sampling and More Examples.

gkrreg v0.4.0: Implements the Gaussian Kernel Robust Regression method proposed by De Carvalho, Lima Neto and Ferreira (2017), which re-weights observations iteratively using the Gaussian kernel so that poorly-fitted observations receive small weights, yielding resistance to Y-space outliers, X-space outliers and leverage points. Provides three estimators for the kernel width hyper-parameter: Caputo, pairwise median, and residual variance. Inference is accomplished via an analytic sandwich variance estimator or via bootstrap. Six real datasets from the robust regression literature are included to facilitate reproducible comparisons. See the vignette.

picreg v0.1.4 Implements the Pivotal Information Criterion developed by Sardy, van Cutsem, and van de Geer. PIC is a general framework to improve on BIC and LASSO for fitting sparse regression linear models in which the regularization parameter 𝜆 is selected automatically from a pivotal statistic. Functions fit the resulting estimators across six response distributions, Gaussian, binomial, Poisson, exponential, Gumbel, and Cox, and three sparsity-inducing penalties; (LASSO), the Smoothly Clipped Absolute Deviation (SCAD), and Minimax Concave Penalty (MCP). See the vignette for an introduction.

SimplexRegression v0.1.5: Fits and analyzes simplex regression models with either fixed or parametric mean link functions. Implements the simplex probability density function, cumulative distribution function, quantile function, random number generation, and variance evaluation. Offers several fixed and parametric link functions for the mean submodel, tools for residual analysis and diagnostic plotting, hypothesis testing procedures, and influence measures such as Cook’s distance and leverage. Includes the Scout Score criterion for model selection, enabling comprehensive inference and diagnostic analysis within the simplex regression framework. See Barndorff-Nielsen and Jorgensen (1991) and Justino and Cribari-Neto (2026) for more details and the vignette for examples.

vbm v0.1.0: Provides methods for variance-based sensitivity analysis and weighting estimators in observational studies based on the methodology by Huang & Pimentel (2025) Includes bootstrap inference, bias bounds estimation, and visualization tools for sensitivity parameters. See the vignette.

Time Seriesbvars v1.0: Provides fast and efficient procedures for Bayesian estimation and forecasting using state-of-the-art vector autoregressions. Includes the model proposed by Chan (2020), a Bayesian vector autoregression with Minnesota priors and a flexible structure of the error term that permits conditional multivariate normal or Student’s t distributions, as well as homoskedastic or heteroskedastic specifications with a common volatility modelled by centred or non-centred Stochastic Volatility. Additional features include predictive analyses using density forecasting and forecast-error variance decompositions. See README for an example.

fable.intermittent v0.1.1: Extends the fable framework to support forecasting methods specifically designed for intermittent time series data, where demand occurs sporadically with many zero values. All methods produce probabilistic forecasts returned as ‘distributional’ objects. The returned forecasts can be used to evaluate accuracy, plot and print the results. Methods include: Harvey, Fernandes (1989), Willemain, Smart, Schwarz (2004) and several others. See the vignette.

muse v0.1.0: Implements the Power / Trend / Seasonal (PTS) model, a unified state-space framework based on the Multiple Source of Error model. It brings the trend, seasonal and irregular component models of Harvey (1989), Durbin and Koopman (2012) and others together under a single estimation, selection and forecasting interface, with an optional Box-Cox power transformation. Models are estimated by maximum likelihood through the Kalman filter and smoother, with automatic component selection by information criteria. See the vignette.

Utilitiesahocorasick v0.2.0: Provides fast multi-pattern string matching using the ’Aho-Corasick algorithm, powered by the Rust aho-corasick crate. It builds reusable automatons for detecting matches, counting matches, locating characters, extracting matched text, and replacing matches in character vectors. See Aho and Corasick (1975) for more information on the Aho-Corasick algorithm and the vignette for an example.

mx.crypto v0.2.0: Provides Olm and Megolm encryption ratchet primitives for the Matrix messaging protocol, wrapping the vodozemac Rust crate. Provides device-key generation, one-time-key management, 1:1 Olm sessions, and Megolm group sessions. Pairs with the mx.api package, which handles Matrix HTTP transport. See the vignette.

pkgmatch v0.5.4: Provides functions to find R packages from CRAN, rOpenSci, or Bioconductor corpora. Packages can be matched to general text descriptions, to names of installed packages, or to local paths to entire source repositories. The package is used to list the most similar packages for each new submission to the rOpenSci software peer-review program. There are three vignettes, including an introduction and Example applications.

To leave a comment for the author, please follow the link and comment on their blog: R Works.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: June 2026 Top 40 New CRAN Packages

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This post describes the R version of Python’s GPopt (https://docs.techtonique.net/GPopt/GPopt.html), a package for‘Bayesian’ optimization of black-box functions and machine learning hyperparameter tuning, using GaussianProcess Regression and other conformalized surrogates. The package is available on GitHub, and through the R universe.

Keep in mind that this package is for Machine Learning hyperparameter tuning: the global minimum won’t always be found, but this isn’t an issue, since it means you aren’t overfitting the training set.

It’s ported the same way as nnetsauce for R was: with uv tocreate an isolated Python virtual environment containing the Python GPopt package, and reticulate to callinto it from R. Every function in this R package is a thin wrapper that returns the underlying Python object;the general rule is: object accesses with .’s in Python are replaced by $’s in R.

See this post for the technique: Finally figured out a way to port python packages to R using uv and reticulate.

Install1. Create a Python virtual environment with uv ```

pip install uv # if necessaryuv venv venvsource venv/bin/activate # on Windows: venv\Scripts\activateuv pip install pip GPopt

`` Keep track of wherevenv/lives – you’ll pass its path asvenv_path` to every function in this package.

Install the R package install.packages("remotes")remotes::install\_github("Techtonique/GPopt\_r") reticulate will be installed automatically as a dependency.

ExamplesMinimizing the Branin functionThis is a standard test function for optimization algorithms. GPOpt is more suitable for expensive black-box functions, but this is a good example to illustrate the usage of the package.

library(GPopt)branin <- function(x) { x1 <- x[1]; x2 <- x[2] term1 <- (x2 - (5.1 * x1^2) / (4 * pi^2) + (5 * x1) / pi - 6)^2 term2 <- 10 * (1 - 1 / (8 * pi)) * cos(x1) term1 + term2 + 10}opt <- GPOpt( lower\_bound = c(-5, 0), upper\_bound = c(10, 15), objective\_func = branin, n\_init = 10, n\_iter = 40, venv\_path = "./venv")opt$optimize(verbose = 1L)print(opt$x\_min) # best parametersprint(opt$y\_min) # best objective value Tuning a scikit-learn model’s hyperparameters library(GPopt)sklearn <- get\_sklearn(venv\_path = "./venv")RandomForestClassifier <- sklearn$ensemble$RandomForestClassifierX <- as.matrix(iris[, 1:4])y <- as.integer(iris$Species) - 1Lmlopt <- MLOptimizer(scoring = "accuracy", cv = 5, venv\_path = "./venv")param\_config <- list( n\_estimators = list(bounds = c(10, 300), dtype = "int"), max\_depth = list(bounds = c(1, 20), dtype = "int"))mlopt$optimize( X\_train = X, y\_train = y, estimator\_class = RandomForestClassifier(), param\_config = param\_config, verbose = 1L)print(mlopt$get\_best\_parameters())print(mlopt$get\_best\_score()) Bayesian optimization with early stopping library(GPopt)opt <- BOstopping( f = branin, bounds = rbind(c(-5, 10), c(0, 15)), venv\_path = "./venv")result <- opt$optimize(n\_iter = 100L) Using a custom (conformalized) surrogate model library(GPopt)sklearn <- get\_sklearn(venv\_path = "./venv")ns <- get\_nnetsauce(venv\_path = "./venv")opt <- GPOpt( lower\_bound = c(-5, 0), upper\_bound = c(10, 15), objective\_func = branin, acquisition="ucb", method="splitconformal", surrogate\_obj = ns$PredictionInterval(sklearn$ensemble$RandomForestRegressor()), venv\_path = "./venv")opt$optimize(verbose = 1L) To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: GPopt for R: Bayesian and conformal optimization of black-box functions and hyperparameter tuning

View Details

[This article was first published on Statforbiology, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Biological phenomena are often studied by examining how a numerical variable, usually called the response (e.g., the weight or height of an organism), is affected by another variable, usually called the predictor (e.g., time or fertiliser application rate). These ‘predictor-response’ relationships are commonly described using models, which express the response as a mathematical function of the predictor. In general, a model can be written as:

where is the response, is the predictor, and is a collection of parameters, usually denoted by symbols, such as the letters of the Roman or Greek alphabet. The element is the function, which determines the shape of the relationship when plotted on an x–y graph.

Because mathematical modelling plays such a central role in biology and many other scientific disciplines, biologists need to be familiar with the most important mathematical functions. More importantly, they need to be able to “read” these functions and use their parameters to describe, interpret, and quantify biological processes. With this aim in mind, I have compiled a collection of the mathematical functions most commonly encountered in biology, explaining the meaning of their parameters, with particular emphasis on their biological interpretation rather than their mathematical properties.

Curve shapesFunctions are often classified according to the shape they exhibit when plotted on an x–y graph. This approach is adopted, for example, in Ratkowsky (1990), and the classification presented below is largely based on that work.

Polynomials
1. Straight line function
2. Quadratic polynomial function
Concave/Convex curves (no inflection)
1. Exponential function
2. Asymptotic function / Negative exponential function
3. Power function
4. Logarithmic function
5. Rectangular hyperbola
Sigmoidal curves
1. Logistic function
2. Gompertz function
3. Modified Gompertz function
4. Log-logistic function
5. Weibull (type-1) function
6. Weibull (type-2) function
Curves with maxima/minima
1. Peaked sigmoidal function
2. Bragg function
3. Lorentz function
4. Beta function

I have chosen to include a relatively large number of functions, so this post is necessarily rather long. To make navigation easier, you can first inspect Figure 1 to identify the type of curve of interest and then use the links above to jump directly to the corresponding section.

Throughout this post, denotes the predictor, denotes the response, and the letters of the Roman alphabet (, , , …) denote the model parameters, which determine the shape of the function.

Figure 1: The shapes of the most important functions. The different colors indicate the possible different shapes of the same function, with different parameters. In the case of the Bragg/Lorentz function, the red color indicate the Bragg function and the blue color indicate the Lorentz function (same parameter values).PolynomialsPolynomials form a class of their own because their flexible shapes allow them to approximate a wide variety of biological processes, at least over a restricted range of the predictor. The mathematical tractation is very simple, but their main limitation is that they cannot describe asymptotic processes, which are extremely common in biology. Moreover, as the polynomial degree increases, their shapes become increasingly complex and often lack a meaningful biological interpretation.

Straight-line functionThe simplest polynomial is the straight line, with equation:

where is the value of when , and is the slope, that is, the change in associated with a one-unit increase in . When , increases as increases; when , it decreases.

Because of their simplicity, straight lines are mainly used to approximate biological responses over a limited range of the predictor. Should you need it, an example of fitting a straight line to research data is given elsewhere at this link.

Quadratic polynomial functionThe quadratic polynomial is described by the equation:

This equation describes a U-shaped curve (a parabola), which may open upwards or downwards depending on the sign of . If , the parabola opens upwards; otherwise, it opens downwards.

The parameter represents the value of when , whereas and determine how the response changes as the predictor varies. To better understand the role of these parameters, we can examine the first derivative of Eq. (2). In R, this can be obtained with the D() function, which differentiates an expression with respect to a specified variable:

D(expr = expression(a + b*X + c*X^2), name = "X")

b + c * (2 * X) The derivative is not constant but varies with the value of and, in particular, represent the variation of around the point . The stationary point, that is, the point where the derivative is zero, occurs at:

This point is a minimum when and a maximum when .

The corresponding response is

In practice, biologists often use only one branch of the parabola to approximate biological responses showing either a concave or a convex trend over a limited range of the predictor. Check this link for an example of fitting.

Concave/Convex curves (no inflection points)Exponential functionThe most common parameterisation of an exponential function is:

Depending on the sign of , Eq. 3 describes a concave-up monotonically increasing shape (exponential growth; ) or a concave-up monotonically decreasing shape (exponential decay; ).

The parameter represents the value of when , whereas the meaning of can be understood by considering the first derivative:

D(expression(a * exp(k * X)), "X")

a * (exp(k * X) * k) In algebraic terms, it is:

and thus:

Therefore, represents the relative rate of change (increase/decrease), throughout the domain, which is often known in growth analysis as the Relative Growth Rate (RGR).

Other equivalent parameterisations are:

and:

Equations 3-5 are equivalent, as can be shown by setting and :

Another, slightly different, parameterisation is commonly used in bioassay studies, mainly to describe exponential decay processes:

where corresponds to in the previous models and .

For all the exponential decay equations presented above, as . A lower asymptote, , can be incorporated into Eq. 6 to account for situations in which the response does not approach zero as the predictor tends to infinity:

Exponential functions are often used to describe the growth of populations under non-limiting environmental conditions or the degradation of xenobiotics in the environment (first-order degradation kinetics). In both cases, represents time and is therefore restricted to non-negative values, while (or, equivalently, ) must be different from zero. Check the post at this link for an example of fitting.

Asymptotic functionThis function appears in several different parameterisations and is also known as the monomolecular growth function, the Mitscherlich law, or the von Bertalanffy law. Owing to its biological interpretation, the most widely used parameterisation is:

It describes a monotonically increasing, concave-down curve approaching a horizontal asymptote as tends to infinity. The parameter represents the maximum attainable value of (the plateau), while is the value of when (the initial value). The parameter is proportional to the rate at which the response approaches the plateau. Indeed, the first derivative is

D(expression(a - (a - b) * exp (- m * X)), "X")

(a - b) * (exp(-m * X) * m) Considering Eq.8, we can write:

and then:

The previous expression shows that the relative rate of change in the response (the RGR in growth analysis) is not constant, as it is for the exponential function. Instead, it depends on the attained value of . In particular, the RGR is greatest at the beginning of the process, when is smallest, and gradually approaches zero as approaches the plateau .

Another closely related parameterisation is often encountered, in which is replaced by , by , and by . Some simple algebraic manipulations are also introduced to make the model consistent with the parameterisation commonly adopted for biological assay models:

In some applications, is reparameterised through its logarithm to facilitate model fitting:

For all these equations, setting (or equivalently ) yields a curve passing through the origin. In this case, the most common parameterisation is:

which is known as the negative exponential function.

The asymptotic function is widely used in biology, not only for growth analysis but also, for example, in weed competition studies. The negative exponential function has also been used to model the absorbed Photosynthetically Active Radiation () as a function of leaf area index (). In this context, represents the incident PAR (), while is the light extinction coefficient. Check the post at this link for an example of fitting.

Power functionThe power function is also known as Freundlich function or allometric function. The most common parameterisation is:

Its shape is highly flexible and is determined by the value of the parameter (see Figure 1). When , the response increases with and the curve is concave down. When , decreases as increases and the curve is concave up. Finally, when , increases with and the curve is concave up. These three cases are illustrated in Figure 1 using different colours. The function is defined only for and has no horizontal asymptote as .

The biological interpretation of the parameters and is not straightforward. Both influence the slope of the curve, as can be seen by examining the first derivative:

D(expression(a * X^b), "X")

a * (X^(b - 1) * b) The power function is mathematically equivalent to an exponential function of , since:

The power function (named as the Freundlich equation) is widely used in agricultural chemistry, for example to model the sorption of xenobiotics in soil. It is also used to describe the relationship between the number of plant species and the sampled area (the species–area relationship). Check the post at this link for an example of fitting.

Logarithmic functionThis function is linear in :

Because of the logarithmic transformation, the function is only defined for . The parameter determines the shape of the curve. When , the response increases with and the curve is concave down. Conversely, when , decreases as increases and the curve is concave up, as shown in Figure 1.

The interpretation of the parameters is fairly straightforward. The parameter is the value of the response when . Changing while keeping constant simply shifts the curve vertically without altering its shape, so that the difference between any two curves remains constant for every value of (Fig. 2).

The parameter controls the slope of the curve and, more specifically, it is equal to the slope at , as can be seen from the expression for the first derivative:

D(expression(a + b*log(X)), "X")

b * (1/X) Changing while keeping constant produces curves that all intersect at (Fig. 2).

Figure 2: Effects of changing parameter values on the shape of logarithmic curves. The parameter is 1 for the red curves and 3 for the black curves, while is 1 for the solid lines (-1 on the right panel) and 0.5 for the dotted lines (-0.5 in the right panel)In biology, logarithmic functions are used, for example, to describe species–area relationships in ecology, enzyme kinetics in biochemistry, and sensory perception in neurobiology. Check the post at this link for an example of fitting.

Rectangular hyperbolaThe rectangular hyperbola is commonly known as the Michaelis-Menten function and is usually parameterised as

The curve is monotonically increasing and concave down, approaching the horizontal asymptote (plateau) . It passes through the origin of the axes ( when ). The parameter is the value of that produces a response equal to . Indeed,

which is readily solved to obtain .

The first derivative is:

D(expression( (a*X) / (b + X) ), "X")

a/(b + X) - (a * X)/(b + X)^2 From this expression, we can see that the initial slope (at ) is .

Equation 14 is not defined for and has no biological meaning for . In most biological applications, however, both and are positive.

An equivalent parameterisation is:

In this form, the parameters and are often replaced by and , respectively. Although the response is not defined for , it tends to zero as .

Another common parameterisation includes the initial slope as an explicit parameter because of its biological relevance. This is obtained by dividing both the numerator and denominator by and noting that , so that :

The Michaelis-Menten function is widely used in pesticide chemistry, enzyme kinetics (Eq. 14), biological assay models (Eq. 15), and weed competition studies, where it is used to describe crop yield loss as a function of weed density (Eq. 16). An example of fitting is shown in another post, at this link.

Sigmoidal functionsSigmoidal functions are S-shaped, with two horizontal asymptotes (a lower and an upper asymptote) and an inflection point. They can be parameterised in countless ways, which may be confusing when selecting the most appropriate model for a particular biological process. In this post, I will use the parameterisations proposed in Ritz et al (2019), which is very consistent and puts all the curves on the same ground. Furthermore, the fitting properties are good, which contribute to good convergence and reliable estimation.

For the sake of simplicity, it is useful to recognise that most sigmoidal models belong to one of three basic families, which differ only in the position of the inflection point relative to the two asymptotes:

Logistic: the ordinate of the inflection point lies halfway between the lower and upper asymptotes; consequently, the S-shape is symmetric (the concave-up and the concave-down portions ‘mirror’ each other).
Gompertz: the ordinate of the inflection point lies closer to the lower asymptote, and, thus, the concave-up portions is ‘shorter’ than the concave-down portion.
Modified Gompertz: the ordinate of the inflection point lies closer to the upper asymptote, and, thus, the concave-up portion is ‘longer’ than than the concave-down portion.

Each of these three equations can describe either increasing or decreasing responses, depending on the sign of one of the model parameters (see below). Their different shapes are compared side by side in Fig. 3.

Figure 3: The different shapes of sigmoidal curves based on x: logistic (red lines), Gompertz (blue lines) and modified Gompertz (green lines).Each of these three families (logistic, Gompertz and modified Gompertz) can be transformed by replacing with , giving raise to three corresponding families:

Log-logistic
Type-I Weibull
Type-II Weibull

Like their linear-scale counterparts, these three families can describe either increasing or decreasing responses, depending on the sign of the same model parameter (see below). In the end, we have twelve different sigmoidal functions to choose from. The choice among them depends primarily on the expected symmetry of the response and on whether the predictor is more naturally interpreted on a linear or logarithmic scale.

Thanks to their versatility, sigmoidal functions have been used for an uncountable number of biological applications. Just to mention a few examples in agriculture, I would like to cite plant growth, dose-response curves in biological assays and the time-course of seed germination. For some examples, take a look at my post at this link.

Logistic functionThe logistic curve is derived from the cumulative logistic distribution function. It is symmetric about its inflection point and can be parameterised as:

where is the upper asymptote, is the lower asymptote, is the value of at the inflection point, and is proportional to the slope at the inflection point. Because the curve is symmetric, also represents the value of that produces a response halfway between and (commonly referred to as the ED50 in biological assays). The parameter may be either positive or negative and, consequently, the response may either increase or decrease as increases.

This equation is known as the four-parameter logistic model. If appropriate, constraints can be imposed on the parameter values. For example, may be fixed at 0, yielding the three-parameter logistic model. If, in addition, is fixed at 1, the model reduces to the two-parameter logistic model.

Gompertz functionThe Gompertz curve can be parameterised in many different ways. I prefer a parameterisation that closely resembles that of the logistic function:

Unlike the logistic function, the Gompertz curve is not symmetric about its inflection point. It exhibits a longer lag phase at the beginning, followed by a progressively steeper increase before gradually approaching the upper asymptote. The parameters have essentially the same interpretation as those of the logistic function, except that , the abscissa of the inflection point, does not correspond to the value of producing a response halfway between and .

As with the logistic function, four-, three-, and two-parameter Gompertz models can be obtained by constraining one or both asymptotes.

Modified Gompertz functionWe have seen that the logistic curve is symmetric about its inflection point, whereas the Gompertz curve exhibits a longer lag phase at the beginning, followed by a progressively steeper increase. A different asymmetric pattern can be obtained by modifying the Gompertz function as follows:

The resulting curve increases rapidly at the beginning but gradually slows down as it approaches the upper asymptote. As with the logistic and Gompertz functions, one or both asymptotes can be constrained ( and/or ), giving rise to four-, three-, and two-parameter versions of the modified Gompertz model.

Log-logistic functionThe log-logistic curve is symmetric with respect to . A log-normal curve has a very similar shape, although it is used much less frequently. In biological assays (and also in germination studies), the log-logistic function is commonly parameterised as

The parameters have the same interpretation as in the logistic function. In particular, represents the value of that produces a response halfway between and (the ED50). It is easy to show that the above equation is equivalent to

Like the logistic function, the log-logistic model can be fitted in four-, three-, or two-parameter versions by constraining one or both asymptotes. It is widely used to describe dose-response relationships in biological assays, seed germination, and crop growth.

Weibull function (type 1)The Type I Weibull function is the logarithmic counterpart of the Gompertz function, being defined on rather than on . It is parameterised as

The parameters have essentially the same interpretation as those of the other sigmoidal functions presented above. In particular, and are the lower and upper asymptotes, respectively, while is the value of corresponding to the inflection point. Unlike the log-logistic function, however, does not correspond to the ED50.

Weibull function (type 2)The Type II Weibull function is closely related to the Type I Weibull function, but it describes a different type of asymmetry, analogous to that of the modified Gompertz function:

The parameters have the same interpretation as those of the Type I Weibull function. In particular, and are the lower and upper asymptotes, respectively, while is the value of at the inflection point. As with the Type I Weibull function, does not correspond to the ED50. One or both asymptotes may be constrained, giving rise to four-, three-, and two-parameter versions of the model.

Another ‘flexible’ sigmoidI would also like to mention another sigmoidal function that has been widely used in biology because of its supposed flexibility: the Richards function:

In this model, the degree of asymmetry is controlled by the parameter . When , the function reduces to the logistic model, whereas, as decreases below 1, the curve progressively resembles a Gompertz function. Figure 4 illustrates the different shapes obtained for different values of .

Despite this apparent flexibility, the Richards function has poor statistical properties for parameter estimation. In particular, the additional parameter often causes strong correlations among the estimates, making the fitting process unstable and the parameter estimates difficult to interpret. Ratkowsky (1990) described it as having “more undesirable nonlinear regression behaviour than almost any nonlinear regression model in common use.” For these reasons, the Richards function will not be considered further in this blog.

Figure 4: The different shapes of the Richard’s function, based on the parameter: (black line), (blue line) and (red line)Curves with maxima/minimaIt is sometimes necessary to describe phenomena where the variable reaches a maximum value at a certain level of the variable, and drops afterwords. For example, growth or germination rates are higher at optimal temperature levels and lower at supra-optimal or sub-optimal temperature levels. Another example relates to bioassays: in some cases, low doses of toxic substances induce a stimulation of growth (hormesis), which needs to be described by an appropriate model. The second order (and higher order) polynomial funcion we have seen earlier is capable of accounting for maxima/minima, but there are a few other interesting functions that may turn out useful in some circumstances.

Peaked sigmoidal functionThe log-logistic decreasing curve can be modified to account for possible hormetic effects at low doses, by combining mathematical switching functions (Schabenberger and Pierce, 2002; pag. 275). The most widespread parameterisation, that was originally devised by Brain and Cousens (Brain, P., Cousens, R., 1989. An equation to describe dose responses where there is stimulation of growth at low doses. Weed Research 29, 93–96) in a slightly different form, is:

The parameters have the same interpretation as in the log-logistic curves, but does not represent the response half-way between the lower and higher asymptote and represents the size of the hormetic effect, which increases as increases, while corresponds to the situation of no hormesis (and the function reduces to a log-logistic.

Bragg functionThis function is connected to the normal (Gaussian) distribution and has a symmetric shape with a maximum equal to , that is reached when and two inflection points. In this model, relates to the slope at the inflection points; the response approaches 0 when approaches :

If we would like to have lower asymptotes different from 0, we should add the parameter , as follows:

The two Bragg functions have proven useful in applications relating to the science of carbon materials.

Lorentz functionThe Lorentz function is similar to the Bragg function, although it has worse statistical properties (Ratkowsky, 1990). The equation is:

We can also allow for lower asymptotes different from 0, by adding a further parameter:

Beta functionThe beta function derives from the beta density function and it has been adapted to describe phenomena taking place only within a minimum and a maximum threshold value (threshold model). One typical example is seed germination, where the germination rate (GR, i.e. the inverse of germination time) is 0 below the base temperature level and above the cutoff temperature level. Between these two extremes, the GR increases with temperature up to a maximum level, that is reached at the optimal temperature level.

The equation is:

where is the maximum level for the expected response , and are, respectively, the minumum and maximum threshold levels, is the abscissa at the maximum expected response level and is a shape parameter. The above function is only defined for and it returns 0 elsewhere.

ConclusionsHere we are; I have discussed more almost 30 functions, which are commonly used to model biological processes. These functions can be found in several other different parameterisations and I suggest you read the book chapter by Miguez et al. (2018), for other interesting information.

Thanks for reading! And … don’t forget to check out my new book!

Prof. Andrea Onofri
Department of Agricultural, Food and Environmental Sciences
University of Perugia (Italy)
Send comments to: andrea.onofri@unipg.it

Further readings1. Miguez, F., Archontoulis, S., Dokoohaki, H., Glaz, B., Yeater, K.M., 2018. Chapter 15: Nonlinear Regression Models and Applications, in: ACSESS Publications. American Society of Agronomy, Crop Science Society of America, and Soil Science Society of America, Inc. 2. Ratkowsky, D.A., 1990. Handbook of nonlinear regression models. Marcel Dekker Inc., New York, USA. 3. Ritz, C., Jensen, S. M., Gerhard, D., Streibig, J. C., 2019. Dose-Response Analysis Using R. CRC Press 4. Schabenberger, O., Pierce, F.J., 2002. Contemporary statistical models for the plant and soil sciences. Taylor & Francis, CRC Press, Books.

This post was originally published in this blog on 2019-01-08

To leave a comment for the author, please follow the link and comment on their blog: Statforbiology.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Some useful equations for biological processes

View Details

[This article was first published on Jason Bryer, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. I find that when teaching statistics (and probability) it is often helpful to simulate data first in order to get an understanding of the problem. The Monty Hall problem recently came up in a class so I implemented a function to play the game.

The Monty Hall problem results from a game show, Let’s Make a Deal, hosted by Monty Hall. In this game, the player picks one of three doors. Behind one is a car, the other two are goats. After picking a door the player is shown the contents of one of the other two doors, which because the host knows the contents, is a goat. The question to the player: Do you switch your choice?

For more information, be sure to see the Wikipedia article.

Below we implement a function that will simulate a single play of this game. You can play interactively, or if you specify the pick and switch parameters this can be looped in order to simulate the results.

monty\_hall <- function(pick, switch) { interactive <- FALSE if(missing(pick)) { interactive <- TRUE cat('Pick your door:') pick <- LETTERS[menu(c('A', 'B', 'C'))] } else { if(!pick %in% LETTERS[1:3]) { stop('pick must be either A, B, or C') } } doors <- c('win', 'lose', 'lose') doors <- sample(doors) # Shuffle the doors names(doors) <- LETTERS[1:3] if(doors[pick] == 'win') { show <- sample(names(doors[!names(doors) %in% pick]), size = 1) } else { show <- doors[!names(doors) %in% pick] == 'lose' show <- names(which(show == TRUE)) } if(missing(switch)) { interactive <- TRUE cat(paste0('Showing door ', show, '. Do you want to switch your choice?')) switch <- menu(c('yes', 'no')) == 1 } if(switch) { pick <- names(doors)[!names(doors) %in% c(show, pick)] } win <- unname(doors[pick] == 'win') if(interactive) { if(win) { cat('You win!') } else { cat('Sorry, you lost.') } invisible(win) } else { return(win) }} We can play a single game:

monty\_hall()

Pick your door:1: A2: B3: CSelection: 2Showing door A. Do you want to switch your choice?1: yes2: noSelection: 1You win! Let’s now simulate 1,000 games. We will use two vectors, mh_switch and mh_no_switch, to store the results after switching doors or not, respectively. For each iteration, the initial door pick is randomly selected.

n\_games <- 1000mh\_switch <- logical(n\_games)mh\_no\_switch <- logical(n\_games)for(i in 1:n\_games) { pick <- sample(LETTERS[1:3], size = 1) mh\_switch[i] <- monty\_hall(pick = pick, switch = TRUE) mh\_no\_switch[i] <- monty\_hall(pick = pick, switch = FALSE)} The probability of winning if we switch the door is:

mean(mh\_switch)

[1] 0.671 The probability of winning if we do not switch the door is:

mean(mh\_no\_switch)

[1] 0.328 It should be noted that the theoretical probability of winning if you switch is 2/3, and is 1/3 if you don’t switch.

To leave a comment for the author, please follow the link and comment on their blog: Jason Bryer.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simulating Monty Hall’s Problem

View Details

[This article was first published on Jason Bryer, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. When working with distributions in R, each distribution has four functions, namely:

dXXX – density function.
rXXX – generate random number from this distribution.
pXXX – returns the area to the left of the given value.
qXXX – returns the quantile for the given value/area.

Where XXX is the distribution name (e.g. norm, binom, t, etc.).

remotes::install\_github('jbryer/VisualStats') The VisualStats::plot_distributions() function will generate four plots representing the four R distribution functions. For each subplot points correspond to the first parameter of the corresponding function (note the subplot for the random rXXX function does not have points since this simply returns random values from that distribution). The arrows correspond to what that function will return.

library(VisualStats)data('distributions', package = 'VisualStats')plot\_distributions(dist = 'norm', xvals = c(-1, 0, 0.5), xmin = -4, xmax = 4) The top two plots (dXXX and rXXX) plot the distribution. The bottom two plots are the cumulative density function for the given distribution. The CDF describes the probability that a random variable (X) will be less than or equal to a specific value (x), written as F(x) = P(X ≤ x). The CDF provides a complete view of a random variable’s distribution by accumulating probabilities up to that point.

plot\_distributions(dist = 'binom', xvals = c(1, 3), xmin = 0, xmax = 10, args = list(size = 10, prob = 0.35)) The VisualStats package also has a Shiny application that allows you to interactively plot the 17 distributions available in base R.

Screenshot of the distributions Shiny application To leave a comment for the author, please follow the link and comment on their blog: Jason Bryer.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Plotting Distributions in R

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.At rOpenSci, our Code of Conduct (CoC) committee works to support a healthy, welcoming, and inclusive community. A big part of this work is making sure that the processes we follow are transparent, consistent, and fair. Over the years, we’ve developed a set of templates that guide us through different stages of incident response and reporting.

We believe these templates may be useful for other communities and projects in open science that are establishing or refining their own CoC processes. Today, we are sharing them with you.

Why templates?Handling Code of Conduct issues can be emotionally demanding and logistically complex. Templates help us by:

Providing structure: ensuring we don’t miss important steps.
Improving consistency: so similar situations are handled in similar ways.
Supporting transparency: both internally and externally, we can clearly show how decisions are made.
Reducing stress: in sensitive moments, having a clear guide helps committee members focus on people, not paperwork.
Easing onboarding and documenting processes: new committee members can quickly get up to speed, and we maintain a written record of how we handle different situations.

The templatesOur templates cover both internal workflows (helping the committee document and process incidents) and community-facing communication (keeping our community informed in a consistent way). Together, they give us a clear structure and schedule for everything from receiving a report to sharing aggregated updates.Here’s an overview of the templates we rely on at rOpenSci:

CoC update report: a framework for annual updates to the community about the committee’s activities.
CoC transparency report: a public-facing report that shares in a yearly aggregate way the types of incidents handled and outcomes, while protecting privacy.
Internal incident summary: a confidential document for committee use, summarizing the key facts and steps taken in a specific incident.
Internal handling incident report: introduced in an earlier blog post, this guides us through the process of responding to a report from start to finish.
CoC reporting form: the form community members use to submit a report.
Incident report email acknowledgement: a standard response we send to confirm receipt of a report, so reporters know their concern has been heard and is being handled.

How you can use themWe know every community has different values, resources, and contexts, so consider these templates a toolbox rather than a one-size-fits-all solution. You might use them as-is, or treat them as a starting point to design your own processes.

You can access all the templates together in the GitHub repository with community management tools. You can fork them, adapt them, and make them work for your own community context. We’d love to hear how you adapt or improve them for your own communities.

AcknowledgementsWe want to thank all past and present members of the rOpenSci Code of Conduct Committee.Their work has made it possible for us to share these templates openly today.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci Code of Conduct Committee Templates

View Details

[This article was first published on jakub::sobolewski, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Don’t fall into the trap of writing scenarios until you run out of ideas.

This approach misses the fundamental economics of testing. Like any investment, BDD scenarios follow the law of diminishing returns. The first scenario should capture the bulk of user value. The second adds meaningful behavior. By the fifth, you’re chasing edge cases that belong in unit tests.

Understanding this cadence transforms how you approach BDD. Instead of exhaustive scenario coverage, you focus on iterative value delivery where each scenario must justify its existence.

Level-up your testing game! Grab your copy of the R testing roadmap.

Start With the Golden ScenarioWrite the scenario that captures your system’s core promise first.

For document management software, this means the approval workflow that brings immediate value to users. Everything else is secondary.

Feature: Document Approval Workflow As a document reviewer I want to review and approve documents So that I can ensure quality before publication Scenario: Document reviewer approves a document Given a document "Project Proposal" has been submitted for approval When I approve the document Then the document status should be "Approved" This scenario embodies the primary user journey. It validates that documents can be submitted, reviewed, and approved. Get this first specification working, and you start delivering value immediately.

Notice how the scenario focuses on behavior, not implementation. We don’t specify database tables, API endpoints, or UI elements. We describe what users experience.

Add the Second Most Valuable BehaviorThe second scenario should address the next essential user need.

In document approval, rejection is equally critical. Users need feedback when documents don’t meet standards.

Scenario: Document reviewer rejects a document with feedback Given a document "Project Proposal" has been submitted for approval When I reject the document And I enter the comment "Missing cost breakdown for Q3" Then the document status should be "Rejected" And the document should have the comment "Missing cost breakdown for Q3" This scenario introduces feedback mechanisms. It ensures rejected documents don’t disappear into a black hole but generate actionable communication.

Two scenarios now cover the complete approval cycle. You’ve captured 70-85% of user value with minimal investment.

Reuse scenario stepsWe start building a vocabulary of how to describe behavior of the system:

Given a document {string} has been submitted for approvalThen the document status should be {string} Another parametrized step could be:

When I {word} the document Every, even complex system can be described with a surprisingly small set of steps. If you don’t get enough reuse, you are probably over-specifying the scenarios, possibly revealing implementation details. That’s the hint to make them more abstract, but keep them precise.

Our set of scenarios should grow faster than the library of steps we need to implement to run them.

The Law of Diminishing ReturnsBeyond two or three scenarios, each addition provides less value while increasing maintenance and runtime cost.

After the third scenario, you’re in diminishing returns territory. Each new scenario costs as much to run as the valuable ones, but contributes mere percentage points of additional coverage. Is it worth waiting extra few minutes in your CI pipeline for a scenario that tests a rare edge case?

Don’t write scenarios for edge cases that should live in unit tests.

Push Edge Cases Down to Unit TestsComplex validation rules, error conditions, and boundary cases belong in fast unit tests.

Consider these potential BDD scenarios that should actually be unit tests:

Documents with special characters in names
Approval workflows with 5+ reviewers
Network timeouts during submission
File size limits and format validation
Concurrent approval attempts

These scenarios test implementation details rather than user-visible behavior. They run slower, break more often, and provide minimal business insight.

Unit tests handle these cases better:

test\_that("document validation rejects oversized files", { # Arrange document <- create\_document(size\_mb = 50) # Act result <- validate\_document(document) # Assert expect\_false(result$is\_valid) expect\_equal(result$error, "File size exceeds 25MB limit")}) Fast, focused, and maintainable.

Focus on Iterative Value DeliveryThis approach aligns perfectly with iterative design principles.

You start with the minimum viable behavior that provides user value. Ship it. Get feedback. Then add the next most valuable scenario.
Each iteration delivers working software that real users can evaluate. You avoid building comprehensive features that nobody wants.
BDD scenarios become your value compass. If a scenario doesn’t represent behavior that users would miss if broken, it probably shouldn’t exist.

Implementation: Cucumber or Internal DSLYou can implement this cadence with Cucumber or a custom Domain Specific Language (DSL).

Cucumber provides the standard Gherkin syntax and step definition mapping:

library(cucumber)given("a document {string} has been submitted for approval", function(name, context) { context$driver$submit\_document(name)})when("I {word} the document", function(action, context) { context$driver$perform\_action\_on\_document(action)})then("the document status should be {string}", function(expected\_status, context) { actual\_status <- context$driver$get\_document\_status() expect\_equal(actual\_status, expected\_status)}) Alternatively, build an internal Domain Specific Language that captures the same behavior:

given\_document\_submitted <- function(name, driver) { driver$submit\_document(name)}when\_approving\_document <- function(name, driver) { driver$perform\_action\_on\_document("approve")}then\_document\_status\_is <- function(expected\_status, driver) { actual\_status <- driver$get\_document\_status() expect\_equal(actual\_status, expected\_status)}test\_that("Document Approval Workflow", { driver <- new\_driver() given\_document\_submitted("Project Proposal", driver) when\_approving\_document("Project Proposal", driver) then\_document\_status\_is("Approved", driver)}) Both approaches work. Choose based on team preferences and tooling constraints.

For more details on implementing the driver and internal DSL, check out how I did it in this tutorial.

Build Valuable Things FirstThis cadence helps you deliver value faster and more predictably.

Instead of spending weeks writing comprehensive scenario suites, you identify and implement the highest-value behaviors first. Users see progress immediately.

You avoid the trap of gaming test coverage. Every scenario earns its place by representing genuine user value.

Most importantly, you focus on behavior rather than implementation. This keeps your tests resilient to code changes while ensuring they validate what actually matters to users. If your system changes implementation details, those changes won’t break your BDD scenarios as long as the behavior remains consistent.

This cadence isn’t about writing fewer tests.

It’s about writing most valuable specifications first and pushing complexity to the appropriate testing layer.

To leave a comment for the author, please follow the link and comment on their blog: jakub::sobolewski.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Cadence of Behavior-Driven Development

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Deutsche Bank Research Institute stated in its published report that Bitcoin has undergone a process similar to what gold experienced over the past 100 years.

According to the report, Bitcoin’s increasing adoption and reduced volatility may transform it into a reserve asset that central banks could hold by 2030.

The uncertainty graph below confirms the analysis mentioned above. Especially over the last two years, gold and bitcoin have converged in terms of monthly returns distribution.

Chart Code:

library(tidyverse)library(tidyquant)library(ggdist)#Golddf\_gold <- tq\_get("GC=F") %>% tq\_transmute(select = close, mutate\_fun = periodReturn, period = "monthly", col\_rename = "gold\_returns") %>% drop\_na()#Bitcoindf\_btc <- tq\_get("BTC-USD") %>% tq\_transmute(select = close, mutate\_fun = periodReturn, period = "monthly", col\_rename = "btc\_returns") %>% drop\_na()#Merging the datasetsdf\_merged <- df\_gold %>% left\_join(df\_btc) %>% filter(date >= as.Date("2020-01-01")) %>% drop\_na() %>% pivot\_longer(-date) %>% mutate(year = year(date) %>% as\_factor())#Uncertainty Distribution Plotdf\_merged %>% ggplot(aes(y = value, x = year, fill = name)) + stat\_slab(aes(thickness = after\_stat(pdf*n)), scale = 0.7) + stat\_dots(side = "bottom", scale = 0.7, slab\_linewidth = NA) + scale\_y\_continuous(labels = scales::percent) + scale\_fill\_manual(values = c("darkorange","goldenrod")) + labs(x = "", y = "", fill = "", title = "Comparison of Monthly Returns: <span style = 'color:goldenrod;'>Gold</span> vs. <span style = 'color:darkorange;'>Bitcoin</span>") + theme\_minimal(base\_family = "Roboto Slab", base\_size = 20) + theme(axis.text = element\_text(face = "bold"), plot.title = ggtext::element\_markdown(size = 18, hjust = 0.5, face = "bold"), axis.text.x = element\_text(angle = 45, hjust = 1, vjust = 1), legend.position = "none", plot.background = element\_rect(fill = "azure", color = "azure"), panel.background = element\_rect(fill = "snow", color = "snow")) To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Uncertainty Analysis: Gold vs. Bitcoin

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. If this post is useful to you I kindly ask a minimal donation on Buy Me a Coffee. It shall be used to continue my Open Source efforts. The full explanation is here: A Personal Message from an Open Source Contributor.

You can send me questions for the blog using this form and subscribe to receive an email when there is a new post.

cpp4r helps you to interact with R objects using C++ code. It is a fork of the cpp11 package with identical syntax and similar goals.

Code: https://github.com/pachadotdev/cpp4r.
Documentation: https://cpp4r.org/.

cpp4r can be used as a replacement for cpp11 in existing or new packages. Think of cpp11 and cpp4r as MySQL and MariaDB: they are almost identical, but cpp4r has some extra features.

After discussing some pull requests with Hadley Wickham from Posit, it was mentioned that I should create my own fork to add the following features:

Convert ordered and unordered C++ maps to R lists.
Roxygen support on C++ side.
Allow dimnames atribute with matrices on C++ side.
Support nullable external_ptr<>.
Use values added to a vector with push_back() immediately.
Support bidirectional passing of complex numbers/vectors.
Provide flexibility with data types (e.g., cpp4r’s as_integers() and as_doubles() accept logical inputs while cpp11’s do not).
Some internal optimizations for better speed (e.g., https://github.com/r-lib/cpp11/pull/463 and https://github.com/r-lib/cpp11/pull/430).

Using cpp4r in a packageTo add cpp4r to an existing package, install it first:

remotes::install\_github("pachadotdev/cpp4r")# orpak::pkg\_install("pachadotdev/cpp4r") Then put your C++ files in the src/ directory and add the following to your DESCRIPTION file:

LinkingTo: cpp4r Then decorate C++ functions you want to expose to R with [[cpp4r::register]].

cpp4r is a header only library with no hard dependencies and does not use a shared library, so it is straightforward and reliable to use in packages without fear of compile-time and run-time mismatches.

Alternatively, you can vendor the current installed version of cpp4r headers into your package with cpp4r::vendor(). This ensures the headers will remain unchanged until you explicitly update them.

Getting startedSee the documentation to get started using cpp4r in your scripts, particularly if you are new to C++ programming.

Getting helpPlease open an issue or email me. I will do my best to respond before 48 hours.

ContributingContributions are welcome! Please see the internals vignette for details about design choices and coding style.

Code of ConductPlease note that the cpp4r project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing cpp4r: A C++ Interface for R’s C Interface

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Dear rOpenSci friends, it’s time for our monthly news roundup! You can read this post on our blog. Now let’s dive into the activity at and around rOpenSci!

rOpenSci HQ10 quick tips for making your software outlive your jobOur community manager Yanina Bellini Saibene participated in the paper “10 quick tips for making your software outlive your job”.

“Loss of key personnel has always been a risk for research software projects. Key members of the team may have to step away due to illness or burnout, to care for a family member, from a loss of financial support, or because their career is going in a new direction. Today, though, political and financial changes are putting large numbers of researchers out of work simultaneously, potentially leaving large amounts of research software abandoned. This article presents ten tips to help researchers ensure that the software they have built will continue to be usable after they have left their present job – whether in the course of voluntary career moves or researcher mobility, but particularly in cases of involuntary departure due to political or institutional changes.”

From Ideas to Action: Champions start their trainingThe training phase is off to a strong start!

So far, Champions have taken part in five workshops, including two on Git and GitHub, one on code style – which we opened up not only to Champions and mentors but also to everyone who applied – and two on R package development. Each session has been a chance to learn, share, and grow together as a community. And we’re just getting started: the next workshops will dive into software peer review and community building – key skills for every Champion’s journey!

As usual you can find the materials on our training page.

Request for feedbackWe recently published a blog post requesting feedback for a prototype of an organization-level dashboard for tracking the health and maintenance of an organization’s R packages. We’re still looking for feedback and suggestions, so please read the blog post if you haven’t already, and help us with your ideas.

rOpenSci featured on the Science Ouverte blog of Institut PasteurIn her latest contribution to the Science Ouverte blog, María Gutiérrez Sánchez explores how the rOpenSci Champions Program is helping to foster more open, inclusive, and multilingual science:

The program strengthens scientific communities in Latin America through training, mentorship, and networking around open-source software development. More than just broadening diversity in the R community, the initiative aims to redistribute power in the global open science ecosystem, recognizing that sustainable solutions must emerge from within the communities themselves.

Read the full article in French (original) or Spanish.

CoworkingRead all about coworking!

Tuesday October 7th, 09:00 Americas Pacific (16:00 UTC), “Ship your R package to the R-multiverse!” with Steffi LaZerte and cohosts Will Landau & Maëlle Salmon.
- Learn more about the R-multiverse;
- Submit your package to the R-multiverse;
- Chat with Will Landau, and learn more about the R-multiverse and how to submit your package.
Tuesday November 3rd, 09:00 Australia Western (01:00 UTC), “Code Review with rOpenSci” with Steffi LaZerte and cohost Liz Hare.
- Explore resources for Code review;
- Sign up to volunteer to do software peer-review at rOpenSci;
- Chat with Liz Hare, and learn more about rOpenSci’s code review and the resources available.

And remember, you can always cowork independently on work related to R, work on packages that tend to be neglected, or work on what ever you need to get done!

Software New packagesThe following four packages recently became a part of our software suite:

ssarp, developed by Kristen Martinet: Create Species- and Speciation-Area Relationships using occurrence records or presence-absence matrices. It has been reviewed by Tom Matthews and Joel Nitta.
rixpress, developed by Bruno Rodrigues: Streamlines the creation of reproducible analytical pipelines using default.nix expressions generated via rix for reproducibility. Define derivations in R, Python or Julia, chain them into a composition of pure functions and build the resulting pipeline using Nix as the underlying end-to-end build tool. Functions to plot a DAG representation of the pipeline are included, as well as functions to load and inspect intermediary results for interactive analysis. User experience heavily inspired by the targets package. It has been reviewed by William Michael Landau and Anthony Martinez.
hdcuremodels, developed by Kellie J. Archer together with Han Fu: Provides functions for fitting various penalized parametric and semi-parametric mixture cure models with different penalty functions, testing for a significant cure fraction, and testing for sufficient follow-up as described in Fu et al (2022) and Archer et al (2024). False discovery rate controlled variable selection is provided using model-X knock-offs. It is available on CRAN. It has been reviewed by Tung Lam Nguyen and Panagiotis Papastamoulis.
dataset, developed by Daniel Antal: The dataset package helps create semantically rich, machine-readable, and interoperable datasets in R. It extends tidy data frames with metadata that preserves meaning, improves interoperability, and makes datasets easier to publish, exchange, and reuse in line with ISO and W3C standards. It is available on CRAN. It has been reviewed by Marcelo Perlin, Anna Márta Mester, and Mauro Lepore.

Discover more packages, read more about Software Peer Review.

New versionsThe following fourteen packages have had an update since the last newsletter: sits (v1.5.3-1), c14bazAAR (5.2.0), comtradr (v1.0.4), dataspice (v1.1.1), ghql (v0.1.2), magick (v2.9.0), paleobioDB (v1.0.1), rgbif (v3.8.3), rinat (v0.1.10), rredlist (v1.1.1), rsvg (v2.7.0), spatsoc (v0.2.10), tarchetypes (0.13.2), and targets (1.11.4).

Software Peer ReviewThere are sixteen recently closed and active submissions and 4 submissions on hold. Issues are at different stages:

Two at ‘6/approved’:
- rixpress, Build Reproducible Analytical Pipelines With Nix. Submitted by Bruno Rodrigues.
- SSARP, SSARP (Species-/Speciation-Area Relationship Projector). Submitted by kmartinet.
- One at ‘5/awaiting-reviewer(s)-response’:
- pkgmatch, Find R Packages Matching Either Descriptions or Other R Packages. Submitted by mark padgham.
- Three at ‘4/review(s)-in-awaiting-changes’:
- openFDA, openFDA API. Submitted by Simon Parker.
- mantis, Multiple Time Series Scanner. Submitted by Phuong Quan.
- galamm, Generalized Additive Latent and Mixed Models. Submitted by Øystein Sørensen. (Stats).
- Five at ‘3/reviewer(s)-assigned’:
- babelquarto, Renders a Multilingual Quarto Book. Submitted by Maëlle Salmon.
- reviser, Tools for Studying Revision Properties in Real-Time Time Series Vintages. Submitted by Marc Burri.
- partialling.out, Residuals from partial regressions. Submitted by Marc Bosch. (Stats).
- distionary, Create and Evaluate Probability Distributions. Submitted by Vincenzo Coia, Ph.D., P.Stat..
- read.abares, Provides simple downloading, parsing and importing of Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES) data sources. Submitted by Adam H. Sparks.
- Five at ‘1/editor-checks’:
- rcrisp, Automate the Delineation of Urban River Spaces. Submitted by Claudiu Forgaci. (Stats).
- coevolve, Fit Bayesian Generalized Dynamic Phylogenetic Models using Stan. Submitted by Scott Claessens. (Stats).
- priorsense, Prior Diagnostics and Sensitivity Analysis. Submitted by Noa Kallioinen. (Stats).
- SPARQLchunks, Run SPARQL Chunks and Inline Functions to Retrieve Data. Submitted by André Ourednik.
- capybara, Fast and Memory Efficient Fitting of Linear Models With High-Dimensional. Submitted by Mauricio “Pachá” Vargas Sepúlveda.

Find out more about Software Peer Review and how to get involved.

On the blog* rOpenSci Champions Second Cohort: Projects Wrap-Up by Yanina Bellini Saibene. Our second cohort of Champions completed the program. In this blog post, we share each champion’s projects, their achievements and outreach activities. * From Surviving to Thriving: A Convening to Reclaim and Sustain Open Science Communities by Kari L. Jordan, Erin Becker, Daniela Saderi, Vanessa Fairhurst, Patricia Herterich, Noam Ross, Yanina Bellini Saibene, Leah Wasser, and Yo Yehudi. Five leading open science organizations — The Carpentries, OLS, rOpenSci, pyOpenSci, and PREreview — are coming together for a strategic convening, supported by The Navigation Fund, to address shared challenges of underfunding, fragmentation, and unsustainable volunteer models. * A Gentle Introduction to Open Science by Steffi LaZerte. Steffi reviews a talk on open science. Don’t forget that it takes courage; be kind to yourself. Other languages: Une introduction en douceur à la science ouverte (fr).

Tech Notes* Help Us Design a New Git(Hub) Organizational Dashboard by Mark Padgham. * All the Ways to Programmatically Edit or Parse R Markdown / Quarto Documents by Maëlle Salmon, Christophe Dervieux, and Zhian N. Kamvar. Overview of programmatic ways to analyze and edit Markdown files: Markdown, R Markdown, Quarto, Hugo files, you name it. Other languages: Todas las Formas de Editar o Analizar Programáticamente Documentos R Markdown / Quarto (es), Toutes les façons d’éditer ou d’analyser programmatiquement des documents R Markdown / Quarto (fr). This post was discussed on the R Weekly Highlights podcast hosted by Eric Nantz and Mike Thomas.

Calls for contributionsCalls for maintainersIf you’re interested in maintaining any of the R packages below, you might enjoy reading our blog post What Does It Mean to Maintain a Package?.

USAboundaries (and USAboundariesdata), historical and contemporary boundaries of the United States of America. Issue for volunteering.

Calls for contributionsRefer to our help wanted page – before opening a PR, we recommend asking in the issue whether help is still needed.

Package development cornerSome useful tips for R package developers.

Are your function names unique?Thanks to an idea from Egor Kotov, via https://github.com/ropensci-review-tools/pkgcheck/issues/142, the pkgcheck package now contains a stand-alone function to use during package development, to quickly check whether your function names are unique:

pkgcheck::fn\_names\_on\_cran (c ("min", "max"))#> package version fn\_name#> 161627 matlab2r 1.1.0 max#> 161628 matlab2r 1.1.0 min#> 178817 mosaic 1.8.3 max#> 178821 mosaic 1.8.3 min#> 234203 rapportools 1.1 max#> 234207 rapportools 1.1 min R-universe badge through usethisThe latest version of usethis includes a handy function use_r_universe_badge() that indicates what version of your package is available on R-universe.

New testthat vignettesThe development version of the testthat package features new vignettes including an useful overview of “Testing challenging functions”.

AI newsletter by PositPosit’s Sara Altman and Simon Couch started a newsletter about AI developments both within and outside of their company.

Last wordsThanks for reading! If you want to get involved with rOpenSci, check out our Contributing Guide that can help direct you to the right place, whether you want to make code contributions, non-code contributions, or contribute in other ways like sharing use cases. You can also support our work through donations.

If you haven’t subscribed to our newsletter yet, you can do so via a form. Until it’s time for our next newsletter, you can keep in touch with us via our website and Mastodon account.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci News Digest, September 2025

View Details

[This article was first published on R Works, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Causal Inferencegadjid v0.1.0: Enables efficient Rust implementations of graph adjustment identification distances available in R. These distances (based on ancestor, optimal, and parent adjustment) count how often the respective adjustment identification strategy leads to causal inferences that are incorrect relative to a ground-truth graph when applied to a candidate graph instead. See Henckel, Würtzen, Weichwald (2024) for background and README to get started.

Datadatacommons v0.1.0: Implements a wrapper to the Google Data Commons API V2 that provides programmatic access to statistical and demographic data from dozens of sources organized in a knowledge graph. See the vignette.

imfweo v0.1.0: Provides tools to download, process, and analyze data from the International Monetary Fund’s World Economic Outlook (WEO) database, including functions for downloading complete WEO releases, accessing specific economic indicators for selected countries, and listing available data. See README.

letsRept v1.0.1: Provides tools to retrieve and summarize taxonomic information and synonymy data for reptile species using data scraped from The Reptile Database website. See the vignette.

pubchem.bio v1.0.1: Provides functions to parse the biologically relevant compounds from all PubChem compounds, using biological database sources, pathway presence, and taxonomic relationships. See the vignette.

redlist v0.2.0: Provides an interface to access data from the International Union for Conservation of Nature (IUCN) Red List, allowing users to retrieve up-to-date information on species’ conservation status in support of biodiversity research and conservation efforts. There is an Introduction and a vignette on Visualization.

verdadecu v1.0.0: Provides access to data collected by the Ecuadorian Truth Commission, including datasets documenting human rights violations from 1984-2008, containing victim information, violation types, perpetrators, and geographic distribution. See the vignette.

Differential PrivacyGRANDpriv v0.1.3: Implements a novel method for privatizing network data using differential privacy and includes functions for generating synthetic networks based on a latent space model, applying differential privacy to network latent positions, and evaluating the utility of privatized networks. See Liu, Bi, & Li (2025) for details and README for examples.

Ecologyecodive v2.0.0: Provides functions to compute alpha and beta diversity metrics using concurrent C threads. Metrics include UniFrac, Faith’s phylogenetic diversity, Bray-Curtis dissimilarity, Shannon diversity index. Functions also parse newick trees into phylo objects and rarefy feature tables. There are six vignettes, including an Introduction and Selecting a Beta Diversity Metric.

Environmental Studiesffaframework v0.1.1: Implements the the FFA framework proposed by Vidrio- Sahagún et al. (2024) to support systematic and reproducible workflows for both stationary and nonstationary flood frequency analysis, with applications extending to other hydroclimate extremes, such as precipitation frequency analysis. There are six vignettes, including Change Point Detection and Notstationary FFA.

Epidemiologymlspatial v0.1.0: Provides tools for the integration, visualization, and modelling of spatial epidemiological data using the method described in Azeez & Noel (2025). Users can import and pre-process shapefiles and associated demographic or disease incidence data, generate richly annotated thematic maps, and apply predictive models to identify spatial patterns and risk factors. See the vignette.

Geologystratcols v1.0.0: Provides tools to quantify stratigraphic disorder using the metrics defined by Burgess (2016), including a range of utility tools to construct and manipulate stratigraphic columns. There are three vignettes, including an Introduction and Quantifying stratigraphic order.

Geneticstidypopgen v0.3.2: Provide a tidy grammar of population genetics, facilitating the manipulation and analysis of data on biallelic single nucleotide polymorphisms (SNPs). Functions scale to very large genetic datasets by storing genotypes on disk, and performing operations on them in chunks, without ever loading all data in memory. See Carter et al. (2025) for details. There are five vignettes, including an Introduction and The grammar of population genetics.

Genomicsibdfindr v0.3.1: Implements continuous-time hidden Markov models (HMMs) to infer identity-by-descent (IBD) segments shared by two individuals from their single-nucleotide polymorphism (SNP) genotypes. Provides posterior probabilities at each marker (forward-backward algorithm), prediction of IBD segments (Viterbi algorithm), and functions for visualizing results. Supports both autosomal data and X-chromosomal data. See README for examples.

readyomics v0.1.2: Provides a flexible and streamlined pipeline for formatting, analyzing, and visualizing omics data (e.g., transcriptomics, proteomics, metabolomics) and includes tools for shaping input data, fitting linear or mixed-effect models, extracting key contrasts, and generating publication-quality plots. See the vignette.

Health Technology AssessmentsearchAnalyzeR v0.1.0: Provides comprehensive analytics, reporting, and testing capabilities for systematic review search strategies, focusing on validating search performance, generating standardized PRISMA-compliant reports, and ensuring reproducibility in evidence synthesis. Features include precision-recall analysis, cross-database performance comparison, benchmark validation against gold standards, sensitivity analysis, temporal coverage assessment, automated report generation, and statistical comparison of search strategies. See the vignettes Comparing Search strategies and Real PubMed Search Analysis.

Machine LearningLCCkNN v0.1.0: Implements the kK-NN algorithm, an adaptive k-nearest neighbor classifier that adjusts the neighborhood size based on local data curvature. The method estimates local Gaussian curvature by approximating the shape operator of the data manifold. This approach aims to improve classification performance, particularly in datasets with limited samples. Look here for examples.

Medical StatisticsE2E v0.1.0: Provides a framework to build and evaluate diagnosis or prognosis models using stacking, voting, and bagging ensemble techniques with various base learners. There are five vignettes, including Getting Started and Advanced Features & Customization.

kfre v0.0.2: Implements the Kidney Failure Risk Equation Tangri et al. (2011) and Tangri et al. (2016) to compute two and five year kidney failure risk. Includes helper functions to classify chronic kidney disease (CKD) stages and end-stage renal disease (ESRD) outcomes, and evaluate and plot model performance. See the [vignette]https://cran.r-project.org/web/packages/kfre/vignettes/getting-started.html) to get started.

metaSVR v0.1.0: Provides a hybrid modeling framework combining Support Vector Regression with metaheuristic optimization algorithms, including the [Archimedes Optimization Algorithm], Hashim et al. (2021); Coot Bird Optimization, Naruei & Keynia (2021); as well as several others. Functions enable automatic tuning of SVR hyperparameters (cost, gamma, and epsilon) to enhance prediction performance. See Setiawan et al. (2021) and Liu et al. (2018) for implementation details and README for an example.

StatisticsAdapDiscom v1.0.0: Implements adaptive direct sparse regression for high-dimensional multimodal data with heterogeneous missing patterns and measurement errors. The method supports flexible block configurations and includes robust variants for heavy-tailed distributions and fast implementations for large-scale applications. See Diakité et al. (2025) for background and the vignette to get started.

allocation v0.1.0: Implements several exact methods for allocating optimal sample sizes when designing stratified samples. These methods are discussed in Wright (2012) and Wright (2017). See the vignette.

BSTFA v0.1.0: Implements Bayesian spatio-temporal factor analysis models for multivariate data observed across space and time. The package provides MCMC tools for model fitting, spatial and temporal interpolation, and visualization of latent factors and loadings. See Berrett et al. (2020) and the vignette for examples.

calms v1.0-1: Provides a Shiny application to conduct comprehensive analysis of latent means including the examination of group equivalency, propensity score analysis, measurement in variance analysis, and assessment of latent mean differences of equivalent groups with invariant data. Ho et al. (2011), Rosseel (2012), and Hammack-Brown et al. (2021) for background and the vignette for examples.

CCI v0.3.4: Implements a tool for performing computational testing for conditional independence between variables in a dataset by using permutations in combination with Monte Carlo cross-validation to generate null distributions and test statistics. See Thorjussen et al. (2024) for details and README for examples.

ExactVaRTest v0.1.3: Provides fast dynamic-programming algorithms in C++/Rcpp for the exact finite-sample distributions and p-values of Christoffersen (1998) independence and conditional-coverage VaR back tests. For completeness, it also provides the exact unconditional-coverage test following Kupiec (1995) via a closed-form binomial enumeration. See the vignette.

propertee v1.0.1: Provides functions to facilitate direct adjustment for experiments and observational studies that are compatible with a range of study designs and covariance adjustment strategies. Functions use explicit specification of clusters, blocks, and treatment allocations to furnish probability of assignment-based weights targeting any of several average treatment effect parameters. There are three vignettes, including Introduction and Regression Discontinuity Design.

senseweight v0.0.1: Provides tools to conduct interpretable sensitivity analyses for weighted estimators to evaluate the sensitivity of the underlying weighting estimators to omitted moderators or confounders. See Huang (2024) and Hartman and Huang (2024) for background. There are two vignettes External Validity and Survey.

WData v0.1.0: Provides functions to estimate the density and cumulative distribution functions from biased samples of continuous distributions. Includes the estimators proposed by Bhattacharyya et al. (1988) and Jones (1991) for density, and Bose and Dutta (2022) for distributions. See README for examples.

SurveysCATAcode v1.0.0: Provides tools to assist with data transformations and subjective decisions often required on “check-all-that-apply” surveys. Includes functions for exploring response patterns, data transformations, applying a set of decision rules for coding responses, and summarizing response frequencies. See the vignette.

Time Seriesdsp v1.2.0: Provides efficient MCMC algorithms for dynamic shrinkage processes, which extend global-local shrinkage priors to the time series setting by allowing shrinkage to depend on its own past. These priors yield locally adaptive estimates, useful for time series and regression functions with irregular features. See Kowal et al. (2019), Wu et al. (2024), and Cho and Matteson (2024) for background on the methods implemented and README to get started.

unfold v1.0.0: Implements an algorithm that reframes time series forecasting as a geometric learning problem through variational mapping that expands future temporal dynamics from folded high-dimensional geometric distance spaces. It turns time series into a 4D block of pairwise distances between reframed windows and maps those distances to the next reframed window to produce predictive functions. Look here for the details.

ToxicologyinvitroTKstats v0.0.13: Provides tools for processing and analyzing in vitro toxicokinetic measurements in a standardized and reproducible pipeline. Functions perform frequentist and Bayesian estimation on a variety of in vitro toxicokinetic measurements, including chemical fraction unbound in the presence of plasma (f_up), intrinsic hepatic clearance (Clint, uL/min/million hepatocytes), and membrane permeability for oral absorption (Caco2). See Wambaugh et al. (2019) for background. There are seven vignettes, including Data Guide and Clint.

UtilitiesCBTF v0.5.0: Provides simple runner (the mechanism responsible for executing the fuzzing process) for fuzz-testing functions in an R package’s public interface. Fuzz testing helps identify functions lacking sufficient argument validation, and uncovers problematic inputs that, while valid by function signature, may cause issues within the function body. See README for examples.

multideploy v0.1.1: Provides tools to deploy file changes across multiple GitHub repositories using the GitHub Web API, allow synchronizing common files, and continuous Integration workflows. See the vignette.

Visualizationggdiagram v0.1.1: Provides object oriented functions to create diagrams with geometric objects that have computed properties with information about themselves (e.g., their area) or about their relationships with other objects (e.g, the distance between their edges). See the vignette.

ggdiceplot v0.1.0: Provides ggplot2 extensions, including geom_dice() for creating dice-based visualizations where each dot position represents a specific categorical variable. Look here for examples.

gridify v0.7.4: Implements a tool to create enriched figures and tables by providing a way to add text around them through predefined or custom layouts. Any input such as ggplot, gt, or flextable which is convertible to grob. Functions are based grid graphics. See Murrell (2018) for details. There are five vignettes, including Get Started and Multi-Page and Multi-Figure Examples.

terminalgraphics v0.1.1: Defines a graphics device and functions for graphical output in terminal emulators that support graphical output. Currently supports the Terminal Graphics Protocol and Sixel. Look here for examples.

TheseusPlot v0.1.1: Provides tools for decomposing and visualizing differences in rate metrics between two groups into contributions from individual subgroups. A Theseus plot combines the stepwise progression of a waterfall plot with the comparative bars of a bar chart, offering an intuitive way to understand subgroup-level effects. See README for examples.

To leave a comment for the author, please follow the link and comment on their blog: R Works.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: August 2025 Top 40 New CRAN Packages

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

You can send me questions for the blog using this form and subscribe to receive an email when there is a new post.

I have created a new R package called spuriouscorrelations that aims to help educators to explain why correlation does not imply causation. What I had in mind were AP Statistics courses and college-level introductory statistics courses.

The package includes a dataset with 15 spurious correlations. You can install it from CRAN with:

```

install.packages("spuriouscorrelations", repos = "https://cran.rstudio.com")library(spuriouscorrelations)

``` Let’s plot one of the spurious correlations, for example, the correlation between the number of people who drowned by falling into a pool and the number of films Nicolas Cage appeared in:

library(dplyr)library(ggplot2)spurious\_correlations %>% distinct(var1\_short, var2\_short)

```

A tibble: 15 × 2 var1_short var2_short 1 US spending on science Suicides 2 Falling into a pool drownings Nicholas Cage 3 Cheese consumed Bedsheet tanglings 4 Divorce rate in Maine Margarine consumed 5 Age of Miss America Murders by steam 6 Arcade revenue Computer science doctorates 7 Space launches Sociology doctorates 8 Mozzarella cheese consumption Engineering doctorates 9 Fishing boat deaths Kentucky marriages 10 US crude oil imports from Norway Railway train collisions 11 Chicken consumption US crude oil imports 12 Swimming pool drownings Nuclear power plants 13 Japanese cars sold Suicides by crashing 14 Spelling bee letters Spider deaths 15 Math doctorates Uranium stored

```

nic\_cage <- filter(spurious\_correlations, var2\_short == "Nicholas Cage")cor(nic\_cage$var1\_value, nic\_cage$var2\_value)

[1] 0.6660043

ggplot(nic\_cage, aes(x = var1\_value, y = var2\_value)) + geom\_point(size = 3) + geom\_smooth(method = "lm", se = FALSE, color = "blue") + labs( title = "Spurious Correlation: Drownings vs. Nicolas Cage Films", x = "Number of Drownings by Falling into a Pool", y = "Number of Films Nicolas Cage Appeared In" ) + theme\_minimal() With a correlation of 67%, we can even fit a linear model:

lm\_model <- lm(var2\_value ~ var1\_value, data = nic\_cage)summary(lm\_model)

Call:lm(formula = var2\_value ~ var1\_value, data = nic\_cage)Residuals: Min 1Q Median 3Q Max -0.9308 -0.5926 -0.1020 0.4836 1.6026 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -5.37515 2.86726 -1.875 0.0936 .var1\_value 0.07620 0.02845 2.678 0.0253 *---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Residual standard error: 0.8678 on 9 degrees of freedomMultiple R-squared: 0.4436, Adjusted R-squared: 0.3817 F-statistic: 7.174 on 1 and 9 DF, p-value: 0.02527 With a explained variance of 38% (adjusted R-squared), we can say that the number of drownings is a statistically significant predictor of the number of films Nicolas Cage appeared in. However, this is a spurious correlation, and there is no causal relationship between these two variables, even with a p-value of 0.025.

Now let’s compare with a double y-axis plot:

```

install.packages("tintin", repos = "https://cran.rstudio.com")library(tidyr)library(tintin)cor_val <- cor(nic_cage$var1_value, nic_cage$var2_value)# Align the two series visuallyv1 <- nic_cage$var1_valuev2 <- nic_cage$var2_valuefun_adjust <- function(v1, v2) { s1 <- sd(v1, na.rm = TRUE) s2 <- sd(v2, na.rm = TRUE) a <- ifelse(s2 == 0, 1, s1 / s2) b <- mean(v1, na.rm = TRUE) - a * mean(v2, na.rm = TRUE) c(a = as.numeric(a), b = as.numeric(b))}adjust <- fun_adjust(v1, v2)scale_a <- adjust["a"]scale_b <- adjust["b"]y1_title <- as.character(unique(nic_cage$var1))y2_title <- as.character(unique(nic_cage$var2))nic_cage_long <- nic_cage %>% select(year, var1_value, var2_value) %>% pivot_longer( cols = c(var1_value, var2_value), names_to = "variable", values_to = "value" ) %>% mutate( variable_label = case_when( variable == "var1_value" ~ y1_title, variable == "var2_value" ~ y2_title ), # apply transform to var2 for plotting: plot_value = a * var2 + b plot_value = ifelse(variable == "var2_value", value * scale_a + scale_b, value) )# make a double y axis plot with year on the x axisggplot(nic_cage_long, aes(x = year)) + geom_line(aes(y = plot_value, color = variable_label, group = variable_label), linewidth = 1.5) + geom_point(aes(y = plot_value, color = variable_label), size = 3) + labs( x = "Year", y = y1_title, title = sprintf("%s\nvs\n%s\n", y1_title, y2_title), subtitle = sprintf("Correlation: %.2f", cor_val), color = "" ) + # display all years on the x axis scale_x_continuous(breaks = nic_cage$year) + # primary y axis is the var1 scale # secondary shows var2 original scale by inverse-transforming scale_y_continuous( sec.axis = sec_axis(~ (. - scale_b) / scale_a, name = y2_title) ) + theme_minimal(base_size = 13) + theme(legend.position = "top") + # use tintin color palette scale_colour_manual( values = tintin_pal(option = "the black island")(2), name = "" ) + # center title and subtitle theme( plot.title = element_text(hjust = 0.5, size = 16, face = "bold"), plot.subtitle = element_text(hjust = 0.5) )

``` How about other spurious correlations? Here is one:

engineering\_doctorates <- filter(spurious\_correlations, var2\_short == "Engineering doctorates")cor\_val <- cor(engineering\_doctorates$var1\_value, engineering\_doctorates$var2\_value)v1 <- engineering\_doctorates$var1\_valuev2 <- engineering\_doctorates$var2\_valueadjust <- fun\_adjust(v1, v2)scale\_a <- adjust["a"]scale\_b <- adjust["b"]y1\_title <- as.character(unique(engineering\_doctorates$var1))y2\_title <- as.character(unique(engineering\_doctorates$var2))engineering\_doctorates\_long <- engineering\_doctorates %>% select(year, var1\_value, var2\_value) %>% pivot\_longer( cols = c(var1\_value, var2\_value), names\_to = "variable", values\_to = "value" ) %>% mutate( variable\_label = case\_when( variable == "var1\_value" ~ y1\_title, variable == "var2\_value" ~ y2\_title ), # apply transform to var2 for plotting: plot\_value = a * var2 + b plot\_value = ifelse(variable == "var2\_value", value * scale\_a + scale\_b, value) )# make a double y axis plot with year on the x axisggplot(engineering\_doctorates\_long, aes(x = year)) + geom\_line(aes(y = plot\_value, color = variable\_label, group = variable\_label), linewidth = 1.5) + geom\_point(aes(y = plot\_value, color = variable\_label), size = 3) + labs( x = "Year", y = y1\_title, title = sprintf("%s\nvs\n%s\n", y1\_title, y2\_title), subtitle = sprintf("Correlation: %.2f", cor\_val), color = "" ) + # display all years on the x axis scale\_x\_continuous(breaks = nic\_cage$year) + # primary y axis is the var1 scale # secondary shows var2 original scale by inverse-transforming scale\_y\_continuous( sec.axis = sec\_axis(~ (. - scale\_b) / scale\_a, name = y2\_title) ) + theme\_minimal(base\_size = 13) + theme(legend.position = "top") + # use tintin color palette scale\_colour\_manual( values = tintin\_pal(option = "the black island")(2), name = "" ) + # center title and subtitle theme( plot.title = element\_text(hjust = 0.5, size = 16, face = "bold"), plot.subtitle = element\_text(hjust = 0.5) ) I can go ad nauseam with these spurious correlations. The point is that correlation does not imply causation, and we should be careful when interpreting correlations.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Spurious Correlations in R – Correlation is not Causation

View Details

[This article was first published on rstats-tips.net, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Unformatunately Piwik Pro has discontinued their free plan as of the end of 2025.If I wanted to user Piwik Pro I would have to pay at least 420,00 € per year.

As the author of the CRAN-hosted R-package piwikproRI have to decide whether I should continue maintaining the package or not.

My employer still uses Piwik Pro and I will continue using the package for my workbut I can’t guarentee that I will continue maintaining the package becauseI can’t test it anymore at home.

To leave a comment for the author, please follow the link and comment on their blog: rstats-tips.net.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Piwik Pro doesn’t offer a free plan anymore

View Details

[This article was first published on r on Everyday Is A School Day, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Learned RNA-seq workflow using C. difficile data from a published study . Processed raw reads through fastp → kallisto → DESeq2 pipeline. Results matched the original paper’s findings, with clear differential expression between mucus and control conditions .

Motivations: To be honest, I don’t really know what RNA-seq was until I learnt more about it and its potential! On our previous learning processes, we’ve looked at Learning Antimicrobial Resistance (AMR) genes with Bioconductor, Phylogenetic Analysis, Building DNA sequence Alignment, Assemblying DNA Sequence, Learning BLAST and MLST, Exploring Long Sequence ONT Workflow. Notice that all these are DNA related. This article, we’ll explore the world of RNA! In my simplistic view, RNA-seq enables us to explore and discover transcriptomes of certain conditions, whether it can tell us a bit more of the gene expression based on certain conditions. The potential is huge! Imagine if you can tell the difference between colonization vs a true infection in a clinical setting, I wonder if differential expression analysis of transcriptome can provide us a bit more information! Differentiating infection vs contamination (?more accurate HAI definition ). Let’s dive into the shallow pool of the unknown world of RNA-Seq and at least learn the basics! Let’s go!

Game Plan: Let’s look at Clostridioides difficile and see if we can find any raw cDNA sequences on ncbi. After some searching, I found this Clostridioides difficile-mucus interactions encompass shifts ingene expression, metabolism, and biofilm formation that might be potentially a good one to look at and see if we can somewhat reproduce what’s found.

Disclaimer: I am not a biostatistician, neither do I work in a lab. I’m attempting to understand the bioinformatics workflow of an RNA-seq. If you find any information displayed here is wrong, please let me know so I can learn. Please verify the information presented here as well. This is a documentation of my learning process and a note for my future self to reproduce the workflow I’ve explored

Objectives: * What is RNA-Seq? + Let’s Look At Existing Data * The Workflow + QC of raw read + Getting Reference of Transcriptome + Assemble + PCA + DESeq2 + Volcano plot * Opportunities for improvement * Lessons Learnt

What is RNA-Seq? RNA-Seq, or RNA sequencing, is a powerful technique used to analyze the transcriptome of a cell or organism. The transcriptome refers to the complete set of RNA molecules, including messenger RNA (mRNA), ribosomal RNA (rRNA), transfer RNA (tRNA), and non-coding RNAs, that are present in a cell at a specific time. RNA-Seq allows researchers to study gene expression patterns, identify differentially expressed genes, and discover novel transcripts. read more on wiki. It’s usually challenging to sequence from RNA itself due to its instability and susceptibility to degradation. Therefore, RNA is typically converted into complementary DNA (cDNA) using a process called reverse transcription before sequencing. This cDNA is then used as the input for sequencing platforms. However, for short RNA sequence, nowadays with ONT, it does provide flowcells that can directly sequence RNA without the need for conversion to cDNA.

Let’s Look At Existing Data Let’s dive into the bioproject. To download the raw sequence, we need to go through several clicks. And also understand what some of these abbreviations means.

Sequence Read Archive (SRA):
SRR – Individual sequencing runs (raw data files).
SRX – Experiments (groups related runs).
SRS – Samples (biological specimens).
SRP – Projects/Studies (collections of related experiments).

Gene Expression Omnibus (GEO):
GSM – Individual samples with expression data.
GSE – Series/datasets (collections of related samples).
GPL – Platforms (microarray or sequencing platform descriptions).
GDS – Curated datasets (processed GSE data).

Assembly Database:
ASM – Genome assemblies.
GCA – GenBank assemblies.
GCF – RefSeq assemblies.

Core Sequence Databases:
NC_ – RefSeq chromosomes/complete genomes.
NM_ – RefSeq mRNA sequences.
NP_ – RefSeq protein sequences.
XM_/XP_ – Model/predicted RefSeq sequences.
AC_ – GenBank finished genomic sequences.

BioProject & BioSample:
PRJNA – BioProject accessions (research projects).
SAMN – BioSample accessions (biological specimens).

Alright, with that out of the way, we’re interested in SRR, which should be under SRA. We can see on the bioproject page that there is SRA experiments. Click on that and it’ll bring you to a page of SRXs. Since they are very BIG files, in terms of gigs, we can’t just download from the website, we need sra-tools

```

Install follow this link instruction ## https://github.com/ncbi/sra-tools/wiki/02.-Installing-SRA-Toolkit## download SRRs examplefasterq-dump --progress --threads 4 SRR27792604

`` Now, repeat the above for the otherSRRsand you’ll get all the raw sequences infastqformat. With our practice, we’ll only downloadSRR27792597,SRR27792603,SRR27792598, andSRR27792604. Two mucus and two controls. It will take sometime, which is why i like the--progress` parameter. Alright, after we’ve downloaded the sequences, let’s move on to our official workflow.

files <- c("kallisto\_SRR27792597/abundance.h5", "kallisto\_SRR27792603/abundance.h5", "kallisto\_SRR27792598/abundance.h5", "kallisto\_SRR27792604/abundance.h5") The Workflow

TL;DR Quality control (fastp) -> Create transcriptome reference (kallisto) -> Assemble Raw Sequence (kallisto) -> Analyse (PCA & DESeq2)

QC of raw read ```

install fastp# run QCfastp -i SRR27792597_1.fastq -h SRR27792597_1.html --verbose --stdout --thread 8 > /dev/null

`` If you’re like me who likes to know what’s going on when running, make sure to include--verboseand I found--threadis very helpful, otherwise it took sometime. Repeat for allfastq` files and inspect.

What is Considered Acceptable? I’ll be honest, I’m not sure. According to Claude, Read Quality Metrics such as Quality Scores (Phred scores):Q30+: >80% is good, >90% is excellent, Q20+: Should be >95%. Total Reads: Depends on our goals: Differential expression: 10-30M reads per sample, whereas Novel transcript discovery: 50-100M+ reads. Contamination and Artifacts: Adapter Content: <5%: Good, 5-20%: Moderate (should trim), >20%: High contamination. Duplication Rate: <30%: Low (genomic DNA-like), 30-70%: Normal for RNA-seq, >70%: Potentially problematic. Sequence Composition:GC Content: Should match expected organism: Human/Mouse: ~42%, E. coli: ~50%, Yeast: ~38%; Per-base Quality: Should stay >Q28 across read length. Red Flags to Watch For Poor quality data: <10M reads for differential expression<80% Q20 bases, Severe 3’ quality drop (below Q20), Unusual GC content spikes, 20% adapter contamination after trimming.

Our QC report is not included, but a quick glance, it looks pretty good!

Getting Reference of Transcriptome On the article’s supplement, we found that they used FN545816.1 as reference genome. We can download them from here. Click download and include gff as well.

```

Install gffread if you don't have itbrew install gffread # on macOS# install kallisto# go here https://pachterlab.github.io/kallisto/download.html for info# extract transcriptome sequencesgffread -w transcriptome.fa -g GCA_000027105.1_ASM2710v1_genomic.fna genomic.gffkallisto index -i transcriptome.idx transcriptome.fa

`` You should havetranscriptome.idxandtranscriptome.fain your current working directory. What this does is that it extracts the transcriptome sequences from the genome based on thegfffile, in our case it’s a specificClostridioides difficile R20291assembly with accessionFN545816.1. Thegfffile contains information about the locations of genes and other features on the genome. The-wflag specifies the output file for the transcriptome sequences, and the-gflag specifies the input genome sequence file. The resultingtranscriptome.fafile contains the nucleotide sequences of all transcripts annotated in thegff` file.

Now that we have the reference. Let’s assemble our raw sequence to something readable.

Assemble ```

assemble, make sure to include all the other sampleskallisto quant -i transcriptome.idx \ -o kallisto_SRR27792604 \ --threads 8 \ SRR27792604_1.fastq SRR27792604_2.fastq

`` We’d have to go through all the raw sequences we downloaded and assemble them. We’ll then see folders such askallisto_SRR27792604. And inside the folder, we seeabundance.h5andabundance.tsv. The reason i preferkallistooverSTAR` is mainly because of the speed. When I first tried STAR it was very slow and no progress bar. The output of these files are actually quite small. Let’s take a look!

library(tximport)library(DESeq2)# Create file paths for all samplesfiles <- c("kallisto\_SRR27792597/abundance.h5", "kallisto\_SRR27792603/abundance.h5", "kallisto\_SRR27792598/abundance.h5", "kallisto\_SRR27792604/abundance.h5")# Name themnames(files) <- c("mucus\_rep1", "control\_rep1", "mucus\_rep2", "control\_rep2")# Import datatxi <- tximport(files, type = "kallisto", txOut = TRUE)coldata <- data.frame( condition = c("mucus", "control", "mucus", "control"), # or however they're grouped row.names = names(files))dds <- DESeqDataSetFromTximport(txi, colData = coldata, design = ~ condition) Just as a note, there are another way we can create from matrix and count as well using tsv.

PCA ```

Perform variance stabilizing transformationvsd <- vst(dds, blind = FALSE)# Plot PCAplotPCA(vsd, intgroup = "condition")

``` Wow, it looks like the treatment and control are separated quite well! Let’s dive straight into differential expression analysis. If we look at figure 2B on the article, it looks very similar, even though we haven’t include the full sequences.

DESeq2 library(tidyverse)# Run DESeq2 analysisdds <- DESeq(dds)# Get resultsres <- results(dds, alpha = 0.01, tidy = T)# View summarysummary(res)## row baseMean log2FoldChange lfcSE ## Length:3570 Min. : 0 Min. :-8.42507 Min. :0.06850 ## Class :character 1st Qu.: 326 1st Qu.:-0.51037 1st Qu.:0.08493 ## Mode :character Median : 1313 Median : 0.00074 Median :0.10664 ## Mean : 12154 Mean :-0.01934 Mean :0.16100 ## 3rd Qu.: 4999 3rd Qu.: 0.53025 3rd Qu.:0.16896 ## Max. :3272163 Max. : 3.75721 Max. :4.98958 ## NA's :1 NA's :1 ## stat pvalue padj ## Min. :-80.75478 Min. :0.0000000 Min. :0.0000000 ## 1st Qu.: -4.39763 1st Qu.:0.0000000 1st Qu.:0.0000000 ## Median : 0.00566 Median :0.0001034 Median :0.0002068 ## Mean : -0.71728 Mean :0.1229994 Mean :0.1366817 ## 3rd Qu.: 3.44035 3rd Qu.:0.0970157 3rd Qu.:0.1293422 ## Max. : 28.59200 Max. :0.9978324 Max. :0.9978324 ## NA's :1 NA's :1 NA's :1# positiveres |> filter(log2FoldChange > 0) |> arrange(padj, desc(log2FoldChange)) |> head(10)## row baseMean log2FoldChange lfcSE stat## 1 gene-CDR20291\_1626 2872.357 2.999390 0.10490313 28.59200## 2 gene-CDR20291\_2014 32895.724 2.043527 0.07250627 28.18414## 3 gene-CDR20291\_0509 54339.793 1.925179 0.07046733 27.32016## 4 gene-CDR20291\_2174 13209.067 1.957986 0.07539631 25.96926## 5 gene-CDR20291\_2495 24664.096 2.321013 0.08978192 25.85168## 6 gene-CDR20291\_0508 8811.393 2.021692 0.07967500 25.37423## 7 gene-CDR20291\_2738 9037.752 1.859512 0.07781080 23.89787## 8 gene-CDR20291\_1446 4410.271 1.982683 0.08581234 23.10487## 9 gene-CDR20291\_2871 3862.148 1.912250 0.08437678 22.66322## 10 gene-CDR20291\_2017 5272.620 1.784124 0.08208607 21.73479## pvalue padj## 1 8.448830e-180 1.076924e-177## 2 9.148740e-175 1.053286e-172## 3 2.444054e-164 2.643282e-162## 4 1.102039e-148 9.832939e-147## 5 2.329704e-147 2.027979e-145## 6 4.856404e-142 3.939206e-140## 7 3.223217e-126 2.170502e-124## 8 4.136645e-118 2.636372e-116## 9 1.033340e-113 6.250829e-112## 10 9.621927e-105 5.283178e-103# negativeres |> filter(log2FoldChange < 0) |> arrange(padj, log2FoldChange) |> head(10)## row baseMean log2FoldChange lfcSE stat pvalue## 1 gene-CDR20291\_3145 195320.938 -8.425071 0.10432907 -80.75478 0## 2 gene-CDR20291\_2142 14441.266 -5.528640 0.08551642 -64.65004 0## 3 gene-CDR20291\_1557 21279.449 -4.484919 0.11271527 -39.78981 0## 4 gene-CDR20291\_0332 11550.169 -4.080126 0.08702073 -46.88683 0## 5 gene-CDR20291\_0331 4196.984 -3.907433 0.09507966 -41.09641 0## 6 gene-CDR20291\_2206 6931.606 -3.821015 0.08700146 -43.91897 0## 7 gene-CDR20291\_1078 28192.247 -3.531056 0.08204064 -43.04033 0## 8 gene-CDR20291\_2237 6355.265 -3.491135 0.08384686 -41.63704 0## 9 gene-CDR20291\_0330 6786.174 -3.481683 0.08283612 -42.03097 0## 10 gene-CDR20291\_2238 6034.666 -3.430734 0.08338778 -41.14193 0## padj## 1 0## 2 0## 3 0## 4 0## 5 0## 6 0## 7 0## 8 0## 9 0## 10 0 Interesting thing about the pvalue, we need to look at adjust pval instead because we are doing multiple comparison. Adjust pvalue is via Benjamini-Hochberg method. This adjust the multiplicity of comparison.

Volcano plot library(ggplot2)library(ggrepel)# Define specific genes to labelgenes\_to\_label <- c("gene-CDR20291\_1626", "gene-CDR20291\_0508", "gene-CDR20291\_1446", "gene-CDR20291\_2495", "gene-CDR20291\_0455", "gene-CDR20291\_0876", "gene-CDR20291\_0877", "gene-CDR20291\_1275", "gene-CDR20291\_0875", "gene-CDR20291\_3145")# Create labeled volcano plotres |> filter(!is.na(padj)) |> mutate(label = ifelse(row %in% genes\_to\_label, str\_extract(row, "\\d+$"), "")) |> mutate(color = case\_when( log2FoldChange < -1 & padj <= 0.01 ~ "negative", log2FoldChange > 1 & padj <= 0.01 ~ "positive", TRUE ~ "neutral" )) |> ggplot(aes(x = log2FoldChange, y = -log10(padj), color = color)) + geom\_point(alpha = 0.2) + geom\_vline(xintercept = c(-1, 1), linetype = "dashed") + geom\_hline(yintercept = -log10(0.01), linetype = "dashed") + geom\_text\_repel(aes(label = label), size = 5, # box.padding = 0.3, max.overlaps = 20) + labs(title = "Volcano Plot - Mucus vs Control", x = "Log2 Fold Change", y = "-Log10 Adjusted P-value") + scale\_color\_manual(values = c("positive" = "red", "negative" = "blue", "neutral" = "grey")) + theme\_bw() + theme(legend.position = "none") If we look at figure 2A, it again looks very similar! To interpret the volcano plot, we basically look at the the top right and top left of the plot. These are the ones that are significantly differentially expressed. The top right are the upregulated genes, whereas the top left are the downregulated genes. We can see that gene-CDR20291_1626 is the most upregulated gene, whereas gene-CDR20291_3145 is the most downregulated gene.

Looking at NCBI, looks like 1626 is a putative sodium/phosphate cotransporter [Clostridioides difficile R20291]. And 3145 is probable protease. What this means is that Cdiff when exposed to mucus when compared to control, the putative sodium/phosphate cotransporter expressed gene was found more (in mucus group), whereas the probable protease expressed gene was found less (in mucus group).

Opportunities for improvement * I should probably rewrite the above to a script and with function so that in the future we can easily reproduce any DE analysis * include esearch or entrez to get accession metadata for more accurate label * include heatmap * pathway analysis. Yes, we can DE analyze these genes, but what do they actually mean? Do they use nutrients differently etc.

Lessons Learnt * learnt the basics of sra-tools fasterq-dump,fastp, kallisto,DESeq2. * took me a while to find the right reference isolate. Found in on their supplement material and got the right one. For future reference, don’t assume they use the popular refseq, look through their procedure and get that specifically. * learnt from raw RNA-seq QC * learnt to interpret volcano plot

If you like this article:

please feel free to send me a comment or visit my other blogs
please feel free to follow me on BlueSky, twitter, GitHub or Mastodon
if you would like collaborate please feel free to contact me

To leave a comment for the author, please follow the link and comment on their blog: r on Everyday Is A School Day.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Learning And Exploring The Workflow of RNA-Seq Analysis – A Note To Myself

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

You can send me questions for the blog using this form and subscribe to receive an email when there is a new post.

This video covers the basics of R and C++ syntax and why data types matter a lot on C++ side while R is more permissive. The code is available here. I had to record again because of an audio issue.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Creating a simple R package with C++ code to sum “a + b” using different data types

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.China aims to increase its influence in the global bullion market by directing friendly countries to store their gold reserves within its borders. This move is part of Beijing’s efforts to reduce its reliance on the dollar and promote the global use of the yuan.

Goldman Sachs predicts that if just 1% of corporate bonds shift to gold, prices could rise to $5,000. However, according to the XGBoost model, both gold and silver near the upper bands suggest that it’s not a good time to buy at those levels.

Source code:

library(tidymodels)library(tidyverse)library(tidyquant)library(timetk)library(modeltime)#Gold Futures (GC=F)df\_gold <- tq\_get("GC=F") %>% select(date, gold = close)#Silver Futures (SI=F)df\_silver <- tq\_get("SI=F") %>% select(date, silver = close)#Creating the survey datadf\_survey <- df\_gold %>% left\_join(df\_silver) %>% pivot\_longer(-date, names\_to = "id", values\_to = "value") %>% mutate(id = toupper(id)) %>% filter(date >= last(date) - months(36)) %>% drop\_na()#Train/Test Splittingsplits <- df\_survey %>% time\_series\_split(assess = "15 days", cumulative = TRUE)#Recipe#The step\_normalize() function is breaking the decision splits.#Reducing the model's accuracy led to its removal.rec\_spec <- recipe(value ~ ., training(splits)) %>% step\_string2factor("id") %>% step\_mutate\_at(id, fn = droplevels) %>% step\_timeseries\_signature(date) %>% step\_rm(date) %>% step\_dummy(all\_nominal\_predictors(), one\_hot = TRUE) %>% step\_zv(all\_predictors()) %>% step\_corr(all\_predictors())#Preprocessed data variablesrec\_spec %>% prep() %>% bake(new\_data = NULL) %>% glimpse() #Workflow fitwflw\_fit <- workflow() %>% add\_model( boost\_tree("regression") %>% set\_engine("xgboost") ) %>% add\_recipe(rec\_spec) %>% fit(training(splits))#Create a Modeltime Tablemodel\_tbl <- modeltime\_table(wflw\_fit)#Calibrating by IDcalib\_tbl <- model\_tbl %>% modeltime\_calibrate( new\_data = testing(splits), id = "id" )#Measuring Test Accuracy#Global Accuracycalib\_tbl %>% modeltime\_accuracy(acc\_by\_id = FALSE) %>% table\_modeltime\_accuracy(.interactive = FALSE)#Local Accuracycalib\_tbl %>% modeltime\_accuracy(acc\_by\_id = TRUE) %>% table\_modeltime\_accuracy(.interactive = TRUE)#Prediction intervals were used similarly to the Relative Strength Index (RSI).calib\_tbl %>% modeltime\_forecast( new\_data = testing(splits), actual\_data = testing(splits), conf\_by\_id = TRUE) %>% group\_by(id) %>% plot\_modeltime\_forecast( .facet\_ncol = 1, .interactive = FALSE, .line\_size = 1.5 ) + labs(title = "Global Modeling with XGBoost", subtitle = "<span style = 'color:dimgrey;'>Predictive Intervals</span> of <span style = 'color:red;'>XGBoost</span>", y = "", x = "") + scale\_y\_continuous(labels = scales::label\_currency()) + scale\_x\_date(labels = scales::label\_date("%b %d"), date\_breaks = "4 days") + theme\_tq(base\_family = "Roboto Slab", base\_size = 16) + theme(plot.subtitle = ggtext::element\_markdown(face = "bold"), plot.title = element\_text(face = "bold"), plot.background = element\_rect(fill = "snow"), strip.text = element\_text(face = "bold", color = "black"), strip.background = element\_rect(fill = "azure"), axis.text= element\_text(face = "bold"), legend.position = "none") To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Global Modeling with XGBoost: Gold vs. Silver

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

You can send me questions for the blog using this form and subscribe to receive an email when there is a new post.

A tutorial on creating a simple R package that includes C++ code for a basic operation using different data types (int and size_t).

This video covers the basics of R and C++ syntax and why data types matter a lot on C++ side while R is more permissive. The code is available here.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Creating a simple R package with C++ code to sum “a + b” using different data types

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop titled Bayesian Optimization for Sequential Decisions with Multi-Arm Bandits, which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Bayesian Optimization for Sequential Decisions with Multi-Arm Bandits

Date: Thursday, October 23rd, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Jordan Nafa is a Data Scientist and Bayesian Statistician who has previously worked at Game Data Pros, where he designed and built production systems for Bayesian optimization and experimentation in major mobile and console games. He was previously a PhD Candidate in Political Science at the University of North Texas, where he taught undergraduate courses in causal inference, statistics, and American political behavior.

Description: This workshop introduces Bayesian optimization and multi-armed bandits. It covers sequential decision problems, Thompson Sampling, and multi-arm bandits. The workshop is designed for those with a basic understanding of traditional experimental design and analysis approaches, such as those commonly applied in A/B testing, who want to learn about Bayesian methods for sequential decision-making. The workshop includes practical examples of implementing multi-arm bandit algorithms and their generalizations in R and Python using Stan and PyMC.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

Please note that the registration confirmation is sent 1 day before the workshop to all registered participants rather than immediately after registration

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Bayesian Optimization for Sequential Decisions with Multi-Arm Bandits was first posted on September 23, 2025 at 6:10 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Bayesian Optimization for Sequential Decisions with Multi-Arm Bandits

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Ten years of blog postsA few months ago—26 July 2025 to be precise—was the tenth anniversary of my first blog post. Over that time it turns out I’ve written about 225 blog posts, and an astonishing (to me) 350,000 words. That’s after you take out the code.

Free Range Statistics is an old-fashioned blog, with a single author and very much representing the ideal of a “web log” just recording things of interest to me. It’s not a comprehensive personal blog (I never have posts just about my travel, family life, etc.), but focused on issues that somehow relate to statistics—ranging from the abstract and methodological, through to specific applications of the type “here’s a fun chart of some interesting historical or current data I saw”. It’s strictly non-monetised; open to the world to read for free, and will never make paid endorsements. I’ll go a bit into what’s kept me motivated later, but the spoiler is that, like art, blogging is in my opinion something best done primarily for your own interests and needs, and if anyone else likes it that’s a bonus.

The ten years of blog history hasn’t been an even one, but has had some ebbs and flows. We can see this in this chart of number of blog posts per month over time.

Code for these charts is at the bottom of the post. Two things worth noting about this one are how I’ve turned the months with zero posts into hollow circles to de-emphasise them visually while still including the zeroes in the modelling; and used for the trend line a continuous single model over all years instead of a separate model fit to each year-facet, which would be the easy default but does not really make sense given how time is continuous and all.

The low point of post frequency was 2021 and 2022, when life events got in the way. I was very busy in my day job as Chief Data Scientist for Nous Group, and this also was fairly hands-on technical itself which reduced my motivation to write code out-of-hours to relax. I was also playing a lot of Elite Dangerous in this period, right up until 2024 (when the civil unrest in Noumea led me to drop that cold turkey). Mid 2018 and mid 2022 both saw me change jobs and countries. In 2025 I’ve had health challenges, but these seem to be under control and I’m getting into a better modus vivendi with them.

The past couple of years has seen a subtle but material uptick in my posting frequency, and I think this is going to continue. I’ve got quite a backlog of half-finished posts to write about. These are on topics ranging from synthetic controls, to power and p-values, to lots of empirical stuff on the Pacific.

One thing that’s happened over time is the posts have gotten longer and, perhaps, more thorough over time. Certainly they are much more likely to be crafted over weeks or even months (or years in some cases), rather than knocked out in a single Saturday morning as used to be the case. Back when I wrote 45 posts in 2016—nearly one a week—they were short, very single topic, no great level of detail. More recently I am more inclined to try to thoroughly tease something out, particularly when I am learning for myself or trying to consolidate my understanding of something. A good example would be my recent set of posts on modelling fertility rates, which I had to split into two, one on the substance and one on the grab bag of things I learned on the way.

Here’s a connected scatter plot that lets us see both word count and posting frequency together, with some very crude characterisations of characteristic themes I was writing about at the time:

While one does one’s art for its own sake, there’s no denying it’s interesting to see what other people read in my blog too. I get a modest but steady trickle of around 60 unique visitors and 80-100 pages read a day. That is, modest compared to say Heather Armstrong’s peak numbers of about 300,000 visitors a day at the peak of mummy blogging, but quite a few more than I thought I’d get when I set out (which would have been, to be honest, in round numbers, around zero).

At its high point back when Twitter more or less worked, I wrote more frequently and was doing election forecasts, I think I got about 70% more traffic than now, but it’s hard to tell, with changing approaches to tracking visitors.

I used to have an automated “most popular” listing but changes in analytics services over the blog’s lifetime degraded this and I’ve pulled it. But from a more ad hoc examination using partial data from some mixed sources (too complicated to talk about here), here are some posts that have been most read recently:

Stepwise selection of variables in regression is Evil (2025)
Skill v luck in determining backgammon winners (2016)
Relative risk ratios and odds ratios (2018)
Weighted versus unweighted percentiles (2023)
Log transforms, geometric means, and estimating population totals (2023)
Dual axis time series plots may be ok sometimes after all (2016)
Extrapolation is tough for trees! (2016)
Snakes and ladders (2024)

This is interesting and I think is probably showing some external searches are turning up my blog on basic methodological questions. This must be dominating over social media or RSS feeds pulling in visitors when I publish a new post. I’m pleased to say each of these posts above does indeed have something useful in it—roughly defined as meaning I sometimes go back to them myself to see what I thought. So I hope other people are finding them of some use at the end of their random web search too.

If I had a longer series of analytics data I’m sure my various election-related posts pages, and time series modelling posts, would be in the genuine top hits. At one point it looked like some of my comparisons of forecasting methods were in the required reading for some courses, they were getting so many hits.

Blog benchmarksI did some cursory internet research into blog longevity, to see how my 10 years stands up in comparison. ChatGPT1 first assured me that research said the 60-70% of blogs are abandoned after one year (attributed to Herring et al) and that the median life was four months (Mishne and de Rijke) or 50% stopped after one month (same alleged authors).

These all sound plausible! And maybe these authors did find that. But I can’t (with limited time and access, admittedly) find them doing so. Application of intensive interrogation techniques to ChatGPT revealed that these were things that it thought sounded plausible as things these people might have written, rather than it could actually find real, published papers that contained these numbers.

Truly, ChatGPT is like an enthusiastic, immensely well-read but very unreliable research assistant who has had a couple of drinks, whose outputs should all be prefaced with “I seem to remember reading or hearing somewhere….” and treated with a heap of scepticism.

In terms of real findings I can actually source, some research from back when blogs were cool and before short-form social media really took off found that a quarter of blogs only last one post. Back in 2003, apparently, “the typical blog is written by a teenage girl who uses it twice a month to update her friends and classmates on happenings in her life.” These days, I do not think such people write blogs or even micro-blogs, but post videos on TikTok or equivalent.

A 2012 study of research blogs—closer in form and motivation to my own than the more personal blogs that make (or made) up the bulk of the blogosphere—found 84% of research blogs published under the author’s own name; 86% in English; and 72% by one or two male authors. So I’m in the majority in those respects.

At around 1,500 words each, my blog posts are much longer than the average of 200-300 words found by Susan Herring and others in a 2004 study.

Much of the research above is dated. Effectively it precedes the rise of video-based influencers. Short-form video (TikTok etc), podcasts, general video, and short-form text (X, Bluesky, LinkedIn etc) seem to dominate over written blogs these days. I have no interest in producing any of these things except the short-form text / social media sites.

There are still apparently a million or so active blogs, many of them forming a more stable piece of infrastructure underneath the froth of these more modern forms. This is basically how I engage with Bluesky, Mastodon and LinkedIn too, in terms of the relationship with my blog. I write in the blog, and use the social media to publicise that writing.

Why I write my blogTen years is a success, I guess. While I couldn’t find a citable source, I’m well prepared to believe that most blogs are abandoned after a few months. So what kept me motivated to keep writing for ten years?

My motivations have certainly evolved over time as I settled into a rhythm of writing and publishing posts. Compared to when I set out, I can give a much more accurate picture of why I’m really doing this:

It helps me exercise and extend my hands-on technical craft—something that doesn’t happen naturally in the managerial roles in my day job, but is still useful for executing those roles even in a directorial and decision-making rather than hands-on capacity.
I can learn things, with the motivation for extra discipline (I really want to be confident I’m getting some unfamiiar thing right if I’m going to post it) that comes from doing so in public. A number of times I have had my course corrected by positive engagements after posting a blog, either on social media or in the comments section.
Sometimes it’s just fun, and relaxing, to play around with data and code. Particularly when I read something interesting and want to check “wait, is that for real?”. Or when I just want to make a cool animation.
I can try out stuff we might (or might not) want to use at work but, for whatever reason, needs me to give it a go myself in a way that doesn’t fit in with my normal work responsibilities yet can be drawn on if helpful.
Sometimes (but not very often) I actually want to make an intervention in the public sphere and communicate some facts and ideas. How important this motivation is has varied over time, and it’s never been particularly important. There were periods when I published election forecasts for New Zealand that had no other equivalent at the time, and some Covid modelling in Australia, where communicating actual issues was the most important thing for my blog. But these didn’t (and almost certainly, couldn’t possibly, given my energy and interest levels) last. Perhaps the high point blog post that I really wanted people to read was my exposure of Surgisphere, which made me Twitter-famous for a few days and was an important contribution to an investigation by the Guardian and then retraction of an article in the Lancet (surprisingly but gratifyingly rapidly).
My day job is helped by networking, and my interest and skills in data and code is one tool I can use in a small way to do that. I’m certainly not into blogging for fame (or I hope I would do differently and better than I am), but I do seek to use my posts in a certain way to broaden and strengthen my professional networks. I publicise my posts on Bluesky and LinkedIn, sometimes Facebook (and until 2024 on Twitter). They are a way of getting myself known to niche audiences, and very occasionally a way to achieve an objective for my day job by publicising something cool we’re doing, positions we’re recruiting, or an issue we’re concerned about.

Technical stuff about the blogWhen I set up my blog I really, really hated the non-data technical stuff about getting it to work, having the fonts right, working out how domain names work, deciding on layout, etc. I had to read quite a few blogs on how to set up blogs, and vowed to myself not to become one of them. So I have relatively few posts on the back end of my blog. But ten years on, there is some (small) possible interest in what works for me, so here is how my blog works under the hood:

It is hosted on GitHub pages but has its own domain name. This (the GitHub part) is free, and gives me plenty of control over formatting, and works well with Jekyll.
I use Jekyll and resisted upgrading to Hugo when it comes along. In things like this, “there’s a time for change, which is when it can no longer be resisted”. If it ain’t broke, don’t fix it.
It is a Git repository within a repository. The source code is the important one that I work on and has a _working folder with all the R and other technical scripts, and a _posts folder with Markdown or HTML files for the actual posts.
When I build the site it appears in the _site folder of the source code repository. _site is also a Git repository and, when it is all good to go, I push that to the https://github.com/ellisp/ellisp.github.io repository on GitHub, which is automatically published on GitHub pages.
I write all the Markdown or HTML pages by hand. I use HTML when things get too complicated layout-wise for Markdown (not very often).
I don’t use RMarkdown or similar for this blog (knitting results in with the code and text) because I prefer to have complete, manual control of where I put a code chunk, plot or table. And my creative process is very much “work on the analysis” and then “write it up”, which is well supported by having a separate R script with the analysis and a Mardown or HTML file with the write-up.
I created and use the frs R package with a few supporting functions, most important of which is the svg_png() function. It uses the method described in this post. It helps SVG files look good with Google fonts and working across platforms. It also saves near-identical PNG and SVG versions of images, so I can have PNG fall-backs for browsers that don’t show SVGs (this was a real issue 10 years ago, I don’t know about now).
There are some things like syntax highlighting, the domain name, link to Disqus for comments section, that involved a bunch of mucking around that I’m pleased to say I’ve forgotten completely what I had to do.

Yeah, blog to live, don’t live to blog. That’s true in general, but never more so than in thinking about the stuff that makes it possible to blog.

Word count codeHere’s the code that produced the charts shown earlier in this post:

library(tidyverse)library(stylo) # for delete.markuplibrary(glue)library(ggtext)#---------------Import and process blog posts-------------blog\_names <- list.files("../\_posts", full.names = TRUE)blogs <- tibble()for(i in 1:length(blog\_names)){ blogs[i, "full"] <- paste(readLines(blog\_names[i]), collapse = " ") blogs[i, "filename"] <- gsub("../\_posts/", "", blog\_names[i], fixed = TRUE)}blogs <- blogs |> mutate(no\_jekyll = gsub("\\{\\% highlight R.*?\\%\\}.*?\\{\\% endhighlight \\%\\}", " ", full), txt = "")# delete markup only works on one string at a time, seems easiest to do it in a loop:for(i in 1:nrow(blogs)){ blogs[i, ]$txt <- delete.markup(blogs[i, ]$no\_jekyll, markup.type = "html")}# a few more basic stats per blog post:blogs <- blogs |> mutate(word\_count = stringi::stri\_count\_words(txt), word\_count\_with\_tags = stringi::stri\_count\_words(no\_jekyll), date = as.Date(str\_extract(filename, "^[0-9]*-[0-9]*-[0-9]*")), month = month(date), year = year(date))#---------------Minimal anaylsis----------------# Summary aggregatesblog\_sum <- blogs |> summarise(number\_blogs = n(), words\_with\_tabs = sum(word\_count\_with\_tags), total\_words = sum(word\_count), mean\_words = mean(word\_count), median\_words = median(word\_count), max\_words = max(word\_count), min\_words = min(word\_count))# Shortest blog (turns out to be one just announcing a work shiny app):blogs |> arrange(word\_count) |> slice(1) |> pull(txt)#------------------Graphics for use in blog-------------------------the\_caption <- "Source: https://freerangestats.info"# Time series plot showing number of posts by month:d1 <- blogs |> group\_by(year, month) |> summarise(number\_blogs = n()) |> ungroup() |> complete(year, month, fill = list(number\_blogs = 0)) |> # remove October, November, December in 2025 (as time of writing is September 2025): filter(!(year == 2025 & month %in% 10:12)) |> # remove months blog did not exist: filter(!(year == 2015 & month %in% 1:6)) |> group\_by(year) |> mutate(year\_lab = glue("{year}: {sum(number\_blogs)} posts"), is\_zero = ifelse(number\_blogs == 0, "Zero", "NotZero")) # model a smooth curve to the whole data set (don't want)# to do this with geom\_smooth in the plot as then it has# break every year:mod <- loess(number\_blogs ~ I(year + month / 12), data = d1, span = 0.15)d1$fitted <- predict(mod)# draw time series plot of number of blogs:d1 |> ggplot(aes(x = month, y = number\_blogs)) + facet\_wrap(~year\_lab) + geom\_line(aes(y = fitted), colour = "grey80") + geom\_point(colour = "steelblue", size = 2.5, aes(shape = is\_zero)) + expand\_limits(y = 0) + scale\_x\_continuous(breaks = 1:12, labels = month.abb) + scale\_shape\_manual(values = c("Zero" = 1, "NotZero" = 19)) + theme(panel.grid.minor = element\_blank(), axis.text.x = element\_text(angle = 45, hjust = 1), legend.position = "none") + labs(x = "", y = "Number of blog posts", title = "Ten years of Free Range Statistics blogging", subtitle = glue("{nrow(blogs)} posts and {comma(blog\_sum$total\_words)} words, in just over ten years."), caption = the\_caption)# Connected scatter plot comparing average word count to number of posts:blogs |> mutate(number\_months = case\_when( year == 2015 ~ 6, year == 2025 ~8.5, TRUE ~ 12 )) |> group\_by(year, number\_months) |> summarise(avg\_word\_count = mean(word\_count, tr = 0.1), number\_blogs = n()) |> ungroup() |> mutate(blogs\_per\_month = number\_blogs / number\_months) |> ggplot(aes(x = blogs\_per\_month, y = avg\_word\_count, label = year)) + geom\_path(colour = "grey80") + geom\_text(colour = "grey50") + scale\_y\_continuous(label = comma) + expand\_limits(x = 4.5) + annotate("text", fontface = "italic", hjust = 0, colour = "darkblue", x = c(4, 3.4, 2.1), y = c(1165, 1350, 1880), label = c("Time series", "Elections", "Covid") ) + # add day jobs annotate("text", fontface = "italic", hjust = 0, colour = "brown", x = c(3.1, 2.5, 0, 1.1), y = c(1130, 1675, 1420, 1330), label = c("NZ economics", "Consultant", "Chief Data Scientist", "Pacific") ) + labs(x = "Blog posts per month", y = "Average words per blog post", title = "Ten years of Free Range Statistics blogging", subtitle = "Annotated with important (but not necessarily dominant) <span style = 'color:darkblue'>themes</span> and <span style = 'color:brown'>day-jobs</span> for different phases.", caption = the\_caption) + theme(plot.subtitle = element\_markdown()) 1. This is my first use of a large language model for any purpose with this blog. I can categorically say Free Range Statistics will never use generative AI to produce either words or code. To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Ten year anniversary of Free Range Statistics by @ellis2013nz

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

You can send me questions for the blog using this form and subscribe to receive an email when there is a new post.

A tutorial on creating a simple R package that includes C++ code for a basic operation.

This video covers the basics of R and C++ syntax and why data types matter a lot on C++ side while R is more permissive. The code is available here.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Creating a simple R package with C++ code to sum “a + b”

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. You can send me questions for the blog using this form and subscribe to receive an email when there is a new post.

Dear fellow developers and data scientists,

If everyone reading this gave just the price of a coffee, I could focus fully on open source work for our community. But not everyone can or will contribute, and that’s okay.

For years, I’ve built and maintained open source R packages—cpp11armadillo, capybara, gravity, tabulapdf, pointblank, and more—that many people use daily. Every GitHub issue and email gets a response. This is my way of giving back as my own research benefits from a wide range of R packages and tools developed by others.

I’m a PhD student from Chile currently in Canada. My scholarship payments have been delayed for four months because of an administrative error starting with my name spelled wrong.

I wrote about this one month ago and I’m still in the same situation. Every time I contact Service Canada by phone and email, I get the same “we will provide an answer in 48 hours” message, but nothing changes. Rent and food can’t wait. To get by, I’ve been selling guitar pedals I design and build (inspired by Queen and Dr. Brian May): https://www.instagram.com/maplebooster/.

I want my software and datasets to stay open. Before considering paywalls or commercializing, I’m asking the community for support. If my packages have helped you, please consider a small contribution.

Support My Work on Buy Me a Coffee

Despite these challenges, I’m not slowing down. I just updated Open Trade Statistics with 2023 data and a new dashboard. Your support—donations, sharing, feedback, or kind words—means a lot. It keeps me going through code debugging while I hear on the phone “all of our agents are busy, please call later.”

To everyone who’s helped with code, bug reports, or encouragement: thank you. Every bit of support helps keep this work going. The open source world thrives on mutual support—today, I’m asking for yours.

With appreciation and hope,

Pacha

P.S. If you can’t contribute financially, sharing this message with others who might benefit from my work is also a huge help. Open source is about supporting each other.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A Personal Message from an Open Source Contributor

View Details

[This article was first published on jakub::sobolewski, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Testing functions that rely on external dependencies is hard.

Your tests become slow, fragile, and unreliable when they depend on external APIs, file systems, or services. Worse yet, some dependencies like Sys.time() return values that change constantly, making consistent testing nearly impossible.

The solution is simple: wrap external dependencies in your own functions and stub them with testthat::local_mocked_bindings.

Why Wrap External Dependencies?External dependencies make testing painful in three ways:

First, they can be unpredictable. APIs go down. File systems change. Network requests timeout.
Second, they can be uncontrollable. You can’t force Sys.time() to return a specific value or make a random API response predictable.
Third, they can be slow. Real database queries and HTTP requests add seconds to test suites that should run in milliseconds.

Wrapping dependencies solves all three problems at once.

The Pattern: Wrap, Stub, TestHere’s how to build testable functions with external dependencies:

Step 1: Wrap External CallsInstead of calling Sys.time() directly, create a wrapper function:

get\_current\_time <- function() { Sys.time()}calculate\_elapsed\_time <- function(start\_time) { current <- get\_current\_time() difftime(current, start\_time, units = "secs")} Step 2: Test with StubsUse local_mocked_bindings to replace your wrapper with a predictable stub:

test\_that("calculate\_elapsed\_time returns time difference", { # Arrange local\_mocked\_bindings( get\_current\_time = function() as.POSIXct("2023-01-01 12:30:00") ) start\_time <- as.POSIXct("2023-01-01 12:00:00") # Act result <- calculate\_elapsed\_time(start\_time) # Assert expect\_equal(as.numeric(result), 1800) # 30 minutes = 1800 seconds}) Look how clean that test is. No setup. No teardown. No flaky timing issues.

Why local_mocked_bindings Encourages Good DesignFor a long time I’ve only used mockery for stubbing, but local_mocked_bindings starts to grow on me.

The key insight: you get the most out of local_mocked_bindings when you don’t use the .package argument.

I bet this was intentional design of this interface: you get the cleanest test code, when not using extra arguments. Without .package you can only stub functions that are defined in the current namespace. This forces you to apply good design principles: wrap external dependencies in your own functions.

Try to stub Sys.time directly and you’ll need:

```

Messy - requires .package argumentlocal_mocked_bindings( Sys.time = function() as.POSIXct("2023-01-01 12:30:00"), .package = "base")

``` But wrap it first:

```

Clean - no .package neededlocal_mocked_bindings( get_current_time = function() as.POSIXct("2023-01-01 12:30:00"))

``` The function teaches you better design (if you pay attention).

Three Benefits of WrappingWrapping external dependencies gives you three powerful capabilities:

Stubbing for Testing: Replace unpredictable external calls with controlled test doubles using local_mocked_bindings.
Dependency Injection: It opens doors for injecting different implementations for different environments. Maybe production uses Sys.time() but your staging environment reads from a mock time server. Then in tests we can use dependency injection of a fake, or still use a stub with local_mocked_bindings.
Easy Migration: Change implementations without touching calling code. Today you read time from the system clock, tomorrow from a solar clock camera, next week from an atomic time API.

The Sys.time() ProblemSys.time() exemplifies why wrapping matters.

Unlike random numbers (controlled by set.seed()), time always changes. Every test run gets different values. You can’t make time-dependent functions deterministic without replacing the time source.

Consider a function that calculates business hours:

is\_business\_hour <- function() { current\_hour <- hour(Sys.time()) current\_hour >= 9 && current\_hour <= 17} How do you test this? You can’t control when your tests run.

Wrap the time dependency:

get\_current\_time <- function() { Sys.time()}is\_business\_hour <- function() { current\_hour <- hour(get\_current\_time()) current\_hour >= 9 && current\_hour <= 17} Now testing becomes trivial:

test\_that("is\_business\_hour returns TRUE during business hours", { # Arrange local\_mocked\_bindings( get\_current\_time = function() as.POSIXct("2023-01-01 14:00:00") # 2 PM ) # Act result <- is\_business\_hour() # Act expect\_true(result)})test\_that("is\_business\_hour returns FALSE outside business hours", { # Arrange local\_mocked\_bindings( get\_current\_time = function() as.POSIXct("2023-01-01 22:00:00") # 10 PM ) # Act result <- is\_business\_hour() # Assert expect\_false(result)}) Perfect control. Perfect reliability.

Real-World ExampleHere’s how the pattern works with more complex dependencies:

```

Wrapper functions for external dependenciesget_system_info <- function() { Sys.info()}get_package_versions <- function(path) { if (!rlang::is_installed("yaml")) { stop("Packages \"yaml\" not installed", call. = FALSE) } if (!rlang::is_installed("here")) { stop("Package \"here\" not installed", call. = FALSE) } yaml::read_yaml(here::here(path, "renv.lock"))}get_test_results <- function(...) { testthat::test_local(..., stop_on_failure = FALSE)}# Function that uses wrapped dependenciesgenerate_system_report <- function(project_path = ".") { system_info <- get_system_info() packages <- get_package_versions(project_path) tests <- get_test_results(project_path) list( os = system_info[["sysname"]], r_version = system_info[["version"]], package_count = length(packages$Packages), test_status = all(tests$passed) )}

``` Testing becomes straightforward:

test\_that("generate\_system\_report creates complete report", { # Arrange local\_mocked\_bindings( get\_system\_info = function() c(sysname = "Linux", version = "4.0.0"), get\_package\_versions = function(path) list(Packages = list(a = 1, b = 2)), get\_test\_results = function(...) data.frame(passed = c(TRUE, TRUE)) ) # Act report <- generate\_system\_report() # Assert expect\_equal(report$os, "Linux") expect\_equal(report$package\_count, 2) expect\_true(report$test\_status)}) Three external dependencies controlled with three simple stubs. No real file system access. No actual test execution. No system introspection.

Interface Over ImplementationThe wrapper pattern creates an interface between your code and external dependencies.

Interfaces are powerful because they separate “what” from “how”. Your code knows what it needs (current time, system info, test results) but doesn’t care how those needs are met.

In production, get_current_time() calls Sys.time(). In tests, it returns a fixed timestamp. In a specialized environment, it might read from a network time protocol server or even that solar clock camera.

Change the implementation without changing a single line of calling code.

Clean Tests WinCompare these two approaches:

Without wrapping:

```

Brittle, slow, unpredictabletest_that("time calculation works", { start <- Sys.time() Sys.sleep(0.1) result <- calculate_duration(start) expect_gt(result, 0.1) # Flaky assertion})

``` With wrapping:

```

Reliable, fast, predictabletest_that("calculate_duration returns time difference between start and current time", { # Arrange local_mocked_bindings( get_current_time = function() as.POSIXct("2023-01-01 12:30:00") ) start <- as.POSIXct("2023-01-01 12:00:00") # Act result <- calculate_duration(start) # Assert expect_equal(result, 1800)})

``` The second test runs in microseconds, never fails randomly, and expresses intent clearly.

Start Wrapping TodayNext time you write a function that touches the outside world, wrap the external call. Your future testing self will thank you.

The pattern is simple: wrap external dependencies, stub in tests, enjoy clean and reliable test suites that run fast and pass consistently.

To leave a comment for the author, please follow the link and comment on their blog: jakub::sobolewski.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Clean R Tests with local\_mocked\_bindings and Dependency Wrapping

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Read it in: Français.

Sunny and Steffi showing off their R hex stickers!

This summer I had a wonderful time attending the Society for Canadian Ornithologists meeting in Saskatoon, Canada.It was super exciting to run into Sunny Tseng, rOpenSci Champion and fellow ornithologist!It’s not often that I am able to run into both types of colleagues (R developers and ornithologists) at the same conference, so I cherish these experiences.

Back before I joined rOpenSci I started my professional career as a behavioural ecologist and ornithologist, studying the effects of urban noise and habitat on communication in Chickadees (Tits for the non-North Americans).While I’ve shifted into R consulting and supporting Yani here at rOpenSci with our amazing community, today I still work mostly for other ornithologists or bird-related conservation non-profits.

This means that the Society for Canadian Ornithologists is still very much my ‘home’ organization and I’m always excited to go to meetings and reconnect with my colleagues.But ornithology conference or not, I always try to bring a bit of R, Open Science, and rOpenSci to the meeting .

In the past I’ve organized symposia on topics such as ” R for Ornithologists: Perspectives from useRs to programmeRs (to birdeRs)” or ” How can ornithologists find R packages?“, and have helped run workshops on R packages for ornithology (like bbsBayes2).

This year, I felt like I should branch out a bit beyond R and decided to talk about ” Practical tips for Open Science in Ornithology“.My goal was encourage ornithologists (or other scientists) to give Open Science a try.To realize that it doesn’t require a background in Computer Science, or buckets of money!And because ornithology has many scientists already working with open data, open methods, and open science, to share some ideas for how to they might encourage more of their colleagues to join in.

For the rOpenSci community, this may feel very much “back to basics”.The conversations we have are more often about how can you be better at open science?How can you make that study completely reproducible?How can we improve open code quality1?

Slide acknowledging that open science can be challenging
(Grey-crowned crane – Kuribo, CC BY-SA 2.0)

But I think we sometimes forget that in addition to time, skills, and sometimes funding, to practice open science takes a great deal of courage2.

For those just getting started (like students), or those without open science support (like researchers at small institutions), practicing open science can seem overwhelming and extremely intimidating.It’s scary to think about the criticisms if you don’t “do open science right”, or the consequences if you make a mistake.

I wanted to focus on baby steps, to make it clear that it’s making the effort that’s important, not getting it perfect!

So in addition to some practical advice, I gave four general tips:

Don’t try to do everything
- Be kind to yourself
- You’re allowed to smart small and grow!
- Do one new thing
- On every project, try to expand your skill set, learn one more new thing
- It’s okay to be nervous
- Mistakes happen, just remember that by practicing open science, you’re already awesome!
- Work with others
- Find your community

I think the best thing about giving this talk, was the number of students who approached me afterwards, excited (and nervous) to start sharing their code online.Full of questions from practical ones about using GitHub to general ones about how ‘good’ your code needs to be to be shared.My answer to this last question was “If you used it, share it.(But annotate like mad!)”

What are your favourite tips for encouraging open science?

AcknowledgementsThanks to Alex Koiter for ideas and brainstorming for my talk, as well as for his awesome (and much longer) talk, Open and Reproducible Soil Science.

Trisovic A, Lau MK, Pasquier T, et al.A large-scale study on research code quality and execution.Sci Data 9, 60 (2022).https://doi.org/10.1038/s41597-022-01143-6 ︎
Insecurity:Gomes DGE, Pottier P, Crystal-Ornelas R, et al.Why don’t we share data and code? Perceived barriers and benefits to public archiving practices.Proc Biol Sci 289(1987):20221113 (2022).https://doi.org/10.1098/rspb.2022.1113 ︎ To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A Gentle Introduction to Open Science

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Any-mean-model + GARCH(1, 1) for probabilistic stock forecastingIn this blog post, we will explore the combination of any model with GARCH(1,1) for probabilistic stock forecasting. This approach allows us to capture both the conditional mean and conditional variance of stock returns. We will demonstrate the implementation using Python and the ahead package.

Ref: https://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity

This function generates the data frame for the animationgenerate_dirichlet_animation_data <- function(alpha1_values, alpha2_values, alpha3_values, n_samples = 2000) { library(MCMCpack) # Triangle vertices v1 <- c(1, 0) v2 <- c(0, 1) v3 <- c(0, 0) # Projection function project_to_triangle <- function(x1, x2, x3) { x <- x1 * v1[1] + x2 * v2[1] + x3 * v3[1] y <- x1 * v1[2] + x2 * v2[2] + x3 * v3[2] data.frame(x = x, y = y) } # Generate animation data animation_data <- do.call(rbind, lapply(seq_along(alpha1_values), function(i) { alpha <- c(alpha1_values[i], alpha2_values[i], alpha3_values[i]) samples <- rdirichlet(n_samples, alpha) projected <- project_to_triangle(samples[,1], samples[,2], samples[,3]) projected$alpha1 <- alpha[1] projected$alpha2 <- alpha[2] projected$alpha3 <- alpha[3] projected$frame <- i projected })) return(animation_data)}# This function creates the animated plotplot_dirichlet_evolution <- function(animation_data) { library(ggplot2) library(gganimate) library(grid) # for arrow units # Triangle vertices v1 <- c(1, 0) v2 <- c(0, 1) v3 <- c(0, 0) # Label positions label_df <- data.frame( x = c(v1[1], v2[1], v3[1]), y = c(v1[2], v2[2], v3[2]), label = c("(1,0,0)", "(0,1,0)", "(0,0,1)"), nudge_x = c(-0.08, 0.1, 0), nudge_y = c(0, 0, 0.08) ) # Triangle outline triangle_df <- data.frame( x = c(v1[1], v2[1], v3[1], v1[1]), y = c(v1[2], v2[2], v3[2], v1[2]) ) # Build plot p_animation <- ggplot(animation_data, aes(x = x, y = y)) + geom_point(alpha = 0.3, size = 0.8, color = "steelblue") + geom_density_2d(color = "red", alpha = 0.7) + geom_path(data = triangle_df, aes(x = x, y = y), color = "black", size = 1) + annotate("segment", x = 1, y = 0, xend = 0, yend = 1, arrow = arrow(length = unit(0.2, "cm")), color = "darkgreen", size = 1.2) + xlim(0, 1) + ylim(0, 1) + geom_text(data = label_df, aes(x = x, y = y, label = label), nudge_x = label_df$nudge_x, nudge_y = label_df$nudge_y, size = 4, fontface = "bold", color = "black") + labs( title = "Dirichlet Prior Evolution", subtitle = "", x = "Projected X", y = "Projected Y", caption = "Transition from uniform to concentrated distribution" ) + theme_minimal() + theme( plot.title = element_text(size = 16, hjust = 0.5), plot.subtitle = element_text(size = 14, hjust = 0.5), axis.title = element_text(size = 12), plot.caption = element_text(size = 10, hjust = 0.5) ) + transition_states(frame, transition_length = 1, state_length = 2) + ease_aes('sine-in-out') return(p_animation)}#| message: false#| warning: false#| code-fold: true#| code-summary: "Animation Code"#| set.seed(42)# Parametersalpha1_values <- seq(1, 0.1, by = -0.01) # starts high, ends lowalpha2_values <- seq(1, 0.1, by = -0.01) # starts low, ends highalpha3_values <- seq(1, 0.1, by = -0.01) # starts low, ends highanimation_data <- generate_dirichlet_animation_data(alpha1 = alpha1_values, alpha2 = alpha2_values, alpha3 = alpha3_values, n_samples = 2000)plot_dirichlet_evolution(animation_data)

``` These next two plots, the first and last frames of the animation, clearly show how the density moves from being uniformly distributed over the simplex to being concentrated at the vertices of the simplex. When modeling the development of a multi-state Markov chain, as I was doing in the post I alluded to above, it is common practice to select a uniform Dirichlet prior with . However, if you believe that the process is likely to start off uniformly distributed among the states, then a prior with might be appropriate. If you had reason to believe that the process would start off concentrated on particular states, then you might explore using an asymmetric distribution by setting different values for the . The code driving these animations might be helpful.

Show the code ggplot(subset(animation\_data, frame == 1), aes(x = x, y = y)) + geom\_point(alpha = 0.3, color = "darkblue") + ggtitle("Initial Frame: Uniform Prior: alpha = (1,1,1)") Show the code ggplot(subset(animation\_data, frame == max(animation\_data$frame)), aes(x = x, y = y)) + geom\_point(alpha = 0.3, color = "red") + ggtitle("Final Frame: Concetrated Prior: alpha = (.1,.1,.1)") This concentration of density as increases is very apparent in this next simulation as moves from (0.1, 0.1, 0.1) to (10.0, 10.0, 10.0). Here we see the distribution concentrating on the mean, = (1/3, 1/3, 1/3).

Show the code ```

Define alpha trajectoriesalpha1 <- seq(.1, 10, by = 0.1)alpha2 <- seq(.1, 10, by = 0.1)alpha3 <- seq(.1, 10, by = 0.1)animation_data <- generate_dirichlet_animation_data(alpha1, alpha2, alpha3, n_samples = 2000)anim2 <- plot_dirichlet_evolution(animation_data)anim2

``` Note that the animation passes through (0.5, 0.5, 0.5) which is the Jeffreys prior for the Dirichlet distribution.

Variance and Differential EntropyThe Wikipedia article for the Dirichlet distribution prominently displays the distribution’s differential entropy:

where and are defined above and is the digamma function. (This equation triggered my mention of the digamma function above.) But please be advised that differential entropy defined as: for continuous distributions is not the same as the Shannon entropy for discrete distributions and does not have a similar interpretation. Among other things, differential entropy can be negative, in not invariant under a change of variables, and probably doesn’t conform to any intuition you may have developed about maximum entropy. The plot on the left below shows the behavior of the differential entropy for the symmetric Dirichlet distribution we have been considering as moves from (0.1, 0.1, 0.1) to (5.0, 5.0, 5.0). Note that the entropy keeps increasing beyond the point which corresponds to the uniform distribution over the simplex.

Show the code ```

Define the number of dimensions for the symmetric Dirichlet distributionn <- 3# Define a function to calculate the entropy of a symmetric Dirichlet distributiondirichlet_entropy <- function(alpha, n) { # This formula is for a symmetric Dirichlet distribution # where all parameters a_i are equal to alpha. # The formula uses the digamma function (psi) and the log-gamma function (lgamma). # H(alpha) = log(Beta(alpha)) + (n * alpha - n) * psi(n * alpha) - n * (alpha - 1) * psi(alpha) # log(Beta(alpha)) = n * lgamma(alpha) - lgamma(n * alpha) # The final formula is simplified for symmetric case. # lgamma(x) is the log-gamma function # digamma(x) is the psi function, the first derivative of lgamma(x) entropy_val <- lgamma(n * alpha) - n * lgamma(alpha) + (n * alpha - n) * digamma(n * alpha) - n * (alpha - 1) * digamma(alpha) return(entropy_val)}# Create a sequence of alpha values from 0.1 to 5alpha_values <- seq(0.1, 5, by = 0.01)# Calculate the entropy for each alpha valueentropy_values <- sapply(alpha_values, dirichlet_entropy, n = n)# Create a data frame for plottingplot_data <- data.frame(alpha = alpha_values, entropy = entropy_values)# Plot the entropy valuesggplot(plot_data, aes(x = alpha, y = entropy)) + geom_line(size = 1.2, color = "steelblue") + labs( title = "Entropy of a Symmetric Dirichlet Distribution", subtitle = paste("for n =", n, "dimensions"), x = "Alpha (Concentration Parameter)", y = "Entropy" ) + geom_vline(xintercept = 1, linetype = "dashed", color = "red", size = 1) + annotate("text", x = 1.1, y = max(entropy_values) * 0.9, label = "Alpha = 1 (Uniform Distribution)", color = "red", hjust = 0, size = 4) + theme_minimal(base_size = 14) + theme(plot.title = element_text(hjust = 0.5), plot.subtitle = element_text(hjust = 0.5))

Show the code k <- seq(.1, 10, by = 0.1)mean_dir <- numeric(length(k))var_dir <- numeric(length(k))for (i in seq_along(k)) { alpha <- k[i] sum_alpha <- alpha * 3 mean_dir[i] <- alpha / sum_alpha var_dir[i] <- (alpha * (sum_alpha - alpha)) / (sum_alpha^2 * (sum_alpha + 1))}df <- data.frame(k = k, mean = mean_dir, variance = var_dir)ggplot(df, aes(x = k, y = variance)) + geom_line(color = "blue", size = 1) + labs( title = "Variance of Dirichlet Distribution vs Alpha Parameter", x = "Alpha Parameter (k)", y = "Variance" ) ``` The plot on the right shows the variance of the Dirichlet distribution as a function increasing . We see that the variance decreases towards zero as increases. This is reflected in the second animation above which shows the distribution concentrating on the mean. Uncertainty is going to zero but differential entropy is shooting off towards to infinity. If you are nevertheless intrigued by differential entropy, you may want to have a look at the references I have included below.

References Bela A. Frigyik, Amol Kapila, and Maya R. Gupta, Introduction to the Dirichlet Distribution and Related Processes * Thomas M. Cover and Joy A. Thomas, Elements of Information Theory*, Wiley-Interscience Edition: 2nd Edition, (2006) * Rameshwar Gupta and Donald St. P. Richards, The History of the Dirchet and Liouville Distributions international Statistical Review (2001) * Jiayu Lin, On the Dirichlet Distribution

To leave a comment for the author, please follow the link and comment on their blog: R Works.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A Note on the Dirichlet Distribution

View Details

[This article was first published on coding-the-past, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In this post, you will learn what a T Test is and how to perform it in R. First, you’ll see a simple function that lets you perform the test with just one line of code. Then, we will explore the intuition behind the test, building it step by step with data about the Titanic passengers. Enjoy the reading!

What is a T-Test?A t-test is a statistical procedure used to check whether the difference between two groups is significant or just due to chance. In this post, we’ll look at data from Titanic passengers, dividing them into males and females. Suppose we want to test the hypothesis that men and women had the same average age. If our data shows that women were, on average, 2 years younger than men, we need to ask: is this a real difference, or could it have happened randomly? The t-test helps us answer this question.
Why is a T-Test important?A t-test is important when we want to draw conclusions about a population based on a sample. For example, imagine we are studying the demographics of ship passengers at the beginning of the twentieth century and want to use the Titanic sample to generalize findings to a broader population of passengers.

Of course, such inferences may be biased, since Titanic passengers might not perfectly represent all ship passengers of that era. Nevertheless, the sample can still provide valuable insights, as long as the context of both the sample and the population is carefully considered and clearly explained.

The Titanic passengersWe are going to use the titanic R library to access data about Titanic passengers. Specifically, we will work with a subset of passengers contained in the titanic_train dataset. Below, you will find the code to load the data, calculate the mean and standard deviation of age for males and females, and show how many passengers are men and women.

content_copy Copy
library(titanic) data('titanic\_train')df <- titanic\_train %>% select(Sex, Age) %>% na.omit()df %>% group\_by(Sex) %>% summarize(mean(Age), sd(Age), n())

| Sex | mean(Age) | sd(Age) | n | | --- | --- | --- | --- | | female | 27.9 | 14.1 | 261 | | male | 30.7 | 14.7 | 453 |

We can see that there is a difference of 2.8 years between the average age of men and women on the Titanic. Below, you can also check the distribution of ages.

content_copy Copy
ggplot()+ geom\_density(aes(x=df$Age, color = df$Sex), size = 0.7)+ scale\_color\_discrete("")+ xlab("Age")+ ylab("Density") It seems indeed that the distributions are very similar. In this case, our best option is to carry a T Test out to see if they are really so similar.

T test in RA T test can be performed in R in a very easy way. There is a function called t.test, whose first argument is a formula, in our case, we would like to know how age varies across different genders. Thomas Leeper wrote a very clear explanation about formulas in this page. Important for us is that the formula is composed by a dependent variable on the left (Age), followed by “~” and one or more independent variables on the right (Sex). The second argument is simply the dataframe with the data we want to test. This test assumes the two samples are independent and that age is approximately normally distributed, which we confirmed by the density plot above.

content_copy Copy
t.test(Age ~ Sex, data = df) How to interpret these results?

The p-value of 0.0118 means that if there were truly no difference in the average age between male and female passengers (i.e., if the null hypothesis were true), there would be only a 1.18% chance of observing a difference as large as the one we found or larger. Since this p-value is less than 0.05, we reject the null hypothesis at the 95% confidence level, suggesting that a real difference exists. However, if we had chosen a 99% confidence level, we would not reject the null hypothesis, because the p-value is greater than 0.01.
Our confidence interval tells us that if we took many samples like the one we have, in 95% percent of the times, we would obtain a difference between averages between -0.62 and -5. This confidence interval does not include 0 and therefore we reject the null hypothesis and accept the hypothesis that there is a difference between the average age of men and women.
T test with BootstrapA T test with bootstrap is a good way of understanding the concepts needed to interpret the results of the T test above. Everything relies on the Central Limit Theorem according to which if I draw many samples of a population and calculate the mean of each sample, then the distribution of all these means will:

(i) follow a normal distribution;

(ii) the mean of the sample means will approximate the population mean;

(iii) the standard deviation of this distribution will be called standard error.

In our example, we have one sample of passengers. Imagine we could collect many of those samples. If we could do that, then the means of all samples would approximate the population parameter. Bootstrap is a technique to virtually create as many samples as we want from our unique sample. In our example, we have 712 ages after eliminating NAs. We could resample 712 observations from these values allowing them to repeat. That is the basic idea behind bootstrapping.

In order to do that procedure, we will create a function that will resample our data frame. The first line of code uses slice_sample to randomly select n rows of our dataframe allowing for the same row to be chosen more than one time. Note that n is the number of rows of the dataframe. After that, we use dplyr to calculate the mean by gender. Note that we are actually interested in the difference between the male mean and the female mean. That’s what the two last lines of code do.

content_copy Copy
diff\_means <- function(data) { sample\_df <- data %>% slice\_sample(n = nrow(data), replace = TRUE) means <- sample\_df %>% group\_by(Sex) %>% summarize(mean\_age = mean(Age, na.rm = TRUE)) male\_mean <- means %>% filter(Sex == "male") %>% pull(mean\_age) female\_mean <- means %>% filter(Sex == "female") %>% pull(mean\_age) return(male\_mean - female\_mean)} Now we can use the replicate function to execute our function for n times. For our purpose 1000 times is enough. Note that replicate works like a for loop. Before we do that, however, let us make a small adjustment so that we can also calculate our p-value. The p-value assumes the null hypothesis is true. Therefore, before resampling our data, let us make the difference between means be 0. For that, let us subtract the difference observed, 2.81, from the ages of all males.

content_copy Copy
df\_null <- df %>% mutate(Age = ifelse(Sex=="male", Age-2.81, Age)) set.seed(1308)diffs <- replicate(1000, diff\_means(df\_null))sd(diffs)mean(diffs)ggplot()+ geom\_histogram(aes(x = diffs), color = "white", fill = "#2E3031")+ geom\_vline(xintercept = -2.8, color = "#A33F3F")+ geom\_vline(xintercept = 2.8, color = "#A33F3F")+ scale\_color\_discrete("")+ xlab("Age Differences (Null Hypothesis)")+ ylab("Number of Individuals")+ theme\_bw() Executing the commands above we get that the mean of the sampling distribution - as the distribution of the sample means is called - is approximately 0, as expected, and its standard deviation is 1.1.

The histogram above shows us how the sample differences would look like if the null hypothesis were true. The red lines show the difference we observed in reality. Do you think it is likely to observe what we observed under the null hypothesis? It is actually not and you can calculate it with the code below:

content_copy Copy
sum(diffs>=2.81)/1000sum(diffs<=-2.81)/1000 The code computes the number of samples whose means were more extrem than 2.8 (male age - female age) or -2.8 (female age - male age). This results in 9 samples out of 1.000, or 0.9%. This estimate is very close to the p-value found using the R function t.test. Again we can reject the null hypothesis and conclude that there is a difference between the average age of men and women.

In addition to helping us better understand the test, the bootstrap method has the advantage of not assuming that the age distribution follows a normal distribution. This is another benefit of using this approach.

Please, use the comments below if you did not understand a specific point of the test or if you have a suggestion to improve the test.

To leave a comment for the author, please follow the link and comment on their blog: coding-the-past.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: T test in R

View Details

[This article was first published on Online College Math Teacher, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Rene Descartes walks into a bar, by Jerry TuttleI recently told the old Rene Descartes joke to a math class: Rene Descartes walks into a bar. The bartender asks, “Would you like a beer?” Descartes pauses for a moment and then replies, “I think not.” Then poof – he disappears. Of course I naively I assumed my students had been exposed to the Descartes quote, “I think, therefore I am.” Philosopher and mathematician Rene Descartes wrote this in French in 1637, and later in Latin as cogito, ergo sum.”

After explaining the Descartes quote, I think the students understood the joke. Well, maybe it’s not that funny.

But perhaps funnier to math people than you realize, is: this joke is logically flawed because the punchline is the inverse tothe original conditional statement, and an inverse is not logically equivalent to the original statement.

Let P and Q be declarative sentences that can be definitively classified as either true or false. Then define:

Conditional Statement: If P, then Q.
Converse: If Q, then P.
Inverse: If not P, then not Q.
Contrapositive: If not Q, then not P

Two conditional statements are defined as logically equivalent when their truth values are identical for every possible combination of truth values for their individual declarative sentences.

The above table shows statement and contrapositive have the same truth values in columns 3 and 6, and so are logically equivalent. Statement and inverse are not logically equivalent.

The Descartes quote is, “If I think, therefore I am”, or “If P then Q”. The punchline is, “If I don’t think, therefore I am not”, or “If not P, then not Q”. The punchline is the inverse, and is not logically equivalent to the quote. If P is false, then “if P then Q” is true regardless of the value of Q. So Q can be either true or false.

Occasionally on television someone, often a police detective, will make a statement where they confuse a statement with its converse or inverse, and I have been known to yell at the television.

Descartes is known for developing analytic geometry, which uses algebra to describe geometry. Descartes’ rule of signs counts the roots of a polynomial by examining sign changes in its coefficients.

And before someone else feels the need to say this, I will: “Don’t put Descartes before the horse.” This is perhaps the punchline to changing the original joke to “A horse walks into a bar … ”

The following is R code to create truth tables. Logical is a variable type in R. Conditional statements in R are createdusing the fact that “If P then Q” is equivalent to “Not P or Q”. I am defining the logic rules for statement, converse, inverse, contrapositive, but I could have defined the rules for more complicated statements as well.

# Define the possible values for P and Q
P <- c(TRUE, TRUE, FALSE, FALSE)
Q <- c(TRUE, FALSE, TRUE, FALSE)

# Calculate the 4 logical rules: statement, converse, inverse, contrapositive

(Note that “if P then Q” is equivalent to “Not P or Q”.)

P_implies_Q <- !P | Q
Q_implies_P <- !Q | P
not_P_implies_not_Q <- P | !Q
not_Q_implies_not_P <- Q | !P

expand.grid(P, Q) would also be a good start, but I wanted a specific ordering

Create a data frame to display the truth table

truth_table <- data.frame(
P = P,
Q = Q,
P -> Q = P_implies_Q,
Q -> P = Q_implies_P,
!P -> !Q = not_P_implies_not_Q,
!Q -> !P = not_Q_implies_not_P
)

# Print the truth table
colnames(truth_table) <- c("P", "Q", "statement", "converse", "inverse", "contrapositive")
print(truth_table)

P_variable <- "I think"
Q_variable <- "I am"

colnames(truth_table) <- c(P_variable, Q_variable, "statement", "converse", "inverse", "contrapositive")
print(truth_table)

End

To leave a comment for the author, please follow the link and comment on their blog: Online College Math Teacher.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Rene Descartes walks into a bar

View Details

[This article was first published on R Code – Geekcologist, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This post has been written by André L. Luza & Vinicius A. G. Bastazini*

In the face of the current biodiversity crisis, understanding how ecological communities respond to species loss is more critical than ever. In a recent study, “Coping with Collapse: Functional Robustness of Coral-Reef Fish Network to Simulated Cascade Extinction“ (see also a previous post on this study) , we shed light on this issue by examining the cascading effects of species extinctions within coral-fish networks in the Southwestern Atlantic. In this post, we will show how to calculate functional network robustness in R, using the methods we developed in our study. The example below is just illustrative, and it is not intend to produce the same results as those reported in our manuscript (see also the GitHub tutorial).

The basis of the framework is the species-habitat network approach (Marini et al. 2019; Bastazini et al. in press), however it can be directly applied to any type of bipartite networks formed by interacting species. The illustrated framework is depicted in the figure below. In step (1), we modeled the occupancy probability of fish as a function of coral and turf algae cover using site occupancy modeling. Based on the model output, we classified species as either coral-associated or co-occurring fish. In step (2), coral and fish species were connected based on the predicted site occupancy probability of each coral-associated fish (fish with different colors in the center of the network) relative to the cover of each coral species (corals with different colors, in the left). Then, coral-associated and co-occurring fish in subnetwork 2 were connected based on Pearson’s correlation values between fish site occupancy probability. Once the networks are constructed, we applied a removal algorithm that eliminated corals and subsequently calculated the direct and indirect effects of coral species removal on network robustness at each elimination step (t=0,t=1,…,|A|). Lost links are shown in red. In step (3), we related fish species composition and species traits at each elimination step. In step (4), we computed the loss in trait space area following corals and fish removal. The area delimited by the black polygon depicts the trait space area at t=0, and the area delimited by the red polygon depicts the trait space area at t=1. Finally, in step (5), we applied a hyperbolic function (non-linear model) to the simulated species elimination data, analyzing both the remaining taxonomic diversity (TD, represented on the first y-axis with a solid curve) and functional diversity (FD, represented on the second y-axis with a dashed curve) along the gradient of coral elimination (x-axis). Note that the approach is based on the analysis of two bipartite networks. Thus, the approach described here can be directly applied to assess the robustness of bipartite networks.

Analytical framework. Please see Luza et al. (2024) for further details

We will simulate some data to illustrate our approach. Consider a set of eight species in the Partite A, 42 species in the Partite B, and 21 species in the Partite C. The subnetwork 1 will be composed by the Partites A and B, and the subnetwork 2 will be composed by the Partites B and C. To fill matrix cells with values, we will gather values constrained between [0,1], resembling fish occupancy probability (ψ) (subnetwork 1) and co-occurrence (Pearson’s correlation ρ) (subnetwork 2) from a Beta distribution with shape parameters a=0.5 and b=1. This will produce a distribution of values high density close to zero.

set.seed(2456)nspA <-8 # sp in Partite AnspB<-42 # sp in Partite BnspC<-21 # sp in Partite Cnspp <- nspB+nspC # all species# create subnetwork 1subnetwork1 <- matrix(rbeta(nspA*nspB, shape1=.5, shape2 = 1),byrow=F, nrow=nspA, ncol=nspB)subnetwork1 [subnetwork1>1]<-1 # set to 1 values larger than 1# create subnetwork 2subnetwork2 <- matrix(rbeta(nspB*nspC,shape1=.5, shape2 = 1),byrow=F, nrow=nspB, ncol=nspC)subnetwork2 [subnetwork2>1]<-1 # set to 1 values larger than 1# set rownames() & colnames()# subnetwork 1rownames(subnetwork1) <- paste0 ("coral", seq(1,nspA))colnames(subnetwork1) <- paste0 ("fish", seq(1,nspB))# subnetwork 2rownames(subnetwork2) <- paste0 ("fish", seq(1,nspB))colnames(subnetwork2) <- paste0 ("fish", seq(nspB+1,nspB+nspC))# establish a threshold of rho = 0.8 for a coocurring speciessubnetwork2[subnetwork2<0.8]<-0# rm empty colssubnetwork2 <- subnetwork2 [, which(colSums(subnetwork2)>0)]# arrange the matrices# ordering (degree) --------------------require(bipartite); require(igraph)subnetwork1<-sortweb(subnetwork1,sort.order="dec")subnetwork2<-sortweb(subnetwork2,sort.order="dec")# define affinitiesrequire(reshape)subnetwork1\_df <- melt(subnetwork1,as.is=T)subnetwork1\_df$aff <- ifelse (subnetwork1\_df$value > 0.9, 1,0)colnames(subnetwork1\_df) <- c("coral", "fish", "value", "aff") Note that in the last step we melted the object “subnetwork_1” and created the object “subnetwork1_df”. We did so to make this new object the base for filter data and to establish a binary variable depicting the relationship between species of partites A and B. Now that we produced the matrices and sorted them out, we can illustrate the tripartite network.

plotweb2(data.matrix(subnetwork1), data.matrix(subnetwork2), method = "normal", empty=T, col.interaction = "gray", ybig=1, labsize = 0.75, spacing=0.01, lab.space =0.1, method2="normal", spacing2=0.01, empty2=T, col.interaction2 = "orange" , col.pred2 = "orange", col.prey2 = "gray80") As we already established the links between species in Partites B and C (fish in our empirical data), we will simulate one trait data set to be used in the trait-based analyzes. The trait values (n=6 traits) of species in the Partites B and C were gathered from Normal distribution with average μ=0 and standard deviation σ=0.2.

```

produce trait datatrait_data <- as.data.frame ( replicate (6, rnorm (n = nspp, mean = 0,sd = 0.2) ) )# set rownames() and colnames()rownames(trait_data) <- paste0 ("fish", seq(1,nspp))colnames(trait_data) <- paste0 ("trait", seq(1,ncol(trait_data)))

``` After getting the trait values, we can start the trait space analyses based on the Partites B and C. First we will calculate the distances between species based on their traits, and run one ordination analysis (Principal Coordinate Analysis, ‘dudi.pco’ function of the package ade4) to summarize these distances into different vectors. We will use the Gower distance (package cluster) as it enables us to handle several types of traits (continuous, categorical, binary traits).

```

trait distance matrix (gower)require(cluster)gower_matrix <- daisy (trait_data, metric=("gower"))# Building the functional space based on a PCOA require(ade4)pco<-dudi.pco(quasieuclid(gower_matrix), scannf=F, nf=10) # quasieuclid() transformation to make the gower matrix as euclidean. nf= number of axis

``` Now we will build the complete trait space. The area of this polygon is the Functional Diversity-FD measure for (t=0), when no species elimination has taken place. We will apply the function ‘chull’ (package grDevices) to the data of the two first PCoA axes, which will link species located in the boundaries of the trait space. Then the function ‘Polygon’ (package sp) will be used to get the polygon, enabling the calculation of the polygon area (FD). These two functions will also be used to group species and obtain FD for other groups (e.g. coral-associated fish, co-occurring fish).

```

complete trait spaceall_pool <- cbind (pco$li[,1:2], ext = F, sp = rownames(trait_data))# convex hulla_pool <- all_pool [chull(all_pool[,1:2], y = NULL),] # its convex hull

``` Then we can start the species removal process. Within the ‘lapply’ loop, there is a sequential selection of rows of the matrix ‘subnetwork1’. As this matrix was already sorted, the selection obeys the degree criterion. The ‘rm_corals’ object define which species in partite A will be selected (from the species highest to the lower degree). Then we selected the species of partite B which have affinity with the selected species (‘aff==1 & coral %in% rm_corals’), define the group of species (using ‘chull’), and calculate FD (using ‘Polygon’ and extracting the ‘@area’). After we create, the ‘RFS’ object, which contains the result of the division of the difference between the complete and simplified trait space area by the complete trait space area. If no trait space was lost then RFS=1. As we were interested in how much remains of the trip space, we calculated 1−RFS to be stored in our output ‘res’.

```

--------------------------------------# removal based on degree of Partite A# calculate functional trait space loss# based on RFS (Luza et al. 2022)require(sp)RFS_corals <- lapply (seq (1, nspA), function (ncoral) { # choose the spp of Partite A to remove rm_corals <- rownames (subnetwork1)[1:ncoral] # coral-associated fish coral_associated <- subnetwork1_df[which(subnetwork1_df$aff == 1 & subnetwork1_df$coral %in% rm_corals ),] # reduced space setB<-cbind(all_pool, ext1=ifelse(all_pool$sp %in% unique(coral_associated$fish),T,F)) pk <-setB[which(setB$ext1==F),] f <- pk [chull(pk, y = NULL),] # hull # quantifying reduction in functional space # https://chitchatr.wordpress.com/2015/01/23/calculating-the-area-of-a-convex-hull/ chull.poly.complete <- Polygon(a_pool[,1:2], hole=F) # complete space chull.area.complete <- chull.poly.complete@area # polygon area # coral chull.poly.coral <- if (nrow (f)==0) {0} else {Polygon(f[,1:2], hole=F)} # reduced space chull.area.coral <- if (nrow (f)==0) {0} else {chull.poly.coral@area} # area # calculating reductions across all corals RFS<-data.frame (corals=(chull.area.complete-chull.area.coral)/chull.area.complete) # reduction in functional space area # results into a dataframe res <- list (remaining_associated_SR = 1-length(unique(coral_associated$fish))/nspB, # proportion of species remaining in partite B remaining_SR_all = 1-length(unique(coral_associated$fish))/nspp, # proportion of species remaining relative to the total RFS = 1-(RFS), # 1-RFS coral.associated = coral_associated$fish[order(coral_associated$fish)], # removed fish corals.removed = rm_corals, # removed corals space.coral = f # trait space to plot ) ; # return res})

``` By running this we will end up with a list with length equal to the number of species in partite A. This output comprise the simulated losses in partite B (coral-associated) produced by the direct losses of species in partite A (corals). Then we melt the list to have the relevant results.

```

total functional lossFD_loss_total <- lapply (RFS_corals, function (i) i$RFS)FD_loss_total<- unlist(FD_loss_total)# loss species richness of partite B (coral associated fish)CA_loss_total <- lapply (RFS_corals, function (i) i$remaining_associated_SR)CA_loss_total<- do.call (rbind, CA_loss_total)# total SR loss ALL_loss_total <- lapply (RFS_corals, function (i) i$remaining_SR_all)ALL_loss_total<- do.call (rbind, ALL_loss_total)# removal of partite A species (Corals)loss_corals <- lapply (RFS_corals, function (i) 1-length(i$corals.removed)/nspA)loss_corals<- do.call (rbind, loss_corals)

``` We now proceed by creating a data frame for the robustness analyzes. First, we create data for t=0, where no loss has taken place yet. Subsequently, we bind to this data frame, the results of the direct species loss.

```

analysis dataset# first step of no loss (t=0)analysis_dataset<- (data.frame (remain_all=1, remain_fish_associated=1, remain_coral=1, remain_FD = 1))# bind removalsanalysis_dataset<-rbind (analysis_dataset, data.frame (remain_all=ALL_loss_total, remain_fish_associated=CA_loss_total, remain_coral=loss_corals, remain_FD = FD_loss_total))# show the headinghead(analysis_dataset)## remain_all remain_fish_associated remain_coral remain_FD## 1 1.0000000 1.0000000 1.000 1.0000000## 2 0.9523810 0.9285714 0.875 1.0000000## 3 0.9047619 0.8571429 0.750 1.0000000## 4 0.8571429 0.7857143 0.625 0.8387945## 5 0.8412698 0.7619048 0.500 0.8387945## 6 0.8095238 0.7142857 0.375 0.8387945

``` Now we will simulate the influence of the indirect loss of corals. We start by selecting the species in partite A (corals) and B (coral-associated fish) to be removed. Then we select associated fish plus those associated with them by checking which species has the sum of ρ>0. These species will be removed and the remaining proportion of species richness (SR) and FD will be calculated.

```

simulating secondary (indirect) extinctions -----------------RFS_corals_secondary_extinctions <- lapply (seq (1, nspA), function (ncoral) { # choose the spp of Partite A to remove rm_corals <- rownames (subnetwork1)[1:ncoral] # coral-associated fish coral_associated <- subnetwork1_df[which(subnetwork1_df$aff == 1 & subnetwork1_df$coral %in% rm_corals ),] # select associated fish plus those associated with them ass_cs <- subnetwork2[unique(coral_associated$fish),] ass_cs <- ass_cs[,colSums(ass_cs)>0] all_to_remove <- unlist(dimnames(ass_cs)) # reduced space setB<-cbind(all_pool, ext1=ifelse(all_pool$sp %in% all_to_remove,T,F)) pk <-setB[which(setB$ext1==F),] f <- pk [chull(pk, y = NULL),] # quantifying reduction in functional space # https://chitchatr.wordpress.com/2015/01/23/calculating-the-area-of-a-convex-hull/ chull.poly.complete <- Polygon(a_pool[,1:2], hole=F) # complete space chull.area.complete <- chull.poly.complete@area # area # coral chull.poly.coral <- if (nrow (f)==0) {0} else {Polygon(f[,1:2], hole=F)} # reduced space chull.area.coral <- if (nrow (f)==0) {0} else {chull.poly.coral@area} # area # calculating reductions across all corals RFS<-data.frame (corals=(chull.area.complete-chull.area.coral)/chull.area.complete) # reduction in functional space area # results res <- list (all_to_remove = all_to_remove, richness_remaining = length(all_to_remove)/nspp, richness_remaining_associated = length(unique(coral_associated$fish))/nspp, RFS = 1-RFS, coral.associated = coral_associated$fish[order(coral_associated$fish)], corals.removed = rm_corals, space.coral = f ) ; # return res})

``` As before, we selected the results we want, melt the lists, and bind to the data frame for robustness analyses.

```

total lossloss_total_secondary <- lapply (RFS_corals_secondary_extinctions, function (i) i$RFS)loss_total_secondary<- do.call (rbind, loss_total_secondary)# richnessloss_total_secondary_SR <- lapply (RFS_corals_secondary_extinctions, function (i) i$richness_remaining)loss_total_secondary_SR<- do.call (rbind,loss_total_secondary_SR)# richness associated fishloss_total_secondary_SR_CA <- lapply (RFS_corals_secondary_extinctions, function (i) i$richness_remaining_associated)loss_total_secondary_SR_CA<- do.call (rbind,loss_total_secondary_SR_CA)# bind indirect extinctions in the analysis datasetanalysis_dataset<-cbind(analysis_dataset, data.frame (remain_RFS_secondary = c(1,unlist(loss_total_secondary)), remain_SR_secondary = c(1,1-unlist (loss_total_secondary_SR)), remain_SR_secondary_associated = c(1,1-unlist (loss_total_secondary_SR_CA)), ext.lower=1, no = seq(0,nspA)/nspA ) )

``` Next we present the function and fit the non-linear model to the data. The object that the function requires is a data frame with the proportion of species remaining in partite A and the remaining SR or FD in the other partites. The function also have a function to plot.

```

run function of hyperbolic curve -------------------------------------------------# the function# Fit hyperbolic functionfit.hyperbolica<-function (object, plot.it = T, ...) { #if (class(object) != "bipartite") # stop("This function cannot be meaningfully applied to objects of this class.") N <- colSums(object) #if (all(object[-nrow(object), 2] == 1)) { # y <- -object[, 3]} else {y <- -object[, 2]} y <- object[,3]#(sum(y) - cumsum(y))/sum(y) x <- (object[, "no"]/max(object[, "no"])) fit <- try(nls(y ~ 1 - x^a, start = list(a = 0.5))) # originally set to 1, but used here 0.5 to make the example work if (class(fit) == "try-error") fit <- nls((y + rnorm(length(y), s = 0.001)) ~ 1 - x^a, start = list(a = 1)) # option to plot if (plot.it) { par(mar = c(5, 5, 1, 1)) plot(x, y, xlab = "Fraction of eliminated corals", ylab = "Fraction of surviving fish", axes = TRUE, type = "n", cex.lab = 1) points(x, y)#, ...) lines(seq(0, 1, length=9), predict(fit, newdata = data.frame(x = seq(0, 1, length=9))), col = "red", lwd = 2) } return(list(exponent = as.numeric(coef(fit)[1]), x=x, y=y, model=fit, preds=predict(fit, newdata = data.frame(x = seq(0, 1, length=9)))))}par(mfrow=c(2,2))# run functionhyper_curve_secondary <- fit.hyperbolica (analysis_dataset[,c("no", "ext.lower", "remain_all")], plot.it = F)# you can plot if you wanthyper_curve_associated <- fit.hyperbolica (analysis_dataset[,c("no", "ext.lower", "remain_SR_secondary")], plot.it = F)hyper_curve_associated_RFS <- fit.hyperbolica (analysis_dataset[,c("no", "ext.lower", "remain_FD")], plot.it = F)hyper_curve_secondary_RFS <- fit.hyperbolica (analysis_dataset[,c("no", "ext.lower", "remain_RFS_secondary")], plot.it = F)

``` After fitting the model to the data, we bind the predictions to the analysis dataset, and estimated network robustness by integrating (summing up) infinitesimally small values of the spline interpolated using the fitted hyperbolic function applied to the minimum and maximum of the proportion of removed partite A species/corals.

```

predictionsanalysis_dataset$pred_remain_SR_all <- hyper_curve_secondary$predsanalysis_dataset$pred_remain_SR_secondary <- hyper_curve_associated$predsanalysis_dataset$pred_remain_RFS_associated <- hyper_curve_associated_RFS$predsanalysis_dataset$pred_remain_RFS_secondary <- hyper_curve_secondary_RFS$preds# robustness# srSR1 <- (integrate(splinefun(hyper_curve_secondary_RFS$x, analysis_dataset$pred_remain_SR_all), min(hyper_curve_secondary_RFS$x), max(hyper_curve_secondary_RFS$x), subdivisions = max(100L, length(hyper_curve_secondary_RFS$x))))# SR secondarySR2 <- (integrate(splinefun(hyper_curve_secondary_RFS$x, analysis_dataset$pred_remain_SR_secondary), min(hyper_curve_secondary_RFS$x), max(hyper_curve_secondary_RFS$x), subdivisions = max(100L, length(hyper_curve_secondary_RFS$x))))# FD cora lassociatedFD1 <- (integrate(splinefun(hyper_curve_secondary_RFS$x, analysis_dataset$pred_remain_RFS_associated), min(hyper_curve_secondary_RFS$x), max(hyper_curve_secondary_RFS$x), subdivisions = max(100L, length(hyper_curve_secondary_RFS$x))))# fd otehrFD2 <- (integrate(splinefun(hyper_curve_secondary_RFS$x, analysis_dataset$pred_remain_RFS_secondary), min(hyper_curve_secondary_RFS$x), max(hyper_curve_secondary_RFS$x), subdivisions = max(100L, length(hyper_curve_secondary_RFS$x))))# create a table to show robustness resultsrequire(dplyr); require(knitr)data.frame(rbind(SR_direct = SR1$value, SR_indirect = SR2$value, FD_direct = FD1$value, FD_indirect = FD2$value )) %>% kable(format = "pipe",col.names = "Robustness (R)")

``` Finally, we can now plot the robustness curves for direct and indirect effects of the removal of partite A species on the other partities.

```

plotrequire(ggplot2)# plotggplot(analysis_dataset[order(analysis_dataset$remain_coral,decreasing=F),], aes (x= (1-remain_coral), y = remain_fish_associated)) + # functional diversity # direct loss geom_line( aes(y=pred_remain_RFS_associated),linetype=1,size=1.2,col="orange") + geom_ribbon( aes(ymax=pred_remain_RFS_associated), ymin=0, linetype=1, size=1.2, col = NA, alpha=0.3, fill="orange") + geom_point( aes(y=remain_FD), size=3, stroke=2, col="orange", fill="white", shape=23, alpha=1) + # indirect loss geom_line( aes(y=pred_remain_RFS_secondary),linetype=1,size=1.2, col = "orange3") + geom_ribbon( aes(ymax=pred_remain_RFS_secondary), ymin=0, linetype=1, size=1.2, col = NA, alpha=0.4, fill="orange3") + geom_point( aes(y=remain_RFS_secondary),size=3, stroke=2, col="orange3", fill="white", shape=21, alpha=1) + # taxonomic diversity geom_ribbon( aes(ymax=pred_remain_SR_all), ymin=0, linetype=1, size=1.2, col = NA, alpha=0.3, fill="gray50") + geom_line( aes(y=pred_remain_SR_all), linetype=1,size=1.2,col="gray50") + geom_point( aes(y=remain_all),size=3, stroke=2, col="gray50", fill="white", shape=23, alpha=1) + #geom_line( aes(y=remain_fish_associated100),linetype=1,size=1.2,col="orange") + geom_line( aes(y=pred_remain_SR_secondary),linetype=1,size=1.2,col="black") + geom_ribbon( aes(ymax=pred_remain_SR_secondary), ymin=0, linetype=1, size=1.2, col = NA, alpha=0.8, fill="black") + geom_point( aes(y=remain_SR_secondary), size=3, stroke=2, col="black", fill="white", shape=21, alpha=1) + scale_y_continuous( # Features of the first axis name = "Remaining Fish Taxonomic Diversity", # Add a second axis and specify its features sec.axis = sec_axis(trans=~.1, name="Remaining Fish Functional Diversity") ) + xlab ("Proportion of corals eliminated") + theme_bw() + theme (axis.title=element_text(size=14), axis.text = element_text(size=12), plot.title = element_text(size=18))

``` Attack Tolerance Curves (ATC). The shaded area below each curve depicts the hyperbolic function curve fitted to the data shown in the two Y-axes. Yellow tones represent losses in functional diversity, while grey and black represent losses in taxonomic diversity. Diamonds and circles denote the direct and indirect effects of coral species extinction on such biodiversity dimensions.

References

Bastazini VAG, Gianuca AT, Vizentin-Bugoni J, Gonçalves MSS & Dias RA . In press. Identificando o potencial de conservação de áreas úmidas usando ferramentas da teoria de redes complexas. In: Gonçalves, MSS, Bastazini VAG, Andretti C, Lanés LE, Volcan M (orgs). Biodiversidade e conservação de ecossistemas aquáticos do sul do Brasil. Editora USEB.

Luza AL, Bender MG, Ferreira CEL, Floeter SR, Francini-Filho RB, Longo GO, Pinheiro HT, Quimbayo JP, Bastazini VAG. 2024. Coping with collapse: Functional robustness of coral-reef fish network to simulated cascade extinction. Global Change Biology 30: e17513. https://doi.org/10.1111/gcb.17513

Marini L, Bartomeus I, Rader R, Lami F. 2019. Species–habitat networks: A tool to improve landscape management for conservation. Journal of Applied Ecology 56: 923–928. https://doi.org/10.1111/1365-2664.13337

To leave a comment for the author, please follow the link and comment on their blog: R Code – Geekcologist.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Estimating ecological network robustness with R: A functional approach*

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Introduction to File Manipulation in LinuxLinux file manipulation is a fundamental skill for managing data efficiently. This guide will introduce you to essential commands like cp, mv, mkdir, rm, and ln, which are crucial for handling files and directories. I hope with this blog post you will learn something just like I did. Remember, I too and learning as I go. So if you are a seasoned Linux user, please feel free to provide feedback in the comments.

Understanding the Linux File SystemBefore getting into commands, it’s important to understand the Linux file system’s hierarchical structure, which organizes files and directories.

Basic Commands OverviewCommand Options Table

| Command | Option | Description | | --- | --- | --- | | cp | -r | Recursively copy directories and their contents. | | -i | Prompt before overwriting files. | | -u | Copy only when the source file is newer than the destination file or when the destination file is missing. | | mv | -i | Prompt before overwriting files. | | -u | Move only when the source file is newer than the destination file or when the destination file is missing. | | mkdir | -p | Create parent directories as needed. | | rm | -r | Recursively remove directories and their contents. | | -i | Prompt before every removal. | | -f | Force removal without prompt. | | ln | -s | Create symbolic links instead of hard links. | | -f | Remove existing destination files. |

Copying Files and Directories with cpThe cp command is used to copy files and directories. Learn its syntax and options to efficiently duplicate data.

Syntax and Options* Basic syntax: cp [options] source destination * Use -r for recursive copying of directories.

Examples of Use* Copy a file: cp file1.txt file2.txt * Copy a directory: cp -r dir1/ dir2/

Moving and Renaming Files with mvThe mv command moves or renames files and directories.

Syntax and Options* Basic syntax: mv [options] source destination * Use -i to prompt before overwriting.

Examples of Use* Move a file: mv file1.txt /new/location/ * Rename a file: mv oldname.txt newname.txt

Creating Directories with mkdirThe mkdir command creates new directories.

Syntax and Options* Basic syntax: mkdir [options] directory_name * Use -p to create parent directories as needed.

Examples of Use* Create a directory: mkdir new_directory * Create nested directories: mkdir -p parent/child/grandchild

terminal@terminal-temple dir2 $ mkdir -p parent/child/grandchildterminal@terminal-temple dir2 $ lsfun parentterminal@terminal-temple dir2 $ cd parentterminal@terminal-temple parent $ lschildterminal@terminal-temple parent $ cd childterminal@terminal-temple child $ lsgrandchildterminal@terminal-temple child $ cd grandchildterminal@terminal-temple grandchild $ ls Removing Files and Directories with rmThe rm command deletes files and directories.

Syntax and Options* Basic syntax: rm [options] file_name * Use -r to remove directories and their contents.

Examples of Use* Remove a file: rm file1.txt * Remove a directory: rm -r directory_name

terminal@terminal-temple dir2 $ rm -r parentterminal@terminal-temple dir2 $ lsfun Creating Links with lnThe ln command creates links between files.

Hard Links vs. Soft Links Hard links: Direct pointers to the data on disk. * Soft links (symbolic links)*: Pointers to the file name.

Examples of Use* Create a hard link: ln file1.txt link1.txt * Create a symbolic link: ln -s file1.txt symlink1.txt

Using Wildcards in LinuxWildcards are special characters used in commands to match multiple files or directories. They simplify file manipulation by allowing you to specify patterns instead of explicit names.

Wildcard Characters Table

| Wildcard | Meaning | | --- | --- | | * | Matches any number of characters, including none. | | ? | Matches exactly one character. | | [ ] | Matches any one of the enclosed characters. | | [! ] | Matches any character not enclosed. | | [[:class:]] | Matches any character in the specified class. |

Commonly Used Character Classes Table

| Character Class | Meaning | | --- | --- | | [:digit:] | Matches any digit. | | [:lower:] | Matches any lowercase letter. | | [:upper:] | Matches any uppercase letter. | | [:alpha:] | Matches any letter. | | [:alnum:] | Matches any alphanumeric character |

Wildcard Examples Table

| Pattern | Matches | | --- | --- | | *.txt | All files ending with .txt | | file?.txt | Files like file1.txt, fileA.txt but not file12.txt | | data[0-9].csv | Files like data1.csv, data9.csv | | report[!0-9].doc | Files like reportA.doc, reportB.doc but not report1.doc | | *[[:lower:]123] | Files with lowercase letters or digits 1, 2, or 3 |

Creating A SandboxTo practice file manipulation safely, create a sandbox directory to experiment with commands without affecting important data.

Creating The Directory1. Create a new directory: mkdir sandbox

terminal@terminal-temple ~ $ lsDocuments Downloads Music my\_new\_directory Picturesterminal@terminal-temple ~ $ mkdir sandboxterminal@terminal-temple ~ $ lsDocuments Downloads Music my\_new\_directory Pictures sandboxterminal@terminal-temple sandbox $ mkdir dir1 terminal@terminal-temple sandbox $ mkdir dir2terminal@terminal-temple sandbox $ lsdir1 dir2 Copying Some Files2. Copy some files into the sandbox directory.

terminal@terminal-temple sandbox $ cp ../my\_new\_directory/my\_new\_subdirectory/new\_file.txt sandbox.txtterminal@terminal-temple sandbox $ lsdir1 dir2 sandbox.txtterminal@terminal-temple sandbox $ ls -ltotal 2drwxr-xr-x 2 terminal staff 64 Sep 27 07:44 AM dir1drwxr-xr-x 2 terminal staff 64 Sep 27 07:45 AM dir2-rwxr--r-- 1 terminal staff 0 Sep 27 07:50 AM sandbox.txt Moving Files3. Move a file from one directory to another.

terminal@terminal-temple sandbox $ mv sandbox.txt funterminal@terminal-temple sandbox $ lsdir1 dir2 funterminal@terminal-temple sandbox $ mv fun dir1terminal@terminal-temple sandbox $ lsdir1 dir2terminal@terminal-temple sandbox $ cd dir1terminal@terminal-temple dir1 $ ls -ltotal 0-rwxr--r-- 1 terminal staff 0 Sep 27 07:54 AM funterminal@terminal-temple sandbox $ mv dir1/fun dir2terminal@terminal-temple sandbox $ cd dir2terminal@terminal-temple dir2 $ lsfunterminal@terminal-temple dir2 $ ls -ltotal 0-rwxr--r-- 1 terminal staff 0 Sep 27 07:54 AM fun Understanding Recursive OperationsRecursive operations are essential for managing directories and their contents effectively. When a command operates recursively, it processes all files and subdirectories within a specified directory. This is particularly useful for tasks that involve entire directory trees, such as copying, moving, or deleting files en masse.

Key Points: Recursive Option (-r or -R): Many Linux commands, such as cp, rm, and chmod, offer a recursive option to apply actions to all files within a directory and its subdirectories. * Use Cases: Recursively copying directories (cp -r source/ destination/), deleting directories (rm -r directory_name), or changing permissions (chmod -R 755 directory). * Caution*: Recursive commands can potentially affect a large number of files, so it’s crucial to use them carefully to avoid unintended changes or data loss.

Common Mistakes and How to Avoid ThemWhen manipulating files and directories, beginners often encounter pitfalls that can lead to data loss or system issues. Here’s how to avoid these common mistakes:

Key Mistakes: Accidental Deletion: Using rm without caution can lead to permanent data loss. * Overwriting Files*: Commands like cp and mv can overwrite files without warning.

Prevention Tips: Interactive Prompts: Use the -i option with commands like rm and cp to prompt before overwriting or deleting files (e.g., rm -i file.txt). * Backups: Regularly back up important data to prevent loss. * Double-Check Commands*: Before executing, review command syntax and options, especially for recursive operations.

Practical Examples and Use CasesUnderstanding practical applications of these commands will enhance your file management skills:

Examples: Batch File Operations: Use cp and mv for batch operations on multiple files using wildcards (e.g., cp *.txt backup/). * Directory Organization: Utilize mkdir to organize files into directories (e.g., mkdir -p projects/2024/january). * File Cleanup*: Regularly use rm to clean up temporary files and maintain system efficiency.

Advanced Tips for Efficient File ManagementEnhance your command-line proficiency with these advanced techniques:

Tips: Wildcard Combinations: Use wildcards to efficiently target multiple files (e.g., rm *.log removes all log files). * Command Chaining: Combine commands using && or ; to execute multiple tasks in sequence (e.g., mkdir new_dir && cd new_dir). * Scripting*: Write shell scripts to automate repetitive tasks, improving efficiency and reducing errors.

Troubleshooting Common IssuesAddressing common issues can save time and prevent frustration:

Solutions: Command Not Found: Ensure that the command is installed and correctly spelled. * Permission Denied: Use sudo to execute commands with elevated privileges if necessary and if you are sure you know what you are doing. * File Not Found*: Verify file paths and names, especially when using relative paths.

Security ConsiderationsSecurity is crucial when manipulating files, particularly on shared or sensitive systems:

Key Considerations: File Permissions: Use chmod to set appropriate permissions, restricting access to sensitive files. * Ownership: Use chown to set correct ownership, especially when files are shared among multiple users. * Safe Deletion*: Consider using tools like shred for securely deleting files.

Conclusion and Best PracticesMastering file and directory manipulation is vital for effective Linux system management. By understanding command syntax, using options wisely, and adhering to best practices like regular backups and cautious use of recursive operations, you can efficiently manage your files while minimizing the risk of errors or data loss.

Best Practices:* Regularly back up important data. * Use interactive prompts to confirm destructive actions. * Employ wildcards and scripting for efficient file management. * Pay attention to file permissions and ownership for security.

By following these guidelines and continuously practicing, you’ll develop robust file management skills that are essential for any Linux user.

Quick Takeaways* Master basic commands: cp, mv, mkdir, rm, ln. * Use options wisely to enhance command functionality. * Practice safe file manipulation to avoid data loss.

FAQs1. What is the difference between cp and mv? * cp copies files, while mv moves or renames them. 2. How do I create a directory in Linux? * Use the mkdir command, e.g., mkdir new_directory. 3. Can I recover files deleted with rm? * Generally, no. Use caution and consider backups. 4. What are hard links and soft links? * Hard links point directly to data; soft links point to file names. 5. How do I avoid accidental file deletion? * Use the -i option with rm to prompt before deletion.

Your TurnI hope this guide helps you master file manipulation in Linux. Please share your feedback and share this article with others who might find it useful!

References* Linux Command Line Basics * GNU Core Utilities * Linux Documentation Project

Happy Coding!

A Command Line To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering File and Directory Manipulation in Linux: A Beginner’s Guide

View Details

[This article was first published on R – Win Vector LLC, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Nina Zumel shares the following puzzle from the December 1908 issue of The Strand Magazine:

100 bushes of corn are distributed to 100 people such that every man receives 3 bushels, every woman 2 bushels, and every child 1/2 a bushel. How many men, women, and children are there?

Check out some of the background and how to solve it here.

To leave a comment for the author, please follow the link and comment on their blog: R – Win Vector LLC.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The 100 Bushels Puzzle

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Dear rOpenSci friends, it’s time for our monthly news roundup!

You can read this post on our blog.Now let’s dive into the activity at and around rOpenSci!

rOpenSci HQCommunity call recording: Navigating the R ecosystem using R-universeVideo and resources.

Learn more about R-Universe and how you can use it to improve your R package development workflow.

In this community call, Jeroen Ooms provided details on what R-Universe is and an update on what you can do with it today.He also discussed the future of R-Universe and how it can be used to navigate the R ecosystem.

R-Universe list pagesR-Universe now features a list of all datasets in all packages and a list of all vignettes from CRAN, Bioconductor, and others.

Another important global table is the package scoreboard that lets you explore the scores used by R-Universe search engine for ranking, which is based on:

Stars: Number of stars on GitHub;
Downloads per month from CRAN or Bioconductor mirrors;
Scripts: (NEW) number of files on GitHub that mention ’library(pkgname)’;
Dependents: recursive reverse dependencies;
Unique contributors;
Yearly commits.

Resources from rOpenSci training sessionsWe’ve added a searchable table of past training sessions to our resources hosted on the rOpenSci website.Explore the full list of our resources.

Noam Ross, rOpenSci, and communities empowering open scienceDataLatam Podcast interviewed rOpenSci director Noam Ross.Learn about his work at the intersection of health and the environment and his journey with rOpenSci.

Materials: Screen Reader Accessible Tools and Resources for Learning and Working with RWe have now published the recording of the webinar that walks through learning and using R with screen readers — and it’s bilingual (English & Turkish)! with subtitles in English.

Webinar video featuring Liz Hare, PhDand Alican Cagri Gokcek

Also, don’t miss the detailed technical note by Liz Hare with all the necessary resources to get started.

The rOpenSci community at upcoming eventsrOpenSci Community Manager Yanina Bellini Saibene will deliver a keynote talk at BioNT Community Event & CarpentryConnect-Heidelberg 2024, on November 14th in Heidelberg, Germany.

Will Landau will deliver a keynote talk at LatinR (Latin American Conference About the Use of R in R&D). The conference is happening online from November 18-22.

CoworkingRead all about coworking!

Join us for social coworking & office hours monthly on first Tuesdays!Hosted by Steffi LaZerte and various community hosts.Everyone welcome.No RSVP needed.Consult our Events page to find your local time and how to join.

Tuesday, October 1st, 14h00 Europe Central (12:00 UTC), “Using R to tell (data science) stories” with cohost Apoorv Anand and Steffi LaZerte.
- Explore how others use R to tell data stories
- Start work on a project using R to craft a narrative through data
- Chat with our cohost to discuss and explore the use of R in data stories
Tuesday, November 5th, 09:00 Americas Pacific (17:00 UTC), “Getting to know Openscapes” with cohost Stefanie Butland and Steffi LaZerte.
- Look up Openscapes
- Chat with our cohost to learn more about the Openscapes organization, and how they use R to automate creating open education infrastructure

And remember, you can always cowork independently on work related to R, work on packages that tend to be neglected, or work on what ever you need to get done!

Software New packagesThe following two packages recently became a part of our software suite:

eDNAjoint, developed by Abigail G. Keller: Models integrate environmental DNA (eDNA) detection data and traditional survey data to jointly estimate species catch rate (see package vignette: https://bookdown.org/abigailkeller/eDNAjoint_vignette/). Models can be used with count data via traditional survey methods (i.e., trapping, electrofishing, visual) and replicated eDNA detection/nondetection data via polymerase chain reaction (i.e., PCR or qPCR) from multiple survey locations. Estimated parameters include probability of a false positive eDNA detection, a site-level covariates that scale the sensitivity of eDNA surveys relative to traditional surveys, and catchability coefficients for traditional gear types. Models are implemented with a Bayesian framework (Markov chain Monte Carlo) using the Stan probabilistic programming language. It has been reviewed by Chitra M Saraswati and Saras Windecker.
rix, developed by Bruno Rodrigues together with Philipp Baumann: Simplifies the creation of reproducible data science environments using the Nix package manager, as described in Dolstra (2006) . The included rix() function generates a complete description of the environment as a default.nix file, which can then be built using Nix. This results in project specific software environments with pinned versions of R, packages, linked system dependencies, and other tools. Additional helpers make it easy to run R code in Nix software environments for testing and production. It is available on CRAN. It has been reviewed by David Watkins and Jacob Wujiciak-Jens.

Discover more packages, read more about Software Peer Review.

New versionsThe following thirteen packages have had an update since the last newsletter: comtradr (v1.0.2), DataPackageR (v0.16.1), datefixR (v1.7.0), epubr (v0.6.5), git2rdata (v0.4.1), GSODR (v4.1.2), jagstargets (1.2.1), mregions2 (v1.1.1), occCite (v0.5.8), osmapiR (v0.2.1), robotstxt (v0.7.15), tarchetypes (0.10.0), and tiler (v0.3.2).

Software Peer ReviewThere are fourteen recently closed and active submissions and 6 submissions on hold. Issues are at different stages:

Three at ‘6/approved’:
- eDNAjoint, Joint Modeling of Traditional and Environmental DNA Survey Data. Submitted by Abigail Keller.
- chopin, CHOPIN: Computation for Climate and Health research On Parallelized INfrastructure. Submitted by Insang Song.
- rix, Rix: Reproducible Environments with Nix. Submitted by Bruno Rodrigues.
- Two at ‘5/awaiting-reviewer(s)-response’:
- rsi, Efficiently Retrieve and Process Satellite Imagery. Submitted by Michael Mahoney.
- agromet, Índices y Estadísticos Climáticos e Hidrológicos. Submitted by Paola Corrales.
- One at ‘4/review(s)-in-awaiting-changes’:
- cancerprof, API Client for State Cancer Profiles. Submitted by Brian Park.
- Five at ‘3/reviewer(s)-assigned’:
- gigs, Assess Fetal, Newborn, and Child Growth with International Standards. Submitted by Simon Parker. (Stats).
- emodnet.wfs, Access EMODnet Web Feature Service data through R. Submitted by Maëlle Salmon.
- gigs, Assess Fetal, Newborn, and Child Growth with International Standards. Submitted by Simon Parker. (Stats).
- sits, Satellite Image Time Series Analysis for Earth Observation Data Cubes. Submitted by Gilberto Camara.
- fwildclusterboot, Fast Wild Cluster Bootstrap Inference for Linear Models. Submitted by Alexander Fischer. (Stats).
- Two at ‘2/seeking-reviewer(s)’:
- QuadratiK, A Collection of Methods Using Kernel-Based Quadratic Distances for. Submitted by Giovanni Saraceno. (Stats).
- galamm, Generalized Additive Latent and Mixed Models. Submitted by Øystein Sørensen. (Stats).
- One at ‘1/editor-checks’:
- mbquartR, Finding Manitoba Quarter Sections. Submitted by Alex Koiter.

Find out more about Software Peer Review and how to get involved.

On the blog* Deposits In The Wild by Collin Schwantes. Using the deposits and frictionless packages to create FAIR data for a complex project. * Resources For Using R With Screen Readers by Liz Hare. Tools and resources to use R with screen readers. Other languages: Recursos para utilizar R con lectores de pantalla (es), Ekran Okuyucularla R Kullanımı İçin Kaynaklar (tr). * Capturing Screenshots Programmatically With R by Maëlle Salmon. * Labels For Technical Writing Projects by Greg Wilson. Greg discusses the challenges of organizing non-software projects in GitHub and shares the label system he uses for technical writing projects like books and lessons.

Calls for contributionsCalls for maintainersIf you’re interested in maintaining any of the R packages below, you might enjoy reading our blog post What Does It Mean to Maintain a Package?.

medrxivr, access and search MedRxiv and BioRxiv preprint data. Issue for volunteering.
USAboundaries (and USAboundariesdata), historical and contemporary boundaries of the United States of America . Issue for volunteering.
historydata, datasets for historians. Issue for volunteering.

Calls for contributions* Help make qualtRics better! Code contributions wanted * Help make assertr better! Come close issues * Help users and developers of targets by answering questions! * Help make waywiser better! User requests wanted

Also refer to our help wanted page – before opening a PR, we recommend asking in the issue whether help is still needed.

Package development cornerSome useful tips for R package developers.

Reminder: R Consortium Infrastructure Steering Committee (ISC) Grant Program Accepting Proposals until October 1st!The R Consortium Call for Proposals might be a relevant funding opportunity for your package!If you can’t prepare your proposal in time, the next call should start March 1st 2025.

How to demonstrate an error in an R Markdown vignetteAn useful piece of information shared by Hugo Gruson in our Slack workspace.To demonstrate an error in a vignette, try setting purl = FALSE in chunks with error = TRUE.More info in a knitr issue.

Better titles for roxygen2 “family” cross-referencesDo you use the “family” roxygen2 tag for grouping and cross-linking manual pages?Did you know you can override the default title from, for instance, “Other blabla” to “blabla functions and related data”?If not, refer to roxygen2 documentation.

Create and use a custom roxygen2 tagRead more about creating a custom tag that adds a new section to a manual page in this blog post by Maëlle Salmon.

Materials from posit::conf(2024) workshopsMaterials from posit::conf(2024) workshops have been posted.Especially relevant to package developers: From R User to R Programmer by Emma Rand and Ian Lyttle; and Package Development: The Rest of the Owl by Jenny Bryan whose introduction slidedeck features GitHub search tips.

Some tips when searching GitHub issuesSpeaking of GitHub search…

Michael Chirico shared the following tip on Mastodon: “[you can use] the ‘in:title’ operator in GitHub search to restrict a search to issues/PRs matching a query in the title”.

Jenny Bryan answered with another tip: “involves:USERNAME” for “I was once part of a conversation about this”.

A usethis function to create a test helper fileThe new usethis function usethis::use_test_helper("mocks") will create a helper file under tests/testthat/helper-mocks.R.

A devtools function for building the README and vignettesNot a new function, but in case you missed it…The devtools::build_rmd() and devtools::build_readme() functions render R Markdown files using a temporary install of your current development package.

If you haven’t subscribed to our newsletter yet, you can do so via a form. Until it’s time for our next newsletter, you can keep in touch with us via our website and Mastodon account.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci News Digest, September 2024

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Disclaimer: While my work in this series draws inspiration from the IBCS® standards, I am not a certified IBCS® analyst or consultant. The visualizations and interpretations presented here are my personal attempts to apply these principles and may not fully align with the official IBCS® standards. I greatly appreciate the insights and framework provided by IBCS® and aim to explore and learn from their approach through my own lens.

As we reach the final letter in the IBCS acronym — S for Structure — this chapter marks the concluding episode of our series. Just like the final piece of an architectural blueprint brings the whole design together, structure is what holds everything in a report in place, ensuring it functions cohesively.

Whether in architecture or data science, a solid structure is indispensable. A well-structured report, just like a sturdy building, ensures clarity, ease of understanding, and actionable insights. The International Business Communication Standards (IBCS) emphasize structure as a cornerstone of effective reporting, advocating for designs that minimize confusion and promote consistent, logical organization.

Structured reporting serves several essential purposes:

Clarity: It leads the reader step-by-step, from high-level insights to detailed data, reducing cognitive overload.
Consistency: By using a clear, repetitive format, readers know exactly where to find key information, speeding up comprehension.
Comprehensive Coverage: A structured approach ensures that nothing important is missed, and that all aspects of the data are presented logically and cohesively.

In this chapter, we will explore how to apply IBCS principles of structure in report design, focusing on consistent items, non-overlapping structures, logical hierarchies, and both deductive and inductive reasoning. By adhering to these guidelines, you can ensure your reports are not only structured but also highly effective in delivering meaningful insights.

Using Consistent ItemsOne of the core principles of structured reporting is consistency. Just as an architect uses standardized materials across a building to ensure uniformity and stability, reports must use consistent elements throughout to avoid confusion and ensure clarity. In IBCS-aligned reports, consistency goes beyond visual appeal — it helps the reader follow the logic and structure with minimal mental effort.

Here are key areas where consistency is crucial:

Consistent Use of Terms and MetricsWhen reporting, it’s important to define key terms, metrics, and labels early on and apply them uniformly throughout the report. For instance, if “Revenue” is used in one section, it should not be referred to as “Income” in another unless explicitly defined. This standardization ensures that readers interpret the data in the same way every time they encounter a term.

Consistent Report ElementsJust like an architect uses the same type of windows, doors, and beams across a building, structured reports need to employ uniform elements such as charts, tables, and graphs. For example, if a specific chart format is used to display monthly sales in one chapter, the same format should be used to display similar data elsewhere. This not only helps in maintaining uniformity but also enhances the report’s usability, as readers can easily compare data across sections.

Consistent Visual DesignVisual design plays an important role in guiding the reader’s attention. Icons, fonts, colors, and layout should remain consistent to avoid distractions. This includes using the same iconography for recurring themes, consistent color coding for different types of data (e.g., red for losses, green for gains), and maintaining uniform font sizes and types for headings and content. This consistency in design contributes to the overall readability and professionalism of the report.

By ensuring consistency in all these elements, we create a seamless experience for the reader, allowing them to focus on the content without being distracted by variations in terminology, design, or data representation.

Using Consistent Grammar and StructureIn addition to consistent items, maintaining uniform grammar and sentence structure across reports is crucial for readability and comprehension. Just as a building needs symmetry and balance to provide ease of navigation, consistent wording and structure in your reports reduce cognitive load, making it easier for readers to interpret the information.

Grammar ConsistencyThe tone, grammatical tense, and voice of your reports should remain consistent. For example, if one section uses passive voice (“The revenue was calculated”), other sections should follow the same style. Switching between active and passive voice, or between past and present tense, can confuse readers and break the flow of the report.
Imperative or Declarative Statements: If you start by using declarative sentences (“Sales increased by 15%”), ensure the rest of the report uses the same approach. Alternatively, if the report uses an imperative style (“Consider this growth trend”), the same style should be maintained throughout.
Consistent Sentence StructureKeep the structure of your statements similar across different sections. For example, use the same sequence for comparative statements: always start with the metric being compared, followed by the time frame or group, and finally the comparison itself. This allows readers to easily scan and absorb information without reinterpreting each sentence.
Example: “Revenue increased by 10% in Q1 compared to Q4” versus “In Q1, the revenue was 10% higher than in Q4.” While both sentences convey the same information, switching between these structures in the same report can cause unnecessary friction for the reader.
Harmonizing Bullet Points and ListsWhen using lists or bullet points, maintain a parallel structure. All items should follow the same grammatical pattern, either starting with a verb, a noun, or an adjective.

Example:

“Increased sales”
“Expanded customer base”
“Launched new products”

Versus:

“Sales have increased”
“There is an expansion in the customer base”
“The new products were launched”

Maintaining a uniform grammatical structure for bullet points and lists helps in retaining the flow and uniformity of the report.

Using Consistent Graphical ElementsJust as an architect ensures that visual harmony exists in a building’s design, the same applies to the graphical elements in a report. Icons, colors, and chart designs should follow a uniform style across all reports. This consistency in visual representation improves readability and reinforces the structure of the data.

Consistent Use of Icons and SymbolsIcons and symbols are valuable tools for visualizing key points, but they can only be effective if used consistently. If one icon represents a concept (e.g., a dollar sign for revenue), it should always represent that concept across the entire report. Using multiple icons for the same idea creates confusion and reduces the effectiveness of the visual aids.
Example: If a red arrow is used to denote a decrease in one chart, ensure that the same red arrow is used in all other charts or tables to indicate a decrease. Consistent icon usage aids quick recognition and interpretation.
Standardized Color SchemeColor is one of the most powerful tools in reporting, but inconsistency can quickly lead to misinterpretation. Adhere to a predefined color palette where specific colors have specific meanings (e.g., red for losses, green for gains). This allows readers to interpret charts and tables intuitively without needing to consult a legend repeatedly.
Example: Use a color scheme where:
Red = negative performance
Green = positive performance
Blue = neutral or baseline indicators

This consistent use of colors ensures that readers can quickly grasp the meaning of data across all visual elements.

Uniform Chart StylesAll charts should follow the same design principles, whether they are bar charts, pie charts, or line graphs. For example, if one section of a report uses a bar chart to display quarterly revenue, any other section presenting revenue data should use the same chart type and layout to maintain visual continuity.
Axes and Scales: Ensure that the scales on axes are consistent across charts where comparisons are made. Discrepancies in scale between charts showing similar data can mislead readers and skew their understanding.
Labels and Titles: Use uniform fonts, sizes, and placement for all chart labels and titles to maintain a cohesive look.
Visual Representation of Data TypesDifferent data types (e.g., percentages, absolute figures, trends) should always be represented in the same format across all charts. For example, percentages might always be shown as pie charts, while trends over time could always be displayed as line graphs. This helps readers immediately recognize the type of data they are looking at.

Building Non-Overlapping Measures and DimensionsTo ensure clarity and prevent confusion, it is essential that the measures and dimensions presented in a report are non-overlapping. Just as in architecture where different materials serve distinct structural purposes, each measure and dimension in a report should provide unique insights, without redundancy or overlap. This approach ensures that readers receive a clear, structured view of the data without misinterpretation.

Non-Overlapping MeasuresMeasures should be defined carefully to avoid reporting the same information in multiple formats. Redundant or overlapping measures, such as reporting both “Net Income” and “Net Earnings” without distinction, can confuse readers and obscure the actual insights the report aims to provide.
Example: If you present both “Operating Profit” and “EBITDA” in your report, clearly differentiate their definitions and roles. Otherwise, choose the most relevant measure for the context. Avoid presenting two metrics that convey essentially the same information unless a comparison is needed.
Non-Overlapping DimensionsDimensions such as time, geography, and product categories should also be non-overlapping in their presentation. This ensures that readers can focus on specific insights without needing to cross-reference data from different sections unnecessarily.
Example: A section on “Sales by Region” should not overlap with a section on “Sales by Product Line” unless it is necessary to show how the two interact. Ensure that each dimension has a distinct role in the analysis, preventing any blurred lines between the two.
MECE (Mutually Exclusive and Collectively Exhaustive) PrincipleThe MECE principle is a key guideline in structuring non-overlapping measures and dimensions. It ensures that all relevant categories are covered without overlap, and that together, they represent the full scope of the data. Each measure or dimension should fall into a mutually exclusive bucket, meaning no two categories overlap, while collectively, all possible scenarios should be addressed.
Example: When reporting product categories, ensure each category (e.g., electronics, apparel, furniture) is distinct and that all products are covered within one of these categories. If some items don’t fit into a predefined category, create an “Other” or “Miscellaneous” category to cover these exceptions, ensuring that the classification is collectively exhaustive.
Including a “Remainder” CategoryWhen it’s difficult to categorize every element in a report, add a catch-all “Other” or “Miscellaneous” category. This ensures that any outliers or unclassifiable data points are accounted for, preventing gaps in the analysis.
Example: In a report on sales by product category, if 90% of the data falls under predefined categories like electronics or apparel but 10% does not, place that 10% in an “Other” category. This ensures completeness without forcing data into ill-fitting categories.

Building HierarchiesJust as buildings are designed with structural hierarchies, from the foundation to the roof, reports should follow a logical hierarchy that organizes information from the most general to the most specific. A well-designed hierarchy guides the reader naturally through the report, making it easier to digest complex information.

Logical Hierarchy in Report StructureReports should be organized in a top-down manner, starting with high-level overviews and then drilling down into more detailed information. This mirrors the natural flow of human understanding, where readers first seek the big picture before diving into specifics.
Example: Begin with a summary of total sales, then proceed to break down sales by region, product, or time. Each level of the hierarchy should build upon the previous one, providing more granular insights as the reader progresses through the report.
Parent-Child RelationshipsEach section of the report should have a clear parent-child relationship, where more detailed insights are nested under broader categories. For example, a parent category such as “Revenue” may have child sections like “Revenue by Product Line” and “Revenue by Region.” This clear demarcation allows the reader to navigate smoothly through the report’s structure without confusion.
Example: If you begin with “Revenue by Region,” the next section could be “Revenue by City” under each region, progressively narrowing down the scope of data presented.
Visual Hierarchies in Tables and ChartsTables, charts, and other visual elements should reflect this hierarchy as well. For instance, a table that lists countries should sort them by continent (the parent category), with individual countries (child categories) organized underneath. In charts, this can be represented through color coding or grouping, showing hierarchical relationships visually.
Example: A bar chart showing sales per country could group countries by continent, with color distinctions representing each continent, making the hierarchy clear at a glance.
Hierarchical NarrativesThe narrative or text accompanying data should also follow a hierarchical structure. Begin with general insights or conclusions, then provide supporting data. This helps readers follow the logical flow of your analysis.
Example: Start with a high-level observation such as “Revenue grew by 15% this year.” Then, break it down into specific insights: “This growth was driven primarily by a 25% increase in North America, while Europe saw a more modest 5% gain.”

By structuring reports in a way that reflects these hierarchies, you help readers navigate complex data more efficiently, enhancing their understanding and enabling quicker decision-making.

Using Deductive Logical FlowIn structured reporting, using a deductive logical flow ensures that the report follows a clear, reasoned progression, much like an architect’s blueprint guiding the construction of a building. Deductive reasoning starts with broad, general statements or findings and gradually narrows down to specific details or supporting evidence. This top-down approach aligns with how most readers naturally process information, helping them grasp conclusions before diving into detailed data.

From General to SpecificA deductive flow begins with overarching conclusions or key insights, followed by supporting data and analysis. This method allows readers to understand the main takeaways upfront, without needing to sift through granular data to draw their own conclusions.
Example: Start by presenting an overall statement like “Total sales increased by 10% year-over-year.” Then, drill down into supporting details, such as “The electronics sector saw the largest growth at 15%, while apparel experienced a modest increase of 5%.”
Establishing a Logical SequenceEnsure that the report follows a logical, sequential order. For example, present insights at the company level first, then break them down by region or product. This builds a coherent narrative that helps readers understand how each level of detail relates to the bigger picture.
Example: First, summarize company-wide performance metrics. Next, detail performance by geographic region, followed by an analysis of individual product lines within those regions.
Supporting Conclusions with DataAfter introducing a key conclusion or insight, immediately follow it with data that supports the claim. This ensures that readers don’t need to search through different sections to find evidence that backs up the report’s conclusions.
Example: “North American sales grew by 20%, driven largely by the expansion of online sales channels. As seen in Figure 3, online sales accounted for 60% of the total revenue in this region, up from 45% last year.”
Maintaining Flow Between SectionsEach section of the report should naturally lead into the next, with transitions that link broad insights to increasingly specific details. This flow ensures that readers stay engaged and can follow the logical progression of the analysis.
Example: After presenting company-wide revenue figures, the next section could seamlessly introduce regional breakdowns with a transition like, “While the company saw an overall increase, the most significant growth was observed in North America, as detailed in the following section.”

Using Inductive Reasoning for Grouping ContentWhile deductive reasoning moves from general to specific, inductive reasoning works in the opposite direction — grouping individual data points or observations to build a general conclusion. In reporting, inductive reasoning is valuable when you want to synthesize patterns from detailed data to draw broader insights. Just as a structural engineer compiles data from individual stress tests to determine overall building safety, a report can use inductive reasoning to group content into meaningful conclusions.

Grouping Data to Form ConclusionsInductive reasoning starts with specific observations and patterns and uses these to form more general insights. By grouping related data together, reports can highlight broader trends or conclusions that might not be immediately obvious from individual data points.
Example: After analyzing quarterly revenue from individual regions, the report might identify that “Emerging markets, particularly in Southeast Asia and South America, saw a significant increase in revenue over the past year, suggesting that these regions are key drivers of growth.”
Using Categories and ClustersWhen working with large datasets, grouping similar data points into clusters or categories helps to organize the report’s findings. By looking at data in clusters, readers can more easily identify patterns and correlations, leading to more informed conclusions.
Example: Group customer satisfaction ratings by geographic region to reveal that “While North America shows consistently high ratings, customer satisfaction in Europe has dipped over the past two quarters, likely due to shipping delays.”
Identifying Trends from Grouped DataInductive reasoning allows reports to focus on identifying patterns that emerge from grouped data. This is particularly useful for trend analysis, where understanding the bigger picture depends on seeing how individual data points relate to each other.
Example: A report on product performance might analyze the sales of individual products and group them by category. “Within the electronics category, wireless devices have outperformed wired counterparts by 30% this year, indicating a clear consumer preference for wireless technology.”
Using Inductive Flow to Build Broader InsightsBy presenting detailed data first and then building up to broader insights, inductive reasoning helps guide readers through the report in a way that feels logical and intuitive. This approach is useful when the key conclusions depend on seeing detailed evidence or specific patterns.
Example: Start with sales data for individual products, followed by group-level trends, then conclude with an overall insight like “Products with sustainable packaging are consistently seeing higher sales growth, suggesting that sustainability is a key purchasing factor.”

Visually Structuring Notes, Tables, and VisualsThe way data is visually structured in tables, charts, and even notes is crucial for guiding the reader’s attention and ensuring clarity. Just as blueprints must be carefully organized to communicate design ideas effectively, reports need to use clear visual structures to make data easy to digest. Visually structuring content allows readers to quickly grasp important information without getting lost in the details.

Structuring Tables for ReadabilityTables are a key element in most reports, and their visual structure can greatly impact how quickly readers understand the data. Use consistent, logical formatting, such as aligned columns, row shading, and clearly labeled headers, to enhance readability.
Example: Ensure that tables have adequate spacing between rows, bold headers, and use shading or color-coding to highlight key data points (e.g., profit margins or percentage changes). This will help readers quickly identify the most important figures.
Using Visual Hierarchy in ChartsCharts and graphs should follow a visual hierarchy, guiding the reader from the most important information to the details. For example, emphasize the key data points by using larger fonts, brighter colors, or bold lines, while using subtler colors or lines for less critical information.
Example: In a line graph showing revenue over time, use a thick line in a prominent color for the current year’s data, while past years’ data is shown in thinner, muted lines for comparison. This makes it clear where the reader’s focus should be.
Organizing Notes and AnnotationsNotes, footnotes, and annotations in reports should be clearly structured and visually separated from the main content. Use a consistent style for all notes, placing them either at the bottom of tables and charts or as side notes, ensuring they do not disrupt the flow of the main report.
Example: If a chart includes an asterisk (*) to denote a specific condition (e.g., “Excludes one-time charges”), ensure that the explanation is clearly visible in the footnote section beneath the chart, using the same font size and color throughout the report.
Aligning Visuals with Report FlowEnsure that visuals, such as tables and charts, are placed in the appropriate sections of the report and are aligned with the narrative. This helps reinforce key points without interrupting the logical flow of the report. Each visual should serve a clear purpose and support the text around it.
Example: Place a bar chart illustrating quarterly revenue right after the text that discusses revenue trends. The chart should be labeled and formatted to reinforce the conclusion mentioned in the narrative.
Consistent Use of Formatting Across Visual ElementsJust as textual elements should follow consistent formatting, so too should visuals. Fonts, colors, line styles, and grid lines should remain consistent across all charts and tables, creating a unified look that enhances readability and professionalism.
Example: If one chart uses blue and green to differentiate between product lines, all subsequent charts in the report should use the same color scheme for those product lines. This consistency allows readers to make comparisons across different charts more easily.

With this chapter on Structure, we’ve reached the final letter in the IBCS acronym, marking the completion of this series. Much like the final touches on an architectural masterpiece, structure holds everything together, ensuring that each element of the report works in harmony. From maintaining consistent items and grammar to building clear hierarchies and using logical flows, a well-structured report guarantees clarity, comprehension, and actionable insights for decision-makers.

Throughout this series, we’ve explored how the International Business Communication Standards (IBCS) guide us toward reporting that is precise, clear, and effective. By applying these standards, you’re not just producing reports — you’re creating a foundation of trust, transparency, and efficiency in communication. We hope this journey has been insightful and that it has equipped you with the tools to enhance your reports, bringing them to the next level of professionalism and clarity.

We want to extend our thanks to IBCS for providing the guiding principles that made this series possible, and to you, our readers, for joining us on this journey.

What’s Next?Next week, we’ll continue our exploration into effective reporting with a tutorial on how to choose the right chart for specific purposes. Understanding which visual best communicates your data is critical to making your reports not just informative, but powerful tools for insight. Stay tuned for practical tips and examples to enhance your chart selection process!

Blueprint for SUCCESS: The Architecture of Structured Reports was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Blueprint for SUCCESS: The Architecture of Structured Reports

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionData visualization is a crucial aspect of data analysis, allowing us to understand and communicate complex data insights effectively. Among various visualization techniques, boxplots stand out for their ability to summarize data distributions. This guide will walk you through creating horizontal boxplots using base R and ggplot2, tailored for beginner R programmers.

Understanding BoxplotsComponents of a BoxplotA boxplot, also known as a whisker plot, displays the distribution of data based on a five-number summary: minimum, first quartile, median, third quartile, and maximum. It highlights the data’s central tendency and variability, making it easier to identify outliers.

When to Use BoxplotsBoxplots are particularly useful for comparing distributions across different groups. They are ideal when you want to visualize the spread and skewness of your data.

Horizontal Boxplots: An OverviewAdvantages of Horizontal BoxplotsHorizontal boxplots enhance readability, especially when dealing with categorical data labels that are lengthy. They also provide a clear visualization of distribution patterns across groups.

Use CasesHorizontal boxplots are commonly used in scenarios such as comparing test scores across different classes, analyzing sales data across regions, or visualizing the distribution of survey responses.

Setting Up R EnvironmentInstalling R and RStudioBefore creating boxplots, ensure that you have R and RStudio installed on your computer. You can download R from CRAN and RStudio from RStudio’s website.

Required PackagesTo create boxplots, you need to install the ggplot2 package for enhanced visualization capabilities. You can install it using:

install.packages("ggplot2") Creating Horizontal Boxplots in Base RBasic SyntaxIn base R, you can create a boxplot using the boxplot() function. To make it horizontal, set the horizontal parameter to TRUE.

Customizing BoxplotsBase R allows customization of boxplots through various parameters, such as col for color and main for the title.

Step-by-Step Guide: Base RLoading DataFor this example, we’ll use the built-in mtcars dataset. Load it using:

data(mtcars) Plotting Horizontal Boxplots boxplot( mpg ~ cyl, data = mtcars, horizontal = TRUE, main = "Horizontal Boxplot of MPG by Cylinder", col = "lightblue" ) Customizing AppearanceYou can further customize your plot by adjusting axis labels, adding a grid, or changing colors:

boxplot( mpg ~ cyl, data = mtcars, horizontal = TRUE, main = "Horizontal Boxplot of MPG by Cylinder", col = "lightblue", xlab = "Miles Per Gallon", ylab = "Number of Cylinders" ) Introduction to ggplot2Why Use ggplot2?ggplot2 offers a high-level approach to creating complex and aesthetically pleasing visualizations. It is part of the tidyverse, making it compatible with other data manipulation tools.

Basic Conceptsggplot2 uses a layered approach to build plots, where you start with a base layer and add elements like geoms, scales, and themes.

Creating Horizontal Boxplots with ggplot2Basic SyntaxTo create a boxplot in ggplot2, use geom_boxplot() and flip it horizontally using coord_flip().

Using coord_flip()``coord_flip() swaps the x and y axes, creating a horizontal boxplot.

Step-by-Step Guide: ggplot2Loading DataWe continue with the mtcars dataset.

Plotting Horizontal Boxplots library(ggplot2)ggplot(mtcars, aes(x = factor(cyl), y = mpg)) + geom\_boxplot(fill = "lightblue") + coord\_flip() + theme\_minimal() + labs( title = "Horizontal Boxplot of MPG by Cylinder", x = "Number of Cylinders", y = "Miles Per Gallon" ) Customizing AppearanceYou can enhance your plot by adding themes, colors, and labels:

ggplot(mtcars, aes(x = factor(cyl), y = mpg, fill = factor(cyl))) + geom\_boxplot() + coord\_flip() + theme\_minimal() + labs( title = "Horizontal Boxplot of MPG by Cylinder", x = "Number of Cylinders", y = "Miles Per Gallon", fill = "Cylinder") + theme\_minimal() Advanced Customizations in ggplot2Adding Colors and ThemesUse scale_fill_manual() for custom colors and explore theme() options for layout adjustments.

Faceting and GroupingFaceting allows you to create multiple plots based on a factor, using facet_wrap() or facet_grid().

ggplot(mtcars, aes(x = factor(cyl), y = mpg, fill = factor(gear))) + geom\_boxplot() + coord\_flip() + facet\_wrap(~ gear, scales = "free") + theme\_minimal() Comparing Base R and ggplot2Pros and Cons Base R: Simpler and requires fewer dependencies, but less flexible for complex plots. * ggplot2*: More powerful for complex visualizations, but has a steeper learning curve.

Performance ConsiderationsFor larger datasets, ggplot2 may be slower due to its complexity, but it provides more options for customization and aesthetics.

Common Errors and TroubleshootingDebugging Tips* Ensure all required packages are installed and loaded. * Check for typos in function names and parameters. * Verify data types and structures are compatible with plotting functions.

FAQs1. What is the purpose of a horizontal boxplot? * Horizontal boxplots improve readability and are useful when dealing with long category labels. 2. How do I flip a boxplot in ggplot2? * Use coord_flip() to switch the axes and create a horizontal boxplot. 3. Can I customize the colors of my boxplot in R? * Yes, both base R and ggplot2 allow color customization using parameters like col and fill. 4. What are common errors when creating boxplots in R? * Common errors include mismatched data types and missing package installations. 5. How do I compare multiple groups using boxplots? * Use the fill aesthetic in ggplot2 or multiple boxplot() calls in base R to compare groups.

Practical ExamplesExample 1: Analyzing a Simple DatasetCreate a horizontal boxplot to compare student test scores across different classes.

Example 2: Complex Data VisualizationUse ggplot2 to visualize sales data distributions across regions, incorporating facets and themes for clarity.

Visual EnhancementsAdding AnnotationsEnhance your plots by adding text annotations with annotate() in ggplot2.

Using Custom ThemesExperiment with ggplot2’s built-in themes or create your own using theme().

ConclusionCreating horizontal boxplots in R is a valuable skill for visualizing data distributions. Whether you choose base R for simplicity or ggplot2 for its advanced capabilities, mastering these techniques will enhance your data analysis toolkit. Experiment with different datasets and customization options to discover the full potential of boxplots.

Encourage EngagementWe’d love to hear your feedback! Share your experiences with horizontal boxplots in R on social media and tag us. If you have questions or tips, leave a comment below.

References1. Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. 2. R Documentation. (n.d.). Boxplot. Retrieved from R Documentation. 3. ggplot2 Documentation. (n.d.). Retrieved from ggplot2.

Some Extra ReadingsHere are some other great resources:

“R for Data Science” by Hadley Wickham & Garrett Grolemund
- This book is a great resource for beginners and provides an introduction to data science using R, including data visualization with ggplot2.
“ggplot2: Elegant Graphics for Data Analysis” by Hadley Wickham
- A comprehensive guide focused specifically on ggplot2, teaching you how to create a wide range of visualizations, including boxplots.
“The R Graphics Cookbook” by Winston Chang
- This cookbook offers practical recipes for visualizing data in R, covering both base R graphics and ggplot2.
R Documentation and Cheat Sheets
- The official R documentation and ggplot2 cheat sheets are invaluable for quick reference and deeper exploration of functions and customization options. Some Tutorials
“Visualize This: The FlowingData Guide to Design, Visualization, and Statistics” by Nathan Yau
- While not R-specific, this book provides insights into the principles of data visualization, which can enhance your overall understanding of creating effective visualizations.
R-bloggers
- A community blog site that aggregates content related to R programming, including tutorials and examples on creating boxplots and other visualizations.

These resources offer a mix of theoretical knowledge and practical application, helping you build a solid foundation in R programming and data visualization.

Happy Coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Create Horizontal Boxplots in Base R and ggplot2

View Details

[This article was first published on R | r4stats.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I have just finished updating my reviews of graphical user interfaces for the R language. These include BlueSky Statistics, jamovi, JASP, R AnalyticFlow, R Commander, R-Instat, Rattle, and RKward. The permanent link to the article that summarizes it all is https://r4stats.com/articles/software-reviews/r-gui-comparison/.
I list the highlights below as this post to reach all the blog aggregators. If you have suggestions for improving any of the reviews, please let me know at muenchen.bob@gmail.com.

With so many detailed reviews of Graphical User Interfaces (GUIs) for R available, which should you choose? It’s not too difficult to rank them based on the number of features they offer, so I’ll start there. Then, I’ll follow with a brief overview of each.

I’m basing the counts on the number of dialog boxes in each category of the following categories:

Ease of Use
General Usability
Graphics
Analytics
Reproducibility

This data is trickier to collect than you might think. Some software has fewer menu choices, depending on more detailed dialog boxes instead. Studying every menu and dialog box is very time-consuming, but that is what I’ve tried to do to keep this comparison trustworthy. Each development team has had a chance to look the data over and correct errors.

Perhaps the biggest flaw in this methodology is that every feature adds only one point to each GUI’s total score. I encourage you to download the full dataset to consider which features are most important to you. If you decide to make your own graphs with a different weighting system, I’d love to hear from you in the comments below.

Ease of UseFor ease of use, I’ve defined it primarily by how well each GUI meets its primary goal: avoiding code. They get one point for each of the following abilities, which include being able to install, start, and use the GUI to its maximum effect, including publication-quality output, without knowing anything about the R language itself. Figure one shows the result. R Commander is abbreviated Rcmdr, and R AnalyticFlow is abbreviated RAF. The commercial BlueSky Pro comes out on top by a slim margin, followed closely by JASP and RKWard. None of the GUIs achieved the highest possible score of 14, so there is room for improvement.

Installs without the use of R
Starts without the use of R
Remembers recent files
Hides R code by default
Use its full capability without using R
Data editor included
Pub-quality tables w/out R code steps
Simple menus that grow as needed
Table of Contents to ease navigation
Variable labels ease identification in the output
Easy to move blocks of output
Ease reading columns by freezing headers of long tables
Accepts data pasted from the clipboard
Easy to move header row of pasted data into the variable name field

Figure 1. The number of ease of use features offered by each R GUI.General UsabilityThis category is dominated by data-wrangling capabilities, where data scientists and statisticians spend most of their time. It also includes various types of data input and output. We see in Figure 2 that both BlueSky versions and R-Instat come out on top not just due to their excellent selection of data-wrangling features but also for their use of the rio package for importing and exporting files. The rio package combines the import/export capabilities of many other packages, and it is easy to use. I expect the other GUIs will eventually adopt it, raising their scores by around 20 points.

Operating systems (how many)
Import data file types (how many)
Import from databases (how many)
Export data file types (how many)
Languages displayable in UI (how many, besides English)
Easy to repeat any step by groups (split-file)
Multiple data files open at once
Multiple output windows
Multiple code windows
Variable metadata view
Variable types (how many)
Variable search/filter in dialogs
Variable sort by name
Variable sort by type
Variable move manually
Model Builder (how many effect types)
Magnify GUI for teaching
R code editor
Comment/uncomment blocks of code
Package management (comes with R and all packages)
Output: word processing features
Output: R Markdown
Output: LaTeX
Data wrangling (how many)
Transform across many variables at once (e.g., row mean)
Transform down many variables at once (e.g., log, sqrt)
Assign factor labels across many variables at once
Project saves/loads data, dialogs, and notes in one file

Figure 2. The number of general usability features in each R GUI.GraphicsThis category consists mainly of the number of graphics each software offers. However, the other items can be very important to completing your work. They should add more than one point to the graphics score, but I scored them one point since some will view them as very important while others might not need them at all. Be sure to see the full reviews or download the Excel file if those features are important to you. Figure 3 shows the total graphics score for each GUI. R-Instat has a solid lead in this category. In fact, this underestimates R-Instat’s ability if you include its options to layer any “geom” on top of another graph. However, that requires knowing the geoms and how to use them. That’s knowledge of R code, of course.

When studying these graphs, it’s important to consider the difference between the relative and absolute performance. For example, relatively speaking, R Commander is not doing well here, but it does offer over 25 types of plots! That absolute figure might be fine for your needs.

Continued…

The post R GUI Reviews Updated first appeared on r4stats.com.To leave a comment for the author, please follow the link and comment on their blog: R | r4stats.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R GUI Reviews Updated

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Mastering R Programming for Data Science: Tips and Tricks appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Mastering R Programming for Data Science: Tips and Tricks, As data science continues to evolve, the demand for R programming skills has become a necessary tool for data professionals to extract insights from data rapidly.

R is a powerful language that offers a wide range of tasks, including statistics, visualization, and machine learning.

Mastering R Programming for Data ScienceIn this article, we will explore the top tips and tricks for mastering R programming in data science.

Tip 1: Leverage the Tidyverse Package

The Tidyverse package is a set of R packages that makes it easy to perform data manipulation and visualization tasks.

Familiarize yourself with packages such as dplyr, ggplot2, and tidyr to promote the correct way of working with data.

Tip 2: Adopt Functional Programming Concepts

Functional programming concepts are essential for writing sustainable and reusable code.

Understand topics such as anonymous functions, higher-order functions, and function composition to write code that is easy to read and maintain.

Machine Learning Archives » Data Science Tutorials

Tip 3: Master the Art of Data Visualization

Data visualization is a crucial step in data exploration and representation.

Master the art of building insightful visualizations using packages such as ggplot2 and plotly.

Tip 4: Apply Statistical Models

R is famous for its importance in the statistics field.

Apply sophisticated statistical methods with tools like stats, lme4, and caret to develop advanced-level skills.

Tip 5: Harness the Power of Machine Learning

R processing has libraries dedicated to machine learning packages like caret, randomForest, and xgboost.

Examine supervised and unsupervised learning algorithms, such as regression, classification, clustering, and dimension reduction.

Additional Tips for Good R Programming

Use vectorization to develop streamlined and efficient code.
Optimize memory usage by avoiding overcopying of objects and employing wise data structures.
Document your code to ensure smooth readability and maintainability.
Explore R Markdown for reproducible research and reports generation.
Engage with the R community by attending forums, meetups, and webinars.

Learning Resources for R Programming

Online courses: Register at websites like Coursera, DataCamp, or Udemy to gain from their structured R programming courses.
Books: “R for Data Science” by Hadley Wickham and Garrett Grolemund or “Machine Learning with R” by Brett Lantz could be a great option.
Documentation: The official R documentation is an excellent resource for understanding R programming from basic to advanced levels.

ConclusionMastering R programming is essential for anyone who wants to develop a career in data science.

By following these tips and tricks, you’ll be empowered to process data, make attractive charts and graphs, build solid models, and apply learning algorithms without any trouble.

How to Measure Execution Time in R
geom_point requires the following missing aesthetics:
How to read or export large datasets in R
Export Data Frames into Multiple Excel Sheets in R
How to Change Background Color in ggplot2
How to split vector and data frame in R
How to Split data into train and test in R
How to Read rda file in R (with Example)

The post Mastering R Programming for Data Science: Tips and Tricks appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering R Programming for Data Science: Tips and Tricks

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Introduction to Variables in CIn the world of C programming, variables play a crucial role. They are human-readable names that refer to specific memory locations where data is stored. Understanding how to declare and use variables effectively is foundational for any programmer. This guide will walk you through the basics of variables in C, helping you become proficient in managing and using data within your programs.

Understanding Memory and VariablesHow Variables Map to MemoryEvery variable in C is a storage location with a specific data type, which determines the size and layout of the variable’s memory; the range of values that can be stored; and the set of operations that can be applied to the variable.

Memory Allocation for VariablesWhen you declare a variable, the compiler allocates memory for it. The amount of memory allocated depends on the data type of the variable. Understanding this concept is essential for efficient memory management and optimization in C programming.

Types of Variables in CPrimitive Data TypesC supports several primitive data types, including:

int: Used for integers.
char: Used for characters.
float: Used for floating-point numbers.
double: Used for double-precision floating-point numbers.

User-Defined Data TypesC also allows the creation of user-defined data types, such as:

struct: A structure is a user-defined data type that groups different data types.
union: Similar to a structure, but members share the same memory location.
enum: An enumeration is a data type consisting of a set of named values.

Declaring Variables in CSyntax of Variable DeclarationTo declare a variable in C, specify the data type followed by the variable name. For example:

int age;char initial;float salary; Examples of Variable DeclarationsConsider the following declarations:

int score = 90;char grade = 'A';double pi = 3.14159; Variable Naming ConventionsRules for Naming VariablesNaming variables in C must follow these rules: – Must begin with a letter or an underscore (_). – Can contain letters, digits, and underscores. – Case-sensitive.

Best Practices for Naming* Use meaningful names (e.g., totalCost instead of x). * Avoid using reserved keywords. * Maintain consistency in naming conventions (e.g., camelCase or snake_case).

Scope and Lifetime of VariablesLocal vs Global Variables Local Variables: Declared inside a function or block and accessible only within it. * Global Variables*: Declared outside all functions and accessible throughout the program.

Static and Dynamic Variables Static Variables: Retain their value between function calls. * Dynamic Variables*: Allocated and deallocated during runtime using pointers.

Initializing VariablesDefault InitializationVariables declared without an initial value have undefined content. Always initialize variables to avoid undefined behavior.

Explicit InitializationAssign a value at the time of declaration:

int count = 0;float temperature = 36.5; Using Variables in ExpressionsArithmetic OperationsVariables can be used in arithmetic operations:

int sum = a + b;float product = x * y; Logical OperationsVariables also participate in logical operations:

if (isAvailable && isAffordable) { printf("Purchase possible!");} Common Errors with VariablesUninitialized VariablesUsing a variable before initializing it can lead to unpredictable results.

Type Mismatch ErrorsEnsuring variables are used with compatible types prevents type mismatch errors.

Advanced Variable ConceptsPointers and VariablesPointers store memory addresses of variables. They are crucial for dynamic memory management.

Arrays and VariablesArrays are collections of variables of the same type. They allow structured data storage and manipulation.

Debugging Variable IssuesTools for DebuggingUse debugging tools like GDB to trace variable values and program execution.

Common Debugging Techniques* Print statements to monitor variable values. * Breakpoints to pause execution and inspect variables.

Optimizing Variable UsageMemory Management TipsEfficient memory usage reduces program overhead. Use appropriate data types and free unused memory.

Performance ConsiderationsOptimize variable usage by minimizing redundant variables and operations.

Practical ExamplesSimple Programs Using Variables ```

include int main() { int num1 = 5, num2 = 10; int sum = num1 + num2; printf("Sum: %d\n", sum); return 0;}

``` Real-world ApplicationsVariables are used to store user inputs, perform calculations, and manage state in complex applications.

Here are a couple of examples:

```

include // Global variableint global_count = 0;void increment_count() { // Static variable static int call_count = 0; call_count++; global_count++; printf("Function called %d times\n", call_count); printf("Global count: %d\n", global_count);}int main() { // Local variables int local_var = 5; float pi = 3.14159; char grade = 'A'; printf("Local variable: %d\n", local_var); printf("Pi: %.2f\n", pi); printf("Grade: %c\n", grade); increment_count(); increment_count(); increment_count(); int num1 = 5, num2 = 10; int sum = num1 + num2; printf("Sum: %d\n", sum); return 0;}

``` Output:

Local variable: 5Pi: 3.14Grade: AFunction called 1 timesGlobal count: 1 Function called 2 timesGlobal count: 2 Function called 3 timesGlobal count: 3 Sum: 15 And another example from Chapter 5 Adding Variables To Your Programs from the book “C Programming for the Absolute Beginner”, Third Edition, Perry and Miller:

```

include // Code snippet from Chapter 5 Adding Variables To Your Programs// C Programming Absolute Beginner's Guide, Third Edition, Perry and Millermain() { char first_initial, middle_initial; int number_of_pencils; int number_of_notebooks; float pencils = 0.23; float notebooks = 2.89; float lunchbox = 4.99; first_initial = 'J'; middle_initial = 'R'; number_of_pencils = 7; number_of_notebooks = 4; printf("%c%c needs %d pencils, %d notebooks, and 1 lunchbox\n", first_initial, middle_initial, number_of_pencils, number_of_notebooks); printf("The total cost is $%.2f\n", number_of_pencils * pencils + number_of_notebooks * notebooks + lunchbox); return 0;}JR needs 7 pencils, 4 notebooks, and 1 lunchboxThe total cost is $18.16

`` FAQs about Variables in C1. **What are the rules for variable names in C?** * Variable names must start with a letter or underscore, are case-sensitive, and cannot be a reserved keyword. 2. **How do I initialize a variable in C?** * Use the assignment operator during declaration, e.g.,int count = 0;`. 3. What is the difference between local and global variables? * Local variables are limited to the function/block, while global variables are accessible throughout the program. 4. How do I avoid uninitialized variable errors? * Always assign an initial value when declaring a variable. 5. Can I change the data type of a variable in C? * No, once declared, a variable’s data type cannot be changed.

ConclusionVariables are the building blocks of C programming, enabling you to store and manipulate data efficiently. By understanding their types, scope, and lifecycle, you can write more robust and maintainable code. Practice writing programs using variables to solidify your understanding and enhance your programming skills.

Your Turn!We hope this guide has been helpful in understanding variables in C. If you have any questions or feedback, please share them in the comments or on social media!

References1. Kernighan, B. W., & Ritchie, D. M. (1988). The C Programming Language. Prentice Hall. 2. Harbison, S. P., & Steele, G. L. (2002). C: A Reference Manual. Prentice Hall. 3. Prata, S. (2013). C Primer Plus. Addison-Wesley Professional.

Happy Coding!

An Example To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Adding Variables to Your C Code: A Beginner’s Guide

View Details

[This article was first published on rstats on Irregularly Scheduled Programming, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In this post I explore some differences between R, python, julia, and APL interms of mutability, and try to make something that probably shouldn’t exist.

I watched this code_report video which describesa leetcode problem;

You are given an integer array nums, an integer k, and an integer multiplier.

You need to perform k operations on nums. In each operation:

Find the minimum value x in nums. If there are multiple occurrences of the minimum value, select the one that appears first.

Replace the selected minimum value x with x * multiplier.

Return an integer array denoting the final state of nums after performing all k operations.

Conor’s python solution in the video was

def getFinalState(nums, k, m): for \_ in range(k): i = nums.index(min(nums)) nums[i] *= m return numsx = [2, 1, 3, 5, 6]k = 5mult = 2getFinalState(x, k, mult)## [8, 4, 6, 5, 6] and, as always, I wanted to see how I’d do that in R. I came up with this

getFinalState = function(nums, k, m) { for (i in 1:k) { m <- which.min(nums)[1] nums[m] <- mult * nums[m] } nums}x <- c(2, 1, 3, 5, 6)k <- 5mult <- 2getFinalState(x, k, mult)## [1] 8 4 6 5 6 It’s worth noting that I can’t use a map in this function because iterationsare dependent; the minimum value at any iteration depends on the previousvalues.

I also had a chance to discuss this solution with some APL’ers at a meetup anda J solution was presented, but I don’t think I wrote it down.

My solution is nearly word-for-word the same as the python solution with acouple of notable exceptions arising from the difference between the twolanguages:

First, R has which.min() as a built-in rather than needing to query the indexof the minimum value (and two references to nums). Also, R has no compoundassignment like x *= 2 which modifies in-place - the closest thing I can thinkof is the %<>% operator in {magrittr} (not re-exported in {dplyr} because thisbehaviour is considered bad practice in R, despite not really being “in-place”)

library(magrittr)m <- data.frame(x = 1:6, y = letters[1:6])m## x y## 1 1 a## 2 2 b## 3 3 c## 4 4 d## 5 5 e## 6 6 fm %<>% head(2)m## x y## 1 1 a## 2 2 b although I can certainly see the case for it - this operator avoids repeatingthe variable being used and assigned, because the alternative using thetraditional pipe is

m <- data.frame(x = 1:6, y = letters[1:6])m## x y## 1 1 a## 2 2 b## 3 3 c## 4 4 d## 5 5 e## 6 6 fm <- m %>% head(2)m## x y## 1 1 a## 2 2 b One could argue that writing out even a longer variable name twice still makesit clear that shadowing is taking place; the value is being overwritten witha new value, but it does feel a little frustrating to have to type it out twice

important\_variable <- important\_variable * 2 Back to my R solution, the indexing at a specific set of values got me thinkingthat it would be clean if we could pass a function to [ so that we couldwrite

nums[which.min] <- value (maybe not so much for this example where m is used twice, but it piqued myinterest)

Let’s say I want to set all the even values of a vector to some other value.That’s easy enough to do

x[x %% 2 == 0] <- 0 but I don’t love that it requires two references to x, which may (should?) bea much longer name

important\_variable[important\_variable %% 2 == 0] <- 0 I want something like x[f] <- y to set the values of x where f(x) isTRUE to y. This seemed like it might be possible, maybe with a functionmethod to [<-, but [<- dispatches on the class of x, not what’s inside[, so no dice. In theory (which will never happen) the built-in [<- couldhave some branch logic for dealing with a function passed as the indices to bemodified, but I’m not about to go rebuilding R from source myself just to playwith that idea.

Nonetheless, if I define some functions that do accomplish this

is\_even <- function(z) z %% 2 == 0set\_if <- function(x, f, value) { x[f(x)] <- value x} then I can try this out on a vector

a <- 1:10a## [1] 1 2 3 4 5 6 7 8 9 10set\_if(a, is\_even, 0)## [1] 1 0 3 0 5 0 7 0 9 0a # unchanged## [1] 1 2 3 4 5 6 7 8 9 10 It works, but I’m back to having to write a <- do_stuff(a) because a isn’tactually modified by this function.

Ideally, my function would operate the same as this does

a <- 1:10a[is\_even(a)] <- 0a## [1] 1 0 3 0 5 0 7 0 9 0 which does modify a in-place; R is not entirely pure, and does occasionallyallow what looks like direct mutation, though under the hood, it’s not - a newobject is actually created

```

not using a range e.g. 1:n because that's internally # a "compact" representationa <- c(2, 3, 4).Internal(inspect(a))## @63a4d9b05be8 14 REALSXP g0c3 [REF(2)] (len=3, tl=0) 2,3,4a[2] <- 9.Internal(inspect(a))## @63a4d9b0fbf8 14 REALSXP g0c3 [REF(1)] (len=3, tl=0) 2,9,4

``` Note that the memory address has changed.

If I was working with a language which did support (enable?) modify-in-placethen that might look like

def is\_even(x): return x % 2 == 0def set\_if(x, f, value): for i in range(len(x)): if f(x[i]): x[i] = valuea = list(range(10))a## [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]set\_if(a, is\_even, 0)a## [0, 1, 0, 3, 0, 5, 0, 7, 0, 9] Now, that’s not always a great thing. In such a language with mutable structures(e.g. lists) we can do maddening things like this

x = [3, 4, 5]y = xy is x## Truey[1] = 9x # still 'bound' to y## [3, 9, 5] Here, is means “are these two things identical in the sense of referring tothe same block of memory”, noting that literals (e.g. single numbers) arereferenced that way, but tuples aren’t

abc = (11, 99)xyz = (11, 99)abc is xyz## Falseabc == xyz## True The big question is can I hack together some solution that does work in-placein R? Yeah, with some ill-advised calls

set\_if <- function(x, f, value) { # can't use <<- because the value passed in as the x argument isn't # necessarily named 'x' in the parent scope .x <- x .x[f(.x)] <- value e <- parent.env(environment()) assign(deparse(substitute(x)), .x, pos = e) invisible(.x)}a <- 1:10a## [1] 1 2 3 4 5 6 7 8 9 10set\_if(a, is\_even, 0)a## [1] 1 0 3 0 5 0 7 0 9 0 As I note in the comment there, I can’t use the super-assignment arrow <<-inside this function because I don’t know the name of the variable I’m updating;it needs to be deparsed from the incoming argument.

This means that it works regardless of the name of the variable being modified

b <- 10:20b## [1] 10 11 12 13 14 15 16 17 18 19 20set\_if(b, is\_even, 0)b## [1] 0 11 0 13 0 15 0 17 0 19 0 I tried to think of some other languages which might support this sort of in-placeset_if(x, f, value) modification and (Dyalog) APL was worth a thought.

⍝ create a vector from 1 to 10 x←⍳10 x1 2 3 4 5 6 7 8 9 10 ⍝ the function {0=2|⍵} calculates a boolean vector with ⍝ 1 where the value is even {0=2|⍵} x0 1 0 1 0 1 0 1 0 1 ⍝ the `@` operator takes a value (or function) on the left and ⍝ a function (or boolean values) on the right and applies it to the ⍝ other argument on the right 0@{0=2|⍵} x 1 0 3 0 5 0 7 0 9 0 ⍝ alternatively a point-free function defined as the negation (`~`) of a ⍝ binding (`∘`) of the value 2 to modulo (`|`); the negation is needed ⍝ otherwise this returns the result of the modulo, not where it is 0 0@(~2∘|)⍳101 0 3 0 5 0 7 0 9 0 ⍝ x is, however, unchanged as APL is typically immutable x1 2 3 4 5 6 7 8 9 10 So there’s no way to do the in-place modification. it is nice, though, that0@(~2∘|)x only refers to x once.

Julia makes a nice distinction between functions which mutate arguments andthose which don’t; (by convention) the former are named ending with anexclamation mark, e.g.

vec = collect(1:5)## 5-element Vector{Int64}:## 1## 2## 3## 4## 5# non-mutatingreverse(vec)## 5-element Vector{Int64}:## 5## 4## 3## 2## 1vec## 5-element Vector{Int64}:## 1## 2## 3## 4## 5# mutatingreverse!(vec)## 5-element Vector{Int64}:## 5## 4## 3## 2## 1vec## 5-element Vector{Int64}:## 5## 4## 3## 2## 1 In julia, the iseven() function is already built-in, but vectorisation isexplicit via a broadcast operator . and the setting of even values to 0looks like

x = collect(1:10);x[iseven.(x)] .= 0;x## 10-element Vector{Int64}:## 1## 0## 3## 0## 5## 0## 7## 0## 9## 0 which looks very much like the R version with some dots where scalar functionsare vectorised. If I don’t use the last . to perform vectorised assignment,the error tells me that the failure involved the setindex! function which doessound like what I want, but this doesn’t work

setindex!(x, 0, iseven.(x)) because it’s trying to assign the value 0 multiple times and I only provided oneof them. Instead,

x = collect(1:10);setindex!(x, zeros(Int8, 5), iseven.(x));x## 10-element Vector{Int64}:## 1## 0## 3## 0## 5## 0## 7## 0## 9## 0 does work, but I had to manually count how many 0 entries this requires, so the[ approach seems cleaner. Either way, I’ve had to explicitly calculateiseven(x) and pass that result somewhere.

Since Julia allows users to extend methods, I could do that modification myself!

import Base.setindex! function setindex!(A::Vector{Int64}, v::Int64, f::Function) A[f.(A)] .= vend## setindex! (generic function with 240 methods)x = collect(1:10);setindex!(x, 0, iseven);x## 10-element Vector{Int64}:## 1## 0## 3## 0## 5## 0## 7## 0## 9## 0 which I could just as easily call set_if!

set\_if! = setindex!;x = collect(1:10);set\_if!(x, 0, iseven);x## 10-element Vector{Int64}:## 1## 0## 3## 0## 5## 0## 7## 0## 9## 0 Nice! I do wonder if I can “hack” (ahem, extend) Julia’s [ to get my prizedx[f] = 0 solution but I doubt it’s worth it when the above does the rightthing.

I don’t imagine I’ll package up my set_if() anywhere, and I should probablyeven avoid using it myself, but it’s been an interesting journey thinking aboutthis stuff. Maybe there’s a better way to do it? Maybe there’s a language whichbetter supports something like that? If you know, or you have comments orsuggestions, I can be found onMastodon or use the comment section below.

devtools::session_info() ```

─ Session info ───────────────────────────────────────────────────────────────## setting value## version R version 4.3.3 (2024-02-29)## os Pop!_OS 22.04 LTS## system x86_64, linux-gnu## ui X11## language (EN)## collate en_AU.UTF-8## ctype en_AU.UTF-8## tz Australia/Adelaide## date 2024-09-25## pandoc 3.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)## ## ─ Packages ───────────────────────────────────────────────────────────────────## package * version date (UTC) lib source## blogdown 1.19 2024-02-01 [1] CRAN (R 4.3.3)## bookdown 0.36 2023-10-16 [1] CRAN (R 4.3.2)## bslib 0.8.0 2024-07-29 [1] CRAN (R 4.3.3)## cachem 1.1.0 2024-05-16 [1] CRAN (R 4.3.3)## callr 3.7.3 2022-11-02 [3] CRAN (R 4.2.2)## cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.3)## crayon 1.5.2 2022-09-29 [3] CRAN (R 4.2.1)## devtools 2.4.5 2022-10-11 [1] CRAN (R 4.3.2)## digest 0.6.37 2024-08-19 [1] CRAN (R 4.3.3)## ellipsis 0.3.2 2021-04-29 [3] CRAN (R 4.1.1)## evaluate 0.24.0 2024-06-10 [1] CRAN (R 4.3.3)## fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.3.3)## fs 1.6.4 2024-04-25 [1] CRAN (R 4.3.3)## glue 1.7.0 2024-01-09 [1] CRAN (R 4.3.3)## htmltools 0.5.8.1 2024-04-04 [1] CRAN (R 4.3.3)## htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.3.2)## httpuv 1.6.12 2023-10-23 [1] CRAN (R 4.3.2)## icecream 0.2.1 2023-09-27 [1] CRAN (R 4.3.2)## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.3.3)## jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.3.3)## JuliaCall 0.17.5 2022-09-08 [1] CRAN (R 4.3.3)## knitr 1.48 2024-07-07 [1] CRAN (R 4.3.3)## later 1.3.1 2023-05-02 [1] CRAN (R 4.3.2)## lattice 0.22-5 2023-10-24 [4] CRAN (R 4.3.1)## lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.3.3)## magrittr * 2.0.3 2022-03-30 [1] CRAN (R 4.3.3)## Matrix 1.6-5 2024-01-11 [4] CRAN (R 4.3.3)## memoise 2.0.1 2021-11-26 [1] CRAN (R 4.3.3)## mime 0.12 2021-09-28 [1] CRAN (R 4.3.3)## miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.3.2)## pkgbuild 1.4.2 2023-06-26 [1] CRAN (R 4.3.2)## pkgload 1.3.3 2023-09-22 [1] CRAN (R 4.3.2)## png 0.1-8 2022-11-29 [1] CRAN (R 4.3.2)## prettyunits 1.2.0 2023-09-24 [3] CRAN (R 4.3.1)## processx 3.8.3 2023-12-10 [3] CRAN (R 4.3.2)## profvis 0.3.8 2023-05-02 [1] CRAN (R 4.3.2)## promises 1.2.1 2023-08-10 [1] CRAN (R 4.3.2)## ps 1.7.6 2024-01-18 [3] CRAN (R 4.3.2)## purrr 1.0.2 2023-08-10 [3] CRAN (R 4.3.1)## R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.3)## Rcpp 1.0.11 2023-07-06 [1] CRAN (R 4.3.2)## remotes 2.4.2.1 2023-07-18 [1] CRAN (R 4.3.2)## reticulate 1.34.0 2023-10-12 [1] CRAN (R 4.3.2)## rlang 1.1.4 2024-06-04 [1] CRAN (R 4.3.3)## rmarkdown 2.28 2024-08-17 [1] CRAN (R 4.3.3)## rstudioapi 0.15.0 2023-07-07 [3] CRAN (R 4.3.1)## sass 0.4.9 2024-03-15 [1] CRAN (R 4.3.3)## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.2)## shiny 1.7.5.1 2023-10-14 [1] CRAN (R 4.3.2)## stringi 1.8.4 2024-05-06 [1] CRAN (R 4.3.3)## stringr 1.5.1 2023-11-14 [1] CRAN (R 4.3.3)## urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.3.2)## usethis 3.0.0 2024-07-29 [1] CRAN (R 4.3.3)## vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.3.3)## xfun 0.47 2024-08-17 [1] CRAN (R 4.3.3)## xtable 1.8-4 2019-04-21 [1] CRAN (R 4.3.2)## yaml 2.3.10 2024-07-26 [1] CRAN (R 4.3.3)## ## [1] /home/jono/R/x86_64-pc-linux-gnu-library/4.3## [2] /usr/local/lib/R/site-library## [3] /usr/lib/R/site-library## [4] /usr/lib/R/library## ## ─ Python configuration ───────────────────────────────────────────────────────## python: /home/jono/.virtualenvs/r-reticulate/bin/python## libpython: /usr/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.so## pythonhome: /home/jono/.virtualenvs/r-reticulate:/home/jono/.virtualenvs/r-reticulate## version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]## numpy: [NOT FOUND]## ## NOTE: Python version was forced by VIRTUAL_ENV## ## ──────────────────────────────────────────────────────────────────────────────

``` To leave a comment for the author, please follow the link and comment on their blog: rstats on Irregularly Scheduled Programming.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: In-Place Modifications

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This week, I released Techtonique web app, a tool designed to help you make informed, data-driven decisions using Mathematics, Statistics, Machine Learning, and Data Visualization. As of September 2024, the tool is in its beta phase (subject to crashes) and will remain completely free to use until December 24, 2024. After registering, you will receive an email. CHECK THE SPAMS.A few selected users will be contacted directly for feedback, but you can also send yours.

The tool is built on Techtonique and the powerful Python ecosystem. At the moment, it focuses on small datasets, with a limit of 1MB per input. Both clickable web interfaces and Application Programming Interfaces (APIs, see below) are available.

Currently, the available functionalities include:

Data visualization. Example: Which variables are correlated, and to what extent?
Probabilistic forecasting. Example: What are my projected sales for next year, including lower and upper bounds?
Machine Learning (regression or classification) for tabular datasets. Example: What is the price range of an apartment based on its age and number of rooms?
Survival analysis, analyzing time-to-event data. Example: How long might a patient live after being diagnosed with Hodgkin’s lymphoma (cancer), and how accurate is this prediction?
Reserving based on insurance claims data. Example: How much should I set aside today to cover potential accidents that may occur in the next few years?

As mentioned earlier, this tool includes both clickable web interfaces and Application Programming Interfaces (APIs).
APIs allow you to send requests from your computer to perform specific tasks on given resources. APIs are programming language-agnostic (supporting Python, R, JavaScript, etc.), relatively fast, and require no additional package installation before use. This means you can keep using your preferred programming language or legacy code/tool, as long as it can speak to the internet. What are requests and resources?

In Techtonique/APIs, resources are Statistical/Machine Learning (ML) model predictions or forecasts.
A common type of request might be to obtain sales, weather, or revenue forecasts for the next five weeks. In general, requests for tasks are short, typically involving a verb and a URL path — which leads to a response.

Below is an example. In this case, the resource we want to manage is a list of users.

Request type (verb): GET
- | URL Path: http://users | Endpoint: users | API Response: Displays a list of all users |
- | URL Path: http://users/:id | Endpoint: users/:id | API Response: Displays a specific user |
Request type (verb): POST
- | URL Path: http://users | Endpoint: users | API Response: Creates a new user |
Request type (verb): PUT
- | URL Path: http://users/:id | Endpoint: users/:id | API Response: Updates a specific user |
Request type (verb): DELETE
- | URL Path: http://users/:id | Endpoint: users/:id | API Response: Deletes a specific user |

In Techtonique/APIs, a typical resource endpoint would be /MLmodel. Since the resources are predefined and do not need to be updated (PUT) or deleted (DELETE), every request will be a POST request to a /MLmodel, with additional parameters for the ML model.
After reading this, you can proceed to the /howtoapi page.

To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Techtonique web app for data-driven decisions using Mathematics, Statistics, Machine Learning, and Data Visualization

View Details

[This article was first published on schochastics - all things R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Most of the posts on this blog include a lot of code an little text besides explaining what the code does. Although I “write for a living” (scientific papers and stuff), I am not a big fan of producing walls of text. “A line of code says more than a thousand words”, you know…

I want to fight those demons though and for that reason, I decided to add a non-code (yet R related) series of posts to this blog. The purpose of these posts will be to shine some light on “behind the scene” activities around R package development which are mostly invisible to users, not always fun, but nevertheless necessary.

The first part of the series is about the recent archival of my R package timeless due to not complying with CRAN policies. Having a package archived on CRAN essentially means that you cannot install it via

install.packages("timeless") anymore. This is not the end of the world, but not everybody might know how to install the package regardless of the archival and in worst case it might create a sense of distrust. In reality, though, the reasons for archival of a package are usually quite benign, even when the reason sounds harsh, like in my case.

The R package timelessI introduced timeless package on this blog not to long ago. I created it when I was trying to learn and experiment with Rust. timeless is a fast general purpose date/time converter similar to anytime. It turned out to be quite a bit faster than anytime, yet I would still consider anytime to be more robust than timeless. Dates and times can be horribly complex to parse.

The “about to be archived” emailEnd of August, I received one of those scary CRAN emails about a package being on the short list to be archived. 1

The ‘Using Rust’ part of the policy says

“Please report the version of rustc used (as R does for C, C++ and Fortran compilers) in the installation log especially if compilation fails, so best reported before starting compilation (as R does).”

The ‘especially’ part was added from bitter experience of no version being reported on installation failure.

On our check runs [redacted] did not report a version, and

[redacted] timeless

only reported after a successful compilation.

Please correct before 2024-09-06 to safely retain the package on CRAN.

I was confused by the mail. I understood that I need to report the version of Rust used, which I apparently do, but only after successful compilation. I consulted the CRAN Rust policy to gain a better understanding. The following gave a specific instruction for the DESCRIPTION file.

The package should declare

SystemRequirements: Cargo (Rust’s package manager), rustc

as they can be packaged separately, and can have different versions. So state carefully any version requirement

I naïvely thought I just need to add the version of Rust to the SystemRequirements field in the DESCRIPTION file.

SystemRequirements: Cargo (Rust's package manager), rustc (>= 1.67.1) Without giving it further thought I submitted the new version to CRAN and quickly received the magical words

Thanks, on its way to CRAN.

So I thought I fixed the issue and moved on.

A few days after the archival deadline I randomly ended up on the CRAN landing page of timeless, just to find out that the package was indeed archived. The stated reason was

Archived on 2024-09-11 for policy violations. This actually does sound bad, like I actively refused to follow CRAN policies. I was quite surprised about this because there were no reminders or messages that the package was archived.

Obviously my fix was not enough (It always felt a bit too simple anyway). I decided to not pursue this to the bitter end and let the package live on GitHub only for now. But still, I was curious how to actually solve this. For that, I needed to look into the basics of Rust code in R packages again.

Using Rust in R packages and the root of the issueTo call rust code in R (packages), you should install the rextendr package. Setting up a package to call Rust is then super easy.

library(rextendr)usethis::create\_package("myrustpkg")rextendr::use\_extendr() use_extendr() sets up everything necessary in your package to call Rust. Now you “just” need to write the Rust code.

Here is where I went wrong when I was trying to fix the issue of timeless. I only consulted the CRAN policies and cried at social media for help. What I should have done is ask the maintainers of rextendr for guidance. I only did so when it was too late. The maintainers were obviously well aware of the issue and have been discussing a solution.

To not violate CRAN policies, the used Rust version for compilation needs to be visible in the install logfile like in the example below.

I was thinking of trying to implement a hacky solution for myself, but quickly decided that I can wait until a fix is available in the rextendr package on CRAN. Until then, timeless will remain on GitHub only.

AftermathThere was some discussion afterwards about whether or not CRAN should communicate such decisions earlier and better. Personally, I would prefer to get one email too much rather than one to little, but I also see that constant (potentially false negative) reminders can also be annoying for maintainers. Ultimately, it should of course be the task of package maintainers to keep an eye on potential issues, but a little more support would be great. For my own purposes, I have built a dashboard (Code) which updates ones a week via GitHub Actions. The dashboard gives me a quick overview of activity on GitHub and current check results of packages on CRAN. That way, I see if I need to react to anything.

See the red errors for graphlayouts? that’ll be the topic of the next post in this series.

Footnotes1. If you are interested, there exists a dashboard that keeps track of all packages currently being threatened to be archived.︎

ReuseCC BY 4.0CitationBibTeX citation: @online{schoch2024, author = {Schoch, David}, title = {Tales from {Open} {Source} {Development} {I:} {Your} Package Is Archived}, date = {2024-09-25}, url = {http://blog.schochastics.net/posts/2024-09-25\_tales-from-os-dev-001/}, langid = {en}} For attribution, please cite this work as:Schoch, David. 2024. “Tales from Open Source Development I: YourPackage Is Archived.” September 25, 2024. http://blog.schochastics.net/posts/2024-09-25_tales-from-os-dev-001/. To leave a comment for the author, please follow the link and comment on their blog: schochastics - all things R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Tales from Open Source Development I: Your package is archived

View Details

[This article was first published on Rstats – quantixed, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.One of the joys of posting a preprint is seeing that people are viewing, downloading and (hopefully) reading your paper. On bioRxiv you can check out the statistics for your paper in the metrics tab.

We posted a preprint recently and it clocked up over 1,000 views in the first day or so. This made me wonder: is that a lot of views or not? How does it compare to other preprints in our category? I wrote some code to find out. It turns out our paper was the third most viewed cell biology preprint in September. My co-authors and I were very grateful for the interest!

Anyway, this post is about the code and how to get the necessary data to look at preprint view metrics. I’ll use it to look at the more interesting question of preprint view dynamics in our category “Cell Biology” this year. Read on for the R stuff, or skip to the findings.

tl;dr a cell biology preprint gets 50% of its lifetime views in the first 4-5 days.

OutlineWe’ll use rbiorxiv to get a list of peprints of interest. Then we’ll use this list to scrape the metrics data using rvest. Finally, we’ll wrangle the data and make some plots using ggplot etc.

There are three metrics summarised for each month. Abstract, full text and PDF. Abstract is usually the biggest number for two reasons: first, unless people click on a link to the PDF or the full text specifically, the landing page is the abstract. So typically a reader would view the abstract and then click on PDF or full text; second, full text takes a day or so to render so this number is usually the lowest number.

The final thing to note is that if a preprint is posted on the last day of the month, it will have one day’s worth of metrics, whereas a preprint posted at the beginning of the month will have a full month’s worth – more on this later.

The code ```

Load packageslibrary(rbiorxiv)library(tidyverse)library(rvest)# get datadf <- biorxiv_content(from = "2024-01-01", to = "2024-09-17", limit = "*", format = "df")# filter for category is "cell biology"df_all <- subset(df, df$category == "cell biology")# select version 1 onlydf_all <- subset(df_all, df_all$version == 1)# convert jatsxml column to url by deleting .source.xml from the end of eachdf_all$url <- gsub(".source.xml", ".article-metrics", df_all$jatsxml)

``` This gives us urls to scrape for all 1st version preprints in the cell biology category in 2024. Note that subsequent versions of the preprint give the same data, so we select the 1st version to reduce the amount of scraping.

```

for every url in df_all get the metrics# create an empty data frame first to store the resultsusage_metrics_df <- data.frame()# loop through each url and extract the usage metricsfor (i in 1:nrow(df_all)) { ex_paper <- tryCatch( read_html(df_all$url[i]), error = function(e) { message(paste("Error reading URL:", df_all$url[i])) return(NULL) } ) if (is.null(ex_paper)) { next } usage_metrics <- ex_paper %>% html_node(".highwire-stats") %>% html_table() if (ncol(usage_metrics == "NULL") != 4) { next } # clean up the table usage_metrics_clean <- usage_metrics %>% rename(month = 1, abstract_views = 2, full_views = 3, pdf_views = 4) %>% mutate(across(c(abstract_views, full_views, pdf_views), ~ as.numeric(gsub(",", "", .)))) %>% mutate(month = as.Date(paste0("01-", month), format = "%d-%B")) usage_metrics_clean$doi <- df_all$doi[i] # add the DOI to the cleaned metrics # add the cleaned metrics to the list usage_metrics_df <- rbind(usage_metrics_df, usage_metrics_clean)}# merge with df_all to get the title and authorsfinal_df <- merge(df_all, usage_metrics_df, by = "doi", all.y = TRUE)final_df$total_views <- final_df$abstract_views + final_df$full_views + final_df$pdf_views

``` OK, now we have all the metrics data. Let’s combine views of abstract, full text and PDF by summing them to get an idea of how many clicks a preprint received.

```

for each doi, sum the views columnsfinal_sum_df <- final_df %>% group_by(doi) %>% summarise( abstract_views = sum(abstract_views, na.rm = TRUE), full_views = sum(full_views, na.rm = TRUE), pdf_views = sum(pdf_views, na.rm = TRUE), total_views = sum(total_views, na.rm = TRUE), date = as.Date(date[1]) )ggplot(final_sum_df, aes(x = date, y = total_views)) + geom_point(alpha = 0.25) + labs(x = "Posting date", y = "Total views") + scale_y_log10(limits = c(100, 30000)) + theme_classic(9)ggsave("Output/Plots/total_views.png", width = 8, height = 6)

``` InsightsFor each preprint the total views received over all months is typically between 1000-5000 (note the log scale). The values are quite steady from January to June and there’s a fall off in September. More on this below but this is a clue to the half-life of interest in a preprint. Another thing to note is that there a dip in Total Views around 4th July which I am assuming is because the US (which is a major fraction of the audience here) is away from work. Again this is a hint about a preprint’s half-life.

```

the column "month" shows the first of the month, as yyyy-mm-01 string,# convert to show last day of that monthfinal_df$month <- as.Date(final_df$month, format = "%Y-%m-%d")# add one month and then take away one dayfinal_df$month <- as.Date(format(as.Date(final_df$month, format = "%Y-%m-%d") + months(1) - days(1), "%Y-%m-%d"))# calculate the number of days between the date and the monthfinal_df$days_from_month <- as.numeric(final_df$month - as.Date(final_df$date))# also make a column where "date" is converted to the first of that monthfinal_df$month_date <- as.Date(format(as.Date(final_df$date, format = "%Y-%m-%d"), "%Y-%m-01"))# change month_date to be the name of the month & factor properlyfinal_df$month_date <- factor( month.abb[as.numeric(format(final_df$month_date, "%m"))], levels = month.abb)ggplot(final_df, aes(x = days_from_month, y = total_views, group = doi)) + geom_path(alpha = 0.10) + labs(x = "Days since posting", y = "Total Views") + scale_y_log10() + facet_wrap(~month_date) + theme_classic(9)ggsave("Output/Plots/views_days_facet.png", width = 8, height = 6)

``` We can look at each preprint’s metrics by month and plot them out versus the days since posting. The initial dynamics are similar regardless of when in the year a preprint was posted. The views are high in the first month and then drop away.

In the final code block we’ll dig into these initial dynamics.

```

for final_df calculate the total views for each doi and each month as a# fraction of the total for that doi# take the doi and total_views column from final_sum_df, merge with final_df,# then normalise total_views by total_views_yfinal_sum_df2 <- final_sum_df %>% select(doi, total_views) %>% rename(total_views_y = total_views)final_alt_df <- merge(final_df, final_sum_df2, by = "doi", all.x = TRUE)final_alt_df$normalised_total_views <- final_alt_df$total_views / final_alt_df$total_views_yggplot(final_alt_df, aes(x = days_from_month, y = normalised_total_views, group = doi)) + geom_path(alpha = 0.10) + labs(x = "Days since posting", y = "Fraction of total views") + geom_vline(xintercept = 30, linetype = "dashed") + theme_classic(9)ggsave("Output/Plots/views_days_frac.png", width = 8, height = 6)# generate the mean of normalised_total_views by days since postingfinal_alt_df %>% group_by(days_from_month) %>% summarise(mean_normalised_total_views = mean(normalised_total_views, na.rm = TRUE)) %>% ggplot(aes(x = days_from_month, y = mean_normalised_total_views)) + geom_point(alpha = 0.25) + labs(x = "Days since posting", y = "Mean fraction of total views") + lims(y = c(0, 1)) + theme_classic(9)ggsave("Output/Plots/views_days_frac_avg.png", width = 8, height = 6)# final_df has the total views for each article for per month,# first turn that into a cumulative sum for each paperfinal_df_cumsum <- final_df %>% group_by(doi, date) %>% mutate(cumsum_total_views = cumsum(total_views))final_alt_df2 <- merge(final_df_cumsum, final_sum_df, by = c("doi"), all.x = TRUE)final_alt_df2$normalised_cumsum_total_views <- final_alt_df2$cumsum_total_views / final_alt_df2$total_views.yfinal_alt_df2 %>% group_by(days_from_month) %>% summarise(mean_normalised_total_views = mean(normalised_cumsum_total_views, na.rm = TRUE)) %>% ggplot(aes(x = days_from_month, y = mean_normalised_total_views)) + geom_point(alpha = 0.5) + labs(x = "Days since posting", y = "Mean fraction of total views") + lims(y = c(0, 1)) + theme_classic(9)ggsave("Output/Plots/views_days_frac_cumsum_avg.png", width = 8, height = 6)

``` When we look at the Total Views as a fraction of the total for the preprint’s lifetime, we can see that a preprint gathers about 75% of its lifetime views in the first month! After that, the following month’s metrics decline dramatically as the average preprint bumbles along picking up its final views.

Plotted a different way, by looking at the cumulative views as a fraction of the total, we can see that in the first 4-5 days after posting, a preprint gets half of its lifetime views.

There’s some caveats here. We’re only looking at Cell Biology, and only in first nine months of 2024. Conversely, Cell Biology is a large category so it should have a good range of hot papers and less popular work, but it’s not so large that it has distinct subdomains (like Neuroscience) that may affect preprint dynamics.

Obviously, with the attention on preprints being pretty much immediate, posting them close to major holidays – or maybe even close to the weekend – is likely to affect how many people will see your preprint.

—

The post title comes from the Belaire album “Exploding, Impacting”.

To leave a comment for the author, please follow the link and comment on their blog: Rstats – quantixed.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploding, Impacting: looking at bioRxiv preprint view dynamics with R

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Since April, Nvidia has tried to hold above the point forecasting line. As predictive intervals’ slopes are slightly up; could Nvidia continue an uptrend?

Source code:

library(tidyverse)library(timetk)library(tidymodels)library(modeltime)library(ggthemes)#Nvidia (monthly)df\_nvidia <- tq\_get("NVDA", to = "2024-08-29") %>% tq\_transmute(select = close, mutate\_fun = to.monthly, col\_rename = "nvidia") %>% mutate(date = as.Date(date))#FEDFUNDSdf\_fedfunds <- read\_csv("fedfunds.csv")#Tidying the datadf\_fedfunds\_tidy <- df\_fedfunds %>% janitor::clean\_names() %>% select(date = release\_date, fedfunds = actual) %>% #Converts string to date object mutate(date = case\_when( !is.na(parse\_date(date, format = "%b %d, %Y")) ~ parse\_date(date, format = "%b %d, %Y"), !is.na(parse\_date(date, format = "%d-%b-%y")) ~ parse\_date(date, format = "%d-%b-%y") )) %>% #Removes the first three blank rows slice\_tail(n = -3) %>% mutate(date = floor\_date(date, "month") %m+% months(1), fedfunds = str\_remove(fedfunds, "%") %>% as.numeric()) %>% #makes regular time series by filling the time gaps pad\_by\_time(date, .by = "month") %>% fill(fedfunds, .direction = "up") %>% distinct(date, .keep\_all = TRUE)#Mergs all the data setsdf\_merged <- df\_nvidia %>% left\_join(df\_fedfunds\_tidy)#Modeling#Splitting tha datadf\_split <- df\_merged %>% time\_series\_split(assess = "1 year", cumulative = TRUE)df\_train <- training(df\_split)df\_test <- testing(df\_split)#Bootstrapping for tuningset.seed(12345)df\_folds <- bootstraps(df\_train, times = 100)#Preprocessing#Preprocessing for Boosting ARIMArec\_arima\_boost <- recipe(nvidia ~ ., data = df\_train) %>% step\_date(date, features = c("year", "month")) %>% step\_dummy(date\_month, one\_hot = TRUE) %>% step\_normalize(all\_numeric\_predictors())#Preprocessing for XGBoost and MARSrec\_mars\_xgboost <- rec\_arima\_boost %>% step\_rm(date)#Models#Boosted ARIMA Regression#(https://business-science.github.io/modeltime/reference/arima\_boost.html)mod\_arima\_boost <- arima\_boost( min\_n = tune(), learn\_rate = tune(), trees = tune() ) %>% set\_engine(engine = "auto\_arima\_xgboost")# Model 1: auto\_arima ----arima\_reg() %>% set\_engine(engine = "auto\_arima") %>% fit(nvidia ~ ., data = df\_merged)#Multivariate adaptive regression splines (MARS) via earth#(https://parsnip.tidymodels.org/reference/details\_mars\_earth.html)mod\_mars <- mars(num\_terms = tune(), prune\_method = tune()) %>% set\_engine("earth", nfold = 10) %>% set\_mode("regression") #Boosted trees via xgboost#(https://parsnip.tidymodels.org/reference/details\_boost\_tree\_xgboost.html)mod\_boost\_tree <- boost\_tree(mtry = tune(), trees = tune(), min\_n = tune(), learn\_rate = tune()) %>% set\_engine("xgboost") %>% set\_mode("regression") #Workflow setswflow\_arima\_boost <- workflow\_set( preproc = list(rec\_arima\_boost = rec\_arima\_boost), models = list(ARIMA\_boost = mod\_arima\_boost) ) wflow\_mars\_xgboost <- workflow\_set( preproc = list(rec\_mars\_xgboost = rec\_mars\_xgboost), models = list(MARS = mod\_mars, XGBoost = mod\_boost\_tree) )#Combining all the workflowswflow\_all <- bind\_rows( wflow\_arima\_boost, wflow\_mars\_xgboost ) %>% #Making the workflow ID's a little more simple: mutate(wflow\_id = str\_remove(wflow\_id, "(rec\_arima\_boost\_)|(rec\_mars\_xgboost\_)"))#Tuning and evaluating all the modelsgrid\_ctrl <- control\_grid( save\_pred = TRUE, parallel\_over = "everything", save\_workflow = TRUE )grid\_results <- wflow\_all %>% workflow\_map( seed = 98765, resamples = df\_folds, grid = 10, control = grid\_ctrl )#Accuracy of the grid resultsgrid\_results %>% rank\_results(select\_best = TRUE, rank\_metric = "rsq") %>% select(Models = wflow\_id, .metric, mean)#Finalizing the model with the best parametersbest\_param <- grid\_results %>% extract\_workflow\_set\_result("ARIMA\_boost") %>% select\_best(metric = "rsq")wflw\_fit <- grid\_results %>% extract\_workflow("ARIMA\_boost") %>% finalize\_workflow(best\_param) %>% fit(df\_train)#Calibration datadf\_cal <- wflw\_fit %>% modeltime\_calibrate(new\_data = df\_test)#Predictive intervals for Boosted ARIMAdf\_cal %>% modeltime\_forecast(actual\_data = df\_merged %>% filter(date >= last(date) - months(12)), new\_data = df\_test) %>% plot\_modeltime\_forecast(.interactive = FALSE, .legend\_show = FALSE, .line\_size = 1, .color\_lab = "", .title = "Predictive Intervals for Nvidia") + labs(subtitle = "Monthly Stock Prices<br><span style = 'color:red;'>Boosted ARIMA Point Forecasting Line</span>") + scale\_x\_date(breaks = c(make\_date(2023,8,1), make\_date(2024,4,1), make\_date(2024,8,1)), labels = scales::label\_date(format = "%Y %b"), expand = expansion(mult = c(.1, .1))) + theme\_wsj(base\_family = "Bricolage Grotesque", color = "grey", base\_size = 12) + theme(legend.position = "none", plot.subtitle = ggtext::element\_markdown(size = 17, face = "bold")) To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Comparing Many Models: An Uptrend for Nvidia?

View Details

[This article was first published on RStudioDataLab, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.To leave a comment for the author, please follow the link and comment on their blog: RStudioDataLab.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Installation, Integration, and Usage Tips | FAQs

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionOutliers can significantly skew your data analysis results, leading to inaccurate conclusions. For R programmers, effectively identifying and removing outliers is crucial for maintaining data integrity. This guide will walk you through various methods to handle outliers in R, focusing on multiple columns, using a synthetic dataset for demonstration.

Understanding OutliersDefinition and Impact on Data Analysis

Outliers are data points that differ significantly from other observations. They can arise due to variability in the measurement or may indicate experimental errors. Outliers can heavily influence the results of your data analysis, leading to biased estimates and incorrect conclusions.

Common Causes of Outliers

Outliers typically result from data entry errors, measurement errors, or natural variability. Identifying their cause is essential to determine whether they should be removed or retained.

Methods to Identify OutliersVisual Methods: Boxplots and Scatter Plots

Boxplots and scatter plots are simple yet effective visual tools for spotting outliers. Boxplots display the distribution of data and highlight values that fall outside the whiskers, indicating potential outliers.

```

Creating a synthetic datasetset.seed(123)data <- data.frame( Column1 = rnorm(100, mean = 50, sd = 10), Column2 = rnorm(100, mean = 30, sd = 5))# Introducing some outliersdata$Column1[c(5, 20)] <- c(100, 120)data$Column2[c(15, 40)] <- c(50, 55)# Boxplot to visualize outliersboxplot(data$Column1, main="Boxplot for Column1")

```

boxplot(data$Column2, main="Boxplot for Column2") Statistical Methods: Z-score, IQR, and Others

Statistical methods like Z-score and Interquartile Range (IQR) provide a more quantitative approach to identifying outliers. The Z-score measures how many standard deviations a data point is from the mean, while IQR focuses on the spread of the middle 50% of data.

Using the IQR MethodExplanation of the IQR Method

The IQR method identifies outliers by calculating the range within the first and third quartiles (Q1 and Q3). Outliers are typically considered as data points below Q1 - 1.5IQR or above Q3 + 1.5IQR.

Step-by-Step Guide to Applying IQR in R for Multiple Columns

Q1 <- apply(data, 2, quantile, 0.25)Q3 <- apply(data, 2, quantile, 0.75)IQR <- Q3 - Q1print(IQR)

Column1 Column2 12.842233 6.403111

outliers <- (data < (Q1 - 1.5 * IQR)) | (data > (Q3 + 1.5 * IQR))head(outliers)

Column1 Column2[1,] FALSE FALSE[2,] TRUE FALSE[3,] FALSE FALSE[4,] TRUE FALSE[5,] TRUE TRUE[6,] TRUE FALSE

data\_cleaned <- data[!apply(outliers, 1, any), ]head(data\_cleaned)

Column1 Column21 44.39524 26.447973 65.58708 28.766547 54.60916 26.075488 37.34939 21.660299 43.13147 28.0988711 62.24082 27.12327 Using Z-score for Outlier DetectionExplanation of Z-score

A Z-score indicates how many standard deviations a data point is from the mean. A common threshold for identifying outliers is a Z-score greater than 3 or less than -3.

Implementing Z-score in R for Multiple Columns

z\_scores <- scale(data)head(z\_scores)

Column1 Column2[1,] -0.6238919 -0.60719837[2,] -0.3577600 0.22933945[3,] 1.0836030 -0.20616583[4,] -0.1154877 -0.29338416[5,] 3.8563627 -0.81580479[6,] 1.2095846 -0.03176142

outliers <- abs(z\_scores) > 3head(outliers)

Column1 Column2[1,] FALSE FALSE[2,] FALSE FALSE[3,] FALSE FALSE[4,] FALSE FALSE[5,] TRUE FALSE[6,] FALSE FALSE

data\_cleaned <- data[!apply(outliers, 1, any), ]head(data\_cleaned)

Column1 Column21 44.39524 26.447972 47.69823 31.284423 65.58708 28.766544 50.70508 28.262296 67.15065 29.774867 54.60916 26.07548 Removing Outliers from a Single ColumnCode Examples and Explanation

To remove outliers from a single column using the IQR method:

Q1 <- quantile(data$Column1, 0.25)Q3 <- quantile(data$Column1, 0.75)IQR <- Q3 - Q1outliers <- data$Column1 < (Q1 - 1.5 * IQR) | data$Column1 > (Q3 + 1.5 * IQR)data\_cleaned\_single <- data[!outliers, ]head(data\_cleaned\_single)

Column1 Column21 44.39524 26.447972 47.69823 31.284423 65.58708 28.766544 50.70508 28.262296 67.15065 29.774867 54.60916 26.07548 Removing Outliers from Multiple ColumnsCode Examples and Explanation

To apply the same logic across multiple columns:

data\_cleaned <- datafor(col in names(data)) { Q1 <- quantile(data[[col]], 0.25) Q3 <- quantile(data[[col]], 0.75) IQR <- Q3 - Q1 outliers <- data[[col]] < (Q1 - 1.5 * IQR) | data[[col]] > (Q3 + 1.5 * IQR) data\_cleaned <- data\_cleaned[!outliers, ]} Handling Outliers in Multivariate DataTechniques for Multivariate Outlier Detection

In multivariate datasets, outliers can be detected using techniques like Mahalanobis distance, which accounts for correlations between variables.

mahalanobis\_distance <- mahalanobis(data, colMeans(data), cov(data))outliers <- mahalanobis\_distance > qchisq(0.975, df=ncol(data))data\_cleaned\_multivariate <- data[!outliers, ]head(data\_cleaned\_multivariate)

Column1 Column21 44.39524 26.447972 47.69823 31.284423 65.58708 28.766544 50.70508 28.262296 67.15065 29.774867 54.60916 26.07548 Automating Outlier Removal in RWriting Functions to Streamline the Process

You can create a custom function to automate outlier removal using either the IQR or Z-score method:

remove\_outliers <- function(data) { cleaned\_data <- data for(col in names(data)) { Q1 <- quantile(data[[col]], 0.25) Q3 <- quantile(data[[col]], 0.75) IQR <- Q3 - Q1 outliers <- data[[col]] < (Q1 - 1.5 * IQR) | data[[col]] > (Q3 + 1.5 * IQR) cleaned\_data <- cleaned\_data[!outliers, ] } return(cleaned\_data)}# Applying the functiondata\_cleaned\_function <- remove\_outliers(data)cat("Original data:", nrow(data), "| Cleaned data:", nrow(data\_cleaned\_function), "\n")

Original data: 100 | Cleaned data: 97

head(data\_cleaned\_function)

Column1 Column21 44.39524 26.447972 47.69823 31.284423 65.58708 28.766544 50.70508 28.262296 67.15065 29.774867 54.60916 26.07548 Case Study: Real-world ApplicationExample Dataset and Analysis

Consider a synthetic dataset containing columns of normally distributed data with added outliers. Applying the methods discussed can help clean the dataset for better analysis and visualization, ensuring accuracy and reliability in results.

Best Practices for Outlier RemovalWhen to Remove vs. When to Keep Outliers

Not all outliers should be removed. Consider the context and reason for their existence. Sometimes, outliers can provide valuable insights.

Common Pitfalls and How to Avoid ThemMistakes to Avoid in Outlier Detection and Removal

Avoid blanket removal of outliers without understanding their cause. Ensure your data cleaning process is well-documented and reproducible.

Advanced TechniquesMachine Learning Approaches to Handle Outliers

Advanced machine learning techniques, such as isolation forests or autoencoders, can handle outliers more effectively, especially in large datasets.

Tools and Packages in R for Outlier DetectionOverview of Useful R Packages

Several R packages can assist in outlier detection, such as dplyr, caret, and outliers. These tools offer functions and methods to streamline the process.

ConclusionProperly identifying and handling outliers is crucial for accurate data analysis in R. By applying the methods and best practices outlined in this guide, you can ensure your datasets remain robust and reliable.

Quick Takeaways Context Matters: Always consider the context before removing outliers. * Multiple Methods: Use a combination of visual and statistical methods for detection. * Automation*: Automate processes for efficiency and consistency.

FAQs1. What is an outlier in R? An outlier is a data point significantly different from other observations in a dataset. 2. How does the IQR method work in R? The IQR method calculates the range between the first and third quartiles and identifies outliers as points outside 1.5 times the IQR from the quartiles. 3. Can I automate outlier removal in R? Yes, by creating functions or using packages like dplyr for streamlined processing. 4. What are the best R packages for outlier detection? Packages like dplyr, caret, and outliers are useful for detecting and handling outliers. 5. Should I always remove outliers from my dataset? Not necessarily. Consider the context and potential insights the outliers might provide.

Your Turn!We’d love to hear about your experiences with outlier removal in R! Share your thoughts and this guide with your network on social media.

References* GeeksforGeeks: Understanding Outliers * R-bloggers: Outliers and Data Analysis * Stack Overflow: Excluding Outliers in R

Happy Coding!

Z Scoure Outlier Scoring To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Remove Outliers from Multiple Columns in R: A Comprehensive Guide

View Details

[This article was first published on Mad (Data) Scientist, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’m very pleased to announce a new package, dsld, available on CRAN. This is the work of eight talented undergrad students. I provided the concept and some general guidance, but this is their work.

The package is aimed at dealing with discrimination — race, gender, age — in the workplace, education, health care and so on. It consists of analytical, graphical and tabular tools for:

detection of discrimination
addressing discriminatory predictive models

We hope the package will be useful in a variety of application venues, such as:

inspiring motivation in statistics courses
discrimination litigation
internal HR audits
social science research

The usefulness of the package is further enhanced by the availability of a free companion textbook. It uses dsld examples throughout, but its role is to explain the statistical concepts, not to serve as a user manual for the package.

One can acquire a good idea of the nature of the book by reading the example on law school admissions, pp. 34-43.

Needless to say, comments are welcome!

To leave a comment for the author, please follow the link and comment on their blog: Mad (Data) Scientist.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: New R Package: Data Science Looks at Discrimination (dsld)

View Details

[This article was first published on Maëlle's R blog on Maëlle Salmon's personal website, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’ve recently been dealing with legacy code refactoring both in theory and in practice: while I’m continuing some work on the igraph R package, I’ve started reading Working Effectively with Legacy Code by Michael Feathers, that had been in my to-read pile for months. In this post, I’ll summarize some ideas from both the book and my work.

“Cover and modify” with “characterization tests”When you start modifying your rusty code, how do you ensure you do not break existing and important behaviour inadvertently? In his book, Michael Feathers advocates for “cover and modify” (test, then amend) rather than “edit and pray” (that you’re not breaking anyone’s script or reverse dependency). Not necessarily brand-new ideas but nice phrases.

In igraph for instance, when I have to add tests for a function that does not have any test yet, I tend to use code from examples as a starting point. I run the code in an unit test and add expectations that seem relevant to me, and to collaborators more knowledgeable about graphs. Part of this work is very mechanical and probably also very common: running the code to figure out its output, codify its output as expectation. For instance if f() returns a numeric of value 0.42, I’ll add something like testthat::expect_type(output, "double") and testthat::expect_equal(output, 0.42). It turns out such tests have a name in the book, which made me very happy: they are “characterization tests”.

How to characterize the outputThree tips on how to characterize the output:

You can use typeof() and sloop::otype() to help you find out which testthat type/class expectation you should use. The latter is linked from a chapter in Hadley Wickham’s Advanced R that in turn is linked from the testthat docs. I am mildly embarrassed about even needing the former.
You can use constructive::construct() to record an expected value in a more readable way than with dput() – for use in snapshot tests or even directly in testthat::expect_equal().
Michael Feathers notes that characterization tests are a form of documentation, so “you have to think about what will be important to the reader”. I try to order expectations logically, and… I request a review on my pull requests so there might actually be another reader involved in the short term.

How to check coverageTo know whether you’ve covered the code you’re about to modify, you have to assess your tests.

For some (historical I suppose) reason, in Michael Feathers’ book code coverage tooling seems absent. I’ve instead read about “sensing variables” and using the debugger to step through the code.

In R when using devtools we can rely on devtools::test_coverage_active_file() (including showing you an interactive report!).1 It’s allowed me to see some branches my tests weren’t hitting yet. Running it again and again and seeing more lines in green is very satisfying.

The code coverage might be a part of the continuous integration workflows but in the case of igraph it only runs after all other checks (so not quickly).

ConclusionIn this post I’ve presented some ideas around “covering” R code with tests before you “modify” it. Do you have any insight to share that I haven’t… covered?

I first used a few lines that did the same thing (but obviously not as well) because I had forgotten about this function. Don’t do like me! ︎

To leave a comment for the author, please follow the link and comment on their blog: Maëlle's R blog on Maëlle Salmon's personal website.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cover and modify, some tips for R package development

View Details

[This article was first published on R – Win Vector LLC, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.BARUG is back!!! Hope to see you there!

https://www.meetup.com/r-users/events/303488652/?eventOrigin=group_upcoming_events

To leave a comment for the author, please follow the link and comment on their blog: R – Win Vector LLC.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Bay Area use R Meetup is Back!!!!

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium recently interviewed Kristina Riemer, director of the CCT Data Science Team at the University of Arizona, and Eric Scott, Scientific Programmer and Educator in the CCT Data Science Team, the developers behind the volcalc package, to discuss the motivation and development of this innovative tool designed to automate the calculation of chemical compound volatilities. volcalc streamlines the process by allowing users to input a compound and quickly receive its volatility information, eliminating the need for time-consuming manual calculations. Initially created to assist Dr. Laura Meredith in managing a large database of volatile compounds, volcalc has since grown into a more versatile tool under Eric’s leadership, now supporting a wider range of researchers.

Kristina and Eric share insights into the challenges they faced, including managing dependencies, integrating with CRAN and Bioconductor, and refining complex molecular identification methods. They also discuss future enhancements, such as incorporating temperature-specific volatility calculations and expanding the package’s functionality to estimate other compound characteristics. This project was funded by the R Consortium.

Could you share what motivated the development of the volcalc package and how it aligns with the broader goals of the R ecosystem, particularly in scientific computing?

Kristina: I was heavily involved in the initial development of volcalc, and later on, Eric took over the project. We developed volcalc because we began collaborating with Dr. Laura Meredith, who was compiling a database of volatile chemical compounds. At the time, she had around 300 compounds, and her students manually gathered details for each one by examining their representations and calculating various associated values. This process was tedious and prone to errors, so we thought there must be a more efficient and automated way to handle it.

That’s when we came up with the idea of creating a pipeline where someone could input a compound and quickly receive its volatility information, eliminating the need for all the manual labor. The purpose of volcalc was to transform the process from taking months to gather details for 300 compounds to obtaining information for thousands in a much shorter time.

Eric: volcalc was initially developed specifically for a project where the researchers were mainly interested in chemical compounds from the KEGG database (Kyoto Encyclopedia of Genes and Genomes). When I joined the team and learned about the project, I was thrilled because, as a chemical ecologist, I saw its potential. However, I also recognized a limitation: the tool only worked with the KEGG database. This was a drawback because many researchers, including food scientists and others who work with similar compounds, might not find their compounds in that specific database.

This realization inspired me to apply for the R Consortium grant. We saw a significant opportunity to expand volcalc, making it more flexible and applicable to a wider range of researchers. We also wanted to improve its integration within the R ecosystem by adding features like returning the file path of a molecule representation after downloading it, so it could be easily piped into subsequent steps. These enhancements aimed to make the tool more versatile and user-friendly for a broader audience.

What were the most significant challenges you faced during the development of the initial version of volcalc, and how did you overcome them?

Kristina: One of the most challenging aspects of developing volcalc, which continues to be an issue, is managing dependencies. Specifically, we rely heavily on a command-line program to handle much of the processing. Early on, we struggled with how to enable users to run volcalc without needing to install this program on their own computers, as many of our users aren’t familiar with that kind of setup. I spent a lot of time trying to create a reproducible environment using Binder, but I was never able to get it fully working. Even today, there are still issues related to managing these dependencies, which Eric can elaborate on further.

It was incredibly important to have Eric on this project because I don’t have a strong background in chemistry. His ability to come in and figure out some of the intricate details that would have taken me much longer to grasp was a huge advantage. The more we can collaborate with domain experts, the better our results will be.

Eric: One thing that has helped with the dependency challenges is that we’ve started building volcalc on R-Universe, which means binaries are available there. While it’s not on CRAN yet, having these binaries on R-Universe makes installation a bit easier. However, we’ve faced some challenges with dependencies, particularly because two of them are from Bioconductor. We didn’t originally aim to develop this package for Bioconductor, which uses S4 objects and has different standards than CRAN. Our goal was to get it on CRAN, but our first submission was rejected because the license field for the Bioconductor package wasn’t formatted to CRAN’s liking. These differences between Bioconductor and CRAN have created barriers, even though the authors of the Bioconductor package have been very responsive. Their package works fine on Bioconductor, but it doesn’t meet CRAN’s criteria, which has been a frustrating challenge.

Another major challenge in developing volcalc relates to the method we use for estimating volatility. This method involves counting the numbers of different functional groups on molecules—such as hydroxyl groups or sulfur atoms—and assigning coefficients to them. To do this programmatically, we use something called SMARTS, which is essentially like regular expressions but for molecular structures. Regular expressions for text are already challenging, but SMARTS is even more complex because it deals with three-dimensional molecules.

Before I joined the group, the first version of volcalc had most of these functional groups figured out, but not all. I spent a significant amount of time trying to develop SMARTS strings to match additional molecules. Moving forward, I hope that if we implement new versions, we can get help from the community to refine these SMARTS strings, as there are likely people out there who are more skilled at it than I am.

The original project proposal mentions expanding volcalc to work with any chemical compound with a known structure. What are the key technical challenges you anticipate in achieving this goal?

Eric: This task turned out to be less difficult than I initially expected, but let me explain. In the original version of volcalc, before we received the R Consortium funding, the main function started with a KEGG ID—an identifier specific to the KEGG database. The function would download a MOL file, which is a text representation of a molecule corresponding to that ID. It would then identify and count the functional groups in the molecule, and finally, calculate the volatility based on those counts.

The major change we needed to implement to make volcalc more versatile was to decouple these steps. In the current version of volcalc, the functionality to download a MOL file from KEGG is still available, but it’s now separate from the main function that calculates volatility. This means that the inputs for calculating volatility can now be any MOL file, not just ones from KEGG. The file can come from any database, be exported from other software, or even be downloaded manually. Additionally, the tool now supports SMILES, which is another, simpler text-based representation of molecules.

There are various ways to represent chemicals in text, including another format called InChI. The Bioconductor packages we use, ChemmineR and ChemmineOB, have the ability to translate from InChI and other types of chemical representations. However, that feature isn’t available on Windows. So, I decided to keep volcalc focused on SMILES and MOL files. I believe that chemists and other researchers should be able to obtain data in one of these two formats, or use another tool to translate their data into these formats. I didn’t want to overload volcalc with the responsibility of being a chemical representation translator, as that didn’t seem like its primary purpose.

Can you walk us through the process of implementing the SIMPOL algorithm within the volcalcc package?

Kristina: The algorithm itself is fairly simple; it’s just basic math. You need to input some constants, the mass of the compound, and the counts of the functional groups we discussed earlier. Writing the code for this was straightforward and not particularly challenging.

Eric: Each functional group has a coefficient associated with it, which is multiplied by the number of times that group appears in the molecule. These values are then summed up, and the mass of the molecule is factored in as well. The challenging part wasn’t the algorithm itself, which is straightforward—just multiplying by coefficients and adding them up. The real difficulty was interpreting what the authors of the algorithm meant by each of the functional groups. Some were oddly specific, like how the hydroxyl group that is part of a nitrophenol group isn’t supposed to count toward the total number of hydroxyl groups. I spent a lot of time poring over the paper, particularly one table, to fully understand how they defined each group. That interpretation was the hardest part.

What future functionalities or expansions do you see as crucial for volcalc, especially in the context of evolving research needs in chemoinformatics?

Eric: Right now, we’re working on allowing users to specify different temperatures. The paper that describes the SIMPOL.1 method includes equations for how the coefficients of each functional group change with temperature. These changes aren’t always linear, and the contributions of functional groups can shift in importance as the temperature varies. This is an important feature to include because the version of volcalc we currently have uses coefficients calculated at 20°C, based on a table from the original paper. To accommodate other temperatures, we need to integrate another table that provides equations for calculating these coefficients based on temperature, and that’s what we’re working on.

Another key feature we want to leave room for in the future is the ability to add other methods for estimating volatility. SIMPOL.1 is just one type of group contribution method, but there are other approaches described in various papers that use different functional groups, equations, and coefficients. The basic idea remains the same: count the functional groups in a molecule, apply an equation, and estimate volatility. We’re trying to structure the code in a way that makes it easy to incorporate additional methods later, even if we don’t add them right away. I think these are the most important features we’re focusing on right now.

Kristina: We’re focused on the features I mentioned in the near future, but looking further ahead, I could see volcalc expanding to estimate other characteristics of compounds beyond just volatility. While I’m not a chemistry expert or a chemical ecologist, I imagine that those interested in volatility might also be interested in other compound characteristics that currently lack automated tools for estimation. So, it’s possible the package could evolve to include those features.

That said, one of the things I appreciate about the R package ecosystem is that it allows for specialized tools. Since anyone can build what they need, we don’t end up with massive, overly complex packages that try to do everything and become difficult to maintain. It might be better to keep volcalc focused and leave room for separate packages to handle additional functionality. This way, the tools remain manageable and easier to maintain in the long run.

How has it been working with the R Consortium? Would you recommend applying for an ISC grant to other R developers?

Kristina: The application process was straightforward, and I found the grant format to be very practical. It was focused on milestones and product development, which is refreshing compared to many academic research grants that tend to avoid specific deliverables. I highly recommend considering this grant. I believe people often overlook smaller funding sources, but even small amounts can make a big impact on the work you’re doing.

Eric: The first time I applied for an R Consortium grant was as a grad student, and I strongly encourage trainees to apply as well. It was a great experience for me because I could do it independently—my advisor wasn’t involved as one of the authors, and it wasn’t a complex process like applying for an NSF grant. It was straightforward and really rewarding. The only tricky part was figuring out the payment process, but that’s something people can work out.

I’ve noticed there seem to be fewer projects in recent years, and I don’t think it’s due to a lack of funding. It seems like fewer people are applying, which is why I especially encourage others to give it a shot. From what I’ve seen, there’s a very good chance of getting funded if you apply right now.

People should be creative and think broadly about how their project can benefit the broader R community. This doesn’t mean you need to develop the next big thing like R-Universe or CRAN. It can be something smaller, like a package that other R users will find helpful. For example, with our project, volcalc, our main goal was to encourage chemists—who usually use point-and-click software—to start using R. That was enough of a contribution to the R community to get funded. So, I really encourage people to think creatively about what “benefiting the R community” can mean.

About ISC Funded ProjectsA major goal of the R Consortium is to strengthen and improve the infrastructure supporting the R Ecosystem. We seek to accomplish this by funding projects that will improve both technical infrastructure and social infrastructure.

Learn moreThe post Unlocking Chemical Volatility: How the volcalc R Package is Streamlining Scientific Research appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Unlocking Chemical Volatility: How the volcalc R Package is Streamlining Scientific Research

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 544–548

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #544Wake up Cinderellas, we have some cleaning to do. To be exact we need to pick up numbers from random strings and then add them up. It shouldn’t be hard. What do you think?

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/544 Sum of First and Last Numbers.xlsx"input = read\_excel(path, range = "A1:A10")test = read\_excel(path, range = "B1:B2") Transformation result = input %>% mutate(numbers = map(Strings, str\_extract\_all, "\\d+")) %>% unnest(numbers) %>% unnest(numbers) %>% mutate(rn = row\_number(), min = min(rn), max = max(rn), .by = Strings) %>% filter(rn == min | rn == max) %>% summarise(sum = sum(as.numeric(numbers))) Validation identical(result$sum, test$`Answer Expected`)# [1] TRUE Puzzle #545Game in this riddle is weird, but I imagined it can be some kind of group scissors-paper-rock, where there is more than 2 players, and a way to log a match is to write down winners with capital letters and losers with lower case. And we need to check what is the point balance for each player, assuming that each capital letter is plus 1, and lower case is minus 1. Let’s count them down.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/545 Ranking the Players.xlsx"input = read\_excel(path, range = "A2:A11")test = read\_excel(path, range = "C2:E8") Transformation result = input %>% separate\_rows(`Match Results`, sep = "") %>% filter(`Match Results` != "") %>% mutate(Players = str\_to\_lower(`Match Results`), case\_point = if\_else(`Match Results` == toupper(`Match Results`), 1, -1)) %>% summarise(Points = sum(case\_point), .by = Players) %>% mutate(Rank = dense\_rank(desc(Points)) %>% as.numeric()) %>% arrange(Rank, Players) Validation identical(result, test)# [1] TRUE Puzzle #546We have some matrices of digits today, and we need to scan all horizontal and vertical lines. If we meet situation that digits forms odd number concatenated, we need to write it down. Does it sound hard? Not really, but lets try to make it dynamical, scalable and short in code at the same moment.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/546 Pick the Odd Numbers in a Grid.xlsx"input1 = read\_excel(path, range = "A2:B3", col\_names = FALSE) %>% as.matrix()test1 = read\_excel(path, range = "G2:G2", col\_names = FALSE) %>% pull()input2 = read\_excel(path, range = "A5:C7", col\_names = FALSE) %>% as.matrix()test2 = read\_excel(path, range = "G5:G5", col\_names = FALSE) %>% pull()input3 = read\_excel(path, range = "A9:C11", col\_names = FALSE) %>% as.matrix()test3 = read\_excel(path, range = "G9:G9", col\_names = FALSE) %>% pull()input4 = read\_excel(path, range = "A13:D16", col\_names = FALSE) %>% as.matrix()test4 = read\_excel(path, range = "G13:G13", col\_names = FALSE) %>% pull()input5 = read\_excel(path, range = "A18:E22", col\_names = FALSE) %>% as.matrix()test5 = read\_excel(path, range = "G18:G18", col\_names = FALSE) %>% pull() Transformation pick\_odds <- function(M) { all <- as.numeric(apply(rbind(M, t(M)), 1, paste0, collapse = "")) paste(all[all %% 2 == 1], collapse = ", ")} Validation all.equal(pick\_odds(input1), test1) # TRUEall.equal(pick\_odds(input2), test2) # TRUEall.equal(pick\_odds(input3), test3) # could not pull "no cells", byt result of function is also emptyall.equal(pick\_odds(input4), test4) # TRUEall.equal(pick\_odds(input5), test5) # TRUE Puzzle #547We met triangular numbers quite a few times before, but we have “hyper” triangular number today. Just to remind what triangular number is, imagine pool balls or bowling pins. At the beginning of the game they are forming triangle. Each triangular number if taken as set of physical objects (coins, bowling pins etc.) can be formed into nice equilateral triangle.

Today we have to find pairs of triangular numbers for which both sum and difference are also triangular. We need to find first 20 such pairs. I will show you two of many possible ways to achieve the goal. First correct, but showing another set of numbers, and second made to meet exact result as given. It differs in number generation method. In first I took all triangular numbers below certain numbers, in second only first 2000 elements. Check it out.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/547 Sum and Diff both Triangular Numbers.xlsx"test = read\_excel(path, range = "A1:A21") Transformation v1 generate\_triangular\_numbers <- function(limit) { n <- 1 triangulars <- c() while (TRUE) { t\_n <- n * (n + 1) / 2 if (t\_n > limit) break triangulars <- c(triangulars, t\_n) n <- n + 1 } return(triangulars)}is\_triangular\_number <- function(n) { return((sqrt(8 * n + 1) - 1) %% 2 == 0)}tr = generate\_triangular\_numbers(1000000)tr\_pairs = expand.grid(tr, tr) %>% as\_tibble() %>% filter(Var1 != Var2, Var1 < Var2, is\_triangular\_number(Var1 + Var2), is\_triangular\_number(abs(Var1 - Var2))) %>% arrange(Var1, Var2) %>% head(20) %>% unite("result", c(Var1, Var2), sep = ", ", remove = T)print(tr\_pairs) Transformation v2 generate\_triangular\_numbers <- function(n) { triangulars <- numeric(n) for (i in 1:n) { triangulars[i] <- i * (i + 1) / 2 } return(triangulars)}is\_triangular\_number <- function(n) { return((sqrt(8 * n + 1) - 1) %% 2 == 0)}tr = generate\_triangular\_numbers(2000)tr\_pairs = expand.grid(tr, tr) %>% as\_tibble() %>% filter(Var1 != Var2, Var1 < Var2, is\_triangular\_number(Var1 + Var2), is\_triangular\_number(abs(Var1 - Var2))) %>% arrange(Var1, Var2) %>% head(20) %>% unite("result", c(Var1, Var2), sep = ", ", remove = T) Validation (for version v2) identical(test$`Answer Expected`, tr\_pairs$result)#> [1] TRUE Puzzle #548It looks like we have Morse code today, but not it only looks like it from certain distance. Tap Code Cypher is based on grid of numbers. If we need to code something we need to tap coordinates of each letter separately. Pretty nice. Let’s do it.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/548 Tap Code Cipher.xlsx"input1 = read\_excel(path, range = "A1:F6")input2 = read\_excel(path, range = "H1:H10")test = read\_excel(path, range = "I1:I10") Transformation coding\_table = input1 %>% pivot\_longer(-1, names\_to = "letter", values\_to = "code") %>% select(row = 1, col = 2, letter = 3) %>% separate\_rows(letter, sep = "/") %>% mutate(letter = str\_to\_lower(letter))encrypt = function(word) { characters = str\_split(word, "") %>% unlist() coord = map\_dfr(characters, ~{ row = coding\_table %>% filter(letter == .x) %>% pull(row) col = coding\_table %>% filter(letter == .x) %>% pull(col) tibble(row = row, col = col) }) %>% unite("coord", row, col, sep = " ") %>% pull(coord) %>% paste(collapse = " ") coord = str\_split(coord, " ") %>% unlist() %>% map\_dfr(~{ dots = str\_c(rep(".", .x), collapse = "") tibble(dots = dots) }) %>% pull(dots) %>% paste(collapse = " ") return(coord)}result = input2 %>% mutate(`Answer Expected` = map\_chr(Words, encrypt)) Validation identical(test$`Answer Expected`, result$`Answer Expected`)#> [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
On my Github repo there are also solutions for the same puzzles in Python. Check it out!

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWelcome to the world of R programming, where data manipulation is a crucial skill. One common task you may encounter is the need to switch two columns in a data frame. Understanding how to efficiently rearrange data can significantly enhance your data analysis workflow. This guide will walk you through the process of switching columns using Base R, with multiple examples to help you master this essential task.

Understanding Data Frames in RWhat is a Data Frame?A data frame in R is a table or a two-dimensional array-like structure that holds data. It is similar to a spreadsheet or SQL table and is used to store data in rows and columns. Each column in a data frame can have data of different types.

Basic Operations with Data FramesBefore diving into switching columns, it’s important to familiarize yourself with basic operations. You can create data frames using the data.frame() function, access columns using the $ operator, and perform operations like filtering and sorting.

Why Switch Columns?Common Scenarios for Switching ColumnsSwitching columns is often needed when preparing data for analysis. For example, you might want to reorder columns for better visualization or to follow the requirements of a specific analysis tool.

Benefits of Rearranging DataRearranging columns can make data more intuitive and easier to interpret. It can also help in aligning data with documentation or standards that require a specific column order.

Basic Method to Switch Columns in Base RUsing Indexing to Switch ColumnsOne of the simplest ways to switch columns in Base R is through indexing. You can rearrange columns by specifying their order in a new data frame.

```

Example: Swapping two columns by indexdata <- data.frame(A = 1:5, B = 6:10, C = 11:15)data

```

A B C1 1 6 112 2 7 123 3 8 134 4 9 145 5 10 15

data <- data[c(1, 3, 2)]data

A C B1 1 11 62 2 12 73 3 13 84 4 14 95 5 15 10 In this example, columns B and C are swapped by reordering their indices.

Switching Columns by NameUsing Column Names for SwitchingAnother approach is to use column names to switch their positions. This method is useful when you are unsure of the column indices or when working with large data frames.

```

Example: Swapping columns by namedata <- data.frame(A = 1:5, B = 6:10, C = 11:15)data

```

A B C1 1 6 112 2 7 123 3 8 134 4 9 145 5 10 15

data <- data[c("A", "C", "B")]data

A C B1 1 11 62 2 12 73 3 13 84 4 14 95 5 15 10 This method swaps columns B and C by specifying their names directly.

Advanced Techniques for Column SwitchingUsing the subset() FunctionThe subset() function can be employed for advanced column switching, especially when combined with logical conditions.

```

Example: Advanced column swappingdata <- data.frame(A = 1:5, B = 6:10, C = 11:15)data

```

A B C1 1 6 112 2 7 123 3 8 134 4 9 145 5 10 15

data <- subset(data, select = c(A, C, B))data

A C B1 1 11 62 2 12 73 3 13 84 4 14 95 5 15 10 Handling Large Data FramesPerformance ConsiderationsWhen dealing with large data frames, performance becomes a concern. Efficient column switching can help reduce computation time and system memory usage.

Efficient Column Switching TechniquesFor large datasets, consider using in-place operations or packages like data.table that offer optimized data manipulation functions.

Common Mistakes and How to Avoid ThemIndexing ErrorsA common mistake is incorrect indexing, which can lead to unexpected results. Always double-check the indices or names you use.

Name MismatchesEnsure that column names are spelled correctly. Even a small typo can cause errors or incorrect data manipulation.

Practical ExamplesExample 1: Switching Columns in a Small Data Frame small\_data <- data.frame(X = 1:3, Y = 4:6, Z = 7:9)small\_data

X Y Z1 1 4 72 2 5 83 3 6 9

small\_data <- small\_data[c("Z", "Y", "X")]small\_data

Z Y X1 7 4 12 8 5 23 9 6 3 Example 2: Switching Columns in a Large Data FrameFor larger datasets, consider using efficient indexing or parallel processing if supported by your environment.

Using dplyr for Column SwitchingIntroduction to dplyrThe dplyr package in R provides a powerful set of tools for data manipulation, including functions to change column positions.

Example: Using relocate() Function library(dplyr)data <- data.frame(A = 1:5, B = 6:10, C = 11:15)data

A B C1 1 6 112 2 7 123 3 8 134 4 9 145 5 10 15

data <- data %>% relocate(C, .before = B)data

A C B1 1 11 62 2 12 73 3 13 84 4 14 95 5 15 10 Comparing Base R and dplyr ApproachesPros and Cons of Each Method Base R: No additional packages needed, but can be less intuitive for complex operations. * dplyr*: More readable and concise, but requires installing and loading the package.

When to Use Base R vs. dplyrUse Base R for simple tasks or when package installation is not an option. Opt for dplyr for larger projects requiring more advanced data manipulation.

FAQsHow to Switch Multiple Columns at Once?Use indexing or dplyr functions to reorder multiple columns simultaneously.

Can I Switch Non-Adjacent Columns?Yes, specify the desired order using indices or names, regardless of their original positions.

What if Columns Have the Same Name?R does not allow duplicate column names. Ensure each column has a unique name before switching.

How to Switch Columns in a List?Convert the list to a data frame, switch columns, and convert back if needed.

Is It Possible to Switch Rows Instead of Columns?Yes, you can use similar indexing techniques to manipulate rows.

Quick Takeaways* Switching columns in R is simple with indexing or dplyr. * Always validate your column order before and after switching. * Choose the method that best fits your data size and manipulation needs.

ConclusionSwitching columns in R is a fundamental skill for data manipulation. Whether using Base R or dplyr, understanding these techniques enhances your ability to organize and analyze data effectively. Practice with different datasets, and don’t hesitate to explore further learning resources.

Your Turn!We hope you found this guide helpful! Please share your feedback and feel free to share this article with fellow R enthusiasts.

References1. Introduction to R Data Frames 2. dplyr Documentation 3. Efficient Data Manipulation in R

Happy Coding!

Swapping Columns To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Switch Two Columns in R: A Beginner’s Guide

View Details

[This article was first published on Epiverse-TRACE developer space, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Licenses are an important topic within open source. Without licenses, information or code can be publicly available but not legally available for reuse or redistribution. The open source software community’s most common licenses are the MIT license or the GNU GPLv3.

When you read the MIT or GNU license, you can see they are rather specific:

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”) [MIT License]

and

The GNU General Public License is a free, copyleft license for software and other kinds of works.

They aim to cover primarily software, not other forms of information such as, for example, data.

Licensing only code in R packagesGiven the importance of a license for redistribution, CRAN requires R packages to contain an open-source license.1 As one of the most important distributors for R packages historically, this rule has become the de facto standard.

An R package is licensed appropriately when the license is stated in the metadata DESCRIPTION file and, if necessary, in the LICENSE file. We recommend including the LICENSE file, with the license year and exact copyright holder (for example, an organization or a specific individual). Some R package developers choose to include a copy of the license but this is not bundled in the R package when it is built. You can find an example of a DESCRIPTION file here and find an example LICENSE here.

Licensing only data in R packagesIt can be the case that an R package is primarily used to bundle and share a data set, for example to allow a user to easily download the data and load it into R. For example, gapminder and palmerpenguins. As an aside, some in the R developer community2 dissuade the use of R packages used primarily for data, often referred to as “data packages”, and instead advocate to host data online and use the several API packages available in R to access the data.

Software licenses do not apply to data. One piece of data cannot be copyrighted, as facts cannot be copyrighted, but a collection of data in a database can come with rights. In Europe, UK, and Russia, databases can have rights if they are a substantial and original piece of work.3 This is called the “sui generis” database right. The USA and Brazil do not recognize this database right. Data compiled from various databases that themselves are copyrighted would need to follow the licenses of those respective databases.

Data need licenses such as the Public Domain Dedication or the Creative Commons Attribution to maximize redistribution. Such general licenses help minimize differences among countries that do and do not recognize database rights. Given that any copyright limitations (even attribution) mean they only apply when the database right is recognized, we recommend the most minimal option: The Public Domain Dedication. This levels the playing field for reuse and redistribution, no matter the jurisdiction.

Licensing code and data in one R packageBut what to do if your R package has both code and data as primary objects of (roughly) equal importance?

In these cases a software license inadequately covers the data, and a data license inadequately covers the code. Dual licensing can help resolve this issue. This means there is one license for code (for example, MIT license) and another license for the included data (for example, Public Domain Dedication).

After conducting an online search, dual licensing for R packages seems rare. An interesting example of dual licensing is the igraphdata package, which contains several licenses: One for each dataset included in the package. Similar to igraphdata, in our own epiparameter package we dual licensed the code and data as well. We licensed the code using the DESCRIPTION file and used the LICENSE file to license the data under CC0. Concretely, we include this additional text in LICENSE to clarify the dual license and that we recommend citing the original source regardless:

All data included in the epiparameter R package is licensed under CC0 (https://creativecommons.org/publicdomain/zero/1.0/legalcode.txt). This includes the parameter database (extdata/parameters.json) and data in the data/ folder. Please cite the individual parameter entries in the database when used.

When including data in your R package from other sources it is important to check that the license of your package and the data is compatible4, or that the individual data license is clearly stated, as in igraphdata. For epiparameter, we consider model estimates as facts (that is, not copyrightable).

This blogpost helps explain a pattern of dual licensing rarely seen in the wild, and how to implement it in a CRAN conformant manner. By sharing this, we hope that those cases where people want to license both code and data, have a resource we wish we had while exploring this topic. Whether or not data packages and combined code + data packages should exist, is a question for another day.

Footnotes1. For a full list of license accepted by CRAN see: https://svn.r-project.org/R/trunk/share/licenses/license.db and they also accept stating the license as “Unlimited” for unrestricted distribution.︎ 2. A discussion of data and R packages can be found here: https://github.com/ropensci/unconf17/issues/61. This thread is used as an example of some thoughts on packaging data in R but we acknowledge it is from 2017 so the opinions of the individuals in this thread may have changed.︎ 3. To see the legal definition of the database right in Europe, and what constitutes it, see the European Union Directive 96/9/EC︎ 4. See this blog post by Julia Silge on including external data sets into an R package and rectifying incompatibilities with license︎

ReuseCC BY 4.0CitationBibTeX citation: @online{lambert2024, author = {Lambert, Joshua and Hartgerink, Chris}, title = {Dual Licensing {R} Packages with Code and Data}, date = {2024-09-23}, url = {https://epiverse-trace.github.io/posts/data-licensing.html}, langid = {en}} For attribution, please cite this work as:Lambert, Joshua, and Chris Hartgerink. 2024. “Dual Licensing RPackages with Code and Data.” September 23, 2024. https://epiverse-trace.github.io/posts/data-licensing.html. To leave a comment for the author, please follow the link and comment on their blog: Epiverse-TRACE developer space.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Dual licensing R packages with code and data

View Details

[This article was first published on Achim Zeileis, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The recently-proposed extended-support beta regression model in R package betareg is illustrated by simultaneously modeling the occurrence and extent of loss aversion in a behavioral economics experiment.

MotivationTo illustrate the benefits of extended-support beta regression models, suggested in a recent arXiv paper with Ioannis Kosmidis, we revisit the analysis of a behavioral economics experiment conducted and published by Glätzle-Rützler et al. (2015, Journal of Economic Behavior & Organization, doi:10.1016/j.jebo.2014.12.021). The outcome variable is the proportion of tokens invested by high-school students in a risky lottery with positive expected payouts. Glätzle-Rützler et al. focused on the effects of several experimental factors on the mean investments, which reflect the players’ willingness to take risks. In their study they employed linear regression models, estimated by ordinary least squares (OLS) with standard errors adjusted for potential clustering and heteroscedasticity.

Here, we extend the analysis from Glätzle-Rützler et al. by employing a similar model for the mean investments but additionally exploring distributional specifications that allow for a probabilistic, rather than mean-only, interpretation of the effects. From an economic perspective this is of interest because it allows to interpret both the mean willingness to take risks in this experiment, and the probability to behave like a rational Homo oeconomicus, who would invest (almost) all tokens in this lottery because it has positive expected payouts.

The full replication code for the analyses from the arXiv paper is available in lossaversion.R with some auxiliary functions in beta01.R. Below we only provide the most important R snippets to provide a feeling for the workflow in R. The rest of the discussion here highlights the main insights from the analysis.

An aggregated version of the data from all nine rounds of the experiment is available as LossAversion in the betareg package. Interest is in linking the variable invest with the proportion of the total tokens invested in all nine rounds to explanatory information:

grade: Is the player from lower grades 6-8 or upper grades 10-12?
arrangement: Is the player an individual or a team of two?
male: Is (at least one of) the player(s) male?
age: (Average) age of the player(s) in years.

ModelsWe compare four different models for invest which all employ the same equation for the mean submodel.And all except the OLS reference model employ the main effects of the three experimental factors for the dispersion submodel.

Normal linear model (N) with constant variance, corresponding to the OLS approach from the original study. In R, this can be fitted with the base lm() or glm() function.

data("LossAversion", package = "betareg")la\_ols <- glm(invest ~ grade * (arrangement + age) + male, data = LossAversion)summary(la\_ols) 2. Heteroscedastic censored normal model (CN), also known as heteroscedastic two-limit tobit model in econometrics. This can be fitted with the crch package (for censored regression with conditional heteroscedasticity).

library("crch")la\_htobit <- crch(invest ~ grade * (arrangement + age) + male | arrangement + male + grade, data = LossAversion, left = 0, right = 1)summary(la\_htobit) 3. Beta regression (B) after ad-hoc scaling of the investments to the open unit interval (to avoid the boundary observations). This can be fitted with the betareg package.

library("betareg")LossAversion$invests <- (LossAversion$invest * (nrow(LossAversion) - 1) + 0.5)/ nrow(LossAversion)la\_beta <- betareg(invests ~ grade * (arrangement + age) + male | arrangement + male + grade, data = LossAversion)summary(la\_beta) 4. Extended-support beta mixture model (XBX) with the same specification as B but adding an extra exceedance parameter to be estimated (instead of the ad-hoc scaling). This can also be fitted with betareg since version 3.2-0 with XBX regression being automatically selected in case of boundary observations in the response.

la\_xbx <- betareg(invest ~ grade * (arrangement + age) + male | arrangement + male + grade, data = LossAversion)summary(la\_xbx)

If you run the code and compare the model summaries, note that the coefficients from N and CN use an identity link for the mean parameter whereas B and XBX use a logit link. In addition, the log-likelihood, and, hence, AIC and BIC, are comparable only between CN and XBX because those two models have the same support for the response variable, that is the unit interval with point masses at 0 and 1. See also the accompanying arXiv paper for the full summary tables. By and large, the mean parameters for the N and CN models are rather similar and those for B and XBX are rather similar, only with some differences that occur due to using models with point masses on the boundaries (CN and XBX) or not (N and B).

However, instead of studying the individual estimated coefficients in more detail we rather assess the models graphically by visualizing their goodness of fit and different types of fitted effects.

Goodness of fitTo illustrate how different the fitted probability distributions of the four models are, we employ so-called hanging rootograms. These compare the empirical marginal distributions of the response variable (proportion of tokens invested) to the aggregated fitted distributions from the models. The quality of the fit can be judged by the deviation of the hanging bars from the zero reference line.

An object-oriented implementation of rootograms is available, along with other tools for working with probabilistic models, in the topmodels package on R-Forge (hopefully soon to be submitted to CRAN). You can install it from R-Forge or R-universe and then create the rootograms for the four models:

install.packages("topmodels", repos = "https://zeileis.R-universe.dev")library("topmodels")rootogram(la\_ols, breaks = -6:16 / 10, main = "N")rootogram(la\_htobit, main = "CN")rootogram(la\_beta, main = "B")rootogram(la\_xbx, main = "XBX") A more refined version of the plots is shown below. See the full replication script linked above for the code details.

The square root of the expected frequencies are shown as red dots and the square root of the observed frequencies are hanging from the points as gray bars. The dashed lines are the Tukey warning limits at +/- 1. These plots show that models N and B fit poorly in the tails. In contrast, models CN and XBX fit very well with almost all bars hanging close to the zero reference line. The fit for XBX appears to be slightly better than for CN.

EffectsWhile models XBX and CN provide a much better probabilistic fit than their uncensored counterparts B and N, it turns out that the predicted mean investments from all four models are still very similar. But XBX and CN allow for interpretations beyond the mean, including economically relevant interpretations of probability effects.

For illustration, we focus on the team arrangement effect for a subsample with a large share of very rational subjects: male players or teams with at least one male, in grades 10-12, and between 15 and 17 years of age. The figure below shows the estimated arrangement effect for the mean E(Y), i.e., the expected proportion of tokens invested, and for the probability to behave very rationally and invest almost everything, i.e., P(Y > 0.95). The empirical quantities are shown in black for the subsample between 15 and 17 years of age while the model-based effects are shown at an age of 16.

The graphic shows that all models do a reasonable job in estimating E(Y) but the censored models XBX and CN are much better at estimating the probability to behave very rationally. Similarly, it could be shown that the fit for the probability P(Y < 0.05) is also much better for XBX and CN than for B and N but this is not done here, because that probability does not have such an appealing economic interpretation like P(Y > 0.95).

For obtaining the model-based effects, as shown above, the procast() function (for probabilistic forecasts) from the topmodels package can be used. This is again an object-oriented implementation that facilitates obtaining not only moments (such as means and variances) but also entire probability distributions (as S3 objects) and corresponding probabilities, densities, and quantiles.

Here, we only briefly show the code for the fitted XBX model but the same function calls can be applied to the other fitted model objects. First, we set up the new data that only varies arrangement from single to team but keeps all other variables fixed. Then, both kinds of effects are computed with procast().

la\_nd <- data.frame(arrangement = c("single", "team"), male = "yes", age = 16, grade = "10-12")procast(la\_xbx, newdata = la\_nd, type = "mean")## mean## 1 0.4713## 2 0.6861procast(la\_xbx, newdata = la\_nd, type = "cdf", at = 0.95, lower.tail = FALSE)## probability## 1 0.07161## 2 0.18501 Thus, the mean invested proportion goes up from 47.1% to 68.6% for teams vs. single players in this setting, while the probability to behave almost fully rationally increases from 7.2% to 18.5%.

Again, the full code for creating the figure and underlying table is provided in the replication script linked above.

The script also inlucdes some further illustrations, e.g., the comparison with three-part hurdle models for “zero-and-one-inflated” beta regression. However, these models do not work well here: unappealing interpretation, too many parameters, quasi-complete separation of boundary and non-boundary observations. Hence, we do not show the details in this post.

To leave a comment for the author, please follow the link and comment on their blog: Achim Zeileis.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Modeling loss aversion with extended-support beta regression

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.When developing R Shiny apps, making sure they work well and are reliable is important. Rigorous testing isn’t just about finding bugs; it’s about preventing them, saving time, and ensuring everything runs smoothly.

We did the comparison of shinytest2 and Cypress. Check out this blog post to learn more.

Testing checks that every part of the app meets requirements and performs well. It also makes the code easier to maintain and reuse. Testing is crucial for security, helping to find and fix vulnerabilities and fortifying defenses against potential breaches.

In essence, testing is an essential part of Shiny app development, bringing together quality, reliability, and security to create great user experiences.

There are mainly 3 approach to tests:

Unit tests: These are used to test that functions behave as expected. {testthat} is the most popular R package used for this purpose.
Server function tests: These tests run the server function of a Shiny application in an environment that simulates a real client session. They can be used to test reactive components and outputs in the server function of a Shiny application, or a module. The testServer() function that comes with the shiny package is used for this purpose.
Snapshot-based tests: These tests are performed with the {shinytest2} package, which runs the Shiny application in a headless web browser. The web browser connects to the Shiny application and simulates user actions, such as clicking on buttons and setting inputs to particular values. It takes snapshots of the application state and, in future runs of the tests, compares the application state to those saved snapshots.

For unit tests and server function tests, no web browser is involved, and the tests and test expectations are expressed in R code. This means that the tests run quickly and that changing code in one part of an application will generally not affect tests of another part of the application.

The snapshot-based tests require a headless web browser, and, as the name suggests, use snapshots.

Overview of {shinytest2}{shinytest2} uses {testthat} ‘s snapshot-based testing strategy, a technique where the app’s state is captured during tests and compared to previously saved snapshots.

Primary uses of shinytest2 include:

Automated Testing: It allows developers to automate the testing process for their Shiny applications, ensuring that all functionalities work as expected across different scenarios and inputs.
Regression Testing: shinytest2 enables regression testing, which involves running tests automatically to ensure that new changes or updates to the application do not introduce bugs or issues that were not present before. This is done using record_test().
User Interaction Testing: Developers can simulate user interactions with the Shiny app, such as clicking buttons, entering text, and navigating through different pages, to validate the behaviour of the application under various conditions.
Integration with Continuous Integration (CI) Pipelines: Snapshot tests can be integrated into continuous integration pipelines, allowing developers to automatically run them as part of the build and deployment process. This ensures that visual regressions are caught early and prevents them from reaching production.

The benefits of using shinytest2 include:

Automated testing helps developers identify and fix bugs early in the development process, leading to higher code quality and more reliable applications.
Automating testing also saves time compared to manual testing, especially for complex Shiny applications with numerous features and interactions.
shinytest2 helps in detecting regressions quickly by running tests automatically whenever changes are made to the codebase, preventing the introduction of unintended bugs.
With the comprehensive test coverage, developers gain confidence in the reliability and stability of their Shiny applications, facilitating smoother deployments and updates.

Example of a simple app, placed in demo/app.R:

library(shiny)ui <- fluidPage( textInput("name", "What is your name?"), actionButton("greet", "Greet"), textOutput("greeting"))server <- function(input, output, session) { output$greeting <- renderText({ req(input$greet) paste0("Hello ", isolate(input$name), "!") })}shinyApp(ui, server) To test the app, load {shinytest2}, and call record_test() with the path to the app:

library(shinytest2)record\_test("demo/") The above code launches a special Shiny application which displays the Shiny app to be tested, on the left side and some recorder controls on the right sidebar. Then, you interact with the app and “expect values”. This records your clicks and inputs and these values are saved to the snapshot files. Once a recording is completed, it will create or append a new shinytest2 test to the test_file(). The next time you run the test, these values serve as the single source of truth. The snapshot serves as a comparison to what the application displays later.

Diving into shiny::testServerWhile {shinytest2} tests your entire application, the testServer() function offers a streamlined approach to testing server functions and modules within your shiny applications, without the need to execute the entire application. This targeted testing capability enhances efficiency and reliability compared to testing methods used by {shinytest2}, which simulate complete sessions including a headless web browser.

Moreover, integrating testServer() with unit testing frameworks like {testthat} further enhances its utility.

Here is a server function for a simple Shiny application:

server <- function(input, output, session) { get\_square <- reactive({ input$x ^ 2 }) output$txt <- renderText({ paste0("I am ", get\_square()) })} Now, we build up a collection of tests that we can run against our server to confirm that it always behaves correctly. For this, we use the {testthat} framework for its expect_ functions.

```

Bring in testthat for its expectationslibrary(testthat)testServer(server, { session$setInputs(x = 1) expect_equal(myreactive(), 1) expect_equal(output$txt, "I am 1") session$setInputs(x = 2) expect_equal(myreactive(), 4) expect_equal(output$txt, "I am 4")})

``` Code explanation:

The test expression provided here assumes the existence of the variables – input called x, output called txt, and reactive called myreactive(). The objects from inside the server function environment will be made available in the environment of the test expression.
By default, the values to any inputs are NULL. We set values by using the session$setInputs() method.
Lastly, we read the output$txt to check its value. When running inside testServer(), you can simply reference an output and it will give the value produced by the render function.

Note that the ‘session’ object used in testServer() differs from the real session object Shiny uses. This helps to tailor it to be more suitable for testing purposes by modifying or creating new methods such as setInputs().

Interested in learning how to conduct user tests for Shiny applications? Learn more in this presentation.

Testing Shiny ModulestestServer() can also be used for testing the server functions for Shiny modules.

Example of a server code for a module:

myModule <- function(id) { moduleServer(id, function(input, output, session) { get\_square <- reactive({ input$x ^ 2 }) output$txt <- renderText({ paste0("I am ", get\_square()) }) })} The module can be tested in the following way:

testServer(myModule, { session$setInputs(x = 1) expect\_equal(myreactive(), 2)}) If the module support additional parameter, let’s say flag, simply add additional param in args as list:

testServer(myModule2, args = list(flag = TRUE), { session$setInputs(x = 1) expect\_equal(myreactive(), 3)}) If the module returns some value, returned() can be used:

testServer(myModule2, args = list(flag = TRUE), { ... expect\_equal(session$returned(), 3) ...}) The Differences and Similarities between shiny::testServer() and shinytest2The major differences between testing with shinytest2 and shiny::testServer() lie in their approaches, scope, and functionality:

Scope of Testing:
shinytest2: shinytest2 focuses on end-to-end testing of Shiny applications, simulating user interactions with the app’s UI and validating its behavior across different scenarios. It captures snapshots of the entire application state and compares them to detect regressions.
shiny::testServer(): shiny::testServer() is used for unit testing of server-side code within Shiny applications. It allows developers to test individual server functions and modules in isolation, without the need to run the full Shiny application.
Granularity:
shinytest2: Provides a high-level testing approach by interacting with the application as a whole, including the UI elements and user inputs. It offers a comprehensive view of the application’s behavior but may be slower due to simulating the entire session.
shiny::testServer(): Focuses on testing specific server-side functionalities at a granular level. It enables developers to target and test individual server functions or modules directly, leading to faster and more focused testing.
Testing Environment:
shinytest2: Typically involves running tests in a headless web browser environment, simulating user interactions with the application through a virtual browser session.
shiny::testServer(): Runs tests directly within the R environment, without the need for a web browser. It executes server-side code and verifies the output directly within the R session.
Dependencies:
shinytest2: Relies on external packages like {shinytest} and may require additional setup for integration with continuous integration (CI) pipelines or test automation frameworks.
shiny::testServer(): Built-in function provided by the Shiny package, requiring minimal dependencies and setup. It seamlessly integrates with existing testing frameworks like {testthat}.
Use Cases:
shinytest2: Suitable for end-to-end testing, regression testing, and validation of the entire Shiny application, including its UI/UX aspects.
shiny::testServer(): Ideal for unit testing and validating the logic and behavior of server-side code within Shiny applications, facilitating faster development iterations and bug detection.

‍

While {shinytest2}and shiny::testServer() differ in their approaches and scopes of testing, they also share some commonalities in their testing objectives and methodologies:

Testing within R Environment:

Both {shinytest2}and shiny::testServer() enable testing within the R environment, allowing developers to write and execute tests directly within their R scripts or RStudio sessions.

Support for Test Automation:

Both shinytest2 and shiny::testServer() can be integrated with test automation frameworks like {testthat}, enabling developers to automate the execution of tests and incorporate them into their continuous integration (CI) pipelines.

Facilitate Regression Testing:

Both testing methods support regression testing, allowing developers to detect and prevent regressions by running tests against previous versions of the application’s codebase and comparing the results.

Complementary Testing Approaches:

While they serve different purposes, shinytest2 and shiny::testServer() can be used in conjunction to achieve comprehensive testing coverage for Shiny applications. Developers may utilize shinytest2 for end-to-end testing and UI validation while employing shiny::testServer() for unit testing of server-side code.

We recently launched our resources page. Explore our ebooks, Shiny Gatherings and other materials curated by our experts.

Writing shiny::testServer() Tests in RhinoLet’s explore how we can write unit tests using shiny::testServer for a Shiny application developed using the Rhino framework. If you are new to rhino and require a step by step guide to create a Shiny application using Rhino, check out this blog.

Example code for the main.R file

box::use(shiny[NS, fluidPage, numericInput, textOutput, moduleServer, renderText, reactive])ui <- function(id) { ns <- NS(id) fluidPage( numericInput(inputId = ns("num"), value = 5, label = "Enter a value:"), textOutput(outputId = ns("squared")) )}server <- function(id) { moduleServer(id, function(input, output, session) { num\_square <- reactive({ input$num ^ 2 }) output$squared <- renderText({ num\_square() }) })} Now navigate to tests -> testthat -> test-main.R and paste the below code

box::use( shiny[testServer], testthat[expect\_equal, test\_that],)box::use( app/main[server, ui],)test\_that("main server works", { testServer(server, { session$setInputs(num = 2) expect\_equal(num\_square(), 4) expect\_equal(output$squared, "4") })}) Now run the following command in the console

rhino::test\_r() You should get a output as below if the tests run successfully

‍

If you change the expectations in tests, you get an error message as below:

In this example, we have set the expectation to “3”, whereas the output is “4”. So we get a output mismatch error.

Do you know – rhino comes with CI actions so you don’t need to setup CI file for testthat. We’ve got you covered.

Tips and Best PracticesOptimizing tests, avoiding common pitfalls, and integrating shiny::testServer() into existing testing frameworks are all crucial aspects of ensuring efficient and effective testing in Shiny applications. Here are some insights and strategies for each:

Optimizing Tests:1. Identify critical user workflows and prioritize testing on these paths to ensure that essential functionalities are thoroughly tested. 2. Break down tests into smaller, modular components to improve maintainability and reusability. This allows for easier debugging and enhances test scalability. 3. Minimize unnecessary setup and teardown steps in tests to reduce execution time. Only include the necessary setup and assertions required to validate the behavior being tested.

Common Pitfalls to Avoid1. Avoid creating tests that are overly dependent on specific implementation details, as they can become brittle and prone to failure when the underlying code changes. 2. Ensure comprehensive test coverage by testing all critical functionalities and edge cases. Incomplete test coverage may lead to undetected bugs and regressions. 3. Consider the performance implications of tests, especially in the context of end-to-end testing with shinytest2. Optimize tests to minimize execution time and resource usage, particularly for long-running tests. 4. Regularly review and update tests to ensure they remain relevant and effective as the codebase evolves. Neglecting test maintenance can lead to outdated or inaccurate tests that provide false assurance.

Integrating shiny::testServer() into existing testing frameworks:1. Ensure that the existing testing framework supports integration with shiny::testServer(). Popular testing frameworks like testthat are commonly used for this purpose. 2. Write test cases using the existing testing framework to cover server-side functionalities of the Shiny application. Each test case should focus on a specific server function or module. 3. Integrate the tests into the Continuous Integration (CI) pipeline to automatically run them on each code commit or build. This ensures that server-side functionalities are thoroughly tested as part of the development workflow.

ConclusionTesting is a cornerstone of robust and reliable Shiny application development, offering numerous benefits including bug detection, quality assurance, maintainability, and security enhancement. Understanding different testing methodologies is crucial for achieving these goals effectively. Here’s a recap of the key points emphasizing the value of this understanding:

Testing is essential for identifying and rectifying potential bugs and errors before they reach end-users. It acts as a safeguard against bugs, ensures quality assurance, enhances maintainability, and fortifies security.
There are primarily three types of tests for Shiny applications: unit tests, server function tests, and snapshot-based tests. Each type serves a specific purpose and contributes to comprehensive testing coverage.
Shinytest2 leverages snapshot testing to automate the testing process for Shiny applications. It facilitates automated testing, regression testing, user interaction testing, and integration with Continuous Integration (CI) pipelines.
shiny::testServer() offers a streamlined approach to testing server functions and modules within Shiny applications. It is ideal for unit testing server-side code and integrates seamlessly with existing testing frameworks like testthat.
shinytest2 and shiny::testServer() have differences in scope, granularity, testing environment, dependencies, and use cases. However, they also share commonalities in testing objectives and methodologies.

Understanding different testing methodologies empowers developers to build robust, high-quality Shiny applications that meet and exceed user expectations. By embracing testing as an integral part of the development lifecycle, developers pave the way for applications that are reliable, maintainable, and secure.

Did you find this blog post useful? Subscribe to get more helpful blog posts and other resources delivered straight to your inbox every week.

References1. https://shiny.posit.co/r/articles/improve/testing-overview/ 2. https://testthat.r-lib.org/ 3. https://rstudio.github.io/shinytest2/ 4. https://appsilon.github.io/rhino/

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Write Tests with shiny::testServer Similar to shinytest2

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn this post, we will explore systematic sampling in R using base R functions. Systematic sampling is a technique where you select every (k^{th}) element from a list or dataset. This method is straightforward and useful when you want a representative sample without the complexity of more advanced sampling techniques.

Let’s dive into an example to understand how it works.

What is Systematic Sampling?Systematic sampling involves selecting every (k^{th}) element from a dataset after a random start. The value of (k) is calculated as:

where (N) is the population size and (n) is the sample size.

Example: Sampling a DatasetImagine we have a dataset of 1000 elements, and we want to select a sample of 100 elements using systematic sampling.

Generate a Dataset

First, let’s create a dataset with 1000 elements.

set.seed(123) # Setting seed for reproducibility, although with this # example it doesn't matterpopulation <- 1:1000 Here, population is a sequence of numbers from 1 to 1000.

Define Sample Size

Define the number of elements you want to sample.

sample\_size <- 100 3. Calculate Interval (k)

Calculate the interval (k) as the ratio of the population size to the sample size.

k <- length(population) / sample\_size 4. Random Start Point

Choose a random starting point between 1 and (k).

start <- sample(1:k, 1) 5. Select Every (k^{th}) Element

Use a sequence to select every (k^{th}) element starting from the chosen start point.

systematic\_sample <- population[seq(start, length(population), by = k)] 6. Check the Sample

Print the first few elements of the sample to check.

head(systematic\_sample)

[1] 3 13 23 33 43 53 Here is the complete code in one block:

```

Step 1: Generate a Datasetset.seed(123) # Setting seed for reproducibilitypopulation <- 1:1000# Step 2: Define Sample Sizesample_size <- 100# Step 3: Calculate Interval kk <- length(population) / sample_size# Step 4: Random Start Pointstart <- sample(1:k, 1)# Step 5: Select Every k-th Elementsystematic_sample <- population[seq(start, length(population), by = k)]# Step 6: Check the Samplehead(systematic_sample)

``` Try It Yourself!Systematic sampling is a simple yet powerful technique. By following the steps above, you can apply it to your datasets. Experiment with different sample sizes and starting points to see how the samples vary. This method can be particularly useful when dealing with large datasets where random sampling might be cumbersome.

Give it a go and see how systematic sampling can be a handy tool in your data analysis toolkit!

Happy Coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Systematic Sampleing in R with Base R

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’ve always wanted to have a minimal unified interface to XGBoost, CatBoost, LightGBM and sklearn's GradientBoosting, without worrying about the different parameters names aliases. So, I had a lot of fun creating unifiedbooster (which is not part of Techtonique, but is a personal swiss knife tool, under the MIT License).

In unifiedbooster, there are 5 main common parameters for each algorithm:

n_estimators: maximum number of trees that can be built
learning_rate: shrinkage rate; used for reducing the gradient step
max_depth: maximum tree depth
rowsample: subsample ratio of the training instances
colsample: percentage of features to use at each node split

In many situations, these are enough for obtaining robust “baselines” (and the whole documentation can be found here). Additional parameters can be provided thanks to the **kwargs (even though that’s not the main philosophy of the tool).

I present a Python version and an R version.

Python version !pip install unifiedbooster There are many ways to calibrate the boosters, which all rely on GPopt. I’ll present only one today (the other ones in a few weeks): Bayesian optimization.

import unifiedbooster as ubfrom sklearn.datasets import load\_iris, load\_breast\_cancer, load\_winefrom sklearn.model\_selection import train\_test\_splitfrom sklearn.linear\_model import ElasticNetCVfrom sklearn.kernel\_ridge import KernelRidgefrom sklearn.metrics import f1\_score, accuracy\_score, precision\_score, recall\_scorefrom time import timedataset = load\_breast\_cancer()X, y = dataset.data, dataset.target # data setX\_train, X\_test, y\_train, y\_test = train\_test\_split( X, y, test\_size=0.2, random\_state=42) # split data into training set and test set# Find 'good' hyperparameters for LightGBM# Obtain 'best' model's performance on test setres = ub.cross\_val\_optim(X\_train=X\_train, y\_train=y\_train, X\_test=X\_test, y\_test=y\_test, model\_type="lightgbm", # or 'lightgbm', 'gradientboosting', 'catboost' type\_fit="classification", scoring="accuracy", n\_estimators=250, cv=5, # numbers of folds in cross-validation verbose=1, seed=123)print(res) Creating initial design... ...Done. Optimization loop... 190/190 [██████████████████████████████] - 45s 237ms/stepresult(best\_params={'learning\_rate': 0.9611431739764045, 'max\_depth': 1, 'rowsample': 0.597564697265625, 'colsample': 0.508392333984375, 'model\_type': 'lightgbm', 'n\_estimators': 250}, best\_score=-0.9780219780219781, test\_accuracy=0.9736842105263158) How do we verify what we’ve just did?

```

Initialize the unified clf clf = ub.GBDTClassifier(**res.best_params)# Fit the modelclf.fit(X_train, y_train)# Predict on the test sety_pred = clf.predict(X_test)# Evaluate model's accuracy on test setprint(accuracy_score(y_test, y_pred))0.9736842105263158

``` Classification report

from sklearn.metrics import classification\_reportprint(classification\_report(y\_test, y\_pred)) precision recall f1-score support 0 0.98 0.95 0.96 43 1 0.97 0.99 0.98 71 accuracy 0.97 114 macro avg 0.97 0.97 0.97 114weighted avg 0.97 0.97 0.97 114 Confusion matrix

import seaborn as snsimport matplotlib.pyplot as pltfrom sklearn.metrics import confusion\_matrixconf\_matrix = confusion\_matrix(y\_test, y\_pred)sns.heatmap(conf\_matrix, annot=True, fmt='g', xticklabels=clf.classes\_, yticklabels=clf.classes\_, )plt.ylabel('Prediction',fontsize=13)plt.xlabel('Actual',fontsize=13)plt.title('Confusion Matrix',fontsize=17)plt.show() R versionIn the same environment as the Python environment:

utils::install.packages("reticulate")library("reticulate")unifiedbooster <- import("unifiedbooster") Get data:

utils::install.packages("palmerpenguins")library("palmerpenguins")penguins\_ <- as.data.frame(palmerpenguins::penguins)# replacing NA's by the medianreplacement <- median(palmerpenguins::penguins$bill\_length\_mm, na.rm = TRUE)penguins\_$bill\_length\_mm[is.na(palmerpenguins::penguins$bill\_length\_mm)] <- replacementreplacement <- median(palmerpenguins::penguins$bill\_depth\_mm, na.rm = TRUE)penguins\_$bill\_depth\_mm[is.na(palmerpenguins::penguins$bill\_depth\_mm)] <- replacementreplacement <- median(palmerpenguins::penguins$flipper\_length\_mm, na.rm = TRUE)penguins\_$flipper\_length\_mm[is.na(palmerpenguins::penguins$flipper\_length\_mm)] <- replacementreplacement <- median(palmerpenguins::penguins$body\_mass\_g, na.rm = TRUE)penguins\_$body\_mass\_g[is.na(palmerpenguins::penguins$body\_mass\_g)] <- replacement# replacing NA's by the most frequent occurencepenguins\_$sex[is.na(palmerpenguins::penguins$sex)] <- "male" # most frequent# one-hot encodingpenguins\_mat <- model.matrix(species ~., data=penguins\_)[,-1]penguins\_mat <- cbind(penguins$species, penguins\_mat)penguins\_mat <- as.data.frame(penguins\_mat)colnames(penguins\_mat)[1] <- "species"y <- as.integer(penguins\_mat$species) - 1LX <- as.matrix(penguins\_mat[,2:ncol(penguins\_mat)])n <- nrow(X)p <- ncol(X)set.seed(123)index\_train <- sample(1:n, size=floor(0.8*n))X\_train <- X[index\_train, c("islandDream", "islandTorgersen", "flipper\_length\_mm")]y\_train <- y[index\_train]X\_test <- X[-index\_train, c("islandDream", "islandTorgersen", "flipper\_length\_mm") ]y\_test <- y[-index\_train] Find hyperparameters:

res <- unifiedbooster$cross\_val\_optim(X\_train=X\_train, y\_train=y\_train, X\_test=X\_test, y\_test=y\_test, model\_type="xgboost", type\_fit="classification", scoring="accuracy", n\_estimators=100L, cv=5L, # numbers of folds in cross-validation verbose=1L, seed=123L)print(res) check

```

Initialize the unified clf clf = do.call(unifiedbooster$GBDTClassifier, res$best_params)# Fit the modelclf$fit(X_train, y_train)# Predict on the test sety_pred = clf$predict(X_test)# Evaluate model's accuracy on test setprint(mean(y_test == y_pred))

``` To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Auto XGBoost, Auto LighGBM, Auto CatBoost, Auto GradientBoosting

View Details

[This article was first published on R on Publishable Stuff, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This is just a post to brag about that the CopenhagenR useR group is alive and kicking, again.

After COVID-19, the group (like so many other meetups) was on hiatus for a couple of years and without an organizer. In 2023, I thought I would try starting it again and, while it took a little while, I’m happy that I got together five great meetups for the spring 2024 season! Here’s a little bit about what went down.

CopenhagenR gratefully acknowledges the R Consortium as a sponsor. Also, a great thanks to Prosa, who generously provide a location for most meetups this spring 2024 season.

• State of the R, the New Stuff – Niels Ole DamFirst out was Niels Ole Dam who, very appropriately, set out to cover what has happened in the R world the last couple of years. His highlights included Quarto, webR, the GT package, and many other things. Check out his very sleek slides here.

• Reproducible Workflows with R and sequencing data – Adrian GeisslerNext out was Adrian Geissler, postdoctoral researcher at the University of Copenhagen. His presentation focused on how to handle data management, reproducible workflows, and large scale computing in the life sciences using R for computation and snakemake for orchestration. Here are the presentation slides and a screencast of the presentation.

• Simplify making shiny apps with teal – Dawid Kałędkowski // R and Software Freedom – Ramarro MarroneIn this double-header evening, Dawid Kałędkowski kicked things off with showcasing his package Teal — a shiny-based interactive exploration framework for quickly creating reproducible dashboards. Next up, Ramarro Marrone tackled the hot topic of software freedom, and went through which parts of the R world were freer and which could be considered non-free. This talk resulted in a heated debate that continued over the following coffee/cake/beer.

• Visualizing 440 bicycle rides using R – Gregers Kjerulf Dubrow Gregers Kjerulf Dubrow presented a very personal analysis of all bike trips he made in Copenhagen in 2023, as tracked by the Strava fitness app. The first part of the talk detailed how to import and clean Strava data in R, the second part took a deep dive into the time-location bike data of the more than 440 bicycle rides (including a data anomaly that, in the end, turned out to have had a very real and dangerous cause). The full bike analysis is available on Gregers’ blog.

• Animating a melody as a mathematical object – Charles T. GrayRounding off the season, Charles T. Gray, former professional musician turned data scientist, gave us a peek into the musical world through the lens of graphs, nodes, and edges. Her presentation explored animating a midi file as graph using R packages like pyramidi, ggraph, and gganimate. Check out a post-version of her talk on her blog.

To leave a comment for the author, please follow the link and comment on their blog: R on Publishable Stuff.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: CopenhagenR, the 2024 spring season

View Details

[This article was first published on rstats on Irregularly Scheduled Programming, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’ve been using a lot of programming languages recently and they all have theirquirks, differentiating features, and unique qualities, but one thing most ofthem have is that they handle strings as a collection of characters. R doesn’t,it has a “character” type which is 0 or more characters, and that’s what we calla “string”, but what if it did have iterable strings?

For comparison, here’s some Python code

for i in "string": print(i)string and some Haskell

upperA = map (\c -> if c == 'a' then 'A' else c)upperA "banana""bAnAnA" and some Julia

[x+1 for x in "HAL"]3-element Vector{Char}: 'I': ASCII/Unicode U+0049 (category Lu: Letter, uppercase) 'B': ASCII/Unicode U+0042 (category Lu: Letter, uppercase) 'M': ASCII/Unicode U+004D (category Lu: Letter, uppercase) In each of these cases the string is treated as a collection of individualcharacters. Many languages make this distinction, going so far as using differentquotes to distinguish them; e.g. double quotes for strings "string" and singlequotes for individual characters 's'. This makes a even more sense when thelanguage supports types in that a string has a String type that is composedof 0 or more Char types.

R is dynamically typed, so we don’t strictly enforce type signatures, and is anarray language, so it has natural support for arrays (vectors, lists, matrices).So why are strings not collections of characters?

My guess is that for the majority of use-cases, it wasn’t necessary – a lot ofthe time when we read in text data we want the entirety of the string and don’twant to worry about dealing with a collection on top of the collection of stringsthemselves. Plus, if you really need the individual characters you can split thetext up with strsplit(x, "").

But if you do want to work with individual characters, callingstrsplit(x, "")[[1]] throughout your code gets ugly. I solved the Exercismproblem ‘Anagram’ in R and really didn’t like how it looked

anagram <- function(subject, candidates) { # remove any same words and inconsistent lengths nonsames <- candidates[tolower(candidates) != tolower(subject) & nchar(subject) == nchar(candidates)] if (!length(nonsames)) return(c()) # no remaining candidates s\_letters <- sort(tolower(strsplit(subject, "")[[1]])) c\_letters <- sapply(sapply(nonsames, \(x) strsplit(x, "")), sort, simplify = FALSE) # find all cases where the letters are all the same anagrams <- nonsames[sapply(c\_letters, \(x) all(s\_letters == tolower(x)))] # if none found, return NULL if(!length(anagrams)) NULL else anagrams} Two calls to strsplit, then needing to sapply over that collection to sort it…not pretty at all. Here’s a Haskell solutionfrom someone very knowledgeable in our local functional programming Meetup group

import Data.List (sort)import Data.Char (toLower)anagramsFor :: String -> [String] -> [String]anagramsFor xs = filter (isAnagram xs' . map toLower) where xs' = map toLower xsisAnagram :: String -> String -> BoolisAnagram a b | a == b = False | otherwise = sort a == sort b which, excluding the type declarations and the fact that it needs to deal withthe edge case that it has to be a rearrangement, could nearly be a one-liner

import Data.List (sort)import Data.Char (toLower)isAnagram a b = sort (map toLower a) == sort (map toLower b) Wouldn’t it be nice if we could do things like this in R?

The world if R had iterable stringsI don’t expect it would ever happen (maaaybe via some special string handlinglike the bare strings r"(this doesn't need escaping)" but unlikely). Icouldn’t find a package that did this (by all means, let me know if there isone) so I decided to build it myself and see how it could work.

Introducing {charcuterie} - namedpartly because it looks like “cut” “char”, and partly because of charcuterie boardsinvolving lots of little bits of appetizers.

image by Google gemini library(charcuterie) At its core, this is just defining chars(x) as strsplit(x, "")[[1]] andslapping a new class on the output, but big improvements don’t immediately comefrom moonshots, they come from incremental improvements. Once I had this, I wantedto do things with it like sort the individual characters. There is of course asort method for vectors (but not for individual strings) so

sort("string")## [1] "string"sort(c("s", "t", "r", "i", "n", "g"))## [1] "g" "i" "n" "r" "s" "t" One aspect of treating strings as collections of characters is that they shouldalways look like strings, so I needed to modify the sort method to return anobject of this new class, and make this class display collections of charactersas a string. That just involves pasting the characters back together for printing,so now I can have this

s <- chars("string")s## [1] "string"sort(s)## [1] "ginrst" It looks like a string, but it behaves like a collection of characters!

When you do things right, people won’t know you’ve done anything at allI thought about what other operations I might want to do and now I have methods to

sort with sort
reverse with rev
index with [
concatenate with c
print with format and print
slice with head and tail
set operations with setdiff, union, intersect, and a new except
leverage existing vectorised operations like unique, toupper, and tolower

I suspect the concatenation will be the one that raises the most eyebrows… I’vedealt with the way that other languages join together strings beforeand I’m certainly open to what this version should do, but I think it makessense to add the collections as

c(chars("butter"), chars("fly"))## [1] "butterfly" If you need more than one chars at a time, you’re asking for a vector of vectors,which R doesn’t support - it supports a list of them, though

x <- lapply(c("butter", "fly"), chars)x## [[1]]## [1] "butter"## ## [[2]]## [1] "fly"unclass(x[[2]])## [1] "f" "l" "y" This still sounds simple, and it is - the point is that it feels a lot moreergonomic to use this inside a function compared to strsplit(x, "")[[1]] andworking with the collection manually.

I added an entire vignette of examples to the package, including identifying vowels

vowels <- function(word) { ch <- chars(word) setNames(ch %in% chars("aeiou"), ch)}vowels("string")## s t r i n g ## FALSE FALSE FALSE TRUE FALSE FALSEvowels("banana")## b a n a n a ## FALSE TRUE FALSE TRUE FALSE TRUE palindromes

palindrome <- function(a, ignore\_spaces = FALSE) { a <- chars(a) if (ignore\_spaces) a <- except(a, " ") all(rev(a) == a)}palindrome("palindrome")## [1] FALSEpalindrome("racecar")## [1] TRUEpalindrome("never odd or even", ignore\_spaces = TRUE)## [1] TRUE and performing character-level substitutions

spongebob <- function(phrase) { x <- chars(phrase) odds <- seq(1, length(x), 2) x[odds] <- toupper(x[odds]) string(x)}spongebob("you can't do anything useful with this package")## [1] "YoU CaN'T Do aNyThInG UsEfUl wItH ThIs pAcKaGe" YoU CaN’T Do aNyThInG UsEfUl wItH ThIs pAcKaGeOn top of all that, I felt it was worthwhile stretching my R package buildingmuscles, so I’ve added tests with 100% coverage, and ensured it fully passescheck().

I don’t expect this would be used on huge text sources, but it’s useful to mefor silly little projects. If you have any suggestions for functionality thatcould extend this then by all means let me know either inGitHub Issues, the comment sectionbelow, or Mastodon.

devtools::session_info() ```

─ Session info ───────────────────────────────────────────────────────────────## setting value## version R version 4.3.3 (2024-02-29)## os Pop!_OS 22.04 LTS## system x86_64, linux-gnu## ui X11## language (EN)## collate en_AU.UTF-8## ctype en_AU.UTF-8## tz Australia/Adelaide## date 2024-08-03## pandoc 3.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)## ## ─ Packages ───────────────────────────────────────────────────────────────────## package * version date (UTC) lib source## blogdown 1.18 2023-06-19 [1] CRAN (R 4.3.2)## bookdown 0.36 2023-10-16 [1] CRAN (R 4.3.2)## bslib 0.6.1 2023-11-28 [3] CRAN (R 4.3.2)## cachem 1.0.8 2023-05-01 [3] CRAN (R 4.3.0)## callr 3.7.3 2022-11-02 [3] CRAN (R 4.2.2)## charcuterie * 0.0.0.9000 2024-08-03 [1] local## cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.3)## crayon 1.5.2 2022-09-29 [3] CRAN (R 4.2.1)## devtools 2.4.5 2022-10-11 [1] CRAN (R 4.3.2)## digest 0.6.34 2024-01-11 [3] CRAN (R 4.3.2)## ellipsis 0.3.2 2021-04-29 [3] CRAN (R 4.1.1)## evaluate 0.23 2023-11-01 [3] CRAN (R 4.3.2)## fastmap 1.1.1 2023-02-24 [3] CRAN (R 4.2.2)## fs 1.6.3 2023-07-20 [3] CRAN (R 4.3.1)## glue 1.7.0 2024-01-09 [1] CRAN (R 4.3.3)## htmltools 0.5.7 2023-11-03 [3] CRAN (R 4.3.2)## htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.3.2)## httpuv 1.6.12 2023-10-23 [1] CRAN (R 4.3.2)## icecream 0.2.1 2023-09-27 [1] CRAN (R 4.3.2)## jquerylib 0.1.4 2021-04-26 [3] CRAN (R 4.1.2)## jsonlite 1.8.8 2023-12-04 [3] CRAN (R 4.3.2)## knitr 1.45 2023-10-30 [3] CRAN (R 4.3.2)## later 1.3.1 2023-05-02 [1] CRAN (R 4.3.2)## lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.3.3)## magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.3)## memoise 2.0.1 2021-11-26 [3] CRAN (R 4.2.0)## mime 0.12 2021-09-28 [3] CRAN (R 4.2.0)## miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.3.2)## pkgbuild 1.4.2 2023-06-26 [1] CRAN (R 4.3.2)## pkgload 1.3.3 2023-09-22 [1] CRAN (R 4.3.2)## prettyunits 1.2.0 2023-09-24 [3] CRAN (R 4.3.1)## processx 3.8.3 2023-12-10 [3] CRAN (R 4.3.2)## profvis 0.3.8 2023-05-02 [1] CRAN (R 4.3.2)## promises 1.2.1 2023-08-10 [1] CRAN (R 4.3.2)## ps 1.7.6 2024-01-18 [3] CRAN (R 4.3.2)## purrr 1.0.2 2023-08-10 [3] CRAN (R 4.3.1)## R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.3)## Rcpp 1.0.11 2023-07-06 [1] CRAN (R 4.3.2)## remotes 2.4.2.1 2023-07-18 [1] CRAN (R 4.3.2)## rlang 1.1.4 2024-06-04 [1] CRAN (R 4.3.3)## rmarkdown 2.25 2023-09-18 [3] CRAN (R 4.3.1)## rstudioapi 0.15.0 2023-07-07 [3] CRAN (R 4.3.1)## sass 0.4.8 2023-12-06 [3] CRAN (R 4.3.2)## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.2)## shiny 1.7.5.1 2023-10-14 [1] CRAN (R 4.3.2)## stringi 1.8.3 2023-12-11 [3] CRAN (R 4.3.2)## stringr 1.5.1 2023-11-14 [3] CRAN (R 4.3.2)## urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.3.2)## usethis 3.0.0 2024-07-29 [1] CRAN (R 4.3.3)## vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.3.3)## xfun 0.41 2023-11-01 [3] CRAN (R 4.3.2)## xtable 1.8-4 2019-04-21 [1] CRAN (R 4.3.2)## yaml 2.3.8 2023-12-11 [3] CRAN (R 4.3.2)## ## [1] /home/jono/R/x86_64-pc-linux-gnu-library/4.3## [2] /usr/local/lib/R/site-library## [3] /usr/lib/R/site-library## [4] /usr/lib/R/library## ## ──────────────────────────────────────────────────────────────────────────────

``` To leave a comment for the author, please follow the link and comment on their blog: rstats on Irregularly Scheduled Programming.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {charcuterie} – What if Strings Were Iterable in R?

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium is pleased to announce that Katherine Jeschke has been appointed Chair of the Grants Committee.

She will oversee day-to-day grant processes for both pre- and post-awards, ensuring grants meet the R Consortium’s strategic objectives. Katherine will report to the Executive Director and work closely with the Infrastructure Steering Committee and the RUGS program to track grants and help improve the effectiveness of R Consortium grants in supporting the R Community.

Katherine’s non-profit grants and administration skills were honed over more than 25 years of experience in marketing communications, and staff, budget, development, and grants management for non-profits and trade organizations, as well as corporate and public sector consulting organizations.

She holds an MA degree in American History and Museum Administration from George Washington University and a BA degree in Fine Arts and Art History from the University of Maryland.

“Getting acquainted with our various grants and procedures while evaluating the strategic effectiveness of R Consortium grants is a big undertaking, but her background and years of experience should ease her way,” said Joseph Rickert, Executive Director of the R Consortium.

She may be reached at kj.jeschke@posit.co.

The post R Consortium Grants Committee Announces New Chair appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Consortium Grants Committee Announces New Chair

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Rhino, the R package development framework created by Appsilon, has recently released two updates – versions 1.8 and 1.9. In this fireside chat, Kamil Żyła (one of the core developers of Rhino) and Marek Rogala (Appsilon’s Head of Technology) sit down to discuss these updates and the vision behind Rhino.

Watch the Full InterviewMarek: For those who haven’t tried it out yet, what exactly is Rhino?

Kamil: Rhino is an R package that you can install just like any other R package and start using right away. It’s best to think of it as a framework for your whole development process.

Rhino takes care of all the tools around software development that will help you maintain the best possible quality for your R applications.

You get a project structure and setup for all the tools needed for automation, so all those checks which Rhino provides can happen automatically without your input. It’s designed to help you build high-quality applications with all that setup coming out of the box, so you don’t need to worry about it.

Marek: What is the high-level vision of Rhino? How would you describe it?

Kamil: To understand the vision, it’s helpful to know how Rhino was born. It was created out of the need we had at Appsilon. We kept creating projects for our clients, and a lot of work was needed repeatedly.

We would set up a project with all the necessary tools, and this happened time and time again. We realized it would be great to create something so that we wouldn’t have to reinvent the wheel each time. The long-term vision now is to keep investing in improving Rhino based on our experience in projects and the things we are learning, so that it represents our best state of knowledge about how to develop Shiny applications well, with all the best practices.

Marek: Let’s discuss the new features in Rhino 1.8 and 1.9. First, tell us about box::linters.

Kamil: box::linters is a package that allows you to lint your R code for correct usage of the box package. Box provides a proper import system for R, similar to other modern languages. Many tools are not designed to work with box import statements, causing issues like the inability to lint for unused objects.

With time, we started working towards improving the situation and created linters specifically designed to work with the box package. As the idea matured and we added more linters, we realized it would make sense to create a separate package out of them. This is what box::linters is – a package that provides linters to automatically check how well you use box statements. Because it’s a separate package, you can also use it outside of Rhino.

The lintr package is the standard tool used for linting R source code, and it’s extensible. You can define new linters, and the box::linters package defines a couple of new ones specifically designed to work with box.

Marek: What about the improvements to GitHub Actions workflow triggers?

Kamil: This is a small change, but still quite helpful. A Rhino application comes with a GitHub Actions workflow that automatically runs all the linters and tests for the whole application.

Previously, it would trigger each time you push any code to your repository, which we figured is too much sometimes and doesn’t work that well with pull requests. We’ve made some improvements so that the tests only run when you open a pull request. This means that once you decide you want to see the checks run on your code, all you need to do is create a pull request from your branch, and you will have your checks. But it won’t happen on all the pushes to the repository, which can be a significant cost saver because running all these actions takes time.

Marek: So basically, I can imagine I’m working on some feature, and even before I share it with anyone, but I keep pushing the latest version of my changes, like new commits to some branch on GitHub, and each of those commits doesn’t trigger any run of all the checks and tests.

Kamil: Yes, unless you open a pull request, and then it will work like before. When the pull request is open, every push after that triggers a new run.

Marek: Coming to 1.9, you mentioned Sass integration. What was this change about?

Kamil: This is again a small addition, but one that makes things cleaner and more natural. Rhino comes with built-in support for writing custom Sass code. All you need to do is open a file, and you can start writing Sass.

Sass is an extension language for CSS, which gives you some really nice features and makes the development process much more pleasant. Rhino comes with built-in support for it. You just start writing that code, and with a single function, you can compile it into a nice minified file optimized for modern browsers. This file will be included automatically.

Typically, when you work with Shiny applications, the approach to styling will be to use some package which will do most of the work for you. There are many packages out there that do that, and one of them is bslib. bslib is actually used by Shiny underneath for its default appearance, but if you use bslib directly, you get access to much more capabilities. You get more components, you can style your application, add theming, and get access to the latest Bootstrap.

While you could use bslib without too much problem in Rhino before, the setup wasn’t that beautiful because it wasn’t working with custom Sass so well. The addition in Rhino 1.9 is a simple one. There’s a configuration file in Rhino that allows you to define how your Sass is built. A new configuration option called custom allows you to do it in your own way. You can bundle this custom Sass with the bslib theme.

Shiny will take care of the rest. That primarily means you can use variables you define for bslib in your custom Sass. But also, whatever you define in your custom Sass can be used with bslib. For example, you can override Bootstrap mixins if you wanted to. So you really get the best of both worlds and can customize it to the fullest extent you want.

Marek: So Rhino takes the assets from bslib and your custom assets and bundles them together instead of just bundling the custom assets. Is that how it works?

Kamil: Close, but not quite. Actually, the recommended approach now is to let bslib and the Sass package, which is used underneath for compiling Sass, handle your custom Sass for yourself.

Marek: And bslib, by the way, is like getting more mature, and I think it’s becoming a more mainstream approach to all Shiny applications.

Kamil: I think so, definitely. Now with the latest features of Bootstrap and bslib, you can do quite a lot of things that previously you’d have to search for a package to do for you. Like you’d use Shiny Dashboard. Now you have many functions for both defining your layout in the application and utilities like creating cards. So basically, you can build most of your UI using just bslib.

And then you can select a theme, which doesn’t lock you into one default appearance or force you into selecting some package based on what it looks like. You can select a theme for those components, and this makes it really quite customizable.

Marek: Right, so the last one was Prettier. What does this mean for us?

Kamil: Again, a brief introduction. Since the beginning, or at least since Rhino 1.0, we had linters for Sass and JavaScript. While these are not the technologies that you use that much in small Shiny applications, we learned that as the project progresses, typically you will need more customizations. And these are the technologies which are kind of the backbone technologies of the web. So we decided that you need the best tools for them as well.

From the beginning, we had linters for Sass and JavaScript. However, over the past couple of years, the landscape of tooling regarding Sass and JavaScript has changed in general. Now the linters generally focus more on detecting buggy or potentially incorrect code and less on code formatting. Instead, formatting is typically delegated to formatters. And here, Prettier is, I believe, the most popular formatter out there. So we decided to use that one. By running them, you can automatically format all your source files.

Marek: It definitely makes sense, especially since it was always a bit awkward because linters were checking also formatting rules, but then it was like duplicated responsibility, right? Because formatters also worked on the formatting, so it was always kind of weird. Before we wrap up, could you tell me, for someone who wants to try out Rhino, what would be the best steps to start?

Kamil: I think the best step would be to visit our website. You can find it by typing “Rhino Appsilon” in Google, and you’ll definitely get there. You can start trying it right away. The recommended way would be to just go to our website, install it, and start a new project.

First time using Rhino? Check out our step by step tutorial to help you get started.

Wrapping Up All You Need to Know about Rhino’s UpdatesRhino’s updates in versions 1.8 and 1.9 bring exciting improvements to the R package development experience. With features like box::linters, enhanced Sass integration, and Prettier formatting, Rhino continues to provide the best tools and practices for creating high-quality Shiny applications. If you’re interested in streamlining your R package development workflow, give Rhino a try!

Get the latest Rhino updates and everything Shiny delivered to your inbox. Subscribe to Shiny Weekly.

Resources: * Rhino GitHub Repository * Rhino Documentation * Boost Your Shiny App’s Code Quality with {box.linters} in {rhino} 1.8.0 * Rhino 1.9.0 Update on CRAN: New Formatting Tools and Cleaner {bslib} Integration

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: All You Need to Know About Rhino 1.8 and 1.9 Updates: A Fireside Chat

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionCluster sampling is a useful technique when dealing with large datasets spread across different groups or clusters. It involves dividing the population into clusters, randomly selecting some clusters, and then sampling all or some members from these selected clusters. This method can save time and resources compared to simple random sampling.

In this post, we’ll walk through how to perform cluster sampling in R. We’ll use a sample dataset and break down the code step-by-step. By the end, you’ll have a clear understanding of how to implement cluster sampling in your projects.

Example ScenarioLet’s say we have a dataset of students from different schools, and we want to estimate the average test score. Sampling every student would be too time-consuming, so we’ll use cluster sampling.

Step 1: Create a Sample DatasetFirst, let’s create a sample dataset to work with.

```

Load necessary librarieslibrary(dplyr)# Set seed for reproducibilityset.seed(123)# Create sample dataschools <- data.frame( student_id = 1:1000, school_id = rep(1:10, each = 100), test_score = rnorm(1000, mean = 75, sd = 10))# Display the first few rows of the datasethead(schools)

```

student\_id school\_id test\_score1 1 1 69.395242 2 1 72.698233 3 1 90.587084 4 1 75.705085 5 1 76.292886 6 1 92.15065 Step 2: Divide the Population into ClustersOur population is already divided into clusters by school_id. Each school represents a cluster.

Step 3: Randomly Select ClustersNext, we’ll randomly select some clusters. Let’s say we want to select 3 out of the 10 schools.

```

Number of clusters to selectnum_clusters <- 3# Randomly select clustersselected_clusters <- sample(unique(schools$school_id), num_clusters)# Display selected clustersselected_clusters

```

[1] 1 10 2 Step 4: Sample Members from Selected ClustersNow, we’ll sample students from the selected schools.

```

Filter the dataset to include only the selected clusterssampled_data <- schools %>% filter(school_id %in% selected_clusters)# Display the first few rows of the sampled datahead(sampled_data)

```

student\_id school\_id test\_score1 1 1 69.395242 2 1 72.698233 3 1 90.587084 4 1 75.705085 5 1 76.292886 6 1 92.15065 Step 5: Analyze the Sampled DataFinally, we can analyze the sampled data to estimate the average test score.

```

Calculate the mean test scoremean_test_score <- mean(sampled_data$test_score)# Display the mean test scoremean_test_score

```

[1] 74.87889 Explanation of Code Blocks Step 1: We create a sample dataset with 1000 students, each belonging to one of 10 schools. Each student has a test score. * Step 2: The school_id column naturally divides our dataset into clusters. * Step 3: We randomly select 3 out of the 10 schools using the sample function. * Step 4: We filter the dataset to include only students from the selected schools. * Step 5*: We calculate the mean test score of the sampled students to estimate the overall average.

ConclusionCluster sampling is a powerful method for efficiently sampling large populations. By dividing the population into clusters and sampling within those clusters, you can obtain reliable estimates with less effort.

Feel free to try this method on your own datasets. Experiment with different numbers of clusters and sample sizes to see how it affects your results.

Happy coding!

If you have any questions or need further clarification, drop a comment below!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cluster Sampling in R: A Simple Guide

View Details

[This article was first published on r-spatial, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.* Summary * Installation + From github + From CRAN * Getting started * Adjacency matrix * Non-spatial regression + MCMC output + Methods * Spatial regression + A filtering approach + A bivariate model + Predicted values * Future work and support * Appendix * References

[view rawRmd]

SummaryThis post introduces thegeostan R package for spatialanalysis. The package is mainly oriented towards areal data, althoughsome models may also be used for other spatial data types. The packageimplements the spatial error/simultaneous spatial autoregressive (SAR)model, conditional autoregressive (CAR) model, and eigenvector spatialfilter (ESF) models for spatial regression. A version of ESF modellingalso appears in the ecology literature as principle coordinate analysisof neighbor matrices (PCNM) (Griffith and Peres-Neto 2006).

geostan also supports the application of the above regression methodsto hierarchical models for count data, as is common in analyses ofdisease incidence or mortality in small areas (‘disease mapping’).Additional features of the software include models forsampling/measurement error in covariates and methods for handlingcensored count data, such as when mortality or disease counts have beencensored for privacy. The models were built using theStan modeling language, so all inference iscompleted using Markov chain Monte Carlo (MCMC) sampling (StanDevelopment Team 2023; Gabry et al. 2024). The spatial autoregressivemodels use custom-built Stan functions that speed up MCMC samplingconsiderably (Donegan 2021).

This post will walk through an example analysis using international dataon life expectancy and per capita GDP. Package vignettes can be foundwith the onlinedocumentation,including an introduction to spatial weights matrices, exploratoryspatial data analysis, spatial measurement error models, rasterregression, and using geostan to build custom spatial models with Stan.A paper in the Journal of Open SourceSoftware reports these and otherfeatures and provides the recommended citation when using geostan(Donegan 2022).

Installationgeostan is currently on CRAN, although that may not always be thecase. You can also install directly from geostan’s githubrepository.

From githubYou can install from the package github repository:

if (!require('devtools')) install.packages('devtools')devtools::install\_github("connordonegan/geostan") If you are using Windows and installing using install_github, you mayneed to install Rtoolsfirst. Rtools is not needed when installing from CRAN. You may alsocontact the author by e-mail for a pre-compiled version that you can usewithout Rtools.

If you are using Mac and installing with install_github then you mayneed to install Xcode Command Line Tools first.

From CRANUsing your R console, you can install from CRAN as follows:

install.packages("geostan") Getting startedTo begin, load the geostan and sf packages into your R environment,as well as the world data:

library(geostan)library(sf)data(world, package = "spData") The world data contains life expectancy and gross domestic product(GDP) per capita (presumably measured in current $US) for 161 countriesas of 2014, gathered from the World Bank. The rest of this post is goingto be structured around a bivariate analysis of these variables.

We are going to apply the Robinson map projection for the countries:

world <- st\_transform(world, crs = 'ESRI:54030') At least a couple of the missing values can be filled in using WorldBank data, so we will do that:

```

https://data.worldbank.orgfrance <- grep("France", world$name_long)world$gdpPercap[ france ] <- 43068world$lifeExp[ france ] <- 82norway <- grep("Norway", world$name_long)world$gdpPercap[ norway ] <- 97666world$lifeExp[ norway ] <- 82.1

``` And we will also remove Antarctica:

world <- subset(world, name\_long != "Antarctica") Mapping the variables shows the familiar geography of high-, middle-,and low-income countries and a similar geography of longevity:

```

store geometry for countriesworld_geom <- st_geometry(world)# show two maps at once, with nice fontogpar <- par(mfrow = c(2, 1), mar = rep(0, 4))# GDP per capitapars <- map_pars(world$gdpPercap / 1e3)plot(world_geom, col = pars$col, lwd = .2)legend("bottomleft", fill = pars$pal, title = 'GDP per capita\n($1,000s)', legend = pars$lbls, bty = 'n')rm(pars)# life expectancypars <- map_pars(world$lifeExp)plot(world_geom, col = pars$col, lwd = .2)legend("left", fill = pars$pal, title = 'Life Expectancy', legend = pars$lbls, bty = 'n' )

``` Choropleth maps of GDP per capita and life expectancy.

par(ogpar) The map_pars function breaks the variables into quantiles and returnsbreaks, colors, labels for the maps; it can be found at the end of thepost. There will be no discussion of substantive (non-statistical)issues here, for which one can consult any number of texts on globalpower and inequality (e.g., Paul Farmer’s Infections andInequalities).

By conventional methods, the correlation coefficient for life expectancyand log GDP per capita is 0.81:

log\_x <- log10( world$gdpPercap )y <- world$lifeExpcor.test(log\_x, y)## ## Pearson's product-moment correlation## ## data: log\_x and y## t = 17.394, df = 160, p-value < 2.2e-16## alternative hypothesis: true correlation is not equal to 0## 95 percent confidence interval:## 0.7478143 0.8561778## sample estimates:## cor ## 0.8087528 The conventional assessment is based on the proposition that we have 161independent observations. The visible geography of the variables, andany level of social awareness, indicates that these are not independentobservations. Rather, there are various functional regions of countriesthat share basic political-economic conditions. A lot, but not all, ofthe variation can be described as variation across continents andregions. We will want to account for this dependence using a spatialmodel (for background see Chun and Griffith 2012; Donegan 2024). Thefirst step will be to construct a spatial weights matrix.

Adjacency matrixThis section will illustrate use of two geostan functions for creatingan revising a spatial weights matrix: shape2mat and edges. Theshape2mat function may be helpful for some users but one can always dothis using spdep or other methods, especially if shape2mat does notprovide the exact method you’re looking for.

We are going to start by removing the 15 countries that are missingvalues:

```

remove missing valuesworld <- subset(world, !is.na(gdpPercap) & !is.na(lifeExp))## leaving 162 observationsnrow(world)## [1] 162

`` Now we can apply theshape2matfunction to obtain an adjacencymatrixthat encodes spatial adjacency relations for countries into a binaryN-by-N matrix. The function usesspdepto find adjacency relations andreturns results as a sparse matrix (using theMatrix` package):

A <- shape2mat(world, "B", method = "rook")## Contiguity condition: rook## Number of neighbors per unit, summary:## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0.000 2.000 3.000 3.605 5.000 13.000## ## Spatial weights, summary:## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 1 1 1 1 1 1 Visualizing the connections in the matrix is important for uncoveringunexpected results. geostan’s edges function converts the matrixinto a list of nodes and edges that we can plot. For this we need tosupply the function with the adjacency matrix, A, and the associatedspatial object, world:

```

edges with geometryE <- edges(A, shape = world) graph <- st_geometry(E)ogpar <- par(mar = rep(0, 4))# plot countriesplot(world_geom, lwd = .1)# add graph nodesplot(graph, add = TRUE, type = 'p')# add graph edgesplot(graph, add = TRUE, type = 'l')

```

par(ogpar) This reveals quite a few unexpected results. French Guiana is stored inthe world data as part of France (a multi-part polygon); this iscorrect of course but it leads to Brazil and Suriname being listed asneighbors of France, which is not sensible. Besides removing thoseconnections, there are a number of island nations that we might want toconnect to nearby places.

To connect Mozambique to Madagascar, we just replace the zeroes withones in the slots that correspond to those countries. First we grabtheir index positions in the matrix:

moz\_idx <- grep("Mozambique", world$name\_long)mad\_idx <- grep("Madagascar", world$name\_long) And then we assign the correct slots in the matrix a value of 1 (orTRUE), remembering that the adjacency matrix is symmetric:

A[moz\_idx, mad\_idx] <- A[mad\_idx, moz\_idx] <- TRUE This can become tedious but it is important. Before moving on, we willmake a series of adjustments. This will be made a bit easier with thisconvenience function:

connect <- function(country\_a, country\_b, names\_vec = world$name\_long, matrix = A, add = TRUE) { stopifnot( country\_a %in% names\_vec ) stopifnot( country\_b %in% names\_vec ) a\_idx <- which(names\_vec == country\_a) b\_idx <- which( names\_vec == country\_b) matrix[a\_idx, b\_idx] <- matrix[b\_idx, a\_idx] <- add return( matrix )} The following are at least reasonable changes to make; they also ensurethat every country has at least one neighbor:

A <- connect("Mozambique", "Madagascar")A <- connect("Australia", "New Zealand")A <- connect("Philippines", "Malaysia")A <- connect("Japan", "Republic of Korea")A <- connect("Fiji", "Vanuatu")A <- connect("Solomon Islands", "Vanuatu")A <- connect("Solomon Islands", "Papua New Guinea")A <- connect("Australia", "Papua New Guinea")A <- connect("Haiti", "Jamaica")A <- connect("Bahamas", "United States")A <- connect("Dominican Republic", "Puerto Rico")A <- connect("Trinidad and Tobago", "Venezuela")A <- connect("Sri Lanka", "India")A <- connect("Cyprus", "Turkey")A <- connect("Cyprus", "Lebanon")A <- connect("Norway", "Iceland")## remove connections between South American and FranceA <- connect("Suriname", "France", add = FALSE)A <- connect("Brazil", "France", add = FALSE) We should look at the revised adjacency matrix:

graph <- st\_geometry( edges(A, shape = world) )ogpar <- par(mar = rep(0, 4))plot(world\_geom, lwd = .1)plot(graph, add = TRUE, type = 'p')plot(graph, add = TRUE, type = 'l')

par(ogpar) Sometimes it can help to examine the edge list interactively using aproper geographic information system like QGIS. For those who arefamiliar with (non-R) GIS software, you can save the edges list as aGeoPackage and then open it up in your GIS to examine the connections‘by hand’ with a base map or other data:

E <- edges(A, shape = world)st\_write(E, "world.gpkg", layer = "edge list") Non-spatial regressionFitting regression models with geostan is similar to using base R’sglm function: the user provides a model formula, data, and the modelfamily or distribution. We can fit a normal linear model using thestan_glm function:

fit\_lm <- stan\_glm(lifeExp ~ log(gdpPercap), data = world, quiet = TRUE) And we can examine parameter estimates by printing to the console:

print(fit\_lm)## Spatial Model Results ## Formula: lifeExp ~ log(gdpPercap)## Spatial method (outcome): none ## Likelihood function: gaussian ## Link function: identity ## Residual Moran Coefficient: NA ## WAIC: 977.33 ## Observations: 162 ## Data models (ME): none## Inference for Stan model: foundation.## 4 chains, each with iter=2000; warmup=1000; thin=1; ## post-warmup draws per chain=1000, total post-warmup draws=4000.## ## mean se\_mean sd 2.5% 20% 50% 80% 97.5% n\_eff## intercept 20.660 0.089 2.934 15.106 18.148 20.705 23.117 26.431 1082## log(gdpPercap) 5.499 0.010 0.317 4.879 5.235 5.498 5.770 6.108 1081## sigma 4.902 0.007 0.268 4.408 4.676 4.894 5.119 5.468 1458## Rhat## intercept 1.002## log(gdpPercap) 1.002## sigma 1.001## ## Samples were drawn using NUTS(diag\_e) at Thu Aug 1 12:33:33 2024.## For each parameter, n\_eff is a crude measure of effective sample size,## and Rhat is the potential scale reduction factor on split chains (at ## convergence, Rhat=1). The output printed to the console provides a summary of the posteriorprobability distributions of the model parameters. The distributions canalso be visualized using plot(fit_lm):

plot(fit\_lm)## `stat\_bin()` using `bins = 30`. Pick better value with `binwidth`. The mean of the distribution is reported in the mean column. For thosewho are more familiar with concepts from sampling theory, the mean maybe understood as the estimate of the parameter. Each distribution’sstandard deviation is found in the sd column; this describes the widthof the posterior distribution. The sd is analogous to the standarderror of the estimate. The quantiles also summarize the width of theposterior distributions; the 2.5% and 97.5% values form a 95% credibleinterval for the parameter value.

MCMC outputThe effective sample size (ESS), n_eff, tells us how many independentMCMC samples the inference is based on after adjusting for serialautocorrelation in the MCMC samples. This is an important quantity topay attention to and generally one might like to see these numbers above400 (or around 100 samples per MCMC chain). The standard error of themean, se_mean, reports how much MCMC sampling error to expect in themean (se_mean is calculated using n_eff). The R-hat statistic,Rhat, should always be very close to 1, preferably less than 1.01. TheR-hat diagnostic tests that the MCMC chains are all depicting the samedistribution. If they diverge from one another, it either means that youneed to draw a higher number of MCMC samples (run the chains for longer)or that there is a problem fitting the model to your data.

By default, geostan models run four independent MCMC chains for 3,000iterations each, half of which is discarded as warm-up. The number ofiterations is controlled by the iter argument, the default beingiter = 3e3. For some models this may be too low and you will want toincrease this. Other times this might be more than is needed in whichcase you can reduce the computation time by using fewer iterations. Whatmatters most is not your number of iterations but your ESS and R-hatstatistics. When it comes to reporting results, it is generally best touse at least the default of four MCMC chains (chains = 4).

MethodsA number of familiar methods are available for working with geostanmodels including fitted, resid, and predict.

The fitted method returns a data.frame with summaries of the fittedvalues. The probability distribution for each fitted value is summarizedby its posterior mean, standard deviation, and quantiles:

fdf <- fitted(fit\_lm)head(fdf)## mean sd 2.5% 20% 50% 80% 97.5%## fitted[1] 70.22990 0.3969583 69.46571 69.88939 70.23141 70.55628 71.00631## fitted[2] 63.46359 0.5884022 62.31542 62.98235 63.45552 63.94977 64.64037## fitted[3] 79.33702 0.6206999 78.07451 78.81257 79.34308 79.86151 80.53088## fitted[4] 80.36367 0.6675295 79.01816 79.79764 80.37163 80.93076 81.65248## fitted[5] 76.02492 0.4883834 75.05750 75.61649 76.03594 76.44239 76.95946## fitted[6] 67.88820 0.4339864 67.04641 67.52555 67.88695 68.24651 68.74736 The resid method behaves similarly. Examining the Moran scatter plotusing the residuals shows a moderate degree of positive SA as well assome skewness:

rdf <- resid(fit\_lm)moran\_plot(rdf$mean, A) Spatial regressionOptions for spatial regression models currently include conditionalautoregressive (CAR), simultaneous autoregressive (SAR/spatial error),and eigenvector spatial filtering (ESF). For count data, commonvariations on the intrinsic autoregressive (ICAR) model are alsoavailable.

All of the spatial models require at least a spatial weights matrix asinput. All additional requirements for data preparation are handled bygeostan’s prep_ functions: prep_car_data, prep_icar_data,prep_sar_data.

The make_EV function is used to create Moran’s eigenvectors for ESFregression; if you want to create your own eigenvectors (say, followingthe PCNM method) you can provide those directly to the ESF model (see?stan_esf).

For the CAR model, we always provide the binary adjacency matrix asinput to prep_car_data. See the prep_car_data documentation foroptions. Here we will fit an intercept-only CAR model to the lifeexpectancy data:

cars <- prep\_car\_data(A)fit\_car <- stan\_car(lifeExp ~ 1, data = world, car\_parts = cars, iter = 1e3, quiet = TRUE)print(fit\_car)## Spatial Model Results ## Formula: lifeExp ~ 1## Spatial method (outcome): CAR ## Likelihood function: auto\_gaussian ## Link function: identity ## Residual Moran Coefficient: -0.356467 ## WAIC: 994.65 ## Observations: 162 ## Data models (ME): none## Inference for Stan model: foundation.## 4 chains, each with iter=1000; warmup=500; thin=1; ## post-warmup draws per chain=500, total post-warmup draws=2000.## ## mean se\_mean sd 2.5% 20% 50% 80% 97.5% n\_eff Rhat## intercept 70.261 0.081 2.664 65.007 68.274 70.280 72.325 76.104 1080 1.002## car\_rho 0.981 0.000 0.010 0.958 0.973 0.983 0.990 0.996 1245 1.003## car\_scale 7.827 0.011 0.449 7.000 7.441 7.806 8.202 8.761 1564 1.001## ## Samples were drawn using NUTS(diag\_e) at Thu Aug 1 12:33:38 2024.## For each parameter, n\_eff is a crude measure of effective sample size,## and Rhat is the potential scale reduction factor on split chains (at ## convergence, Rhat=1). Notice that using iter = 1000 was more than adequate for inference inthis case.

The CAR model has a spatial dependence parameter car_rho. Thisparameter does not have an interpretation similar to a correlationcoefficient, and it is often near 1; this is not a problem unless onemisinterprets it or desires a value similar to the correlationcoefficient. The spatial dependence parameter in the SAR model doesprovide that kind of interpretation.

A filtering approachReturning to the correlation coefficient estimated above, one way toadjust our estimate for spatial dependence is to filter out the spatialtrend from each of the two variables and then calculate the correlationcoefficient using the detrended values (Chun and Griffith 2012, 71).This spatial ‘filtering’ or ‘pre-whitening’ method is not particularlycommon in practice but its a good trick to know given the familiarity ofthe correlation coefficient. We will use it here to demonstrate somebasic features of the software.

The spatial trend term can be extracted from any spatial geostan modelusing the spatial method.

theta <- spatial(fit\_car)$meanpars <- map\_pars(theta)ogpar <- par(mar = rep(0, 4))plot(st\_geometry(world), col = pars$col, lwd = .2)legend("left", fill = pars$pal, title = 'Spatial trend (LE)', legend = pars$lbls, bty = 'n' )

par(ogpar) We can obtain detrended values most simply by taking the residuals froman intercept-only spatial model:

```

lifeExpt detrendeddy <- resid(fit_car)$mean# log per capita GDP detrendedfit_carx <- stan_car(log(gdpPercap) ~ 1, data = world, car = cars, iter = 1e3, quiet = TRUE)dx <- resid(fit_carx)$mean

`` Usingcor.test` with those provides an estimate of correlation adjustedfor spatial autocorrelation:

```

adjusted correlationcor.test(dx, dy)## ## Pearson's product-moment correlation## ## data: dx and dy## t = 9.1418, df = 160, p-value = 2.666e-16## alternative hypothesis: true correlation is not equal to 0## 95 percent confidence interval:## 0.4744113 0.6786559## sample estimates:## cor ## 0.5857574

``` The adjusted estimate of .59 is considerably different from the naiveestimate of .80 and is outside the naive confidence intervals. (Theadjusted estimate is .62 if we use SAR models.)

A bivariate modelHere we will use the SAR model to illustrate its use. Fitting thespatial error or SAR model requires nearly the same steps as above.

Unlike prep_car_data, be sure to row-standardize the adjacency matrixbefore passing it to prep_sar_data.

W <- row\_standardize(A)sars <- prep\_sar\_data(W) When fitting the model, we are going to add centerx = TRUE to centerthe covariate. (Internally this will callcenter(x, center = TRUE, scale = FALSE).) This will not changecoefficient estimates but it does often improve MCMC samplingefficiency, sometimes considerably so. It does change interpretation ofthe intercept: the intercept will be an estimate of the average lifeexpectancy (or the expected life expectancy when all covariates are attheir average values).

fit\_sar <- stan\_sar(lifeExp ~ log(gdpPercap), data = world, sar\_parts = sars, centerx = TRUE, iter = 1e3, quiet = TRUE) Lets plot the results this time:

plot(fit\_sar)## `stat\_bin()` using `bins = 30`. Pick better value with `binwidth`. The spatial dependence parameter is around 0.7, which indicatesmoderately strong SA. The mean life expectancy is about 71 (probablysomewhere between about 69 and 74). And the coefficient for log GDP isaround 4 (or somewhere between 3 and 5). The residual variation has astandard deviation of around 3.6 years.

If we scale both variables before fitting the bivariate spatialregression model (so that their variances both equal 1) then we getapproximately the same estimate as the adjusted correlation coefficient(above). The credible interval is slightly wider because uncertainty inrho is (appropriately) incorporated here:

world <- transform(world, sx = scale(log(gdpPercap), scale = T, center = T), sy = scale(lifeExp, scale = T, center = T) )fit\_scaled <- stan\_sar(sy ~ sx, data = world, sar\_parts = sars, iter = 1e3, quiet = TRUE)print(fit\_scaled)## Spatial Model Results ## Formula: sy ~ sx## Spatial method (outcome): SAR ## Likelihood function: auto\_gaussian ## Link function: identity ## Residual Moran Coefficient: -0.0555795 ## WAIC: 224.16 ## Observations: 162 ## Data models (ME): none## Inference for Stan model: foundation.## 4 chains, each with iter=1000; warmup=500; thin=1; ## post-warmup draws per chain=500, total post-warmup draws=2000.## ## mean se\_mean sd 2.5% 20% 50% 80% 97.5% n\_eff Rhat## intercept 0.017 0.003 0.121 -0.213 -0.081 0.015 0.110 0.259 1374 1.001## sx 0.584 0.002 0.062 0.462 0.533 0.586 0.636 0.706 1550 1.001## sar\_rho 0.703 0.002 0.057 0.577 0.656 0.708 0.753 0.801 1224 1.000## sar\_scale 0.438 0.001 0.027 0.391 0.415 0.436 0.461 0.496 1541 1.000## ## Samples were drawn using NUTS(diag\_e) at Thu Aug 1 12:33:49 2024.## For each parameter, n\_eff is a crude measure of effective sample size,## and Rhat is the potential scale reduction factor on split chains (at ## convergence, Rhat=1). Predicted valuesWe can visualize the model results by plotting the expected lifeexpectancy across the full range of GDP per capita. We use the predictfunction for this. As input, it requires our fitted model and adata.frame with covariate values.

We will start by creating a data.frame with GDP per capita values thatspan from the minimum to maximum values in the world data:

gdp <- range(world$gdpPercap)min\_gdp <- gdp[1]max\_gdp <- gdp[2]pdf <- data.frame(gdpPercap = seq(min\_gdp, max\_gdp, length.out = 200)) The column names in this data.frame have to match the variable namesthat were present in the data that we first provided to the model. Inthis case, the name of the columns should match those from the worlddata. Likewise, we provide the new GDP data on its original(un-transformed) scale, just as we did when we fit the model usingstan_sar (the log transformation will be applied by predict becauseit is specified in the model formula). Because we centered the covariateusing the centerx = TRUE argument, we will also allow the predictfunction to handle the centering automatically using information that isstored with the fitted model (stan_sar$x_center).

Now we pass this new data to predict:

preds <- predict(fit\_sar, newdata = pdf) The output includes our pdf data plus some new columns. The newcolumns provide a summary of the predicted values. As usual, the meanis the estimate and the estimate is accompanied by other values that canbe used to taken as credible intervals for the predicted value. Theoutput reflects uncertainty in the model parameter estimates.

head(preds)## gdpPercap mean sd 2.5% 20% 50% 80% 97.5%## 1 597.1352 60.10281 1.525231 57.31512 58.81438 60.02981 61.40785 63.23217## 2 1201.4715 62.90912 1.324012 60.47457 61.78458 62.84984 64.02423 65.60880## 3 1805.8079 64.54460 1.222436 62.29812 63.50132 64.50656 65.56321 67.07337## 4 2410.1442 65.70331 1.159626 63.56161 64.71873 65.67025 66.65184 68.13579## 5 3014.4805 66.60137 1.117121 64.51219 65.64677 66.57454 67.52918 68.94410## 6 3618.8169 67.33477 1.086904 65.28404 66.40366 67.29468 68.21982 69.56564 These ‘predicted’ values represent the expectation of the outcomevariable at the given level of the covariates. So we would expect actualobservations to form a cloud of points around the ‘predicted’ values. Tocalculate these predicted values, the predict function only includescovariates and the intercept, it does not include any spatialautocorrelation components. Its purpose is to examine implications ofthe coefficient estimates on recognizable scales of variation, not topredict values for particular places. (The log-linear model can also beinterpreted in terms of percent changes in the covariate, such as ’a 10%increase in GDP per capita, e.g., from 10,000 to 11,000, is associatedwith around 4 * log(11/10) = 0.38 additional years of life expectancyon average.)

```

scale GDPpreds <- transform(preds, gdpPercap = gdpPercap / 1e3)yrange <- c(min(preds$`2.5%`), max(preds$`97.5%`))plot(preds$gdpPercap, preds$mean, t = 'l', ylim = yrange, axes = F, xlab = "GDP per capita ($1,000s)", ylab = "Life expectancy")axis(1)axis(2)# add credible intervalslines(preds$gdpPercap, preds$`2.5%`, lty = 3)lines(preds$gdpPercap, preds$`97.5%`, lty = 3)

``` Per this dataset, about 50% of the world population lives in countrieswith GDP per capita below $12,300.

Future work and supportYou can submit any questions, requests, or issues on the package issuespage or thediscussionspage. geostanis still actively being developed so users are encouraged to check thepackage news page forupdates.

If you are interesting contributing to the package you are encouraged tosend an e-mail to the author or use the discussions page. You can submita pull request with any bug fixes. Contributions that would make thepackage more useful to fields other than geostan’s current focus(human geography and public health), such as ecology, would beespecially welcome.

Appendix ```

function for getting colors, breaks, and labels for mappingmap_pars <- function(x, brks = quantile(x, probs = seq(0, 1, by = 0.2), na.rm = TRUE), cols = c("#A8554EFF", "gray95", "#5D74A5FF")) { # put x values into bins x_cut <- cut(x, breaks = brks, include.lowest = TRUE) # labels for each bin lbls <- levels( cut(x, brks, include.lowest = TRUE) ) # colors rank <- as.numeric( x_cut ) max_rank <- max( rank , na.rm = TRUE ) pal_fun <- colorRampPalette( cols ) pal <- pal_fun( max_rank ) colors <- pal[ rank ] # return list ls <- list(brks = brks, lbls = lbls, pal = pal, col = colors) return( ls )}

``` ReferencesChun, Yongwan, and Daniel A Griffith. 2012. “Spatial Statistics andGeostatistics: Theory and Applications for Geographic InformationScience and Technology.”

Donegan, Connor. 2021. “Building Spatial Conditional Autoregressive(CAR) Models in the Stan Programming Language.” https://osf.io/3ey65/.

———. 2022. “Geostan: An R Package for Bayesian Spatial Analysis.”Journal of Open Source Software 7 (79): 4716.https://doi.org/10.21105/joss.04716.

———. 2024. “Plausible Reasoning and Spatial-Statistical Theory: ACritique of Recent Writings on ‘Spatial Confounding’.” GeographicalAnalysis Early view. https://doi.org/10.1111/gean.12408.

Gabry, Jonah, Ben Goodrich, Martin Lysy, and Andrew Johnson. 2024.Rstantools: Tools for Developing R Packages Interfacing with ’Stan’.https://CRAN.R-project.org/package=rstantools.

Griffith, Daniel A, and Pedro R Peres-Neto. 2006. “Spatial Modeling inEcology: The Flexibility of Eigenfunction Spatial Analyses.” Ecology87 (10): 2603–13.

Stan Development Team. 2023. Stan User’s Guide. https://mc-stan.org.

To leave a comment for the author, please follow the link and comment on their blog: r-spatial.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Spatial analysis with geostan

View Details

[This article was first published on R – Win Vector LLC, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Richard Layton recently shared a neat article: A subtle flaw in pull(). This is the usual loss of reliable programmable semantics just to avoid a few quote marks (at the cost of many more force eval and paste marks). It is well considered and well written. Please check it out!

To leave a comment for the author, please follow the link and comment on their blog: R – Win Vector LLC.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Pulling a Loose Thread on Pull()

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.PharmaRUG, China organizer Joe Zhu, spoke with the R Consortium about the growing R community and the increasing use of R in the pharmaceutical industry in China. The group has contributed to the pharmaceutical R community through several R packages. Since its establishment last year, the group has organized large-scale hybrid events. Joe also shared some tools and techniques for smoothly organizing and running hybrid events.

Please share about your background and involvement with the RUGS group.

I have a PhD in statistics and studied in New Zealand for my undergraduate and postgraduate degrees in statistics. My PhD work focused on theoretical coalescent theory and probabilistic modeling for phylogenetics models. I also completed a postdoc at Oxford, focusing on statistical genomics for the human genome and malaria parasite genome projects. During this time, I developed open source software tools for statistical genomics, primarily using R as a front end and developing C++ software.

For the past four years, I’ve worked at Roche, where I started leading a major collaboration initiative in pharma three years ago. I’ve created TLG (table, listing, and figures) for regulatory submissions to the FDA. Throughout this initiative, we have open sourced around 30 software packages, including formatters, rtables, rlistings and tern. Last year, we submitted these packages to CRAN.

At first, we open sourced the project on GitHub and then submitted it to CRAN. I’m heavily involved in one of China’s R user groups, PharmaRUG. We use the group to share posts about developments in the area, and we organize events and conferences. In March last year, we hosted the first event with over 100 people on-site and around 100 online. The event covered topics like R package usage in the pharma industry. Later that year, we organized another event called “Open Source Clinical Reporting summeR“.

Lately, I have been busy organizing several events. I recently gave a talk (about R package dependencies as directed in acyclic graphs) at a conference hosted by the R community in China. Early next month, on August 1st, I will attend a pharma conference where I will conduct a workshop on good practices in software package development. The conference schedule is quite packed for me as I also have a session on how teams operate and collaborate within the Pharma industry to develop R packages. On the third day of the conference, I will organize a series of 11 data visualization talks, one of which is about Python. Most of the talks will focus on using R, except for one discussion on Python.

Can you share what the R community is like in China?

We have opened up seats for students to join our events in the pharmaceutical industry. In the past, fewer than 20 students, mostly from academia, have joined us for these conferences. The events include big names like Roche, Johnson & Johnson, Novartis, Boehringer Ingelheim, and Sanofi and local companies such as Fosun, Hengrui, and Legend Biotech. There is a big R community in China across academia and industry. Our user group primarily focuses on the pharma industry. Our WeChat channel has nearly a thousand subscribers, and our group chat has almost 500 members. It’s a very active community.

Later this year, we will collaborate with the “R in Pharma” for the October conference. Daniel Sabanes Bove and I have contacted Harvey and Phil, and we will organize an APAC track, including India, China, Japan, Australia, Singapore, and Korea.

Any techniques you recommend using for planning for or during the event? (Github, zoom, other) Can these techniques be used to make your group more inclusive to people that are unable to attend physical events in the future?

We have created a GitHub account called PharmaRUG. We use this platform to share websites, posts, slides, and videos related to our events. The Pharma RUG 2024 conference was particularly successful this year, thanks to the support from the R Consortium. We also utilize WeChat groups to call for speakers and interact with others. In addition to GitHub and WeChat, we use Tencent Docs to share documents. This is particularly useful in China, where using company-specific platforms like Google or Microsoft can be hindered by firewalls. Tencent Docs works perfectly in China, making sharing and synchronizing documents easy.

Can you share some valuable tips for organizing succesful Hybrid events?

We have a series of planning sessions where we actively communicate using WeChat. We meet at a community center where everyone is open, and we have preset meetings. We test the audio and everything beforehand. This is our second year organizing these events, so we have gained more experience. We are now familiar with the standards and know what needs to be done. For example, when two companies, like MNCs, use different systems, we find it better to use one shared system to ensure everything is synchronized.

We’ve found that Microsoft Teams is easy to use for setting up meetings and scheduling them ahead of time. For live demos, we recommend pre-recording the demos and taking questions. In the case of hybrid sessions with multiple locations, we prioritize asking and answering questions based on the primary and secondary locations, as well as online participation. If we cannot answer questions quickly, we host Q&A sessions afterward and share them online.

I believe that for the event to be successful, timing is crucial. We must stick to the schedule because it’s a hybrid event. However, we should also allow for some flexibility when unexpected things come up. We haven’t created a YouTube account yet because YouTube isn’t accessible in China. One alternative could be setting up a Bilibili web page and account to share the videos. All our files are currently on GitHub, which is convenient. We need to trim the videos to smaller sizes to fit GitHub’s file size limits, maybe at four and a half speeds or similar.

What trends do you currently see in R language and your industry?

So, SAS has dominated the software space for the Pharma industry for decades. While it used to be used for exploratory and research purposes, there have been successes with using Office to support missions in recent years. Roche also has success stories in this area. There are several initiatives, with PharmaVerse being a significant player. Roche is part of PharmaVerse, taking inspiration from the tidyverse multiverse concept. The end-to-end clinical reporting process is considered in this space, from data preparation to TLG generations. A lot has happened in the past three to four years, especially in China last year. There’s been significant development in China, and you can see a shift from SAS to R in the tools used. At the PharmaSUG meeting, which was previously dominated by SAS users, in the past few years, a quarter to one-third of the tools are using languages other than SAS. It’s clear that things are moving away from SAS towards software languages like R.

This year, I don’t have the complete statistics with me right now, but you do see a lot of topics. In my session, I’m sharing, and you know, many talks use visualization because it’s much likable. So, the trend is that R is becoming more acceptable than before, from PLCs to things in production. There are very high standards for codes and validation.

In the end, I would like to thank my dear friends and colleagues for their support and for making this happen

Yan Qiao, Associate Director of Scientific Programming, Beigene,
Baoqin Li, China Head Clinical & Statical Programming, Johnson & Johnson
Dong Guo, China head of Stats Analyst, Eli-lilly&Company
Yun Ma, Director, clinical data Sciences, Boehringer Ingelheim (china) Investment Co.
Yanli Chang, Head of Data Operations China, Novartis

How do I Join?

R Consortium’s R User Group and Small Conference Support Program (RUGS) provides grants to help R groups organize, share information, and support each other worldwide. We have given grants over the past four years, encompassing over 68,000 members in 33 countries. We would like to include you! Cash grants and meetup.com accounts are awarded based on the intended use of the funds and the amount of money available to distribute.

Learn moreThe post Pharma RUG: The Rise of R in China’s Pharmaceutical Industry appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Pharma RUG: The Rise of R in China’s Pharmaceutical Industry

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Filter a Vector in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Filter a Vector in R is a fundamental skill that can be applied to a wide range of data analysis tasks.

In this article, we’ll explore four different methods for filtering a vector in R, along with practical examples.

Method 1: Filter for Elements Equal to Some Value

The first method involves filtering a vector for elements that are equal to a specific value. This can be achieved using the == operator.

```

filter for elements equal to 8x[x == 8]

`` For example, let’s create a vectorx` and filter it for elements that are equal to 8:

Separate a data frame column into multiple columns-tidyr Part3 (datasciencetut.com)

```

create vectorx <- c(1, 2, 2, 4, 6, 8, 8, 8, 102, 150)#filter for elements equal to 8x[x == 8][1] 8 8 8

``` Method 2: Filter for Elements Based on One Condition

The second method involves filtering a vector based on a single condition. This can be achieved using the < or > operator.

```

filter for elements less than 8x[x < 8]

`` For example, let’s create a vectorx` and filter it for elements that are less than 8:

```

create vectorx <- c(1, 2, 2, 4, 6, 8, 8, 8, 102, 115)#filter for elements less than 8x[x < 8][1] 1 2 2 4 6

``` Method 3: Filter for Elements Based on Multiple Conditions

The third method involves filtering a vector based on multiple conditions. This can be achieved using the | operator.

Step-by-Step Data Science Coding Course

```

filter for elements less than 8 or greater than 12x[(x < 8) | (x > 12)]

`` For example, let’s create a vectorx` and filter it for elements that are less than 8 or greater than 12:

x <- c(1, 2, 2, 4, 6, 8, 8, 8, 12, 15) Method 3: Filter for Elements Based on Multiple Conditions

The third method involves filtering a vector based on multiple conditions. This can be achieved using the | operator.

```

filter for elements less than 8 or greater than 12x[(x < 8) | (x > 12)]

`` For example, let’s create a vectorx` and filter it for elements that are less than 8 or greater than 12:

```

create vectorx <- c(1, 2, 2, 4, 6, 8, 8, 8, 12, 15)#filter for elements less than 8 or greater than 12x[(x < 8) | (x > 12)][1] 1 2 2 4 6 15

``` Method 4: Filter for Elements in the List

The fourth method involves filtering a vector for elements that are in a list. This can be achieved using the %in% operator.

```

filter for elements equal to values in listx[x %in% c(2, 6, 12)]

`` For example, let’s create a vectorxand filter it for elements that are equal to values in the listc(2, 6, 12)`:

```

create vectorx <- c(1, 2, 2, 4, 6, 8, 8, 8, 12, 15)#filter for elements equal to values in listx[x %in% c(2, 6, 12)][1] 2 2 6 12

``` ConclusionThese four methods provide a comprehensive overview of how to filter a vector in R.

By mastering these techniques, you’ll be able to extract specific subsets of data from your vectors and analyze them more effectively.

Is It Difficult to Learn Data Science
Data Manipulation Techniques with dplyr
COUNTIF Function in R
Kendall’s Rank Correlation in R-Correlation Test
Calculate Confidence Intervals in R
Check if the Column Contains a String or not
How to calculate Power Regression in R (Step-by-Step Guide)
Most Winning Numbers in Kerala Lottery
How to Choose Appropriate Clustering Method for Your Dataset

The post Filter a Vector in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Filter a Vector in R

View Details

[This article was first published on Layton R blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. SummaryIn the current version of dplyr, if x is not a column name in data frame d, then pull(d, x) attempts to look up the value of x in the environment instead of returning NULL or an error. There are ways to augment pull() to yield the expected results, though base R alternatives [[ and $ may also be used reliably and predictably.

A current dplyr article, “Programming with dplyr”, states that the pull() function, like select(), uses “tidy selection” for working with data frame columns (Wickham et al., 2023). Accordingly, the select() documentation includes <tidy-select> in its arguments list—but the pull() documentation (unexpectedly) does not.

The thumbnail image Pull by Jeremy Brooks (2014) is licensed under CC BY-NC 2.0.

This may be simply a minor documentation error, but close examination reveals a subtle flaw in pull() that users should be aware of—if column name x does not exist in data frame d, then pull(d, x) unexpectedly attempts to operate on the name x in the environment, if present, instead of returning a NULL or an error.

In this post, I compare the behavior of pull() to that of [[, $, and subset() for extracting a single column x from a data frame when column x exists and when it does not, while the environment contains another name x and the value it references.

To distinguish between a name in the environment and the value(s) it references and the name of a data frame column and the value(s) it references, I borrow some terminology from the dplyr article cited above:

env-variables live in an environment, usually created with <-.
data-variables live in a data frame.

For example, in the following code chunk, I create a env-variable, df, that contains two data-variables, x and y. I then use $ to extract the data-variable x from the env-variable df.

df <- data.frame(x = c("ex", "ex"), y = c("why", "why"))df#> x y#> 1 ex why#> 2 ex whydf$x#> [1] "ex" "ex" For reference, the R and package versions I’m using are as follows.

R.version$version.string#> [1] "R version 4.4.1 (2024-06-14 ucrt)"library("dplyr")packageVersion("dplyr")#> [1] '1.1.4'packageVersion("tidyselect") # for all\_of()#> [1] '1.2.1'packageVersion("magrittr") # for %>%#> [1] '2.0.3'packageVersion("rlang") # for !!#> [1] '1.1.4' Extracting a single columnI start each example with two assignments: the name d referencing the data frame created above; and the name x, referencing a string value.

d <- dfx <- "y" For extracting a single column as a vector from a data frame, the following are roughly equivalent. Each should extract the data-variable x from the env-variable d.

d[["x"]] d$x pull(d, x) Using [[ or $The base R operator [[ matches a character value to the column names of a data frame and returns the matching column, if any, as a vector. No partial matching is the default behavior, that is, d[["x"]] is equivalent to d[["x", exact = TRUE]]. The return is the expected data-variable x in vector form.

d[["x"]]#> [1] "ex" "ex" The base R operator $ is similar: d$name is equivalent to d[["name", exact = FALSE]] (partial matching enabled). In this case, partial matching is not relevant; we extract the expected column.

d$x#> [1] "ex" "ex"identical(d[["x"]], d$x)#> [1] TRUE If name x does not exist in d,

d <- select(df, -x) both [[ and $ return NULL.

d[["x"]]#> NULLd$x#> NULL Both [[ and $ produce reliable and easily predictable results. As expected in the examples above, the env-variable x and the value it references, "y", are irrelevant to [[ and $. This turns out (sometimes) not to be the case with pull().

Using pull()Reset. (Binding the name x to the value "y" again is unnecessary, but I repeat the assignment with each reset just to remind us that the environment contains this name and value.)

d <- dfx <- "y" dplyr::pull() extracts a column as a vector, similar to $. Here, the data-variable x is correctly extracted from d.

pull(d, x)#> [1] "ex" "ex" If name x does not exist in d,

d <- select(df, -x) then, like $, we expect pull() to return NULL (or error)—but it doesn’t.

pull(d, x)#> [1] "why" "why" pull(), not finding the data-variable x in d, has unexpectedly operated on the env-variable x and used its value to pull the y data-variable from d, exactly as if we had written pull(d, y)

identical(pull(d, x), pull(d, y))#> [1] TRUE —behavior surely contrary to a user’s expectations. In general, one expects such behavior only when deliberately using syntax designed to use env-variables to extract data-variables from data frames (the topic of the section below on programming safely).

To borrow a conclusion from John Mount (2018), the unfortunate coincidence that the name x has a value in the environment should be irrelevant to pull().

Some backgroundI was re-reading John Mount’s opinion/tutorial piece (cited above) that demonstrated that dplyr::select() at the time had the same sort of flaw as the one I discuss in this post. Running Mount’s examples today show that the flaw in select() has since been corrected.

I had been working with pull() in another context and John’s article prompted me to compare standard evaluation (SE) and non-standard evaluation (NSE) approaches to the task for which pull() is designed, inspiring me to write this post.

Mount also showed that base R subset() is known to have a similar mal-feature. To illustrate, I set up subset() to extract one column as a vector.

Reset.

d <- dfx <- "y" The argument drop = TRUE yields a vector when a single column is selected.

subset(d, select = x, drop = TRUE)#> [1] "ex" "ex" If name x does not exist in d,

d <- select(df, -x) then, like pull(), the column y is returned instead the expected NULL or error.

subset(d, select = x, drop = TRUE)#> [1] "why" "why" However, the subset() documentation does include the following warning about the potential for “unanticipated consequences” of subset’s non-standard evaluation (NSE) interface:

Warning

This is a convenience function intended for use interactively. For programming it is better to use the standard subsetting functions like [, and in particular the non-standard evaluation of argument subset can have unanticipated consequences.

The pull() documentation does not include such a warning.

Using pull() safelyTo use pull() safely to extract column x, we have three forms (at least) that currently return the expected results, including NULL or an error when the column doesn’t exist.

Reset.

d <- dfx <- "y" 1. Quote the data-variable name.

pull(d, "x")#> [1] "ex" "ex" 2. Use the all_of() selection helper with the column name in quotes.

pull(d, all\_of("x"))#> [1] "ex" "ex" 3. Use the .data pronoun with $.

pull(d, .data$x)#> [1] "ex" "ex" If name x does not exist in d,

d <- select(df, -x) all three forms ignore the env-variable x and return the expected errors.

pull(d, "x")#> Error in `pull()`:#> ! Can't extract columns that don't exist.#> ✖ Column `x` doesn't exist.pull(d, all\_of("x"))#> Error in `pull()`:#> ℹ In argument: `all\_of("x")`.#> Caused by error in `all\_of()`:#> ! Can't subset elements that don't exist.#> ✖ Element `x` doesn't exist.pull(d, .data$x)#> Error in `pull()`:#> ℹ In argument: `x`.#> Caused by error in `.data$x`:#> ! Column `x` not found in `.data`. The safer syntax may be inconvenient enough to defeat the purpose of non-standard evaluation in the first place: being able to type column names without quotation marks and looking nice in pipes. For the moment, the alternatives [[ or $ may be more attractive.

Of course, like subset(), one can treat pull() as a convenience function best used interactively where the existence of the desired column can be confirmed before pulling.

Programming safelyWhen programming, it is often useful to have an env-variable that references a character vector populated with column names you expect a function to find and operate with. In the examples below, we use the env-variable x in function arguments to pull the column specified by its value, in this case, "y".

Reset.

d <- dfx <- "y" Create function f to operate on the env-variable var using square brackets [[ to extract the column specified by the value of var. The y column is returned as desired.

f <- function(dframe, var) { dframe[[var]]}f(d, x)#> [1] "why" "why" Function g using pull() and the all_of() selection helper yields a similar result.

g <- function(dframe, var) { dframe %>% pull(all\_of(var))}g(d, x)#> [1] "why" "why" Function h using pull() and the .data pronoun also yields the desired result.

h <- function(dframe, var) { dframe %>% pull(.data[[var]])}h(d, x)#> [1] "why" "why" As does function q using the rlang injection operator !!.

q <- function(dframe, var) { dframe %>% pull(!!var)}q(d, x)#> [1] "why" "why" And lastly, if name x does not exist in d,

d <- select(df, -y) all four functions return a NULL or an error.

f(d, x)#> NULLg(d, x)#> Error in `pull()`:#> ℹ In argument: `all\_of(var)`.#> Caused by error in `all\_of()`:#> ! Can't subset elements that don't exist.#> ✖ Element `y` doesn't exist.h(d, x)#> Error in `pull()`:#> ℹ In argument: `y`.#> Caused by error in `.data[["y"]]`:#> ! Column `y` not found in `.data`.q(d, x)#> Error in `pull()`:#> ! Can't extract columns that don't exist.#> ✖ Column `y` doesn't exist. ConclusionIt appears to be an oversight that pull() attempts to operate on an env-variable if the intended data-variable doesn’t exist. Workarounds exist though base R alternatives [[ and $ may also be used reliably and predictably.

ReferencesMount, J. (2018). A subtle flaw in some popular R NSE interfaces. Blog post. https://win-vector.com/2018/09/23/a-subtle-flaw-in-some-popular-r-nse-interfaces/Wickham, H., François, R., Henry, L., Müller, K., & Vaughan, D. (2023). dplyr: A grammar of data manipulation. https://CRAN.R-project.org/package=dplyr To leave a comment for the author, please follow the link and comment on their blog: Layton R blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A subtle flaw in pull()

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionAs a blogger who uses R for content creation, I’ve found it incredibly useful to automate some of the repetitive tasks. One such task is creating Quarto Markdown (QMD) files for new blog posts. To simplify this, I’ve added a custom R function that not only creates the necessary file structure. Let’s take a look at this function and how you can integrate it into your own workflow.

The FunctionHere’s the function I’ve been using:

create\_qmd\_file <- function(filename = "index.qmd", title = "", date = Sys.Date(), categories = c("code", "rtip")) { # Define the base path base\_path <- paste0(getwd(),"/posts") # Convert date to string and create the full directory path date\_str <- as.character(date) full\_path <- file.path(base\_path, date\_str) # Create the directory if it doesn't exist if (!dir.exists(full\_path)) { dir.create(full\_path, recursive = TRUE) message("Directory created: ", full\_path) } # Define the full file path file\_path <- file.path(full\_path, filename) # Create the content to be written in the file content <- paste0( "---\n", 'title: "', title, '"\n', 'author: "Steven P. Sanderson II, MPH"\n', 'date: "', date\_str, '"\n', "categories: [", paste(categories, collapse = ", "), "]\n", "toc: TRUE\n", "---\n\n", '<script src="https://giscus.app/client.js"\n', ' data-repo="Yours\_Here"\n', ' data-repo-id="Yours\_Here"\n', ' data-category="Comments"\n', ' data-category-id="Yours\_Here"\n', ' data-mapping="url"\n', ' data-strict="0"\n', ' data-reactions-enabled="1"\n', ' data-emit-metadata="0"\n', ' data-input-position="top"\n', ' data-theme="dark"\n', ' data-lang="en"\n', ' data-loading="lazy"\n', ' crossorigin="anonymous"\n', ' async>\n', '</script>\n' ) # Write the content to the file writeLines(content, file\_path) message("QMD file created: ", file\_path)} How It Works1. Setting the Base Path: The function starts by defining the base path where blog posts will be stored, appending "/posts" to the current working directory. This centralizes all posts in one location. 2. Creating Directories: It then converts the date to a string and uses it to create a directory path. If this directory doesn’t exist, the function creates it. This helps in organizing posts by date. 3. File Path Definition: The function then defines the full path for the QMD file, defaulting the filename to “index.qmd” if none is provided. 4. Content Creation: The main content for the QMD file is generated next. This includes a YAML front matter section with metadata like title, author, date, and categories. The function also adds a script for Giscus, which handles the comments section. 5. File Writing: Finally, the function writes the generated content to the specified file path and informs you that the file has been created.

Automating with .RprofileTo make this function available every time you start your project, you can use the .Rprofile file. This file is sourced whenever you start a new R session, making it perfect for setting up your environment.

Here’s the relevant .Rprofile setup:

source(paste0(getwd(),"/create\_qmd\_file.R")) By sourcing the create_qmd_file.R script, the function is loaded automatically, so you don’t have to manually source it each time.

Give It a Try!This function has saved me a lot of time, and I encourage you to try something similar if you’re managing content or projects with R. You can customize the function to suit your specific needs, whether that’s adjusting the metadata, changing the file structure, or adding other automated features. It’s a great way to streamline your workflow and focus more on the creative aspects of your work.

If you have any questions or run into any issues, feel free to reach out. Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Automate Your Blog Workflow with a Custom R Function: Creating QMD Files

View Details

[This article was first published on R on Nicola Rennie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In September 2023, I wrote a blog post aboutcreating typewriter-styled maps in {ggplot2}. It described the process of creating an elevation map where, instead of using colours to denote the different elevation levels, different letters of the alphabet were used. By choosing the correct font, it gives the impression that the map was created using a typewriter. In this blog post, I’ll walk you through the same process to create a typewriter-styled image (instead of an elevation map).

Image processing in RWe’re going to start with a normal image and transform it into a typewriter-styled image, so the first thing we need to do is make sure we can work with image files in R. There are two packages that tend to be the go-to packages for simple image processing in R. The{magick} package provides bindings to the ImageMagick image processing library and the{imager} package is based onCImg, a C++ library by David Tschumperlé. Both packages have their strengths, and it’s easy to use both at the same time via the cimg2magick() and magick2cimg() conversion functions in {imager}. In this blog post, we’ll use the {imager} package but you could definitely do something similar using the {magick} package instead.

Let’s decide which image to use. I’d recommend you don’t start with an insanely high resolution image – simply to reduce the processing time when you’re first trying out how functions work! It doesn’t matter what orientation or aspect ratio the image has. Let’s use the following photo of a bridge:

(bonus points if you know where this is!)

Loading image filesLet’s start by loading the {imager} package. We can then load an image into R using the load.image() function, where you simply provide the file path to the image. It works with PNG, JPG, and BMP images out of the box.

| 12 | library(imager)img <- load.image("image.jpg") |

Note: if you run plot(img), you’ll see the image plotted on standard plot axes in your graphics pane.

Rescaling imagesLet’s start by rescaling the size of the image. If you print the img object, you’ll see the original size of the image in pixels:

| 1 | Image. Width: 4032 pix Height: 3024 pix Depth: 1 Colour channels: 3 |

For our plot of the image, each pixel will be represented by a letter rather than a small coloured square. There are currently 12,192,768 (4032*3024) pixels - that’s a lot of letters! Let’s reduce this number (essentially making the image more pixelated) using the resize() function. We define a rescale variable that states how much smaller it will become. Here, we’ll use 20 but you might choose to use a different value depending on how closely you want your typewriter-styled image to represent the original image. We resize both the size_x and size_y by this rescale factor to maintain the original aspect ratio.

| 123456 | rescale <- 20img <- resize( im = img, size\_x = round(width(img) / rescale), size\_y = round(height(img) / rescale)) |

Now the image is 202 x 151 pixels, and if you run plot(img), you’ll see that the image looks a bit blurrier:

We also need an ordered variable that we’ll map the different letters to. This means that we need a single, continuous variable that we split into bins. You can think of the image as currently having three continuous variables since it’s a colour image: R, G, and B representing the amount of red, green, and blue in each pixel. We could pick just one of these to plot. However, it would make more sense to convert the image to black and white and use the luminance (brightness) of the pixel as the continuous variable.

We can use the grayscale() function to convert to a black and white image. By default, this returns an image with just the luminance data in it (and not the RGB data).

| 1 | img <- grayscale(img) |

Converting to a matrixWe want to extract the luminance values from img so that we can group and then plot them. The img object currently has a cimg class which isn’t very easy to work with if you don’t want to do any further processing of the image. Luckily, it can easily be converted to a matrix using the as.matrix() function. We’ll add some row and column names (based on the number of the row or column) to make it easier to convert to a tibble() in the next step.

| 123 | m <- as.matrix(img)colnames(m) <- 1:ncol(m)rownames(m) <- 1:nrow(m) |

Data processingNow that we have a numeric matrix, we’re essentially in the same place as we were when we had the elevation matrix in thecreating typewriter-styled maps in {ggplot2} blog. What we need to do now, is convert the matrix into a format we can use for plotting with {ggplot2} and map the numeric values to different letters.

Data wranglingHere we’ll use {tidyverse} functions for data processing, but you can also do these steps in base R if you prefer. We start by converting from a matrix to a tibble, and making the row names of the matrix into a column called x. This will be the x-coordinates of each letter we want to plot. We then pivot the data into long format - ending up with three columns: x, y, and value containing the x- and y- coordinates for each letter and the numeric value that will be represented by the letter. We also make sure that all three columns are actually numeric.

Choosing a fontBefore we go on to plotting, we need to decide on:

which font we are going to use
how many different letters we need
which letters those are

Here, we’ll use the Special Elite font available through Google Fonts, as we did in the typewriter map blog post. This font has a typewriter-look, and will work well for this because it’s a monospace font. Since it’s a Google Font, it’s also very easy to get it working in R using the{showtext} package.

We load the {showtext} package, and then pass the font name into the font_add_google() function. Running showtext_auto() and showtext_opts(dpi = 300) switches on the use of {showtext} for fonts, and specifies what resolution our plot will be in.

| 1234 | library(showtext)font\_add\_google("Special Elite")showtext\_auto()showtext\_opts(dpi = 300) |

Mapping values to lettersLet’s define a vector of which letters we’re going to use. We’ll use a lower case l, and upper case I, H, and M to denote four different levels of luminance from lowest to highest. When you look at how the characters are printed, M uses a lot of ink (and is a dark letter) whereas l uses very little (and is a light letter).

| 1 | chars <- c("l", "I", "H", "M") |

Let’s also create a lookup table of how our letters map to the different levels of luminance:

| 1234 | chars\_map <- data.frame( value = rev(seq\_len(length(chars))), value\_letter = chars) |

Our look up table looks like this:

| 12345 | value value\_letter1 1 l2 2 I3 3 H4 4 M |

Now let’s turn the continuous luminance data into four levels (1, 2, 3, and 4) using the ntile() function from {dplyr}. The ntile() function breaks the input vector into n buckets and returns an integer vector denoting which bucket each value falls into. We can then left_join() our bucketed luminance data to the chars_map look-up table we’ve already created.

Now we’re ready for plotting!

Plotting with {ggplot2}The plotting is probably the easiest part of this whole process. Let’s start with the ggplot() function (as we would almost any plot made with {ggplot2}). Then, we only really need to use geom_text()! We map the x and y values in the plot_df data to the x and y axes and specify that the value_letter should be used as the label inside the aes() call.

We also need to remember to use the family argument to apply our chosen font - previously loaded in as "Special Elite". Pick a colour of your choice - we’ll use the default black as you would see in a traditional typewriter! The size of the letters also needs to be adjusted to make sure they don’t overlap - this will probably take some trial and error, and it depends on the size of the image, and the rescale value that was chosen earlier.

| 123456789 | p <- ggplot() + geom\_text( data = plot\_df, mapping = aes(x = x, y = y, label = value\_letter), family = "Special Elite", colour = "black", size = 2.5 )p |

What you’ll notice immediately about this plot of the image is that it’s upside down. That’s because, in most traditional plots, the lowest values on the y-axis are at the bottom. Here, our y values represent row numbers and so the smallest values should be at the top. We can add scale_y_reverse() to flip the axis upside down. We also apply coord_fixed() to make sure our image doesn’t get squashed, setting expand = FALSE to remove the white space from around the edge.

Finally, we edit the theme to get rid of the grid lines you would be more likely to need on a bar chart. Using theme_void() removes all theme elements, and sets a transparent background. We can override this by changing the plot.background values in the theme() function - setting the background fill and border colour to white.

| 1 2 3 4 5 6 7 8 910 | p <- p + scale\_y\_reverse() + coord\_fixed(expand = FALSE) + theme\_void() + theme( plot.background = element\_rect( fill = "white", colour = "white" ) ) |

That looks better - if you zoom in really close, you’ll see that each pixel is indeed a letter!

We can also save a copy of our image to a file. We can save our typewriter-styled map in the same size our original image by extracting the width() and height() of the image object - making sure to change the units to pixels!

| 123456 | ggsave(p, filename = "font\_image.png", width = rescale * width(img), height = rescale * height(img), units = "px") |

Let’s compare the original image with the typewriter-styled version side-by-side:

From afar, it might just look like a pixelated, black and white version. But up close, viewers can be surprised by the fact that it’s actually individual letters!

Useful resourcesYou might be wondering what the point to all this is (other than some making some unusual prints for your home decor). There might not be a direct point, but it’s a fun way to learn about image processing in R and understand how values associated with images can be accessed and manipulated.

If you want to learn a little bit more about image processing in R:

Marco Gandolfo has written a blog post aboutImage processing in R which introduces some basic functions in both {imager} and {magick}.
Thedocumentation for the {magick} package has lots of examples you can try to get started.
If you’re looking for a package in R which does a specific type of image processing, have a look at theCRAN Task Views for Medical Imaging. It has a section for General Image Processing and though most have medical applications, many also work for non-medical image processing tasks.

To leave a comment for the author, please follow the link and comment on their blog: R on Nicola Rennie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Creating typewriter-styled images in R

View Details

[This article was first published on R on Stats and R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Motivation * Coarse data * Simulated example * Real data example * References MotivationA group of researchers from the Data Science Institute (DSI) at Hasselt University developed a new statistical model to estimate the incubation period of a pathogenic organism based on coarse data. The incubation period of an infectious disease (defined as the time elapsed between infection and the manifestation of first symptoms) is of great importance as it permits to shed light on the epidemic potential of a disease and to optimize the length of quarantine periods to freeze transmission. The article (Gressani et al. 2024) was recently published in the American Journal of Epidemiology* with practical implementation of the methodology accessible through the EpiLPS package (Gressani et al. 2022).

Coarse dataWhat makes estimation of incubation times so challenging in the first place? The devil lies in the data. True infection times are stealthy and rarely observed. In information-theoretic jargon this phenomenon is called “imperfect information’’ but statisticians prefer to call it censoring. To be more precise, infection times are interval censored, i.e. one part of the collected data contains exposure intervals (\mathcal{E}=[t^{E_L},t^{E_R}]) reported by individuals that are part of the study, where (t^{E_L}) and (t^{E_R}) stand for the left and right bound, respectively, of the exposure window. The other part of the data contains symptom onset times (t^{\mathcal{S}}). This is a more easily accessible piece of information -people tend to remember the day when first symptoms appeared- and so the timing of symptom onset is assumed to be exactly observed. Subtracting the exposure bounds from the symptom onset time, one obtains the incubation interval (\mathcal{I}=[t^{\mathcal{I}_L}, t^{\mathcal{I}_R}]) with lower bound (t^{\mathcal{I}_L}=t^{\mathcal{S}}-t^{E_R}) and upper bound (t^{\mathcal{I}_R}=t^{\mathcal{S}}-t^{E_L}), characterizing the coarse data structure which will be the main model input.

Simulated exampleIn EpiLPS, the estimIncub() routine is designed to compute an estimate of the incubation density based on the methodology of Gressani et al. (2024). Giving a detailed account of the methodology would be out of scope for this blog and the reader is redirected to the article for technicalities. In a nutshell, it is a Bayesian approach making use of (penalized) B-splines, Laplace approximations and Markov chain Monte Carlo (MCMC) methods to derive a semi-parametric estimate of the incubation density. An attractive feature of the estimIncub() routine for the end-user is the minimalistic input it requires to work, namely:

x: A data frame containing the lower and upper bound of the incubation interval.
K: An integer specifying the number of B-splines to smooth the incubation density.
niter: The number of MCMC samples required.

In practice, only the data frame x is required as the remaining inputs are assigned default values. Computationally, the routine requires a small amount of resources as costly subroutines are coded in C++ and integrated in R via the Rcpp package. The structure of x is quite simple. It should be a data frame with two columns containing the left bound (t^{\mathcal{I}_L}) of the incubation interval (in the first column) and the right bound (t^{\mathcal{I}_R}).

Let’s start with a simple example where x is simulated based on a data generating mechanism assuming a known incubation distribution. This can be achieved with the incubsim() routine in EpiLPS. We choose x to be generated according to a Lognormal incubation distribution with a mean of 5.5 days and a standard deviation of 2.1 days following Ferretti et al. (2020). Simulation of (n=40) observations with an average exposure window of 2 days is implemented as follows:

set.seed(2024)simdat <- incubsim(incubdist = "LogNormal", n = 40, coarseness = 2)gt(head(simdat$Dobsincub, 5))

| tL | tR | | --- | --- | | 3.724500 | 5.510304 | | 4.381377 | 6.654224 | | 4.588020 | 5.483614 | | 3.847835 | 6.003948 | | 3.522282 | 5.033487 |

By typing simdat$Dobsincub, the user has access to the generated incubation intervals (expressed in days), corresponding here to a data frame with two columns and (n=40) rows. This is the data frame that is injected in the estimIncub() routine:

fit <- estimIncub(x = simdat$Dobsincub, verbose = TRUE)## ----------------------------------------------------------------## Time elapsed: 1.339 seconds.## Fitted density is Log-Normal with meanlog=1.617 and sdlog=0.317.## Mean incubation period (days): 5.298 with 95% CI: 5.086-5.615.## 95th percentile (days): 8.484 with 95% CI: 8.121-9.089.## ---------------------------------------------------------------- The output in the R console can easily be interpreted. It tells us that the model chooses a Lognormal density fit for the incubation period with a mean of 5.3 days (95% CI: 5.0-5.6 days). More detailed summary statistics are accessible by typing fit$stats, such as the posterior standard deviation and additional percentiles. What happens under the hood? Basically, the model computes a semi-parametric fit to the data and compares it with classic parametric fits (Lognormal, Weibull and Gamma) used for incubation modeling. The candidate with the lowest Bayesian information criterion (BIC) wins the game and is finally selected (here the Lognormal distribution). The incubation windows and the fitted incubation density can be obtained by typing:

grid.arrange(plot(fit, typ = "incubwin"), plot(fit, type = "pdf"), nrow = 1) Real data exampleThe flexible Bayesian methodology is illustrated on SARS-CoV-2 symptom onset and exposure window data extracted from cases in Vietnam. The dataset was analyzed in Bui et al. (2020) and is publicly available on the GitHub repository provided in the article (https://github.com/longbui/Covid19IncubVN; last accessed July 17, 2024). The dataset contains information about (n=19) cases identified from January 23, 2020 to April 13, 2020. After continuity corrections (required to change calendar dates into continuous time points), the left and right incubation bounds are given by:

```

Left incubation boundtL <- c(0.504, 2.983, 5.343, 6.969, 6.990, 2.570, 6.870, 1.263, 0.693, 1.869, 1.151, 2.748, 1.209, 1.161, 4.982, 4.017, 2.170, 9.805, 1.659)# Right incubation boundtR <- c(1.491, 7.762, 11.777, 12.821, 11.359, 8.551, 24.954, 4.144, 6.408, 7.083, 4.191, 10.326, 9.942, 7.254, 12.690, 10.825, 14.478, 16.592, 18.649)# Data setdataVietnam <- data.frame(tL = tL, tR = tR)

`` TheestimIncub()routine provides a Weibull fit with a mean incubation period of 6.7 days (95% CI: 5.8-7.4 days) and the standard deviation (extracted by typingincubfit$stats`) is 3.3 days (95% CI: 3.0-3.8 days). A figure of the incubation windows and the fitted Weibull density (with 95% credible interval) is also provided.

set.seed(2024)incubfit <- EpiLPS::estimIncub(x = dataVietnam, verbose = TRUE, tmax = 25)## ----------------------------------------------------------------## Time elapsed: 1.13 seconds.## Fitted density is Weibull with shape=2.142 and scale=7.521.## Mean incubation period (days): 6.661 with 95% CI: 5.753-7.375.## 95th percentile (days): 12.552 with 95% CI: 11.931-14.087.## ----------------------------------------------------------------grid.arrange(plot(incubfit, typ = "incubwin"), plot(incubfit, type = "pdf"), nrow = 1) ReferencesGressani, O., Torneri, A., Hens, N. and Faes, C. (2024). Flexible Bayesian estimation of incubation times. American Journal of Epidemiology (Accepted manuscript). https://doi.org/10.1093/aje/kwae192

Gressani, O., Wallinga, J., Althaus, C. L., Hens, N. and Faes, C.(2022). EpiLPS: A fast and flexible Bayesian tool for estimation of thetime-varying reproduction number. PLoS Comput Biol 18(10):e1010618. https://doi.org/10.1371/journal.pcbi.1010618

Eddelbuettel, D. and Francois, R. (2011). Rcpp: Seamless R and C++ Integration. Journal of Statistical Software, 40(8), 1–18. https://doi.org/10.18637/jss.v040.i08

Ferretti, L. et al. (2020). Quantifying SARS-CoV-2 transmission suggests epidemic control withdigital contact tracing. Science 368, eabb6936. https://doi.org/10.1126/science.abb6936

Bui LV, Nguyen HT, Levine H, Nguyen HN, Nguyen T-A, Nguyen TP, et al. (2020) Estimation of the incubation period of COVID-19 in Vietnam. PLoS ONE 15(12): e0243889. https://doi.org/10.1371/journal.pone.0243889

To leave a comment for the author, please follow the link and comment on their blog: R on Stats and R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: EpiLPS for estimation of incubation times

View Details

[This article was first published on R Archives - Remix Institute, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

“We can only see a short distance ahead, but we can see plenty there that needs to be done.”

― Alan Turing

AI virtual assistants have become indispensable tools for both personal and professional settings. While Siri and Alexa are household names, the latest advancements in AI-powered virtual assistants offer capabilities that far surpass these early pioneers.

Remix Institute’s founder, Douglas Davila-Pestana, had a good point when he posted on X: “Even though Amazon sold millions of Alexa devices, I’ve never seen anyone actually use an Alexa.”

Alexa used to be so popular that it became a go-to Christmas gift, often chosen when people had no idea what else to give. Now it’s an obsolete virtual assistant in the age of smarter AI assistants like ChatGPT and LegalMente AI’s Para.

So, let’s explore a few innovative AI virtual assistants on the market today that make Siri and Alexa look like children’s toys.

The New Wave of Virtual AssistantsVirtual assistants today are not just about setting reminders or playing music. They are integrated into sophisticated workflows, assisting in complex tasks and providing specialized support across various industries.

Healthcare: Personalized Patient CareOne of the most promising advancements is in healthcare. AI-powered virtual assistants are now capable of managing patient records, scheduling appointments, and even providing preliminary diagnoses based on patient symptoms. For instance, Docus AI Doctor offers personalized health guidance with 24/7 availability. It provides diagnoses, treatment options, and health reports based on user conversations. The AI is backed by natural language models and validated by over 300 top doctors, empowering individuals to manage their health concerns conveniently and helping those who have difficulty accessing in-person healthcare.

Financial Management: Streamlining Expense ManagementIn the realm of financial management, AI-powered virtual assistants are revolutionizing how businesses handle expenses, and Fyle stands out as a top recommendation for AI-driven expense management. This versatile assistant, compatible with iOS, Android, and desktop devices, offers advanced features such as an automatic data extraction engine that effortlessly organizes expense data from receipts and invoices, a custom approval hierarchy for streamlined expense approvals, and a comprehensive expense audit trail ensuring transparency and compliance, making it one of the leading solutions for intelligent expense management.

Digital Management: Pioneering Personalized AIDelphi AI and AI.XYZ are transforming personalized AI assistance. Delphi AI creates interactive, personalized digital replicas or “clones,” encapsulating individuals’ knowledge and personality, allowing users to engage dynamically with their audience across platforms like websites and social media. Ideal for influencers, educators, and business leaders, Delphi’s clones provide tailored responses and robust analytics for refining content strategy and monetization. Users maintain control over interactions and data, ensuring security and customization. Meanwhile, AI.XYZ offers personalized AI assistance by learning about users’ lifestyles to provide proactive support, enhancing everyday activities such as planning, communication, and personal wellness for a seamless and efficient user experience.

Case Study: Para – LegalMente AI’s AI Paralegal AssistantLegalMente AITM uses artificial intelligence to reduce the cost of legal work for small businesses, startups, healthcare, and individuals. LegalMente AI’s Para, is an AI Paralegal assistant that exemplifies the next generation of virtual legal assistants.

Features of Para:Legal Question Answering:Para can provide accurate and timely answers to legal questions so you don’t have to spend hundreds of dollars to ask a lawyer.

Business Insights:Para offers valuable insights to support informed business decisions.

Document Analysis:Para efficiently analyses contracts, data files, and various legal documents, regardless of file format.

Business Formation Assistance:Para helps guide users through the process of forming a business in the US such as an LLC or Corporation.

Impact of Para:Para is fine-tuned with specialized expertise in the legal domain and guardrailed to prevent hallucinations. It’s more reliable to ask legal questions to Para compared to Google Gemini or ChatGPT. Para also maintains political neutrality and will politely decline to answer any questions related to politics, ensuring focused and unbiased legal assistance. Para assists small businesses, startups, healthcare providers, and individuals with their legal work without the burden of huge legal bills.

Meet Para, your free AI Paralegal assistant.

ConclusionAI-powered virtual assistants have moved beyond basic functionalities, such as playing music or telling the weather, to become critical tools in various domains. Whether it’s healthcare, digital clones, customer service, or legal assistance, these advancements are making tasks easier, more efficient, and more personalized. As AI technology continues to evolve, we can anticipate even more innovative applications that will reshape how we interact with our digital environments.

Remix Institute MembershipIf you liked this article, join our free Membership as a Stádas Genesis member for access to elite professionals and exclusive courses on getting started with R, Julia, data science, AI, and six-figure job opportunities. Elevate your skills and gain an edge in the industry. Sign up now for the elite learning experience.

Sign Up for Our Free, Elite AI Membership and CoursesThe post AI-Powered Virtual Assistants: Beyond Siri and Alexa appeared first on Remix Institute.

To leave a comment for the author, please follow the link and comment on their blog: R Archives - Remix Institute.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: AI-Powered Virtual Assistants: Beyond Siri and Alexa

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Split a Vector into Chunks in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Split a Vector into Chunks in R can be a useful technique for manipulating and analyzing data.

In this article, we’ll explore how to use the split() function in R to split a vector into chunks.

Basic Syntax:Split a Vector into Chunks in R

The basic syntax for splitting a vector into chunks in R is:

ggpairs in R » Data Science Tutorials

chunks <- split(my\_vector, cut(seq\_along(my\_vector), n, labels=FALSE) Where:

my_vector is the vector you want to split
n is the number of chunks you want to split the vector into
labels=FALSE specifies whether to use labels for the chunks or not

Example: Splitting a Vector into Chunks

Let’s create a vector with 12 elements and split it into 4 chunks:

Step-by-Step Data Science Coding Course

```

Create vectormy_vector <- c(12, 2, 54, 37, 46, 18, 92, 83, 18, 102, 85, 94)# View length of vectorlength(my_vector)[1] 12# Split vector into four chunkschunks <- split(my_vector, cut(seq_along(my_vector), 4, labels=FALSE))# View chunkschunks$`1`[1] 12 2 2 54$`2`[1] 37 46 18$`3`[1] 92 83 18$`4`[1] 102 85 94

``` From the output, we can see that each chunk contains an equal number of elements.

Accessing Specific Chunks

We can access a specific chunk using brackets:

```

Access second chunk onlychunks[2]$`2`[1] 37 46 18

``` Splitting into Different Numbers of Chunks

We can change the value of n to split the vector into a different number of chunks. For example, let’s split the vector into six chunks:

```

Split vector into six chunkschunks <- split(my_vector, cut(seq_along(my_vector), 6, labels=FALSE))# View chunkschunks$`1`[1] 12 2 2$`2`[1] 54 37$`3`[1] 46 18$`4`[1] 92 83$`5`[1] 18 102$`6`[1] 85 94Now we have six chunks, each containing an equal number of elements.

``` Conclusion

In this article, we’ve learned how to split a vector into chunks in R using the split() function.

We’ve seen how to specify the number of chunks and access specific chunks using brackets. By mastering this technique, you can easily manipulate and analyze large datasets in R.

Is It Difficult to Learn Data Science
Data Manipulation Techniques with dplyr
COUNTIF Function in R
Kendall’s Rank Correlation in R-Correlation Test
Calculate Confidence Intervals in R
Check if the Column Contains a String or not
How to calculate Power Regression in R (Step-by-Step Guide)
Most Winning Numbers in Kerala Lottery
How to Choose Appropriate Clustering Method for Your Dataset

The post Split a Vector into Chunks in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Split a Vector into Chunks in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, fellow R useRs! Today, we’re going to discuss a fascinating topic that bridges the gap between VBA (Visual Basic for Applications) and R. We’ll explore how to get a list of all open workbooks in Excel using VBA and then call this VBA code from R. This can be particularly useful if you’re working with multiple Excel files and need to manage them efficiently from R.

PrerequisitesStep 1: Writing the VBA CodeFirst, let’s write a simple VBA macro to list all open workbooks. Open Excel, press Alt + F11 to open the VBA editor, and insert a new module. Here’s the VBA code:

Sub ListAllOpenWorkbooks() Dim wb As Workbook Dim wbNames As String wbNames = "Open Workbooks:" & vbCrLf For Each wb In Application.Workbooks wbNames = wbNames & wb.Name & vbCrLf Next wb MsgBox wbNamesEnd Sub Explanation:* Sub ListAllOpenWorkbooks(): This starts our macro. * Dim wb As Workbook: Declares a variable wb to represent each workbook. * Dim wbNames As String: Declares a string variable to store the names of open workbooks. * For Each wb In Application.Workbooks: Loops through each open workbook. * wbNames = wbNames & wb.Name & vbCrLf: Appends the name of each workbook to the wbNames string. * MsgBox wbNames: Displays the names of all open workbooks in a message box.

Step 2: Saving the VBA MacroSave your VBA macro by clicking File > Save. Make sure to save your Excel file as a macro-enabled workbook (.xlsm).

Step 3: Calling the VBA Macro from RNow, let’s move to R. We’ll use the RDCOMClient package to interact with Excel and call our VBA macro. If you haven’t installed this package yet, you can do so using:

install.packages("RDCOMClient", repos = "http://www.omegahat.net/R") Here’s the R code to call our VBA macro:

library(RDCOMClient)# Create a COM object to interact with Excelexcel\_app <- COMCreate("Excel.Application")# Make Excel visible (optional)excel\_app[["Visible"]] <- TRUE# Open the workbook containing the VBA macroworkbook <- excel\_app[["Workbooks"]]$Open("C:\\path\\to\\your\\workbook.xlsm")# Run the VBA macroexcel\_app$Run("ListAllOpenWorkbooks")# Close the workbook without savingworkbook$Close(FALSE)# Quit Excelexcel\_app$Quit() Explanation:* library(RDCOMClient): Loads the RDCOMClient package. * COMCreate(“Excel.Application”): Creates a COM object to interact with Excel. * excel_app[[“Visible”]] <- TRUE: Makes Excel visible (optional). * excel_app[[“Workbooks”]]$Open(“C:.xlsm”): Opens the workbook containing the VBA macro. Replace “C:\path\to\your\workbook.xlsm” with the actual path to your workbook. * excel_app$Run(“ListAllOpenWorkbooks”): Runs the VBA macro. * workbook$Close(FALSE): Closes the workbook without saving changes. * excel_app$Quit(): Quits Excel.

Here are some sample outputs for me:

Many Open WorkbooksTwo Open WorkbookConclusionAnd there you have it! You’ve successfully listed all open workbooks using VBA and called this macro from R. This approach can be incredibly powerful for automating tasks that involve both R and Excel. Give it a try and see how what happens!

Feel free to experiment with the code and adapt it to your needs.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to List All Open Workbooks Using VBA and Call It from R

View Details

[This article was first published on R-bloggers | A Random Walk, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.IntroductionWhen fitting a nonlinear regression model in R with nls(), the first step is to select an appropriate regression model to fit the observed data, the second step is to find reasonable starting values for the model parameters in order to initialize the nonlinear least-squares (NLS) algorithm. In some cases, choosing starting values is straightforward, for instance when there is some physical interpretation to the model or the least squares objective function is highly regular and easy to optimize. In other cases, this can be more challenging, especially when the least squares objective consists of large plateaus with local minima located in small narrow canyons (see e.g. Transtrum, Machta, and Sethna (2011)), or when the parameters live on very different scales. To illustrate, consider the following Hobbs’ weed infestation example dataset from (Nash 1979) and (Nash 2014):

```

Hobbs' weed infestation data hobbs_weed <- data.frame( x = 1:12, y = c(5.308, 7.24, 9.638, 12.866, 17.069, 23.192, 31.443, 38.558, 50.156, 62.948, 75.995, 91.972))

`` Using the standard Gauss-Newton algorithm to fit the nonlinear modely ~ b1 / (1 + b2 * exp(-b3 * x))with a basicnls()` call, correct convergence of the algorithm is (perhaps) surprisingly sensitive to the choice of starting values. If not properly initialized, the parameters tend to run away to infinity, also known as parameter evaporation:

```

initial attempt nlsnls( formula = y ~ b1 / (1 + b2 * exp(-b3 * x)), data = hobbs_weed, start = c(b1 = 100, b2 = 10, b3 = 1), trace = TRUE)#> 34094.85 (3.11e+00): par = (100 10 1)#> 11186.49 (3.86e+00): par = (78.18911 3.290385 0.4759647)#> 5112.607 (4.69e+01): par = (80.9356 3.413601 0.1187296)#> 5032.864 (3.53e+01): par = (114.9669 5.478666 0.1037978)#> 4937.315 (2.78e+01): par = (184.1488 9.704171 0.09525315)#> 4798.508 (2.41e+01): par = (346.9847 19.72996 0.09116909)#> 4624.151 (2.16e+01): par = (1325.61 80.41635 0.08932604)#> 3859.425 (1.91e+01): par = (23585.2 1465.277 0.09589796)#> 1605.353 (1.32e+01): par = (7709100 479913.4 0.1272658)#> Error in nls(formula = y ~ b1/(1 + b2 * exp(-b3 * x)), data = hobbs_weed, : singular gradient

``` This is in fact an example where the Gauss-Newton direction initially points to the boundary of the parameter space (see also Figure 7 in Transtrum, Machta, and Sethna (2011)). Instead, using a damped least squares algorithm (Levenberg-Marquardt) that combines both the Gauss-Newton and gradient descent directions, we expect the parameters to evaporate less quickly:

library(gslnls)## initial attempt gsl\_nlsgsl\_nls( fn = y ~ b1 / (1 + b2 * exp(-b3 * x)), data = hobbs\_weed, start = c(b1 = 100, b2 = 10, b3 = 1), algorithm = "lm")#> Nonlinear regression model#> model: y ~ b1/(1 + b2 * exp(-b3 * x))#> data: hobbs\_weed#> b1 b2 b3 #> 196.1863 49.0916 0.3136 #> residual sum-of-squares: 2.587#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 15 #> Achieved convergence tolerance: 1.554e-14 In an attempt to reduce the dependence of the NLS algorithm on a single set of (poorly selected) starting values, this post demonstrates a new multistart procedure available1 in R-package gslnls, which can be useful when we only have limited knowledge regarding the expected parameter values or when we wish to automate nonlinear model fits across multiple different datasets.

A common approach in R to avoid the need for user-supplied NLS starting values is to make use of so-called selfStart models (SSasymp(), SSfpl(), SSmicmen(), etc.) that include an initialization function to return reasonable starting values for the nonlinear model given the available data. The initialization function typically considers a simpler approximate or linearized version of the nonlinear model for which parameters can be estimated without starting values, using e.g. lm(). This is the model fitting approach that is utilized by drc and nlraa among other packages. If we intend to fit a nonlinear model for which a selfStart implementation is already available (or straightforward to implement), this is definitely the recommended approach, since the starting values obtained from a selfStart model are usually well-informed and most NLS routines will not have trouble obtaining the correct parameter estimates from these starting values.

If no selfStart model is available, another approach is to repeatedly call nls() using different starting values drawn as random or fixed points from a pre-defined grid. This is implemented by nls2::nls2() using one of the methods "brute-force", "grid-search", or "random-search". The new multistart procedure in gsl_nls() tries to improve on naive random or grid-based multistart optimization, which can be a very time consuming process, especially if the number of parameters is large (curse of dimensionality) or the scale of the parameter ranges to evaluate is quite broad. As in any multistart global optimization procedure, the ideal approach would be to start a local NLS optimizer within each basin of attraction2 exactly once to reach all existing local minima with minimal computational effort. In practice, we try to avoid running too many local optimizers in the same basin of attraction (resulting in the same local minimum) by exploring the parameter space for promising starting values that might converge to an unseen (local) optimum. The multistart algorithm implemented in gsl_nls() is a modified version of the algorithm in (Hickernell and Yuan 1997) that works both with or without a pre-defined grid of starting ranges. Before describing the details of the multistart procedure, below are several examples illustrating its usage with gsl_nls().

NLS examplesHobbs’ weed infestation exampleRevisiting the Hobbs’ weed infestation example above, we fit the nonlinear model with gsl_nls() using a (fixed) set of starting ranges for the parameters instead of individual starting values:

```

multistart attempt gsl_nlsgsl_nls( fn = y ~ b1 / (1 + b2 * exp(-b3 * x)), data = hobbs_weed, start = list(b1 = c(0, 1000), b2 = c(0, 1000), b3 = c(0, 10)))#> Nonlinear regression model#> model: y ~ b1/(1 + b2 * exp(-b3 * x))#> data: hobbs_weed#> b1 b2 b3 #> 196.1863 49.0916 0.3136 #> residual sum-of-squares: 2.587#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 3 #> Achieved convergence tolerance: 2.842e-14

``` Alternatively, we can leave one or more starting ranges undefined, in which case they are updated dynamically during the multistart optimization:

```

multistart attempt 2 gsl_nlsgsl_nls( fn = y ~ b1 / (1 + b2 * exp(-b3 * x)), data = hobbs_weed, start = list(b1 = NA, b2 = NA, b3 = NA))#> Nonlinear regression model#> model: y ~ b1/(1 + b2 * exp(-b3 * x))#> data: hobbs_weed#> b1 b2 b3 #> 196.1863 49.0916 0.3136 #> residual sum-of-squares: 2.587#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 4 #> Achieved convergence tolerance: 5.329e-15

`` NIST StRDGauss1exampleSecond, we consider theGauss1regression test problem as listed in the NIST StRD Nonlinear Regression archive. The observed data takes the shape of a camel’s back consisting of two Gaussians on a decaying exponential baseline subject to additive Gaussian noise. The data, nonlinear model and target parameter values are included in thegslnlspackage and available through the functionnls_test_problem()`:

gauss1 <- nls\_test\_problem("Gauss1")## datastr(gauss1$data)#> 'data.frame':250 obs. of 2 variables:#> $ y: num 97.6 97.8 96.6 92.6 91.2 ...#> $ x: num 1 2 3 4 5 6 7 8 9 10 ...## model + target parametersgauss1[c("fn", "target")]#> $fn#> y ~ b1 * exp(-b2 * x) + b3 * exp(-(x - b4)^2/b5^2) + b6 * exp(-(x - #> b7)^2/b8^2)#> <environment: 0x562f10e8ceb8>#> #> $target#> b1 b2 b3 b4 b5 b6 #> 98.77821087 0.01049728 100.48990633 67.48111128 23.12977336 71.99450300 #> b7 b8 #> 178.99805021 18.38938902 Due to the large number of parameters in the model y ~ b1 * exp(-b2 * x) + b3 * exp(-(x - b4)**2 / b5**2) + b6 * exp(-(x - b7)**2 / b8**2), finding starting values from which nls() is able to correctly fit the model is a tedious task. Below is an initial attempt at solving the NLS problem with the NL2SOL algorithm (algorithm = "port") and all parameters bounded from below by zero:

```

initial attempt nlsnls( formula = gauss1$fn, data = gauss1$data, start = c(b1 = 100, b2 = 0, b3 = 100, b4 = 75, b5 = 25, b6 = 75, b7 = 125, b8 = 25), lower = 0, algorithm = "port")#> Error in nls(formula = gauss1$fn, data = gauss1$data, start = c(b1 = 100, : Convergence failure: singular convergence (7)

`` As seen from this example, most parameter starting values –except forb7– are not that far from their target values. However,nls()still fails to converge due to a single poorly selected starting value for the parameterb7`.

Using the multistart procedure in gsl_nls(), we can combine fixed starting values or ranges when good initial guesses for the parameters are available together with missing starting values for the parameters that are lacking such information. This avoids the need to select poor starting values for certain parameters, which may cause the NLS optimization to fail as in our previous attempt to fit the model.

```

multistart attempt gsl_nlsgsl_nls( fn = gauss1$fn, data = gauss1$data, start = list(b1 = 100, b2 = c(0, 1), b3 = NA, b4 = NA, b5 = NA, b6 = NA, b7 = NA, b8 = NA), lower = 0)#> Nonlinear regression model#> model: y ~ b1 * exp(-b2 * x) + b3 * exp(-(x - b4)^2/b5^2) + b6 * exp(-(x - b7)^2/b8^2)#> data: gauss1$data#> b1 b2 b3 b4 b5 b6 b7 b8 #> 98.7782 0.0105 100.4899 67.4811 23.1298 71.9945 178.9981 18.3894 #> residual sum-of-squares: 1316#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 4 #> Achieved convergence tolerance: 2.274e-13

`` Lubricant dataset (Bates & Watts)As a final example, consider the Lubricant dataset from (Bates and Watts 1988, Appendix 1, A1.8), which measures the kinematic viscosity of a lubricant as a function of temperature (°C) and pressure (atm). The Lubricant data, model and target parameter are included as an NLS test problem ingslnlsand can be retrieved withnls_test_problem()similar to the previous example. Here,x1encodes the temperature predictor (in °C) andx2` is the pressure (atm) predictor.

lubricant <- nls\_test\_problem("Lubricant")## datastr(lubricant$data)#> 'data.frame':53 obs. of 3 variables:#> $ y : num 5.11 6.39 7.39 5.79 5.11 ...#> $ x1: num 0 0 0 0 0 0 0 0 0 0 ...#> $ x2: num 0.001 0.741 1.407 0.363 0.001 ...## model + target parameterslubricant[c("fn", "target")]#> $fn#> y ~ b1/(b2 + x1) + b3 * x2 + b4 * x2^2 + b5 * x2^3 + (b6 * x2 + #> b7 * x2^3) * exp(-x1/(b8 + b9 * x2^2))#> <environment: 0x562f0ab01c30>#> #> $target#> b1 b2 b3 b4 b5 #> 1054.54053186 206.54577890 1.46031479 -0.25965483 0.02257371 #> b6 b7 b8 b9 #> 0.40138428 0.03528405 57.40463143 -0.47672110 The nonlinear model contains a relatively large number of parameters, similar to the Gauss1 example, and the model fitting is further complicated by the large differences in magnitude of the target parameters. Without a more systematic approach, e.g. by linearizing parts of the model to initialize parameters as in (Bates and Watts 1988, Ch 3.6), it is difficult to obtain a good model fit with nls() by naively trying different sets of starting values. Here, we again use the NL2SOL algorithm (algorithm = "port"), selecting starting values that are all close to the target parameters, but which still cause nls() to fail.

```

initial attempt nlsnls( formula = lubricant$fn, data = lubricant$data, start = c(b1 = 1000, b2 = 200, b3 = 1, b4 = -0.01, b5 = 0.01, b6 = 0.01, b7 = 0.01, b8 = 50, b9 = -0.01), algorithm = "port")#> Error in nls(formula = lubricant$fn, data = lubricant$data, start = c(b1 = 1000, : Convergence failure: false convergence (8)

`` Attempting to solve the NLS problem with the multistart procedure ingsl_nls()`, we are able to obtain correct NLS convergence without any knowledge of the target parameters by leaving all starting values unspecified:

```

multistart attempt gsl_nlsgsl_nls( fn = lubricant$fn, data = lubricant$data, start = c(b1 = NA, b2 = NA, b3 = NA, b4 = NA, b5 = NA, b6 = NA, b7 = NA, b8 = NA, b9 = NA))#> Nonlinear regression model#> model: y ~ b1/(b2 + x1) + b3 * x2 + b4 * x2^2 + b5 * x2^3 + (b6 * x2 + b7 * x2^3) * exp(-x1/(b8 + b9 * x2^2))#> data: lubricant$data#> b1 b2 b3 b4 b5 b6 b7 #> 1054.54080 206.54584 1.46031 -0.25965 0.02257 0.40138 0.03528 #> b8 b9 #> 57.40467 -0.47672 #> residual sum-of-squares: 0.08702#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 6 #> Achieved convergence tolerance: 5.69e-16

``` Multistart algorithm detailsBefore evaluating more NLS test problems, this section provides a more comprehensive overview of the implemented multistart algorithm. The implementation is primarily based on the global optimization algorithm in (Hickernell and Yuan 1997), but slightly modified to the context of trust-region nonlinear least squares. The multistart algorithm consists of multiple major iterations. At the start of major iteration (M), a set of pseudo-random starting points is sampled inside the grid of starting ranges. Starting points with an almost singular (approximate) Hessian matrix are discarded immediately, as these are unlikely to converge to a local optimum. For the remaining points, a few iterations of the NLS optimizer are applied in order to distinguish promising from unpromising starting points. At each major iteration, only the (q) most promising starting points are kept, and if a starting point is not discarded from this set for at least (s) major iterations, a full NLS optimization routine is executed using this starting point. If a new optimal solution is found, the number of optimal stationary points NOSP is incremented and the number of found worse stationary points NWSP is reset to zero. If the obtained solution has already been observed before, or if it converges to a local minimum that is worse than the current optimal solution, the number of worse stationary points NSWP is incremented. If the number of found worse stationary points (NWSP) becomes much larger than the number of optimal starting points (NOSP), then it is likely that we have exhausted the search space and are unable to further improve the current optimal solution. The following pseudo-code provides a high-level description of the multistart procedure. For simplicity, we first consider the scenario in which all (p) parameter starting ranges ([l_1, u_1] \times \ldots \times [l_{p}, u_{p}]) are pre-defined and fixed.

INITIALIZE
- Choose parameters (N \geq q \geq 1), (L \geq \ell \geq 1), (s \geq 1), (r > 1), maxstart (\geq 1), minsp (\geq 1). Set (M = 0), NOSP = NWSP = 0, and initialize a scaling vector (\boldsymbol{D} = (0.75, \dots, 0.75) \in [0, 1]^p).
SAMPLE
- Sample (N) initial (p)-dimensional points (\boldsymbol{\theta}_1,\ldots, \boldsymbol{\theta}_N) inside the grid of starting ranges ([l_1, u_1] \times \ldots \times [l_{p}, u_{p}]) from a pseudo-random Sobol sequence.
- For (i = 1,\ldots,N) and (k = 1,\ldots,p), rescale the sampled points inside the grid of starting ranges favoring points near zero according to3: (\theta^*_{i,k} = \frac{\text{sgn}(\theta_{i,k})}{D_k} \cdot ((|\theta_{i,k}| - l_k^+ - u_k^- + 1)^{D_k} - 1) + l_k^+ - u_k^-).
REDUCE 1 AND CONCENTRATE
- Discard all points for which (\det\left(J(\boldsymbol{\theta}^_i)^TJ(\boldsymbol{\theta}^_i)\right) < \epsilon), with (\epsilon > 0) small and (J(\boldsymbol{\theta}^_i)) the Jacobian evaluated at (\boldsymbol{\theta}^_i).
- For each remaining point apply (a small number) (\ell) iterations of the NLS optimizer, obtaining concentrated points (\boldsymbol{\theta}_1^{},\ldots,\boldsymbol{\theta}_{N_1}^{})
REDUCE 2
- Identify the first (q) order statistics of (F(\boldsymbol{\theta}_1^{}),\ldots,F(\boldsymbol{\theta}_{N_1}^{})), with (F(\boldsymbol{\theta}_i^{})) the NLS objective evaluated at (\boldsymbol{\theta}_i^{}). Denote (I_q) for the set of indices (of size (q)).
- Update counter (S(i) = S(i) + 1) if (i \in I_q), otherwise set (S(i) = 0).
OPTIMIZE
- For each (i) with (S(i) \geq s), if (F(\boldsymbol{\theta}_i^{}) < (1 + \delta) \cdot F(\boldsymbol{\theta}^{opt})), apply (L) iterations of the NLS optimizer, obtaining (\boldsymbol{\theta}_i^{*}), and set (S(i) = 0).
- If (F(\boldsymbol{\theta}_i^{}) < F(\boldsymbol{\theta}^{opt})), set (\boldsymbol{\theta}^{opt} = \boldsymbol{\theta}_i^{}), and update the scaling vector (\boldsymbol{D}) by (D_k = \left(\frac{\min_\kappa \Delta_\kappa}{\Delta_k}\right)^\alpha), with exponent (\alpha \in [0, 1]) and (\boldsymbol{\Delta} = \text{diag}(J(\boldsymbol{\theta}^{}_i)^TJ(\boldsymbol{\theta}^{}_i))), i.e. the diagonal damping matrix as used by Marquardt’s scaling method. Update the number of found optimal stationary points NOSP = NOSP + 1, and reset the number of found worse stationary points NWSP = 0.
- If (F(\boldsymbol{\theta}_i^{***}) \geq F(\boldsymbol{\theta}^{opt})), set NWSP = NWSP + 1.
REPEAT
- If NWSP (\geq) (r \cdot) NSP and NSP (\geq) minsp (minimum required number of stationary points) then stop with success. Otherwise, if (M <) maxstart, increment (M = M + 1) and return to step 2. sampling new pseudo-random points for each (i) with (S(i) = 0), reusing (\boldsymbol{\theta}_i = \boldsymbol{\theta}_i^{**}) if (S(i) > 0).

It is important to point out that there is no guarantee that the NLS objective (F(\boldsymbol{\theta}^{opt})) at the solution returned by the multistart procedure indeed evaluates to the global minimum inside the grid of starting ranges. This may for instance be due to the rescaling function in step 2., which is a type of inverse logistic function scaled to the starting range ([l_k, u_k]). The scaling exponent (D_k) is calculated from the damping matrix in Marquardt’s scaling method, thereby rescaling each parameter differently based on an approximate measure of its order of magnitude. If the damping matrix that is used to calculate the (D_k)’s is highly –but incorrectly– confident about the order of magnitudes of certain parameters, we may not explore the parameter starting ranges ([l_k, u_k]) sufficiently broadly in the subsequent major iterations. Increasing the number of sampled points (N) at the start of each major iteration may help to overcome such issues. If we suspect that the returned optimal solution (\boldsymbol{\theta}^{opt}) is only a local optimizing solution, we can always force the multistart procedure to continue searching until a better optimum is found by increasing minsp (minimum required number of stationary points).

Missing starting rangesIf no fixed starting values or ranges are defined for certain parameters, as demonstrated in the above examples, then the missing ranges ([l_k, u_k]) are initialized to the unit interval and dynamically increased or decreased in each major iteration of the multistart algorithm. The decision to increase or decrease the limits of a parameter’s starting range is driven by the minimum and maximum parameter values obtained from the (q) best-performing concentrated points (step 3. and 4.) with indices included in (I_q). These typically provide a rough indication of the order of magnitude of the parameter range in which to search for the optimal solution. If the dynamic parameter ranges fail to grow sufficiently large to include the global optimizing solution, it may help to increase the values of (N), (r), maxstart or minsp to avoid early termination of the algorithm at the cost of increased computation effort.

NLS test problemsAt the moment of writing this post, 59 NLS test problems are included in gslnls originating primarily from the NIST StRD Nonlinear Regression archive, (Bates and Watts 1988) and (Moré, Garbow, and Hillstrom 1981). This collection of test problems contains 33 regression problems, with nonlinear models defined as a formula and the number of parameters and observations fixed (p, n fixed). The other 26 problems are NLS optimization problems, ported from the Fortran library TEST_NLS. For these problems the nonlinear models are defined as a function and some of the models allow for the number of parameters and observations to be freely varied, only requiring that the number of parameters does not exceed the number of observations/residuals (p <= n free and p == n free). The table below lists all 59 test problems as returned by nls_test_list() including their default number of observations and parameters as set in gslnls.

Table 1: NLS test problems| Dataset name | Reference | # Observations (n) | # Parameters (p) | Data constraint | Model expression | | --- | --- | --- | --- | --- | --- | | Misra1a | NIST StRD (1978) | 14 | 2 | p, n fixed | formula | | Chwirut2 | NIST StRD (1978) | 54 | 3 | p, n fixed | formula | | Chwirut1 | NIST StRD (1978) | 214 | 3 | p, n fixed | formula | | Lanczos3 | NIST StRD (1978) | 24 | 6 | p, n fixed | formula | | Gauss1 | NIST StRD (1978) | 250 | 8 | p, n fixed | formula | | Gauss2 | NIST StRD (1978) | 250 | 8 | p, n fixed | formula | | DanWood | NIST StRD (1978) | 6 | 2 | p, n fixed | formula | | Misra1b | NIST StRD (1978) | 14 | 2 | p, n fixed | formula | | Kirby2 | NIST StRD (1978) | 151 | 5 | p, n fixed | formula | | Hahn1 | NIST StRD (1978) | 236 | 7 | p, n fixed | formula | | Nelson | NIST StRD (1978) | 128 | 3 | p, n fixed | formula | | MGH17 | NIST StRD (1978) | 33 | 5 | p, n fixed | formula | | Lanczos1 | NIST StRD (1978) | 24 | 6 | p, n fixed | formula | | Lanczos2 | NIST StRD (1978) | 24 | 6 | p, n fixed | formula | | Gauss3 | NIST StRD (1978) | 250 | 8 | p, n fixed | formula | | Misra1c | NIST StRD (1978) | 14 | 2 | p, n fixed | formula | | Misra1d | NIST StRD (1978) | 14 | 2 | p, n fixed | formula | | Roszman1 | NIST StRD (1978) | 25 | 4 | p, n fixed | formula | | ENSO | NIST StRD (1978) | 168 | 9 | p, n fixed | formula | | MGH09 | NIST StRD (1978) | 11 | 4 | p, n fixed | formula | | Thurber | NIST StRD (1978) | 37 | 7 | p, n fixed | formula | | BoxBOD | NIST StRD (1978) | 6 | 2 | p, n fixed | formula | | Ratkowsky2 | NIST StRD (1978) | 9 | 3 | p, n fixed | formula | | MGH10 | NIST StRD (1978) | 16 | 3 | p, n fixed | formula | | Eckerle4 | NIST StRD (1978) | 35 | 3 | p, n fixed | formula | | Ratkowsky3 | NIST StRD (1978) | 15 | 4 | p, n fixed | formula | | Bennett5 | NIST StRD (1978) | 154 | 3 | p, n fixed | formula | | Isomerization | Bates and Watts (1988) | 24 | 4 | p, n fixed | formula | | Lubricant | Bates and Watts (1988) | 53 | 9 | p, n fixed | formula | | Sulfisoxazole | Bates and Watts (1988) | 12 | 4 | p, n fixed | formula | | Leaves | Bates and Watts (1988) | 15 | 4 | p, n fixed | formula | | Chloride | Bates and Watts (1988) | 54 | 3 | p, n fixed | formula | | Tetracycline | Bates and Watts (1988) | 9 | 4 | p, n fixed | formula | | Linear, full rank | Moré, Garbow, and Hillstrom (1981) | 10 | 5 | p <= n free | function | | Linear, rank 1 | Moré, Garbow, and Hillstrom (1981) | 10 | 5 | p <= n free | function | | Linear, rank 1, zero columns and rows | Moré, Garbow, and Hillstrom (1981) | 10 | 5 | p <= n free | function | | Rosenbrock | Moré, Garbow, and Hillstrom (1981) | 2 | 2 | p, n fixed | function | | Helical valley | Moré, Garbow, and Hillstrom (1981) | 3 | 3 | p, n fixed | function | | Powell singular | Moré, Garbow, and Hillstrom (1981) | 4 | 4 | p, n fixed | function | | Freudenstein/Roth | Moré, Garbow, and Hillstrom (1981) | 2 | 2 | p, n fixed | function | | Bard | Moré, Garbow, and Hillstrom (1981) | 15 | 3 | p, n fixed | function | | Kowalik and Osborne | Moré, Garbow, and Hillstrom (1981) | 11 | 4 | p, n fixed | function | | Meyer | Moré, Garbow, and Hillstrom (1981) | 16 | 3 | p, n fixed | function | | Watson | Moré, Garbow, and Hillstrom (1981) | 31 | 6 | p, n fixed | function | | Box 3-dimensional | Moré, Garbow, and Hillstrom (1981) | 10 | 3 | p <= n free | function | | Jennrich and Sampson | Moré, Garbow, and Hillstrom (1981) | 10 | 2 | p <= n free | function | | Brown and Dennis | Moré, Garbow, and Hillstrom (1981) | 20 | 4 | p <= n free | function | | Chebyquad | Moré, Garbow, and Hillstrom (1981) | 9 | 9 | p <= n free | function | | Brown almost-linear | Moré, Garbow, and Hillstrom (1981) | 10 | 10 | p == n free | function | | Osborne 1 | Moré, Garbow, and Hillstrom (1981) | 33 | 5 | p, n fixed | function | | Osborne 2 | Moré, Garbow, and Hillstrom (1981) | 65 | 11 | p, n fixed | function | | Hanson 1 | Salane (1987) | 16 | 2 | p, n fixed | function | | Hanson 2 | Salane (1987) | 16 | 3 | p, n fixed | function | | McKeown 1 | McKeown (1975) | 3 | 2 | p, n fixed | function | | McKeown 2 | McKeown (1975) | 4 | 3 | p, n fixed | function | | McKeown 3 | McKeown (1975) | 10 | 5 | p, n fixed | function | | Devilliers and Glasser 1 | Salane (1987) | 24 | 4 | p, n fixed | function | | Devilliers and Glasser 2 | Salane (1987) | 16 | 5 | p, n fixed | function | | Madsen example | Madsen (1988) | 3 | 2 | p, n fixed | function |

For each test problem, the data, nonlinear model and target parameter values can be retrieved using nls_test_problem(), as also illustrated above for the Gauss1 and Lubricant datasets. The nls_test_problem() function includes suggested starting values for all regression problems and for optimization problems when using the default number of parameters and residuals (p = NA, n = NA). For the optimization problems, a function calculating the (n \times p) Jacobian matrix is also returned. This function can be passed to the jac argument of gsl_nls() in order to use analytic evaluation of the gradient in the NLS algorithm.

Example regression problem (Misra1a) ```

nls model + data(misra1a <- nls_test_problem("Misra1a"))#> $data#> y x#> 1 10.07 77.6#> 2 14.73 114.9#> 3 17.94 141.1#> 4 23.93 190.8#> 5 29.61 239.9#> 6 35.18 289.0#> 7 40.02 332.8#> 8 44.82 378.4#> 9 50.76 434.8#> 10 55.05 477.3#> 11 61.01 536.8#> 12 66.40 593.1#> 13 75.47 689.1#> 14 81.78 760.0#> #> $fn#> y ~ b1 * (1 - exp(-b2 * x))#> #> #> $start#> b1 b2 #> 5e+02 1e-04 #> #> $target#> b1 b2 #> 2.389421e+02 5.501564e-04 #> #> attr(,"class")#> [1] "nls_test_formula"## nls model fitwith(misra1a, gsl_nls( fn = fn, data = data, start = start ))#> Nonlinear regression model#> model: y ~ b1 * (1 - exp(-b2 * x))#> data: data#> b1 b2 #> 2.389e+02 5.502e-04 #> residual sum-of-squares: 0.1246#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 16 #> Achieved convergence tolerance: 6.745e-15

Example optimization problem (`Rosenbrock`)

nls model fitwith(nls_test_problem("Rosenbrock"), gsl_nls( fn = fn, y = y, start = start, jac = jac ))#> Nonlinear regression model#> model: y ~ fn(x)#> x1 x2 #> 1 1 #> residual sum-of-squares: 9.762e-19#> #> Algorithm: multifit/levenberg-marquardt, (scaling: more, solver: qr)#> #> Number of iterations to convergence: 16 #> Achieved convergence tolerance: 7.696e-14

`` Benchmark NLS fitsTo conclude, we benchmark the performance of the multistart algorithm by computing NLS model fits for each of the 59 test problems using the multistart algorithm with no starting values provided, i.e. all starting values are set toNA. As trust region method we choose respectively: the default Levenberg-Marquardt algorithm (algorithm = "lm"); the double dogleg algorithm (algorithm = "ddogleg"); and the Levenberg-Marquardt algorithm with geodesic acceleration (algorithm = "lmaccel"). The maximum number of allowed iterationsmaxiteris set to $10\ 000$, all other tuning parameters in thecontrolargument are kept at their default values according togsl_nls_control(). For comparison, we also compute single-start NLS model fits using the default Levenberg Marquardt algorithm (algorithm = "lm"), with as naive choice of starting values a vector of all ones $(1, \ldots, 1)$, similar tonls()when argumentstart` is missing.

The table below displays the NLS model fit results for each individual test problem using the following status colors:

success ; the NLS routine converged successfully and the residual sum-of-squares (SSR) under the fitted parameters coincides (up to a small numeric tolerance) with the SSR under the target parameter values4. The total runtime is displayed in seconds, timed on a modern laptop computer (Intel i7-8550U CPU, 1.80GHz) using a single core.
false convergence ; the NLS routine converged successfully, but the residual SSR under the fitted parameters is larger than the SSR under the target parameter values.
max. iterations ; the NLS routine failed to converge within the maximum number of allowed iterations.
failed/non-zero exit ; the NLS routine failed to converge and returns an error or an NLS object with a non-zero exit code.

We observe that the naive single-start model fits manage to correctly fit about half of the test problems (27 out of 59), suggesting that these test problems are straightforward to optimize and do not require well-informed starting values. The multistart model fits using the double dogleg method improve upon the naive single-start model fits achieving correct convergence for 51 out of 59 test problems. The multistart Levenberg-Marquardt model fits correctly converge for a few more test problems (56 out of 59). Finally, the most robust results are obtained with the multistart model fits using the Levenberg-Marquardt algorithm with geodesic acceleration, which correctly fit all 59 test problems without initializing proper starting values or starting ranges!

| | lm/single-start | lm/multi-start | ddogleg/multi-start | lmaccel/multi-start | | --- | --- | --- | --- | --- | | Misra1a | | 0.4s | 0.2s | 0.2s | | Chwirut2 | | 0.4s | 0.3s | 0.4s | | Chwirut1 | | 0.5s | 0.4s | 0.7s | | Lanczos3 | 0.02s | 2s | 2s | 2s | | Gauss1 | | 4s | | 2s | | Gauss2 | | 4s | | 2s | | DanWood | 0.003s | 0.3s | 0.3s | 0.3s | | Misra1b | | 0.4s | 0.6s | 0.6s | | Kirby2 | 0.01s | 0.3s | 0.3s | 0.4s | | Hahn1 | | | | 2s | | Nelson | | 0.3s | 0.6s | 0.2s | | MGH17 | | 0.9s | 0.3s | 1s | | Lanczos1 | 0.02s | 3s | 2s | 1s | | Lanczos2 | 0.02s | 2s | 2s | 2s | | Gauss3 | | 4s | | 2s | | Misra1c | | | | 0.8s | | Misra1d | | 2s | 2s | 0.5s | | Roszman1 | | 4s | | 5s | | ENSO | | 6s | 5s | 5s | | MGH09 | 0.004s | 0.4s | 0.4s | 0.5s | | Thurber | | 1s | 1s | 2s | | BoxBOD | | 0.3s | 0.2s | 0.4s | | Ratkowsky2 | | 0.1s | 0.1s | 0.3s | | MGH10 | | 2s | 0.7s | 1s | | Eckerle4 | | 0.4s | 0.4s | 0.5s | | Ratkowsky3 | | 0.4s | 0.3s | 0.4s | | Bennett5 | | | | 1s | | Isomerization | 0.008s | 0.6s | 0.6s | 0.6s | | Lubricant | 0.01s | 2s | 2s | 3s | | Sulfisoxazole | | 0.3s | 0.4s | 0.5s | | Leaves | | 1s | 0.9s | 0.3s | | Chloride | | 0.3s | 0.4s | 0.6s | | Tetracycline | 0.005s | 0.5s | 0.3s | 0.4s | | Linear, full rank | 0.002s | 0.2s | 0.2s | 0.2s | | Linear, rank 1 | 0.002s | 0.5s | 0.3s | 0.4s | | Linear, rank 1, zero columns and rows | 0.002s | 0.4s | 0.3s | 0.3s | | Rosenbrock | 0.002s | 0.3s | 0.1s | 0.3s | | Helical valley | 0.002s | 0.5s | 0.2s | 0.3s | | Powell singular | 0.002s | 0.3s | 0.2s | 0.3s | | Freudenstein/Roth | | 0.4s | 0.2s | 0.4s | | Bard | 0.001s | 0.3s | 0.2s | 0.3s | | Kowalik and Osborne | 0.002s | 0.3s | 0.2s | 0.2s | | Meyer | | 1s | 0.5s | 1s | | Watson | 0.004s | 0.9s | 0.4s | 0.5s | | Box 3-dimensional | 0.002s | 0.1s | 0.2s | 0.4s | | Jennrich and Sampson | 0.003s | 0.2s | 0.2s | 0.3s | | Brown and Dennis | 0.008s | 0.6s | 0.5s | 0.7s | | Chebyquad | 0.009s | 1s | 0.5s | 1s | | Brown almost-linear | 0.001s | 0.7s | 0.6s | 0.9s | | Osborne 1 | | 0.4s | 0.2s | 0.5s | | Osborne 2 | | 1s | | 1s | | Hanson 1 | | 0.2s | 0.2s | 0.4s | | Hanson 2 | | 0.2s | 0.2s | 0.2s | | McKeown 1 | 0.002s | 0.2s | 0.2s | 0.3s | | McKeown 2 | 0.003s | 0.3s | 0.2s | 0.4s | | McKeown 3 | 0.004s | 0.4s | 0.3s | 0.4s | | Devilliers and Glasser 1 | | 0.4s | 0.4s | 0.6s | | Devilliers and Glasser 2 | | 0.6s | 0.7s | 1s | | Madsen example | 0.004s | 0.2s | 0.2s | 0.3s | | # Successful fits | 27/59 | 56/59 | 51/59 | 59/59 |

ReferencesBates, D. M., and D. G. Watts. 1988. “Nonlinear Regression Analysis and Its Applications.” Wiley.Hickernell, F. J., and Y. Yuan. 1997. “A Simple Multistart Algorithm for Global Optimization.” OR Transactions 1 (2): 1–11.Madsen, K. 1988. “A Combined Gauss-Newton and Quasi-Newton Method for Non-Linear Least Squares.” Institute for Numerical Analysis, DTU.McKeown, J. J. 1975. “Specialised Versus General-Purpose Algorithms for Minimising Functions That Are Sums of Squared Terms.” Mathematical Programming 9: 57–68.Moré, J. J., B. S. Garbow, and K. E. Hillstrom. 1981. “Testing Unconstrained Optimization Software.” ACM Transactions on Mathematical Software (TOMS) 7 (1): 17–41.Nash, J. C. 1979. Compact Numerical Methods for Computers: Linear Algebra and Function Minimisation. Bristol, UK: Adam Hilger.———. 2014. Nonlinear Parameter Optimization Using r Tools. UK: Wiley.NIST StRD. 1978. “NIST Statistical Reference Datasets (StRD) Archive.” Online source. https://www.itl.nist.gov/div898/strd/nls/nls_main.shtml.Salane, D. E. 1987. “A Continuation Approach for Solving Large-Residual Nonlinear Least Squares Problems.” SIAM Journal on Scientific and Statistical Computing 8 (4): 655–71.Transtrum, M. K., B. B. Machta, and J. P. Sethna. 2011. “Geometry of Nonlinear Least Squares with Applications to Sloppy Models and Optimization.” Physical Review E 83 (3).

multistart optimization requires gslnls version >= 1.3.0.︎
a basin of attraction refers to the basin surrounding a local minimum of the objective function, such that starting from any point inside the basin the local optimizer converges to the same local minimum.︎
(f^+ = f \vee 0) and (f^- =-f \vee 0) denote the positive and negative part of (f).︎
The reason for using the residual sum-of-squares (SSR) to check for correct convergence of the NLS model fits is that several problems have multiple distinct parameter solutions that result in the same (optimal) SSR.︎ To leave a comment for the author, please follow the link and comment on their blog: R-bloggers | A Random Walk.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Multistart nonlinear least squares fitting with {gslnls}

View Details

[This article was first published on Getting Genetics Done, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.My new blog/newsletter (“Paired Ends”) is now at blog.stephenturner.us. I’ll be posting semi-regular updates and literature highlights in bioinformatics, computational biology, and data science, along with the occasional post on programming. Head over to blog.stephenturner.us to subscribe by email, or add the RSS feed to your favorite reader app.

Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.To leave a comment for the author, please follow the link and comment on their blog: Getting Genetics Done.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Moving to blog.stephenturner.us (Paired Ends)

View Details

[This article was first published on R | r4stats.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.BlueSky Statistics is a free and open-source graphical user interface for the powerful R language. There is also a commercial “Pro” version that offers tech support, priority feature requests, and many powerful additional features. The Pro version has been beefed up considerably with the new features below. These features apply to quality control, general statistics, team collaboration, project management, and scripting. Many are focused on quality control and Six Sigma as a result of requests from organizations migrating from Minitab and JMP. However, both versions of BlueSky Statistics offer a wide range of statistical, graphical, and machine-learning methods.

The free version saves every step of the analysis for full reproducibility. However, repeating the analysis is a step-by-step process. The Pro version can now rerun the entire set at once, substituting other datasets when needed.

You can obtain either version at https://BlueSkyStatistics.com. A detailed review is available at https://r4stats.com/articles/software-reviews/bluesky/. If you plan to attend the Joint Statistical Meetings (JSM) in Portland next week, stop by Booth 406 to get a demonstration. We hope to see you there!

Copy and Paste data from Excel

Copy data from Excel and paste it into the BlueSky Statistics data grid. This is in addition to the existing mechanism of bringing data through file import for various file formats into BlueSky Statistics to perform data analysis.

Undo/Redo data grid edits

Single-item and muti-items data element edits can be discarded by undo and restored by redo operations.

Project save/open to save/open all work (all open datasets and output analysis)

Analysis performed can be saved into one or more projects. Each project contains all the datasets along with all the analyses and any R code from the editor. The projects can be exported and shared (sent as .bsp, which is a zip file; “bsp” is an abbreviation of BlueSky Statistics Project) with other BlueSky Statistics users. The users can import projects, see all the datasets and analyses stored in the projects, and subsequently add/modify/rerun all the analyses.

Enhanced cleaning/adjustment of copied/imported Excel/CSV data on the Datagrid

Dataset > Excel Cleanup

There are a few enhancements made to offer additional data cleanup/adjustment options to the existing Excel Cleanup dialog to clean/adjust (i.e., rows. Columns, data type, etc.) data on the BlueSky Statistics data grid, irrespective of how the data was loaded into the data grid with the file open option or by copying and pasting from Excel/CSV file.

Renaming output tabs

Double-clicking on the output tab will open a dialog box asking for the new name. The user can type in a name to rename the output tab.

Enhanced Pie Chart and Bar Chart

Graphics > Pie Charts > Pie Chart
Graphics > Bar Chart

The pie chart and bar chart have been enhanced to show % and counts on the plot.

Scatterplot Matrix

Graphics > Scatterplot Matrix

The Scatter Plot Matrix dialog has been added.

Scatterplot with mean and confidence interval bar

Graphics > Scatterplot > Scatter Plot with Intervals

A Scatter Plot dialog with mean and confidence interval bar has been made available with an unlimited number of grouping variables for the X-axis to group a numeric variable for the Y-axis.

Enhanced Scatterplot with both horizontal and vertical reference lines

Graphics > Scatterplot > Scatter Plot Ref Lines

The Scatterplot dialog has been enhanced so that users can add an unlimited number of reference lines (horizontal and vertical axis) to the plot.

Enhancements to BlueSky Statistics R Editor and Output Syntax/Code Editor

For R-programmers many enhancements have been made to the BlueSky R Editor and the output syntax/code editor to improve ease of use and productivity with tooltips, find and replace, undo/redo, comment/uncomment blocks, etc.

Enhanced Normal Distribution Plot

Distribution > Normal > Normal Distribution Plot with Labels

The normal distribution plot will show the computed probability and x values on the plot for the shaded area for x value and quantiles, respectively
Plot one tail (left and right), two tails, and other ranges

Automatic randomization of generating normal sample distribution

Distribution > Normal > Sample from Normal Distribution

In addition to setting a seed value for reproducibility, the default option has been set to randomize automatically the sample data generation every time.

Automatic randomization of design creations of all DoE designs

DOE > Create Design > ….

In addition to setting a seed value for reproducibility, the default option has been set to randomize the creation of any DoE design every time automatically.

Enhanced Distribution Fit analysis

Analysis > Distribution Analysis > Distribution Fit P-value

The distribution fit analysis has been enhanced to compute AD, KS, and CVM tests and show test statistics, as well as corresponding p-values. These assist users in determining the best fit in addition to the existing AIC and BIC values.

Moreover, an option has been introduced for users to see only the comparison of distributions and skip displaying the analysis of the individual distribution fit analysis.

Tolerance Intervals

Six Sigma > Tolerance Intervals

A new Tolerance Intervals analysis has been introduced. The tolerance interval describes the range of values for a distribution with confidence limits calculated to a particular percentile of the distribution. These tolerance limits, taken from the estimated interval, are limits within which a stated proportion of the population is expected to occur.

Equivalence (and Minimal Effect) test

Analysis > Means > Equivalence test

This new feature tests for mean equivalence and minimal effects.

Nonlinear Least Square – all-purpose Non-Linear Regression modeling

Model Fitting > Nonlinear Least Square

Performs non-linear regression with flexibility and many user options to model, test, and plot.

Polynomial Models with different degrees

Model Fitting > Polynomial

Computes and fits an orthogonal polynomial model with a specified degree. Also, optionally compares multiple Polynomial models of different degrees side by side.

Enhanced Pareto Chart

Six Sigma > Pareto Chart > Pareto Chart

A new option has been added for data that does not have a count column but only has the raw data. Automatically computes cumulative frequency from Raw Data for plotting.

Frequency analysis with an option to draw a Pareto chart

Analysis > Summary > Frequency Plot

A new dialog has been introduced to plot (optionally) the Pareto Chart from the frequency table and, if desired, display the frequency table on the Datagrid.

MSA (Measurement System Analysis) Enhancements

Gage Study Design Table

Six Sigma > MSA > Design MSA Study

Users can generate a randomized design experiment table for any combination of the number of operators, parts, and replications to set up a Gage study table to perform experiments and collect the results to analyze the accuracy of the Gage under study with analysis like Gage R&R, Gage Bias, etc.

Enhanced Gage R&R

Six Sigma > MSA > Gage R&R

Many enhancements and options have been introduced to the Gage of R&R dialog and the underlying analysis

Report header table
Enlarged graphs
Nested gage data analysis, in addition to crossed
Usage of historical process std dev to estimate Gage Evaluation values (%StudyVar table)
Show %Process

Enhanced Gage Attribute Analysis

Six Sigma > MSA > Attribute Analysis

Many enhancements and options have been introduced to the Attribute Analysis dialog and the underlying analysis

Report header table
Accuracy and classification rate calculations, in addition to agreement and disagreement
Optional Cohen’s Kappa stats (between each pair of raters) in addition to Fleiss Kappa (multi-raters)

Enhanced Gage Bias Analysis

Six Sigma > MSA > Gage Bias Analysis

Many enhancements and options have been introduced to the Gage Bias Analysis dialog and the underlying analysis

Efficient single dialog with options for linearity and type-1 tests for one or more References
A new option – “Method to use for estimating repeatability std dev”
Cg and Cgk – calculated for different Reference values in one go
Run charts for every reference value and an overall run chart for all reference values
Usage of historical std dev to calculate RF (Reference Figure)
%RE, %EV are introduced, and all tables show how the computed values compared to the required/cut-off values specified by users on the dialog

PCA (Process Capability Analysis) Enhancements

Enhanced Process Capability Analysis (for normal data)

Six Sigma > Process Capability > Process Capability

pp_l = pp_k and ppU = ppk is shown when a one-sided tolerance is used
Removed underscores to only show Ppl, Ppk, Ppu, Cp, Cpk, .. etc
A new option – “Do not use unbiasing constant to estimate std dev for overall process capability indices” to compute overall Ppk (Ppl)
Underlying charts (xbar.one) renamed to MR or I Chart based on SD or MR
Handling of missing values
Customizable number of decimals to show on the plot
Standard Deviation label on the plot marked as “Overall StdDev” and “Within StdDev”

Process Capability Analysis for non-normal data

Six Sigma > Process Capability > Process Capability (Non-Normal)

A new dialog has been introduced to perform process capability analysis for non-normal data.

Multi-Vari graph

Six Sigma > Multi-Vari Chart

A new option has been added to adjust horizontal and vertical position offset to place/move the values for the data points on the plot.

Enhanced Shewhart Charts

Six Sigma > Shewhart Charts > …….

A new option has been added to all Shewhart Charts dialogs: the ability to add any number of spec/reference lines to the chart specified by the user.

The post BlueSky Statistics Enhancements first appeared on r4stats.com.To leave a comment for the author, please follow the link and comment on their blog: R | r4stats.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: BlueSky Statistics Enhancements

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Mastering the table() Function in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Mastering the table() Function in R, The table() function in R is a powerful tool for creating frequency tables, allowing you to quickly summarize the distribution of variables in your data.

In this article, we’ll explore the basics of table() and demonstrate its applications through practical examples.

Step-by-Step Data Science Coding Course

Syntax:Mastering the table() Function in R

The basic syntax of the table() function is:

table(x) Where x is a vector or a data frame.

Example 1: Frequency Table for One Variable

Let’s start with an example that demonstrates how to create a frequency table for the position variable in our data frame:

```

Create data framedf <- data.frame(player = c('AddJ', 'Bodkjgb', 'Chdgad', 'Dadgjdsn', 'dsjghdric', 'Frandgsk'), position = c('A', 'B', 'B', 'B', 'B', 'A'), points = c(51, 52, 52, 81, 70, 50))# View data framedf# Calculate frequency table for position variabletable(df$position)

``` The output will be a vector containing the frequency of each level of the position variable.

A B 2 4 Example 2: Frequency Table of Proportions for One Variable

In this example, we’ll use prop.table() to create a frequency table of proportions for the position variable:

```

Calculate frequency table of proportions for position variableprop.table(table(df$position))

``` The output will be a vector containing the proportion of each level of the position variable.

A B 0.3333333 0.6666667 Example 3: Frequency Table for Two Variables

Let’s create a frequency table for the position and points variable:

```

Calculate frequency table for position and points variabletable(df$position, df$points)

``` The output will be a matrix containing the frequency of each combination of levels of the position and points variables.

50 51 52 70 81 A 1 1 0 0 0 B 0 0 2 1 1 Example 4: Frequency Table of Proportions for Two Variables

In this example, we’ll use prop.table() to create a frequency table of proportions for the position and points variable:

```

Calculate frequency table of proportions for position and points variableprop.table(table(df$position, df$points))

``` The output will be a matrix containing the proportion of each combination of levels of the position and points variables.

50 51 52 70 81 A 0.1666667 0.1666667 0.0000000 0.0000000 0.0000000 B 0.0000000 0.0000000 0.3333333 0.1666667 0.1666667 Tips and Variations

You can use additional arguments with table() to specify specific levels or subsets of your data.
You can use prop.table() to create frequency tables of proportions instead of frequencies.
You can use options() to specify how many decimals to display in your proportion table.
You can use table() with other types of data structures, such as lists or matrices.

Conclusion

In conclusion, the table() function is a powerful tool in R that allows you to quickly create frequency tables and summarize the distribution of variables in your data.

By mastering this function, you can gain valuable insights into your data and make informed decisions.

With its flexibility and versatility, table() is an essential tool for any R programmer.

Error in apply: dim(X) must have a positive length
What is an adaptive clinical trial?
Highest Paying Data Science Skills-You should know!
How to Identify Outliers-Grubbs’ Test in R
Repeated Measures of ANOVA in R Complete Tutorial
Kerala lottery scams – How to avoid them?
Data-must-be-a-data-frame-or-other-object-coercible-by-fortify-not-an-s3-object-with-class-uneval
Change Legend Position in R Plots with examples
How to Perform Univariate Analysis in R

The post Mastering the table() Function in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering the table() Function in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn data analysis and manipulation, handling text data is a common task. One of the essential operations you might need to perform is converting strings to lowercase. In R, this is easily done using the tolower() function. Let’s explore how to convert your text data into lowercase, along with practical examples and a real-world use case.

The tolower() FunctionThe tolower() function converts all characters in a string to lowercase. Here’s the basic syntax:

tolower(string) * string: This is the input string or character vector that you want to convert to lowercase.

Why Convert to Lowercase?Converting strings to lowercase is useful for standardizing text data. It helps in comparison and searching, ensuring consistency, especially when dealing with user inputs, names, or categories.

ExamplesExample 1: Converting a Single String text <- "Hello World!"lower\_text <- tolower(text)print(lower\_text)

[1] "hello world!" Example 2: Converting a Vector of Strings fruits <- c("Apple", "Banana", "Cherry")lower\_fruits <- tolower(fruits)print(lower\_fruits)

[1] "apple" "banana" "cherry" Example 3: Handling Mixed Case Strings mixed\_case <- "ThiS Is A MiXeD CaSe StrIng."lower\_case <- tolower(mixed\_case)print(lower\_case)

[1] "this is a mixed case string." Practical Use: Checking User’s Favorite ColorA practical application of converting strings to lowercase is in user input validation. Let’s consider a simple function that checks a user’s favorite color and responds accordingly. By converting the input to lowercase, we can ensure that the function handles different cases uniformly.

Here’s the function:

```

Function to check user's favorite colorcheck_favorite_color <- function(color) { color <- tolower(color) # Convert input to lowercase if (color == "blue") { return("Blue is my favorite color!") } else if (color == "red") { return("Red is not a good choice!") } else { return("That's a nice color too!") }}# Test the functionprint(check_favorite_color("BLUE")) # Works with uppercase

```

[1] "Blue is my favorite color!"

print(check\_favorite\_color("Red")) # Works with mixed case

[1] "Red is not a good choice!"

print(check\_favorite\_color("green")) # Works with lowercase

[1] "That's a nice color too!" In this function, we use tolower() to ensure that the input is in lowercase, making it easier to compare against predefined color choices. This approach helps handle inputs consistently, regardless of how the user types them.

Understanding the CodeThe tolower() function converts uppercase characters to lowercase in a given string or vector of strings. It only affects alphabetic characters, leaving other characters unchanged. This makes it an essential tool for standardizing text data.

Try It OutNow it’s your turn! Experiment with different strings or scenarios where converting to lowercase can simplify your code and improve data consistency. Whether it’s for user input validation, data cleaning, or any other purpose, mastering this simple function can be incredibly useful in your R programming journey.

Feel free to share your experiences or any interesting use cases you’ve come across.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering String Conversion to Lowercase in R

View Details

[This article was first published on Achim Zeileis, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Python package ‘colorspace’ with tools for manipulating and assessing colors and palettes is now available from PyPI, accompanied by a documentation web page and an arXiv paper.

CitationReto Stauffer, Achim Zeileis (2024). “colorspace: A Python Toolbox for Manipulating and Assessing Colors and Palettes.” arXiv.org E-Print Archive arXiv:2407.19921 [cs.GR]. doi:10.48550/arXiv.2407.19921

AbstractThe Python colorspace package provides a toolbox for mapping between different color spaces which can then be used to generate a wide range of perceptually-based color palettes for qualitative or quantitative (sequential or diverging) information. These palettes (as well as any other sets of colors) can be visualized, assessed, and manipulated in various ways, e.g., by color swatches, emulating the effects of color vision deficiencies, or depicting the perceptual properties. Finally, the color palettes generated by the package can be easily integrated into standard visualization workflows in Python, e.g., using matplotlib, seaborn, or plotly.

Software* Package (PyPI): https://pypi.org/project/colorspace/ * Documentation: https://retostauffer.github.io/python-colorspace/ * Interactive apps: https://hclwizard.org/ * Repository (GitHub): https://github.com/retostauffer/python-colorspace/

MotivationColor is an integral element of visualizations and graphics and is essential for communicating (scientific) information. However, colors need to be chosen carefully so that they support the information displayed for all viewers (see e.g., Tufte 1990; Ware 2004; Wilke 2019). Therefore, suitable color palettes have been proposed in the literature (e.g., Brewer 1999; Ihaka 2003; Crameri, Shephard, and Heron 2020) and many software packages transitioned to better color defaults over the last decade. A prominent example from the Python community is matplotlib 2.0 (Hunter, Dale, Firing, Droettboom, and the Matplotlib Development Team 2017) which replaced the classic “jet” palette (a variation of the infamous “rainbow”) by the perceptually-based “viridis” palette. Hence a wide range of useful palettes for different purposes is provided in a number of Python packages today, including cmcramery (Rollo 2024), colormap (Cokelaer 2024), colormaps (Patel 2024), matplotlib (Hunter 2007), palettable (Davis 2023), or seaborn (Waskom 2021).

However, in most graphics packages colors are provided as a fixed set. While this makes it easy to use them in different applications, it is usually not easy to modify the perceptual properties or to set up new palettes following the same principles. The colorspace package addresses this by supporting color descriptions using different color spaces (hence the package name), including some that are based on human color perception. One notable example is the Hue-Chroma-Luminance (HCL) model which represents colors by coordinates on three perceptually-based axes: Hue (type of color), chroma (colorfulness), and luminance (brightness). Selecting colors along paths along these axes allows for intuitive construction of palettes that closely match many of the palettes provided in the packages listed above.

In addition to functions and interactive apps for HCL-based colors, the colorspace package also offers functions and classes for handling, transforming, and visualizing color palettes (from any source). In particular, this includes the simulation of color vision deficiencies (Machado Oliviera, and Fernandes 2009) but also contrast ratios, desaturation, lightening/darkening, etc.

The colorspace Python package was inspired by the eponymous R package (Zeileis, Fisher, Hornik, Ihaka, McWhite, Murrell, Stauffer, and Wilke 2020). It comes with extensive documentation at https://retostauffer.github.io/python-colorspace/, including many practical examples. Selected highlights are presented in the following.

Key functionalityHCL-based color palettesThe key functions and classes for constructing color palettes using hue-chroma-luminance paths (and then mapping these to hex codes) are:

qualitative_hcl: For qualitative or unordered categorical information, where every color should receive a similar perceptual weight.
sequential_hcl: For ordered/numeric information from high to low (or vice versa).
diverging_hcl: For ordered/numeric information around a central neutral value, where colors diverge from neutral to two extremes.

These functions provide a range of named palettes inspired by well-established packages but actually implemented using HCL paths. Additionally, the HCL parameters can be modified or new palettes can be created from scratch.

As an example, the figure below depicts color swatches for four viridis variations. The first pal1 sets up the palette from its name. It is identical to the second pal2 which employes the HCL specification directly: The hue ranges from purple (300) to yellow (75), colorfulness (chroma) increases from 40 to 95, and luminance (brightness) from dark (15) to light (90). The power parameter chooses a linear change in chroma and a slightly nonlinear path for luminance.

In pal3 and pal4 the most HCL properties are kept the same but some are modified: pal3 uses a triangular chroma path from 40 via 90 to 20, yielding muted colors at the end of the palette. pal4 just changes the starting hue for the palette to green (200) instead of purple. All four palettes are visualized by the swatchplot function from the package.

The objects returned by the palette functions provide a series of methods, e.g., pal1.settings for displaying the HCL parameters, pal1(3) for obtaining a number of hex colors, or pal1.cmap() for setting up a matplotlib color map, among others.

from colorspace import palette, sequential\_hcl, swatchplotpal1 = sequential\_hcl(palette = "viridis")pal2 = sequential\_hcl(h = [300, 75], c = [40, 95], l = [15, 90], power = [1., 1.1])pal3 = sequential\_hcl(palette = "viridis", cmax = 90, c2 = 20)pal4 = sequential\_hcl(palette = "viridis", h1 = 200)swatchplot({"Viridis (and altered versions of it)": [ palette(pal1(7), "By name"), palette(pal2(7), "By hand"), palette(pal3(7), "With triangular chroma"), palette(pal4(7), "With smaller hue range") ]}, figsize = (8, 1.75)); An overview of the named HCL-based palettes in colorspace is depicted below.

from colorspace import hcl\_paletteshcl\_palettes(plot = True, figsize = (20, 15)) Palette visualization and assessmentTo better understand the properties of palette pal4, defined above, the following figure shows its HCL spectrum (left) and the corresponding path through the HCL space (right).

The spectrum in the first panel shows how the hue (right axis) changes from about 200 (green) to 75 (yellow), while chroma and luminance (left axis) increase from about 20 to 95. Note that the kink in the chroma curve for the greenish colors occurs because such dark greens cannot have higher chromas when represented through RGB-based hex codes. The same is visible in the second panel where the path moves along the outer edge of the HCL space.

pal4.specplot(figsize = (5, 5));pal4.hclplot(n = 7, figsize = (5, 5)); Color vision deficiencyAnother important assessment of a color palette is how well it works for viewers with color vision deficiencies. This is exemplified below by depicting a demo plot (heatmap) under “normal” vision (left), deuteranomaly (colloquially known as “red-green color blindness”, center), and desaturated (gray scale, right). The palette in the top row is the traditional fully-saturated RGB rainbow, deliberately selected here as a palette with poor perceptual properties. It is contrasted with a perceptually-based sequential blue-yellow HCL palette in the bottom row.

The sequential HCL palette is monotonic in luminance so that it is easy to distinguish high-density and low-density regions under deuteranomaly and desaturation. However, the rainbow is non-monotonic in luminance and parts of the red-green contrasts collapse under deuteranomaly, making it much harder to interpret correctly.

from colorspace import rainbow, sequential\_hclcol1 = rainbow(end = 2/3, rev = True)(7)col2 = sequential\_hcl("Blue-Yellow", rev = True)(7)from colorspace import demoplot, deutan, desaturateimport matplotlib.pyplot as pltfig, ax = plt.subplots(2, 3, figsize = (9, 4))demoplot(col1, "Heatmap", ax = ax[0,0], ylabel = "Rainbow", title = "Original")demoplot(col2, "Heatmap", ax = ax[1,0], ylabel = "HCL (Blue-Yellow)")demoplot(deutan(col1), "Heatmap", ax = ax[0,1], title = "Deuteranope")demoplot(deutan(col2), "Heatmap", ax = ax[1,1])demoplot(desaturate(col1), "Heatmap", ax = ax[0,2], title = "Desaturated")demoplot(desaturate(col2), "Heatmap", ax = ax[1,2])plt.show() Integration with Python graphics packagesTo illustrate that colorspace can be easily combined with different graphics workflows in Python, the code below shows a heatmap (two-dimensional histogram) from matplotlib and multi-group density from seaborn. The code below employs an example data set from the package (using pandas) with daily maximum and minimum temperature. For matplotlib the colormap (.cmap(); LinearSegmentedColormap) is extracted from the adapted viridis palette pal3 defined above. For seaborn the hex codes from a custom qualitative palette are extracted via .colors(4).

from colorspace import dataset, qualitative\_hclimport matplotlib.pyplot as pltimport seaborn as snsdf = dataset("HarzTraffic")fig = plt.hist2d(df.tempmin, df.tempmax, bins = 20, cmap = pal3.cmap().reversed())plt.title("Joint density daily min/max temperature")plt.xlabel("minimum temperature [deg C]")plt.ylabel("maximum temperature [deg C]")plt.show()pal = qualitative\_hcl("Dark 3", h1 = -180, h2 = 100)g = sns.displot(data = df, x = "tempmax", hue = "season", fill = "season", kind = "kde", rug = True, height = 4, aspect = 1, palette = pal.colors(4))g.set\_axis\_labels("temperature [deg C]") g.set(title = "Distribution of daily maximum temperature given season")plt.show() Dependencies and availabilityThe colorspace is available from PyPI at https://pypi.org/project/colorspace. It is designed to be lightweight, requiring only numpy (Harris et al. 2020) for the core functionality. Only a few features rely on matplotlib, imageio (Klein et al. 2024), and pandas (The Pandas Development Team 2024). More information and an interactive interface can be found on https://hclwizard.org/. Package development is hosted on GitHub at https://github.com/retostauffer/python-colorspace. Bug reports, code contributions, and feature requests are warmly welcome.

References* Brewer CA (1999). “Color Use Guidelines for Data Representation.” In Proceedings of the Section on Statistical Graphics, American Statistical Association, pp. 55–60. Alexandria, VA. * Cokelaer T (2024). Colormap. Version 1.1.0, Python Package Index (PyPI), URL https://pypi.org/project/colormap/. * Crameri F, Shephard GE, Heron PJ (2020). “The Misuse of Colour in Science Communication.” Nature Communications, 11(5444), 1–10. doi:10.1038/s41467-020-19160-7. * Davis M (2023). palettable: Color Palettes for Python. Version 3.3.3, Python Package Index (PyPI), URL https://pypi.org/project/palettable/. * Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, Wieser E, Taylor J, Berg S, Smith NJ, Kern R, Picus M, Hoyer S, van Kerkwijk MH, Brett M, Haldane A, del Río JF, Wiebe M, Peterson P, Gérard-Marchant P, Sheppard K, Reddy T, Weckesser W, Abbasi H, Gohlke C, Oliphant TE (2020). “Array Programming with NumPy.” Nature, 585(7825), 357–362. doi:10.1038/s41586-020-2649-2. * Hunter JD (2007). “Matplotlib: A 2D Graphics Environment.” Computing in Science & Engineering, 9(3), 90–95. doi:10.1109/mcse.2007.55. * Hunter JD, Dale D, Firing E, Droettboom M, the Matplotlib Development Team (2017). “What’s New in Matplotlib 2.0 (Jan 17, 2017), Changes to the Default Style.” Accessed 2024-07-22, URL https://matplotlib.org/stable/users/prev_whats_new/dflt_style_changes.html. * Ihaka R (2003). “Colour for Presentation Graphics.” In K Hornik, F Leisch, A Zeileis (eds.), Proceedings of the 3rd International Workshop on Distributed Statistical Computing, Vienna, Austria. ISSN 1609-395X, URL https://www.R-project.org/conferences/DSC-2003/Proceedings/Ihaka.pdf. * Klein A, Wallkötter S, Silvester S, Rynes A, actions-user, Müller P, Nunez-Iglesias J, Harfouche M, Schrangl L, Dennis, Lee A, Pandede, McCormick M, OrganicIrradiation, Rai A, Ladegaard A, van Kemenade H, Smith TD, Vaillant G, jackwalker64, Nises J, Komarčevič M, rreilink, Barnes C, Zulko, Hsieh PC, Rosenstein N, Górny M, scivision, Singleton J (2024). Imageio/Imageio: V2.34.2. doi:10.5281/zenodo.12514964. Version 2.34.2, Zenodo. * Machado GM, Oliviera MM, Fernandes LAF (2009). “A Physiologically-Based Model for Simulation of Color Vision Deficiency.” IEEE Transactions on Visualization and Computer Graphics, 15(6), 1291–1298. doi:10.1109/tvcg.2009.113. * Patel P (2024). Colormaps. Version 0.4.2, Python Package Index (PyPI), URL https://pypi.org/project/colormaps/. * Rollo C (2024). cmcrameri: Python Wrapper around Fabio Crameri’s Perceptually Uniform Colormaps. Version 1.9, Python Package Index (PyPI), URL https://pypi.org/project/cmcrameri/. * The Pandas Development Team (2024). pandas-Dev/Pandas: Pandas. doi:10.5281/zenodo.10957263. Version 2.2.2, Zenodo. * Tufte E (1990). Envisioning Information. Graphics Press, Cheshire. * Ware C (2004). “Color.” In Information Visualization: Perception for Design, chapter 4, pp. 103–149. Morgan Kaufmann Publishers Inc. * Waskom ML (2021). “seaborn: Statistical Data Visualization.” Journal of Open Source Software, 6(60), 3021. doi:10.21105/joss.03021. * Wilke CO (2019). Fundamentals of Data Visualization. O’Reilly Media. ISBN 1492031089. URL https://clauswilke.com/dataviz/color-basics.html. * Zeileis A, Fisher JC, Hornik K, Ihaka R, McWhite CD, Murrell P, Stauffer R, Wilke CO (2020). “colorspace: A Toolbox for Manipulating and Assessing Colors and Palettes.” Journal of Statistical Software, 96(1), 1–49. doi:10.18637/jss.v096.i01.

To leave a comment for the author, please follow the link and comment on their blog: Achim Zeileis.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: colorspace: A Python Toolbox for Colors and Palettes

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#195–196

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #197Today we have some warehouse management issue. We were provided with log of products stocked in three stores in three months. You would say, Nice, but we don’t have any stock levels in there just INs and OUTs. We have to fix it and find Start and End Stock for each move of products assuming that we started on January. Try to find nice way to do it.

Loading libraries and data library(tidyverse)library(readxl)path = "Power Query/PQ\_Challenge\_197.xlsx"input = read\_xlsx(path, range = "A1:E21")test = read\_xlsx(path, range = "H1:N21") Transformation result <- input %>% group\_by(Item, Store) %>% mutate(data = accumulate(`Stock IN` - `Stock OUT`, `+`), `Start Stock` = lag(data, default = first(`Stock IN`)), `End Stock` = data) %>% ungroup() %>% select(-data) Validation identical(result, test)#> [1] TRUE Puzzle #198I don’t know if scenario like for today is really used in any industry, but anyway it is nice case to show how we manipulate data. We need to find max value per month, and then for each row in month, place cumulative sum of those maxes. I used little self join here, but of course it can be done many other ways as well.

Loading libraries and data library(readxl)library(tidyverse)path = "Power Query/PQ\_Challenge\_198.xlsx"input = read\_excel(path, range = "A1:B20")test = read\_excel(path, range = "D1:F20") Transformation result = input %>% mutate(month = month(Date)) %>% summarise(Max = max(Value), .by = month) %>% mutate(`Running Total` = cumsum(Max)) %>% right\_join(input %>% mutate(month = month(Date)), by = "month") %>% select(Date, Value, `Running Total`) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.When it comes to time series forecasting in R, one thing you don’t lack is options. There are dozens of algorithms and their variations you can choose from, and doing so is usually overwhelming to newcomers.

That’s where this article chimes in. In the next 15 minutes, you’ll go through dataset preprocessing and simple forecasting methods to seasonal ARIMA models. You’ll also learn how to evaluate and compare time series models through evaluation metrics specific to this type of algorithms.

Buckle in, it’s gonna be a long one.

Don’t know the first thing about Time Series Analysis in R? Make sure to read our latest prequel in this series.

Table of contents:* Data Preprocessing for Time Series Forecasting in R * Time Series Forecasting in R – The Complete Guide * How to Evaluate Time Series Forecasting Models in R * Summing up Time Series Forecasting in R

Data Preprocessing for Time Series Forecasting in RJust like in the previous article, you’ll use the Airline passengers dataset. It’s not the most exciting one, but requires almost no preprocessing. That’s exactly what you want when first starting out.

You’ll need the following R packages to follow along. If one or more aren’t available on your system, simply install them by running install.packages(“”) from the R console:

library(zoo)library(ggplot2)library(forecast)library(Metrics) Alright, assuming you have the dataset downloaded and the packages installed, you’re good to proceed!

Dataset Loading and Data Type ConversionThe biggest “issue” with the dataset is that it displays the date information in YYYY-MM format. It’s an issue because R won’t be able to infer it by default, so you’ll have to assist.

The zoo package has an as.yearmon() function that allows you to pass in a year-month string. It will be converted to an appropriate date format automatically:

data <- read.csv("airline-passengers.csv")data$Month <- as.yearmon(data$Month)max(data$Month)head(data) Image 1 – Head of the Airline passengers datasetAs you can see, there are only two columns – time period and the number of passengers in thousands. The data spans from January 1949 to December 1960, so we have a decent chunk of data to work with.

Train/Test SplitThe next step you want to do when forecasting time series data in R is to split the original dataset into training and testing subsets. The idea is to build your model(s) only on training data and then compare its predictions to the test set (actual data).

But with time series, you don’t want to split the data randomly. Time is a crucial component here, and you always want to predict the future in a sequential matter.

The following snippet shows you how to extract the last two years for testing purposes:

test\_set\_start\_date <- as.yearmon("Jan 1959")train\_set <- subset(data, Month < test\_set\_start\_date)test\_set <- subset(data, Month >= test\_set\_start\_date)dim(train\_set)dim(test\_set) Image 2 – Train/Test set dimensionalityThere are 12 months in a year (no missing records), so you now have 10 years of data for training and 2 years for evaluation.

To further drive this point home, let’s visualize both series:

ggplot() + geom\_line(data = train\_set, aes(x = Month, y = Passengers, color = "Training"), size = 1) + geom\_line(data = test\_set, aes(x = Month, y = Passengers, color = "Testing"), size = 1) + labs( title = "Airline Passengers - Training and Testing Sets", x = "Date", y = "Passengers" ) + scale\_color\_manual(values = c("Training" = "#12355B", "Testing" = "#D72638"), name = "Airline passengers") + theme\_minimal() + theme(plot.title = element\_text(size = 20)) Image 3 – Train and test set visualizedYou can now clearly see how the testing set continues where the training set left off. Splitting the data randomly as you would do for traditional machine learning datasets wouldn’t make sense.

The increasing trend and seasonal components are clearly visible. Most people travel by plane during the summer months, and the peaks only get higher with time. It will be interesting to see if time series algorithms can capture that.

‍

Note: We’ll use the above code snippet for visualization purposes throughout the article. The code snippet will always be slightly modified to accommodate a different number of lines shown. To keep the code portions minimal, we won’t share the visualization code.

Time Series Forecasting in R – The Complete GuideThis section will walk you through different time series forecasting algorithms ranging in complexity. Let’s start simple by using no forecasting algorithm whatsoever.

Simple and Naive ForecastingPredicting future values as a flat line of training set average is typically used to make a baseline in time series forecasting. The premise is simple – you don’t need a sophisticated algorithm if a simple average does the job.

That might be the case for time series data that shows no trend or seasonality, but will be far from an optimal solution in our case.

To calculate this average flat line, simply get the mean value of a training set and repeat it N times, where N is the length of the test set:

train\_set\_avg <- mean(train\_set$Passengers)simple\_avg\_predictions <- data.frame( Month = test\_set$Month, Passengers = rep(train\_set\_avg, nrow(test\_set))) Image 4 – Forecasting with simple averagingNot impressive at all and the error looks huge, but let’s not worry about metrics just yet. Just from the looks of it, it doesn’t seem to follow the test set well.

Moving AveragesFor the rest of the article, you’ll want to convert the Passengers numeric vector to a specific time series object. The forecast::ts() function is here for you. Since the data captures the fluctuations within a year, you can specify freqency = 12:

series\_ts <- ts(train\_set$Passengers, frequency = 12)series\_ts You now have a specific time series object to work with. It’s also much easier to inspect:

Image 5 – Converting a column to a time series objectNow, onto moving averages. You can think of these as a specific technique used to smooth out fluctuations and highlight long-term trends within the data. You select a window size (the number of data points used to calculate the average), calculate the average of a window, and then move the window over the series.

There are many flavors of moving averages, but we’ll stick with a simple one here. The others may put more weight on different data points in the window.

Moving averages are useful for smoothing out the noise due to averaging and identifying general trends. They fall behind when there are sudden changes and anomalies in the data. It’s also worth noting that a window size can significantly impact the results you get, so it’s worth playing around with this parameter.

‍We’ll do just that – create the model and forecasts for moving averages of window sizes 3, 6, and 12:

ma3 <- ma(series\_ts, order = 3, centre = FALSE)ma6 <- ma(series\_ts, order = 6, centre = FALSE)ma12 <- ma(series\_ts, order = 12, centre = FALSE)ma3\_forecast <- forecast(ma3, h = nrow(test\_set))ma6\_forecast <- forecast(ma6, h = nrow(test\_set))ma12\_forecast <- forecast(ma12, h = nrow(test\_set))ma\_forecast\_df <- data.frame( Month = test\_set$Month, MA3 = ma3\_forecast$mean, MA6 = ma6\_forecast$mean, MA12 = ma12\_forecast$mean)ma\_forecast\_df You now have a dataframe with actual values and predictions:

Image 6 – Moving average forecastsWhen visualized, it’s easy to see how moving averages work. The larger the window size you choose, the fewer “peaks” and “valleys” you’ll capture. You’re essentially smoothing out the series with the aim of keeping the trend and somewhat removing the seasonal effects:

Image 7 – Moving average forecasts visualizedThat’s moving averages for you. Up next, let’s dive into exponential smoothing.

Exponential SmoothingUnlike moving averages, exponential smoothing algorithms will assign exponentially decreasing weights to historical data. In plain English, this means that more recent data points are more important for forecasting. This fact makes exponential smoothing algorithms adaptable and useful in dynamic environments, as more weight is placed on recent observations.

In practice, you’ll encounter three types of exponential smoothing algorithms:

Simple Exponential Smoothing (SES) – A method that assigns exponentially decreasing weights to past observations. It’s typically used when data shows no trend or seasonality.
Double Exponential Smoothing (DES) – Also known as Holt’s method, it extends SES by capturing trends in the data. It’s useful for forecasting time series data that has a clear trend but no seasonality.
Triple Exponential Smoothing (TES) – Also known as the Holt-Winters method, it extends DES by capturing seasonality. It’s typically used when time series data shows both trend and seasonality, which most time series datasets do.

By knowing this, it seems like TES would be a perfect candidate for forecasting on the Airplane passengers dataset!

But let’s test all three. The HoltWinters() method runs triple exponential smoothing by default, but you can turn off beta and gamma parameters to build SES and DES models:

ses <- HoltWinters(series\_ts, beta = FALSE, gamma = FALSE)des <- HoltWinters(series\_ts, gamma = FALSE)tes <- HoltWinters(series\_ts)ses\_forecast <- forecast(ses, h = nrow(test\_set))des\_forecast <- forecast(des, h = nrow(test\_set))tes\_forecast <- forecast(tes, h = nrow(test\_set))exsm\_forecast\_df <- data.frame( Month = test\_set$Month, SES = ses\_forecast$mean, DES = des\_forecast$mean, TES = tes\_forecast$mean)exsm\_forecast\_df Once again, you have a dataframe with actual data and predictions for every algorithm:

Image 8 – Exponential smoothing forecastsThe theory makes sense as soon as the predictions are visualized. SES is essentially a flat line that doesn’t capture either trend or seasonality. DES captures the trend correctly, but TES looks almost identical to the actual data from the test set:

Image 9 – Exponential smoothing forecasts visualizedThe HoltWinters() function also allows you to change how the seasonal component is combined with the modeling formula. You can either add it (additive) or multiply (multiplicative) it with the rest.

Let’s create a model for both to see how they differ:

tes\_seasonal\_add <- HoltWinters(series\_ts, seasonal = "additive")tes\_seasonal\_mul <- HoltWinters(series\_ts, seasonal = "multiplicative")tes\_seasonal\_add\_forecast <- forecast(tes\_seasonal\_add, h = nrow(test\_set))tes\_seasonal\_mul\_forecast <- forecast(tes\_seasonal\_mul, h = nrow(test\_set))exsm\_tes\_forecast\_df <- data.frame( Month = test\_set$Month, TES = tes\_forecast$mean, TESAdd = tes\_seasonal\_add\_forecast$mean, TESMul = tes\_seasonal\_mul\_forecast$mean) It seems like HoltWinters() uses an additive seasonality option by default. Switching it to multiplicative better captures the peaks in the series, but is slightly worse on the valleys:

Image 10 – Further exponential smoothing explorationAs said earlier, let’s not worry about the actual correctness just yet. There’ll be a dedicated section later in the article.

But first, let’s go over a slightly more complex family of time series forecasting models – ARIMA.

ARIMA ModelsCovering all there is to ARIMA models would require a couple of articles at least, so we’ll go only the basics for now.

ARIMA stands for Auto Regressive Integrated Moving Average and is a popular family of algorithms to capture a wide range of patterns in the data. The I (Integrated) part is an interesting one – it assumes your data isn’t stationary and stands for the order of differencing required to make it stationary.

The Auto Regressive (AR) part says that the value of a variable at a given time point is modeled as a linear function of its previous variables, or lags. You already know what the Moving Average (MA) is.

The Arima() function in R allows you to create a full ARIMA model, or only to use it partially. For example, you can tweak the parameters to build only AR, MA, and ARMA models. The parameters you’ll typically see when it comes to ARIMA are named (p, d, q).

To make matters even more complicated, ARIMA can also capture seasonal patterns. This algorithm variation is called SARIMA. It has a somewhat more complex parameter structure – (p, d, q)(P, D, Q)m. You already know what the first set of parameters covers. The uppercased parameters are here to add the seasonal components with a seasonal period of m. In the case of Airline passengers, that would be 12.

The good thing about R is that you don’t have to worry about manual parameter configuration. The auto.arima() function automates the entire process for you, as it automatically chooses the best-resulting values for both seasonal and non-seasonal models.

The following code snippet builds 6 models – AR, MA, ARMA, manual ARIMA, auto ARIMA without seasonality, and auto ARIMA with seasonality:

ar\_model <- Arima(series\_ts, order = c(1, 0, 0))ma\_model <- Arima(series\_ts, order = c(0, 0, 1))arma\_model <- Arima(series\_ts, order = c(1, 0, 1))arima\_model <- Arima(series\_ts, order = c(1, 1, 1))auto\_arima\_no\_season\_model <- auto.arima(series\_ts, seasonal = FALSE)auto\_arima\_season\_model <- auto.arima(series\_ts, seasonal = TRUE)ar\_forecasts <- forecast(ar\_model, h = nrow(test\_set))ma\_forecasts <- forecast(ma\_model, h = nrow(test\_set))arma\_forecasts <- forecast(arma\_model, h = nrow(test\_set))arima\_forecasts <- forecast(arima\_model, h = nrow(test\_set))auto\_arima\_no\_season\_forecasts <- forecast(auto\_arima\_no\_season\_model, h = nrow(test\_set))auto\_arima\_season\_forecasts <- forecast(auto\_arima\_season\_model, h = nrow(test\_set))arima\_forcast\_df <- data.frame( Month = test\_set$Month, AR = ar\_forecasts$mean, MA = ma\_forecasts$mean, ARMA = arma\_forecasts$mean, ARIMA = arima\_forecasts$mean, AutoARIMANoSeason = auto\_arima\_no\_season\_forecasts$mean, AutoARIMASeason = auto\_arima\_season\_forecasts$mean)arima\_forcast\_df These are the results you’ll get:

Image 11 – ARIMA forecastsWhen inspected visually, it’s easy to see that seasonal ARIMA, or SARIMA performs the best:

Image 12 – ARIMA forecasts visualizedLooking at the above chart reveals one fact that’s often overlooked in data science – Just because a model is more complex, it doesn’t mean it’s better. SARIMA looks like a top performer here, but the predicted line still looks worse than the one yielded by exponential smoothing models.

Let’s verify if that’s the case by going through an evaluation process.

How to Evaluate Time Series Forecasting Models in RBy now, you’ve only looked at the results visually and tried to eyeball which prediction line is the closest to the actual data. It’s not the worst approach, but we’d rather look at the concrete numbers.

First things first, you should collect all of your model predictions into a single dataframe object, alongside the ground truth data:

all\_model\_data <- data.frame( Month = test\_set$Month, Passengers = test\_set$Passengers, AVG = simple\_avg\_predictions$Passengers, MA = ma\_forecasts$mean, MA3 = ma3\_forecast$mean, MA6 = ma6\_forecast$mean, MA12 = ma12\_forecast$mean, SES = ses\_forecast$mean, DES = des\_forecast$mean, TES = tes\_forecast$mean, TESAdd = tes\_seasonal\_add\_forecast$mean, TESMul = tes\_seasonal\_mul\_forecast$mean, AR = ar\_forecasts$mean, ARMA = arma\_forecasts$mean, ARIMA = arima\_forecasts$mean, AutoARIMANoSeason = auto\_arima\_no\_season\_forecasts$mean, AutoARIMASeason = auto\_arima\_season\_forecasts$mean)all\_model\_data We’ve built a lot of models today, that’s for sure:

Image 13 – Actual data vs. predictions from all modelsOnto the evaluation now.

With time series, you typically want to use metrics like Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Squared Error (RMSE). MAE will show you how far off the model is in terms of thousands of passengers on average. MAPE will show the same but expressed as a percentage. RMSE will be similar to MAE, but will penalize larger errors more due to squaring.

Overall, you can easily loop over the columns and use the Metrics package to calculate these 3 values for all algorithms:

mae\_values <- c()mape\_values <- c()rmse\_values <- c()for (col in names(all\_model\_data)[3:ncol(all\_model\_data)]) { mae\_values <- c(mae\_values, mae(all\_model\_data$Passengers, all\_model\_data[[col]])) mape\_values <- c(mape\_values, mape(all\_model\_data$Passengers, all\_model\_data[[col]])) rmse\_values <- c(rmse\_values, rmse(all\_model\_data$Passengers, all\_model\_data[[col]]))}model\_test\_set\_metrics <- data.frame( Model = names(all\_model\_data)[3:ncol(all\_model\_data)], MAE = mae\_values, MAPE = mape\_values, RMSE = rmse\_values)model\_test\_set\_metrics Here are the results we got:

Image 14 – Model evaluation through MAE, MAPE, and RMSEOur baseline (predictions as a flat line of train set averages) is by far the worst model in all three metrics – that’s good, as it means it makes sense to invest time into building time series models.

ARIMA model without seasonality seems to be performing better than SARIMA, even though SARIMA looks better on the chart. That’s because of the overexaggerated valleys.

‍But the stars of the show are your simple exponential smoothing models. Triple exponential smoothing (Holt-Winters) seems to do best for this dataset, being wrong for about 35.8K passengers on average, or 6.7%. That’s impressive!

Summing up Time Series Forecasting in ROverall, there’s a lot that goes into time series forecasting in general, but R hides 99% of math and abstractions from you. It’s a good thing if you’re a practitioner, as the only thing you need to build highly accurate models is a couple of lines of code and a curious mind. If you want to dive deeper into the math, the functions shown today allow you to tweak all the possible formula parameters.

Evaluating the performance of your time series models is also a breeze. Just make sure you choose a metric that’s relevant for time series data.

You now know how to build basic time series models in R. But what about machine learning? Can you apply regression or boosting algorithms to time series data? Make sure to stay tuned to the Appsilon blog, and we’ll make sure to let you know.

Looking to make your time series charts animated and interactive? Be sure to check out our guide on R Highcharts visualization package.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Time Series Forecasting In R: From Moving Averages To Seasonal ARIMA

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, R users! Today, we’ll dive into a common text manipulation task: extracting strings before a space. This is a handy trick for dealing with names, addresses, or any text data where you need to isolate the first part of a string.

We’ll explore three approaches: using base R, stringr, and stringi. Each method offers its unique advantages, so you can choose the one that fits your style best.

ExamplesBase R ApproachLet’s start with base R. The sub function is a versatile tool for pattern matching and replacement. To extract the string before a space, we can use a regular expression.

```

Sample datatext <- c("John Doe", "Jane Smith", "Alice Johnson")# Extract strings before the first spacefirst_part_base <- sub(" .*", "", text)# Display the resultprint(first_part_base)

```

[1] "John" "Jane" "Alice" In this example, the sub function replaces the space and everything after it with an empty string, effectively extracting the first part of each string.

Using stringrNext, let’s see how stringr simplifies this task. The stringr package, part of the tidyverse, provides a consistent and easy-to-use interface for string manipulation.

```

Load stringr packagelibrary(stringr)# Sample datatext <- c("John Doe", "Jane Smith", "Alice Johnson")# Extract strings before the first spacefirst_part_stringr <- str_extract(text, "^[^ ]+")# Display the resultprint(first_part_stringr)

```

[1] "John" "Jane" "Alice" Here, str_extract is used with a regular expression to match and extract the part of the string before the first space. The ^[^ ]+ pattern matches the beginning of the string () followed by one or more characters that are not a space ([^ ]+).

Using stringiFinally, let’s use stringi, a powerful package for advanced string operations. stringi functions are optimized for performance, making it a great choice for handling large datasets.

```

Load stringi packagelibrary(stringi)# Sample datatext <- c("John Doe", "Jane Smith", "Alice Johnson")# Extract strings before the first spacefirst_part_stringi <- stri_extract_first_regex(text, "^[^ ]+")# Display the resultprint(first_part_stringi)

```

[1] "John" "Jane" "Alice" With stringi, stri_extract_first_regex performs similarly to str_extract from stringr, using the same regular expression pattern.

ConclusionEach method—base R, stringr, and stringi—offers a straightforward way to extract strings before a space. Whether you prefer the simplicity of base R, the tidyverse consistency of stringr, or the performance optimization of stringi, you have powerful tools at your disposal.

I encourage you to try these examples on your own datasets. Text manipulation is a fundamental skill in data analysis, and mastering these techniques will enhance your ability to clean and prepare data for analysis.

Feel free to share your experiences and any additional tips you might have in the comments. Happy coding!

```

To run the examples, just copy and paste the code blocks into your R script or R console.# Let me know how it goes!

``` Until next time, keep exploring the wonders of R!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extracting Strings Before a Space in R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.ShinyProxy has emerged as a powerful solution for deploying Shiny applications, empowering organizations to share interactive data visualizations and analyses.

With its robust feature set, ShinyProxy has simplified the process of managing and scaling Shiny apps for a wide range of use cases. The ShinyProxy 3.1.1 release marks a significant leap forward, introducing new features that optimize performance, enhance resource management, and expand deployment options.

The two main scalable hosting solutions are Posit Connect and ShinyProxy. We did the research for the comparison, you do the reading.

This article explores the key advancements in ShinyProxy 3.1.1 and how they empower organizations to unlock new levels of scalability and efficiency.

Impact Of New FeaturesShinyProxy 3.1.1 introduces several powerful features designed to address the growing demands of Shiny application deployments. These features offer a compelling set of advantages:

Enhanced Resource Efficiency: The new container sharing feature allows multiple users to utilize a single container, dramatically optimizing resource allocation and reducing overall server load.
Faster User Experience: Pre-initialization of containers ensures that applications are ready for use, minimizing wait times for users and providing a seamless and responsive experience.
Expanded Deployment Options: The addition of AWS ECS support opens up new possibilities for serverless deployments, providing organizations with greater flexibility and scalability.
Streamlined Container Management: Improved container naming conventions and an updated Docker library enhance usability and ensure compatibility with the latest Docker releases.
Enhanced Control and Security: The ability to set user limits at a global and application-specific level provides administrators with greater control over resource usage and strengthens security measures.

We prepped everything you need to know about running your Shiny apps in a container. Learn more in our comprehensive guide.

More Efficient Container Management With Container SharingTraditionally, ShinyProxy operated on a one-user-per-container model. While this approach offers isolation and customization benefits, it can lead to increased resource consumption, especially when dealing with a large number of users. ShinyProxy 3.1.1 introduces the groundbreaking concept of container sharing, allowing multiple users to share a single container.

This new approach in container management significantly impacts resource utilization and scalability:

Reduced Resource Consumption: Container sharing drastically reduces the number of containers required to serve a given number of users, leading to substantial savings in CPU, memory, and other resources.
Increased Scalability: With container sharing, ShinyProxy can now efficiently handle a much larger number of concurrent users with the same hardware infrastructure.
Faster Startup Times: Although container sharing might slightly increase the initial startup time of a container, subsequent users connecting to the shared container experience significantly faster startup times compared to launching a new container.

To ensure that user-specific information is still accessible within a shared container environment, ShinyProxy utilizes HTTP headers to transmit crucial data like usernames and groups directly to the application. This approach eliminates the reliance on environment variables, which are insufficient in shared containers.

Configuration:To enable container sharing, you can specify the seats-per-container property in your application’s configuration. For example:

proxy: specs: - id: my-shared-app display-name: My Shared App container-cmd: ["R", "-e", "shiny::runApp('/app')"] container-image: openanalytics/shinyproxy-demo container-network: sp-net seats-per-container: 5 ‍

This configuration allows up to five users to share a single container instance for the “my-shared-app” application.

Achieving Instant App Readiness With Pre-InitializationShinyProxy 3.1.0 introduced the concept of pre-initialization, a powerful feature that dramatically improves application startup times. Instead of waiting for a container to launch when a user requests an application, ShinyProxy can pre-initialize a pool of containers ready for immediate use.

When a user requests an application, they are instantly assigned to one of these pre-initialized containers, eliminating the wait time associated with container startup. This results in a seamless and highly responsive user experience.

The impact of pre-initialization is particularly significant in scenarios with:

High User Volume: Pre-initialization ensures that a pool of readily available containers can handle sudden spikes in user demand, preventing delays and ensuring a smooth experience for all users.
Resource-Intensive Applications: For applications that require substantial resources to start, pre-initialization minimizes the perceived wait time for users, making the application feel more responsive.

Configuration:To enable pre-initialization, you can configure the minimum-seats-available property in your application’s configuration:

proxy: specs: - id: my-pre-initialized-app display-name: My Pre-Initialized App container-cmd: ["R", "-e", "shiny::runApp('/app')"] container-image: openanalytics/shinyproxy-demo container-network: sp-net minimum-seats-available: 3 ‍

This configuration ensures that at least three pre-initialized container instances are always available for the “my-pre-initialized-app” application.

Deeper Dive Into The Other New FeaturesContainer Re-useWhether a (pre-initialized) container can be re-used by a different user (has no effect when minimum-seats-available isn’t specified).

allow-container-re-use: true ‍

Normally ShinyProxy removes the container after first use. It can only be used when seats-per-container equals 1. When set to false, you benefit from pre-initialized containers, while still having the guarantee that a container is used by a single user only..

‍

Scale Down ControlShinyProxy waits to scale-down after a scale-up (defaults to two minutes). This means that if a user stops using an app, the container is released only if there was no scale-up in 2 minutes.

scale-down-delay: 2 ‍

By increasing this value (in minutes), you can prevent too many scale-up and scale-downs in a short period.

‍

For more details on container sharing and pre-initialization, check out the official documentation.

Custom Container NamesSimplifies container management by allowing administrators to customize container names using SpEL expressions. This provides greater flexibility and control in identifying and managing containers within a deployment.

‍

Configuration:

container-name: 'sp-container-#{proxy.id}-#{container.index}' ‍

Up-to-Date Docker LibraryShinyProxy 3.1.1 includes an updated Docker library, ensuring compatibility with the latest Docker releases and addressing potential compatibility issues with older libraries. This enhancement contributes to the platform’s overall stability and reliability.

User LimitsAdministrators can now set limits on the total number of running applications, both globally and on a per-application basis. This feature empowers organizations to better manage resources and prevent potential overloads, particularly in public-facing or resource-constrained environments.

Configuration:

Global limit: Define the max-running-containers in the proxy section of your configuration file.
Per-application limit: Use the max-instances property within the specific app’s configuration.

AWS ECS BackendThe introduction of AWS ECS (Elastic Container Service) support expands ShinyProxy’s deployment options, allowing organizations to leverage the power of serverless deployments using AWS Fargate. This backend provides enhanced scalability, eliminates server management overhead, and integrates seamlessly with other AWS services like CloudWatch and IAM. A ready-to-use Terraform deployment example is available to simplify the setup process.
A downside of AWS ECS is that in most cases it takes more time to start the container compared to the other backends. But of course it can be used together with the pre-initialized container feature, reducing the startup time to less than a second.

Enhanced Monitoring (Not explicitly mentioned but valuable)ShinyProxy 3.1.0 introduced enhanced monitoring capabilities, particularly for pre-initialized and shared containers. New dashboards provide comprehensive insights into the performance and resource usage of these containers, enabling administrators to effectively monitor and manage their deployments.

ConclusionShinyProxy 3.1.1 sets a new standard for deploying and managing Shiny applications. The groundbreaking features of container sharing and pre-initialization, coupled with expanded deployment options through AWS ECS support, mark a promising advancement in scalability, performance, and resource efficiency.

These enhancements, along with usability improvements and enhanced monitoring capabilities, empower organizations to unlock the full potential of Shiny applications and cater to increasingly demanding deployment scenarios. You can check out full release notes in this documentation.

Want to get more insights on the latest in R/Shiny and data science as a whole? Sign up today for our weekly newsletter!

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: ShinyProxy 3.1.1: What’s New And How It Improves Scalability And Performance Of Your Shiny Applications

View Details

[This article was first published on The Manipulative Gerbil: Playing around with Energy Data, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In the US, the Federal Government’s National Renewable Energy Lab released the 2024 “Annual Technology Baseline” a couple of weeks ago. The large, yearly dataset does a few things. For one, it collects data on constructed electricity generators by type of fuel/renewable and collates the components of their cost. By factoring in all of the expected outlays for a specific type of generator, we can arrive at the “Levelised Cost of Energy” (LCOE); the exact calculation can differ but essentially, the units of the LCOE are “units of currency for a unit of energy,” eg, $/MWh. This allows people to Interestingly for people interested in intermittent renewables like solar and wind, NREL data also divides categorises regions by the strength of their wind or solar resources, on a 1 to 10 scale (1 being best and 10 being least good). I wanted to be able to think of a state at random within the US and ask “In which counties would I have the best (or worst?) opportunities to build a solar farm and/or a wind farm?”. To do this, I would need to be able to tie climatology data (from our trusty friend nasapower) and categorise it based on the system devised by NREL. Using the R “simple features” object type, it’s possible to transform this into easily readable maps, too.

| | | Where in Arizona do wind turbines cost the least (the most) to operate? |

The way in which I’ve written the scripts gives users the choice between Canada, the United States and Mexico–but this is maybe a flurry on my part, since NREL bases its ATB data on findings from projects in the contiguous 48 states. In the meantime, it’s possible that these scripts could be built on to cover the world more globally (see below). There are a number of ways this project can be made better, including, to wit: 1. Expanding the data sources to cover a greater number of countries 2. Factoring in the grid connection costs, a major obstacle for renewables projects but which do not form a part of the LCOE 3. Allowing a different calculation of the LCOE I made the scripts available on my GitHub, here. To run this locally on your machine, To leave a comment for the author, please follow the link and comment on their blog: The Manipulative Gerbil: Playing around with Energy Data.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Visualising NREL’s Annual Technology Baseline ("ATB") data for wind and solar energy using R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionToday, we’re going to walk through an example of fitting a linear model in R, summarizing the results, and exporting the findings to an Excel file. This workflow is useful for documenting and sharing your statistical analysis.

Let’s break down the code step by step.

ExampleStep 1: Loading the Necessary LibrariesFirst, we need to load the openxlsx library, which helps us create and manipulate Excel files. If you don’t have it installed, you can get it using install.packages("openxlsx").

library(openxlsx) This line of code loads the openxlsx library into R so we can use its functions later.

Step 2: Fitting the Linear ModelNext, we fit a linear model using the built-in mtcars dataset. We model mpg (miles per gallon) based on all other available variables in the dataset.

model <- lm(mpg ~ ., data = mtcars) Here, lm stands for linear model. The mpg ~ . part means we want to predict mpg using all other variables in the mtcars dataset.

Step 3: Summarizing the ModelWe obtain a summary of our linear model, which includes details like coefficients, R-squared values, and the F-statistic.

model\_summary <- summary(model) This code generates a summary of the linear model we just created, giving us important statistics about the model’s performance.

Step 4: Extracting Key ComponentsWe extract essential parts of the summary for easy access and to organize them in our Excel file.

coefficients <- model\_summary$coefficientsr\_squared <- model\_summary$r.squaredadj\_r\_squared <- model\_summary$adj.r.squaredf\_statistic <- model\_summary$fstatisticp\_value <- pf( f\_statistic[1], f\_statistic[2], f\_statistic[3], lower.tail = FALSE )model\_formula <- paste0( model\_summary[["terms"]][[2]], " ", model\_summary[["terms"]][[1]], " ", model\_summary[["terms"]])[[3]] * coefficients: The estimated coefficients of the model. * r_squared: How well the model explains the variability of the data. * adj_r_squared: Adjusted version of R-squared for the number of predictors. * f_statistic: Overall significance of the model. * p_value: Probability value indicating the significance of the F-statistic. * model_formula: The formula used to fit the model.

Step 5: Creating and Populating the WorkbookNow, we create a new Excel workbook and add a worksheet to it. We then write our extracted model summary components to this worksheet.

wb <- createWorkbook()addWorksheet(wb, "Model Summary")writeData(wb, "Model Summary", "Coefficients", startRow = 1, startCol = 1)writeData(wb, "Model Summary", coefficients, startRow = 2, startCol = 1, rowNames = TRUE)writeData(wb, "Model Summary", "R-Squared", startRow = 2 + nrow(coefficients) + 2, startCol = 1)writeData(wb, "Model Summary", r\_squared, startRow = 2 + nrow(coefficients) + 2, startCol = 2)writeData(wb, "Model Summary", "Adjusted R-Squared", startRow = 2 + nrow(coefficients) + 3, startCol = 1)writeData(wb, "Model Summary", adj\_r\_squared, startRow = 2 + nrow(coefficients) + 3, startCol = 2)writeData(wb, "Model Summary", "F-Statistic", startRow = 2 + nrow(coefficients) + 4, startCol = 1)writeData(wb, "Model Summary", f\_statistic[1], startRow = 2 + nrow(coefficients) + 4, startCol = 2)writeData(wb, "Model Summary", "p-Value", startRow = 2 + nrow(coefficients) + 5, startCol = 1)writeData(wb, "Model Summary", p\_value, startRow = 2 + nrow(coefficients) + 5, startCol = 2)writeData(wb, "Model Summary", "Model Formula", startRow = 2 + nrow(coefficients) + 6, startCol = 1)writeData(wb, "Model Summary", model\_formula, startRow = 2 + nrow(coefficients) + 6, startCol = 2) * createWorkbook(): Creates a new Excel workbook. * addWorksheet(wb, "Model Summary"): Adds a new sheet named “Model Summary” to the workbook. * writeData: Writes data to the specified location in the sheet. Here, we write various parts of the model summary in different rows and columns.

Step 6: Saving the WorkbookFinally, we save our workbook to a file named lm_model_summary.xlsx.

saveWorkbook( wb, file = paste0(getwd(),"/lm\_model\_summary.xlsx"), overwrite = TRUE ) This line saves the workbook to your working directory with the specified file name.

Here is a screenshot:

Excel ScreenshotConclusionThis example shows how to fit a linear model in R, extract meaningful summary statistics, and save those results in an Excel file. It’s a simple yet powerful way to document your analyses and share them with others.

Feel free to modify the code to fit your own datasets and models. Experimenting with different variables and models can provide deeper insights into your data. Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Linear Models with R and Exporting to Excel

View Details

[This article was first published on Albert Rapp, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Line charts are one of the most fundamental chart type out there. That’s why there’s a lot of tips for line charts out there. Today, I’m going to walk you through a couple of techniques that you can use to make your line chart nicer. Here, I’ll provide you with all the code chunks. All explanations can be found in my corresponding YT video:

Get the data library(tidyverse)sp500\_data\_wide <- gt::sp500 |> select(date, open, close) |> filter(year(date) == 2014, month(date) == 1)sp500\_data\_wide## # A tibble: 21 × 3## date open close## <date> <dbl> <dbl>## 1 2014-01-31 1791. 1783.## 2 2014-01-30 1777. 1794.## 3 2014-01-29 1790. 1774.## 4 2014-01-28 1783 1792.## 5 2014-01-27 1791. 1782.## 6 2014-01-24 1827. 1790.## 7 2014-01-23 1842. 1828.## 8 2014-01-22 1845. 1845.## 9 2014-01-21 1841. 1844.## 10 2014-01-17 1844. 1839.## # ℹ 11 more rows Bring into a nice format for ggplot sp500\_data <- sp500\_data\_wide |> pivot\_longer( cols = -date, names\_to = 'type', values\_to = 'price' ) Create a basic line chart sp500\_data |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) Apply a theme sp500\_data |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank() ) Use nicer colors sp500\_data |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank() ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) Use meaningful labels sp500\_data |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank() ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) Make labels into currency sp500\_data |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank() ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) + scale\_y\_continuous( labels = scales::label\_dollar() ) Use direct labels instead of legend sp500\_data |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + geom\_text( data = sp500\_data |> slice\_head(n = 1, by = type), aes(label = type), hjust = 0, vjust = 0, family = 'Source Sans Pro', size = 10, nudge\_x = 0.1 ) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank(), legend.position = 'none' ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) + scale\_y\_continuous( labels = scales::label\_dollar() ) + scale\_x\_date( limits = c( make\_date(2014, 1, 1), make\_date(2014, 2 ,3) ) ) Make nicer labels sp500\_data\_with\_nicer\_labels <- sp500\_data |> mutate( type = if\_else( type == 'open', 'Opening price', 'Closing price' ) ) sp500\_data\_with\_nicer\_labels|> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + geom\_text( data = sp500\_data\_with\_nicer\_labels |> slice\_head(n = 1, by = type), aes(label = type), hjust = 0, vjust = 0, family = 'Source Sans Pro', size = 10, nudge\_x = 0.1 ) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank(), legend.position = 'none' ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) + scale\_y\_continuous( labels = scales::label\_dollar() ) + scale\_x\_date( limits = c( make\_date(2014, 1, 1), make\_date(2014, 2, 8) ) ) Place labels closer to the lines sp500\_data\_with\_nicer\_labels |> ggplot(aes(date, price, col = type)) + geom\_line(linewidth = 1.25) + geomtextpath::geom\_textline( data = sp500\_data\_with\_nicer\_labels |> filter(type == 'Opening price'), aes(label = type), hjust = 0.76, vjust = 0, family = 'Source Sans Pro', size = 8 ) + geomtextpath::geom\_textline( data = sp500\_data\_with\_nicer\_labels |> filter(type == 'Closing price'), aes(label = type), hjust = 0.77, vjust = 1, family = 'Source Sans Pro', size = 8, text\_smoothing = 40, offset = unit(-14, 'mm') ) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank(), legend.position = 'none' ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) + scale\_y\_continuous( labels = scales::label\_dollar() ) Highlight area between lines. sp500\_data\_with\_nicer\_labels |> ggplot(aes(date, price, col = type)) + ggbraid::geom\_braid( data = sp500\_data\_wide, aes( y = NULL, ## Overwrite the inherited aes from ggplot() col = NULL, ymin = open, ymax = close, fill = open < close ), alpha = 0.6 ) + geom\_line(linewidth = 1.25) + geomtextpath::geom\_textline( data = sp500\_data\_with\_nicer\_labels |> filter(type == 'Opening price'), aes(label = type), hjust = 0.76, vjust = 0, family = 'Source Sans Pro', size = 8 ) + geomtextpath::geom\_textline( data = sp500\_data\_with\_nicer\_labels |> filter(type == 'Closing price'), aes(label = type), hjust = 0.77, vjust = 1, family = 'Source Sans Pro', size = 8, text\_smoothing = 40, offset = unit(-14, 'mm') ) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank(), legend.position = 'none' ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + scale\_fill\_manual( values = c('TRUE' = '#0072B2', 'FALSE' = '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) + scale\_y\_continuous( labels = scales::label\_dollar() ) ## `geom\_braid()` using method = 'line' Add a callout label box sp500\_data\_with\_nicer\_labels |> ggplot(aes(date, price, col = type)) + ggbraid::geom\_braid( data = sp500\_data\_wide, aes( y = NULL, ## Overwrite the inherited aes from ggplot() col = NULL, ymin = open, ymax = close, fill = open < close ), alpha = 0.6 ) + geom\_line(linewidth = 1.25) + geomtextpath::geom\_textline( data = sp500\_data\_with\_nicer\_labels |> filter(type == 'Opening price'), aes(label = type), hjust = 0.76, vjust = 0, family = 'Source Sans Pro', size = 8 ) + geomtextpath::geom\_textline( data = sp500\_data\_with\_nicer\_labels |> filter(type == 'Closing price'), aes(label = type), hjust = 0.77, vjust = 1, family = 'Source Sans Pro', size = 8, text\_smoothing = 40, offset = unit(-14, 'mm') ) + ggforce::geom\_mark\_circle( data = tibble( date = make\_date(2014, 1, 24), price = 1820 ), aes( col = NULL, label = 'This area signals whether the\nclosing price or opening price was\nhigher on a given day' ), fill = 'white', color = 'grey20', alpha = 1, x0 = make\_date(2014, 1, 13), y0 = 1805, label.family = 'Source Sans Pro', label.colour = 'grey20', label.hjust = 0, label.fontsize = 12, label.fontface = 'plain', con.colour = 'grey20', con.cap = unit(1, 'mm'), expand = 0.011 ) + theme\_minimal( base\_size = 20, base\_family = 'Source Sans Pro' ) + theme( panel.grid.minor = element\_blank(), legend.position = 'none' ) + scale\_color\_manual( values = c('#0072B2', '#D55E00') ) + scale\_fill\_manual( values = c('TRUE' = '#0072B2', 'FALSE' = '#D55E00') ) + labs( x = element\_blank(), y = element\_blank(), title = 'SP500 Prices in January 2014' ) + scale\_y\_continuous( labels = scales::label\_dollar() ) ## `geom\_braid()` using method = 'line' To leave a comment for the author, please follow the link and comment on their blog: Albert Rapp.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Quick dataViz techniques for nicer line charts with ggplot

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionToday, let’s dive into a nifty R package called taskscheduleR that can automate running your R scripts. Whether you need to execute a task every hour or just once a day, taskscheduleR has you covered. This package leverages the Windows Task Scheduler, making it a breeze to schedule and automate repetitive tasks directly from R. Let’s walk through a couple of examples from my new book, “Extending Excel with Python and R”.

ExamplesExample 1: Hourly Script ExecutionFirst, let’s set up a task that runs a script every hour. Here’s the code:

library(taskscheduleR)# Create a task scheduler job that runs the script every hourtaskscheduler\_create( taskname = "Hello World Hourly", rscript = "hello\_world.R", schedule = "0 * * * *") In this snippet, we use the taskscheduler_create() function to create a new task. Let’s break down the arguments:

taskname: A unique name for the task, in this case, “Hello World Hourly”.
rscript: The path to the R script you want to run, here it’s “hello_world.R”.
schedule: This is the cron expression for scheduling. 0 * * * * means the script will run at the start of every hour.

Example 2: Daily Script Execution at a Specific TimeNow, let’s set up a task that runs the script once a day at 10:00 AM. Here’s how you can do it:

```

Create a task scheduler job that runs the script once a day at 10:00 AMtaskscheduler_create( taskname = "Hello World Daily", rscript = "hello_world.R", schedule = "0 10 * ")

`` In this example, thescheduleargument0 10 * ` ensures the script runs daily at 10:00 AM.

Why Automate?Scheduling routine tasks can save you a lot of time and reduce the chances of forgetting to run an important script. Whether it’s updating a report, fetching data, or performing backups, automation ensures that these tasks run consistently and accurately without manual intervention.

Give It a Try!I encourage you to experiment with taskscheduleR and see how it can fit into your workflow. Whether you’re new to automation or looking for ways to optimize your existing processes, this package offers a simple yet powerful solution.

If you found this helpful and want to learn more about extending your Excel capabilities with R and Python, check out my new book, “Extending Excel with Python and R”. It’s packed with practical examples and detailed explanations to help you get the most out of these powerful tools.

Amazon: Extending Excel with Python and R
Packt: Extending Excel with Python and R

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Automate Your R Scripts with taskscheduleR

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Nowadays, most data professionals choose either R or Python when it comes to a programming language of choice. But what if you need both? Do you have to constantly switch between environments and IDEs? That’s one option, sure, but leaves a lot to be desired.

To solve this problem, Posit came up with Positron IDE – a development environment for data scientists, supporting R and Python out of the box. The IDE is currently in public beta, meaning anyone can install it.

Today you’ll learn what this new IDE is all about and how it compares to existing solutions.

Want to work with Appsilon? Here’s our exact process for delivering impactful products.

Table of contents:* Hello Positron IDE – Key Features You Must Know * Running Code in Positron IDE – Tips and Tricks * Frequently Asked Questions about Positron IDE * Current Limitations and Future Potential * Summing up Positron IDE

Hello Positron IDE – Key Features You Must KnowPositron is a next-generation data science IDE delivered by Posit. It’s still in active development, so it’s expected to see some features not working properly (more on this later). But, as mentioned in the introduction, it’s in public beta, which means you’re free to take it for a spin!

You can download the latest Positron release from the official GitHub releases page.

In essence, Positron is a fork of a famous IDE – Visual Studio Code. If you’re familiar with it, Positron should feel right at home. It has some neat features delivered out of the box, but you could configure most of these through plugins on a fresh VSCode installation.

Up next, let’s discuss some key features you must know about.

RStudio Meets Visual Studio CodeHere’s what you’ll see when you first launch Positron:

‍

Image 1 – Positron IDE welcome screenIt certainly looks like a combination of RStudio and Visual Studio Code! You’ve got your familiar sidebar for navigation and extensions, but also your four-panel view for code, console, plots, and variables.

The top left panel allows you to start working on your data science projects – either in R or Python, through a notebook or file. Positron automatically detects installed programming languages and their version, but also picks up any virtual environments you’ve previously created:

‍

Image 2 – File/project creation in Positron IDEUp next, let’s explore this multi-language and multi-format support in more detail.

Multi-Language SupportThe big selling point of Positron IDE is that it comes configured for R and Python out of the box – Jupyter Notebooks included. This means you don’t have to set everything up from scratch, which in the case of R and Jupyter is not as easy as it sounds.

To create a new R script, click on the New File button on the welcome screen and select R File. Writing and running code works just like in RStudio – Command/Control + Enter will run the cell on which your cursor is located:

Image 3 – Working with R files in PositronThe same approach to writing and running code works in Python scripts – write any code block you want and hit Command/Control + Enter to run it:

‍

Image 4 – Working with Python files in PositronStill, we think Jupyter notebooks allow maximum flexibility. You can create a notebook with a default programming language profile (R or Python), but you can then change the language for each cell.

Because of this flexibility, you can also sprinkle text/markdown content between your cells to provide resources or explanations:

‍

Image 5 – Working with Jupyter Notebooks in PositronAnd that’s the basics of programming language and format support in Positron. Up next, let’s discuss some more advanced features.

Ark KernelArk is an R kernel for Jupyter applications and is a default interface between R and the Positron IDE. Previously, you’d have to install IRkernel if you wanted to run R in Jupyter Notebooks. You also had the option to install Ark, but it now ships by default with Positron.

What Ark does for you is provide structured interaction between R language and Jupyter, enable IntelliSense features through the LSP server, and sophisticated step-debugging of R functions through a DAP server. Now, Ark doesn’t support breakpoints so far, but you can use debug(), debugonce(), or browser() as substitutes.

Let’s talk more about the IntelliSense part. For example, you can hit Tab to display autocomplete suggestions:

‍

Image 6 – Autocomplete suggestionsThis also works if you want to, for example, list all functions available in a given package:

‍

Image 7 – Autocomplete suggestions (2)Sure, it’s nothing groundbreaking, and is something you’ll take for granted if coming from RStudio, but Jupyter and R don’t always work nicely. Ark kernel is a way to fix that, and thanks to Positron, you don’t have to install and configure it manually.

But by far the most important quality of life improvement for developers is box imports. Let’s talk about them next.

Box ImportsBox imports are somewhat of a mess in VSCode and RStudio. In VSCode, you can’t Control/Command + Click to get to the source function definition, and in RStudio, it only works when calling the function (not when importing it). Positron has thankfully solved this problem.

Here’s an example – we have a my\_square\_function() exported in r\_code/functions.R:

‍

Image 8 – Box imports exampleAs you can see, you can now get to the function definition both from the import and from the function calls!

And that’s the basics of Positron for you! Up next, we’ll go over some tips and tricks for running code in this new IDE.

Running Code In Positron IDE – Tips And TricksPositron, being a mixture of RStudio and Visual Studio Code, combines the best features of both worlds. This section will take you through useful keyboard shortcuts and features for working with data.

Useful ShortcutsWhile you don’t necessarily need to use keyboard shortcuts, they significantly improve developer productivity. The following image illustrates what it takes to run a single cell without keyboard shortcuts:

‍

Image 9 – Running Jupyter cell without shortcutsManually clicking on the play button will become pretty annoying after a couple of iterations. Instead, you can opt for one of the following:

Control + Enter: Run the current cell without creating a new one
Shift + Enter: Run the current cell and create a new cell for the same programming language

While in a notebook environment, you might also find the following shortcuts useful:

C: Copy cell
X: Cut cell
V: Paste cell
D: Delete cell
Command/Control + P: Navigate between files

Note that they are different from the usual jupyter notebook keyboard shortcuts, but you can see the full list of available shortcuts in Settings – Keyboard Shortcuts. Every shortcut is adjustable.

DataFrame ViewerDataframes are the core of all data science workflows, so having an IDE that can display all relevant information about them is a must-have feature.

Positron allows you to print the dataframe content to the R console by calling R-specific functions, such as head():

‍

Image 10 – Printing the top 6 rows of a dataframeBut the more interesting feature is the dataframe viewer. Once your dataframe is declared, you’ll see it in the Variables panel. You can expand the variable to view all columns and their respective values, or you can click on the table column to inspect the dataframe in an Excel-like fashion:

‍

Image 11 – Dataframe inspectionAs you can see, you can sort the values, apply filters, inspect missing values, and much more – straight from the GUI.

Plot ViewerAn amazing feature of RStudio is the plot viewer. You have a dedicated panel for visualizations, and you can easily cycle through multiple charts. Positron has the same feature, arguably with a somewhat updated interface:

‍

Image 12 – Plot inspectionCreating a new chart won’t delete the old one, as you can easily navigate between them using the right-side panel:

‍

Image 13 – Plot inspection (2)Up next, let’s discuss the powerful and improved variable inspector.

Variable InspectorBeing able to inspect complex objects, such as plots, is an essential feature for debugging code and making sure everything works as expected. RStudio also has this feature, but Positron allows you to dig deeper and has a sleeker-looking user interface:

‍

Image 14 – Variable inspectionAs you can see, you can drill down into all the small pieces that are combined to make complex objects.

Frequently Asked Questions About Positron IDEPositron IDE is currently in open beta, so this section will answer the most commonly asked questions posted by the community.

Positron is a fork of VSCode, so why not just use VSCode?R support in VSCode exists, but it tends to break every couple of patches due to interactions with other extensions. Positron solves this problem by supporting R out of the box and having a brand new Jupyter kernel, Ark, with blazing-fast IntelliSense. Additionally, plain VSCode doesn’t have a dedicated R console.

Is Positron a replacement for RStudio?No, Posit is going to support both Positron and RStudio moving forward.

GitHub Copilot and Remote SSH plugins aren’t working – What can I do?Positron IDE is currently in open beta, so you can expect occasional hiccups. Since it’s a VSCode fork, you’d expect that every VSCode plugin would work. That’s currently not the case for GitHub Copilot due to Microsoft licensing, A couple of other must-have plugins, such as Remote SSH and DevContainers also don’t work. The remote SSH plugin is in Posit’s roadmap for Positron, so they should have it sorted out soon.

Are all RStudio features supported in Positron?Most RStudio features are available in Positron, but the support is currently lacking for inline Quarto output, R Markdown profiling, Sweave, RStudio Add-in support, and even for easy publishing Shiny/Quarto apps/reports to Posit Connect or Shinyapps.io.

Current Limitations And Future PotentialWith everything said today, let’s now make a quick TL;DR of everything you might find annoying about Positron IDE public beta release coming from RStudio or Visual Studio Code:

Feature support – Unlike RStudio, Positron doesn’t support inline Quarto output, R Markdown profiling, Sweave, RStudio Add-in support, and support for easy app and report publishing.
Plugin support – Positron is a fork of VSCode, but major VSCode plugins such as Copilot, Remote SSH, and DevContainers are not working.
No wow factor – Being based on VSCode, Positron doesn’t feel like anything new, so many users that already have a properly configured environment don’t have the urge to switch (or even give it a try).

Still, Positron is currently in public beta, so it’s not 100% fair to compare it to a product that’s been in a stable release for years. Many things can, and hopefully will, change before it’s out for good.

Summing Up Positron IDETo summarize, Positron is a new IDE from Posit, the authors of RStudio. It’s not designed to be a replacement for RStudio, as both will be maintained moving forward. It’s a new product, aiming to be a one-stop-shop for data science workflows (R and Python). In the current public beta release, it’s still a bit rough around the edges, as the plugins we rely on daily simply don’t work.

If you’d like to learn more about Positron, make sure to visit the official GitHub repository. If you want to take it out for a spin, download the latest version from their release page. We encourage you to do so, and also to share your feedback with us (comment section below), and with Posit.

Have you already tried the Positron IDE? Do you think it’s a viable RStudio/VSCode replacement for data science workflows? We’re eager to hear your opinion in our Slack community.

Looking to containerize your R Shiny application? This article reveals if you should go with Docker, Podman, or Singularity.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing Positron: A New, Yet Familiar IDE For R And Python

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Complexity is often an unavoidable reality in design, particularly in fields like product design and user experience (UX). Whether it’s intricate user interactions, multi-layered functionalities, or sophisticated user interfaces, navigating complexity is a fundamental challenge for designers.

Make your interfaces more effective by learning how to speak your user’s language. Find out how in Yuri’s blog post.

High-fidelity prototyping emerges as a powerful tool in this journey, offering a robust framework to tackle complexity head-on. In this article, we’ll delve into the key considerations when designing for complexity through high-fidelity prototyping.

Key 1: Understanding ComplexityIn projects like this, complexity goes hand in hand with the functionality that needs to be addressed. This is especially true regarding a unified system that connects different markets with their peculiarities and needs.

Key components of the system were meant to be Risk Identification, Demand Simulation, and Sales Visualization. At the same time, the system had to be uniform and universal.

Understanding that each market had its peculiarities in production and supply chains added to the complexities. This, in turn, is reflected in the data structure. All of this was overlaid onto a role model that included a Planner, Market Leader, and Regional Leader.

To avoid underestimation, we agreed to conduct a series of preliminary meetings with market representatives to update requirements and collect differences. The result was a consolidated document that we referred to throughout the project.

Summary of AchievementsKey Point: Underestimating project complexity carries risks of budget increase or termination of cooperation.

Key 2: Deep Delve into Business Processes That Are UniqueEvery business has its unique features and processes specific to it. Internal procedures and ways of working are quite complex to understand for an outsider. To understand the process from the user’s perspective, it was important to first understand what the numbers meant, how to read tables, and where to look for patterns.

We conducted a series of in-depth meetings dedicated to business nuances. We went through user flows and process details. At that point, we already had a complete picture of how the process worked and were able to identify pain points and areas for improvement as we progressed.

Key Project InsightsKey Point: It is critically important to know the process as well as the manager responsible for it. This way, you can be on the same page as the client and offer truly effective solutions.

‍

Discover the key steps to creating dashboards that users love. Learn 7 essential UX design techniques for Shiny apps and elevate your design game!

Key 3: Think Outside the BoxAt this point, we determined that risk visualization was a key module of the system and an entry point. So, we focused on finding the most suitable solution. We needed to demonstrate the significance of risk considering the cost-to-volume ratio and the coefficient of additional factors.

After several iterations, we came up with the idea of visualization through bubbles where the size indicated significance. Taking into account user roles, we added two levels. For implementation, we chose the Reactflow library, which fully suited the task. We also considered the user’s familiar pattern and added a tabular view for risks.

Bubble Chart‍

Key Point: Explore the market of innovative tools and propose solutions that go beyond the standards, but do not forget about familiar patterns for users.

‍

Key 4: User’s Always RightIn cases where the prototype needs to cover all markets and all user models, it is important to test hypotheses at early stages. If you do not do this, the risk of not meeting expectations is quite high. Organizing different meetings in different time zones is a difficult task for a manager. However, we did this and got several important lessons.

Despite the warning that we were using unrealistic data in the prototype, the testers still tried to interpret the prototype by reading the data. This led to an incorrect evaluation of the prototype. Nevertheless, we were able to gather the necessary data for improvement. We took into account the previous experience, and the following sessions were more meaningful.

Cultural differences among respondents should also be considered. We encountered a problem of interpreting the same information from different perspectives. The compromise was an extended risk table with a heat map where users could visually identify the degree of risks and their impact on the supply chain.

User EngagementKey Point: Conduct feedback sessions as often as your schedule allows. Take into account the cultural differences of the respondents.

Interested in designing accessible research? Check out this blog post to learn best practices with practical case studies.

Key 5: Scalability and FlexibilityComplex automation systems are inherent in large companies. In our case, it was a top-10 company in the Fortune 500. We worked on the PoC and understood that the system would need to expand in case of success, so we took this into account early and proposed using the popular AntD component library adapted to the company’s brand book.

We also discussed further system development with clients and collected a backlog of potential features. Considering this, independent components were added to the system in the left panel without complicating the current concept.

Key point: Considering the project’s scalability potential and foreseeing future requirements is a strategic method for preventing potential issues in the future.

Summing Up Designing for ComplexityIn conclusion, high-fidelity prototyping is a tool for visualizing and testing concepts and a strategic approach to solving complex problems.

Understanding the project’s unique features, deep analysis of business processes, creative approach to choosing tools and solutions, constant feedback from users, and consideration of potential scalability and flexibility form the basis of successful prototyping.

Adhering to these principles allows designers to effectively address challenges and create products and interfaces that truly meet users’ and business tasks’ needs.

Did you find this blog post useful? Learn more about how you can make your Shiny app beautiful in this blog post.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Designing for Complexity: 5 Key Considerations in High-Fidelity Prototyping

View Details

[This article was first published on Epiverse-TRACE developer space, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. We are continuing our post series on S3 object orientation and interoperability in R. We have previously discussed what makes a good S3 class and how to choose a good parent for it, as well as when to write or not write a custom method. We have highlighted in particular how classes inheriting from data.frames can simplify user experience because of familiarity, and reduce developer workload due to the pre-existing S3 methods.

We have detailed how to improve compatibility with the tidyverse by explaining:

how functions taking data.frames or data.frames subclass should also allow compatibility with tibble, which can be done in a few steps
how to ensure class attributes are preserved whenever possible while using dplyr functions.

Here, we are going to explore how to start adding support in the ecosystem for the new S3 classes while minimizing user-facing breaking changes. We have previously delved into this topic with our post “Convert Your R Function to an S3 Generic: Benefits, Pitfalls & Design Considerations” and this is a wider and higher-level view of the same topic.

The strategy presented here is the variation of a common concept in web development and the web ecosystem: progressive enhancement. This philosophy aims to support browsers with a common set of essential features, and even richer features for browser with the most recent updates. It makes sense to think about this philosophy with the prism of introducing new classes to a new software ecosystem as it has the similar constraints of multiple stakeholders with different interests and timelines. The application of progressive enhancement in this context means that users or packages that have not (yet) adopted the new classes are not penalized compared to users or packages that have.

Adding class support to function inputs via progressive enhancementThe goal here is to allow functions to accept the new classes as inputs, while keeping the old behaviour unchanged for unclassed objects (or with a different class than the new one).

This can conveniently be done in an almost transparent way by converting the old function to the S3 generic, and using the default method to handle the old behaviour. The practical steps, and minor caveats, have been previously described in the post “Convert Your R Function to an S3 Generic: Benefits, Pitfalls & Design Considerations”.

A before / after type image showing the conversion of a function to a generic with a default method keeping the exisiting behaviour.For a different, additional, example, we can consider a function working on patient-level data, which previously only accepted a data.frame as input:

```

' Compute length of stay in hospital on a patient-level dataset#'#' @param data A data.frame containing patient-level data#' @param admission_column The name of the column containing the admission date#' @param discharge_column The name of the column containing the discharge date#'#' @returns A numeric vector of hospitalization durations in dayscompute_hospitalization_duration <- function(data, admission_column, discharge_column) { difftime( data[[discharge_column]], data[[admission_column]], units = "days" )}

`` We want to add support forlinelistobjects, as defined in the linelist package.linelistobjects inherit fromdata.frameand contain an additionaltagsattribute. In particular,linelistobjects can have adate_admissionanddate_discharge` tag. This means we can use the tags to automatically detect the columns to use.

But we want the function to keep working for standard data.frames, tibbles, etc. We can follow the steps described in the previous post to convert the function to a generic, and add a default method to handle the old behaviour:

compute\_hospitalization\_duration <- function(data, ...) { UseMethod("compute\_hospitalization\_duration")}compute\_hospitalization\_duration.default <- function(data, admission\_column, discharge\_column) { difftime( data[[discharge\_column]], data[[admission\_column]], units = "days" )}compute\_hospitalization\_duration.linelist <- function(data, ...) { x <- linelist::tags\_df(data) compute\_hospitalization\_duration( data = x, admission\_column = "date\_admission", discharge\_column = "date\_discharge" )} If the function was already a generic, then a new method for the new class should be added, leaving everything else unchanged.

Adding class support to function outputs via progressive enhancementAdding class support to function outputs is often more challenging. A common option is to add a new argument to the function, which would be a boolean indicating whether the output should be of the new class or not. But this doesn’t fit in the view of progressive enhancement, as it would require users to change their code to benefit from the new classes, or to suffer from breaking changes.

While the new argument approach is sometimes indeed the only possible method, there are some situations where we can have an approach truly following the progressive enhancement philosophy.

In particular, this is the case when the old output was already inheriting from the parent of the new class (hence the importance of carefully choosing the parent class). In this situation, the new attributes from the new class should not interfere with existing code for downstream analysis.

In this case, let’s consider a function that was previously returning an unclassed data.frame with patient-level data:

create\_patient\_dataset <- function(n\_patients = 10) { data <- data.frame( patient\_id = seq\_len(n\_patients), age = sample(18:99, n\_patients, replace = TRUE) ) return(data)} We want to start returning a linelist object. Because linelist objects are data.frames (or tibbles) with an extra attr, it can be done in a transparent way:

create\_patient\_dataset <- function(n\_patients = 10) { data <- data.frame( patient\_id = seq\_len(n\_patients), age = sample(18:99, n\_patients, replace = TRUE) ) data <- linelist::make\_linelist( data, id = "patient\_id", age = "age" ) return(data)}inherits(data, "data.frame") For a more realistic example, you can also see the work in progress to integrate the new contactmatrix standard format for social contact data to the contactdata package.

This is however only true if code in downstream analysis follows good practices in checking for the class of an object 1. If existing code was testing equality of the class to a certain value, it will break when the new class value is appended. This is described in a post on the R developer blog, when base R was adding a new array class value to matrix objects. Class inheritance should never be tested via class(x) == "some_class". Instead, inherits(x, "some_class") or is(x, "some_class") should be used to future-proof the code and allow appending an additional in the future.

ConclusionObject oriented programming and S3 classes offer a convenient way to iteratively add interoperability in the ecosystem in a way that is minimally disruptive to users and developers. Newly classed input support can be added via custom methods (after converting the existing function to a generic if necessary). Newly classed output support can be added via progressive enhancement, by ensuring that the new class is a subclass of the old one and that downstream code uses good practices to test class inheritance.

Thanks to James Azam and Tim Taylor for their very valuable feedback on this post.

Footnotes1. This is now enforced in R packages by R CMD check, and via the class_equals_linter() in the lintr package.︎

ReuseCC BY 4.0CitationBibTeX citation: @online{gruson2024, author = {Gruson, Hugo}, title = {Improving {Ecosystem} {Interoperability} {Iteratively} via {Progressive} {Enhancement}}, date = {2024-07-05}, url = {https://epiverse-trace.github.io/posts/progressive-enhancement/}, langid = {en}} For attribution, please cite this work as:Gruson, Hugo. 2024. “Improving Ecosystem InteroperabilityIteratively via Progressive Enhancement.” July 5, 2024. https://epiverse-trace.github.io/posts/progressive-enhancement/. To leave a comment for the author, please follow the link and comment on their blog: Epiverse-TRACE developer space.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Improving Ecosystem Interoperability Iteratively via Progressive Enhancement

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Shiny in Production is returning to the Catalyst, Newcastle upon Tyne,for its third instalment this October. We’ve expanded the itinerary thisyear, with four workshops to choose from as well as a day of talks, withspeakers soon to be announced. Full details of the workshop are below,and you can head over to the conferencewebsite to register.Join us for an immersive experience tailored for both beginners andadvanced users of Shiny and other web-based R packages.

The first day of the conference (Wednesday 9th October), will consist ofthe four parallel workshops, followed by a drinks reception in theevening, a great opportunity for networking and debriefing from theday’s learning.

Registernow

Level up your plots: Tips, tricks and resources for crafting compelling visualisations – Cara ThompsonData visualisations are a great asset in getting people talking aboutyour findings. From making the patterns in the data easy to see, tomaking a big visual statement and keeping people talking beyond the endof your presentation, transforming your plots from functional toaesthetically pleasing and visually compelling is about so much morethan making things pretty.

In this workshop, we’ll explore how we can make the most of colours,different plot types, text, and interactivity to maximise the impact ofour visualisations. Here’s where we’re looking to boost your datavizconfidence:

crafting intuitive dataviz-friendly colour palettes withoutcompromising on accessibility (or creativity!)
selecting the right type of dataviz for your data and your story
making the most of typography to optimise text hierarchy andreadability
using annotations wisely to both help interpretation and declutter thevisualisations
turning your ggplot into an interactive plot for additional dataexploration
packaging up your decisions, easy reuse across plots (and projects!)

This is intended as a hands-on workshop, so bring along a laptop, a plotyou’re working on or a research question, and some data. Throughout theworkshop, I will highlight free resources for each of these aspects ofdataviz development. The aim is for you to leave with a plot that you’dbe happy to publish, and with some resources you can continue to buildon.

About the speakerCara is a freelance data consultant with an academic background,specialising in dataviz and in “enhanced” reproducible outputs. Shelives in Edinburgh, Scotland, and is passionate about maximising theimpact of other people’s expertise.

Building Responsive Shiny Applications – Pedro SilvaThe diverse range of devices used for modern web browsing presentschallenges when designing an application that works well for all users.Enter responsive design: the practice of building fluid web pages that“work” on huge 4k and 5k monitors, tiny smartphones and all things inbetween. This course will look at responsive design principles and bestpractices for Shiny developers, covering page layout, easy-to-addwidgets and some simple CSS tricks for when built-in solutions don’tquite cut it.

By the end of the workshop, participants will…

know what responsive web design is
know how to use flexible grids to adjust page layout for mobile,tablet and desktop
be able to use HTML5 elements and Shiny Widgets to use limited spaceefficiently and effectively
know how to add CSS and JavaScript snippets to an app for finercustomisations
understand how to test Shiny apps on various screen sizes from desktopto mobile

About the speakerPedro is a full stack developer with over 15 years of experience in thefield, loves front-end and R Shiny development, and is a moonlightpractitioner of JavaScript dark arts.

Asynchronous Shiny – Russ HydeImagine you couldn’t register to attend “Shiny in Production” if someoneelse was in the process of registering, and you had to wait until theyhad finished before you could click to “Buy tickets on EventBrite”. Thiskind of “blocking” shouldn’t happen in modern web applications but issurprisingly common in Shiny applications. It happens because a single Rprocess handles all of the server-side processing for multiple users—onelong-running task can prevent any other task from proceeding, hamperinginteractivity both between and within user-sessions.

Fortunately, Shiny’s support for asynchronous programming can alleviatethis problem. In the asynchronous approach, you start tasks runningwithout having to wait for them to complete. But, this requires a changein mindset for many programmers and there are a few concepts tounderstand before you can take advantage of this approach. So, what areyou waiting for? Sign up for this workshop!

By the end of the workshop, participants will…

understand how within-session and between-session blocking can arisein a Shiny app
understand the basics of asynchronous computation
solve between-session blocking with future/promise
solve blocking the modern way, with ExtendedTask

About the speakerRuss has previously worked in molecular biology and bioinformatics. Heholds a PhD in Molecular Physiology and MSc in Mathematics. Russ is anauthor of several CRAN packages and mentor on the R-for-data-sciencecommunity.

Building Apps for Humans – Clarissa BarrattFrameworks like Shiny and Dash can help those with a scientific ormathematical background communicate their research in a way that’sinteractive and engaging. But while these tools can make constructing agraphical user interface quicker and easier, there’s no guarantee thatthe end product is going to be optimised for human use.

This workshop is aimed at scientists (and the curious) that areinterested in learning some basics of human-computer interaction andgaining an understanding for how science itself can assist with thedevelopment of better user interfaces that, in turn, lead to improveduser experiences.

By the end of the workshop, participants will…

understand the benefits that come from designing applications with thehuman mind in mind
know how the layout, colour, size and motion of interface andgraphical components can be used to enhance (or detract from) a user’sexperience
understand the importance of providing users with feedback so they cantell both whether their actions have been successful and what thecurrent state of the application is
be able to identify some common problems found in web applications

About the speakerWhile working towards her PhD in applied mathematics Clarissa discoveredher love of science communications. Her goal is to make data scienceaccessible to everyone, and to encourage people to engage with thegoings on at Jumping Rivers.

What’s next?Early bird tickets for the conference are still available till the endof July, so don’t miss out! The full line up of speakers will beannounced in the coming weeks. Still not convinced? Head over to ourYouTube channel to take alook at lineups from previous years to see what we have in store.

Registernow

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Shiny in Production 2024: Workshops

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Have you ever sat through a meeting, staring at a slide filled with jargon and complex charts, and found yourself utterly lost? Or perhaps you’ve read a report that was so vague you had no idea what the key takeaways were. Miscommunication in business can lead to costly mistakes, poor decision-making, and missed opportunities. This is why clarity in business communication is so crucial.

The IBCS SUCCESS framework is designed to tackle this very issue. It stands for International Business Communication Standards and aims to improve the clarity, consistency, and efficiency of business communication. Each letter in SUCCESS represents a principle to guide your communication efforts: Say, Unify, Condense, Check, Express, Simplify, and Structure. In this series of articles, we’ll dive deep into each principle, starting with the most fundamental one: Say.

“Saying what you mean” sounds straightforward, but in practice, it requires careful consideration of your goals, your audience, and your message. It’s about making sure that your audience can easily understand what you’re trying to communicate without any room for ambiguity.

When I first started in my career, I often fell into the trap of using complex language and convoluted explanations, thinking it made me sound more professional. However, I quickly learned that my messages were getting lost, and my colleagues were often left confused. It wasn’t until I started focusing on clarity — saying what I meant in a simple, direct manner — that my communication truly improved. This principle not only enhanced my effectiveness but also built trust and credibility with my team.

Know Your ObjectiveUnderstanding your own goals is the first step in clear communication. When you know exactly what you want to achieve, you can shape your message to align with that objective. This clarity of purpose not only helps you stay focused but also ensures that your audience receives a clear and consistent message.

Consider a presentation about quarterly performance. Is your goal to inform your team about past performance, highlight areas needing improvement, or persuade them to adopt a new strategy? Each goal would shape the presentation differently. Clear goals act as a roadmap, guiding what information to include, how to structure your message, and what tone to use.

Knowing your audience is equally important. Tailoring your language and message to fit your audience ensures effective and engaging communication. For instance, presenting a financial report to accountants requires a different approach than presenting the same report to marketing professionals. Understanding your audience’s background helps you determine the depth and complexity of the information to present.

In my early career, I often found myself overwhelmed by the amount of information I wanted to include in my presentations. It wasn’t until I started setting clear objectives that I could streamline my content and deliver more effective presentations. By focusing on my goals, I was able to cut out extraneous information and keep my audience engaged and informed.

Introduce Your MessageSetting the stage with context and background information is crucial for helping your audience understand why your message matters. Think of this step as setting the scene in a story. Before diving into the main plot, you need to provide some background.

For instance, if you’re presenting on flight delays, start with an overview of the airline industry, highlighting recent trends and challenges. This sets the stage and helps your audience understand the relevance of the data you will present.

Clearly defining the problem at hand is the next step. A well-defined problem helps your audience understand the significance of your message and sets the direction for the rest of your communication. For example, “Despite a significant number of flights, our data shows that delays are a persistent issue, with an average delay of 45 minutes per flight.”

Raising the key question your message will address is also important. This question should be directly related to the problem you’ve just explained and should set the stage for the solutions or insights you will provide. For example, “Given the persistent delays, what steps can we take to improve our on-time performance?”

In my professional journey, I found that taking the time to define and explain problems clearly has always led to more productive discussions and better-informed decision-making. When you provide context, explain the problem, and raise a key question, you create a clear and engaging introduction that prepares your audience for the main points.

Deliver Your MessageDelivering your message involves clearly identifying what your message means for your audience and either explaining the situation or suggesting solutions. There are three types of messages you can deliver: detection, explanation, and suggestion.

A detection identifies and presents an observed fact or trend. For instance, “We have detected that carrier X has the highest average delays.” An explanation provides reasons or insights behind the detected facts or trends. For example, “The high delays for carrier X are primarily due to maintenance issues and scheduling conflicts.” A suggestion offers actionable recommendations or solutions based on the detected facts and explanations. For example, “We suggest implementing a more rigorous maintenance schedule and optimizing flight schedules to reduce delays for carrier X.”

One of the most effective strategies in clear communication is to lead with your main point. Stating your main point at the beginning helps capture your audience’s attention and provides a clear focus for the rest of your communication. For example, instead of saying, “After analyzing the flight data, we found several interesting patterns that could indicate operational issues,” say, “Our analysis shows that carrier X has the highest average delays, indicating significant operational issues that need to be addressed.”

When you state your message upfront, you reduce the risk of your audience getting lost in the details. It also allows you to use the rest of your communication to support and elaborate on your main point, making your message more coherent and impactful.

Support Your MessageSupporting your message with solid evidence builds trust with your audience. Using credible and relevant data and examples helps make your message more convincing. For instance, providing data that shows carrier X has an average delay of 45 minutes, which is higher than the industry average, supports your claim and adds credibility.

Language precision is critical in clear communication. Using precise words helps eliminate ambiguity and ensures that your audience understands exactly what you mean. For example, instead of saying, “There were several delays last month,” say, “There were 123 flight delays last month, with an average delay of 45 minutes per flight.”

Highlighting key points helps ensure that your audience remembers the most important aspects of your message. This can be done through visual aids, textual emphasis, or repetition. For example, using bold text or bullet points to highlight important information, or repeating key points throughout your communication, helps reinforce the message.

Citing your sources adds credibility to your message. Always name your sources when presenting data or quoting information. This transparency builds trust and allows your audience to verify the information if needed. For example, “According to the Bureau of Transportation Statistics, carrier X had an average delay of 45 minutes last year.”

Linking comments and additional information helps connect different parts of your communication and provides a deeper understanding. This can involve referencing previous points, connecting to external resources, or providing further reading.

In my experience, using well-sourced data has always made a significant difference in how my audience receives my message. It shows that you’ve done your homework and that your recommendations are based on solid information.

Summarize Your MessageOne of the most effective ways to ensure your message is understood and remembered is through repetition. By repeating your key points, you reinforce the main ideas and help your audience retain the information.

Repetition is a powerful tool in communication. Studies have shown that people are more likely to remember information that is repeated. When you repeat your key points, you create multiple opportunities for your audience to absorb and understand your message. For example, summarizing your findings at the beginning, revisiting them throughout your presentation, and concluding with a final summary ensures that your audience retains the important information.

Explaining the consequences of your message helps your audience understand the significance and what it means for them. Understanding the consequences of your message helps your audience grasp its importance and urgency. It also provides a clear rationale for any recommendations or actions you suggest. For instance, explaining how flight delays impact customer satisfaction, operational efficiency, and revenue helps your audience understand why the issue is important and what the potential outcomes are.

Summarizing your message effectively involves repeating key points and explaining the consequences. These strategies ensure that your audience retains the important information and understands its significance. Clear communication is a journey that starts with knowing your objective and audience, introducing your message effectively, delivering it with clarity, supporting it with solid evidence, and summarizing it for retention and understanding. By following these principles, you can master the art of saying what you mean in business communication.

Practical Applications and ExamplesApplying the principles of clear communication in real-world scenarios can significantly enhance the effectiveness of your business communications. In this chapter, we’ll explore some practical applications and examples that illustrate how to transform ambiguous statements into clear, actionable messages.

Before and After ExamplesTransforming ambiguous statements into clear, actionable messages is a powerful exercise in improving communication clarity. Let’s look at a few examples from various industries.

Before: “Our Q4 performance didn’t meet expectations, and several departments showed mixed results. We need to address various issues to improve our outcomes in the future.”

After: “In Q4, our revenue decreased by 10% compared to Q3, primarily due to lower sales in the marketing and sales departments. To improve future outcomes, we need to increase our marketing efforts and provide additional training for the sales team.”

In the revised version, the message is clear and actionable. The reader knows exactly what happened and what steps are needed to address the issues.

Before: “The project is behind schedule due to several unforeseen issues.”

After: “The project is currently two weeks behind schedule due to delays in receiving critical components from our supplier. To get back on track, we need to expedite these deliveries and allocate additional resources to the assembly team.”

Again, the revised message provides specific details and a clear plan of action.

Best Practices for Report WritingWriting clear and concise reports is essential for effective business communication. Here are some best practices:

Start with a Summary: Begin your report with an executive summary that outlines the key points and conclusions. This helps readers quickly grasp the main message.
Organize Logically: Structure your report in a logical order, with clear headings and subheadings. This makes it easier for readers to follow your argument.
Use Clear Language: Avoid jargon and complex language. Use short sentences and simple words to convey your message clearly.
Include Visuals: Use charts, graphs, and tables to illustrate key points. Visuals can help make complex information more accessible and engaging.
Proofread Carefully: Errors can undermine your credibility. Always proofread your report to catch any mistakes.

Effective Presentation TechniquesPresenting information clearly in meetings is just as important as writing clear reports. Here are some strategies for effective presentations:

Know Your Audience: Tailor your presentation to the knowledge level and interests of your audience. Use language and examples that will resonate with them.
Start with the Main Point: Begin your presentation with a clear statement of your main message. This helps capture your audience’s attention and sets the stage for the rest of your talk.
Use Visual Aids: Visual aids, such as slides or handouts, can help reinforce your message. Make sure they are clear and easy to read.
Engage Your Audience: Encourage questions and interaction. This helps keep your audience engaged and ensures they understand your message.
Practice: Rehearse your presentation to ensure you are comfortable with the material and can deliver it smoothly.

Tools and Techniques for Ensuring ClarityEnsuring clarity in your communication often requires the use of various tools and techniques. These can help you streamline your message, visualize data effectively, and maintain consistency. In this chapter, we’ll explore some of the most effective tools and techniques available, including the graphical capabilities of R and Python, and modern AI tools for proofreading and editing.

Editing and ProofreadingOne of the simplest yet most effective ways to ensure clarity is through thorough editing and proofreading. Here are some tips:

Take a Break: After writing your message, take a break before you start editing. This helps you see your work with fresh eyes.
Read Aloud: Reading your text aloud can help you catch awkward phrasing and unclear sentences.
Use Editing Tools: Tools like Grammarly or Hemingway can help identify complex sentences, passive voice, and grammatical errors.
Get Feedback: Have a colleague review your message. A second pair of eyes can catch mistakes you might have missed.

Feedback and IterationSeeking feedback and iterating on your communication is crucial for improvement. Here’s how to effectively incorporate feedback:

Ask Specific Questions: When seeking feedback, ask specific questions about clarity, tone, and structure.
Be Open to Criticism: Constructive criticism is valuable. Use it to refine your message.
Iterate: Don’t be afraid to revise your message multiple times. Each iteration should improve clarity and impact.

Software and ToolsSeveral software tools can help ensure clarity in your writing and reporting. Here are a few you might find useful:

Microsoft Word and Google Docs: Both offer robust editing features, including grammar and style suggestions.
Grammarly: This tool provides advanced grammar and style checks, helping you write more clearly.
Hemingway Editor: Hemingway highlights complex sentences and common errors, encouraging simpler and clearer writing.
AI Tools: Modern AI tools like ChatGPT and others can provide real-time proofreading, suggest edits, and even help you generate content. These tools are becoming increasingly sophisticated and can be a valuable resource for improving clarity and coherence in your writing.

Graphical Capabilities of R and PythonData visualization is a powerful way to enhance clarity in your communication. Both R and Python offer robust libraries for creating clear and impactful visualizations.

R: Libraries like ggplot2, plotly, and lattice allow you to create a wide range of visualizations, from simple bar charts to complex multi-dimensional plots. These tools help you present data in a visually appealing and easy-to-understand manner.
Python: Libraries like matplotlib, seaborn, and plotly provide extensive capabilities for data visualization. Python’s versatility makes it a favorite for creating both static and interactive visualizations.

Using these tools, you can transform raw data into meaningful visual narratives that enhance your audience’s understanding and retention.

Benefits of Clear CommunicationClear communication brings numerous benefits, both for individuals and organizations. In this chapter, we’ll explore some of these benefits and how they can positively impact your work.

Enhanced Understanding and Decision-MakingClear communication leads to better understanding. When your audience understands your message, they are more likely to make informed decisions. This can lead to more effective problem-solving and strategic planning.

For example, a clear report on project delays and their causes can help management take appropriate actions to get the project back on track. Similarly, clear communication during team meetings ensures everyone is on the same page, reducing misunderstandings and increasing efficiency.

Building Trust and CredibilityClarity builds trust. When you communicate clearly, your audience is more likely to trust your message. This is particularly important in business, where trust is a key component of successful relationships.

For instance, clear and transparent communication with clients about project progress and any issues that arise builds credibility and trust. Clients appreciate honesty and clarity, which can lead to stronger, long-term relationships.

Improving Efficiency and ProductivityClear communication saves time and reduces errors. When messages are clear, there is less need for follow-up questions and clarifications, allowing everyone to work more efficiently.

For example, clear instructions to a team can eliminate the need for repeated explanations and corrections, thereby increasing productivity. Additionally, clear documentation ensures that everyone has access to the same information, reducing the likelihood of mistakes.

Real-world examples of increased efficiency through clarity include streamlined workflows, quicker resolution of issues, and improved overall performance.

In this article, we’ve explored the principle of “Say” within the IBCS SUCCESS framework. We’ve covered how to know your objective, introduce your message, deliver it effectively, support it with evidence, and summarize it for clarity and impact. We’ve also looked at practical applications, tools, and the benefits of clear communication.

Clear communication is a journey that starts with understanding your goals and audience, crafting a clear message, and supporting it with solid evidence. By following these principles, you can enhance your communication skills, build trust and credibility, and improve efficiency and productivity in your organization.

Remember, saying what you mean is not just about using simple words; it’s about being deliberate and thoughtful in how you convey your message. Keep practicing, seek feedback, and continually refine your approach to master the art of clear communication.

References and Further ReadingTo continue improving your business communication skills and deepen your understanding of the principles discussed in this series, here are some recommended books, articles, and resources.

Books and Articles on Business Communication“Made to Stick: Why Some Ideas Survive and Others Die” by Chip Heath and Dan Heath

This book explores what makes ideas memorable and how you can apply these principles to your communication to make your messages stick with your audience.

“The Pyramid Principle: Logic in Writing and Thinking” by Barbara Minto

A guide on how to structure your communication logically and effectively, using the pyramid principle to present information clearly and persuasively.

“Influence: The Psychology of Persuasion” by Robert Cialdini

This book delves into the psychology of influence and persuasion, offering valuable insights into how you can craft messages that resonate and persuade.

“Business Communication: Building Critical Skills” by Kitty O. Locker and Stephen Kyo Kaczmarek

A comprehensive textbook covering all aspects of business communication, from writing and speaking to listening and teamwork.

Articles from Harvard Business Review (HBR)

HBR offers a wealth of articles on effective communication, leadership, and business strategy. Regularly reading these can provide you with new insights and techniques.

IBCS ResourcesIBCS (International Business Communication Standards) Website

The official IBCS website (www.ibcs.com) provides detailed guidelines, examples, and templates for applying the SUCCESS framework in your business communications.

“IBCS® Standards Version 1.2: Conceptual, Perceptual and Semantic Design of Comprehensible Business Reports, Presentations, and Dashboards”

This resource outlines the IBCS standards in detail, offering practical advice on how to implement these rules in your reports and presentations.

Workshops and Training

IBCS offers various workshops and training programs to help individuals and organizations improve their communication skills using the SUCCESS framework.

Case Studies and Examples

The IBCS website and associated publications provide numerous case studies and examples that illustrate how the standards can be applied in real-world scenarios.

Online Courses and TutorialsCoursera and edX

Both platforms offer courses on business communication, data visualization, and effective presentation skills. These courses often include practical exercises and peer reviews to help you practice and refine your skills.

LinkedIn Learning

LinkedIn Learning offers a variety of courses on business communication, from basic writing and presentation skills to advanced techniques for persuasive communication.

Data Visualization Courses

Courses focused on data visualization, such as those offered by DataCamp and Udacity, can help you master the graphical capabilities of R and Python, enhancing your ability to present data clearly and effectively.

Mastering the art of clear communication is an ongoing journey. By continually seeking new knowledge and practicing the principles we’ve discussed, you can significantly improve your ability to convey your ideas clearly and effectively. Whether through books, articles, training, or online courses, there are ample resources available to help you on this path.

Thank you for following along with the second episode of our series on the IBCS SUCCESS framework, focusing on the principle of “Say.” We hope it has provided you with valuable insights and practical strategies for improving your business communication.

Stay tuned for the next episode, where we will explore the principle of “Unify,” delving into how consistency in visual language can enhance the clarity and impact of your messages.

Say What You Mean: Mastering Clarity in Business Communication was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Say What You Mean: Mastering Clarity in Business Communication

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Isabella Velasquez, co-organizer of R-Ladies Seattle, recently spoke with the R Consortium about her journey with R and the group’s recent activities. Isabella started as a beginner but has become a key figure in the R community thanks to the supportive and collaborative learning environment. R-Ladies Seattle regularly hosts in-person, hybrid, and online events, such as casual happy hours, lightning talks, and collaborations with other user groups. The group engages its members through creative activities and uses tools like GitHub for event planning. Their commitment to inclusivity and continuous learning helps maintain a dynamic and supportive community for R users in Seattle.

R-Ladies Seattle is seeking speakers for an upcoming lightning talks session. If you are interested in presenting, please contact Isabella.

Personal website | Twitter | Mastodon

Please share your background and involvement with the RUGS group.

I first encountered R when I started my graduate program in 2014. I was pursuing a Master’s in Analytics in Chicago. The program mainly revolved around using R. At that time, I was a complete beginner and had to start from the basics, like installing R. Since the program was fairly new, there wasn’t a well-structured curriculum for introducing R. It was assumed that students would either pick it up or already have some knowledge. The coursework focused on R from there.

My older brother Gustavo was a great resource when I started learning R and picking up the necessary skills. He is a highly proficient R user, so I asked him for help. He introduced me to many tools that made it much easier for a beginner to work in RStudio and pick up the tidyverse syntax. The main course curriculum was open to different approaches to using R, which provided flexibility in learning the tools and skills that interested me the most.

After completing my program in 2016, I landed my first job as a data analyst. I began using R and regularly working with data. Back then, Twitter was buzzing with activity. I stayed enthusiastic and continued learning from the community. My brother and I collaborated to solve problems and acquire new skills. At work, my team had diverse tool proficiency; some were adept in Excel, while others had data expertise. Eventually, we formed a learning community and collectively mastered R. We utilized R to generate presentations, reports, visualizations, and clean up data. It was fantastic to have a small community at work and a larger one outside through social media.

One of my colleagues, Chaya Jones, at my previous workplace, where I worked as a data analyst, was one of the original co-organizers for R-Ladies Seattle. The R user group had just started in 2018, and she invited me to join and give one of the earliest presentations for R-Ladies Seattle. Over time, the membership grew, and eventually, I became one of the co-organizers. My role involved coordinating events, finding speakers, and other related tasks.

I am very fortunate because I got into R in a friendly and collaborative environment. During graduate school, I collaborated with classmates and gained valuable knowledge from my brother. Later, I landed a new data analyst job and had a whole team of people who were interested in learning and using R. It has truly been a joy, and I feel appreciative for how well things have worked out and for the length of time that I’ve been able to use R.

Can you share what the R community is like in Seattle?

As I mentioned, my workplace involved various programming languages, but there were quite a few R users. We used to have these small study groups where we discussed creating an R Markdown template for our company and shared Shiny apps and other similar things. The field I worked in was education, but in Seattle, you see a lot of R being used in bioinformatics and scientific research related to diseases. It’s very popular among those groups, and they have solid user groups where they grow and learn together. Many members of R Ladies Seattle are from organizations like Fred Hutch, where the emphasis on using R is very strong, which is pretty great.

Every month, R-Ladies Seattle hosts a casual happy hour. We have good chips and salsa, and it’s a great opportunity for members to join, chat, and have a good time. Additionally, after the Cascadia R Conference in June, we will have a social hour where people can keep the conversation going in a relaxed setting. We will also host a social hour at the end of posit::conf in August, and R-Ladies who didn’t attend the conference are more than welcome to join and hang out. We organize many social events, so there are plenty of opportunities to connect with us.

We have some exciting events related to R coming up. We are currently looking for speakers for a lightning talk session, where individuals can quickly share the projects they are working on or a tool they love. It’s a low-pressure way to join in, and we welcome anyone who would like to sign up and participate.

Our focus is primarily on in-person and hybrid events. While we have seen an explosion of online events after COVID-19, it’s important to uphold the Seattle community by providing opportunities for local participation. Generally, our events are held in person, with occasional hybrid events. However, we are also excited about organizing online events with Seattle residents in mind at a convenient time for the Pacific Time Zone. Our offering of in-person, hybrid, and online events provides a unique experience for our user group in Seattle.

Any techniques you recommend using for planning for or during the event? (Github, zoom, other) Can these techniques be used to make your group more inclusive to people who are unable to attend physical events in the future?

One thing we implemented was to think of events that could generate high engagement. For example, with our Hex sticker, we organized a competition for all R-Ladies members to participate by submitting their designs, followed by a voting process. It was a lot of fun, both creatively and in terms of getting everyone involved. We strive to come up with similar engaging activities. Additionally, when appropriate, we reach out to other user groups in the area to explore collaboration opportunities, co-hosting events, or simply support each other’s promotions to foster a strong sense of community.

One method we have tried for planning events is using GitHub discussions to log our ideas for events and determine which events are the most popular based on comments and upvotes. This helps guide our future event planning.

What trends do you currently see in R language and your industry? Any trends you see developing shortly?

Every year, we send out a survey to inquire about people’s interests and what they like to see. The responses are usually a mix of technical content, with many requests for intermediate-level information. There’s a lot of interest from people in Seattle who use R in some capacity or are members of R-Ladies Seattle. They are looking for opportunities to upskill based on their existing knowledge of R. Additionally, there are many requests for information about career advancement and available positions.

There are various job titles related to data, such as data scientist and data engineer. Many people have questions about the career prospects in this field, including the potential for advancement and available options. These are common topics of discussion.

Please share about a project you are currently working on or have worked on in the past using the R language. Goal/reason, result, anything interesting, especially related to the industry you work in?

I now work at Posit, formerly RStudio, in a marketing role. I still get to work with R a lot, which is great for creating dashboards to track various metrics. I’ve been focusing on defining metrics of success and similar tasks.

Recently, I created a dashboard in Shiny that refreshes daily to compile the information I need for my to-do list. Every morning, I check my Shiny dashboard to see my daily tasks. It pulls information from my project management tool, so I only have to update one place to see an aggregated view of my month. It was fun to do this in R with Shiny.

Recently, I worked on creating a custom template in Quarto for the upcoming R Medicine Conference website. The website is built entirely on Quarto, a new tool similar to R Markdown. My work specifically involved designing the events page to display previous events and provide links to the event page and YouTube playlist. It’s exciting to learn and work with new tools like Quarto.

How do I Join?R Consortium’s R User Group and Small Conference Support Program (RUGS) provides grants to help R groups organize, share information, and support each other worldwide. We have given grants over the past four years, encompassing over 68,000 members in 33 countries. We would like to include you! Cash grants and meetup.com accounts are awarded based on the intended use of the funds and the amount of money available to distribute.

Learn moreThe post Diving into R with Isabella Velasquez: Perspectives from R-Ladies Seattle appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Diving into R with Isabella Velasquez: Perspectives from R-Ladies Seattle

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Introduction
In social sciences, variables of interest are often conceptualized as latent variables—hidden continuous variables measured through Likert scale questions, typically categorized as Strongly disagree, Disagree, Neutral, Agree, and Strongly agree. Researchers frequently aim to uncover these latent variables using various statistical techniques.

Accurate modeling of survey data is crucial for comparative analysis through simulation, especially when applying statistical techniques that require metric data. The latent2likert package addresses this need by providing an effective algorithm to simulate Likert response variables from hypothetical latent variables. This post introduces the features of the latent2likert package.

Simulating Likert Scale ResponsesUsing the rlikert function, you can generate random responses to Likert scale questions based on specified means and standard deviations of latent variables, with optional settings for skewness and correlations.

Reproducing Rating-Scale DataFrom existing survey data, you can estimate the values of latent parameters using the estimate_params function. You can then generate new responses using the estimated parameters to create a new dataset with very similar properties.

Further ReadingFor more detailed information and practical examples, please refer to the package website and vignette. The implemented algorithms are described in the function reference.

Related R PackagesTo simulate Likert scale responses, the draw_likert function from the fabricatr package can recode a latent variable into a Likert response variable by specifying intervals that subdivide the continuous range. However, the latent2likert package offers an advantage by automatically calculating optimal intervals that minimize distortion between the latent variable and the Likert response variable for both normal and skew normal latent distributions, eliminating the need to manually specify the intervals.

There are also alternative approaches that do not rely on latent distributions. One method involves directly defining a discrete probability distribution and sampling from it using the sample function in R or the likert function from the wakefield package. Another approach is to specify the means, standard deviations, and correlations among Likert response variables. For this, you can use LikertMakeR or SimCorMultRes to generate correlated multinomial responses.

Additionally, you can define a data-generating process. For those familiar with item response theory, the mirt package allows users to specify discrimination and difficulty parameters for each response category.

Introducing latent2likert v1.2.1: Converting Latent Variables into Likert Scale Responses was first posted on July 3, 2024 at 6:36 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing latent2likert v1.2.1: Converting Latent Variables into Likert Scale Responses

View Details

[This article was first published on ZAHIER NASRUDIN, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Introduction Purpose of the tutorial: To demonstrate a quick and straightforward implementation of time series clustering using the widyr package in R * What is time series clustering?*: Grouping time series data into clusters where data points in the same cluster group are more similar to each other than to those in other clusters. For example, if we have monthly sales data, time series clustering can help identify stores with similar sales patterns over time.

Load libraryCode library(tidyverse)library(widyr) Import data About the data*:

+ Fake dataset that can be downloaded from my GitHub.
+ Contains 832 rows & 3 columns


    - **Columns**:


        * `year` (`<date>`): Date information for each observation.
        * `storecode` (`<chr>`): Unique identifier for each store.
        * `sales` (`<dbl>`): Sales figures for each store.

Importing data: Using read_csv()

Code store\_list <- read\_csv("https://raw.githubusercontent.com/zahiernasrudin/datasets/main/sample\_store.csv") * Glimpse of the dataset:

| year | storecode | sales | | --- | --- | --- | | 2022-12-01 | A4P1Q1 | 22432 | | 2023-01-01 | A4P1Q1 | 22425 | | 2023-02-01 | A4P1Q1 | 20710 | | 2023-03-01 | A4P1Q1 | 23054 | | 2023-04-01 | A4P1Q1 | 23912 | | 2023-05-01 | A4P1Q1 | 22782 |

Clustering with widyrUsing widely_kmeans for time series clustering:

Code ```

Perform k-means clustering using widely_kmeanscluster_group <- store_list %>% widely_kmeans(item = storecode, feature = year, value = sales, k = 3)# Join the clustering results back to the original datastore_list_with_cluster <- left_join(store_list, cluster_group)

`` * **Defineitem`**:

+ **Description**: Item to cluster. In the context of our dataset, this would be the `storecode`

Define feature:
- Description: Feature column (dimension in clustering). In our case, the feature is the time component, which is represented by year column
- Define value:
- Description: Value column. In our dataset, this would be the sales
- Define k:
- Description: Number of clusters. This should be chosen based on the specific requirements of your analysis or determined using evaluation metrics. For the sake of simplicity in this tutorial, we will use 3 clusters.
- Joining Results: The clustering results are joined back to the original dataset.

Evaluating Clustering Results* We can visualize the clustering results using ggplot2.

Code library(ggthemes)store\_list\_with\_cluster |> ggplot(aes(x = year, y = sales, group = storecode, colour = cluster)) + geom\_line(show.legend = F) + scale\_y\_continuous(labels = scales::comma) + facet\_wrap(vars(cluster)) + scale\_color\_solarized() * There you have it, a simple way to implement time series clustering using the widyr package in R. Of course, there is much more you can explore and refine in your clustering analysis. For comprehensive documentation and further exploration of the widyr package, visit the widyr page itself: widyr Documentation.

To leave a comment for the author, please follow the link and comment on their blog: ZAHIER NASRUDIN.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Time Series Clustering in R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Have you been looking for a more efficient way to create ADaM (Analysis Data Model) datasets for your clinical trial submissions? Look no further than Admiral, an open-source R package.

In our latest Shiny Gathering, hosted in collaboration with Pharmaverse, featured Ben Straub, Principal Programmer at GSK and one of the maintainers of the package.

Collaborative projects led by pharmaceutical companies are propelling advancements in research. Explore R Packages for Clinical Trial Data in our blog post.

Ben shared his insights and expertise on the admiral R package, highlighting its utility for creating ADaM datasets in clinical submissions. Here’s a quick overview of the session for those who missed it.

The Admiral Package: An OverviewAdmiral is an R package designed to facilitate the creation of ADaM datasets, a critical component in clinical drug submissions. Ben walked through the package’s history, its alignment with CDISC standards, and its collaborative development involving multiple pharmaceutical companies.

Watch the VideoKey Features of Admiral1. Modular Functions: Admiral offers modular functions for adding variables and records to datasets. This modularity simplifies the process of constructing ADaM datasets, making it easier to follow and less prone to errors. 2. Documentation and Examples: The package includes extensive documentation and examples, aiding users in understanding and utilizing its functions effectively. 3. Unit Testing: Admiral incorporates unit testing to ensure code reliability, crucial in the highly regulated pharmaceutical industry. 4. GitHub Integration: The package leverages GitHub for collaborative development, with features like issue tracking, pull requests, and continuous integration ensuring high-quality code and documentation.

Practical DemonstrationBen provided a practical demonstration of using the admiral package, showcasing how to derive variables and merge datasets. He highlighted the simplicity and efficiency of admiral functions compared to base R and tidyverse alternatives, particularly in reducing repetitive coding tasks.

Deriving Variables: Using the derive_vars functions, users can easily add new variables to datasets.
Adding Records: The derive_param_computed functions allow for the addition of new records based on existing dataset information.

Future Goals for AdmiralBen outlined some of the future goals for the admiral package, including:

Reducing Complexity: Simplifying the datetime functions to enhance maintainability and reduce code complexity.
Improving Error Messaging: Enhancing user messaging to provide clearer guidance when errors occur, making it easier for users to troubleshoot issues.

Contributing to AdmiralHe encouraged the community to contribute to the admiral package, highlighting resources like the dummy issue for onboarding new contributors. He emphasized the collaborative nature of admiral’s development and the valuable learning opportunities it offers.

ConclusionAdmiral represents a significant step forward in using R for clinical submissions. By providing a standardized, well-documented, and collaboratively developed solution, it’s helping to pave the way for more R-based submissions in the pharmaceutical industry alongside other open-source packages in the Pharmaverse.

Here are some helpful links:

Join the Pharmaverse Slack channel
Subscribe to the Pharmaverse newsletter
Blog post about the latest admiral 1.1 update

Our next Shiny Gathering promises to share more insights for the pharma industry. Sign up for our newsletter to find out the details.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Recap: Exploring Clinical Submissions With Admiral: An R-Based ADaM Solution With Ben Straub

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhen working with large datasets in Excel, the ability to zoom in and out quickly can significantly enhance your productivity. If you often find yourself adjusting the zoom level manually, why not automate it with VBA? In this blog post, we’ll explore how to use the zoom functionality in VBA to control the zoom level of your worksheets efficiently.

Why Use Zoom in Excel?Zooming in Excel allows you to get a closer look at your data or see more of your worksheet at once. Whether you’re focusing on a specific section of data or getting an overview of your entire sheet, zooming can help you navigate and analyze your information more effectively.

Getting Started with Zoom in VBATo control the zoom level in Excel using VBA, you’ll primarily work with the ActiveWindow.Zoom property. This property allows you to set the zoom level to any value between 10% and 400%.

Basic Zoom In and Zoom OutLet’s start with simple macros to zoom in and zoom out.

Zoom In:

Sub ZoomIn() Dim currentZoom As Integer currentZoom = ActiveWindow.Zoom If currentZoom < 400 Then ActiveWindow.Zoom = currentZoom + 10 End IfEnd Sub Zoom Out:

Sub ZoomOut() Dim currentZoom As Integer currentZoom = ActiveWindow.Zoom If currentZoom > 10 Then ActiveWindow.Zoom = currentZoom - 10 End IfEnd Sub In these macros, ZoomIn increases the current zoom level by 10%, while ZoomOut decreases it by 10%. The code ensures that the zoom level stays within the permissible range of 10% to 400%.

Setting a Specific Zoom LevelSometimes, you might need to set the zoom level to a specific percentage. You can do this easily with the following macro:

Sub SetZoomLevel(zoomLevel As Integer) If zoomLevel >= 10 And zoomLevel <= 400 Then ActiveWindow.Zoom = zoomLevel Else MsgBox "Please enter a zoom level between 10 and 400." End IfEnd Sub You can call this macro with any desired zoom level. For example:

Sub ZoomToSpecificLevel() Call SetZoomLevel(150) ' Sets the zoom level to 150%End Sub Resetting the Zoom LevelIf you need to reset the zoom level to its default setting (usually 100%), you can use the following macro:

Sub ResetZoom() ActiveWindow.Zoom = 100End Sub Applying Zoom to a Specific WorksheetThe above examples modify the zoom level of the currently active window. If you want to set the zoom level for a specific worksheet, you can activate that sheet first and then set the zoom level:

Sub ZoomSpecificSheet(sheetName As String, zoomLevel As Integer) Worksheets(sheetName).Activate If zoomLevel >= 10 And zoomLevel <= 400 Then ActiveWindow.Zoom = zoomLevel Else MsgBox "Please enter a zoom level between 10 and 400." End IfEnd Sub Using Zoom with User FormsZoom functionality isn’t limited to worksheets. You can also control the zoom level of user forms in VBA. This is especially useful if your user form contains detailed information or numerous controls.

Sub ZoomUserForm(zoomLevel As Double) With UserForm1 .Zoom = zoomLevel End WithEnd Sub Call this macro with a zoom level between 10 and 400 to adjust the user form’s zoom.

ConclusionWhether you need to zoom in for a closer look, zoom out to see more data, or set a specific zoom level, VBA provides a simple and powerful way to control your view.

By incorporating these macros into your workflow, you can easily adjust your zoom settings without leaving the keyboard. Experiment with these examples, and you’ll soon find the perfect zoom levels to enhance your productivity in Excel.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering Zoom Functionality in Excel with VBA

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We are excited to welcome Beatriz Milz and Margaret Siple to our team of Associate Editors for rOpenSci Software Peer Review.They join Laura DeCicco, Julia Gustavsen, Anna Krystalli, Mauro Lepore, Noam Ross, Maëlle Salmon, Emily Riederer, Adam Sparks, and Jeff Hollister.rOpenSci has been running a rigorous and collegial software peer review system since 2015.Editors manage the review process, performing initial package checks, identifying reviewers, and moderating the process until the package is accepted by reviewers as part of rOpenSci’s software suite.To address the scope and volume of packages submitted for review, it’s critical that we have a team of editors with complementary expertise.Beatriz brings her experiences in environmental science and active community engagement.Margaret improves open data science and software development practices at the US Government’s National Oceanic and Atmospheric Administration (NOAA).

Meet our new editors!

Beatriz MilzBeatriz Milz received her PhD in Environmental Science from the University of São Paulo (USP) in Brazil. Her research is related to water resources management, transparency, and open data. She is an R-Ladies co-organizer of the São Paulo chapter in Brazil. She is organizing the translation to Portuguese of the second edition of the book “R for Data Science” (this translation is being done voluntarily by people from the community).Beatriz has contributed to packages available on CRAN, including dados, odbr, and geobr. She reviewed the qualR and frictionless packages for rOpenSci. She was a mentor in the first cohort of the rOpenSci Champions Program.

Beatriz on GitHub, Website, rOpenSci.

I first heard of rOpenSci in 2019, at Latin-R, during a community panel. I got interested because rOpenSci was aligned with things that are important to me, and started to follow its activities on social media and explore some of the packages. In 2022 I was first invited to review a package, and I was really excited: that was the first time that I had the opportunity to contribute to this community! rOpenSci represents to me not only a great community but also a way to contribute to the R ecosystem, encouraging best practices of code and Open Science. Now I am very happy to have the opportunity to contribute as an editor!

Beatriz Milz

Margaret SipleMargaret is a Research Fish Biologist at the National Oceanic and Atmospheric Administration (NOAA)’s Alaska Fisheries Science Center. She has a PhD in Aquatic and Fishery Sciences from the University of Washington. Her research focuses on advancing ecosystem-based fisheries management by improving the statistical models used to assess marine fish stocks and identify important habitat. She also works on advancing open data science practices within NOAA and improving the accessibility, documentation, and review of NOAA’s internally-developed software packages through the Fisheries Integrated Toolbox. She developed the R package mmrefpoints.

Margaret on Twitter, GitHub, website, rOpenSci.

I found my way to rOpenSci during graduate school and since reviewing my first R package for them, I have seen rOpenSci’s approach as a “north star” of R package development and documentation. rOpenSci’s rigorous (and well-communicated!) standards and open-source format set an example that we follow in my groups at NOAA, and the supportive and active rOpenSci community has been an inspiration to me throughout my career. I am honored to be able to contribute as a member of the editorial board.

Margaret Siple

Submit your package!Are you considering submitting your package for review? These resources will help.

About rOpenSci Software Peer Review;
Browse the online book rOpenSci Packages: Development, Maintenance, and Peer Review;
Read public software review threads on GitHub

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Announcing New Software Peer Review Editors: Beatriz Milz and Margaret Siple

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Submission Working Group is making significant strides in promoting the use of R for regulatory submissions in the pharmaceutical industry. At PharmaSUG 2024, held from May 19-22 in Baltimore, MD, the group’s impact was evident through various insightful presentations and discussions.

One highlight was Ben Straub’s presentation, “Piloting into the Future: Publicly available R-based Submissions to the FDA,” which showcased the growing adoption of R in both industry and regulatory settings. Straub emphasized the vibrant R community and its diverse packages that enhance statistical analysis and data visualization, highlighting R’s role in facilitating efficient and transparent FDA submissions.

Additionally, André Veríssimo and Ismael Rodriguez’s presentation, “Automating SDTM Using R: A Practical Guide,” demonstrated the advantages of using R for automating the creation of Study Data Tabulation Model (SDTM) datasets. They provided a detailed guide on implementing automation techniques and sharing best practices and real-world applications to improve data management workflows with R.

These presentations underscored the R Submission Working Group’s contributions to advancing the use of R in regulatory processes promoting greater efficiency, reproducibility, and transparency in pharmaceutical data management.

The post R Consortium’s Submission Working Group: Advancing R for Regulatory Success at PharmaSUG 2024 appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Consortium’s Submission Working Group: Advancing R for Regulatory Success at PharmaSUG 2024

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#195–196

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #195Is it told that analysts spend big chunk of their time on cleaning data. And sometimes it is indeed a mess. We have list of orders made by people, with products, prices and amounts written “how they liked”. Sometimes concatenated by commas, colons and even ampersands. And we need to clean it up, check which vendor gave us each product, and finally get to know how much we need to pay for vendors. Let’s do it.

Loading libraries and data library(tidyverse)library(readxl)path = "Power Query/PQ\_Challenge\_195.xlsx"input1 = read\_xlsx(path, range = "A1:C5")input2 = read\_xlsx(path, range = "A8:B11") test = read\_xlsx(path, range = "F1:G4") Transformation result1 = input1 %>% mutate(across(everything(), ~str\_split(.x, "\\W+"))) %>% unnest(cols = everything()) %>% mutate(total = as.numeric(`Unit Price`) * as.numeric(Quantity)) %>% select(Items, total)result2 = input2 %>% mutate(across(everything(), ~str\_split(.x, "\\W+"))) %>% unnest(cols = everything()) %>% mutate(part = n(), .by = Items)result = result2 %>% left\_join(result1, by = "Items") %>% mutate(paid\_by\_stockist = total/part) %>% summarise(`Amount Paid` = sum(paid\_by\_stockist, na.rm = T), .by = Stockist) Validation identical(result, test)# [1] TRUE Puzzle #196Sometimes there are documents that have to be done certain way, and there is no other possibility (I mean technically can be done, but obligation is obligation). And one of those in my experience is school class schedules and reports. And now we have something similar to both of them at once to transpose from nice table.

Loading libraries and data library(tidyverse)library(readxl)path = "Power Query/PQ\_Challenge\_196.xlsx"input = read\_xlsx(path, range = "A1:C11")test = read\_xlsx(path, range = "F1:O5") Transformation result = input %>% mutate(class1 = Class) %>% pivot\_wider(names\_from = Subject, values\_from = c(class1, Marks), names\_sep = "-") %>% select(-Class) %>% rename\_with(~str\_remove(., "class1-"), starts\_with("class1-")) %>% select(sort(names(.), decreasing = FALSE)) %>% select(1:3,9:10, everything()) Validation identical(result, test)#> [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.When it comes to finding an R package capable of making interactive visualizations out of the box while also working flawlessly with R Shiny, you don’t have that many options. Sure, there’s Highcarts, but what if you’re looking for something more specialized for time series?

Well, that’s where R Dygraphs chime in! This package serves as an R interface to a popular JavaScript library, and it works like a charm in R scripts and R Shiny. We’ll explore both options today.

‍After reading, you’ll know how to:

Plot R time series objects with R Dygraphs
Style R Dygraphs charts, add title and axis labels, show multiple series, and customize aesthetics
Tweak the chart interactivity
Use R Dygraphs in R Shiny

What are race charts and how do you make one in R? Read our guide on visualizing stock data over time with gganimate.

Table of contents:* R Dygraphs – How to Get Started * Deep Dive Into R Dygraphs – Visualize Stock Data * R Dygraphs in R Shiny – Build an App From Scratch * Summing up R Dygraphs

R Dygraphs – How To Get StartedThe Dygraphs package is available on CRAN, which means you can get it by running your typical install.packages() command. Make sure to run this first before proceeding with the article:

install.packages("dygraphs") And now to start using it, you’ll only need one line of code. Well, after the package import, of course. The following code snippet will create an interactive time series chart of the famous Airline passengers dataset:

library(dygraphs)dygraph(datasets::AirPassengers) Image 1 – First R dygraph chartCan you believe it only took one line of code? R Dygraphs does so much for you behind the scenes. As it turns out, tweaking the chart also requires much less code than the alternative charting packages. Let’s dive into that next.

Deep Dive Into R Dygraphs – Visualize Stock DataIn this section, you’ll learn the essentials behind R Dygraphs. We’ll cover extensive chart creation and styling, multi-series plots, and interactivity – all on stock data pulled straight from the web.

How to Get Stock Data in RThe first order of business is data gathering. Since we’re working with stock data, the quantmod package is your friend (install if necessary). It allows you to pull stock data from Yahoo Finance for a given ticker and time frame.

The returned stock dataframe will contain the time period (daily level) as an index column, and several other attributes showing the trade volume, open, close, and adjusted price for the day. We only care about the volume and the adjusted price:

library(quantmod)get\_stock\_data <- function(ticker, start\_date, end\_date) { data <- getSymbols(ticker, src = "yahoo", from = start\_date, to = end\_date, auto.assign = FALSE) # Removes the ticker name from column names colnames(data) <- gsub(paste0(ticker, "\\."), "", colnames(data)) data <- data[, c("Volume", "Adjusted")] return(data)}aapl <- get\_stock\_data(ticker = "AAPL", start\_date = "2023-01-01", end\_date = "2024-01-01")tail(aapl, 10) The above code snippet declares a function for fetching stock data and prints the last 10 values for the AAPL ticker (Apple):

Image 2 – Last 10 days of AAPL stock dataNow in R Dygraphs, you only need to call the dygraph() function on the column you want to visualize – Adjusted price in our case:

dygraph(aapl$Adjusted) Image 3 – Apple stock visualizedHovering over the points on the line will show you the corresponding X and Y axis values:

Image 4 – R Dygraphs interactivityis few and far between. It’s great to see Dygraphs doesn’t belong to this group, and does everything with so few lines of code.

Dygraph Customization and Chart OptionsLet’s take a look at some basic tweaking options first. The dygraph() function can take a couple of optional parameters. For example, main, xlab, and ylab control the labels for the title, X-axis, and Y-axis.

The width and height parameters will set the chart to have a fixed size, rather than its content resizing with the window:

dygraph( aapl$Adjusted, main = "Apple Stock Price (AAPL)", xlab = "Time period", ylab = "Adjusted price (USD)", width = 1200, height = 800) Image 5 – Adding title and axis labelsWe’ll keep only main, xlab, and ylab moving forward.

The cool thing about R Dygraphs is that it allows you to use the pipe operator (%>%) to chain function calls. For example, you can add an additional dySeries() function to control how the value axis looks and behaves.

We’ll change the line color and add square points to each data point:

dygraph(aapl$Adjusted, main = "Apple Stock Price (AAPL)", xlab = "Time period", ylab = "Adjusted price (USD)") %>% dySeries(color = "#0198f9", drawPoints = TRUE, pointSize = 3, pointShape = "square") Image 6 – Styling line chartsNow we’re getting somewhere!

Dygraphs also make it possible to plot multiple series on a single chart. To demonstrate, we’ll plot the scaled version of the Volume attribute as a filled and stepped area chart. The scaling down process is more or less mandatory since their values are on different orders of magnitude.

While here, let’s also change the labels shown on the series legend:

aapl$VolumeScaled <- aapl[, "Volume"] / 1000000dygraph(aapl[, c("Adjusted", "VolumeScaled")], main = "Apple Stock Price (AAPL) and Trade Volume") %>% dySeries("Adjusted", label = "Adjusted Price (USD)", color = "#0198f9", drawPoints = TRUE, pointSize = 3, pointShape = "square") %>% dySeries("VolumeScaled", label = "Trade Volume (M)", stepPlot = TRUE, fillGraph = TRUE, color = "#FF9900") Image 7 – Multiple plots on one chartThis kind of chart might work, but scaling up/down one variable to accommodate the other leaves a lot to be desired.

Luckily, R Dygraphs has the option to display multiple axis ticks. All you need to do is to set axis = “y2” on the series you’re working on second:

aapl <- get\_stock\_data(ticker = "AAPL", start\_date = "2023-01-01", end\_date = "2024-01-01")dygraph(aapl) %>% dySeries("Adjusted", label = "Adjusted Price (USD)", color = "#0198f9", drawPoints = TRUE, pointSize = 3, pointShape = "square") %>% dySeries("Volume", label = "Trade Volume (M)", stepPlot = TRUE, fillGraph = TRUE, color = "#FF9900", axis = "y2") Image 8 – Multiple Y-axis plot ticksYou’ve now successfully plotted two variables with different orders of magnitude!

And finally, let’s take a look into interactivity. We’ll fetch stock price information for a couple of additional tickers – Microsoft and Amazon – and concatenate everything into a single XTS object:

get\_stock\_data <- function(ticker, start\_date, end\_date) { data <- getSymbols(ticker, src = "yahoo", from = start\_date, to = end\_date, auto.assign = FALSE) colnames(data) <- gsub(paste0(ticker, "\\."), "", colnames(data)) data <- data[, "Adjusted"] return(data)}start\_date <- "2023-01-01"end\_date <- "2024-01-01"stock\_data <- get\_stock\_data("AAPL", start\_date, end\_date)names(stock\_data) <- ("AAPL")stock\_data$MSFT <- get\_stock\_data("MSFT", start\_date, end\_date)stock\_data$AMZN <- get\_stock\_data("AMZN", start\_date, end\_date)tail(stock\_data, 10) Image 9 – Multiple stock dataYou can now use the dyHighlight() function to control the highlighting behavior. For example, the below code snippet will increase the width of the highlighted line, add a circle marker, and decrease the opacity of the lines not currently highlighted:

dygraph(stock\_data, main = "Stock Price Comparison") %>% dyHighlight( highlightSeriesOpts = list(strokeWidth = 3), highlightCircleSize = 5, highlightSeriesBackgroundAlpha = 0.5 ) Image 10 – Multiple line chart with highlightsYou can also see how highlighting a point on one series also highlights the same point in time on others. R Dygraphs does this synchronization automatically – you don’t have to lift a finger!

And now with the basics of time series visualization under your belt, let’s take a look at implementation in R Shiny!

R Dygraphs In R Shiny – Build An App From ScratchThe idea behind the Shiny application you’re about to build is simple:

Give the user the option to select a stock ticker from a predefined list
Allow the user to change start/end dates for the stock analysis time window
Display historic stock prices, trade volume, and daily return percentage

To use Dygraphs in R Shiny, you’ll want to leverage the built-in dygraphOutput() function in the UI, and renderDygraph() server.

The get\_stock\_data() function will get the stock data for a given ticker and date range for you, and it will also calculate daily returns.

Moving further, the UI code is self-explanatory. It just contains the required components for input and output elements.

The server function is where things get interesting. First, we need to update date inputs on the fly through the observe() function, so that the user can’t select an end date that’s lower than the start date (and vice versa). Chart data is stored in stock\_data reactive value and is referenced in chart code down below:

library(dplyr)library(shiny)library(dygraphs)# Constantsallowed\_tickers <- c("AAPL", "AMZN", "GOOGL", "META", "MSFT", "NVDA", "TSLA")allowed\_min\_date <- as.Date("2010-01-01")allowed\_max\_date <- Sys.Date() - 1# Custom functions for getting and organizing dataget\_stock\_data <- function(ticker, start\_date, end\_date) { data <- getSymbols(ticker, src = "yahoo", from = start\_date, to = end\_date, auto.assign = FALSE) colnames(data) <- gsub(paste0(ticker, "\\."), "", colnames(data)) data <- data[, c("Volume", "Adjusted")] data <- data.frame(Date = index(data), coredata(data)) # In addition to what we had before, also calculate daily returns data <- data %>% arrange(Date) %>% mutate(DailyReturn = (Adjusted / lag(Adjusted) - 1) * 100) %>% na.omit() return(as.xts(data))}ui <- fluidPage( sidebarLayout( sidebarPanel( tags$h3("Stock price analyzer"), tags$hr(), selectInput(inputId = "inTicker", label = "Stock ticker:", choices = allowed\_tickers, selected = allowed\_tickers[1]), dateInput(inputId = "inStartDate", label = "Start date:", value = as.Date("2023-01-01"), min = allowed\_min\_date, max = allowed\_max\_date), dateInput(inputId = "inEndDate", label = "End date:", value = allowed\_max\_date, min = allowed\_min\_date, max = allowed\_max\_date) ), mainPanel( dygraphOutput("graphPrice"), dygraphOutput("graphVolume"), dygraphOutput("graphReturns") ) ))server <- function(input, output, session) { # Make sure the dates don't get messed up observe({ updateDateInput(session, "inStartDate", max = input$inEndDate - 1) }) observe({ updateDateInput(session, "inEndDate", min = input$inStartDate + 1) }) # Update the data as we go stock\_data <- reactive({ get\_stock\_data(ticker = input$inTicker, start\_date = input$inStartDate, end\_date = input$inEndDate) }) # Display charts output$graphPrice <- renderDygraph({ dygraph(stock\_data()$Adjusted, main = paste0(input$inTicker, " Stock Price (USD)")) %>% dySeries("Adjusted", label = "Adjusted price") }) output$graphVolume <- renderDygraph({ dygraph(stock\_data()$Volume, main = paste0(input$inTicker, " Trade Volume")) %>% dySeries("Volume", label = "Trade volume", stepPlot = TRUE, fillGraph = TRUE, color = "#FF9900") }) output$graphReturns <- renderDygraph({ dygraph(stock\_data()$DailyReturn, main = paste0(input$inTicker, " Daily Returns (%)")) %>% dySeries("DailyReturn", label = "Daily returns", color = "#cc0000") %>% dyLimit(0) })}shinyApp(ui = ui, server = server) You’ll see the following Shiny application after running the code:

Image 11 – R Shiny app with R DygraphsEverything works like a charm! R Dygraphs integrates nicely with R Shiny, and the only thing you have to worry about is the data format going into the visualizations. It has to be an XTS object – a plain dataframe won’t suffice.

Summing Up R DygraphsIf there’s one thing R doesn’t lack, it has to be options for data visualization. Even when R-native ones aren’t enough, there are plenty of alternatives from different programming languages, such as JavaScript.

R Dygraphs is one such option specializing in time series data visualization. It works exceptionally well with XTS objects and sets the record for the least amount of code the developer has to write. All charts are interactive by default, which isn’t always a given with R, and the package has superb R Shiny support. What more do you need?

What are your thoughts on time series visualization with R Dygraphs? Do you use it daily or do you prefer an alternative package? Make sure to let us know in our Slack community.

Is R the right choice for analyzing huge datasets? Here’s how well it does in a 1 billion row challenge.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Dygraphs: How To Visualize Time Series Data In R And R Shiny

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWelcome back, R Programmers! Today, we’ll explore a common task: extracting a substring after a specific character in R. Whether you’re cleaning data or transforming strings, this skill is quite handy. We’ll look at three approaches: using base R, stringr, and stringi. Let’s dive in!

ExamplesUsing Base RBase R provides several functions to manipulate strings. Here, we’ll use sub and strsplit to extract a substring after a specific character.

Example 1: Using subThe sub function allows us to replace parts of a string based on a pattern. Here’s how to extract the part after a specific character, say a hyphen (-).

```

Example stringstring <- "data-science"# Extract substring after the hyphenresult <- sub(".*-", "", string)print(result) # Output: "science"

```

[1] "science" Explanation:

.*- is a regular expression where .* matches any character (except for line terminators) zero or more times, and - matches the hyphen.
"" is the replacement, effectively removing everything up to and including the hyphen.

Example 2: Using strsplitThe strsplit function splits a string into substrings based on a delimiter.

```

Example stringstring <- "hello-world"# Split the string at the hyphenparts <- strsplit(string, "-")[[1]]# Extract the part after the hyphenresult <- parts[2]print(result) # Output: "world"

```

[1] "world" Explanation:

strsplit(string, "-") splits the string into parts at the hyphen, returning a list.
[[1]] extracts the first element of the list.
[2] extracts the second part of the split string.

Using stringrThe stringr package, part of the tidyverse, provides consistent and easy-to-use string functions.

Example 1: Using str_extractThe str_extract function extracts matching patterns from a string.

library(stringr)# Example stringstring <- "apple-pie"# Extract substring after the hyphenresult <- str\_extract(string, "(?<=-).*")print(result) # Output: "pie"

[1] "pie" Explanation:

(?<=-) is a look behind assertion, ensuring the match occurs after a hyphen.
.* matches any character zero or more times.

Example 2: Using str_splitSimilar to strsplit in base R, str_split splits a string based on a pattern.

```

Example stringstring <- "open-source"# Split the string at the hyphenparts <- str_split(string, "-")[[1]]# Extract the part after the hyphenresult <- parts[2]print(result) # Output: "source"

```

[1] "source" Explanation:

str_split(string, "-") splits the string into parts at the hyphen, returning a list.
[[1]] extracts the first element of the list.
[2] extracts the second part of the split string.

Using stringiThe stringi package is another powerful tool for string manipulation, providing high-performance functions.

Example 1: Using stri_extractThe stri_extract function extracts substrings based on patterns.

library(stringi)# Example stringstring <- "front-end"# Extract substring after the hyphenresult <- stri\_extract(string, regex = "(?<=-).*")print(result) # Output: "end"

[1] "end" Explanation:

regex = "(?<=-).*" uses a regular expression where (?<=-) is a lookbehind assertion ensuring the match occurs after a hyphen, and .* matches any character zero or more times.

Example 2: Using stri_splitSimilar to strsplit and str_split, stri_split splits a string based on a pattern.

```

Example stringstring <- "full-stack"# Split the string at the hyphenparts <- stri_split(string, regex = "-")[[1]]# Extract the part after the hyphenresult <- parts[2]print(result) # Output: "stack"

```

[1] "stack" Explanation:

stri_split(string, regex = "-") splits the string into parts at the hyphen, returning a list.
[[1]] extracts the first element of the list.
[2] extracts the second part of the split string.

ConclusionThere you have it—three different ways to extract a substring after a specific character in R. Each method has its own benefits and can be handy depending on your specific needs. Give these examples a try and see which one works best for your data!

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Extract String After a Specific Character in R

View Details

[This article was first published on Code R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Today I’m extremely happy because I’ve finally been able to fulfil a dream of mine.And yes, by the end of this blogpost you might be worried about me for having such a weird, niche and, frankly, dumb dream, but I swear I’m fine!

My dream was to create an R wrapper for the Climate Data Operators (CDO) automatically from its documentation.

CDO is a command-line utility that provides a plethora of functionality common to climate science.Things like computing monthly means, interpolating to grids, computing correlations, etc.It’s incredibly fast and convenient, can work with larger-than-RAM data for many operations as long as one horizontal field fits into memory, and can even run in parallel.

The only rub is that using CDO from R is kind of a pain.It means building system commands by paste0ing files and names of operators and managing the input and output files.Dreadful, unreadable code like this

in\_file <- "input\_file.nc"out\_file <- "climatology.nc"system(paste0("cdo ymonmean ", in\_file, " ", out\_file)) Oof.The stuff of nightmares, indeed.

Having a package that did all that for me including documentation and auto completion would be incredibly cool, but CDO has literally hundreds of different operators; wrapping all of them would be an herculean task.It’s just not worth it…

Is it?

A few years ago I watched a talk by Alicia Schep in which she explained how she built an R interface to the Vega-lite visualisation library.This javascript library creates interactive plots based on a json specification of the various geometries, axis and scales.There was a vegalite R package that created these json files from R, but it was outdated and it took a lot of work to get it up to speed.She realised that keeping the package in sync with Vega-lite development was not sustainable.So she actually created a package that took the vega-lite schema and automatically created the R functions that conformed to them.

Neat idea.A wrapper package created automatically from the documentation…could I apply this concept to my dream of running CDO from within R?

Yes and no.It turns out that the CDO API is very straightforward; each operator has a fixed number of inputs and outputs and potentially a couple of required and/or optional parameters.Creating a function that took the name of the operator, the number of inputs and outputs and the parameters was not that hard — there was a lot of reusable code.But unlike Vega-lite, CDO doesn’t have a nicely machine-readable schema that I could ingest; the documentation is in the form of a massive PDF with a lot of little inconsistencies that made automatic parsing very hard.

Screenshot of the documentation for MERSTAT - Meridional statistical valuesI did try it, though.I found bit of documentation in the CDO source code and with some text-wrangling I was able to extract part of the needed information.But it was hard and I wasn’t using CDO as much, so eventually I gave up.

The dream laid dormant for a few years.

A few months ago I got my PhD and was fortunate enough to immediately get a postdoc position.In my new job I had to get used to completely different infrastructure and I’m back to analysing relatively large datasets that are not as easy to handle with R.The situation thrust me back into CDO’s arms, where our love bloomed again, along with my desire to wrap it in a cosy blanket made of R.

So these few days, with renewed vigour and 4 more years of R experience, I went at it again.I honestly had all but forgotten that I had already tried this so it was a surprise to see that I already had an rdco repo with an very similar implementation waiting for me.I integrated some of that old code with the new one and the new rcdo package was born.

The rcdo package wraps most CDO operators in a pipable interface.Each operator is its own function with parameters as arguments, enabling autocomplete and documentation right in the R session.

For example, let’s take geopotential height from the NCEP reanalysis.

library(rcdo)ncep <- "data/hgt\_ncep.nc" We can compute the monthly annual cycle with the ymonmean operator, which is wrapped by rcdo::cdo_ymonmean()

ncep |> cdo\_ymonmean()## CDO command:## cdo ymonmean [ data/hgt\_ncep.nc ] {{output}} Running this doesn’t do anything other than to print the command, with the output as a place-holder.To execute the operator, one needs to use cdo_execute().

ncep |> cdo\_ymonmean() |> cdo\_excecute()## [1] "/tmp/RtmpNmkEms/file10cc641f84c1f" By default the output is saved in a temporary file.

Operators can be chained.Let’s select just the Southern Hemisphere, only look at 500 hPa and compute the seasonal means.I will also pipe that to metR::ReadNetCDF() to read the data and plot it with ggplot2.

Some operators take two or more inputs, which can be files or other CDO operations.For example, to compute monthly anomalies with respect to the 1981 to 2011 climatology first we need to compute the monthly climatology (cdo_ymonmean()) and then subtract it from the data (cdo_ymonsub()):

climatology <- ncep |> cdo\_selyear(years = 1981:2011) |> cdo\_ymonmean() anomalies <- ncep |> cdo\_ymonsub(ifile2 = climatology) |> cdo\_excecute() Notice that climatology is not a file but a CDO command.Just for fun, let’s read just one level and one month and plot it.

anomalies |> ReadNetCDF("hgt", subset = list(level = 500, time = "2024-01-01")) |> ggplot(aes(lon, lat)) + geom\_contour\_fill(aes(z = hgt)) + scale\_fill\_divergent() Lovely!

Now, the dream is not 100% complete.As I mentioned before, it’s not trivial to parse the CDO documentation, so there are still some issues.Some rcdo functions take extra arguments that do not correspond to their corresponding CDO operator and some arguments are just plain wrong.Some operators are also not supported, such as operators with zero output arguments and some with very hard-to-parse documentation.I’ll be ironing out those kinks, but I’m very proud of how this turned out.

To leave a comment for the author, please follow the link and comment on their blog: Code R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A wrapper for the Climate Data Operators in R

View Details

[This article was first published on ouR data generation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The most traditional approach for analyzing binary outcome data is logistic regression, where the estimated parameters are interpreted as log odds ratios or, if exponentiated, as odds ratios (ORs). No one other than statisticians (and maybe not even statisticians) finds the odds ratio to be a very intuitive statistic, and many feel that a risk difference or risk ratio/relative risks (RRs) are much more interpretable. Indeed, there seems to be a strong belief that readers will, more often than not, interpret odds ratios as risk ratios. This turns out to be reasonable when an event is rare. However, when the event is more prevalent, the odds ratio will diverge from the risk ratio. (Here is a paper that discusses some of these issues in greater depth, in case you came here looking for more.)

I was playing around with ORs and RRs using simstudy and realized that up until now, one could not specify a binary data generating process using an assumption about the underlying RR. (Well, you actually could, but it required the extra step of explicitly creating probability parameters using a RR assumption.) I’ve rectified that in the latest development version, by including a “log” link option for the binary distribution (and for binomial data generation more broadly). Here’s some simulation code to show this in action.

SimulationIn this data set, the treatment indicator is (A). The control group ((A = 0)) will have a (20\%) underlying probability of an outcome. The risk ratio is 1.8, so that the underlying probability of an outcome in the treatment group ((A = 1)) is (1.8 \times 0.20 = 36\%).

library(simstudy)library(data.table)def <- defData(varname = "A", formula = "1;1", dist = "trtAssign") |> defData( varname = "y", formula = "log(0.2) + A * log(1.8)", dist = "binary", link = "log" ) The data generation bears this out:

set.seed(123)dd <- genData(1000, def)## Key: <id>## id A y## <int> <int> <int>## 1: 1 0 0## 2: 2 0 0## 3: 3 0 0## 4: 4 1 0## 5: 5 0 0## --- ## 996: 996 0 1## 997: 997 0 0## 998: 998 0 0## 999: 999 1 0## 1000: 1000 0 0dd[, .(obs\_p = mean(y)), keyby = A]## Key: <A>## A obs\_p## <int> <num>## 1: 0 0.210## 2: 1 0.366 Added bonus: estimating the RR using regressionUnder the traditional approach, we might estimate a logistic regression model:

summary(glm(y ~ A, family = binomial, data = dd))## ## Call:## glm(formula = y ~ A, family = binomial, data = dd)## ## Coefficients:## Estimate Std. Error z value Pr(>|z|) ## (Intercept) -1.3249 0.1098 -12.067 < 2e-16 ***## A 0.7755 0.1438 5.393 6.91e-08 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## (Dispersion parameter for binomial family taken to be 1)## ## Null deviance: 1200.7 on 999 degrees of freedom## Residual deviance: 1170.7 on 998 degrees of freedom## AIC: 1174.7## ## Number of Fisher Scoring iterations: 4 The parameter of interest is (e^{0.7755} = 2.1716), which is a bit higher than the (RR = 1.8) used to generate the data. The reason, of course, is that it is the OR.

We can estimate the RR by using a log-binomial regression model. Like logistic regression, this is implemented using the glm function with the binomial “family”, but with a “log” link instead of a “logistic” link:

fit\_logbin = glm(y ~ A, family = binomial(link="log"), data = dd)summary(fit\_logbin)## ## Call:## glm(formula = y ~ A, family = binomial(link = "log"), data = dd)## ## Coefficients:## Estimate Std. Error z value Pr(>|z|) ## (Intercept) -1.56065 0.08674 -17.99 < 2e-16 ***## A 0.55553 0.10483 5.30 1.16e-07 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## (Dispersion parameter for binomial family taken to be 1)## ## Null deviance: 1200.7 on 999 degrees of freedom## Residual deviance: 1170.7 on 998 degrees of freedom## AIC: 1174.7## ## Number of Fisher Scoring iterations: 6 Here, the estimated parameter is the log(RR), so the estimated (RR =e^{0.5555} = 1.7428), much closer to the true value.

The paper I’ve referenced suggests that binomial regression with a log link does not always reliably converge, so they suggest a modified Poisson regression as a better approach. The modification arises because the standard errors estimated from the Poisson model are too conservative (i.e. too high). We know that the binary data (Bernoulli distribution) variation is proportional to (p(1-p)), whereas the variance for the Poisson model is proportional to (p). Since (0 < p < 1), (p > p(1-p)). This makes sense as the binary outcome data is limited to (0) and (1), but the Poisson data can include values greater than (1).

We can see this by generating data from each distribution:

rb <- rbinom(10000, 1, 0.20)rp <- rpois(10000, 0.20)## dist avg var min max## <char> <num> <num> <num> <int>## 1: binom 0.2 0.16 0 1## 2: pois 0.2 0.19 0 3 Now, let’s estimate a Poisson regression model. Since “log” is the default link for Poisson regression for function glm, we should get a risk ratio estimate similar to the log-binomial regression above, and in fact we do:

fit\_pois <- glm(y ~ A, family = poisson, data = dd)summary(fit\_pois)## ## Call:## glm(formula = y ~ A, family = poisson, data = dd)## ## Coefficients:## Estimate Std. Error z value Pr(>|z|) ## (Intercept) -1.56065 0.09758 -15.994 < 2e-16 ***## A 0.55553 0.12242 4.538 5.68e-06 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## (Dispersion parameter for poisson family taken to be 1)## ## Null deviance: 717.00 on 999 degrees of freedom## Residual deviance: 695.61 on 998 degrees of freedom## AIC: 1275.6## ## Number of Fisher Scoring iterations: 5 The only issue is that we see that the standard error is inflated somewhat, the log-binomial model standard error the treatment parameter was (0.10483), whereas the estimate is (0.12242) in the Poisson model. If we estimate a robust standard error using the function vcovHC in the sandwich package, the standard error is in now line with the log-binomial estimate. (I could simulate many data sets to see what the empirical variation of the estimate is, but I’ll leave that to you as an exercise.)

library(sandwich)data.table( estimate = summary(fit\_pois)$coef["A", "Estimate"], pois.SE = summary(fit\_pois)$coef["A", "Std. Error"], logbin.SE = summary(fit\_logbin)$coef["A", "Std. Error"], robust.SE = sqrt(diag(vcovHC(fit\_pois, type = "HC3")))["A"])## estimate pois.SE logbin.SE robust.SE## <num> <num> <num> <num>## 1: 0.5555258 0.122418 0.104825 0.1050351 And just to doubly make sure that the robust standard errors are reasonable, we can estimate standard errors using a simple bootstrap:

bootstrap\_both <- function(dx) { selected.rows <- dx[, sample(id, replace = TRUE), keyby = A][, V1] ds <- dx[selected.rows] fit\_logbin = glm(y ~ A, family = binomial(link="log"), data = ds) fit\_pois <- glm(y ~ A, family = poisson, data = ds) data.table(bs.logbin = coef(fit\_logbin)["A"], bs.pois = coef(fit\_pois)["A"]) }bs.res <- rbindlist(lapply(1:1500, function(x) bootstrap\_both(dd)))bs.res[, .(bs.se\_logbin = sd(bs.logbin), bs.se\_pois = sd(bs.pois))]## bs.se\_logbin bs.se\_pois## <num> <num>## 1: 0.1043538 0.1043538 It does look like the robust standard errors for the Poisson regression model are indeed pretty robust, and that in this simple case at least, there is no difference between the Poisson and log-binomial models.

References:

Guangyong Zou. “A Modified Poisson Regression Approach to Prospective Studies with Binary Data.” American Journal of Epidemiology. Volume 159, Issue 7, 1 April 2004, Pages 702–706.To leave a comment for the author, please follow the link and comment on their blog: ouR data generation.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Generating binary data by specifying the relative risk, with simulations

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Structural and Predictive Macro Analyses using the R Package bsvars, which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Structural and Predictive Macro Analyses using the R Package bsvars

Date: Thursday, August 1st, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Tomasz Wozniak, Tomasz is an econometrician who is developing new methods for empirical macroeconomic analyses. He codes these algorithms in C++ for R applications using Rcpp and authors the R package bsvars for Bayesian estimation of structural vector autoregressions. He is a senior lecturer at the University of Melbourne and co-organises the annual Melbourne Bayesian Econometrics Workshop.

Description: Quantifying the dynamic effects of well-isolated shocks on macro and financial aggregates is essential for governing institutions, academia, and business. This workshop presents a complete workflow for such analyses and focuses on various methods that facilitate interpretations and visualisations of data insights. It briefly introduces the necessary background on Bayesian Structural VARs. All this is complemented by a series of exercises, ensuring a hands-on learning experience. Please make sure to install the package following the instructions at https://bsvars.github.io/bsvars/#installation

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Structural and Predictive Macro Analyses using the R Package bsvars workshop was first posted on July 1, 2024 at 3:49 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Structural and Predictive Macro Analyses using the R Package bsvars workshop

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Staying ahead of the curve in machine learning often means adapting to unexpected changes. Recently, our team at Appsilon encountered a situation that highlights the importance of constant monitoring and flexible solutions when working with cloud-based Large Language Models (LLMs).

Interested in a demo of our Text2Graph application? Reach out to experts to set up a call today.

Today, we’d like to share our experience with GPT-4 and how it impacted Text2Graph (our R/Shiny application designed to transform your data into insights).

The Scenario: Using LLM APIs In Text2GraphOur Text2Graph platform relies on GPT-4 to generate code based on specific prompts. The core of our solution involved sending a carefully crafted prompt to GPT-4 and expecting the generated code to be neatly wrapped in triple backticks (“`).

The Unexpected Change: Challenges With LLM API UpdatesAfter a recent update to GPT-4, we noticed something peculiar. In about 50% of cases, the model started including the language type within the code block, resulting in outputs like ```r instead of just `````. This subtle change, while seemingly minor, had a significant impact on our application’s ability to process the generated code correctly. Let’s make it clear. The whole prompt was exactly the same, but the output has changed in a systematic way!

Our Solution: Adjusting Prompts For LLM APIsInterestingly, our approach to solving this issue wasn’t to modify our application to handle both “and “r. Instead, we found that adjusting our prompt was the most effective solution. This experience underscores the importance of prompt engineering and the delicate balance between the prompt, the model, and the application processing the output.

The TakeawayThis incident brings us to a crucial point that all developers and companies working with cloud LLMs should keep in mind:

With cloud LLMs, there’s no guarantee that a solution working today will continue to work tomorrow.

Unlike traditional software where you have control over the version and behavior of your tools, cloud-based AI models can be updated at any time, potentially altering their output in ways that might affect your applications.

Interested in learning how we apply machine learning to drug discovery? Check out Crystal Clear Vision, our model for protein crystal detection.

The Importance Of MonitoringThis experience showed us the importance of monitoring production applications that rely on LLMs.

Here are a few key reasons why:

Regular monitoring can help you quickly identify when model outputs start deviating from expected patterns.
By catching issues early, you can adjust your prompts or application logic to maintain consistent performance.
Continuous monitoring helps ensure that your AI-powered solutions remain reliable and trustworthy for your users.
With proper monitoring in place, you can adapt to changes in model behavior swiftly, minimizing downtime or degradation in service quality.

Additional Challenges Of Using LLM APIsWhile building with LLMs, here are some other challenges that should be considered:

Context Window Limitations: LLMs have input size limits, necessitating creative solutions like chunking data or using embeddings.
Latency and Performance Issues: LLMs can be slow, and chaining calls makes latency problems worse. Sometimes you may need to opt-in for a more performant model of a lesser quality.
Prompt Engineering Complexity: Crafting effective prompts requires continuous experimentation. It’s worth pointing out that an optimal prompt for OpenAI’s GPT-4o will be different from Anthropic’s Claude 3.5. It will be even different from the GPT-4 prompt!
Prompt Injection Risks: Guardrails are needed to mitigate security risks.
Product Development Realities: LLMs are tools for features, not complete products, requiring standard design and validation processes.
Legal and Compliance Concerns: Ensuring data privacy and regulatory compliance is essential.

Summing Up Challenges Of Working With Cloud LLMsAs we continue to push the boundaries of what’s possible with AI and LLMs, we should remember that these powerful tools come with their own set of challenges. By being ready to adapt, we can utilize these tools to their full potential. Sometimes it’s worth using additional layers between you and the LLMs with tools like LangChain.

At Appsilon, we’re committed to sharing our experiences and insights as we navigate this exciting and rapidly changing landscape. We hope that by sharing this, we can help other teams better prepare for the unique challenges of working with cloud-based LLMs.

Interested in more insights and best practices in R/Shiny and Machine Learning? Subscribe to our newsletter to stay up to date.

Note: Thank you Pasza Storożenko for providing guidance in writing this article.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Hidden Challenges Of Working With Cloud Large Language Models: A Text2Graph Case Study

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 484–488

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #484I think we all know what Pythageorean Theorem is. Or at least I hope so… And today we need to use properties of right triangle to solve our challenge. We need to find groups of 3 numbers (called Pythagorean triplets), that are about two things. Sum of a squared and b squared are equal c squared, but also sum of a, b and c (means circumference) is equal to given number. There is possibility that given number has more then one triplet, and it is true for one case.

Loading libraries and data library(tidyverse)library(readxl)library(gmp)path = "Excel/484 Pythagorean Triplets for a Sum.xlsx"input = read\_xlsx(path, range = "A2:A10")test = read\_xlsx(path, range = "B2:D10") %>% mutate(across(everything(), as.numeric)) Transformation find\_pythagorean\_triplet <- function(P) { m\_max <- floor(sqrt(P / 2)) possible\_values <- expand\_grid(m = 2:m\_max, n = 1:(m\_max - 1)) %>% filter(m > n, m %% 2 != n %% 2, gcd(m, n) == 1) triplets <- possible\_values %>% pmap(function(m, n) { k <- P / (2 * m * (m + n)) if (k == floor(k)) { a <- k * (m^2 - n^2) b <- k * 2 * m * n c <- k * (m^2 + n^2) return(c(a, b, c)) } else { return(NULL) } }) triplet <- triplets %>% compact() %>% keep(~ sum(.x) == P) if (length(triplet) > 0) { result <- triplet[[1]] } else { result <- c(NA\_real\_, NA\_real\_, NA\_real\_) } tibble(a = result[1], b = result[2], c = result[3])}result = input %>% pmap\_dfr(~ find\_pythagorean\_triplet(..1)) Validation ```

in one case (for 132) I get another but correct result

``` Puzzle #485Sequences, world is full of them, and some have even their name and well researched properties. It is the case with Padovan sequence, which is similar to Fibonacci’s but has little bit bigger step and another initial elements. Check it out.
PS. To run it efficiently and save some time I also use memoise.

Loading libraries and data library(purrr)library(memoise)library(readxl)library(tidyverse)path = "Excel/485 Pandovan Sequence.xlsx"input = read\_excel(path, range = "A1:A10")test = read\_excel(path, range = "B1:B10") Transformation padovan <- function(n) { if (n <= 2) { return(1) } else { return(padovan\_memo(n - 2) + padovan\_memo(n - 3)) }}padovan\_memo = memoise(padovan)result = input %>% mutate(`Answer Expectecd` = map\_dbl(n, padovan\_memo)) Validation identical(result$`Answer Expectecd`, test$`Answer Expectecd`)# [1] TRUE Puzzle #486When we are measuring thing like ranges and distances, sometimes for better and more readable notes, we are shortening down sequences of consecutive numbers using hyphenated range notation. And that is our task today. Let’s get party started.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/486 Create Integer Intervals.xlsx"input = read\_excel(path, range = "A1:A8")test = read\_excel(path, range = "B1:B8") Transformation group\_consecutive = function(number\_string) { numbers <- str\_split(number\_string, ",") %>% unlist() %>% as.numeric() tibble(numbers = sort(numbers)) %>% mutate(group = cumsum(c(TRUE, diff(numbers) != 1))) %>% summarise(range = if\_else(n() > 1, paste0(min(numbers), "-", max(numbers)), as.character(numbers[1])), .by = group) %>% pull(range) %>% paste(collapse = ", ")}result = input %>% mutate(`Answer Expected` = map\_chr(Problem, group\_consecutive)) Validation identical(result$`Answer Expected`, test$`Answer Expected`)#> [1] TRUE Puzzle #487In this puzzle we need to find most frequent characters in given strings. It was like warm up before warm up. With no further words, lets do it.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/487 Maximum Frequency Characters.xlsx"input = read\_xlsx(path, range = "A2:A11")test = read\_xlsx(path, range = "B2:C11") Transformation find\_most\_freq\_char = function(input) { result = input %>% strsplit("") %>% unlist() %>% table() df = data.frame( Characters = names(result), Frequency = as.numeric(result)) %>% filter(Frequency == max(Frequency)) %>% summarise(Characters = paste(sort(Characters), collapse = ", "), .by = Frequency) %>% select(Characters, Frequency) return(df)}result = input$Strings %>% map(find\_most\_freq\_char) %>% bind\_rows() Validation: ```

Validation on eye, all correct. In three cases there is different sorting than# in provided example but with the same characters.

``` Puzzle #488Combinations, possible inputs to given output. I am sure we all love it. Today we are given bunch of numbers and one output. We need to find all combinations of this numbers that will sum up to given target. Code is quite long, but I think also easy to understand. Let me show it.

Loading libraries and data library(gtools)library(tidyverse)library(readxl)path = "Excel/488 Numbers to Meet Target Sum.xlsx"input = read\_excel(path, range = "A1:A10")target = read\_excel(path, range = "B1:B2") %>% pull()test = read\_excel(path, range = "C1:C5") Transformation find\_combinations <- function(numbers, target) { combs <- map(1:length(numbers), ~combinations(length(numbers), ., v = numbers)) valid\_combs <- combs %>% map(as\_tibble) %>% bind\_rows() %>% mutate(sum = rowSums(., na.rm = TRUE)) %>% filter(sum == target) return(valid\_combs)}combinations <- find\_combinations(input$Numbers, target) %>% unite("Combination", - sum, sep = ", ", remove = T, na.rm = T) sort\_numbers <- function(numbers) { paste(sort(as.numeric(strsplit(numbers, ",")[[1]])), collapse = ", ")}test <- test %>% mutate(Combination = map\_chr(`Answer Expected`, sort\_numbers)) Validation identical(sort(combinations$Combination), sort(test$Combination))# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
On my Github repo there are also solutions for the same puzzles in Python. Check it out!

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn the world of time series analysis, Random Walks, Brownian Motion, and Geometric Brownian Motion are fundamental concepts used in various fields, including finance, physics, and biology. Today, we’ll explore these concepts using functions from the healthyR.ts package.

Random WalksA Random Walk is a path that consists of a series of random steps. It’s a simple but powerful concept used to model seemingly unpredictable paths, such as stock prices or animal movements.

Let’s generate and plot some Random Walks using the ts_random_walk() function from healthyR.ts.

Function Syntax ts\_random\_walk( .mean = 0, .sd = 0.1, .num\_walks = 100, .periods = 100, .initial\_value = 1000) * .mean: The desired mean of the random walks. * .sd: The standard deviation of the random walks. * .num_walks: The number of random walks you want to generate. * .periods: The length of the random walk(s) you want to generate. * .initial_value: The initial value where the random walks should start.

Example library(ggplot2)library(healthyR.ts)random\_walk\_data <- ts\_random\_walk( .mean = 0, .sd = 0.1, .num\_walks = 10, .periods = 100, .initial\_value = 1000 )head(random\_walk\_data)

```

A tibble: 6 × 4 run x y cum_y 1 1 1 0.0725 1072.2 1 2 0.182 1267.3 1 3 0.110 1407.4 1 4 0.0275 1445.5 1 5 -0.0546 1367.6 1 6 0.0712 1464.

```

random\_walk\_plot <- random\_walk\_data |> ggplot( mapping = aes( x = x, y = cum\_y, color = factor(run), group = factor(run) ) ) + geom\_line(alpha = 0.8) + ts\_random\_walk\_ggplot\_layers(random\_walk\_data)print(random\_walk\_plot) This code generates 10 random walks over 100 periods, starting from an initial value of 1000. The resulting plot visualizes the paths of these random walks, each represented by a different color.

Brownian MotionBrownian Motion, also known as Wiener Process, is a continuous-time stochastic process that is often used to model random movements in physics and finance.

Function Syntax ts\_brownian\_motion( .time = 100, .num\_sims = 10, .delta\_time = 1, .initial\_value = 0, .return\_tibble = TRUE) * .time: Total time of the simulation. * .num_sims: Total number of simulations. * .delta_time: Time step size. * .initial_value: Initial value of the simulation. * .return_tibble: Return a tibble (TRUE) or a matrix (FALSE).

Example brownian\_data <- ts\_brownian\_motion( .time = 100, .num\_sims = 10, .delta\_time = 1, .initial\_value = 0, .return\_tibble = TRUE)head(brownian\_data)

```

A tibble: 6 × 3 sim_number t y 1 sim_number 1 1 02 sim_number 2 1 03 sim_number 3 1 04 sim_number 4 1 05 sim_number 5 1 06 sim_number 6 1 0

```

brownian\_plot <- ts\_brownian\_motion\_plot( .data = brownian\_data, .date\_col = t, .value\_col = y, .interactive = TRUE)brownian\_plot To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Random Walks and Brownian Motions with healthyR.ts

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, everyone! Today, we’ll be diving into a practical example of how to run a macro when a cell value changes in VBA. This is particularly useful when you need to trigger certain actions based on user input or dynamic data changes in your Excel sheets. Let’s get started!

Setting Up the Worksheet Change EventTo run a macro when a cell value changes, we need to use the Worksheet_Change event. This event is triggered every time a cell value in a specific worksheet changes.

Here’s the basic structure of the Worksheet_Change event:

Private Sub Worksheet\_Change(ByVal Target As Range) ' Your code hereEnd Sub In this event, Target refers to the range of cells that have changed. We’ll use this to identify if the change occurred in the cell or range of cells we are interested in.

Example ScenarioImagine we want to run a macro whenever the value in cell A1 changes. We’ll write a simple macro that displays a message box when this happens.

Step-by-Step Guide1. Open the VBA Editor: * Press ALT + F11 to open the VBA editor. 2. Insert the Code: * In the VBA editor, find the sheet where you want to apply the change event. For example, Sheet1. * Double-click on Sheet1 to open its code window. * Insert the following code:

Private Sub Worksheet\_Change(ByVal Target As Range) ' Check if the changed cell is A1 If Not Intersect(Target, Me.Range("A1")) Is Nothing Then ' Run your macro here Call MyMacro End IfEnd Sub Explanation Worksheet_Change Event: This event gets triggered whenever any cell value in Sheet1 changes. * Intersect Function: We use the Intersect function to check if the changed cell (Target) overlaps with cell A1 (Me.Range("A1")). If there is an intersection (i.e., the changed cell is A1), the condition returns True. * Call MyMacro:* When the condition is True, we call another macro named MyMacro. This is where you define what actions you want to perform when cell A1 changes.

Defining the MacroNext, let’s define the MyMacro that gets called when cell A1 changes. For simplicity, we’ll make it display a message box.

Sub MyMacro() MsgBox "Cell A1 has changed!"End Sub Putting It All TogetherHere’s the complete code for Sheet1:

Close the VBA editor and go back to Excel.
Change the value in cell A1.
You should see a message box saying, “Cell A1 has changed!”

ConclusionBy using the Worksheet_Change event, you can easily set up macros to run whenever specific cell values change. This can be incredibly useful for automating tasks and making your Excel workbooks more dynamic and interactive.

I hope you found this guide helpful! If you have any questions or want to share how you use this in your projects, feel free to leave a comment below.

Happy coding!

Referencehttps://learn.microsoft.com/en-us/office/vba/api/excel.worksheet.change

That’s it for today! Remember to keep experimenting and exploring new ways to automate your Excel tasks using VBA. See you next time!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Run a Macro When a Cell Value Changes in VBA

View Details

[This article was first published on Brenden Smith, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionRecently, I participated in Posit’s 2024 Table Contest. For my submission, which you can view here, I included a leaflet map that acts as a filter in Shiny. This is a cool, dashboard-like feature similar to what you might find in Power BI. I recreated this effect and learned a bit through the process.

I first saw this wonderful blog post by Nathan Day but realized didn’t exactly match the feel I was going for. I adapted his code and added my own preferences (specifically allowing the input map to select multiple polygons and resetting the output table when polygons were “unclicked”). I wanted to share a basic example for others who might want to try this out!

Example DataThe data I will be using for this example can be queried using the CDCPLACES package (see more on GitHub). I will take a sample of county data from the State of Ohio. Here I am filtering only the age-adjusted rates and the measure “ACCESS2” which is the percentage of the population aged 18-64 that lack health insurance. I will also set the CRS for the data using sf::st_transform to avoid warnings when the data is queried.

Code library(leaflet)library(shiny)library(CDCPLACES)library(dplyr)ohio <- get\_places(state = "OH", measure = "ACCESS2", geometry = TRUE) |> filter(datavaluetypeid == "AgeAdjPrv") |> select(year, stateabbr, locationname, measure, data\_value, geometry) |> sf::st\_transform(crs = 4326) UINext, we can get into the UI side of our demo app. This is fairly straightforward. We initiate a fluid page, a title, and a sidebar layout. The sidebar has our leaflet map as a filter. In the main panel, we will output a data table.

I have added a tags$head function to add some custom CSS to the app. This is an optional step, but these two options make the panel transparent, which I think adds a lot to the look and feel of the app.

Code ui <- fluidPage( tags$head( tags$style(HTML(".leaflet-container { background: none; } .well { background: none;}")) ), titlePanel("My Demo App"), sidebarLayout( sidebarPanel( leafletOutput("mapfilter", height = 250) ), mainPanel( DT::DTOutput("table") ) )) ServerNow we can specify the logic of the server to get the result we want. To start we can initialize a few reactive values. This will allow us to update our filtered data and what is displayed on the map. selected_counties will correspond to what is highlighted on the map when we click, filtered_data will be the data frame that is displayed on the main table output.

Code server <- function(input, output, session) { # Initialize reactive values rv <- reactiveValues(selected\_counties = NULL, filtered\_data = ohio) # Initialize reactive values} NoteThe following code chunks are wrapped within the server function call.

OutputsThis section will briefly describe the functions for our outputs: the map filter and the table.

TableThis chunk defines the output corresponding to the id table, and renders a datatable. We input the reactive value of our filtered data with rv$filtered_data, remove the geometry with sf::st_set_geometry(NULL), and send it to DT::datatable() for a simple table display.

Code output$table <- DT::renderDT({ rv$filtered\_data |> sf::st\_set\_geometry(NULL) |> DT::datatable() }) MapFor our map, we follow similar steps. We use the base data frame ohio to create our map. Future steps will show how we update this with our click behavior. highlightOptions here defines how the map reacts to hovering over polygons. It will fill the county the mouse is hovering over.

Code output$mapfilter <- renderLeaflet({ # rendering the filter map leaflet(ohio, # initializing the map options = leafletOptions( zoomControl = FALSE, dragging = FALSE, minZoom = 6, maxZoom = 6 )) |> # then add polygons addPolygons(layerId = ~locationname, label = ~locationname, col = "black", fillColor = "steelblue", weight = 2, fillOpacity = .1, highlight = highlightOptions( fillOpacity = 1, bringToFront = TRUE )) }) Click BehaviorNext, we will define our behavior when the map is clicked. We can break this into two parts, updating the data that is fed into the output table, and changing the display of the input map.

The code chunk below runs when a polygon on our map is clicked. That is the logic of the observeEvent function and its argument input$mapfilter_shape_click. Because our actions all relate to this event, we can wrap all of our code in it. The other step here is to store the input in an object called click.

Code observeEvent(input$mapfilter\_shape\_click, { # this is the logic behind the "click" of the map. click <- input$mapfilter\_shape\_click }) If we were to simply print(click) we would see the following output upon an initial click and a second click of the same polygon:

Initial Click
Second Click

This will inform how we use the input to update our data and map.

We can use a set of if and else statements to store data from click in our reactive values.

The first statement checks to see if the current click$id exists in rv$selected_counties. If it does, it will remove it from the vector.
The next statement checks to see if the click$id is equal to “selected”. Recall that this occurs when the same polygon is selected twice in a row. If this condition is met, we will filter rv$selected_counties by removing the last value in the length of the vector.
Lastly, if the other two conditions are not met, the new and unique click$id is added to rv$selected_counties.

Code if (click$id %in% rv$selected\_counties) { # If selected, remove it rv$selected\_counties <- rv$selected\_counties[rv$selected\_counties != click$id] } else if(click$id == "selected"){ # when a county is clicked again it is removed rv$selected\_counties <- rv$selected\_counties[rv$selected\_counties != tail(rv$selected\_counties, n = 1)] }else { # If not selected, add it rv$selected\_counties <- c(rv$selected\_counties, click$id) } Then we have an update to our map. We can accomplish this with leafletProxy. We will simply add an ifelse function to the argument fillOpacity. This ensures that counties present in our rv$selected_counties will have the proper fill.

Code leafletProxy("mapfilter", session) |> addPolygons(data = ohio, layerId = ~locationname, label = ~locationname, fillColor = "steelblue", col = "black", weight = 2, fillOpacity = ifelse( ohio$locationname %in% rv$selected\_counties, 1, 0.1 ), highlight = highlightOptions( fillOpacity = 1, bringToFront = TRUE) ) Each of these pieces all fit into our observeEvent function for a click on the map, so in our consolidated code it will look like this:

Code observeEvent(input$mapfilter\_shape\_click, { click <- input$mapfilter\_shape\_click if (click$id %in% rv$selected\_counties) { rv$selected\_counties <- rv$selected\_counties[rv$selected\_counties != click$id] } else if(click$id == "selected"){ rv$selected\_counties <- rv$selected\_counties[rv$selected\_counties != tail(rv$selected\_counties, n = 1)] }else { rv$selected\_counties <- c(rv$selected\_counties, click$id) } leafletProxy("mapfilter", session) |> addPolygons(data = ohio, layerId = ~locationname, label = ~locationname, fillColor = "steelblue", col = "black", weight = 2, fillOpacity = ifelse( ohio$locationname %in% rv$selected\_counties, 1, 0.1 ), highlight = highlightOptions( fillOpacity = 1, bringToFront = TRUE) ) }) Lastly, we have one more if else statement in our server. The following code chunk takes the reactive value rv$selected_counties and updates rv$filtered_data which we use to render the table. This logic will cause the data to reset when we have no selected counties (all the shapes are “unclicked”).

Code observe({ # Update table filtering based on selected counties if (!is.null(rv$selected\_counties) && length(rv$selected\_counties) > 0) { # Check if any counties are selected rv$filtered\_data <- ohio |> filter(locationname %in% rv$selected\_counties) } else { rv$filtered\_data <- ohio } }) ConclusionThis post was an excellent way for me to revisit my code and share an interesting and unique Shiny feature. In this process I ended up eliminating quite a few redundancies in my original code and reinforced some of the concepts of reactivity showcased here.

I hope you find this tutorial useful. If you put it to use, please share it with me! I would love to see the work you come up with.

See the full consolidated example code below.

Full CodeCode library(leaflet)library(shiny)library(tigris)library(CDCPLACES)library(dplyr)library(htmltools)ohio <- get\_places(state = "OH", measure = "ACCESS2", geometry = TRUE) |> filter(datavaluetypeid == "AgeAdjPrv") |> select(year, stateabbr, locationname, measure, data\_value, geometry) |> sf::st\_transform(crs = 4326)ui <- fluidPage( tags$head( tags$style(HTML(".leaflet-container { background: none; } .well { background: none;}")) ), # Application title titlePanel("My Demo App"), # Sidebar with a slider input for number of bins sidebarLayout( sidebarPanel( leafletOutput("mapfilter", height = 250) ), # Show a plot of the generated distribution mainPanel( DT::DTOutput("table") ) ))# Define server logic required to draw a histogramserver <- function(input, output, session) { rv <- reactiveValues(selected\_counties = NULL, filtered\_data = ohio) # Initialize reactive value for selected counties observeEvent(input$mapfilter\_shape\_click, { # this is the logic behind the "click" of the map. click <- input$mapfilter\_shape\_click ########## map behavior ################ # If a county is clicked if (click$id %in% rv$selected\_counties) { # If selected, remove it rv$selected\_counties <- rv$selected\_counties[rv$selected\_counties != click$id] } else if(click$id == "selected"){ # when a county is clicked again it is removed rv$selected\_counties <- rv$selected\_counties[rv$selected\_counties != tail(rv$selected\_counties, n = 1)] }else { # If not selected, add it rv$selected\_counties <- c(rv$selected\_counties, click$id) } leafletProxy("mapfilter", session) |> addPolygons(data = ohio, layerId = ~locationname, label = ~locationname, fillColor = "steelblue", # Change fill color based on selection col = "black", weight = 2, fillOpacity = ifelse(ohio$locationname %in% rv$selected\_counties, 1, 0.1), highlight = highlightOptions( fillOpacity = 1, bringToFront = TRUE) ) }) output$mapfilter <- renderLeaflet({ # rendering the filter map leaflet(ohio, options = leafletOptions( # initializing the map zoomControl = FALSE, dragging = FALSE, minZoom = 6, maxZoom = 6 )) %>% addPolygons(layerId = ~locationname, label = ~locationname, # fillColor = "black", col = "black", fillColor = "steelblue", weight = 2, fillOpacity = .1, highlight = highlightOptions( fillOpacity = 1, bringToFront = TRUE )) }) output$table <- DT::renderDT({ rv$filtered\_data |> sf::st\_set\_geometry(NULL) |> DT::datatable() }) observe({ # Update table filtering based on selected counties if (!is.null(rv$selected\_counties) & length(rv$selected\_counties) > 0) { # Check if any counties are selected rv$filtered\_data <- ohio |> filter(locationname %in% rv$selected\_counties) } else { rv$filtered\_data <- ohio } })}# Run the applicationshinyApp(ui = ui, server = server) To leave a comment for the author, please follow the link and comment on their blog: Brenden Smith.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Interactive Map Filter in Shiny

View Details

[This article was first published on R - datawookie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The Big Book of R provides a comprehensive and ever-growing overview of a broad selection of R programming books. It was created and is maintained by Oscar Baruffa. The collection began with approximately 100 books and, with the help of contributions from the R community, has subsequently expanded to over 400. The books are grouped into topics such as geospatial, machine learning, statistics, text analysis, and many more. The Big Book of R is an excellent resource for anyone learning R programming, whether they are a beginner or advanced user.

What we set out to doFathom Data uses R extensively in our consulting work. We took on the migration of the Big Book of R from Bookdown to Quarto as a way of giving back to the R community, to which we owe so much. This migration offers several advantages such as:

Consistency — Quarto merges the functions of R Markdown, Bookdown, Distill, and Xaringan into one system. Instead of multiple YAML configuration files (_bookdown.yml and _output.yml) there’s just a single _quarto.yml file.
Support for Many Languages and Tools — Quarto supports many programming languages (R, Python, JavaScript, and Julia, as well as new languages that might be added in the future) and tools (like knitr, Jupyter, and Observable).
Works Well with Existing Content — Quarto can use most R Markdown documents and Jupyter notebooks without needing changes.
Elegant Outputs — Quarto, built on Pandoc, is an open-source system that lets you combine text and code to make beautifully formatted documents, web pages, blog posts, books, and more.
Managing Bibliographies — Quarto has a smart way to handle bibliographies.

Where we startedWe began with a GitHub repository that contained the R Markdown project for the book. This repository had three main parts: a preface, a page about various R communities, and the main script. The main script pulls book data from Google Sheets, sorts it into chapters, and creates a well-organised report. This report includes detailed information about the authors and books, which is used to render each chapter in the book.

ApproachOur approach involved setting up a new Quarto book project, to which we gradually transferred files. The first two R Markdown (.Rmd) files were converted into Quarto markdown format (.qmd). We didn’t need the index file because Quarto operates with a _quarto.yml file that defines the book structure. All configuration options were transferred from the index to the _quarto.yml file and the rest of the content was incorporated into the index.qmd file.

ChallengesUnlike Markdown, where a single script could render the book by fetching data from Google Sheets, Quarto required a different approach.

Quarto needs each chapter to be its own .qmd file within the project folder, and these files must be listed in _quarto.yml. We achieved this by developing a script that retrieves information from Google Sheets for each chapter, saves each chapter as an individual .qmd file, and creates a list of all the .qmd files as a text file. This made it simpler to update the _quarto.yml file with new or modified chapter titles.

Lessons learned* The {googlesheets4} package worked well to generate .qmd files from Google Sheets. * Projects can successfully be migrated from Markdown or Bookdown to Quarto with the proper configuration steps. * CSS could easily be integrated with Quarto to enhance the book’s presentation. * There is a built-in search bar for Quarto books in the _quarto.yml template.

Finished productThe finished product retains the structural essence of the original Bookdown format, with several enhancements. It features a book cover and CSS styling that improves the visual appeal of the book, which includes an easy to use light and dark mode toggle. Additionally, each chapter is equipped with its own table of contents, making it much easier for readers to navigate.

Why this is an improvementQuarto facilitates easier collaboration with its multilingual and multi-engine support. This is ideal for projects that involve multiple programming languages or the integration of various data sources and analytical tools.

Using a script to organise content into separate .qmd files for each chapter allows for more efficient document management. This modular structure enables independent editing, version control, and reuse of individual chapters without impacting the overall document, making it suitable for large projects with numerous contributors.

Furthermore, Quarto’s support for a variety of output formats and customisations, such as websites, PDFs, and slides, improves the functionality and aesthetics of the outputs.

Future stepsThe configuration of the book simplifies its maintenance. Changes to the _quarto.yml file are required only for significant updates, such as the introduction of a new chapter within Google Sheets. This setup ensures that maintenance of the book remains straightforward and manageable over time.

We’re confident that this change to the underlying infrastructure of the Big Book of R will place the project on a solid foundation, which will ensure that it continues to be one of the best resources for information about R.

To leave a comment for the author, please follow the link and comment on their blog: R - datawookie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Updates to the Big Book of R

View Details

[This article was first published on r-spatial, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.* Introduction * Vector loading + Shapefile (.shp) + GeoPackage (.gpkg) + GeoJSON (.geojson) * Vector saving * Synopsis * Supplement + Zipped shapefile (.shz) + Virtual File Systems + SQL preselection - Columns filtering - Rows filtering - Spatial filtering

[view rawRmd]

Summary:

If you have spatial vector data and are wondering how to load / save itin R, this tutorial is the answer to your questions. It presentspractical examples for the most popular formats using thesf package. We will use freevector layers from Natural Earth asa data source.

IntroductionFor convenience, all necessary files are located in the GitHubrepository:

countries.shp (and related files)
rivers.gpkg
cities.geojson

We can download the mentioned data and interactive notebook (.Rmd)manually from the repository (“Code” button > “Download ZIP”) or usethe following script.

url = "https://github.com/kadyb/sf\_load\_save/archive/refs/heads/main.zip"download.file(url, "sf\_load\_save.zip")unzip("sf\_load\_save.zip") In the first step, we need to download the sf package using theinstall.packages() function, and then use the library() function toload it into the session.

install.packages("sf")library("sf") Vector loadingShapefile (.shp)Let’s start by loading the shapefile format, which actually consists ofseveral files (e.g., .shp, .shx, .dbf, .prj). More information can befound on Wikipedia, butcurrently it is not recommended to use this format due to its manylimitations.

Generally, we can use the read_sf() function to load data. It requiresproviding a path to the file. The file path can be defined in two waysin R and this is the most common source of problems (errors like:Error: Cannot open "file.shp"; The file doesn't seem to exist.).

The first way (easier) is to provide an absolute path, i.e. we mustprovide the exact location where the file is located. For instance:

path = "C:/Users/Krzysztof/Documents/file.shp" However, this isnotthe recommended method, as it makes it impossible to locate files ondifferent operating systems. The second way is to specify a relativepath. In this case, we specify the location of the file relative tothe current working directory (or project). To find out where theworking directory is, we can use the getwd() function, and to changeit the setwd() function. For instance:

getwd()#> "C:/Users/Krzysztof/Documents"path = "file.shp" Let’s load the shapefile using a relative path (all data can be found inthe data folder).

countries = read\_sf("data/countries/countries.shp") We can then print the metadata about this vector layer by referring tothe countries object.

countries## Simple feature collection with 52 features and 168 fields## Geometry type: MULTIPOLYGON## Dimension: XY## Bounding box: xmin: -17.53604 ymin: -34.82195 xmax: 51.41704 ymax: 37.3452## Geodetic CRS: WGS 84## # A tibble: 52 × 169## featurecla scalerank LABELRANK SOVEREIGNT SOV\_A3 ADM0\_DIF LEVEL TYPE TLC ## <chr> <int> <int> <chr> <chr> <int> <int> <chr> <chr>## 1 Admin-0 cou… 0 2 Ethiopia ETH 0 2 Sove… 1 ## 2 Admin-0 cou… 0 3 South Sud… SDS 0 2 Sove… 1 ## 3 Admin-0 cou… 0 6 Somalia SOM 0 2 Sove… 1 ## 4 Admin-0 cou… 0 2 Kenya KEN 0 2 Sove… 1 ## 5 Admin-0 cou… 0 6 Malawi MWI 0 2 Sove… 1 ## 6 Admin-0 cou… 0 3 United Re… TZA 0 2 Sove… 1 ## 7 Admin-0 cou… 0 5 Somaliland SOL 0 2 Sove… 1 ## 8 Admin-0 cou… 0 3 Morocco MAR 0 2 Sove… 1 ## 9 Admin-0 cou… 0 7 Western S… SAH 0 2 Inde… 1 ## 10 Admin-0 cou… 0 4 Republic … COG 0 2 Sove… 1 ## # ℹ 42 more rows## # ℹ 160 more variables: ADMIN <chr>, ADM0\_A3 <chr>, GEOU\_DIF <int>,## # GEOUNIT <chr>, GU\_A3 <chr>, SU\_DIF <int>, SUBUNIT <chr>, SU\_A3 <chr>,## # BRK\_DIFF <int>, NAME <chr>, NAME\_LONG <chr>, BRK\_A3 <chr>, BRK\_NAME <chr>,## # BRK\_GROUP <chr>, ABBREV <chr>, POSTAL <chr>, FORMAL\_EN <chr>,## # FORMAL\_FR <chr>, NAME\_CIAWF <chr>, NOTE\_ADM0 <chr>, NOTE\_BRK <chr>,## # NAME\_SORT <chr>, NAME\_ALT <chr>, MAPCOLOR7 <int>, MAPCOLOR8 <int>, … We can see that this layer consists of 52 features (rows) and 168 fields(columns). The next information is about geometry type, dimension,spatial extent (bounding box) and coordinate reference system (CRS). Inaddition, the first 10 rows were printed.

After loading the data, it is a good idea to present it on a map. Asimple plot() function can be used for this purpose. The countriesobject has many fields (attributes), but to start with we only needgeometry. It can be obtained by using the st_geometry() function.

plot(st\_geometry(countries)) GeoPackage (.gpkg)The next dataset is rivers (linear geometry) saved in GeoPackageformat. It is loaded in exactly the sameway as the shapefile before. Note that this format can consist ofmultiple layers of different types. In this case, we must define whichlayer exactly we want to load. To check what layers are in thegeopackage, use the st_layers() function, and then specify it usingthe layer argument in read_sf(). If the file only contains onelayer, we don’t need to do this.

st\_layers("data/rivers.gpkg")## Driver: GPKG ## Available layers:## layer\_name geometry\_type features fields crs\_name## 1 rivers Multi Line String 228 38 WGS 84rivers = read\_sf("data/rivers.gpkg", layer = "rivers") We can also display metadata as in the previous example.

rivers## Simple feature collection with 228 features and 38 fields## Geometry type: MULTILINESTRING## Dimension: XY## Bounding box: xmin: -16.54233 ymin: -34.34378 xmax: 49.46094 ymax: 35.12311## Geodetic CRS: WGS 84## # A tibble: 228 × 39## dissolve scalerank featurecla name name\_alt rivernum note min\_zoom name\_en## <chr> <int> <chr> <chr> <chr> <int> <chr> <dbl> <chr> ## 1 975River 9 River <NA> <NA> 975 <NA> 7.1 <NA> ## 2 976River 9 River Rung… <NA> 976 <NA> 7.1 Rungwa ## 3 977River 9 River Ligo… <NA> 977 <NA> 7.1 Ligonha## 4 978River 9 River Dong… <NA> 978 <NA> 7.1 Dongwe ## 5 979River 9 River Cuito <NA> 979 <NA> 7.1 Cuito ## 6 980Lake … 9 Lake Cent… <NA> <NA> 980 <NA> 7.1 <NA> ## 7 980River 9 River <NA> <NA> 980 <NA> 7.1 <NA> ## 8 981River 9 River Bagoé <NA> 981 <NA> 7.1 Bagoé ## 9 982River 9 River Hade… <NA> 982 <NA> 7.1 Hadejia## 10 983River 9 River Sous <NA> 983 <NA> 7.1 Sous ## # ℹ 218 more rows## # ℹ 30 more variables: min\_label <dbl>, ne\_id <dbl>, label <chr>,## # wikidataid <chr>, name\_ar <chr>, name\_bn <chr>, name\_de <chr>,## # name\_es <chr>, name\_fr <chr>, name\_el <chr>, name\_hi <chr>, name\_hu <chr>,## # name\_id <chr>, name\_it <chr>, name\_ja <chr>, name\_ko <chr>, name\_nl <chr>,## # name\_pl <chr>, name\_pt <chr>, name\_ru <chr>, name\_sv <chr>, name\_tr <chr>,## # name\_vi <chr>, name\_zh <chr>, name\_fa <chr>, name\_he <chr>, … And make a visualization, but this time we will plot rivers against thebackground of country borders. Adding more layers to the visualizationis done with the add = TRUE argument in plot() function. Note thatthe order in which objects are added is important – the objects addedlast are displayed at the top. The col argument is used to set thecolor of the object.

plot(st\_geometry(countries))plot(st\_geometry(rivers), add = TRUE, col = "blue") GeoJSON (.geojson)The last GeoJSON file contains cities in the world. In this case, wealso use the read_sf() function to load this file.

cities = read\_sf("data/cities.geojson")cities## Simple feature collection with 1287 features and 31 fields## Geometry type: POINT## Dimension: XY## Bounding box: xmin: -17.47508 ymin: -34.52953 xmax: 51.12333 ymax: 37.29042## Geodetic CRS: WGS 84## # A tibble: 1,287 × 32## scalerank natscale labelrank featurecla name namepar namealt nameascii## <int> <int> <int> <chr> <chr> <chr> <chr> <chr> ## 1 10 1 8 Admin-1 capital Bassar <NA> <NA> Bassar ## 2 10 1 8 Admin-1 capital Sotou… <NA> <NA> Sotouboua## 3 10 1 7 Admin-1 capital Meden… <NA> <NA> Medenine ## 4 10 1 7 Admin-1 capital Kebili <NA> <NA> Kebili ## 5 10 1 7 Admin-1 capital Tatao… <NA> <NA> Tataouine## 6 10 1 7 Admin-1 capital L'Ari… <NA> <NA> L'Ariana ## 7 10 1 7 Admin-1 capital Jendo… <NA> <NA> Jendouba ## 8 10 1 7 Admin-1 capital Kasse… <NA> <NA> Kasserine## 9 10 1 7 Admin-1 capital Sdid … <NA> <NA> Sdid Bou…## 10 10 1 7 Admin-1 capital Silia… <NA> <NA> Siliana ## # ℹ 1,277 more rows## # ℹ 24 more variables: adm0cap <int>, capalt <int>, capin <chr>,## # worldcity <int>, megacity <int>, sov0name <chr>, sov\_a3 <chr>,## # adm0name <chr>, adm0\_a3 <chr>, adm1name <chr>, iso\_a2 <chr>, note <chr>,## # latitude <dbl>, longitude <dbl>, pop\_max <int>, pop\_min <int>,## # pop\_other <int>, rank\_max <int>, rank\_min <int>, meganame <chr>,## # ls\_name <chr>, min\_zoom <dbl>, ne\_id <int>, geometry <POINT [°]> In this dataset, there is the featurecla column that indicates thetype of city. So let’s try to print them and then select only statecapitals.

We can print a column (attribute) in two ways, i.e. by specifying thecolumn name in:

Single square brackets – a spatial object will be printed
Double square brackets (alternatively a dollar sign) – only the textwill be printed

cities["featurecla"]## Simple feature collection with 1287 features and 1 field## Geometry type: POINT## Dimension: XY## Bounding box: xmin: -17.47508 ymin: -34.52953 xmax: 51.12333 ymax: 37.29042## Geodetic CRS: WGS 84## # A tibble: 1,287 × 2## featurecla geometry## <chr> <POINT [°]>## 1 Admin-1 capital (0.7890036 9.261)## 2 Admin-1 capital (0.9849965 8.557002)## 3 Admin-1 capital (10.4167 33.4)## 4 Admin-1 capital (8.971003 33.69)## 5 Admin-1 capital (10.4667 33)## 6 Admin-1 capital (10.2 36.86667)## 7 Admin-1 capital (8.749999 36.5)## 8 Admin-1 capital (8.716698 35.2167)## 9 Admin-1 capital (9.500004 35.0167)## 10 Admin-1 capital (9.383302 36.0833)## # ℹ 1,277 more rows# the `head()` function prints only the first 6 elementshead(cities[["featurecla"]])## [1] "Admin-1 capital" "Admin-1 capital" "Admin-1 capital" "Admin-1 capital"## [5] "Admin-1 capital" "Admin-1 capital"# or alternatively# head(cities$featurecla) This layer contains 1287 different cities. To find out what types ofcities these are, we can use the table() function, which willsummarize them.

table(cities[["featurecla"]])## ## Admin-0 capital Admin-0 capital alt Admin-1 capital ## 54 6 609 ## Admin-1 region capital Populated place ## 19 599 We are interested in Admin-0 capital and Admin-0 capital alt typesbecause some countries have two capitals. We make selection as followsusing the | (OR) operator:

sel = cities$featurecla == "Admin-0 capital" | cities$featurecla == "Admin-0 capital alt"head(sel)## [1] FALSE FALSE FALSE FALSE FALSE FALSE As a result of this operation, we got a logical vector with TRUE andFALSE values (if the city is / is not the capital). Now let’s create anew object named capitals, which will contain only capitals.

```

select only those cities that meet the above conditionscapitals = cities[sel, ]capitals["name"]## Simple feature collection with 60 features and 1 field## Geometry type: POINT## Dimension: XY## Bounding box: xmin: -17.47508 ymin: -33.91807 xmax: 47.51468 ymax: 36.80278## Geodetic CRS: WGS 84## # A tibble: 60 × 2## name geometry## ## 1 Lobamba (31.2 -26.46667)## 2 Bir Lehlou (-9.652522 26.11917)## 3 Kigali (30.05859 -1.951644)## 4 Mbabane (31.13333 -26.31665)## 5 Juba (31.58003 4.829975)## 6 Dodoma (35.75 -6.183306)## 7 Laayoune (-13.20001 27.14998)## 8 Djibouti (43.148 11.59501)## 9 Banjul (-16.5917 13.45388)## 10 Porto-Novo (2.616626 6.483311)## # ℹ 50 more rows

`` In the last step, we prepare the final visualization. We can add a title(mainargument), axes (axesargument) and change the backgroundcolor (bgcargument) of the figure. We can also change the pointsymbol (pchargument), set its size (cexargument) and fill color(bg` argument).

plot(st\_geometry(countries), main = "Africa", axes = TRUE, bgc = "deepskyblue", col = "burlywood")plot(st\_geometry(rivers), add = TRUE, col = "blue")plot(st\_geometry(capitals), add = TRUE, pch = 24, bg = "red", cex = 0.8) Vector savingSaving vector data is as easy as loading. There is a dedicatedwrite_sf() function for this purpose and it requires two arguments:

The object we want to save
The path to save with file extension

For example, let’s save our capital object as a GeoPackage (.gpkg),but as an exercise you can save it in other formats as well (you justneed to change the extension).

write\_sf(capitals, "data/capitals.gpkg") SynopsisThe sf package allows loading vector data with the read_sf()function and saving it with the write_sf() function in R. A list ofall supported vector formats can be found on the GDALwebsite.

For more information, see:

sf introductory vignette: Reading, Writing and ConvertingSimple Features
Introduction to sf and stars in Spatial Data Science: WithApplications in R(Pebesma E. & Bivand R., 2023)

SupplementIn the previous part of the tutorial, we looked at simple examples ofloading vector data, while in this section we will check out moreadvanced ways.

Zipped shapefile (.shz)As we noted earlier, a shapefile consists of several files, which can becumbersome. Some solution is to use zipped shapefiles, which is de factoan archive. To create such a file, the extension .shz (or .shp.zip) andthe ESRI Shapefile driver are required. Loading is done in a standardway by specifying the path to the “.shz” file.

write\_sf(capitals, "data/capitals.shz", driver = "ESRI Shapefile") Hooray, only one file on the disk!

Virtual File SystemsGDAL provides some facilities for loading files using some abstractionby Virtual FileSystems. In practice,this means that we can refer directly to the files without firstunpacking or downloading them in R. For example, we can directly openthe shapefile that is in the archive on the website. To do this, we mustuse two prefixes:

/vsicurl/ to download the file
/vsizip/ to unpack the archive

```

URL is file pathurl = "https://raw.githubusercontent.com/OSGeo/gdal/master/autotest/ogr/data/shp/poly.zip"# note that the order of the prefixes is reversef = paste0("/vsizip/", "/vsicurl/", url)read_sf(f)## Simple feature collection with 10 features and 3 fields## Geometry type: POLYGON## Dimension: XY## Bounding box: xmin: 478315.5 ymin: 4762880 xmax: 481645.3 ymax: 4765610## Projected CRS: OSGB36 / British National Grid## # A tibble: 10 × 4## AREA EAS_ID PRFEDEA geometry## ## 1 215229. 168 35043411 ((479819.8 4765180, 479690.2 4765260, 479647 476537…## 2 247328. 179 35043423 ((480035.3 4765558, 480039 4765540, 479730.4 476540…## 3 261753. 171 35043414 ((479819.8 4765180, 479859.9 4765270, 479909.9 4765…## 4 547597. 173 35043416 ((479014.9 4765148, 479029.7 4765110, 479117.8 4764…## 5 15776. 172 35043415 ((479029.7 4765110, 479046.5 4765117, 479123.3 4765…## 6 101430. 169 35043412 ((480083 4765050, 480080.3 4764980, 480134 4764856,…## 7 268598. 166 35043409 ((480389.7 4764950, 480537.2 4765014, 480568 476491…## 8 1634833. 158 35043369 ((480701.1 4764738, 480761.5 4764778, 480825 476482…## 9 596610. 165 35043408 ((479750.7 4764702, 479968.5 4764788, 479985.1 4764…## 10 5269. 170 35043413 ((479750.7 4764702, 479658.6 4764670, 479640.1 4764…

``` SQL preselectionWe can use SQL queries topre-filter features, so only selected objects / attributes will beloaded. This allows us to limit the size of the object in memory andspeed up the operation time. Moreover, we can also make spatialselection, i.e. limit the loading of data only to a selected area.

Columns filteringThe query argument in the read_sf() function is used to pass SQLqueries. Let’s go back to the countries dataset and load only thecolumn with the names of countries (NAME_LONG).

sql = "SELECT NAME\_LONG FROM countries"f = "data/countries/countries.shp"read\_sf(f, query = sql)## Simple feature collection with 52 features and 1 field## Geometry type: MULTIPOLYGON## Dimension: XY## Bounding box: xmin: -17.53604 ymin: -34.82195 xmax: 51.41704 ymax: 37.3452## Geodetic CRS: WGS 84## # A tibble: 52 × 2## NAME\_LONG geometry## <chr> <MULTIPOLYGON [°]>## 1 Ethiopia (((34.0707 9.454592, 34.06689 9.531176, 34.09821 9.679…## 2 South Sudan (((35.92084 4.619332, 35.85654 4.619603, 35.78122 4.61…## 3 Somalia (((46.46696 6.538292, 46.48805 6.558645, 46.50841 6.57…## 4 Kenya (((35.70585 4.619447, 35.70594 4.619962, 35.71152 4.66…## 5 Malawi (((34.96461 -11.57356, 34.65125 -11.57004, 34.61673 -1…## 6 Tanzania (((32.92086 -9.4079, 32.90546 -9.398185, 32.83074 -9.3…## 7 Somaliland (((48.93911 11.24913, 48.93911 11.13674, 48.93911 11.0…## 8 Morocco (((-8.817035 27.66146, -8.818449 27.6594, -8.81292 27.…## 9 Western Sahara (((-8.817035 27.66146, -8.816537 27.66147, -8.752562 2…## 10 Republic of the Congo (((18.62639 3.476869, 18.63455 3.449222, 18.64241 3.32…## # ℹ 42 more rows Rows filteringWe can also select rows using a condition, e.g. population (POP_EST)greater than 25 million.

sql = "SELECT * FROM countries WHERE POP\_EST > 25000000"f = "data/countries/countries.shp"read\_sf(f, query = sql) # 17 countries## Simple feature collection with 17 features and 168 fields## Geometry type: MULTIPOLYGON## Dimension: XY## Bounding box: xmin: -17.01374 ymin: -34.82195 xmax: 50.50392 ymax: 37.09394## Geodetic CRS: WGS 84## # A tibble: 17 × 169## featurecla scalerank LABELRANK SOVEREIGNT SOV\_A3 ADM0\_DIF LEVEL TYPE TLC ## <chr> <int> <int> <chr> <chr> <int> <int> <chr> <chr>## 1 Admin-0 cou… 0 2 Ethiopia ETH 0 2 Sove… 1 ## 2 Admin-0 cou… 0 2 Kenya KEN 0 2 Sove… 1 ## 3 Admin-0 cou… 0 3 United Re… TZA 0 2 Sove… 1 ## 4 Admin-0 cou… 0 3 Morocco MAR 0 2 Sove… 1 ## 5 Admin-0 cou… 0 2 Democrati… COD 0 2 Sove… 1 ## 6 Admin-0 cou… 0 2 South Afr… ZAF 0 2 Sove… 1 ## 7 Admin-0 cou… 0 3 Sudan SDN 0 2 Sove… 1 ## 8 Admin-0 cou… 0 3 Ivory Coa… CIV 0 2 Sove… 1 ## 9 Admin-0 cou… 0 2 Nigeria NGA 0 2 Sove… 1 ## 10 Admin-0 cou… 0 3 Angola AGO 0 2 Sove… 1 ## 11 Admin-0 cou… 0 3 Algeria DZA 0 2 Sove… 1 ## 12 Admin-0 cou… 0 3 Mozambique MOZ 0 2 Sove… 1 ## 13 Admin-0 cou… 0 3 Uganda UGA 0 2 Sove… 1 ## 14 Admin-0 cou… 0 3 Cameroon CMR 0 2 Sove… 1 ## 15 Admin-0 cou… 0 3 Ghana GHA 0 2 Sove… 1 ## 16 Admin-0 cou… 0 2 Egypt EGY 0 2 Sove… 1 ## 17 Admin-0 cou… 0 3 Madagascar MDG 0 2 Sove… 1 ## # ℹ 160 more variables: ADMIN <chr>, ADM0\_A3 <chr>, GEOU\_DIF <int>,## # GEOUNIT <chr>, GU\_A3 <chr>, SU\_DIF <int>, SUBUNIT <chr>, SU\_A3 <chr>,## # BRK\_DIFF <int>, NAME <chr>, NAME\_LONG <chr>, BRK\_A3 <chr>, BRK\_NAME <chr>,## # BRK\_GROUP <chr>, ABBREV <chr>, POSTAL <chr>, FORMAL\_EN <chr>,## # FORMAL\_FR <chr>, NAME\_CIAWF <chr>, NOTE\_ADM0 <chr>, NOTE\_BRK <chr>,## # NAME\_SORT <chr>, NAME\_ALT <chr>, MAPCOLOR7 <int>, MAPCOLOR8 <int>,## # MAPCOLOR9 <int>, MAPCOLOR13 <int>, POP\_EST <dbl>, POP\_RANK <int>, … Spatial filteringFinally, to perform spatial filtering, we must first define the spatialextent / bounding box (st_bbox() function) and specify its coordinatereference system (CRS). Then the bounding box needs to be converted intoa polygon using the st_as_sfc() function and finally converted to aWell-KnowTextrepresentation using st_as_text() function. Therefore, prepared textis passed to the wkt_filter argument. Follow the example below ofloading rivers only in southern Africa:

bbox = st\_bbox(c(xmin = 10, xmax = 40, ymax = -35, ymin = -20), crs = st\_crs(4326))bbox = st\_as\_text(st\_as\_sfc(bbox))bbox## [1] "POLYGON ((10 -20, 40 -20, 40 -35, 10 -35, 10 -20))"f = "data/rivers.gpkg"rivers\_south = read\_sf(f, wkt\_filter = bbox)plot(st\_geometry(rivers\_south), axes = TRUE) To leave a comment for the author, please follow the link and comment on their blog: r-spatial.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to load and save vector data in R

View Details

[This article was first published on Epiverse-TRACE developer space, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. I have recently published a series of blog posts on the reasons why one may want to start using object-oriented programming (and more specifically R S3 classes) to improve interoperability with other tools from the ecosystem.

But there are still questions I have not addressed directly, even if they may have been implicitly included sometimes: what makes a good object class? What good practices in class & function design can improve interoperability?

As you can expect from these questions, this post will present a subjective view on S3 class and method design. I will argue that it is often a good strategy to inherit from existing standards classes, and to leverage this inheritance relationship as much as possible.

Inherit from standard classesA unique feature of R is the availability and centrality of data.frames in the base language, whereas you need extra libraries for a similar functionality in most other languages (e.g., pandas in Python).

data.frame is one of the first “complex” (in the sense of non-atomic) object most R learners will be exposed to and will develop a familiarity with. A good way to leverage this familiarity is to make your subclass a thin wrapper around data.frames.

This means that not only will users be able to get started with your package faster because of this familiarity, but you will also immediately benefit from the huge ecosystem of functions and packages working on data.frames, such as the tidyverse. If you want some examples, this is what collaborators and I did in the linelist, pavo, scoringutils, epichains, and vaccineff R packages.

In some cases, the output is too complex to fit into a data.frame. Even in this situation, I would recommend inheriting from existing, well-established, classes for the same two reasons: familiarity and ecosystem. For example, for the serofoi R package, we have made the decision to inherit from stanfit objects, rather than a custom structure.

Rely on parent methods as much as possibleA follow up recommendation from inheriting from standard classes is to leverage their methods wherever possible.

One of the first changes I made when becoming maintainer of the linelist package was to remove the rename.linelist() and select.linelist() methods. Indeed, they were, or could easily be, behaving identically as the parent rename.data.frame() and select.data.frame() methods. Rather than burdening the codebase and maintenance with an extra unnecessary method, it is much simpler and more robust to rely on the well-tested parent method. In fact, the dplyr documentation explicitly recommends only writing methods for a couple of standard functions (including [.subclass() and names<-.subclass()), which will enable the use of parent methods directly, rather than writing custom methods for each dplyr function.

Similarly, many developers have the reflex to write a custom print.subclass() method as part of the method implementation. While it may be justified in some cases, it is sometimes unnecessary. My recommendation would be to evaluate carefully what benefits the custom method brings over the default parent method.

Enable conversion to standard classesIf after careful consideration, extra metadata makes it too difficult to fit your new class into an existing class, you may sometimes have to define your own class from “scratch” (i.e., often list() in R).

But even in this case, you can still apply some of the ideas proposed earlier. As much as possible, you should provide helpers or methods to enable the streamlined conversion of your method to a standard class.

A good example here is the epiparameter package, which provides a complex S3 class built on lists, including extensive metadata about probability distribution of epidemiological parameters. As such, this custom class cannot be used out of the box in most functions from other packages. But an as.function() method is conveniently provided to enable the conversion of this probability distribution parameters into a density distribution, which can then be used in functions which expect a function object.

ConclusionIn summary, I recommend relying on well-established parent classes such as data.frames or at least providing direct conversion functions to these standard classes, and using parent methods wherever possible rather than writing custom dedicated methods. This should help produce a package:

more easily accessible for users because it uses objects that feel familiar
more maintainable because a lot of method writing is offloaded to the parent class
more likely to be interoperable because standard classes are a good way to pass data between functions or packages

Thanks to Chris Hartgerink, James Azam and Josh Lambert, for their very valuable feedback on this post.

ReuseCC BY 4.0CitationBibTeX citation: @online{gruson2024, author = {Gruson, Hugo}, title = {Choosing the {Right} {Parent} for {R} {Object} {Classes}}, date = {2024-06-26}, url = {https://epiverse-trace.github.io/posts/parent-class/}, doi = {10.59350/fk6nv-1k973}, langid = {en}} For attribution, please cite this work as:Gruson, Hugo. 2024. “Choosing the Right Parent for R ObjectClasses.” June 26, 2024. https://doi.org/10.59350/fk6nv-1k973. To leave a comment for the author, please follow the link and comment on their blog: Epiverse-TRACE developer space.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Choosing the Right Parent for R Object Classes

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Vincent Guyader, organizer of the R Addicts Paris and president of ThinkR, recently updated the R Consortium on the group’s activities. Last year, Vincent discussed the application of R in developing solutions for industrial problems. He emphasized the importance of helping people become fluent in R and leveraging the language to add value to their work. ThinkR is dedicated to enhancing R proficiency in various industries. The R Addicts Paris, one of France’s oldest and largest R user groups with 1,800 members, continues to foster a strong R community under Vincent’s leadership.

Please share your background and involvement with the RUGS group.

My name is Vincent, and I have been using R since my student days. During my studies, I took on freelance R projects for various companies. Currently, I head a company called ThinkR, where we have a team of over 10 experts specializing in everything related to R. Our services include training, consulting, developing Shiny applications, creating R packages, and more. We also collaborate with Posit and handle hardware installations for clients, primarily in France but also in Switzerland, Belgium, and other parts of Europe.

Since 2018, I have been managing the R user group in Paris, known as the R Addicts Paris. It’s one of the oldest and possibly the largest R user groups, with 1,800 members. I aimed to organize meetups every three months, but the next one has been delayed due to internal organizational issues. I genuinely enjoy helping people become fluent in R and use the language to add value to their work.

What challenges do you face in organizing the R Addict Paris group and how do you overcome those challenges?

One of the main challenges is that our users are not professional programmers or developers; they are specialists in fields like biology and finance. They have to shift their mindset to use programming languages. My daily job involves helping these individuals embrace software development. Coming from a genetics and biochemistry background, I understand how challenging this can be for non-developers. However, I love doing this, and I have a dedicated, competent team to assist.

Based on your work with ThinkR, which industries in France do you see using R?

We have clients in various fields across France, including finance, retail, and research. The health sector is particularly prominent. For instance, a company that used SAS a few years ago now uses R & Python. About half of our clients currently use Python. While we provide Python installation on hardware, we don’t offer Python training yet.

We are committed to being the sole organization in France that can certify R users and developers. The French government has authorized us to issue an official certification akin to a diploma. Our goal is to elevate R proficiency across various fields in France. Our clients include businesses and individuals, with many investing their resources to learn proper software and programming skills.

Do you host online or in-person events?

I chose not to host online events. It’s a very opinionated choice because most meetups switched to online formats during the pandemic. At ThinkR, we are a fully remote company, and I spend my day on Zoom. While remote training is effective, I’ve found that in-person events work better for our user group.

One of the main challenges we face as a group is finding female speakers. I try to avoid having only male speakers, but I only get female speakers every fifth or sixth event, which is not enough. I encourage other R user group organizers to recognize our power to give a voice to different kinds of people. I push myself to include more female speakers. Sometimes, I encounter highly qualified women who hesitate to speak, while less experienced men are more willing. It’s challenging, but I strive to maintain a balanced representation.

I consciously avoid engaging with speakers who lack substance, ensuring I have time to encourage qualified women to share their knowledge. Despite my efforts, female representation remains below 20%. A few years ago, my colleague Diane and I tried to connect with the R-Ladies Paris group. Many men are actively engaged there, and I wonder why that is.

There are many skilled women in the R community, which includes biologists and geneticists. There’s no excuse for the lack of female representation. We must remember our influence and endorse individuals who truly represent our values.

What are some trending topics in R in your R User Group?

I’ve noticed a decline in interest in statistics over the past two to five years. During meetups, we rarely discuss statistics. The machine learning and AI fields aren’t well-represented in R, possibly because most people in these fields use Python. It could also be due to regional differences or my network.

You had a Meetup “Raddicts x RTE – {webr} – Shinyproxy and return of the Reconteres 2024” on 19th June, can you share more on the topic covered? Why this topic?

For this event, we had two male speakers. Colin Fay discussed {webr}, a new JavaScript capability for launching user insights in the browser. This is powerful for deploying Shiny applications. Valentin Cadoret talked about new Shinyproxy functionalities, and tools that enhance the deployment of Shiny applications. So we focused heavily on Shiny once again.

Learn moreThe post R Addicts Paris: Promoting Diversity in R appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Addicts Paris: Promoting Diversity in R

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#193–194

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #193Summary again? Yes, but this time not as usual. We have sales and bonus per salesperson per quarter, and as result we also need total per each of those partitions and cummulative summary down the table. Looks simple and if we would have only those things to do that I pointed before it would be really easy. But we have merged two-level headers and that is tricky to manage. We are lucky that there is unpivotr package for this purpose. If you’re going to check this code, focus on first 4 steps of pipe to know how we are helped by unpivotr.

Load libraries and data library(tidyverse)library(readxl)library(unpivotr)path = "Power Query/PQ\_Challenge\_193.xlsx"input = read\_xlsx(path, range = "A1:I6", col\_names = FALSE)test = read\_xlsx(path, range = "A12:F24") Transformation result = input %>% as\_cells() %>% behead("up-left", "Quarter") %>% behead("up", "Category") %>% behead("left", "Persons") %>% select(Persons, Quarter, Category, chr) %>% pivot\_wider(names\_from = Category, values\_from = chr) %>% mutate(across(c(Sales, Bonus), as.numeric), Total = Sales + Bonus) %>% pivot\_longer(cols = Sales:Total, names\_to = "Category", values\_to = "Value") %>% pivot\_wider(names\_from = Quarter, values\_from = Value) %>% mutate(across(c(Q1:Q4), cumsum), .by = Category) %>% mutate(Persons = accumulate(Persons, ~ paste(.x, .y, sep = ", "))[match(Persons, unique(Persons))], .by = Category) %>% mutate(Persons = ifelse(Category == "Sales", Persons, NA\_character\_)) Validation identical(result, test) #> [1] TRUE Puzzle #194Our table looks like we have some kind of sales summary, but done by noting total in register everyday at 3 time points a day, without withdrawing it. But we are weird and we want to know something else. How much this salesperson earned between each check. So we have to “unsummarize” this sequence, which goes like Z along this table. It is pretty easy. Three main procedures: pivot_longer, mutate with lag and pivot_wider. Check it out.

Loading libraries and data library(tidyverse)library(readxl)path = "Power Query/PQ Challenge\_194.xlsx"input = read\_xlsx(path, range = "A1:D10")test = read\_xlsx(path, range = "F1:I10") Transformation result = input %>% pivot\_longer(cols = -c(1), names\_to = "Amt", values\_to = "Value") %>% mutate(val = lag(Value, default = 0), diff = Value - val) %>% select(-c(Value, val)) %>% pivot\_wider(names\_from = Amt, values\_from = diff) Validation identical(result, test) # [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on RStudioDataLab, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The f-test in R is a powerful tool for comparing variances and drawing significant conclusions from your data. Understanding how to perform an F-test can transform your data analysis capabilities, allowing you to determine whether the variances in two samples are statistically different. In this comprehensive guide, we’ll explore the f-test in R, from setting up your environment to interpreting To leave a comment for the author, please follow the link and comment on their blog: RStudioDataLab.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to do F-test in R | Compare variances in Rstudio

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.PyShiny (Shiny for Python) represents a significant advancement in the field of data dashboarding, setting new standards for design, maintainability, and scalability. This framework is not just an extension of R Shiny to Python but a refined approach that encapsulates the best practices learned from years of dashboard development:

Modular Design: PyShiny’s architecture promotes clean, maintainable code through its emphasis on modularity, akin to principles seen in functional programming.
Scalability and Maintainability: Unlike traditional methods in R Shiny, PyShiny enforces scalable and maintainable solutions, making it easier to manage dynamic content like multiple plots.
Lessons from R Shiny: Building on R Shiny’s legacy, PyShiny addresses past limitations by fostering best practices in software design from the outset.
Future-Proof Technology: PyShiny is not just a contemporary solution but a future-forward framework, designed to adapt to the evolving demands of data visualization.

This article encapsulates the essence of PyShiny as not just a tool, but a beacon for the future of data dashboarding, merging quality, innovation, and sustainability. For those looking to harness the power of this advanced framework, Appsilon stands ready to provide expert guidance and support.

Didn’t know Shiny is available for Python? Make sure to read our introduction article first.

PyShiny: A Brief Primer In Shiny For PythonIn the evolving landscape of Python-based dashboarding tools, PyShiny emerges as a noteworthy contender, especially for those familiar with the R Shiny framework. Developed by the team at Posit (formerly RStudio), PyShiny aims to bring the interactive, web application framework capabilities of R Shiny to the Python ecosystem, catering to a broad spectrum of data science and analytics needs.

Orbit SimulationFrameworks like Streamlit and Dash have carved their niches within the Python community, PyShiny distinguishes itself through its unique approach to application structure and reactivity.

‍Streamlit, known for its simplicity and ease of use, is ideal for quickly spinning up data applications. However, its model of re-running the entire script upon any input change poses scalability issues for more complex applications. This reactivity model, or rather the lack thereof, limits Streamlit’s applicability for larger, more interactive applications where efficiency and control over execution flow are paramount.

Dash, on the other hand, offers extensive customization options and is a powerful tool for building sophisticated web applications. Despite its strengths, Dash’s manual trigger system for updates can introduce additional complexity in managing the reactivity and interactivity of components, making the development process more cumbersome for applications requiring high levels of interactivity.

PyShiny aims to bridge these gaps by providing a framework that supports scalable, interactive web applications with a structured approach to reactivity and modularization. It encourages clean design and maintainable code, principles that are sometimes more challenging to adhere to in Streamlit and Dash.

For those familiar with R Shiny, transitioning to PyShiny can be intuitive, yet it’s important to appreciate the nuanced differences and enhancements PyShiny offers. A practical tutorial by Winston Chang (one of the core developers behind Py and RShiny), which guides developers through translating R Shiny applications to PyShiny, serves as an excellent resource for those looking to dive deeper into PyShiny’s capabilities.

As we go further into PyShiny’s approach to dynamic plot management, it’s essential to keep in mind the broader context of Python dashboarding frameworks. PyShiny, with its emphasis on modularity and maintainability, offers an appealing path for developers seeking to build or transition to scalable, interactive web applications.

We recently released Tapyr, a framework for deployment ready PyShiny applications. Learn more about it in this blog post.

The Challenge: Managing Dynamic Content In PyShiny DashboardsAt Appsilon, we hold software craftsmanship in high regard, understanding deeply that the maintainability of code is not just a preference but a critical necessity in the long-term success of any project. A common hurdle we often face in dashboard development is the effective management of dynamic content, such as displaying a variable number of plots or images based on user input.

This task tests not only the flexibility and scalability of a dashboarding framework but also its ability to uphold our standards for clean, manageable codebases. As user interactions within applications grow more complex, it becomes imperative to ensure that the application remains responsive and intuitive, without compromising on performance.

This challenge is a testament to our commitment to finding and utilizing frameworks like PyShiny, which allow us to elegantly handle dynamic elements, seamlessly updating the user interface in real time to reflect changes, all while keeping the underlying code structured and in line with our principles of software excellence.

Interested in steps Appsilon takes to deliver impactful projects? Here’s a blog post on what it’s like working with us.

R Shiny’s Approach: A Case StudyLet’s examine a typical R Shiny example by Winston Chang first. The code snippet demonstrates a Shiny application featuring a sidebar with a slider input to control the number of plots displayed in the main panel.

max\_plots <- 5ui <- fluidPage( headerPanel("Dynamic number of plots"), sidebarPanel( sliderInput("n\_plots", "Number of plots", value=1, min=1, max=max\_plots) ), mainPanel( # This is the dynamic UI for the plots uiOutput("plots") ))server <- function(input, output) { # Insert the right number of plot output objects into the web page output$plots <- renderUI({ plot\_output\_list <- lapply(1:input$n\_plots, function(i) { plotname <- paste("plot", i, sep="") plotOutput(plotname) }) # Convert the list to a tagList - this is necessary for the list of items # to display properly. do.call(tagList, plot\_output\_list) }) # Call renderPlot for each one. Plots are only actually generated when they # are visible on the web page. for (i in 1:max\_plots) { # Need local so that each item gets its own number. Without it, the value # of i in the renderPlot() will be the same across all instances, because # of when the expression is evaluated. local({ my\_i <- i plotname <- paste("plot", my\_i, sep="") output[[plotname]] <- renderPlot({ plot(1:my\_i, 1:my\_i, xlim = c(1, max\_plots), ylim = c(1, max\_plots), main = paste("1:", my\_i, ". n is ", input$n\_plots, sep = "") ) }) }) }}shinyApp(ui, server) This approach hinges on dynamically modifying the output object within the server function to render the appropriate number of plot outputs. The uiOutput in the main panel doesn’t refer to a single plot but to an entire section of the UI designated for these plots. As the user adjusts the slider, the server function responds by generating a list of plot output objects (plotOutput) with uniquely constructed names, which are then rendered and displayed.

While this method is straightforward and widely used, it has limitations in scalability and maintainability. Each plot is essentially added manually by crafting a unique string name and binding it to a renderPlot call within a loop. This manual handling of UI elements can become cumbersome and error-prone as the complexity of the application grows.

Moreover, although R Shiny offers modules as a means to encapsulate and reuse UI and server logic, their use isn’t mandatory. Many developers opt for the direct manipulation of the output object, as seen in this example, due to its simplicity and immediate results.

This short-term convenience usually leads to long-term scalability challenges, as the application’s structure becomes more difficult to manage and extend. This case study underscores the importance of considering not just the immediate ease of implementation but also the future implications on the codebase’s maintainability and adaptability.

Learn more about Shiny for Python – From an R Shiny Developer point of view.

PyShiny’s Elegant SolutionIn contrast to the R Shiny example, Shiny for Python offers a structured and modular approach to managing dynamic content, such as a variable number of plots. PyShiny utilizes the concept of modules extensively, which is a core part of its design philosophy, promoting reusability and maintainability.

import matplotlib.pyplot as pltimport numpy as npfrom shiny import ui, render, req, App, moduleMAX\_PLOTS = 5# UI module for a single plot@module.uidef plot\_ui(): # Returns a placeholder for a plot that will be rendered server-side return ui.output\_plot("plot")# Server module to generate and render each plot@module.serverdef plot\_server(input, output, session, plot\_id: int, max\_plots: int = MAX\_PLOTS): @render.plot def plot(): # Generate data for the plot based on the plot\_id x = np.r\_[1 : (plot\_id + 1)] y = np.r\_[1 : (plot\_id + 1)] fig, ax = plt.subplots() ax.scatter(x, y) ax.set\_xlim(0, max\_plots) ax.set\_ylim(0, max\_plots) ax.set\_title(f"1: {plot\_id}. n is {max\_plots}") return figapp\_ui = ui.page\_fluid( ui.panel\_title("Dynamic number of plots"), ui.layout\_sidebar( ui.sidebar( ui.input\_slider( "n\_plots", "Number of plots", value=1, min=1, max=MAX\_PLOTS ), ), # Main panel to display all plots ui.output\_ui("plots"), ),)def server(input, output, session): # Function to dynamically render UI components (plots) @render.ui def plots(): # Ensure value is an integer and available n\_plots = int(req(input.n\_plots())) # Dynamically create server-side functions for each plot for i in range(1, n\_plots + 1): plot\_server(f"plot\_{i}", plot\_id=i, max\_plots=MAX\_PLOTS) # Dynamically create UI components for each plot # Note that the names must match the server-side functions # But they're close to each other in code return ([plot\_ui(f"plot\_{i}") for i in range(1, n\_plots + 1)],)app = App(app\_ui, server) The provided Python code showcases a PyShiny application that dynamically renders a specified number of plots based on user input, similar to the R Shiny case study. However, the implementation leverages PyShiny’s modules to encapsulate both the UI and server logic for individual plots, demonstrating a cleaner and more scalable solution.

Each plot is represented by a UI module (plot_ui) that defines a placeholder for the plot, and a corresponding server module (plot_server) that contains the logic to generate and render the plot based on a unique plot_id. This separation of concerns allows for each plot component to be self-contained, enhancing code readability and making the application easier to extend and maintain.

The main application (app_ui and server) dynamically constructs the UI components and server logic for the number of plots specified by the user. This dynamic generation of content is handled elegantly within the PyShiny framework, without the need for manually manipulating output names or directly modifying the output object. Instead, the modular structure allows for a clear definition of how each plot should be rendered and where it should be displayed within the application.

This example highlights the framework’s alignment with good software design practices. The modular approach inherently encourages developers to write cleaner, more organized code, which is crucial for building scalable and maintainable applications.

For a deeper dive into the workings of modules in PyShiny, the documentation on shiny modules provides comprehensive insights and examples, further demonstrating the framework’s capabilities and advantages over traditional methods.

By adopting PyShiny’s modular design, developers can achieve a high degree of interactivity and dynamism in their applications, all while maintaining a clean and manageable codebase.

The Design Advantage: Architectural Benefits Of Shiny For PythonPyShiny represents a significant stride towards maturity in the dashboarding domain, epitomizing the lessons learned from its R counterpart. It’s not about reinventing the wheel but rather refining it to roll more smoothly. The framework is built on the foundation of promoting best practices and making it inherently challenging to deviate towards less maintainable code structures.

One of the core principles PyShiny advocates for is modularity. This approach is akin to the principles of functional programming, where the emphasis is on immutability and pure functions. Just as functional programming encourages developers to avoid mutating values to enhance predictability and reduce side effects, PyShiny’s modular design discourages direct manipulation of global state and ad-hoc UI modifications. This constraint, far from being a limitation, actually serves to streamline the development process, ensuring that applications are built with scalability and maintainability in mind from the outset.

Drawing parallels to functional programming, as discussed in our blog post on functional programming in R, the discipline imposed by such paradigms often leads to more robust, testable, and reliable code. In the same vein, PyShiny’s architectural design nudges developers towards cleaner code by structuring applications into discrete, reusable modules. Each module encapsulates a specific functionality, reducing interdependencies and making the codebase more manageable.

Moreover, PyShiny’s emphasis on modularity aligns with the principles of good software design, where each component has a well-defined responsibility. This separation of concerns not only aids in debugging and testing but also facilitates collaboration among developers, as modules can be developed and tested in isolation before being integrated into larger applications.

In essence, PyShiny’s architectural benefits are all about embedding a culture of software craftsmanship. By making it more challenging to write bad code, PyShiny ensures that developers are guided towards practices that will stand the test of time, much like the enduring principles of functional programming have shown.

Wrapping Up: Why PyShiny Represents The FutureIt’s clear as day that PyShiny’s emphasis on modularity, maintainability, and adherence to robust software design principles positions it not just as a contemporary solution, but as a forward-looking framework that’s set to shape the future of dashboard development.

PyShiny’s approach resonates with the ongoing shift in the tech industry towards sustainable and scalable software practices. By incorporating lessons from the past and aligning with modern programming paradigms, PyShiny is poised to be a key player in the realm of interactive web applications.

For those looking to leverage this cutting-edge technology, our team at Appsilon offers comprehensive services to meet a wide range of needs. Whether it’s rapid dashboard development, full-stack engineering support (from setting up a Shiny server to UX optimization), or DevOps advisory for Posit products, we have the expertise and experience to help bring your projects to fruition. Our commitment to quality and innovation ensures that we’re not just developers but partners in your journey towards leveraging the best of what technology has to offer.

If you’re interested in exploring how PyShiny can transform your data visualization and dashboarding efforts, or if you need expert assistance in any aspect of dashboard development, don’t hesitate to reach out to us at Appsilon. Together, we can create solutions that are not only efficient and effective but also future-proof, embodying the best practices that PyShiny champions.

Ready to deploy your Shiny for Python applications? Explore our Tapyr framework that helps you do this seamlessly.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Beyond R Shiny: PyShiny’s Clean Design For Dynamic Plot Management

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. How to Extract Strings Between Specific Characters in RHello, R enthusiasts! Today, we’re jumping into a common text processing task: extracting strings between specific characters. This is a great skill for data cleaning and manipulation, especially when working with raw text data. I’m going to show you how to achieve this using base R, the stringr package, and the stringi package. Let’s go!

Extracting Strings Using Base RBase R provides several ways to extract substrings, including sub and gregexpr. Here, we’ll use sub and gsub for some examples.

Example 1: Base R with subSuppose you have a string and you want to extract the text between two characters, say [ and ].

```

Sample stringtext <- "Extract this [text] from the string."# Using sub to extract text between square bracketsresult <- sub(".\[(.?)\].*", "\1", text)# Print the resultprint(result)

```

[1] "text" Example 2: Base R with gsubNow, let’s extract text between parentheses ( and ).

```

Example stringtext2 <- "This is a sample (extract this part) string."# Extract string between parentheses using base Rextracted_base <- gsub(".$(.)$.*", "\1", text2)print(extracted_base)

```

[1] "extract this part" In these examples, sub and gsub use regular expressions to find the text between the specified characters and replace the entire string with the extracted part. The pattern .*\\[(.*?)\\].* and .*\$(.*)\$.* break down as follows: - .* matches any character (except for line terminators) zero or more times. - \\[ matches the literal [ and \$ matches the literal (. - (.*?) and (.*) are non-greedy matches for any character (.) zero or more times. - \\] matches the literal ] and \$ matches the literal ). - \\1 in the replacement string refers to the first capture group, i.e., the text between [ ] and ( ).

Extracting Strings Using stringrThe stringr package, part of the tidyverse, makes string manipulation more straightforward with consistent functions.

Example 1: Using stringr::str_extract ```

Load the stringr packagelibrary(stringr)# Using str_extract to extract text between square bracketsresult_str_extract <- str_extract(text, "(?<=\[).*?(?=\])")# Print the resultprint(result_str_extract)

```

[1] "text" Example 2: Using stringr to extract text between parentheses ```

Example using stringrextracted_str <- str_extract(text2, "$.*?$")extracted_str <- str_sub(extracted_str, 2, -2)print(extracted_str)

```

[1] "extract this part" The str_extract function extracts the first substring matching a regex pattern. Here, (?<=\\[).*?(?=\\]) and \$.*?\$ use lookbehind (?<=\\[) and lookahead (?=\\]) assertions to match text between [ and ], and simple matching for text between ( and ). str_sub is then used to remove the enclosing parentheses.

Extracting Strings Using stringiThe stringi package provides robust and efficient tools for string manipulation.

Example 1: Using stringi::stri_extract ```

Load the stringi packagelibrary(stringi)# Using stri_extract to extract text between square bracketsresult_stri_extract <- stri_extract(text, regex = "(?<=\[).*?(?=\])")# Print the resultprint(result_stri_extract)

```

[1] "text" Example 2: Using stringi to extract text between parentheses ```

Example using stringiextracted_stri <- stringi::stri_extract_first_regex(text2, "$.*?$")extracted_stri <- stringi::stri_sub(extracted_stri, 2, -2)print(extracted_stri)

```

[1] "extract this part" The stri_extract function from stringi works similarly to str_extract, utilizing regex patterns for text extraction. It’s highly optimized for performance, especially with large datasets. stri_sub is used to remove the enclosing parentheses.

Your Turn!Experimenting with these functions and patterns on your own datasets will help you understand their nuances. Here are a few additional exercises to solidify your understanding:

Extract text between parentheses ( and ).
Extract text between the first and last occurrences of a specific character in a string.
Extract all occurrences of text between two characters in a string.

Feel free to use the examples provided as a template for your own tasks.

Happy coding!

Bonus: Combining MethodsFor more complex scenarios, you might need to combine different methods. Here’s a quick example of how you can handle multiple extractions.

```

Sample string with multiple patternstext_multiple <- "Here is [text1] and here is (text2)."# Using gregexpr and regmatches to extract all matchesmatches <- regmatches( text_multiple, gregexpr("(?<=\[).?(?=\])|(?<=$).?(?=$)", text_multiple, perl = TRUE) )# Print the matchesprint(unlist(matches))

```

[1] "text1" "text2" This example uses gregexpr to find all matches and regmatches to extract them.

Until next time, keep exploring and enjoying the power of R!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Steve's Data Tips and Tricks 2024-06-24 22:00:00

View Details

[This article was first published on Achim Zeileis, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Detailed probabilistic forecast for the match Netherlands vs. Austria at UEFA Euro 2024 in Group D, accompanying a conference presentation at Imagine 2024.

Machine learning ensembleIn a recent blog post, prior to the start of the tournament, probabilistic forecasts for the UEFA Euro 2024 were provided based on a machine learning approach. In short, the approach obtained a number of highly informative inputs about the 24 participating teams before the start of the tournament: Historic match abilities from all national matches in 8 years, bookmaker consensus abilities based on quoted odds from 28 bookmakers, average player ratings from goal contributions of individual players in club and national matches, as well as further team-specific information like market value or FIFA rank etc. Then an ensemble of a random forest, a lasso, and an XGBoost learner were trained on matches from the UEFA Euro 2004–2020. The outcome was a prediction for the mean goals for both teams in all potential matches at the UEFA Euro 2024. Based on these predictions the entire tournament was simulated 100,000 times yielding probabilities for all possible outcomes of the tournament.

Match forecastThe prediction from the machine learning ensemble above for the match Netherlands vs. Austria is summarized in the following table.

| Mean goals | Win probability | | --- | --- | | | 1.3 | 48.6% | | Draw | – | 28.1% | | | 0.8 | 23.4% |

This means that if the Netherlands were to play Austria in lots of matches, the Netherlands are predicted to score 1.3 goals on average in these matches while Austria scores an average of 0.8 goals. Assuming a certain probability distribution for the goals per team in each match, not only the mean goals can be predicted but also the probability for each possible combination of goals by the two teams. The probability distribution employed here is a bivariate independent Poisson model, a relatively simple and standard model that fits empirical scores in football matches very well. The resulting probabilities (for up to five goals per team) are displayed in the heatmap below. Aggregating all probabilities for a Dutch win, a draw, or an Austrian win yields the probabilities shown in the table above (which do not sum to 100% exactly due to rounding).

Conference presentationThe Imagine conference hosted by the Austrian Ministry of Climate Action, Environment, Energy, Mobility, Innovation and Technology celebrates its 10th birthday today. The final highlight of the conference program is a public viewing of the match Netherlands vs. Austria where the forecast above will be presented alongside a live data-driven analysis by colleagues from the Rotterdam University of Applied Sciences. The presentation slides are linked from the screenshot below.

To leave a comment for the author, please follow the link and comment on their blog: Achim Zeileis.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: UEFA Euro 2024 forecast: Netherlands vs. Austria

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Guest blog contributed by Ning Leng, People and Product Leader, Roche-Genentech. Eric Nantz, Director, Eli Lilly and Company; Ben Straub, Principal Programmer, GSK; Sam Parmar, Statistical Data Scientist, Pfizer.

Supporting the science of drug development requires computational tools with careful implementations of core statistical functions and data structures. The R programming language, a general purpose language developed by statisticians that grows dynamically through the contributions of a worldwide community of developers is a common choice for serious statistical work. However, managing new versions of the core R language and the hundreds of specialized libraries (called packages in R) necessary to support multiple development groups in a way that ensures the consistency, reproducibility, and reliability of results poses many practical challenges

The FDA, for example, requires that the software and tools supporting a clinical trial submission are capable of producing reproducible results for an extended period of time. This means submitting code based on a version of R that is sufficiently tested and stable yet new enough to support the critical R packages over the required FDA time horizon.

So, how is the R environment release managed across different healthcare organizations? We interviewed individuals from different pharma companies to learn their internal approaches to keep their R environment up-to-date and secure.

Roche’s Scientific Computing Environment is container based, with clinical reporting done from managed qualified images being released twice per year – roughly timed to capture the last update to an R major version (April release) and a 6 month later update (September release). For each image, R packages undergo a mostly automated risk assessment to document package quality. Automated indicators of package quality include test coverage, thoroughness of documentation, test coverage of exported objects (using covtracer), and may be supplemented with package adoption measured using download counts, author reputation or other peripheral knowledge of the package’s history. Prior to internal publication, a representative sample of reverse dependencies are re-evaluated to safeguard against breaking changes. If the package meets our quality criteria it is published to a continuously updated repository of validated packages corresponding to the image’s R minor version (e.g. x.x). This allows flexibility for teams to roll forward to newer releases of packages within a managed release by moving their renv snapshot to a later date, easing the transition between bi-annual image releases. A generalized version of Roche’s automated process has been open sourced as ‘theValidator‘, and more details on the Roche process was shared within the R Validation case studies series.
Eli Lilly currently updates its qualified R environment only after a new major release of R is available and the corresponding release of Bioconductor (utilizing that version of R) is also available. In a new release, all packages currently installed from the CRAN and Bioconductor repositories are refreshed to their latest versions at the time of the release. Once the new R version is deployed, all packages are frozen for that particular release to ensure stability and reproducibility. Lilly maintains multiple R versions for backward compatibility. Only packages available on CRAN or Bioconductor are permitted for installation in the central package library. Lilly uses a hybrid approach of automation and risk-based assessment when a new package is requested for installation. In the event that a new version of a package is necessary for a project (such as a new Shiny application), the users are encouraged to leverage the renv package created by Posit to create a project-based environment which will not impact the central package library. As technology evolves and the R language becomes more prominent in clinical data analysis, Lilly continues to assess the current and future possibilities of a robust clinical computing environment primed for innovation while adhering to the foundational principles of reproducibility and transparency.
GSK releases “frozen R environments” for clinical reporting work on a 6-12 month cycle. The choice of R version is the latest stable release with at least one patch release of R, e.g. 4.3.1 rather than 4.3.0. As R itself is stable with decades of use, we prefer to focus on package assessment for building of our “frozen R environments.” Packages for this environment can be from external sources (CRAN, Bioconductor) or internally built at GSK, regardless of origin we assess the same way. We pay close attention to author qualification and institutional backing, types and breadth of testing, documentation and examples, and the practice of software development life cycle practices. Once a package is approved in this process it will be included in the frozen environment. Packages change over time, if substantial changes are implemented in the packages, then we re-assess with a focus on those changes for allowing up-versioning of the package in the frozen environment. These frozen environments ensure that clinical reporting can be easily reproduced if needed as all packages versions and the version of R used during the analysis are contained in the frozen environment.
Pfizer releases one new R version every year. We typically target R-x.y.1 releases to pick up patches – so we might consider this a “stable” release. The process of testing, documenting, and deploying R into validated containers is performed every 6 months, with a new release of R once per year, e.g. R-4.3.1, and an update to package set and package versions 6 months later (for the same R version). We take a snapshot date of CRAN to form the basis of our package set for the container build. We try to balance competing priorities of getting latest versions of packages and newest package releases, while maintaining a snapshot and version-controlled release to ensure reproducibility and stability.

Here is what we have: four companies and four somewhat complex bespoke solutions. It seems likely that if we interviewed a hundred representatives from a hundred different companies we would get at least a hundred different solutions. It is also not difficult to imagine that multiple protocols for managing R and package versions imposed a fairly complex project management solution on the FDA as it simultaneously deals with submissions from multiple sponsors.

In the R Consortium’s R Submissions Work Group meeting we have been discussing whether there might be a simple solution for at least dealing with the R versioning problem that might serve as a de facto standard for the industry. One suggestion that has gained some traction is that sponsors use the previous minor and latest patched R version for a submission. For example, if R version 4.4.0 is currently available then it is suggested that a sponsor uses the latest patch version (4.3.z). If R version 4.5.0 becomes available, then it is suggested that a sponsor uses the latest patch version (4.4.z). This ensures that the minor version is stable and most likely available to all stakeholders. Of course, if a version change eliminates a security problem, that might be preferred. (Note that R versions are organized as R x.y.z where, x is the major version, y is the minor version, and z is the patch version.)

We would love to hear what you think. Please, go to Issue number 117 on the GitHub repository of our working group and leave a comment.

The post The Crucial Role of Release Control in R for Healthcare Organizations appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Crucial Role of Release Control in R for Healthcare Organizations

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.See the full agenda and book your tickets today for EARL 2024 and key in RBTBZO at the checkout for a limited (while stocks last) 10% saving on your tickets as a special for R bloggers readers.

The Enterprise Applications of the R Language (EARL) Conference is a cross-sector tech conference focusing on the commercial use of the R programming language. Join us on 3rd – 5th September 2024 in Brighton for its 10th year as we hear from some of the world’s leading practitioners, consultants and industry experts in R, Python and data science.

Yes, you read that correctly! EARL has long been a cornerstone for professionals across industries who leverage R for real-world data challenges, but this year, we’re expanding our horizons by integrating Python. This convergence promises a more comprehensive view of modern data science, bridging statistical expertise with advanced computational capabilities.

New city, new languages – and new hosts too, with Brighton based data and analytics consultancy Datacove and Brighton’s tech sector hosts, Silicon Brighton taking over from the wonderful team at Ascent who have made EARL the event it is today.

As if that wasn’t enough to get you excited, our line-up of speakers includes the one and only Hadley Wickham! Joining him are Christel Swift from the BBC, Steph Locke from Microsoft, and renowned statistician and author Andy Field from the University of Sussex, each offering their unique insights on the industry today – and where we are headed tomorrow!

The conference agenda spans three days of workshops, keynotes, and talks featuring 40+ speakers from a diverse range of diverse and markets. It’s an unparalleled opportunity to learn, network, and engage with leading minds in both R and Python.

EARL 2024 wouldn’t be possible without the generous support of sponsors Posit, Ascent, and the R Consortium – and maybe you too! If you would like to showcase your brand in front of more than 200 of the world’s leading tech professionals, there are still sponsorship and advertising opportunities available. More details can be found on the EARL website, below.

We are excited to bring EARL 2024 to Brighton, a vibrant hub of culture, technology and innovation, and can’t wait to welcome you to what promises to be the most dynamic conference yet. Join us at The Grand Hotel, an elegant and historic venue located on the city’s picturesque seafront, blending classic charm with modern comforts. You’ll be treated to a delicious buffet lunch on both days of the conference, prepared by some of the city’s finest chefs – with tea and coffee also available throughout. Then cross the road for a memorable evening event at the British Airways i360, with stunning panoramic views of the city and coastline from the world’s tallest moving observation tower.

See the full agenda and book your tickets today for EARL 2024 and key in RBTBZO at the checkout for a limited (while stocks last) 10% saving on your tickets as a special for R bloggers readers.

See you in Brighton!

Celebrating a Decade of EARL: Join us in Brighton, UK for EARL 2024! was first posted on June 24, 2024 at 5:36 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Celebrating a Decade of EARL: Join us in Brighton, UK for EARL 2024!

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 479–483

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #479Today we have pretty weird sequence to generate — Recaman’s sequence. It doesn’t make any sense in real life, it is just mathematical concept linked to recursion. Nonetheless we have done more not really practical things here, so let get into the task.
I wanted this generator to be efficient, so I measured time for it. We had 10k of elements to generate, and it tooks only 0.05 sec. One of secrets is pre-allocations of memory. Object storing sequence is not increasing size while working, because we have already created object with N empty slots at the beginning and we are only populating it.

Loading libraries and data library(tidyverse)library(readxl)library(tictoc)library(memoise)path = "Excel/479 Recaman Sequence.xlsx"test = read\_excel(path) Transformation recaman\_sequence <- function(n) { recaman <- integer(n) recaman[1] <- 0 seen <- setNames(logical(n * 3), 0:(n * 3 - 1)) seen[1] <- TRUE for (i in 2:n) { prev\_value <- recaman[i - 1] next\_value <- prev\_value - (i - 1) if (next\_value > 0 && !seen[next\_value + 1]) { recaman[i] <- next\_value } else { next\_value <- prev\_value + (i - 1) recaman[i] <- next\_value } seen[recaman[i] + 1] <- TRUE } return(recaman)}tic()recaman\_sequence(10000)toc()# 0.05 sec elapsedresult = recaman\_sequence(10000) Validation identical(result, test$`Answer Expected`)# [1] TRUE Puzzle #480We already had ciphered text with many different codes, and also deciphered some. Today is time to decipher Ceasar’s Cipher. We have encrypted text and base shift for characters. Little bit tricky, because we need to have it case sensitive. But of course not impossible.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/480 Caesar's Cipher\_Decrypter.xlsx"input = read\_excel(path, range = "A1:B10")test = read\_excel(path, range = "C1:C10") Transformation decrypt\_caesar <- function(encrypted\_text, shift) { shift\_char <- function(char, shift\_value) { if (char %in% letters) { base <- 97 char\_val <- utf8ToInt(char) - base shifted\_val <- (char\_val - shift\_value) %% 26 intToUtf8(shifted\_val + base) } else if (char %in% LETTERS) { base <- 65 char\_val <- utf8ToInt(char) - base shifted\_val <- (char\_val - shift\_value) %% 26 intToUtf8(shifted\_val + base) } else { char } } decrypt\_char <- Vectorize(shift\_char, "char") decrypted\_text <- map2\_chr(str\_split(encrypted\_text, "")[[1]], 0:(nchar(encrypted\_text) - 1), ~ decrypt\_char(.x, shift + .y)) paste0(decrypted\_text, collapse = "")}result = input %>% mutate(`Answer Expected` = map2\_chr(`Encrypted Text`, Shift, decrypt\_caesar)) %>% select(`Answer Expected`) Validation identical(result, test)# [1] TRUE Puzzle #481Today’s challenge is a proof that mathematicians think differently. There is a story about two professors Godfrey Hardy and Srinivasa Ramanujan. One was visiting the other and came with taxi. And one of topic of their discussion was number of this cab. Number 1729 was first discovered by them Taxicab number, which has this interesting property, that it can be written as sum of two cubes, but in more than one way. And we have to check if given numbers are Taxicab ones. It can be very memory and time consuming so I measured it as well.

Loading libraries and data library(tidyverse)library(readxl)library(tictoc)path = "Excel/481 Taxicab Numbers.xlsx"input = read\_excel(path, range = "A1:A10")test = read\_excel(path, range = "B1:B10") Transformation tic()is\_taxicab = function(number) { x = ceiling(number^(1/3)) df = tibble(a = 1:x, b = 1:x) %>% expand.grid() %>% filter(a <= b, a^3 + b^3 == number) check = ifelse(nrow(df) >= 2, "Y", "N") return(check)}result = input %>% mutate(`Answer Expected` = map\_chr(Numbers, is\_taxicab)) toc()# 0.03 sec elapsed Validation identical(result$`Answer Expected`, test$`Answer Expected`)# [1] TRUE Puzzle #482Soccer is hot topic right now at least in Europe thanks to EURO 2024. So we have soccer related task to do. We have something looking like Champions League table, and we need to transform it into crosstable with results. We need to do some reversing to have it all correct, but it is only looking hard. Check it out.

If I would have to recommend one trick to note here, it is regex replacement with capturing group. Pretty smart solution.

Loading libraries and data library(tidyverse)library(readxl)library(janitor)path = "Excel/482 Soccer Result Grid.xlsx"input = read\_excel(path, range = "A2:C12") %>% clean\_names() test = read\_excel(path, range = "E2:J7") Transformation rev\_input = data.frame(team\_1 = input$team\_2, team\_2 = input$team\_1, result = input$result) %>% mutate(result = str\_replace(result, "([0-9]+)-([0-9]+)", "\\2-\\1"))all = bind\_rows(input, rev\_input) %>% pivot\_wider(names\_from = team\_2, values\_from = result) %>% arrange(team\_1) %>% select(sort(c("team\_1", colnames(.)[-1]))) %>% select(Team = team\_1, everything()) %>% mutate(across(everything(), ~ifelse(is.na(.), "X", .))) Validation identical(all, test)# [1] TRUE Puzzle #483Again we have some “drawing”. We need to populate sectors of matrix 20x20 with different but reccurent small sequences. I did it more verbatim way with every step widely written. And later I decided to make it little bit shorter and smarter using purrr functions.

Loading libraries and data library(tidyverse)library(readxl)path = "Excel/483 Generate Matrix.xlsx"test = read\_excel(path, range = "A2:T21", col\_names = F) Transformation — approach 1 seg1 = 5:9seg2 = 0:4seg3 = rev(seg1)seg4 = rev(seg2)pattern1 <- c(seg3, seg2, seg1, seg4)pattern2 <- c(seg4, seg1, seg2, seg3)pattern3 <- c(seg1, seg4, seg3, seg2)pattern4 <- c(seg2, seg3, seg4, seg1)block1 <- matrix(rep(pattern1, 5), nrow = 5, byrow = TRUE)block2 <- matrix(rep(pattern2, 5), nrow = 5, byrow = TRUE)block3 <- matrix(rep(pattern3, 5), nrow = 5, byrow = TRUE)block4 <- matrix(rep(pattern4, 5), nrow = 5, byrow = TRUE)final\_matrix <- rbind(block1, block2, block3, block4) %>% as.data.frame() Validation — approach 1 all.equal(test, final\_matrix, check.attributes = F)# # [1] TRUE Transformation — approach 2 a = 0:4b = 5:9patterns = list(c(rev(b),a, b, rev(a)), c(rev(a),b, a, rev(b)), c(b, rev(a), rev(b), a), c(a, rev(b), rev(a), b))final\_matrix = patterns %>% map( ~ matrix(rep(.x, 5), nrow = 5, byrow = TRUE)) %>% reduce(rbind) %>% as.data.frame() Validation — approach 2 all.equal(test, final\_matrix, check.attributes = F)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
PS. Couple weeks ago, I started uploading on Github not only R, but also in Python. Come and check it.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.So, you’ve mastered the basics of ggplot2 animation and are now looking for a real-world challenge? You’re in the right place. After reading this one, you’ll know how to download and visualize stock data change through something known as race charts.

You can think of race charts as dynamic visualizations (typically bar charts) that display a ranking of different items over time. In our case, we’ll build a stock price comparison animation that shows the average monthly stock price for 7 tickers for the last 10 years.

Let’s start by gathering data. The best part – you can do it straight from R!

Want to use R to process huge volumes of data? Read our detailed comparison of tools and packages for efficiently handling 1 billion rows.

Table of contents:* Data Gathering: How to Download and Organize Stock Data in R * Step 1: Build and Style a Bar Chart for a Single Time Period * Step 2: Chart Animation with R gganimate * Summing up Race Charts in R

‍

Data Gathering: How To Download And Organize Stock Data In RThe process of gathering data typically involves downloading CSV/Excel files from the web or connecting to a database of some sort. That couldn’t be further from the truth when it comes to financial data.

You can download data for any ticker and any time period straight from R! Just make sure you have the quantmod package installed. And speaking of packages, these are the ones you’ll need today:

library(quantmod)library(dplyr)library(lubridate)library(gganimate) If you don’t have them installed, simply run install.packages(“”) from the R console.

Moving on, we’ll declare a vector of 7 tickers for which we want to download daily stock prices: Amazon, Apple, Google, Microsoft, Meta, Tesla, and Nvidia. For each, we’ll grab the last 10 years’ worth of data, indicated with start\_date and end\_date.

To demonstrate how this approach works, we’ll call the quantmod::getSymbols() function to display the first 10 rows of a returned dataframe for the Nvidia stock. It will make adequate API calls to Yahoo Finance for you:

```

Define stock tickers and the time periodtickers <- c("AMZN", "AAPL", "GOOGL", "MSFT", "META", "TSLA", "NVDA")start_date <- Sys.Date() - years(10)end_date <- Sys.Date()# The getSymbols() function returns a data.frame of stock data for a given tickerhead(getSymbols("NVDA", src = "yahoo", from = start_date, to = end_date, auto.assign = FALSE), 10)

``` This is the output you’ll see:

Image 1 – Sample of Nvidia stock pricesWe only care about the Adjusted price and the date, which is represented by the dataframe index. Let’s see how to get the data for all tickers next.

How to Get Historical Data for All TickersThe get\_monthly\_averages() is a custom function that will download stock data for a provided ticker. It will then rename the dataframe columns so that the ticker name is removed from them (needed for the aggregation later).

As mentioned earlier, only the Adjusted price column is relevant, so we’ll construct a new dataframe with its value and the index (time period). We’ll then remove the day information from the date since we only care for monthly averages.

Finally, the mean value for each time period is calculated and returned:

```

Function to download and aggregate stock dataget_monthly_averages <- function(ticker) { # Download stock data from Yahoo Finance stock_data <- getSymbols(ticker, src = "yahoo", from = start_date, to = end_date, auto.assign = FALSE) # Remove ticker prefix from column names colnames(stock_data) <- gsub(paste0(ticker, "\."), "", colnames(stock_data)) # Keep only the 'Adjusted' column from the data stock_data <- stock_data[, "Adjusted"] # Convert the data to a data frame with Date and Adjusted columns stock_data <- data.frame(Date = index(stock_data), Adjusted = coredata(stock_data)) # Add a YearMonth column by flooring the Date to the nearest month stock_data$YearMonth <- floor_date(stock_data$Date, "month") # Group the data by month and calcualte the mean adjusted price for each period monthly_data <- stock_data %>% group_by(YearMonth) %>% summarize(Value = mean(Adjusted, na.rm = TRUE)) %>% ungroup() %>% mutate(Ticker = ticker) return(monthly_data)}

`` With the function to get single stock data out of the way, you can now use thelapply()` magic to apply the function to each of our 7 tickers. Here, we’ll also arrange the data by the time period (ascending) and the average stock value (descending):

data <- lapply(tickers, get\_monthly\_averages)data <- bind\_rows(data)data <- data %>% arrange(YearMonth, desc(Value))head(data, 10) Image 2 – Downloaded stock data for all 7 tickersYou could say the following to interpret this data: Microsoft’s average stock price for the month June of the year 2014 was USD 35.5.

You now have the data, so the only thing left to do is to build some cool visualizations!

Step 1: Build And Style A Bar Chart For A Single Time PeriodRendering an animated chart takes time, so a good piece of advice is to start small by building a visualization for a single time period. This way, you’ll know everything looks exactly the way you want to.

Race charts need one thing to work properly, and that is the rank. In our case, rank represents the position of the ticker’s value when compared to other tickers. In other words, it determines the position the column will have in a bar chart.

While here, we’ll also add a label column (you’ll see why in a bit):

chart\_data <- data %>% group\_by(YearMonth) %>% mutate( Rank = rank(-Value), Label = paste0("$", round(Value, 2)) )head(chart\_data, 10) Here’s what your dataframe should look like now:

Image 3 – Stock data with added Rank and Label columnsMeta had the highest stock price in June 2014, so it’s ranked number 1. The exact opposite is true for Nvidia.

The last thing we want to do before visualizing this data is to declare a custom color palette. We’ve grabbed the official brand color codes so the chart looks more representative:

chart\_colors <- c( AMZN = "#FF9900", AAPL = "#555555", GOOGL = "#0F9D58", MSFT = "#FFB900", META = "#0081FB", TSLA = "#cc0000", NVDA = "#76B900") And now, onto the visualization. As mentioned, we’ll create it for a single time period first, and discuss animation in the following section.

We’re creating a horizontal bar chart with the help of the geom\_tile() function that’s typically used to build heatmaps. The two geom\_text() calls are used to add ticker names and stock prices, both from their respective side of the chart. The rest of the code is related to styling, changing the theme, and playing around with the overall aesthetics:

ggplot(chart\_data %>% filter(YearMonth == "2024-05-01"), aes(Rank, group = Ticker, fill = as.factor(Ticker), color = as.factor(Ticker))) + geom\_tile(aes(y = Value / 2, height = Value, width = 0.9), alpha = 0.9, color = NA) + geom\_text(aes(y = 0, label = Ticker), vjust = 0.2, hjust = 1.3) + geom\_text(aes(y = Value, label = Label, hjust = -0.15)) + coord\_flip(clip = "off", expand = FALSE) + scale\_x\_reverse() + scale\_fill\_manual(values = chart\_colors) + scale\_color\_manual(values = chart\_colors) + theme\_minimal() + theme( axis.text.x = element\_blank(), axis.text.y = element\_blank(), axis.title.x = element\_blank(), axis.title.y = element\_blank(), plot.title = element\_text(size = 18, face = "bold", color = "#424242"), legend.position = "none", plot.margin = margin(1, 2, 1, 2, unit = "cm") ) In the end, this is the chart you should end up with:

Image 4 – Static chart for a single time periodNow let‘s see how to animate it!

Step 2: Chart Animation With R GganimateGood news – nothing much changes in the charting code. You only have to remove the data filtering so all time periods are captured, and also store the entire plot into a variable:

p <- ggplot(chart\_data, aes(Rank, group = Ticker, fill = as.factor(Ticker), color = as.factor(Ticker))) + geom\_tile(aes(y = Value / 2, height = Value, width = 0.9), alpha = 0.9, color = NA) + geom\_text(aes(y = 0, label = Ticker), vjust = 0.2, hjust = 1.3) + geom\_text(aes(y = Value, label = Label, hjust = -0.15)) + coord\_flip(clip = "off", expand = FALSE) + scale\_x\_reverse() + scale\_fill\_manual(values = chart\_colors) + scale\_color\_manual(values = chart\_colors) + theme\_minimal() + theme( axis.text.x = element\_blank(), axis.text.y = element\_blank(), axis.title.x = element\_blank(), axis.title.y = element\_blank(), plot.title = element\_text(size = 18, face = "bold", color = "#424242"), legend.position = "none", plot.margin = margin(1, 2, 1, 2, unit = "cm") ) Now to build the animation, we’ll append the transition\_states() function to our chart to indicate we want the chart to change between different states. Then, the view\_follow() function allows the chart window to follow the data dynamically, but only on the Y-axis.

The labs() function allows us to modify the title through the {closest\_state} property. In simple terms, this updates the time period in the chart title:

anim <- p + transition\_states(YearMonth, transition\_length = 4, state\_length = 1) + view\_follow(fixed\_x = TRUE) + labs(title = "Average monthly stock price in USD ({closest\_state})") Finally, the animate() function will render a 1024×768 GIF with 600 frames in total (more info about different rendering options here):

animate( anim, width = 1024, height = 768, res = 150, nframes = 600, fps = 60, end\_pause = 60, renderer = gifski\_renderer("stock\_race\_chart.gif")) Once the rendering finishes, you’ll see the following chart saved to your disk:

Image 5 – Rendered race chartAnd that’s the basics of race charts for you. Let’s wrap things up next.

Summing Up Race Charts In RData visualization is a powerful tool. Given the option, almost no one would choose to stare at spreadsheets instead of charts. Animating these charts brings the whole thing to a different dimension.

You now know how to breathe life into static and somewhat boring ggplot2 visualizations, which is an amazing skill to have. Animated charts make it easy to capture trends and shifts over time, ensuring your data stories are more engaging and easier to understand. They are especially useful in presentations and reports where you need to highlight key changes and patterns dynamically, rather than relying on a series of static images.

‍What are your thoughts on chart animation in R? Do you use gganimate or some other package? Let us know in our Slack community.

Are you hitting the wall with Excel? Here are 5 ways R can help you improve your business workflows.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Race Charts In R: How To Visualize And Compare Change Over Time With Gganimate

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The Warner Music Group launched a new company in April, Warner Music South Asia targeting region markets that include Bangladesh, Nepal, Pakistan, and Sri Lanka. The company thought that the region was populated and diverse. I will analyze the reasons behind that decision based on some economic data related to South Asia.

library(tidyverse)library(tidyquant)library(timetk)#Population Ages 15 to 64 for South Asia (percent of total)#(https://fred.stlouisfed.org/series/SPPOP1564TOZSSAS)df\_pop\_15\_64 <- tq\_get("SPPOP1564TOZSSAS", get = "economic.data", from = "1960-01-01") %>% rename(value = price) #Ratio of Female to Male Secondary School Enrollment for South Asia#(Ratio of girls to boys)#(https://fred.stlouisfed.org/series/SEENRSECOFMZSSAS)df\_female\_to\_male\_secondary\_enrollment <- tq\_get("SEENRSECOFMZSSAS", get = "economic.data", from = "1960-01-01") %>% rename(value = price)#Constant GDP per capita for South Asia (2010 U.S. Dollars)#(https://fred.stlouisfed.org/series/NYGDPPCAPKDSAS)df\_gdp\_per\_capita <- tq\_get("NYGDPPCAPKDSAS", get = "economic.data", from = "1960-01-01") %>% rename(value = price)#Mobile Cellular Subscriptions in South Asia (Number per 100 People)#(https://fred.stlouisfed.org/series/ITCELSETSP2SAS)df\_mobile\_phone <- tq\_get("ITCELSETSP2SAS", get = "economic.data", from = "1975-01-01") %>% rename(value = price)#Youth Unemployment Rate for South Asia (Percent)#(https://fred.stlouisfed.org/series/SLUEM1524ZSSAS)df\_youth\_unemployment <- tq\_get("SLUEM1524ZSSAS", get = "economic.data", from = "1991-01-01") %>% rename(value = price) #Merging all data setsdf\_merged <- rbind( df\_pop\_15\_64, df\_gdp\_per\_capita, df\_youth\_unemployment, df\_mobile\_phone, df\_female\_to\_male\_secondary\_enrollment ) Now we will draw all the variables above together to compare and find an insight.

```

Plot the time series with facetsp <- plot_time_series( .data = df_merged, .date_var = date, .value = value, .facet_vars = symbol, .title = "South Asia", .interactive = FALSE)#Custom titles for each facetcustom_labels <- c("SPPOP1564TOZSSAS" = "Population Ages 15 to 64 (Percent of total)", "NYGDPPCAPKDSAS" = "Constant GDP per capita (2010 U.S. Dollars)", "SLUEM1524ZSSAS" = "Youth Unemployment Rate (Percent)", "ITCELSETSP2SAS" = "Mobile Cellular Subscriptions (Number per 100 People)", "SEENRSECOFMZSSAS" = "Ratio of Female to Male Secondary School Enrollment")#Use `facet\_wrap` with a custom labelp + facet_wrap(~symbol, scales = "free", ncol = 2, labeller = as_labeller(custom_labels)) + theme(text = element_text(family = "Bricolage Grotesque", face = "bold"))

``` It is seen that mobile phone ownership has risen sharply since 2000 which is critical for a company whose main revenue is mostly streaming. Besides that, all other variables seem plausible to an investable market.

Now, we will model the GDP per capita based on other variables to find which factors impact the most on the development of the region’s economy. In order to model, we will use polynomial support vector machines (SVMs) via kernlab engine.

```

Merging data for modeling#By renaming the predictor and target variablesdf_mod_merged <- df_gdp_per_capita %>% rename(gdp_per_capita = value) %>% select(-symbol) %>% left_join(df_pop_15_64 %>% rename(pop_15_64 = value) %>% select(-symbol)) %>% left_join(df_youth_unemployment %>% rename(youth_unemployment = value) %>% select(-symbol)) %>% left_join(df_mobile_phone %>% rename(mobile_phone = value) %>% select(-symbol)) %>% left_join(df_female_to_male_secondary_enrollment %>% rename(female_to_male_secondary_enrollment = value) %>% select(-symbol)) %>% drop_na() #Bootstraping for tuninglibrary(tidymodels)set.seed(12345)df_folds <- bootstraps(df_mod_merged, times = 1000)#Preprocessingdf_rec <- recipe(gdp_per_capita ~ ., data = df_mod_merged) %>% step_date(date, features = c("year")) %>% step_rm(date) %>% step_normalize(all_numeric()) #Processed datadf_rec %>% prep() %>% bake(new_data = NULL)#Model: Polynomial support vector machines (SVMs) via kernlabdf_spec <- svm_poly( cost = tune(), degree = tune(), scale_factor = tune(), margin = tune()) %>% set_engine("kernlab") %>% set_mode("regression")#Building workflow setslibrary(workflowsets)wflow_svm <- workflow_set( preproc = list(formula = df_rec), models = list(SVM_poly = df_spec) )#Tuning and evaluating all the modelsgrid_ctrl <- control_grid( save_pred = TRUE, parallel_over = "everything", save_workflow = TRUE )grid_results <- wflow_svm %>% workflow_map( seed = 98765, resamples = df_folds, grid = 10, control = grid_ctrl )#Accuracy of the grid resultsgrid_results %>% rank_results(select_best = TRUE, rank_metric = "rsq") %>% select(Models = wflow_id, .metric, mean)# A tibble: 2 × 3# Models .metric mean# #1 formula_SVM_poly rmse 0.166#2 formula_SVM_poly rsq 0.989#Finalizing the model with the best parameterssvm_best_param <- grid_results %>% extract_workflow_set_result("formula_SVM_poly") %>% select_best(metric = "rsq")svm_fit_wflw <- grid_results %>% extract_workflow("formula_SVM_poly") %>% finalize_workflow(svm_best_param) %>% fit(df_mod_merged)#Calibrate the fitted workflow to the full data setlibrary(modeltime)calibration_tbl_svm <- svm_fit_wflw %>% modeltime_calibrate(new_data = df_mod_merged)#Accuracy of the finalized workflowcalibration_tbl_svm %>% modeltime_accuracy(metric_set = metric_set(rmse,rsq))# A tibble: 1 × 5# .model_id .model_desc .type rmse rsq# # 1 1 KERNLAB Test 0.136 0.990

``` Since our model’s accuracy results look sharp, we can move on to the permutation-based variable importance.

```

Variable importancelibrary(DALEXtra)#Fitted workflow for KNNset.seed(98765)knn_wflow_fitted <- workflow() %>% add_recipe(rec_features) %>% add_model(spec_knn) %>% fit(df_train)#Processed data frame for variable importance calculationimp_data <- df_rec %>% prep() %>% bake(new_data = NULL) #Explainer objectexplainer_svm <- explain_tidymodels( svm_fit_wflw %>% extract_fit_parsnip(), data = imp_data %>% select(-gdp_per_capita), y = imp_data$gdp_per_capita, label = "", verbose = FALSE )#Calculating permutation-based variable importance set.seed(1983)vip_svm <- model_parts(explainer_svm, loss_function = loss_root_mean_square, type = "difference", B = 100,#the number of permutations label = "")#Variable importance plotvip_svm %>% mutate(variable = case_when( variable == "date_year" ~ "Year", variable == "pop_15_64" ~ "Population Ages 15 to 64", variable == "youth_unemployment" ~ "Youth Unemployment", variable == "mobile_phone" ~ "Mobile Phone Ownership", variable == "female_to_male_secondary_enrollment" ~ "Female to Male Secondary School Enrollment", TRUE ~ variable )) %>% plot() + scale_y_continuous(expand = expansion(mult = c(.1, .1))) + labs(color = "", x = "", y = "", subtitle = "Higher indicates more important", title = "Factors Affecting GDP per capita for South Asia") + theme_minimal(base_family = "Bricolage Grotesque", base_size = 16) + theme(legend.position = "none", plot.title = element_text(hjust = 0.5, size = 14, face = "bold"), plot.subtitle = element_text(hjust = 0.5, size = 12), panel.grid.minor.x = element_blank(), panel.grid.major.y = element_blank(), plot.background = element_rect(fill = "#ffc900"))

``` The boxplot of the female-to-male ratio in secondary enrollment includes zero, indicating no significant impact of this ratio on GDP per capita. On the other hand, population, youth employment, and mobile phone ownership seem to have a significant impact on GDP per capita in South Asia.

To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Polynomial Support Vector Machines: Why Warner Music Entering the South Asia Market?

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionThis post will introduction to the healthyR.ai package. The healthyR.ai package is a collection of functions that I have developed to help me analyze and visualize data. The package is designed to be easy to use and to provide a wide range of functionality for data analysis. The package is also meant to help and provide some easy boilerplate funcationality for machine learning.

It might be best to view this post in light mode to see the tables better.

InstallationYou can install the released version of healthyR.ai from CRAN with:

install.packages("healthyR.ai") And the development version from GitHub with:

```

install.packages("devtools")devtools::install_github("spsanderson/healthyR.ai")

``` Getting StartedThe Goal

The ultimate goal really is to make it easier to do data analysis and machine learning in R. The package is designed to be easy to use and to provide a wide range of functionality for data analysis. The package is also meant to help and provide some easy boilerplate functionality for machine learning. This package is in its early stages and will be updated frequently.

It also keeps with the same framework of all of the healthyverse packages in that it is meant for the user to be able to use the package without having to know a lot of R. Many rural hospitals do not have the resources to perform this sort of work, so I am working hard to build these types of things out for them for free.

Let’s go through some examples.

library(healthyR.ai)library(tidyverse)library(DT) Now let’s get a list of all the functions that are exposed in the package.

```

Functions and their arguments for healthyRpat <- c("%>%",":=","as_label","as_name","enquo","enquos","expr", "sym","syms","required_pkgs.step_hai_fourier", "required_pkgs.step_hai_fourier_discrete", "required_pkgs.step_hai_hyperbolic", "required_pkgs.step_hai_scale_zero_one", "required_pkgs.step_hai_scal_zscore", "required_pkgs.step_hai_winsorized_move", "required_pkgs.step_hai_winsorized_truncate")tibble(fns = ls.str("package:healthyR.ai")) |> filter(!fns %in% pat) |> mutate(params = purrr::map(fns, formalArgs)) |> group_by(fns) |> mutate(func_with_params = toString(params)) |> mutate( func_with_params = ifelse( str_detect( func_with_params, "\("), paste0(fns, func_with_params), paste0(fns, "(", func_with_params, ")") )) |> select(fns, func_with_params) |> mutate(fns = as.factor(fns)) |> datatable( #class = 'cell-boarder-stripe', colnames = c("Function", "Full Call"), options = list( autowidth = TRUE, pageLength = 10 ) )

`` ExamplesLet’s start off going through an example of using the function,pca_your_repipe`. First, the syntax.

Example – PCA a recipeSyntax pca\_your\_recipe(.recipe\_object, .data, .threshold = 0.75, .top\_n = 5) Arguments* .recipe_object – * .data – The full data set that is used in the original recipe object passed into .recipe_object in order to obtain the baked data of the transform. * .threshold – A number between 0 and 1. A fraction of the total variance that should be covered by the components. * .top_n – How many variables loadings should be returned per PC

ValueA list object with several components.

DetailsThis is a simple wrapper around some recipes functions to perform a PCA on a given recipe. This function will output a list and return it invisible. All of the components of the analysis will be returned in a list as their own object that can be selected individually. A scree plot is also included. The items that get returned are:

pca_transform – This is the pca recipe.
variable_loadings
variable_variance
pca_estimates
pca_juiced_estimates
pca_baked_data
pca_variance_df
pca_rotattion_df
pca_variance_scree_plt
pca_loadings_plt
pca_loadings_plotly
pca_top_n_loadings_plt
pca_top_n_plotly

Working Example library(rsample)library(recipes)splits <- initial\_split(mtcars, prop = 0.8)rec\_obj <- recipe(mpg ~ ., data = training(splits)) |> step\_normalize(all\_predictors())pca\_output <- pca\_your\_recipe( .recipe\_object = rec\_obj, .data = mtcars, .threshold = 0.75, .top\_n = 5 ) Now let’s check the output:

pca\_output

$pca\_transform

── Recipe ──────────────────────────────────────────────────────────────────────

── Inputs

Number of variables by role

outcome: 1predictor: 10

── Operations

• Centering and scaling for: all\_predictors()

• Centering for: recipes::all\_numeric()

• Scaling for: recipes::all\_numeric()

• Sparse, unbalanced variable filter on: recipes::all\_numeric()

• PCA extraction with: recipes::all\_numeric\_predictors()

$variable\_loadings# A tibble: 100 × 4 terms value component id <chr> <dbl> <chr> <chr> 1 cyl -0.394 PC1 pca\_RSbN6 2 disp -0.389 PC1 pca\_RSbN6 3 hp -0.356 PC1 pca\_RSbN6 4 drat 0.321 PC1 pca\_RSbN6 5 wt -0.358 PC1 pca\_RSbN6 6 qsec 0.248 PC1 pca\_RSbN6 7 vs 0.319 PC1 pca\_RSbN6 8 am 0.248 PC1 pca\_RSbN6 9 gear 0.238 PC1 pca\_RSbN610 carb -0.232 PC1 pca\_RSbN6# ℹ 90 more rows$variable\_variance# A tibble: 40 × 4 terms value component id <chr> <dbl> <int> <chr> 1 variance 6.09 1 pca\_RSbN6 2 variance 2.42 2 pca\_RSbN6 3 variance 0.619 3 pca\_RSbN6 4 variance 0.231 4 pca\_RSbN6 5 variance 0.215 5 pca\_RSbN6 6 variance 0.171 6 pca\_RSbN6 7 variance 0.112 7 pca\_RSbN6 8 variance 0.0848 8 pca\_RSbN6 9 variance 0.0409 9 pca\_RSbN610 variance 0.0219 10 pca\_RSbN6# ℹ 30 more rows$pca\_estimates

── Inputs

Number of variables by role

outcome: 1predictor: 10

── Training information

Training data contained 25 data points and no incomplete rows.

── Operations

• Centering and scaling for: cyl, disp, hp, drat, wt, qsec, ... | Trained

• Centering for: cyl, disp, hp, drat, wt, qsec, vs, am, gear, ... | Trained

• Scaling for: cyl, disp, hp, drat, wt, qsec, vs, am, gear, ... | Trained

• Sparse, unbalanced variable filter removed: <none> | Trained

• PCA extraction with: cyl, disp, hp, drat, wt, qsec, vs, am, ... | Trained

$pca\_juiced\_estimates# A tibble: 25 × 3 mpg PC1 PC2 <dbl> <dbl> <dbl> 1 -1.67 -3.54 -0.529 2 0.0945 0.633 2.03 3 0.394 2.31 -1.60 4 0.178 1.88 -1.88 5 1.99 3.29 0.00164 6 1.66 3.79 0.988 7 -0.670 -2.14 -0.503 8 -0.953 -3.45 -0.248 9 2.24 3.59 0.0209 10 0.161 2.47 0.534 # ℹ 15 more rows$pca\_baked\_data# A tibble: 32 × 3 mpg PC1 PC2 <dbl> <dbl> <dbl> 1 0.0945 0.633 2.03 2 0.0945 0.613 1.87 3 0.394 2.76 0.137 4 0.161 0.228 -2.17 5 -0.288 -2.01 -0.623 6 -0.388 0.191 -2.55 7 -1.02 -2.82 0.438 8 0.660 1.91 -1.13 9 0.394 2.31 -1.60 10 -0.205 0.622 0.125# ℹ 22 more rows$pca\_variance\_df# A tibble: 10 × 6 PC var\_explained var\_pct\_txt cum\_var\_pct cum\_var\_pct\_txt ou\_threshold <chr> <dbl> <chr> <dbl> <chr> <fct> 1 PC1 0.609 60.86% 0.609 60.86% Under 2 PC2 0.242 24.19% 0.850 85.05% Over 3 PC3 0.0619 6.19% 0.912 91.24% Over 4 PC4 0.0231 2.31% 0.935 93.55% Over 5 PC5 0.0215 2.15% 0.957 95.70% Over 6 PC6 0.0171 1.71% 0.974 97.41% Over 7 PC7 0.0112 1.12% 0.985 98.52% Over 8 PC8 0.00848 0.85% 0.994 99.37% Over 9 PC9 0.00409 0.41% 0.998 99.78% Over 10 PC10 0.00219 0.22% 1 100.00% Over $pca\_rotation\_df# A tibble: 10 × 10 PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> 1 -0.394 0.0328 -0.146 0.0804 -0.162 0.0655 -0.223 -0.0200 -0.728 2 -0.389 -0.0623 0.0134 -0.0763 0.200 -0.383 0.447 0.0126 -0.369 3 -0.356 0.213 0.240 0.342 -0.195 -0.145 0.270 0.614 0.282 4 0.321 0.295 0.0556 0.336 0.768 -0.125 -0.0333 0.127 -0.204 5 -0.358 -0.124 0.391 -0.367 0.305 -0.347 -0.105 -0.338 0.262 6 0.248 -0.432 0.425 -0.317 0.00567 -0.0377 -0.272 0.568 -0.259 7 0.319 -0.255 0.421 0.505 -0.328 -0.313 0.156 -0.373 -0.163 8 0.248 0.443 -0.217 -0.241 -0.294 -0.693 -0.253 0.0794 -0.0246 9 0.238 0.449 0.346 -0.431 -0.125 0.278 0.509 -0.0781 -0.222 10 -0.232 0.445 0.490 0.151 -0.0541 0.189 -0.494 -0.133 -0.0201# ℹ 1 more variable: PC10 <dbl>$pca\_variance\_scree\_plt

$pca\_loadings\_plt

$pca\_loadings\_plotly$pca\_top\_n\_loadings\_plt

$pca\_top\_n\_plotly Pretty easy as you can see.

Example - Histogram Facet PlotSyntax hai\_histogram\_facet\_plot( .data, .bins = 10, .scale\_data = FALSE, .ncol = 5, .fct\_reorder = FALSE, .fct\_rev = FALSE, .fill = "steelblue", .color = "white", .scale = "free", .interactive = FALSE) Arguments* .data - The data you want to pass to the function. * .bins - The number of bins for the histograms. * .scale_data - This is a boolean set to FALSE. TRUE will use hai_scale_zero_one_vec() to [0, 1] scale the data. * .ncol - The number of columns for the facet_warp argument. * .fct_reorder - Should the factor column be reordered? TRUE/FALSE, default of FALSE * .fct_rev - Should the factor column be reversed? TRUE/FALSE, default of FALSE * .fill - Default is steelblue * .color - Default is ‘white’ * .scale - Default is ‘free’ * .interactive - Default is FALSE, TRUE will produce a plotly plot.

Working Example hai\_histogram\_facet\_plot(mtcars, .interactive = FALSE)

hai\_histogram\_facet\_plot(mtcars, .interactive = FALSE, .scale\_data = TRUE) Example - Boilerplacte FuncationalityNow we are going to go over some simple boilerplate funcationality. I call it boilerplate because you don’t have to change anything if you dont want to. For the boilerplate function there is a corresponding data preprocessor that will get the data into the shape it needs to be in for the algorithm. Let’s take a look.

Working ExampleFirst lets look at the data, then we will look at it after the preprocessor.

head(iris)

Sepal.Length Sepal.Width Petal.Length Petal.Width Species1 5.1 3.5 1.4 0.2 setosa2 4.9 3.0 1.4 0.2 setosa3 4.7 3.2 1.3 0.2 setosa4 4.6 3.1 1.5 0.2 setosa5 5.0 3.6 1.4 0.2 setosa6 5.4 3.9 1.7 0.4 setosa

rec\_obj <- hai\_earth\_data\_prepper(iris, Species ~ .)rec\_obj Now to run it through the boilerplate:

auto\_earth <- hai\_auto\_earth( .data = iris, .rec\_obj = rec\_obj, .best\_metric = "f\_meas", .model\_type = "classification") Now let’s inspect the output:

names(auto\_earth)

[1] "recipe\_info" "model\_info" "tuned\_info" Recipe Information auto\_earth[["recipe\_info"]] Model Information auto\_earth[["model\_info"]]

$model\_specMARS Model Specification (classification)Main Arguments: num\_terms = tune::tune() prod\_degree = tune::tune() prune\_method = noneComputational engine: earth $wflw══ Workflow ════════════════════════════════════════════════════════════════════Preprocessor: RecipeModel: mars()── Preprocessor ────────────────────────────────────────────────────────────────4 Recipe Steps• step\_string2factor()• step\_novel()• step\_dummy()• step\_zv()── Model ───────────────────────────────────────────────────────────────────────MARS Model Specification (classification)Main Arguments: num\_terms = tune::tune() prod\_degree = tune::tune() prune\_method = noneComputational engine: earth $fitted\_wflw══ Workflow [trained] ══════════════════════════════════════════════════════════Preprocessor: RecipeModel: mars()── Preprocessor ────────────────────────────────────────────────────────────────4 Recipe Steps• step\_string2factor()• step\_novel()• step\_dummy()• step\_zv()── Model ───────────────────────────────────────────────────────────────────────GLM (family binomial, link logit): nulldev df dev df devratio AIC iters convergedsetosa 144.779 111 52.6908 110 0.6360 56.69 22 1versicolor 137.505 111 125.5536 110 0.0869 129.60 4 1virginica 144.779 111 15.1575 110 0.8950 19.16 9 1Earth selected 2 of 15 terms, and 1 of 4 predictors (pmethod="none") (nprune=2)Termination condition: Reached nk 21Importance: Petal.Length-unused, Sepal.Length-unused, Sepal.Width-unused, ...Number of terms at each degree of interaction: 1 1 (additive model)Earth GCV RSS GRSq RSqsetosa 0.15145196 16.066078 0.34455933 0.36796602versicolor 0.20252995 21.484449 0.05906052 0.09266277virginica 0.04535734 4.811523 0.80370644 0.81071635All 0.36072282 38.265605 0.46747354 0.48649080$was\_tuned[1] "tuned" Tuned Information auto\_earth[["tuned\_info"]]

$tuning\_grid# A tibble: 7 × 2 num\_terms prod\_degree <int> <int>1 3 22 4 13 5 24 3 15 4 26 2 27 2 1$cv\_obj# Monte Carlo cross-validation (0.75/0.25) with 25 resamples # A tibble: 25 × 2 splits id <list> <chr> 1 <split [84/28]> Resample01 2 <split [84/28]> Resample02 3 <split [84/28]> Resample03 4 <split [84/28]> Resample04 5 <split [84/28]> Resample05 6 <split [84/28]> Resample06 7 <split [84/28]> Resample07 8 <split [84/28]> Resample08 9 <split [84/28]> Resample0910 <split [84/28]> Resample10# ℹ 15 more rows$tuned\_results# Tuning results# Monte Carlo cross-validation (0.75/0.25) with 25 resamples # A tibble: 25 × 4 splits id .metrics .notes <list> <chr> <list> <list> 1 <split [84/28]> Resample01 <tibble [77 × 6]> <tibble [5 × 3]> 2 <split [84/28]> Resample02 <tibble [77 × 6]> <tibble [5 × 3]> 3 <split [84/28]> Resample03 <tibble [77 × 6]> <tibble [5 × 3]> 4 <split [84/28]> Resample04 <tibble [77 × 6]> <tibble [5 × 3]> 5 <split [84/28]> Resample05 <tibble [77 × 6]> <tibble [5 × 3]> 6 <split [84/28]> Resample06 <tibble [77 × 6]> <tibble [5 × 3]> 7 <split [84/28]> Resample07 <tibble [77 × 6]> <tibble [5 × 3]> 8 <split [84/28]> Resample08 <tibble [77 × 6]> <tibble [5 × 3]> 9 <split [84/28]> Resample09 <tibble [77 × 6]> <tibble [5 × 3]>10 <split [84/28]> Resample10 <tibble [77 × 6]> <tibble [5 × 3]># ℹ 15 more rowsThere were issues with some computations: - Warning(s) x5: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x6: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x1: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x1: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x2: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x1: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x5: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x4: While computing multiclass `precision()`, some levels had no pred... - Warning(s) x48: glm.fit: algorithm did not converge, glm.fit: fitted probabilitie... - Warning(s) x2: glm.fit: algorithm did not converge, glm.fit: fitted probabilitie... - Warning(s) x49: glm.fit: fitted probabilities numerically 0 or 1 occurred, glm.fi... - Warning(s) x1: glm.fit: fitted probabilities numerically 0 or 1 occurred, glm.fi...Run `show\_notes(.Last.tune.result)` for more information.$grid\_size[1] 10$best\_metric[1] "f\_meas"$best\_result\_set# A tibble: 1 × 8 num\_terms prod\_degree .metric .estimator mean n std\_err .config <int> <int> <chr> <chr> <dbl> <int> <dbl> <chr> 1 2 1 f\_meas macro 0.124 25 0.0154 Preprocessor1\_Mo…$tuning\_grid\_plot

$plotly\_grid\_plot Metric SetsWith this package there comes some metric set’s that can be computed using the yardstick package. These are the metric sets that are available:

Classification hai\_default\_classification\_metric\_set()

A metric set, consisting of:- `sensitivity()`, a class metric | direction: maximize- `specificity()`, a class metric | direction: maximize- `recall()`, a class metric | direction: maximize- `precision()`, a class metric | direction: maximize- `mcc()`, a class metric | direction: maximize- `accuracy()`, a class metric | direction: maximize- `f\_meas()`, a class metric | direction: maximize- `kap()`, a class metric | direction: maximize- `ppv()`, a class metric | direction: maximize- `npv()`, a class metric | direction: maximize- `bal\_accuracy()`, a class metric | direction: maximize Regression hai\_default\_regression\_metric\_set()

A metric set, consisting of:- `mae()`, a numeric metric | direction: minimize- `mape()`, a numeric metric | direction: minimize- `mase()`, a numeric metric | direction: minimize- `smape()`, a numeric metric | direction: minimize- `rmse()`, a numeric metric | direction: minimize- `rsq()`, a numeric metric | direction: maximize Here is a list of the items currently on it as of writing this article:

Plotting Functions - Functions for plotting.
Clustering Functions - Functions for clustering and analysis.
Boiler Plate Functions - Functions for automatic recipes, workflows, and tuned models.
Dimensionality Reduction - Functions for dimension reduction.
Data Wrangling - Functions for data wrangling.
Data Preprocessors - Functions for data preprocessing.
Recipe Steps - Functions to add recipe steps.
Table Functions - Functions that return tibbles.
Vectorized Functions - Vector functions.
Augmenting Functions - Functions for data augmentation.
Miscellaneous Functions - Miscellaneous utility functions.
Metric Sets - Metric sets for evaluation.

For more detailed information, you can visit the healthyR.ai function reference page.

ConclusionI hope this helped a bit with understanding the healthyR.ai package. It is a very powerful package that can help you with a lot of different tasks. I will be writing more about this package in the future. If you have any questions or comments, please feel free to reach out to me at any of these:

People can get in touch with you through the following social profiles:

LinkedIn: linkedin.com/in/spsanderson
Mastodon: mstdn.social/@stevensanderson
Telegram: t.me/steveondata

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: An Introduction to healthyR.ai

View Details

[This article was first published on Online College Math Teacher, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Something a llttle different: Hexbin maps by Jerry Tuttle I recently became acquainted with hexbin maps, so I thought I would experiment with one. In a hexbin map, each geographical region is represented by an equally sized regular hexagon. (A regular hexagon can tessellate a plane, one of only three regular polygons that can do so besides an equilateral triangle and a square.) A hexbin map can be used to illustrate the distribution of a categorical variable by using colors, at a cost that geographical areas are distorted by size, shape, and orientation. Here is a hexbin map, followed by the same data in a conventional map. The four categories represent which of the two political parties had the larger popular vote in the 2020 presidential election and by what margin over the other party.

# Hexbin map

# Conventional map A hexbin is based on a geoJson file that provides the hexagon boundaries as two-dimensional coordinates. I downloaded a file from https://team.carto.com/u/andrew/tables/andrew.us_states_hexgrid/public/map in geo json format containing the boundaries of the US states. I read it in with read_sf from the R sf (simple features spatial vector data) library. I used a datafile of 2020 presidential election popular votes by state scraped from https://www.presidency.ucsb.edu/statistics/elections/2020 .I binned percent Democraric votes to total votes minus percent Republican votes to total votes into four bins and joined to the sf file. I used ggplot to plot the hexbin map, where label = iso3166_2 provides a two-character state abbreviation. For the conventional map, I used the R usmap library.

A hexbin map is a nice alternative to a conventional map. With a conventional map, it’s hard to read those small states. Howwever, it is not the right visualization tool if the data is quantitative. It is also not the right tool if the size or geography is important. Utah is not the same size as Califirnia in either area or number of voters, and South Carolina is not east of North Carolina.

An alternative to hexagonal areas is to make them squares, but hexagons seem more interesting.

Here is the R code I used:

plot the conventional map first:

library(usmap)

Color maps with data; must have fips column

library(ggplot2)
library(readxl)
data <- read_excel("C:/Users/Jerry/Desktop/R_files/presidential_2020.xlsx")
data$party <- ifelse(data$Dem_Minus_Rep >=0 & data$Dem_Minus_Rep <= 0.10, 'Dem 0 to 10%',
ifelse(data$Dem_Minus_Rep > 0.10, ‘Dem > 10%’,
ifelse(data$Dem_Minus_Rep <0 & data$Dem_Minus_Rep >= -0.10, ‘Rep 0 to 10%’,
ifelse(data$Dem_Minus_Rep < -0.10, 'Rep > 10%’, “Error”))))

party_colors <- c("blue", "lightskyblue", "red", "#FF66CC")
mytitle=”2020 Presidential Election Popular Vote by State”

plot_usmap(data = data, values = “party”, labels = TRUE, label_color = “white”) +
labs(title=mytitle) +
scale_fill_manual(values=party_colors) +
theme(
legend.position=”bottom”,
plot.title = element_text(size=15, face=”bold”),
legend.title = element_text(size=12, face=”bold”),
legend.text = element_text(size=12, face=”bold”))

plot the hexbin map:

Download coordinates from https://team.carto.com/u/andrew/tables/andrew.us_states_hexgrid/public/map

in geo json format

my_sf <- read_sf("C:/Users/Jerry/Desktop/R_files/us_states_hexgrid.geojson")
my_sf <- my_sf %>%
mutate(google_name = gsub(” $United States$”, “”, google_name))

join sf file with data file

my_sf_data <- my_sf %>%
left_join(data, by = c(“google_name” = “STATE”))

ggplot(my_sf_data) +
geom_sf(aes(fill = party), linewidth = 0, alpha = 0.9) +
geom_sf_text(aes(label = iso3166_2), color = “white”, size = 4, alpha = 1) +
theme_void() +
scale_fill_manual(values=party_colors) +
ggtitle(mytitle) +
theme(
legend.position=”bottom”,
plot.title = element_text(size=15, face=”bold”),
legend.title = element_text(size=12, face=”bold”),
legend.text = element_text(size=12, face=”bold”))
# End

############################################################################

To leave a comment for the author, please follow the link and comment on their blog: Online College Math Teacher.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Something a llttle different: Hexbin maps

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhen working with data, exporting your results to an Excel file can be very handy. Today, I’ll show you how to write the iris dataset to an Excel file using R and Python. We will explore three R packages: writexl, openxlsx, and xlsx, and the openpyxl library in Python. Let’s dive in!

Writing Excel Files in RFirst, let’s start with R. We’ll use the well-known iris dataset and write it to a temporary file using three different packages.

Using writexlThe writexl package is straightforward and easy to use for writing data frames to Excel files.

```

Install and load the writexl packageinstall.packages("writexl")library(writexl)# Write iris dataset to a temporary filewritexl::write_xlsx(iris, tempfile())

`` Thewrite_xlsxfunction does exactly what it says: it writes your data frame to an Excel file. Thetempfile()` function creates a temporary file, which is useful for quick testing without cluttering your directory.

Using openxlsxThe openxlsx package provides more flexibility and additional features compared to writexl.

```

Install and load the openxlsx packageinstall.packages("openxlsx")library(openxlsx)# Write the iris dataset to a temporary fileopenxlsx::write.xlsx(iris, tempfile())

`` Withopenxlsx, you can directly write the data frame to an Excel file using thewrite.xlsx` function, making the process simple and efficient.

Using xlsxThe xlsx package is another option that can be useful, though it requires Java.

```

Install and load the xlsx packageinstall.packages("xlsx")library(xlsx)# Write the iris dataset to a temporary filexlsx::write.xlsx(iris, paste0(tempfile(), ".xlsx"))

``write.xlsxfrom thexlsxpackage works similarly to the previous functions but requires the.xlsx` extension to be explicitly added to the temporary file name.

Writing Excel Files in PythonNow, let’s see how to achieve the same with Python using the openpyxl library.

```

Install openpyxl if you haven't already!pip install openpyxlimport openpyxl# Load an existing workbookworkbook = openpyxl.load_workbook("example.xlsx")# Add a new seetworkbook.create_sheet(title = "Sheet1")sheet_name = "Sheet1"sheet = workbook[sheet_name]sheet["A1"] = "Hello, World!"workbook.save("example.xlsx")

``` Here is a concise breakdown of what this script does:

Installs the openpyxl library (if necessary).
Imports the openpyxl library.
Loads an existing Excel workbook named example.xlsx.
Creates a new sheet titled “Sheet1” in the workbook.
Assigns the value “Hello, World!” to cell A1 of the new sheet.
Saves the changes back to “example.xlsx”.

Try It Yourself!Feel free to try these code snippets on your own. Exporting data to Excel is a common task, and knowing different ways to do it can be very useful in your data analysis toolkit. If you want to learn more about integrating Excel with R and Python, consider purchasing the book “Extending Excel with Python and R” for in-depth tutorials and advanced techniques.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Writing Excel Spreadsheets to Disk with R and Python

View Details

[This article was first published on R | Discindo, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I am involved in a Posit Teamdeployment, and one of the things that we are looking into is default R packagesthat should be made available to all users. We are looking to do this because wewould like to avoid people installing, for example tidyverse, in their own locallibraries in order to save on space and to make sure everyone is on the sameversion, at least for the packages that are considered to be a preferreddefault option for working with data in R.

In order to do this we wanted to collect all the packages that are currentlyused, their versions, source repository and similar information. That way wecan see if anything else should be installed for all users, in addition to thebest guess that we should have tidyverse, tidymodels, and shiny.

In order to do this we first have to get the list of installed packages, whichis fairly simple to do:

installed\_packages <- installed.packages() Then, utils::packageDescription can be used to get the packagesdescriptions. For example for getting the package description fordplyr` we can run:

dplyr\_pkg\_desc <- utils::packageDescription('dplyr') The result is a list, and it can be subsetted to see details, for example:

```

dplyr_pkg_desc[1]$Type[1] "Package"> dplyr_pkg_desc[2]$Package[1] "dplyr"> dplyr_pkg_desc[3]$Title[1] "A Grammar of Data Manipulation"> dplyr_pkg_desc[4]$Version[1] "1.1.4" `` At this point, I am thinking that all these description files have the samestructure. Therefore, if I want to get all packages’ version I need tolapplytoget the fourth element and that’s that. It turns out this is not entirely true.Not all packages have the same structure of the description. Seetydir`:

```

tidyr_pkg_desc[1]$Package[1] "tidyr"> tidyr_pkg_desc[2]$Title[1] "Tidy Messy Data"> tidyr_pkg_desc[3]$Version[1] "1.3.1"> tidyr_pkg_desc[4]$Authors@R[1] "c(\n person(\"Hadley\", \"Wickham\", , \"hadley@posit.co\", role = c(\"aut\", \"cre\")),\n person(\"Davis\", \"Vaughan\", , \"davis@posit.co\", role = \"aut\"),\n person(\"Maximilian\", \"Girlich\", role = \"aut\"),\n person(\"Kevin\", \"Ushey\", , \"kevin@posit.co\", role = \"ctb\"),\n person(\"Posit Software, PBC\", role = c(\"cph\", \"fnd\"))\n )" `` Number four is the authors, and version is three. And these are two packages thatare ultimately from the same author. Look atdata.table`:

```

data.table_pkg_desc[1]$Package[1] "data.table"> data.table_pkg_desc[2]$Version[1] "1.15.4"> data.table_pkg_desc[3]$Title[1] "Extension of data.frame"> data.table_pkg_desc[4]$Depends[1] "R (>= 3.1.0)" ``` Now, of course, subsetting works with using the name, instead of the position:

```

dplyr_pkg_desc[["Version"]][1] "1.1.4"> tidyr_pkg_desc[["Version"]][1] "1.3.1"> data.table_pkg_desc[["Version"]][1] "1.15.4" ``` However, to be honest, it rarely comes to my mind to subset lists like this.

package\_names <- installed.packages()[, 1]all\_packages\_data <- lapply(package\_names, utils::packageDescription)version\_number <-lapply(1:length(package\_names), function (x) {all\_packages\_data[[x]][["Version"]]}) The above is possible, and then to cbind all needed fields in a data.frame.

However, looking at the packageDescription documentation, it seems the best wayis to use additional arguments the function. This is neat:

package\_data <- lapply(package\_names,utils::packageDescription,fields = c("Package", "Version", "Built", "Repository")) And then there is another surprise. The results are with classpackageDescription which makes getting to a data.frame, or tibble inthis case, a bit complicated:

package\_data <- purrr::map\_df(package\_names,utils::packageDescription,fields = c("Package", "Version", "Built", "Repository"))Error in `as\_tibble()`:! All columns in a tibble must be vectors.✖ Column `askpass` is a `packageDescription` object. The full solution involves a step of changing the class of the object usingas, and then reassigning the names of each element, because the previous stepremoves them:

package\_data <- lapply(package\_names,utils::packageDescription,fields = c("Package", "Version", "Built", "Repository")) |>lapply(as, Class = "list") |>lapply(setNames, c("Package", "Version", "Built", "Repository")) |>dplyr::bind\_rows() To leave a comment for the author, please follow the link and comment on their blog: R | Discindo.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Few notes on getting R package data from the local library

View Details

[This article was first published on coding-the-past, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.1. What is R?R is a programming language and an environment for statistical computing and visualization. R is not a general-purpose programming language, like Python or Java, because its focus is on statistical computing. The language is very popular in the academic environment and allows for complex calculations and algorithms.

What is Power BI?Power BI is a set of softwares and applications focused on data analysis and visualization for Business Intelligence. For this article, when we talk about Power BI, we refer to Power BI Desktop, a drag and drop application used to transform, analyse and visualize data.

R vs Power BIBelow, a list of the main differences and similarities of R and Power BI is presented for several aspects:
Scope: While R is more suitable for academic and complex statistical data analysis, Power BI is more adequate for quick visual analyses. While R is common in the academic context, it can also be used in companies and industries that leverage data science for decision making. In this case, R would be used to prepare the data, train models and the Power BI to visualize the findings;
Learning Curve: Power BI is user-friendly and allows the creation of beautiful visualizations with a few clicks. R, on the other hand, has a steep learning curve. It requires a lot more training and reading more complex documentation before you can produce effective visualizations;
Interface: R is a written programming language, while most of tasks in Power BI are achieved with drag and drop actions;
Data Visualization: Power BI is limited in its visuals and customization options of reports and graphs, while R is flexible and versatile. There are many more chart types that can be plotted in R compared to Power BI. On the other hand, it is much easier and faster to plot appealing visualizations in Power BI compared to R;
Data Analysis: R provides libraries for advanced statistical operations that allow statistical inference, causal inference, machine learning and more complex analysis. Power BI is more suitable for answering simple Business Intelligence questions.
Price: Both platforms are free, but companies offer paid tools to enrich their functionalities.

R vs Power BI for digital humanitiesR as well as Power BI might be used for digital humanities. R is perfect for analyses and visualizations for a scientific article. It is also the right option if you would like to implement complex algorithms. Power BI is a great fit if you would like to easily produce beautiful plots and enable user interactivity for a broader audience.

In education, for example, Power BI could be used to produce an interactive dashboard exploring the casualties of World War II. This could be used to teach history or bring insights to researchers on possible research questions.

Regarding R, this blog has plenty of examples of how to apply it to the humanities. I recommend this article where you learn about the use of synthetic control to investigate hypothesis in History: ‘When Numbers Meet Stories – an introduction to the synthetic control method in R’

R vs Power BI – ExamplesTo exemplify the differences and similarities of R and Power BI, we will replicate in Power BI the treemap plotted in R in the lesson Treemaps in R.

The data used in R is also available in a CSV file at this link. It is part of a great initiative by Professor Vincent Arel-Bundock to gather many interesting R datasets and make them available in CSV format on this page: R Datasets.

Power BI Desktop is free and you can download it from the Power BI Microsoft official page. To learn more about it and how to get started, please consult this resource.

In the lesson Treemaps in R we learnt how to plot a treemap in R. In this lesson we will plot the same treemap in Power BI. To do that, download the data above and save it in the desired folder.

When you open Power BI, you will see the option to load data from an Excel File. You can choose this option and a window will open to select the file with your data. You can then select all files to see also csv files. Select the cholera.csv file and confirm. You will be offered the option to transform your data in Power Query, a tool aimed at preparing your data before visualization. For this lesson, you can skip this step and load the data without transforming its structure.

On the bar to the right, you will see the variables of your dataset. We would like to create a treemap in which we have bigger rectangles representing the regions of London and smaller rectangles representing the districts within their respective region. The size of the rectangles will inform us about the mortality caused by cholera in a given region and district. These are the relevant variables for us:

region will define our outer rectangles (categories) and will represent regions of London (West, North, Central, South, Kent);
district will define our inner rectangles (details), representing the districts of London;
cholera_drate represents deaths caused by cholera per 10,000 inhabitants in 1849 and will define the size of rectangles

The first step is to select the cholera_drate field, as shown in the image below. You will realise that Power BI automatically creates a bar chart with the sum of all death rates.

Now, click on the bar plot and select the option Treemap in the Visualization tab, as shown in the image below.

The next step is to define which variable will determine the branches of our treemap, that is, the more general category. In our case, it is region. Finally, we define the field determining the leaves of our treemap. In this example, the leaves are the districts inside each region of London. Drag these two fields to category and details as shown below.

That’s it! Without any line of code, you created a treemap that offers a great visual of London cholera death rates by region and district. You have even automatically generated tooltips that provide additional information about each leaf in your tree. You can further format your plot to have your desired colors, fonts and sizes. Read more about how to format a visualization on this page. Below you see the formatted version of the treemap.

As you have seen, compared to R, it is easier to plot a treemap in Power BI. On the other hand, Power BI customization options are limited compared to R. Please, if you have any question or comments, feel free to write below and I wish you a great learning journey!

Conclusions* Both R and Power BI are great tools for data analysis. While R is more suitable for complex and academic applications, Power BI is user-friendly and produces beautiful visualizations with drag-and-drop actions;
Deciding whether to use R or Power BI depends on your goals and requirements, and the two tools can complement each other to produce effective results.

To leave a comment for the author, please follow the link and comment on their blog: coding-the-past.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R vs Power BI

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. If you like these contributions, please consider buying me a coffee.

AboutI added the CP 1919 / PSR B1919+21 Dataset to my GitHub.

This dataset, found in one of my old external drives, corresponds to the famous plot from Radio Observations of the Pulse Profiles and Dispersion Measures of Twelve Pulsars (Craft, 1970). This is broadly known as the Joy Division’s plot from Unknown Pleasures. If you happen to know whom created the provided CSV file, please let me know so I can give proper credit.

The dataset contains “successive pulses from the first pulsar discovered, CP 1919, are here superimposed vertically. The pulses occur every 1.337 seconds. They are caused by rapidly spinning neutron star.” (The Cambridge Encyclopaedia of Astronomy, 1977)

Thanks to Scientific American, there is a complete explanation of the dataset and its origin.

Read pulsar <- readr::read\_csv("https://raw.githubusercontent.com/pachadotdev/cp1919/main/cp1919.csv")

Rows: 24000 Columns: 3── Column specification ────────────────────────────────────────────────────────Delimiter: ","dbl (3): x, y, zℹ Use `spec()` to retrieve the full column specification for this data.ℹ Specify the column types or set `show\_col\_types = FALSE` to quiet this message.

pulsar

```

A tibble: 24,000 × 3 x y z 1 1 1 -0.81 2 2 1 -0.91 3 3 1 -1.09 4 4 1 -1 5 5 1 -0.59 6 6 1 -0.82 7 7 1 -0.43 8 8 1 -0.68 9 9 1 -0.7110 10 1 -0.27# ℹ 23,990 more rows

VisualizeThe Cambridge Encyclopaedia of Astronomy (1977) library(ggplot2)library(ggridges)col1 <- "white"col2 <- "black"ggplot(pulsar, aes(x = x, y = y, height = z, group = y)) + geom_ridgeline( min_height = min(pulsar$z), scale = 0.2, linewidth = 0.5, fill = col1, colour = col2 ) + scale_y_reverse() + theme_void() + theme( panel.background = element_rect(fill = col1), plot.background = element_rect(fill = col1, color = col1), ) The Nature of Pulsars (Scientific American, 1970) col1 <- "#94cee1"col2 <- "white"ggplot(pulsar, aes(x = x, y = y, height = z, group = y)) + geom_ridgeline( min_height = min(pulsar$z), scale = 0.2, linewidth = 0.5, fill = col1, colour = col2 ) + scale_y_reverse() + theme_void() + theme( panel.background = element_rect(fill = col1), plot.background = element_rect(fill = col1, color = col1), ) Joy Division’s Unknown Pleasures (1979) col1 <- "black"col2 <- "white"ggplot(pulsar, aes(x = x, y = y, height = z, group = y)) + geom_ridgeline( min_height = min(pulsar$z), scale = 0.2, linewidth = 0.5, fill = col1, colour = col2 ) + scale_y_reverse() + theme_void() + theme( panel.background = element_rect(fill = col1), plot.background = element_rect(fill = col1, color = col1), ) ``` To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: CP 1919 / PSR B1919+21 Dataset

View Details

[This article was first published on R - datawookie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The ability to specify a message ID in emails sent from the {emayili} package makes it possible to create email threads.

Create a ServerFirst set up the SMTP server details. Credentials are stored in environment variables.

library(emayili)GMAIL\_USERNAME = Sys.getenv("GMAIL\_USERNAME")GMAIL\_PASSWORD = Sys.getenv("GMAIL\_PASSWORD")smtp <- gmail( username = GMAIL\_USERNAME, password = GMAIL\_PASSWORD) Create a Message IDCreate a suitably unique message ID.

id <- c(letters, 0:9) %>% sample(size = 24, replace = TRUE) %>% paste0(collapse = "") %>% paste0("@mx.google.com")print(paste("Message ID:", id))2lcwvut0ji8no0dp33c2pbaf@mx.google.com Send a Message with an IDNow create a message specifying that message ID. Set the sender and receiver to the same email address so that we’ll be able to see the thread in a single inbox.

msg <- envelope( to = GMAIL\_USERNAME %>% address(display = "Receiver"), from = GMAIL\_USERNAME %>% address(display = "Sender"), subject = "Hello!", id = id, # importance = "high", priority = "urgent") %>% return\_path(GMAIL\_USERNAME %>% address(display = "Sender")) Send the message.

smtp(msg, verbose = TRUE) This is what the message source looks like in my email client:

Date: Fri, 21 Jun 2024 22:07:56 -0700 (PDT)X-Google-Original-Date: Sat, 22 Jun 2024 05:07:56 GMTX-Mailer: {emayili}-0.8.0MIME-Version: 1.0To: ReceiverFrom: SenderSubject: Hello!Message-ID: <2lcwvut0ji8no0dp33c2pbaf@mx.google.com>Priority: urgentReturn-Path: Sender Notice that the message ID we specified is included as the Message-ID header.

Reply on Same ThreadNext we’ll respond to that message. You can again specify the message ID for the response, but since we won’t be proceeding any further with this thread we’ll just let the SMTP server generate a message ID. However, we do specify the message ID from the original message in the

msg <- envelope( to = GMAIL\_USERNAME %>% address(display = "Sender"), from = GMAIL\_USERNAME %>% address(display = "Receiver"), subject = "Hello! (reply)") %>% inreplyto(id) %>% references(id) Send the reply.

smtp(msg, verbose = TRUE) This is the source for the reply in my email client:

Message-ID: <66765c68.050a0220.6ffbc.d0a9@mx.google.com>Date: Fri, 21 Jun 2024 22:08:56 -0700 (PDT)X-Google-Original-Date: Sat, 22 Jun 2024 05:08:56 GMTX-Mailer: {emayili}-0.8.0MIME-Version: 1.0To: SenderFrom: ReceiverSubject: Re: Re: Hello! (reply)Return-Path: ReceiverIn-Reply-To: <2lcwvut0ji8no0dp33c2pbaf@mx.google.com>References: <2lcwvut0ji8no0dp33c2pbaf@mx.google.com> Notice the generated message ID but that the ID of the original message is included in both the In-Reply-To and References header fields.

Thread in Email ClientFinally, check that the thread is recognised by email client.

Yes indeed, Thunderbird recognises that the original message and reply are related and renders them as a thread.

To leave a comment for the author, please follow the link and comment on their blog: R - datawookie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Creating Email Threads

View Details

[This article was first published on r on Everyday Is A School Day, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

MCAR, MAR, MNAR, all so confusing.
But with DAG, oh so amusing!
Many technical words, I don’t understand,
but with simulation, I am a fan!
Join me in exploring missing mechanisms,
learn I will with great optimism.

Visualizing Missing Data With DAG & Simulation
Just for kicks, what is the missing mechanism of the title? Scroll all the way down for answer!

Motivations The motivation behind this topic stems from the attempt to understand the missing mechanisms of Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR). I had a hard time understanding its terminology and also definition, until I stumbled upon Understanding missing data mechanisms using causal DAGs mechanism by Cameron Patrick. Very informative and also has a lot of references to guide my journey of simulating these mechanisms in order for me to grasp the surface of the meaning of missingness. It was a bumpy road, and I must say, I don’t think I completely understand it, but I think I’m getting closer. Disclaimer, this is a note for myself with R code to generate the missing mechanisms for all 3 properties and also the underlying mathematical notation. If you find any mistakes in my understanding, please feel free to comment below and guide me to the right path.

Objectives * Motivations * Generate Data * MCAR * MAR * MNAR * Variety of Missingness DAGs/notation/representation * Opportunity for Improvement * References * Lessons Learnt

Generate Data library(tidyverse)library(broom)set.seed(1)n <- 100z <- rnorm(n)x <- 0.2*z + rnorm(n)y <- 0.3*z + 0.5*x + rnorm(n)model <- lm(y~x+z)summary(model)## ## Call:## lm(formula = y ~ x + z)## ## Residuals:## Min 1Q Median 3Q Max ## -2.94359 -0.43645 0.00202 0.63692 2.63941 ## ## Coefficients:## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 0.02535 0.10519 0.241 0.81005 ## x 0.44653 0.10948 4.079 9.29e-05 ***## z 0.33180 0.11877 2.794 0.00628 ** ## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## Residual standard error: 1.043 on 97 degrees of freedom## Multiple R-squared: 0.2339,Adjusted R-squared: 0.2181 ## F-statistic: 14.81 on 2 and 97 DF, p-value: 2.442e-06 A very simple continuous data for x (exposure), y (outcome), and z (common cause). We aren’t going to pay too much attention to z on the DAG, hence it’s omitted, so that we can focus on the missingness mechanism.

Take note that the true x coefficient is 0.4465332 and the std.error is 0.1094785.

Now let’s write a function so that we can visualize the missing values of the different missingness mechanisms we’re about to encounter.

Write a function to Visualize compare\_plot <- function(z=z,x=x,y=y,ym=ym,mechanism) { df <- tibble(y,x,z,ym) plot <- ggplot() + geom\_point(df, mapping=aes(x=x,y=y), color = "red", alpha = 0.5) + geom\_smooth(df, mapping=aes(x=x,y=y), method = "lm", color = "black") + geom\_point(df, mapping=aes(x=x,y=ym), color = "blue", alpha = 0.5) + geom\_smooth(df, mapping=aes(x=x,y=ym), method = "lm", color = "red") + ggtitle(label = mechanism, subtitle = paste0("y observed (blue point), y missing (red point), true x coef (black line), \nx coef with y missing (red line)")) + theme\_bw() return(plot)}

MCAR

X: Exposure.
Y: Complete outcome, sometimes this is also the sum of Y_mis (missing Y) and Y_obs(observed Y).
Yobs: Outcome that is observed.
M: Missingness mechanism of Y, sometimes can be represented as My

\begin{gather} P(M=0|Y_{obs},Y_{mis},\psi) = P(M=0|\psi) \end{gather}

Over here, we assign M = 0 as missing, where M = 1 is not missing. Take note that this on certain textbooks are reversed. Let $\psi$ contain the parameters of the missing data model. With MCAR missing mechanism, we can ignore $Y_{obs}$ and $Y_{mis}$, hence it’s the missingness is random, not caused by variables in the DAG. Example in real life could be data missing from a portable meter due to out of battery.

Alright, let’s similate this in R and visualize it! Take note that when we estimate X coefficient, we will be using complete case analysis (aka listwise deletion, pairwise deletion, available case analysis, deletion methods, complete record analysis). We will not attempt to impute at this point.

```

mcarmy <- rbinom(n, 1, 0.7) |> as.logical()ym <- c()for (i in 1:n) { ym[i] <- ifelse(my[i]==T,y[i],NA)}summary(lm(ym ~ x + z))## ## Call:## lm(formula = ym ~ x + z)## ## Residuals:## Min 1Q Median 3Q Max ## -2.88370 -0.62185 -0.02348 0.67784 2.80147 ## ## Coefficients:## Estimate Std. Error t value Pr(>|t|) ## (Intercept) -0.02645 0.12961 -0.204 0.83894 ## x 0.43957 0.14479 3.036 0.00341 ## z 0.45181 0.14763 3.060 0.00318 ## ---## Signif. codes: 0 '' 0.001 '' 0.01 '' 0.05 '.' 0.1 ' ' 1## ## Residual standard error: 1.08 on 67 degrees of freedom## (30 observations deleted due to missingness)## Multiple R-squared: 0.2798,Adjusted R-squared: 0.2583 ## F-statistic: 13.01 on 2 and 67 DF, p-value: 1.678e-05compare_plot(z,x,y,ym,"MCAR")

`` What’s going on up there? Let’s break it down. We assignedmywith random variable of a binomial distribution with a probability of 70%. So that means, about 70% of the time we will see1, which we had said before thatM = 0will be our missingness mechanism, that means 30% will be missing, randomly. Then, we turn these ones and zeros to logical data. Next, we create an empty vectorymto slot in data whenMy == T, else assignNA`. Then we estimate our model with observed data.

Our X coefficient is quite close to the real one. though with 30% of data missing, you can see that the confidence interval is much wider than the original data. This is also known as loss of precision (increasing the uncertainty of the estimations).

MAR

As you can see on the DAG above, M is affected by X. That means the missingness mechanism is not completely at random. Something else caused it. The mathematical notation would be:

\begin{gather} P(M=0|Y_{obs},Y_{mis},\psi) = P(M=0|Y_{obs},\psi) \end{gather}

The missingness mechanism is the probability of missingness depends on the observed information of any design factor, which also means for MAR, we can ignore Y_mis, and hence our mechanism is $P(M=0|Y_{obs},\psi)$. Now let’s simulate and visualize!

```

MARmy <- rbinom(n,1,plogis(2-2x)) |> as.logical()for (i in 1:n) { ym[i] <- ifelse(my[i]==T,y[i],NA)}summary(lm(ym~x+z))## ## Call:## lm(formula = ym ~ x + z)## ## Residuals:## Min 1Q Median 3Q Max ## -2.8750 -0.3840 0.0631 0.6970 1.8169 ## ## Coefficients:## Estimate Std. Error t value Pr(>|t|) ## (Intercept) -0.1665 0.1279 -1.302 0.196810 ## x 0.2158 0.1596 1.352 0.180365 ## z 0.5004 0.1288 3.886 0.000218 ## ---## Signif. codes: 0 '' 0.001 '' 0.01 '' 0.05 '.' 0.1 ' ' 1## ## Residual standard error: 0.9926 on 75 degrees of freedom## (22 observations deleted due to missingness)## Multiple R-squared: 0.2172,Adjusted R-squared: 0.1964 ## F-statistic: 10.41 on 2 and 75 DF, p-value: 0.0001026compare_plot(z,x,y,ym,"MAR")

`` Let’s explain. We assignedmynow with the influence ofxand then turn it into boolean. Same procedure, if my isTRUE, then slot in data, if not, make itNA. Our estimation now is, same as MCAR, imprecise, with a wider confidence interval, and also theX coefficient` is different from the true estimate. Same thing with the intercept as well. Usually, it is advised that we impute these numbers so that we don’t throw out data.

MNAR / NMAR

\begin{gather} P(M=0|Y_{obs},Y_{mis},\psi) = P(M=0|Y_{obs},Y_{mis},\psi) \end{gather}

Now, this is a very interesting one! I never understood this until I can visualize this on a DAG. Anytime when the variable itself (that is missing) affects the missingness mechanism, it’s MNAR! Another name for MNAR is also Not Missing At Random (NMAR). In real life example would be missing lactate results in patients. For example, lactate is usually not ordered when not indicated, which also usually means the the lactate value is most likely normal or low. Hence, low/normal value of lactate will likely to go missing. Imputation in this setting may produce a biased result.

my <- rbinom(n,1,plogis(-2*x+2*y)) |> as.logical()for (i in 1:n) { ym[i] <- ifelse(my[i]==T,y[i],NA)}summary(lm(ym~x+z))## ## Call:## lm(formula = ym ~ x + z)## ## Residuals:## Min 1Q Median 3Q Max ## -1.74498 -0.59748 0.02568 0.60912 2.15993 ## ## Coefficients:## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 0.5183 0.1362 3.805 0.000364 ***## x 0.6595 0.1502 4.391 5.29e-05 ***## z 0.2861 0.1286 2.224 0.030329 * ## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## Residual standard error: 0.8888 on 54 degrees of freedom## (43 observations deleted due to missingness)## Multiple R-squared: 0.3624,Adjusted R-squared: 0.3388 ## F-statistic: 15.34 on 2 and 54 DF, p-value: 5.287e-06compare\_plot(z,x,y,ym,"MNAR") Now our estimate is REALLY off! Complete case analysis here is not appropriate at all.

Different variety of Missing Data DAGs/notation/representation DAG 1. Enders, Craig K.. Applied Missing Data Analysis (Methodology in the Social Sciences Series) (p. 6). Guilford Publications.This is the DAG that we have adopted on this article. 2. Statistical Rethinking by Richard McElreathHere, Richard uses a concept of dog (missingness mechanism) eating homework as an example. Hence, D is the missingness mechanism. Otherwise looks quite similar to our DAG. 3. [Canonical Causal Diagrams to Guide the Treatment of Missing Data in Epidemiologic] Studies(https://doi.org/10.1093/aje/kwy173)W here represents unmeasured variables which affect the missingness mechanism that are not affected by any of the variables in the study. Missingness mechanisms of the variable were prefixed by M, for example My indicated missingness mechanism of Y variable, Mx of X, Mz1 of Z2.

Representation Through out the blog, we assign M = 0 as missing, where M = 1 is not missing. But different textbooks will have this reversed. Hence, missingness notation/representation and also DAGs are not universally consistent.

Opportunity for improvement * Next time, we will explore a bit more on the solutions to each missingness mechanisms, what is recoverability, ignorabiity? Which imputation technique to use, what kinds are there? * If MNAR is a non-outcome or exposure variable, does it matter much? * use ggdag to draw DAGs next time instead of copy and pasting from dagitty

References * Understanding missing data mechanisms using causal DAGs * van Buuren, Stef. Flexible Imputation of Missing Data, Second Edition (Chapman & Hall/CRC Interdisciplinary Statistics) (p. 36). CRC Press. * Enders, Craig K.. Applied Missing Data Analysis (Methodology in the Social Sciences Series) (p. 6). Guilford Publications. * [Canonical Causal Diagrams to Guide the Treatment of Missing Data in Epidemiologic] Studies(https://doi.org/10.1093/aje/kwy173)

Lessons Learnt * Missingness notation/representation/DAGs are not universally consistent. We have presented a few versions above. * We learnt what MCAR, MAR, and MNAR mean through DAG and simulation * [Canonical Causal Diagrams to Guide the Treatment of Missing Data in Epidemiologic] Studies(https://doi.org/10.1093/aje/kwy173) is a great resource! Full of information

Answer: The missingness mechanism of the title is… MNAR. Only vowels were missing, meaning vowels caused the title to be missing vowels.

If you like this article:

please feel free to send me a comment or visit my other blogs
please feel free to follow me on twitter, GitHub or Mastodon
if you would like collaborate please feel free to contact me

To leave a comment for the author, please follow the link and comment on their blog: r on Everyday Is A School Day.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: V_s__l_ng M_ss_ng D_t_ W_th D_G & S_m_l_t__n

View Details

[This article was first published on R | Discindo, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.With DuckDB releasingversion 1.0.0 on June 3rd, and MotherDuck following with the general availability announcement on June 11th, it is a perfectopportunity to see how both can be used from R. I work for an organization whereR is the default language for doing most of the analytics, so being able to dothis is more than just simple curiosity.

My setupI am running R version 4.4.1 on Linux, with duckdb version 1.0.0.

I have python version 3.12, and I am installing duckb 1.0.0 in a virtual environment.

I installed the duckdb version 1.0.0 binary as well, and installed the motherduck extension.

And, of course, I also have created an account on MotherDuck.

Running duckdb in RThis has been probably covered many times sofar. Nevertheless, for completeness, running duckdb in R is fairly straightforward:

```

Connect to an in-memory DuckDB databasecon <- dbConnect(duckdb::duckdb(), ":memory:")# Write the Iris dataset to the DuckDB in-memory databasedbWriteTable(con, "iris", iris)# Query data from the DuckDB databasedata <- dbGetQuery(con, "SELECT * FROM iris")

``` Of course, there is the possibility of doing things with duckplyr, but we are not going to go into that.

Connecting to MotherDuckMotherDuck documentation has details about connecting to MotherDuck using python, but not for connecting using R.

After asking a few questions on the discord, I learned that the process should be similar.

Let’s see how that looks.

PythonIn a Python 3.12 virtual environment, that has duckdb-1.0.0, getting toMotherDuck is simple, in fact as the documentation says:

import duckdb# connect to MotherDuck using 'md:' or 'motherduck:'con = duckdb.connect('md:') The above results in getting a notification on the terminal:

Attempting to automatically open the SSO authorization page in your default browser.1. Please open this link to login into your account: https://auth.motherduck.com/activate2. Enter the following code: XXXX-XXXX Nothing else is required here. We click in the browser, establish a connection, get a token, etc.

RHowever, doing the same with R doesn’t have the same outcome:

con <- DBI::dbConnect(duckdb::duckdb(), "md:") Creates a local database called md::

ls -lh md\:-rw-r--r-- 1 novica novica 12K jun 21 10:32 md: My best guess here is that the duckdb package for R does not automaticallyfigure out that it should load the motherduck extension, as is probably the casein python.

The approach in R is then similar to what is suggested in the sectionConnecting to MotherDuck after opening a local DuckDB database:

```

Create a local databasecon <- DBI::dbConnect(duckdb::duckdb(), "local.duckdb")# Note: Installing the extension is not nececsary here since# I already have it installed on my system. However, if you don't want# to go to the trouble of installing the duckdb binary and then installing# extensions, then it is possible to do the installation here before# loading with:# DBI::dbExecute(con, "INSTALL 'motherduck';")# Load the Mother Duck extensionDBI::dbExecute(con, "LOAD 'motherduck';")# Verify that the extension is loadedDBI::dbGetQuery(con,"SELECT extension_name, loaded, installed FROM duckdb_extensions() WHEREextension_name = 'motherduck'")# Connect to MotherDuckDBI::dbExecute(con, "PRAGMA MD_CONNECT")

``` At which point the message for authenticating in the browser shows up in the terminal.

After approving the connection, as the friendly message says, the token can bestored in an environment variable to avoid having to log in again.

Then it is a simple matter of querying things on MotherDuck:

```

Query the sample data about air quality that is avaiable on MotherDuckDBI::dbGetQuery(con,"SELECT country_name, city, \"year\", pm10_concentration, pm25_concentration, no2_concentration, FROM sample_data.who.ambient_air_quality WHERE city = 'Skopje' OR city = 'Oslo';")

`` Note that we have to specify the database name:sample_data, and theschema:who` to query the data on MotherDuck.

The results can be assigned to an object in R, which I did. And since it isalways fun to make a plot, here is how it looks for the two cities where Ispend most of my time.

And, that’s a wrap. I mean, a quack.

To leave a comment for the author, please follow the link and comment on their blog: R | Discindo.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Joining the flock from R: working with data on MotherDuck

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Dear rOpenSci friends, it’s time for our monthly news roundup!

You can read this post on our blog.Now let’s dive into the activity at and around rOpenSci!

rOpenSci HQrOpenSci takes over maintenance of the {goodpractice} packageThe {goodpractice} package was started by Gábor Csárdi in 2016 to auto-magically provide advice on good practices for your own R package.rOpenSci’s Dev Guide has recommended using it from the first day we started writing it in 2018.The package is now a central part of our own internal {pkgcheck} system, which is run automatically on all new submissions, and we recommend that all authors use our ‘pkgcheck-action’ GitHub action, which also runs {goodpractice}.

We are pleased to announce that rOpenSci has now taken over maintenance of the {goodpractice} package, thanks to the approval both of the original author Gábor, and the previous maintainers at ascent.io.The package has now been moved to our ropensci-review-tools GitHub organization, which holds all software used in our automated checking system.This also means that documentation for the package is now built by our own documentation system, and will live from here on at docs.ropensci.org/goodpractice/.

rOpenSci at CZI Open Science 2024From June 10 to June 14 Noam Ross, Mauro Lepore and Yanina Bellini Saibene participated on the CZI Open Science 2024 event.

On Wednesday, we showcased the Champions Program, sharing Champions’ projects, training materials, and the results of the two-year pilot. We had the chance to chat and learn about many other projects during these sessions.

Yanina participated in the closing panel on Case Study Session 3: Demonstrating Impact of Open Science to explore the challenges of using traditional academic metrics to measure project impact and emphasize alternative approaches. In her talk, Yani introduced the work done by different rOpenSci members, the tools and metrics we use to capture their stories, and the impact we achieve together.

The rOpenSci community at upcoming eventsMeet rOpenSci team and community members at events in the near future!

Two talks by rOpenSci team members, and more contributions by rOpenSci community members, to look forward to at useR! 2024 in Salzburg, Austria.
- Maëlle Salmon’s keynote talk “How your code might get rusty” on Wednesday, July 10 at 09:20 CEST;
- Jeroen Ooms’ talk “Navigating the R Ecosystem Using R-Universe” on Thursday, July 11, 11:30 – 11:50 CEST.
- Jon Harmon’s virtual talk “Learning Together at the Data Science Learning Community” will go live on the useR!2024 YouTube playlist at 10:30am CDT on July 2;
- Paola Corrales’ and Elio Campitelli’s tutorial “Efficient Data Analysis with data.table” on Monday, July 3 (pre-registration required);
- Elio Campitelli’s talk “Building Bilingual Bridges with Multilingual Manuals” on Tuesday, July 4 at 11:40 CEST.
- Hugo Gruson’s talk ”Building Interoperability in Existing Software Ecosystems with S3 Classes” on Tuesday, July 9 at 14:50, and his poster “A reproducible analysis of CRAN Task Views to understand the state of an R package ecosystem”
- Lluís Revilla’s and Henrik Bengtsson’s poster about CRAN packages archived and a the cranhaven.org R-universe created to reduce the impact of that on users.
- Will Landau’s and Charlie Gao’s talk “Moju-Kapu: How {Mirai} and {Crew} Are Powering the Next Generation of Parallel Computing in R” on Tuesday, July 9 at 11:00 CEST.
- Binod Jung Bogati’s talks “Generate Raw Synthetic Dataset for Clinical Trial – Binod Jung Bogati, Numeric Mind” on Tuesday, July 9 at 13:30 CEST; and “Translate R for Global Reach” on Thursday, July 11 at 12:10 CEST.
- At posit::conf(2024) in Seattle, US, you’ll get to meet some champions and mentors!
- Look for: Andrea Gomez Vargas; Yi-Chin Sunny Tseng; Luis D. Verde Arregoitia; Francisco Cardozo; Jonathan Keane.
- Don’t miss Luis’ lightning talk “Why’d you load that package for?” on Tuesday, Aug 13 at 1:00 PM PDT
- We’re excited to share that rOpenSci community manager Yanina Bellini Saibene will deliver a keynote talk at BioNT Community Event & CarpentryConnect-Heidelberg 2024, on November 14th in Heidelberg, Germany.

CoworkingRead all about coworking!

Tuesday, July 2nd, 14:00 Europe Central (12:00 UTC), Git and GitHub with cohost Zhian Kamvar and Steffi LaZerte.
- Read up on Git and GitHub and how they might serve you
- Start your first Git/GitHub project
- Chat with our cohost about the pros and cons of Git & GitHub, resources for getting started and tips and tricks.
- Tuesday, August 6th, 9:00 America Pacific (16:00 UTC), Building your first R package with cohost Carolina Pradier and Steffi LaZerte.
- Explore how to make R packages
- Plan out that package you’ve always wanted to create
- Chat with our cohost about tips and tricks for making your first R package

And remember, you can always cowork independently on work related to R, work on packages that tend to be neglected, or work on what ever you need to get done!

Software New packagesThe following three packages recently became a part of our software suite:

goodpractice, developed by Mark Padgham together with Karina Marks, Daniel de Bortoli, Gabor Csardi, Hannah Frick, Owen Jones, and Hannah Alexander: Give advice about good practices when building R packages. Advice includes functions and syntax to avoid, package structure, code complexity, code formatting, etc. It is available on CRAN.
mregions2, developed by Salvador Fernandez-Bejarano together with Lotte Pohl: Explore and retrieve marine geo-spatial data from the Marine Regions Gazetteer and the Marine Regions Data Products, including the Maritime Boundaries. It has been reviewed.
rOPTRAM, developed by Micha Silver: The OPtical TRapezoid Model (OPTRAM) derives soil moisture based on the linear relation between a vegetation index and Land Surface Temperature (LST). The Short Wave Infra-red (SWIR) band is used as a proxy for LST. See: Sadeghi, M. et al., 2017. https://doi.org/10.1016/j.rse.2017.05.041 .

Discover more packages, read more about Software Peer Review.

New versionsThe following nine packages have had an update since the last newsletter: goodpractice (v1.0.5), beastier (v2.5.1), c14bazAAR (5.0.0), comtradr (v1.0.1), DataPackageR (v0.16.0), dynamite (1.5.2), readODS (v2.3.0), rgbif (v3.8.0), and targets (1.7.1).

Software Peer ReviewThere are fourteen recently closed and active submissions and 6 submissions on hold. Issues are at different stages:

Three at ‘6/approved’:
- karel, Learning programming with Karel the robot. Submitted by Marcos Prunello.
- rOPTRAM, Derive soil moisture using the OPTRAM algorithm. Submitted by Micha Silver.
- mregions2, Access Data from Marineregions.org: The Marine Regions Gazetteer and the Marine Regions Data Products. Submitted by salvafern.
- Four at ‘4/review(s)-in-awaiting-changes’:
- cancerprof, API Client for State Cancer Profiles. Submitted by Brian Park.
- osmapiR, OpenStreetMap API. Submitted by Joan Maspons.
- rix, Rix: Reproducible Environments with Nix. Submitted by Bruno Rodrigues.
- agromet, Índices y Estadísticos Climáticos e Hidrológicos. Submitted by Paola Corrales.
- Five at ‘3/reviewer(s)-assigned’:
- eDNAjoint, Joint Modeling of Traditional and Environmental DNA Survey Data. Submitted by Abigail Keller.
- chopin, CHOPIN: Computation for Climate and Health research On Parallelized INfrastructure. Submitted by Insang Song.
- rsi, Efficiently Retrieve and Process Satellite Imagery. Submitted by Michael Mahoney.
- sits, Satellite Image Time Series Analysis for Earth Observation Data Cubes. Submitted by Gilberto Camara.
- fwildclusterboot, Fast Wild Cluster Bootstrap Inference for Linear Models. Submitted by Alexander Fischer. (Stats).
- One at ‘2/seeking-reviewer(s)’:
- QuadratiK, A Collection of Methods Using Kernel-Based Quadratic Distances for. Submitted by Giovanni Saraceno. (Stats).
- One at ‘1/editor-checks’:
- gigs, Assess Fetal, Newborn, and Child Growth with International Standards. Submitted by Simon Parker. (Stats).

Find out more about Software Peer Review and how to get involved.

On the blogSoftware Review* From scripts to package. Developing dendroNetwork and learning with rOpenSci by Ronald Visser. How scripts were transformed into a package and what I learned in the process.

Tech Notes* A fresh new look for R-universe! by Jeroen Ooms. We have given the WebUI for R-universe a big refresh. This is the biggest UX overhaul in since the beginning of the project.

Calls for contributionsCalls for maintainersIf you’re interested in maintaining any of the R packages below, you might enjoy reading our blog post What Does It Mean to Maintain a Package?.

historydata, datasets for historians. Issue for volunteering.
USAboundaries (and USAboundariesdata), historical and contemporary boundaries of the United States of America . Issue for volunteering.

Also refer to our help wanted page – before opening a PR, we recommend asking in the issue whether help is still needed.

Package development cornerSome useful tips for R package developers.

Make your functions compa-tibbleDo the functions of your package use data.frame as input?Do not miss Hugo Gruson’s post Make your functions compa-tibble as users of your package might well try and pass a tibble, which you probably don’t want to be a showstopper!

Use lintr to enforce your package’s function preferencesDo you want to commit to using the cli package instead of base R messaging?You can configure the lintr settings for your codebase to pick up usage of certain functions, to inform you along with the preferred replacement.See, as an example, pkgdown’s lintr configuration file and the corresponding GitHub Actions workflow (from r-lib/actions).This neat safeguard makes use of the Undesirable function linter.

More metadata on CRANCRAN pages of packages now show…

Issues which need fixing! Shared by Henrik Bengtsson.
DOIs! Have you seen Achim Zeileis’ announcement that All R packages on CRAN will get an official DOI?In contrast to Zenodo DOIs, it’s the same DOI across all versions.

A pure GitHub preview workflow for pkgdown websitesIf you use a gh-pages branch on GitHub to store the source of your pkgdown website, and use GitHub Pages to deploy it, you could extend that workflow to create (and then clean) subdirectories in that branch to host previews of pull requests.Check out this GitHub Actions workflow file by Garrick Aden-Buie.

Tips for refactoring test filesDo you put the object as close as possible to the related expectation(s)?Read about this, and other, tips for refactoring test files.

One more tool for checking inputs of your R functionsDo you check inputs of your R functions?Beside the aforelinked R-hub blog post by Hugo Gruson, Sam Abbott, Carl Pearson, you might be interested in the experimental stbl package by Jon Harmon.

If you haven’t subscribed to our newsletter yet, you can do so via a form. Until it’s time for our next newsletter, you can keep in touch with us via our website and Mastodon account.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci News Digest, June 2024

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This is part one of a two part series on {vetiver}.

Part 1: Vetiver: First steps inMLOps
Part 2: Vetiver: Model Deployment (This post)

IntroductionIn our previous blog, we provided an overview of MLOps and the{vetiver} package, creating anddeploying a simple model locally. In this post, we’ll show you how todeploy a model to production using PositConnect,SageMaker, and Docker.

What is DockerDocker is an open-source platform that allowsdevelopers to build, deploy, and run containers. These containers bundleapplication source code with the operating system libraries anddependencies needed to run that code.

Previously, we discussed deploying a ShinyApplicationusing Docker. Similarly, we can deploy a set of APIs to access ourmodel.

Data comes in all shapes and sizes. It can often be difficult to know where to start. Whatever your problem, Jumping Rivers can help.

Creating a Docker fileThe {vetiver} package simplifies creating a Dockerfile. We simply run:

vetiver::vetiver\_prepare\_docker( pins::board\_connect(), "colin/k-nn", docker\_args = list(port = 8080)) This command accomplishes several tasks:

Uses the {renv}package to create a list of R package dependencies required to runyour model.
Creates a file named plumber.R containing the necessary code todeploy an API, essentially just vetiver_api().
Generates the Dockerfile.

The Dockerfile includes several components. The first component sets theR version, specifies the package repository, and crucially, installs thenecessary system libraries.

FROM rocker/r-ver:4.4.0ENV RENV\_CONFIG\_REPOS\_OVERRIDE https://packagemanager.rstudio.com/cran/latestRUN apt-get update -qq && apt-get install -y --no-install-recommends \ ... The second component copies the renv.lock file and installs the requiredR packages:

COPY vetiver\_renv.lock renv.lockRUN Rscript -e "install.packages('renv')"RUN Rscript -e "renv::restore()" Finally, we have the plumber/API section

COPY plumber.R /opt/ml/plumber.REXPOSE 8080ENTRYPOINT ["R", "-e", "pr <- plumber::plumb('/opt/ml/plumber.R'); pr$run(host = '0.0.0.0', port = 8080)"] which runs the API on port 8080.

The container is built via

docker build --tag my-first-model . The --tag flag allows you to name your Docker image. You can inspectyour stored Docker images with:

docker image listREPOSITORY TAG IMAGE ID CREATED SIZEmy-first-model latest 792af21c775a About a minute ago 1.33GB To run the image, use

docker run --rm --publish 8080:8080 my-first-model Posit Connect / Sage MakerWe can also trivially publish the model to Posit Connect via

vetiver::vetiver\_deploy\_rsconnect(board = pins::board\_connect(), "colin/k-nn") Similarly, we can publish to SageMaker using the functionvetiver_deploy_sagemaker().

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Vetiver: Model Deployment

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Ever wondered what it takes to develop a product?

Imagine you’re choosing a company to make your product a reality. What makes you choose one over another? Is it their cool website or their experience with enterprises from Fortune500?

When it comes to building quality software solutions, there really aren’t any shortcuts. Explore the benefits of doing it right the first time in this blog post.

Let’s look into what makes Appsilon tick and what goes into our work beyond just writing the code.

Making An ImpactAt Appsilon, making an impact isn’t just a click-bait; it’s at the heart of everything we do. While it may sound abstract, it’s the cornerstone of a successful project. Our journey begins with a deep dive into understanding your business problem and deciphering what project success means to you. This sets the stage for figuring out how to measure progress and ensuring we have the right team for the job.

Once we’re up and running, we’ve got a solid plan and a clear target in sight. Our approach has been tried and tested, helping us navigate any bumps in the road that come our way.

Our ProcessKey Phases of Appsilon’s Project Development ProcessThe DeliveryWe use the SCRUM framework to ensure efficient project execution and preparedness for potential uncertainties.

Appsilon’s Agile Sprint Cycle‍

Our approach includes bi-weekly sprint iterations, daily scrum meetings, sprint planning sessions, and regular sprint reviews. These SCRUM events form a continuous feedback loop, allowing us to tweak our plans as needed.
‍

Moreover, we place great emphasis on thorough documentation and testing, employing unit tests, automated end-to-end tests, and frequent deployments to ensure the utmost quality of our applications.

Appsilon’s Sprint Workflow: A 10-Day Development CycleThis process is field-tested and praised by clients who have worked with us. It’s not just about churning out code; it’s about providing holistic support throughout the project. Our Delivery Managers are the secret ingredient, aiding in requirements gathering and ensuring you’re always in the loop, enabling us to proactively tackle any challenges that may arise.

Effective communication about ROI can transform how your app ideas are perceived. Learn more about how to calculate the ROI for your next R/Shiny project.

Why Appsilon?So, what does it really mean to team up with Appsilon? It’s about jumping into a journey together where your dreams mix with our skills, and your success becomes our goal. More than just making a product, it’s about making a real difference in your business.

“We experience the Appsilon developers to be highly competent. They have been very good at understanding our core problems and finding solutions that increase the quality and robustness of our functions and applications step by step. Appsilon’s control, understanding and continuous improvement of our Posit infrastructure leaves us free to focus on domain. As a result we’re able to raise quality of code and business logic simultaneously.”
– Andrea Nicolaysen Carlsson – Technology Manager Electrodes at Elkem ASA

‍

From understanding what you need to deliver the final product and beyond, we promise to be open, keep you in the loop, and always strive for the best. With our trusted process and dedicated Delivery Managers, we’re here to not just meet but surpass your expectations.

At Appsilon, we’re not just about coding; we’re about finding solutions. Solutions that click, drive progress, and help you reach your goals. If you’re ready to take your project to the next level, let’s do it together.

Interested in seeing how we deliver business-oriented digital solutions? Explore how we transformed digital operations for a Fortune 500 Movie Studio.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How Appsilon Delivers Impactful Products: What To Expect When You Work With Us

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Embracing the Tidyverse Style GuideHey there, fellow R coder! When it comes to writing code in R, making it functional is just the beginning. Trust me, I’ve been there — debugging code at 2 AM, wondering what I was thinking when I wrote that line. This is where the Tidyverse Style Guide comes to the rescue, transforming your functional code into a masterpiece of readability and maintainability.

Why Coding Style MattersImagine reading a book with no punctuation or structure. Nightmare, right? The same goes for code. Good coding style ensures that your future self and your colleagues can comprehend and extend your work. As they say, “Today I know and God knows, but in a week only God will know how this should work.”

Files and DirectoriesFile Naming ConventionsProper file naming is crucial. Imagine rummaging through a folder named “folder2” — frustrating, right? Descriptive, meaningful names make it easier for others to understand the purpose of each file at a glance.

Good Example:

data\_analysis.R Bad Example:

Data Analysis.R Pros: Clear, concise, and consistent naming conventions make files easy to understand and manage, enhancing collaboration and avoiding issues with operating systems.

Cons: Inconsistent naming can lead to confusion, errors, and inefficiencies in managing and collaborating on projects.

Directory StructureA well-organized directory structure helps in navigating the project efficiently. It separates data, scripts, and results, making it easier to locate and manage files.

Good Example:

project/├── data/├── scripts/└── results/ Bad Example:

project/├── folder1/├── folder2/└── random\_folder/ Pros: A clear directory structure improves readability, navigation, and file management. It enhances collaboration by providing a standardized layout.

Cons: Poor organization leads to confusion, difficulty in finding files, increased errors, and reduced collaboration efficiency.

SyntaxIndentation and SpacingThink of indentation and spacing as the grammar of your code. Proper indentation and spacing make your code more readable and maintainable. The tidyverse style guide recommends using two spaces per indentation level and avoiding tabs.

Good Example:

if (condition) { do\_something()} Bad Example:

if(condition){do\_something()} Pros: Using consistent indentation and spacing enhances readability and ensures that your code looks clean and professional. It makes it easier for others to follow your logic.

Cons: Inconsistent indentation makes the code hard to read and understand, leading to potential errors and misinterpretations.

Line Length and BreaksKeeping lines under 80 characters and breaking lines after operators improve code readability, especially on smaller screens.

Good Example:

my\_function <- function(arg1, arg2) { long\_expression <- arg1 + arg2 return(long\_expression)} Bad Example:

my\_function <- function(arg1, arg2) { long\_expression <- arg1 + arg2 return(long\_expression)} Pros: Maintaining a maximum line length and breaking lines appropriately makes your code easier to read and prevents horizontal scrolling.

Cons: Ignoring this practice can lead to cramped and hard-to-follow code, making debugging and collaboration more challenging.

Naming ConventionsAdopting consistent naming conventions, such as snake_case for object names and UpperCamelCase for function names, helps in making the code more predictable and easier to understand.

Good Example:

data\_frame <- data.frame(x = 1:10, y = 10:1) Bad Example:

DataFrame <- data.frame(x = 1:10, y = 10:1) Pros: Consistent naming conventions enhance readability and maintainability by providing a clear and predictable structure to your code. Cons: Inconsistent naming can cause confusion and errors, making it harder for others (and your future self) to understand and work with the code.

FunctionsWriting FunctionsFunctions should have clear, descriptive names and be designed to perform a single task. This improves readability and maintainability.

Good Example:

add\_numbers <- function(a, b) { return(a + b)} Bad Example:

addnumbers <- function(a,b){return(a+b)} Pros: Clear, descriptive names and single-task functions make code easier to understand and maintain.

Cons: Ambiguous names and multifunctional code increase complexity, making it harder to debug and extend.

Function ArgumentsUse default arguments where appropriate and document all arguments and return values. This makes functions more flexible and easier to use.

Good Example:

plot\_data <- function(data, x\_col, y\_col, color = “blue”) { plot(data[[x\_col]], data[[y\_col]], col = color)} Bad Example:

plot\_data <- function(data, x\_col, y\_col, color) { plot(data[[x\_col]], data[[y\_col]], col = color)} Pros: Default arguments provide flexibility and make functions easier to use. Proper documentation aids in understanding.

Cons: Lack of defaults and documentation can lead to misuse and confusion.

Return ValuesEnsure functions always return a value and that the return type is consistent. This makes the behavior of functions predictable and easier to debug.

Good Example:

add\_numbers <- function(a, b) { return(a + b)} Bad Example:

add\_numbers <- function(a, b) { result <- a + b # No return statement} Pros: Consistent return values make functions predictable and easier to integrate.

Cons: Inconsistent or missing return values create ambiguity, making debugging and integration challenging.

PipesUsing PipesPipes, introduced by the magrittr package and widely used in the tidyverse, streamline code by chaining operations in a readable manner.

Good Example:

library(dplyr)data %>% filter(x > 1) %>% summarise(mean\_y = mean(y)) Bad Example:

library(dplyr)summarise(filter(data, x > 1), mean\_y = mean(y)) Pros: Pipes enhance readability by breaking down operations into clear, sequential steps, making complex data transformations easier to follow. Cons: Without pipes, code becomes nested and harder to read, increasing the likelihood of errors and making debugging more difficult.

Pipe PracticesTo ensure clarity, avoid performing complex operations within a single pipe chain. Instead, break down steps to maintain readability. This example is little bit exaggerated, because we have only 6 lines, but it is not unusual to have pipe made of 30 or more lines, and this rule should be used in that case.

Good Example:

data\_cleaned <- data %>% filter(!is.na(x)) %>% mutate(z = x + y)result <- data\_cleaned %>% group\_by(category) %>% summarise(mean\_z = mean(z)) Bad Example:

result <- data %>% filter(!is.na(x)) %>% mutate(z = x + y) %>% group\_by(category) %>% summarise(mean\_z = mean(z)) Pros: Breaking down pipe chains improves readability and makes each step understandable and debuggable.

Cons: Long, complex pipes can be difficult to follow and troubleshoot, reducing code clarity and increasing maintenance difficulty.

ggplot2Breaking Code on OperatorsBreaking code on operators enhances readability and maintains a clean structure. This practice is particularly useful when dealing with long lines of code.

Good Example:

ggplot(data, aes(x = x, y = y)) + geom\_point() + theme\_minimal() + labs(title = “Scatter Plot”, x = “X Axis”, y = “Y Axis”) Pros: Each operation is on a new line, making the code easier to read and modify.

Proper Order of LayersMaintaining a proper order of layers in ggplot2 ensures that each layer is applied correctly, making the visualization more accurate and aesthetically pleasing.

Good Example:

ggplot(data, aes(x = x, y = y)) + geom\_point() + geom\_smooth(method = “lm”) + theme\_minimal() Pros: The smoothing layer is applied on top of the points, and the theme is applied last, ensuring a clean and logical structure.

DocumentationIn-Code DocumentationIn-code documentation using comments helps others (and your future self) understand the logic and purpose of your code. It’s important to strike a balance between too many and too few comments.

Good Example:

```

Calculate the mean of a numeric vectorcalculate_mean <- function(x) { mean(x)}

``` Pros: Provides clear, concise information about the function’s purpose.

Roxygen2 for FunctionsUsing Roxygen2 for documenting functions ensures comprehensive, consistent, and machine-readable documentation. This is particularly useful for creating package documentation.

Good Example:

```

’ Calculate the mean of a numeric vector#’#’ @param x A numeric vector#’ @return The mean of the vector#’ @exportcalculate_mean <- function(x) { mean(x)}

``` Pros: Provides a structured and detailed description, making it easy to generate documentation files automatically.

Good in-code documentation and comprehensive function documentation using Roxygen2 enhance code readability, usability, and maintainability. Poor documentation leads to confusion, errors, and increased time spent understanding and debugging code.

Miscellaneous Style GuidelinesAssignment Using <- Not =The assignment operator <- is preferred over = for clarity and consistency in R code.

Good Example:

x <- 10 Pros: Clear distinction between assignment and equality checks.

Proper SpacingUsing proper spacing, especially near operators, enhances code readability.

Good Example:

result <- a + b Pros: Improves readability and reduces errors.

Avoiding Reserved NamesAvoid using reserved names like c, T, or F as variable names to prevent conflicts with built-in functions and constants.

Good Example:

vec <- c(1, 2, 3) Pros: Avoids conflicts with the c() function.

Code OrganizationOrganizing code using empty lines and breaking long lines helps in maintaining a clean and readable structure.

Good Example:

calculate\_sum <- function(a, b) { result <- a + b return(result) } Pros: Use of empty lines and line breaks improves readability and structure.

ConclusionBy embracing the Tidyverse Style Guide for R coding, you’re not just writing code; you’re crafting a readable, maintainable, and collaborative masterpiece. These guidelines will help you avoid those 2 AM debugging sessions and make your code a joy to work with. Consistent coding style reduces errors, improves project efficiency, and facilitates long-term maintenance. Embrace these guidelines to enhance your coding practices and project success. Happy coding, and remember, good style is the key to long-term coding happiness!

PS. Ugly and unreadable code will work either way, but you will not like to work with this code.

Writing R Code the “Good Way” was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Writing R Code the “Good Way”

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionToday I am going to go over some quick yet practical examples of ways that you can use the healthyR.ts package. This package is designed to help you analyze time series data in a more efficient and effective manner.

Let’s just jump right into it!

Load the libraries library(healthyR.ts)library(dplyr)library(ggplot2)library(tidyr)library(plotly)library(timetk)library(modeltime) Load the dataWe are going to use the timeseries data called BJSales.lead that comes with Base R. We will do this to showcase a couple of things like turning a ts object into a tibble and plotting the data.

```

Load the data, which has no time series information other than it is# a time series object and 150 points in length, so we will go ahead and# create a date column for it and name it date_col.df <- BJsales.lead |> ts_to_tbl() |> mutate(date_col = seq.Date(from = as.Date("1991-01-01"), by = "month", length.out = 150)) |> select(date_col, everything())# Print the first few rows of the datahead(df)

```

A tibble: 6 × 2 date_col value 1 1991-01-01 10.0 2 1991-02-01 10.1 3 1991-03-01 10.3 4 1991-04-01 9.755 1991-05-01 10.3 6 1991-06-01 10.1

`` So far, we have loaded the data and created a date column for it. Now, let’s plot the data. We are going to use thets_vva_plot` function to do this.

```

Plot the dataplt_data <- ts_vva_plot(df, date_col, value)head(plt_data[["data"]][["augmented_data_tbl"]])

```

A tibble: 6 × 3 date_col name value 1 1991-01-01 Value 10.0 2 1991-01-01 Velocity NA 3 1991-01-01 Acceleration NA 4 1991-02-01 Value 10.1 5 1991-02-01 Velocity 0.06006 1991-02-01 Acceleration NA

```

plt\_data[["plots"]][["interactive\_plot"]] Now we have created the augmented data that gets the first order difference of the time series velocity and then the second order difference which gets us the acceleration. The function then creates a ggplot2 plot and a plotly plot of the data. Let’s move on to see the growth rate of this data.

```

Plot the growth rate of the datadf_growth_augment_tbl <- ts_growth_rate_augment( df, value)head(df_growth_augment_tbl)

```

A tibble: 6 × 3 date_col value growth_rate_value 1 1991-01-01 10.0 NA 2 1991-02-01 10.1 0.5993 1991-03-01 10.3 2.48 4 1991-04-01 9.75 -5.52 5 1991-05-01 10.3 5.95 6 1991-06-01 10.1 -1.94

``` Let’s now view the data:

plt <- df\_growth\_augment\_tbl |> pivot\_longer(cols = -date\_col) |> ggplot(aes(x = date\_col, y = value, color = name)) + facet\_wrap(~ name, ncol = 1, scales = "free") + geom\_line() + theme\_minimal() + labs( x = "Date", y = "Value", title = "Growth Rate of Time Series Data", color = "Variable" )print(plt)

ggplotly(plt) Stationary?Is the data stationary? Meaning does the joint probability of the distribution change when shifted in time? Let’s find out.

ts\_adf\_test(df[["value"]])

$test\_stat[1] -1.723664$p\_value[1] 0.6915227 The p-value from this test is 0.692. This means that we can accept the null hypothesis that the data is non-stationary. We can, however, make the data stationary by using a built in function in this package.

auto\_stationary\_df <- auto\_stationarize(df[["value"]])

The time series is not stationary. Attempting to make it stationary...

stationary\_vec <- auto\_stationary\_df[["stationary\_ts"]]ndiffs <- auto\_stationary\_df[["ndiffs"]]trans\_type <- auto\_stationary\_df[["trans\_type"]]test\_stat <- auto\_stationary\_df[["adf\_stats"]][["test\_stat"]]p\_value <- auto\_stationary\_df[["adf\_stats"]][["p\_value"]] The data is now stationary after 1 differencing. The transformation type used was diff. The test statistic was -4.839 and the p-value was 0.01.

Let’s now add the stationary data to the df_growth_augment_tbl and plot it. First in order to do this we are going to have to pad the data since it is shorter than the original data. We will simply add an NA to the vector then attach.

stationary\_vec <- c(rep(NA, ndiffs), stationary\_vec)df\_growth\_augment\_tbl <- df\_growth\_augment\_tbl |> mutate(stationary = stationary\_vec)df\_growth\_augment\_tbl |> pivot\_longer(cols = -date\_col) |> ggplot(aes(x = date\_col, y = value, color = name)) + facet\_wrap(~ name, ncol = 1, scales = "free") + geom\_line() + theme\_minimal() + labs( x = "Date", y = "Value", title = "Growth Rate/Value and Stationary Data of Time Series", color = "Variable" ) It’s close to the growth rate as it is the first order difference of the data.

Now, lets see if there is any lags that are present in the data.

output <- ts\_lag\_correlation(df\_growth\_augment\_tbl, .date\_col = date\_col, .value\_col = value, .lags = c(1,2,3,4,6,12,24))output[["plots"]][["plotly\_lag\_plot"]] To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Practical Examples with healthyR.ts

View Details

[This article was first published on R on Nicola Rennie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Back in September 2022 I submitted {PrettyCols}, an R package containing aesthetically pleasing colour palettes, to CRAN. If you missed it, you can read theblog post introducing the package!

Over a year after the last CRAN release, it was time for an update and this blog post will give you a brief overview of some of the new features and palettes! You can install {PrettyCols} using:

| 1 | install.packages("PrettyCols") |

You can also install the development version fromGitHub using:

| 1 | remotes::install\_github("nrennie/PrettyCols") |

Breaking changesPrevious versions of {PrettyCols} used the name argument in all of the {ggplot2} scale functions to define the name of the palette you wanted to use. Unfortunately, {ggplot2} uses name to define the name of the legend. This meant there was an inconsistency between {PrettyCols} and other {ggplot2} scale functions, and I wanted to fix that in this release. Now, the behaviour is more consistent.

Change: the palette argument is now used to define the name of the colour palette you want to use.

| 1234567 | library(ggplot2)ggplot( data = mtcars, mapping = aes(x = cyl, fill = as.factor(gear))) + geom\_bar() + scale\_fill\_pretty\_d(palette = "Peppers") |

For example, if you have previously scale_fill_manual(name = "Bright") to say you want to use the "Bright" palette, this will result in an error. Please use scale_fill_manual(palette = "Bright") instead. If you didn’t explicitly name the argument for the palette name, your code should still work.

Changing the legend nameTo get around the problem above, earlier versions of {PrettyCols} added a legend_title argument to the scale_*_pretty_*() functions. The legend_title argument still works, but you will get a deprecation warning. To maintain consistency with other {ggplot2} scale functions, you should use name instead.

Change: the name argument sets the legend (scale) name.

| 123456 | ggplot( data = mtcars, mapping = aes(x = mpg, y = disp, colour = wt)) + geom\_point(size = 5) + scale\_colour\_pretty\_c("Roses", name = "New Title") |

Diverging scalesThere were several diverging palettes already within {PrettyCols}, and in version 1.1.0 you can now use the palettes with diverging scales in {ggplot2} more easily.

| 123456789 | ggplot( data = mtcars, mapping = aes(x = mpg, y = disp, fill = wt)) + geom\_point(size = 5, pch = 21) + scale\_fill\_pretty\_div( palette = "PurpleYellows", midpoint = mean(mtcars$wt) ) |

The following diverging scales functions now exist:

scale_colour_pretty_div() and scale_color_pretty_div()
scale_fill_pretty_div()

They work similarly to scale_*_gradient2().

More palettes!The new release contains eight additional palettes, taking the total up to 40 palettes! You can see the complete list of available palettes by running view_all_palettes() or browsing thepackage vignette on available palettes.

Python supportA Python implementation of {PrettyCols} existed in previous releases, but Python support has now moved to a separate GitHub repository. Seegithub.com/nrennie/PrettyPyCols.*

To end this post, here’s another couple of examples of these palettes being used for generative art!

Show code

| 1 2 3 4 5 6 7 8 9101112 | aRt::split\_grid( col\_palette = prettycols("Coast"), grid\_col = prettycols("Coast")[3], n\_x = 6, n\_y = 6)aRt::window\_boxes( col\_palette = prettycols("Disco"), n\_x = 20, n\_y = 20) |

If you find that something in {PrettyCols} isn’t working, or if you’ve got an idea for more features, please raise aGitHub issue!

To leave a comment for the author, please follow the link and comment on their blog: R on Nicola Rennie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: What’s new in {PrettyCols} 1.1.0?

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Following the release and an introductory blog post on Tapyr, our Shiny for Python framework, we recently hosted a Shiny Gathering to further explore its features and capabilities. ‍

Ready to simplify your PyShiny dashboard development? Learn how to get started with Tapyr in this introductory guide.

‍Here’s a summary of the key points covered during the session, presented by Piotr Pasza Storożenko, the Innovation Lead at Appsilon and creator of Tapyr.

Session OverviewPasza guided us through the Tapyr framework, emphasizing its goal to simplify the development of deployment-ready PyShiny applications.

Here Are The Major Takeaways:Getting Started with Tapyr Setup and Installation: Tapyr can be set up both locally and in the cloud. The session demonstrated creating a new repository from the Tapyr template and cloning it. The installation process involves using Poetry for dependency management and packaging in Python. * Running the Example App*: Pasza showcased running the default Tapyr app using Poetry, highlighting the ease of installing necessary packages, activating the environment, and running the application. This example app illustrated Tapyr’s ability to create interactive, Shiny-like applications.

Need help getting started with Tapyr? Take a look at the documentation.

Editing and Customizing the App Customization: Modifying the UI components and server logic in Tapyr is straightforward. Pasza demonstrated how easily developers can customize the app to suit their needs, showcasing Tapyr’s flexibility. * Deployment*: Tapyr is designed for efficient deployment. Developers can deploy their applications to platforms like Posit Connect using simple commands, ensuring their projects are shareable and accessible.

Testing with Tapyr Robust Testing Capabilities*: Tapyr supports both unit tests and end-to-end tests using Pytest and Playwright. This ensures the reliability and functionality of applications through comprehensive testing of individual components and overall user interactions.

Why Tapyr?Tapyr stands out among other Python frameworks like Streamlit or Dash. Tapyr leverages Shiny’s reactive programming model, offering a balanced approach between ease of use and production readiness, making it an excellent choice for developing interactive web applications in Python.

Watch The Full SessionFor a detailed walkthrough of Tapyr, including live demonstrations and further insights from Pasza, watch the full session on YouTube.

Interested in learning more about Tapyr and other open-source packages? Subscribe to our newsletter to have them delivered to your inbox.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Open Source Spotlight Recap: Tapyr (Deployment-Ready PyShiny Template)

View Details

[This article was first published on Tony's Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Introduction“The best predictor for future performance is Expected Goals”, authored by Sander Ijtsma of 11tegen11 in 2015, is one of the most notable articles in public soccer analytics. The write-up provided compelling evidence for the superiority of expected goals (xG) in terms of forecasting team-season outcomes on a game-by-game basis.12 Ijtsma drew out curves for running R-squared values of a handful of season-to-date (“past”) metrics with respect to rest-of-season (“future”) performance indicators–goals ratio and points per game–and found that xG ratio3 tended to be the most predictive.

In 2022, Eliot McKinley wrote a cool piece for American Soccer Analytics (ASA) where he replicated and extended Ijtsma’s analysis. He used a larger data set of more recent seasons–2018 through 2022– and added a look at Major League Soccer (MLS) in addition to the the Big 5 European Leagues, and adding. Further, Eliot presented a novel aspect where he bootstrapped the procedure for generating running R-squared values to generate smoother, more interpretable curves.

In this post I extend this prior art in two ways:

I expand the data set range to 2018 through 2024, sourcing from a reputed public data source–FBref.4 The intention is to make the analysis even more robust and reproducible.
I include additional curves for “gamestate-aware” goals and xG, in an effort to see if one can achieve metrics that are perhaps even more predictive than xG itself.

On the second point, Ijtsma noted that subsetting the past performance tallies to gamestates when the score is either tied or within 1 goal might abate the potential biases introduced by within-match complacency (manifesting in conservative tactics). In fact, he said he had evaluated these things.

I’ve looked at this, but I have decided not to include that subanalysis in this post, for the sake of accessibility… I think this method will show that the phenomenon [where teams exert more or less effort in matches against particular opposition] either does not truly exist, or that its effect is so small that correcting for this will allow more noise and thereby weaken the model.

Unfortunately, I couldn’t find any subsequent blog post where Ijtsma writes about the role of gamestate in the context of his R-squared analysis, so, alas, I’m here to do exactly that.

Now, my point isn’t necessarily to find the absolute best metric possible to use to forecast future team performance. Tiotal Football (with assistance from Eliot) showed that comprehensive event-based metrics like goals added (light blue), completed passes into the area in front of the box (yellow), etc. are even more predictive of future points per game than xG, at least in the MLS.

I do look at a few metrics that go beyond Ijtsma’s set in the Appendix, but that is not the primary focus of this post.

DataI use {worldfootballR} to retrieve match-level figures and shot logs from FBref. As noted in the intro, we’re going to be working with Big 5 and MLS data for the 2018 – 2024 (season-ending) seasons.

Code Setup ```

data ingestionlibrary(worldfootballR)library(dplyr)library(stringr)library(tidyr)## plottinglibrary(ggplot2)library(sysfonts)library(showtext)library(ggtext)library(htmltools)## local devlibrary(withr)library(qs)## parallelization for bootstrap calcslibrary(parallel)library(future)library(furrr)PROJ_DIR <- 'posts/xg-predictor-future-results'## Extract the from "47880eb7" from "https://fbref.com/en/matches/47880eb7/Liverpool-Manchester-City-November-10-2019-Premier-League"extract_fbref_match_id <- function(match_url) { basename(dirname(match_url))}

Data Setup

Discrepancies in worldfootballR's saved datafix_team <- function(x) { dplyr::case_match( x, 'Montreal Impact' ~ 'CF Montréal', 'Sporting Kansas City' ~ 'Sporting KC', .default = x )}COUNTRY <- c('ENG', 'ESP', 'FRA', 'GER', 'ITA', 'USA')GENDER <- 'M'TIER <- '1st'SEASON_END_YEAR <- c(2018:2024)## for match-level shot dataraw_team_summary <- worldfootballR::load_fb_advanced_match_stats( country = COUNTRY, gender = GENDER, tier = TIER, season_end_year = SEASON_END_YEAR, stat_type = 'summary', team_or_player = 'team')#> → Data last updated 2024-05-21 18:39:59 UTCraw_team_passing <- worldfootballR::load_fb_advanced_match_stats( country = COUNTRY, gender = GENDER, tier = TIER, season_end_year = SEASON_END_YEAR, stat_type = 'passing', team_or_player = 'team')#> → Data last updated 2024-05-21 18:39:59 UTC## for shot-level data for gamestate calcsraw_shots <- worldfootballR::load_fb_match_shooting( country = COUNTRY, gender = GENDER, tier = TIER, season_end_year = SEASON_END_YEAR)#> → Data last updated 2024-05-21 17:52:12 UTC## for team info for gamestate calcsraw_match_summaries <- worldfootballR::load_fb_match_summary( country = COUNTRY, gender = GENDER, tier = TIER, season_end_year = SEASON_END_YEAR)#> → Data last updated 2024-05-21 18:14:02 UTC

Match-level data ingestion

Known/valid matches with missing xG## - https://fbref.com/en/matches/e0a20cfe/Hellas-Verona-Roma-September-19-2020-Serie-A: awarded to Hellas Verona## https://fbref.com/en/matches/c34bbc21/Bochum-Monchengladbach-March-18-2022-Bundesliga: awarded to Monchengladbachdrop_bad_matches <- function(df) { df |> dplyr::filter( ## choose non-playoff/relegation weeks (Country != 'USA' & stringr::str_detect(Matchweek, 'Matchweek')) | (Country == 'USA' & Matchweek == 'Major League Soccer (Regular Season)') ) |> ## Fix effectively dup records due to change in MatchURLs dplyr::filter(!(Country == 'GER' & Season_End_Year == 2024 & stringr::str_detect(MatchURL, 'Leverkusen') & !stringr::str_detect(MatchURL, 'Bayer-Leverkusen'))) |> ## Drop MLS 2024 since it's incomplete at time of writing dplyr::filter( !(Country == 'USA' & Season_End_Year == 2024) ) |> ## Drop COVID-19-affected seasonf for all leagues ## (Ligue 1 is espeically odd, as each team has about 7 games missing.) dplyr::filter( Season_End_Year != 2020 )}team_summary <- raw_team_summary |> drop_bad_matches() |> dplyr::transmute( season = Season_End_Year, country = Country, gender = Gender, tier = Tier, # match_week = stringr::str_extract(Matchweek, '[0-9]+') |> as.integer(), ## won't works for MLS ## We'll use this to define our own notion of match week, where we order games ## by date rather than use the FBref matchweek provided at face value, since a ## a match could have been rescheduled, leading to situations where the labeled ## matchweek 2 comes before the matchweek 1, for example. date = Match_Date, match_id = extract_fbref_match_id(MatchURL), team = Team, is_home = Home_Away == 'Home', g = Gls, xg = xG_Expected, shots = Sh, sot = SoT ) |> ## passing metrics for further exploration dplyr::left_join( raw_team_passing |> drop_bad_matches() |> dplyr::transmute( match_id = extract_fbref_match_id(MatchURL), team = Team, ppa = PPA, xag = xAG ), by = dplyr::join_by(match_id, team) ) |> dplyr::mutate( xg_xag = dplyr::coalesce(xg, 0) + dplyr::coalesce(xag, 0) ) |> dplyr::group_by(season, team) |> dplyr::mutate( season_game_count = dplyr::n(), game_idx = dplyr::row_number(date) ) |> dplyr::ungroup()combined_team_summary <- dplyr::left_join( team_summary, team_summary |> dplyr::rename_with( (.x) paste0(.x, '_conceded'), c( g, xg, shots, sot, ppa, xag, xg_xag ) ) |> dplyr::select( match_id, opponent = team, dplyr::ends_with('conceded') ), by = dplyr::join_by(match_id), relationship = 'many-to-many') |> dplyr::filter(team != opponent) |> dplyr::mutate( pts = dplyr::case_when( g > g_conceded ~ 3L, g < g_conceded ~ 0L, g == g_conceded ~ 1L ), pts_conceded = dplyr::case_when( g > g_conceded ~ 0L, g < g_conceded ~ 3L, g == g_conceded ~ 1L ) ) |> dplyr::arrange(team, season, date)dplyr::glimpse(combined_team_summary)#> Rows: 25,416#> Columns: 27#> $ season 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2023, 2…#> $ country "FRA", "FRA", "FRA", "FRA", "FRA", "FRA", "FRA", …#> $ gender "M", "M", "M", "M", "M", "M", "M", "M", "M", "M",…#> $ tier "1st", "1st", "1st", "1st", "1st", "1st", "1st", …#> $ date "2022-08-05", "2022-08-14", "2022-08-21", "2022-0…#> $ match_id "47ab569b", "8b559e8f", "9befca9a", "d8aa49d4", "…#> $ team "Ajaccio", "Ajaccio", "Ajaccio", "Ajaccio", "Ajac…#> $ is_home FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, TRUE, FALS…#> $ g 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 4, 2, 1, 0…#> $ xg 1.5, 0.5, 1.4, 1.7, 1.2, 0.5, 1.2, 0.5, 1.5, 0.2,…#> $ shots 7, 9, 9, 14, 15, 16, 9, 6, 12, 6, 14, 8, 5, 9, 12…#> $ sot 3, 2, 2, 2, 0, 3, 3, 4, 3, 1, 6, 1, 1, 5, 3, 2, 0…#> $ ppa 6, 4, 3, 5, 7, 11, 9, 4, 5, 2, 8, 3, 5, 8, 6, 7, …#> $ xag 0.4, 0.3, 0.6, 0.7, 0.7, 0.4, 1.2, 0.4, 0.7, 0.1,…#> $ xg_xag 1.9, 0.8, 2.0, 2.4, 1.9, 0.9, 2.4, 0.9, 2.2, 0.3,…#> $ season_game_count 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 3…#> $ game_idx 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15…#> $ opponent "Lyon", "Lens", "Rennes", "Lille", "Montpellier",…#> $ g_conceded 2, 0, 2, 3, 2, 1, 1, 0, 3, 1, 1, 3, 1, 2, 2, 0, 2…#> $ xg_conceded 1.3, 1.4, 1.2, 1.7, 1.7, 0.8, 1.3, 0.5, 1.0, 1.8,…#> $ shots_conceded 10, 12, 16, 6, 9, 7, 12, 8, 7, 9, 9, 11, 18, 9, 1…#> $ sot_conceded 5, 3, 7, 4, 4, 4, 3, 4, 3, 1, 2, 6, 5, 5, 5, 4, 4…#> $ ppa_conceded 7, 13, 5, 7, 5, 3, 5, 7, 6, 12, 7, 16, 3, 7, 5, 9…#> $ xag_conceded 0.5, 1.2, 0.8, 0.7, 1.7, 0.7, 1.1, 0.3, 1.0, 0.8,…#> $ xg_xag_conceded 1.8, 2.6, 2.0, 2.4, 3.4, 1.5, 2.4, 0.8, 2.0, 2.6,…#> $ pts 0, 1, 0, 0, 0, 0, 0, 3, 0, 1, 1, 0, 0, 3, 1, 3, 0…#> $ pts_conceded 3, 1, 3, 3, 3, 3, 3, 0, 3, 1, 1, 3, 3, 0, 1, 0, 3…

Gamestate data ingestion match_summaries <- raw_match_summaries |> dplyr::group_by(MatchURL) |> dplyr::mutate( match_summary_rn = dplyr::row_number(dplyr::desc(Event_Time)), match_has_no_penalties = all(Event_Type != 'Penalty') ) |> dplyr::ungroup() |> dplyr::mutate( match_has_no_goals = Away_Score == 0 & Home_Score == 0 ) |> ## Drop non-shot events, e.g. card and substitution events. ## Always keep the first timeline event, so that we're not accidentally dropping matches. dplyr::filter( Event_Type %in% c('Goal', 'Own Goal', 'Penalty') | ## don't drop games with no goals (match_has_no_goals & match_has_no_penalties & match_summary_rn == 1) ) |> dplyr::transmute( season = Season_End_Year, country = Country, gender = Gender, tier = Tier, match_week = stringr::str_extract(Matchweek, '[0-9]+') |> as.integer(), match_id = extract_fbref_match_id(MatchURL), date = lubridate::ymd(Match_Date), home_team = fix_team(Home_Team) , away_team = fix_team(Away_Team), period = as.integer(Event_Half), ## ensure that minutes always has a value minutes = dplyr::case_when( period == 1L & Event_Time > 45L ~ 45L, period == 2L & Event_Time > 90L ~ 90L, .default = Event_Time ) |> as.integer(), minutes_added = dplyr::case_when( period == 1L & Event_Time > 45 ~ Event_Time - 45L, period == 2L & Event_Time > 90 ~ Event_Time - 90L, .default = NA_integer_ ), home_g = as.integer(gsub('[:].$', '', Score_Progression)), ## after event away_g = as.integer(gsub('^.[:]', '', Score_Progression)), is_own_goal = Event_Type == 'Own Goal', team = Team, player = Event_Players )deduped_match_summaries <- match_summaries |> ## Some matches are recorded twice if they were rescheduled dplyr::semi_join( match_summaries |> dplyr::distinct(match_id, date) |> dplyr::group_by(match_id) |> dplyr::slice_max(date, n = 1) |> dplyr::ungroup() )shots <- raw_shots |> dplyr::transmute( season = Season_End_Year, country = Country, gender = Gender, tier = Tier, match_id = extract_fbref_match_id(MatchURL), period = as.integer(Match_Half), ## convert "45+2" to "45" minutes = ifelse( grepl('[+]', Minute), as.integer(gsub('(^[0-9]+)+', '\1', Minute)), as.integer(Minute) ), ## convert "45+2" to "2" minutes_added = ifelse( grepl('[+]', Minute), as.integer(gsub('(^[0-9]+)+', '\2', Minute)), NA_integer_ ), is_home = Home_Away == 'Home', team = fix_team(Squad), player = Player, is_goal = Outcome == 'Goal', is_on_target = !is.na(PSxG), xg = as.double(xG), xgot = as.double(PSxG) )shots_with_own_goals <- dplyr::bind_rows( shots |> dplyr::transmute( match_id, period, minutes, minutes_added, is_home, team, player, is_goal, xg, xgot, is_on_target, is_own_goal = FALSE ), ## synthetic events for own goals deduped_match_summaries |> dplyr::filter( is_own_goal ) |> dplyr::transmute( match_id, period, minutes, minutes_added, is_home = team == home_team, team, player, is_goal = TRUE, xg = NA_real_, xgot = NA_real_, is_on_target = NA, is_own_goal = TRUE ))clean_shots <- shots_with_own_goals |> ## To get meta-information about the game dplyr::inner_join( deduped_match_summaries |> dplyr::distinct(match_id, home_team, away_team), by = dplyr::join_by(match_id), relationship = 'many-to-one' ) |> dplyr::mutate( home_g = dplyr::case_when( ## Note that fotmob would list the away team for an own goal but fbref ## lists the home team (is_goal | is_own_goal) & is_home ~ 1L, is_own_goal & is_home ~ 1L, TRUE ~ 0L ), away_g = dplyr::case_when( (is_goal | is_own_goal) & !is_home ~ 1L, TRUE ~ 0L ), home_xg = dplyr::case_when( is_home ~ dplyr::coalesce(xg, 0), TRUE ~ 0 ## even for own goals ), away_xg = dplyr::case_when( !is_home ~ dplyr::coalesce(xg, 0), TRUE ~ 0 ), home_xgot = dplyr::case_when( is_home ~ dplyr::coalesce(xgot, 0), TRUE ~ 0 ), away_xgot = dplyr::case_when( !is_home ~ dplyr::coalesce(xgot, 0), TRUE ~ 0 ) ) |> dplyr::select( match_id, period, minutes, minutes_added, is_home, is_goal, is_on_target, is_own_goal, player, home_team, away_team, home_g, away_g, home_xg, away_xg, home_xgot, away_xgot ) |> ## Still need to distinct because, even with fixing team names, there have been FBref MatchURL changes dplyr::distinct() |> dplyr::group_by(match_id) |> ## Differentiate between shots in the same minute. dplyr::mutate( shot_idx = dplyr::row_number((minutes + dplyr::coalesce(minutes_added, 0L))) ) |> dplyr::ungroup() |> dplyr::mutate( shot_id = sprintf('%s-%02d', match_id, shot_idx), .before = 1 )restacked_shots <- dplyr::bind_rows( clean_shots |> dplyr::filter(is_home) |> dplyr::transmute( shot_id, match_id, period, minutes, minutes_added, is_home, is_goal, is_on_target, is_own_goal, player, team = home_team, opponent = away_team, g = home_g, g_conceded = away_g, xg = home_xg, xg_conceded = away_xg, xgot = home_xgot, xgot_conceded = away_xgot ), clean_shots |> dplyr::filter(!is_home) |> dplyr::transmute( shot_id, match_id, period, minutes, minutes_added, is_home, is_goal, is_on_target, is_own_goal, player, team = away_team, opponent = home_team, g = away_g, g_conceded = home_g, xg = away_xg, xg_conceded = home_xg, xgot = away_xgot, xgot_conceded = home_xgot ))doublecounted_restacked_shots <- dplyr::bind_rows( restacked_shots |> dplyr::mutate(pov = 'primary', .before = 1), restacked_shots |> ## re-assign to temporary variable names first, so that way we don't accidentlaly overwrite information dplyr::rename( team1 = team, team2 = opponent, g1 = g, g2 = g_conceded, xg1 = xg, xg2 = xg_conceded, xgot1 = xgot, xgot2 = xgot_conceded ) |> ## then formally re-assign columns dplyr::rename( team = team2, opponent = team1, g = g2, g_conceded = g1, xg = xg2, xg_conceded = xg1, xgot = xgot2, xgot_conceded = xgot1 ) |> dplyr::mutate( is_home = !is_home ) |> dplyr::mutate( pov = 'secondary', .before = 1 )) |> dplyr::arrange(shot_id, pov)cumu_doublecounted_restacked_shots <- doublecounted_restacked_shots |> dplyr::group_by( match_id, team ) |> dplyr::mutate( dplyr::across( c(g, g_conceded), list(cumu = cumsum) ) ) |> dplyr::ungroup() |> dplyr::mutate( gamestate = g_cumu - g_conceded_cumu )gamestate_shots <- cumu_doublecounted_restacked_shots |> dplyr::inner_join( deduped_match_summaries |> dplyr::distinct( match_id, season, country, gender, tier, match_week, date, home_team, away_team ), by = dplyr::join_by(match_id) ) |> dplyr::transmute( pov, match_id, season, country, gender, tier, match_week, date, home_team, away_team, team, player, shot_id, period, minutes, minutes_added, time = minutes + dplyr::coalesce(minutes_added, 0L), g, g_conceded, xg, xg_conceded, xgot, xgot_conceded, xgd = xg - xg_conceded, gamestate_gd0 = dplyr::case_when( gamestate == 0 ~ 'neutral', gamestate < 0 ~ 'trailing', gamestate > 0 ~ 'leading' ), gamestate_abs_gd1 = dplyr::case_when( abs(gamestate) <= 1 ~ 'neutral', gamestate < 1 ~ 'trailing', gamestate > 1 ~ 'leading' ) ) |> dplyr::group_by(match_id, team) |> dplyr::arrange(shot_id, .by_group = TRUE) |> dplyr::mutate( pre_shot_gamestate_gd0 = dplyr::lag(gamestate_gd0, default = 'neutral'), pre_shot_gamestate_abs_gd1 = dplyr::lag(gamestate_abs_gd1, default = 'neutral') ) |> dplyr::ungroup() |> dplyr::arrange(team, season, date, shot_id)LAST_MIN_BUFFER <- 3last_min_pad <- gamestate_shots |> dplyr::select( match_id, season, country, gender, tier, match_week, date, team, pre_shot_gamestate_gd0, pre_shot_gamestate_abs_gd1, period, time ) |> dplyr::group_by(match_id, team, period) |> dplyr::slice_max(time, n = 1, with_ties = FALSE) |> dplyr::ungroup() |> dplyr::mutate( xg = 0, xg_conceded = 0, xgot = 0, xgot_conceded = 0, last_regular_min = ifelse(period == 1L, 45L, 90L), time = pmax(last_regular_min + LAST_MIN_BUFFER, time + 1) )padded_gamestate_shots <- dplyr::bind_rows( gamestate_shots, last_min_pad) |> dplyr::arrange(match_id, period, time)gamestate_shots_and_durations <- padded_gamestate_shots |> dplyr::group_by(match_id, team) |> dplyr::mutate( prev_period = dplyr::lag(period), prev_time = dplyr::lag(time) ) |> dplyr::ungroup() |> dplyr::mutate( duration = dplyr::case_when( period == 1L & is.na(prev_period) ~ time - 0L, period == 2L & period != prev_period ~ time - 45L, TRUE ~ time - prev_time ) )aggregate_games_by_gamestate <- function(df, gamestate_col) { gamestate_sym <- rlang::ensym(gamestate_col) agg_shots_by_game_gamestate <- df |> dplyr::group_by( country, gender, tier, team, season, match_id, date, !!gamestate_sym, ) |> ## need all the na.rm = TRUE's since i don't fill in pov for the padded minute records dplyr::summarize( shots = sum(pov == 'primary', na.rm = TRUE), shots_conceded = sum(pov == 'secondary', na.rm = TRUE), sot = sum(pov == 'primary' & xgot > 0, na.rm = TRUE), sot_conceded = sum(pov == 'primary' & xgot_conceded > 0, na.rm = TRUE), dplyr::across( c( g, g_conceded, xg, xg_conceded, xgot, xgot_conceded, duration ), (.x) sum(.x, na.rm = TRUE) ) ) |> dplyr::ungroup() |> dplyr::arrange( country, gender, tier, season, team, season, date, !!gamestate_sym ) agg_shots_by_game_gamestate |> dplyr::distinct( country, gender, tier, season, team, date, match_id ) |> tidyr::crossing( !!gamestate_sym := c('trailing', 'neutral', 'leading') ) |> dplyr::left_join( agg_shots_by_game_gamestate, by = dplyr::join_by( country, gender, tier, season, team, !!gamestate_sym, date, match_id ) ) |> dplyr::mutate( dplyr::across( c( shots, shots_conceded, sot, sot_conceded, g, g_conceded, xg, xg_conceded, xgot, xgot_conceded, duration ), (.x) dplyr::coalesce(.x, 0) ) ) |> tidyr::pivot_wider( names_from = !!gamestate_sym, values_from = c( shots, shots_conceded, sot, sot_conceded, g, g_conceded, xg, xg_conceded, xgot, xgot_conceded, duration ), names_sort = TRUE )}## output has the same format as gamestate_abs_gd1_by_gamegamestate_gd0_by_game <- gamestate_shots_and_durations |> aggregate_games_by_gamestate('pre_shot_gamestate_gd0')gamestate_abs_gd1_by_game <- gamestate_shots_and_durations |> aggregate_games_by_gamestate('pre_shot_gamestate_abs_gd1')dplyr::glimpse(gamestate_abs_gd1_by_game)#> Rows: 29,852#> Columns: 40#> $ country "ENG", "ENG", "ENG", "ENG", "ENG", "ENG", "…#> $ gender "M", "M", "M", "M", "M", "M", "M", "M", "M"…#> $ tier "1st", "1st", "1st", "1st", "1st", "1st", "…#> $ team "Arsenal", "Arsenal", "Arsenal", "Arsenal",…#> $ season 2018, 2018, 2018, 2018, 2018, 2018, 2018, 2…#> $ date 2017-08-11, 2017-08-19, 2017-08-27, 2017-0…#> $ match_id "e3c3ddf0", "37f97732", "e4374144", "2fab7d…#> $ shots_leading 0, 0, 0, 11, 0, 8, 9, 0, 8, 0, 0, 10, 0, 8,…#> $ shots_neutral 27, 18, 1, 6, 11, 8, 16, 9, 22, 17, 4, 4, 1…#> $ shots_trailing 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 2…#> $ shots_conceded_leading 0, 0, 0, 7, 0, 0, 4, 0, 3, 0, 0, 10, 0, 1, …#> $ shots_conceded_neutral 6, 11, 10, 0, 13, 7, 5, 11, 6, 5, 9, 4, 8, …#> $ shots_conceded_trailing 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2…#> $ sot_leading 0, 0, 0, 5, 0, 2, 1, 0, 3, 0, 0, 3, 0, 4, 0…#> $ sot_neutral 10, 6, 0, 4, 2, 4, 7, 6, 11, 5, 1, 2, 2, 2,…#> $ sot_trailing 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 1…#> $ sot_conceded_leading 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…#> $ sot_conceded_neutral 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…#> $ sot_conceded_trailing 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…#> $ g_leading 0, 0, 0, 1, 0, 0, 0, 0, 2, 0, 0, 0, 0, 3, 0…#> $ g_neutral 4, 0, 0, 2, 0, 2, 2, 1, 3, 2, 0, 2, 1, 2, 0…#> $ g_trailing 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1…#> $ g_conceded_leading 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0…#> $ g_conceded_neutral 3, 1, 2, 0, 0, 0, 0, 2, 1, 1, 3, 0, 0, 0, 3…#> $ g_conceded_trailing 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…#> $ xg_leading 0.00, 0.00, 0.00, 1.57, 0.00, 0.65, 0.72, 0…#> $ xg_neutral 2.57, 1.46, 0.09, 0.66, 1.53, 1.56, 1.71, 0…#> $ xg_trailing 0.00, 0.00, 0.48, 0.00, 0.00, 0.00, 0.00, 0…#> $ xg_conceded_leading 0.00, 0.00, 0.00, 0.57, 0.00, 0.00, 0.21, 0…#> $ xg_conceded_neutral 1.50, 0.71, 1.65, 0.00, 0.79, 0.88, 0.20, 1…#> $ xg_conceded_trailing 0.00, 0.00, 1.52, 0.00, 0.00, 0.00, 0.00, 0…#> $ xgot_leading 0.00, 0.00, 0.00, 3.06, 0.00, 0.23, 0.07, 0…#> $ xgot_neutral 3.34, 1.09, 0.00, 1.72, 0.18, 2.01, 2.67, 1…#> $ xgot_trailing 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0…#> $ xgot_conceded_leading 0.00, 0.00, 0.00, 0.63, 0.00, 0.00, 0.00, 0…#> $ xgot_conceded_neutral 2.11, 1.02, 2.21, 0.00, 0.53, 0.26, 0.08, 1…#> $ xgot_conceded_trailing 0.0, 0.0, 2.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0…#> $ duration_leading 0, 0, 0, 70, 0, 26, 38, 0, 22, 0, 0, 57, 0,…#> $ duration_neutral 99, 99, 40, 27, 96, 70, 59, 98, 77, 96, 81,…#> $ duration_trailing 0, 0, 56, 0, 0, 0, 0, 0, 0, 0, 15, 0, 0, 0,… ``` Methods and AnalysisReplicating prior artLet’s begin with replicating Ijtsma’s visualization where he evaluated the running R-squared of the five measures of past performance with respect to two measures of future performance. The five measures of past performance are:

points per game
goal ratio
total shots ratio
shots on target ratio
xG ratio

The two measures of future performance are:

goals ratio
points per game

Calculate rolling R-squared for season-to-date and rest-of-season performance measures accumulate\_team\_summary <- function(df, op, .prefix) { df |> dplyr::arrange(team, season, op(game\_idx)) |> dplyr::group\_by(season, team) |> dplyr::mutate( dplyr::across( c( shots, shots\_conceded, sot, sot\_conceded, g, g\_conceded, xg, xg\_conceded, xgot, xgot\_conceded, g\_neutral, g\_conceded\_neutral, xg\_neutral, xg\_conceded\_neutral, xgot\_neutral, xgot\_conceded\_neutral, ppa, ppa\_conceded, xag, xag\_conceded, xg\_xag, xg\_xag\_conceded, duration\_leading, pts, pts\_conceded ), \(.x) cumsum(dplyr::coalesce(.x, 0)) ) ) |> dplyr::ungroup() |> dplyr::mutate( shot\_ratio = shots / (shots + shots\_conceded), sot\_ratio = sot / (sot + sot\_conceded), g\_ratio = g / (g + g\_conceded), xg\_ratio = xg / (xg + xg\_conceded), xgot\_ratio = xgot / (xgot + xgot\_conceded), shot\_neutral\_ratio = shots\_neutral / (shots\_neutral + shots\_conceded\_neutral), sot\_neutral\_ratio = sot\_neutral / (sot\_neutral + sot\_conceded\_neutral), g\_neutral\_ratio = g\_neutral / (g\_neutral + g\_conceded\_neutral), xg\_neutral\_ratio = xg\_neutral / (xg\_neutral + xg\_conceded\_neutral), xgot\_neutral\_ratio = xgot\_neutral / (xgot\_neutral + xgot\_conceded\_neutral), ppa\_ratio = ppa / (ppa + ppa\_conceded), xag\_ratio = xag / (xag + xag\_conceded), xg\_xag\_ratio = xg\_xag / (xg\_xag + xg\_xag\_conceded), duration\_leading\_per\_game = duration\_leading / game\_idx, ppg = pts / game\_idx ) |> dplyr::mutate( ## replace NaNs with NAs for the cor calculation dplyr::across( dplyr::ends\_with('ratio'), \(.x) tidyr::replace\_na(.x, 0.5) ) ) |> dplyr::rename\_with( .fn = \(.x) paste0(.prefix, '\_', .x), .cols = c( shots, shots\_conceded, sot, sot\_conceded, g, g\_conceded, xg, xg\_conceded, xgot, xgot\_conceded, shots\_neutral, shots\_conceded\_neutral, sot\_neutral, sot\_conceded\_neutral, g\_neutral, g\_conceded\_neutral, xg\_neutral, xg\_conceded\_neutral, xgot\_neutral, xgot\_conceded\_neutral, ppa, ppa\_conceded, xag, xag\_conceded, xg\_xag, xg\_xag\_conceded, duration\_leading, pts, pts\_conceded, dplyr::ends\_with('ratio'), duration\_leading\_per\_game, ppg ) )}calculate\_nested\_r2 <- function(data, col, target\_col) { purrr::map\_dbl( data, \(.x) { cor( .x[[col]], .x[[target\_col]], use = 'complete.obs' )^2 } )}calculate\_rolling\_r2s <- function(combined\_df) { past\_team\_summary <- combined\_df |> accumulate\_team\_summary(`+`, .prefix = 'past') future\_team\_summary <- combined\_df |> accumulate\_team\_summary(`-`, .prefix = 'future') accumulated\_df <- dplyr::inner\_join( past\_team\_summary |> dplyr::select( season, team, country, game\_idx, dplyr::starts\_with('past') ), future\_team\_summary |> dplyr::select( season, team, game\_idx, dplyr::starts\_with('future') ), by = dplyr::join\_by(season, team, game\_idx) ) |> dplyr::mutate( league\_group = ifelse(country == 'USA', 'MLS', 'Big 5'), .keep = 'unused', .before = 1 ) accumulated\_df |> tidyr::nest(data = -c(league\_group, game\_idx)) |> dplyr::mutate( past\_shot\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_shot\_ratio', 'future\_g\_ratio'), past\_sot\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_sot\_ratio', 'future\_g\_ratio'), past\_g\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_g\_ratio', 'future\_g\_ratio'), past\_xg\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_xg\_ratio', 'future\_g\_ratio'), past\_xgot\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_xgot\_ratio', 'future\_g\_ratio'), past\_shot\_neutral\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_shot\_neutral\_ratio', 'future\_g\_ratio'), past\_sot\_neutral\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_sot\_neutral\_ratio', 'future\_g\_ratio'), past\_g\_neutral\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_g\_neutral\_ratio', 'future\_g\_ratio'), past\_xg\_neutral\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_xg\_neutral\_ratio', 'future\_g\_ratio'), past\_xgot\_neutral\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_xgot\_neutral\_ratio', 'future\_g\_ratio'), ## Bonus 1: Non-gamestate, non-shot features past\_ppa\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_ppa\_ratio', 'future\_g\_ratio'), past\_xag\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_xag\_ratio', 'future\_g\_ratio'), past\_xg\_xag\_ratio\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_xg\_xag\_ratio', 'future\_g\_ratio'), ## Bonus 2: duration leading per game past\_duration\_leading\_per\_game\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_duration\_leading\_per\_game', 'future\_g\_ratio'), past\_ppg\_\_future\_g\_ratio = calculate\_nested\_r2(data, 'past\_ppg', 'future\_g\_ratio'), past\_shot\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_shot\_ratio', 'future\_ppg'), past\_sot\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_sot\_ratio', 'future\_ppg'), past\_g\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_g\_ratio', 'future\_ppg'), past\_xg\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_xg\_ratio', 'future\_ppg'), past\_xgot\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_xgot\_ratio', 'future\_ppg'), past\_shot\_neutral\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_shot\_neutral\_ratio', 'future\_ppg'), past\_sot\_neutral\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_sot\_neutral\_ratio', 'future\_ppg'), past\_g\_neutral\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_g\_neutral\_ratio', 'future\_ppg'), past\_xg\_neutral\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_xg\_neutral\_ratio', 'future\_ppg'), past\_xgot\_neutral\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_xgot\_neutral\_ratio', 'future\_ppg'), past\_ppa\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_ppa\_ratio', 'future\_ppg'), past\_xag\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_xag\_ratio', 'future\_ppg'), past\_xg\_xag\_ratio\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_xg\_xag\_ratio', 'future\_ppg'), past\_duration\_leading\_per\_game\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_duration\_leading\_per\_game', 'future\_ppg'), past\_ppg\_\_future\_ppg = calculate\_nested\_r2(data, 'past\_ppg', 'future\_ppg') ) |> dplyr::select(-data)}pivot\_rolling\_r2s <- function(df) { df |> tidyr::pivot\_longer( -c(league\_group, game\_idx), names\_pattern = '(^.*)\_\_(.*$)', names\_to = c('predictor', 'target'), values\_to = 'r2' )}do\_calculate\_rolling\_r2s <- purrr::compose( calculate\_rolling\_r2s, pivot\_rolling\_r2s, .dir = 'forward')join\_team\_summary\_and\_gamestate\_dfs <- function(team\_summary\_df, gamestate\_df) { team\_summary\_df |> dplyr::left\_join( gamestate\_df |> dplyr::select(-c(country, gender, tier, date)), by = dplyr::join\_by(season, match\_id, team) ) |> ## since we didn't have xgot at the match-level, we need to create it from the shot-level data dplyr::mutate( xgot = xgot\_trailing + xgot\_neutral + xgot\_leading, xgot\_conceded = xgot\_conceded\_trailing + xgot\_conceded\_neutral + xgot\_conceded\_leading )}combined\_df\_gd0 <- join\_team\_summary\_and\_gamestate\_dfs( combined\_team\_summary, gamestate\_gd0\_by\_game)combined\_df\_abs\_gd1 <- join\_team\_summary\_and\_gamestate\_dfs( combined\_team\_summary, gamestate\_abs\_gd1\_by\_game)## both dfs have the same formatrolling\_r2s\_gd0 <- combined\_df\_gd0 |> do\_calculate\_rolling\_r2s()rolling\_r2s\_abs\_gd1 <- combined\_df\_abs\_gd1 |> do\_calculate\_rolling\_r2s()rolling\_r2s\_abs\_gd1#> # A tibble: 2,088 × 5#> league\_group game\_idx predictor target r2#> <chr> <int> <chr> <chr> <dbl>#> 1 Big 5 1 past\_shot\_ratio future\_g\_ratio 0.187 #> 2 Big 5 1 past\_sot\_ratio future\_g\_ratio 0.170 #> 3 Big 5 1 past\_g\_ratio future\_g\_ratio 0.165 #> 4 Big 5 1 past\_xg\_ratio future\_g\_ratio 0.182 #> 5 Big 5 1 past\_xgot\_ratio future\_g\_ratio 0.158 #> 6 Big 5 1 past\_shot\_neutral\_ratio future\_g\_ratio 0.188 #> 7 Big 5 1 past\_sot\_neutral\_ratio future\_g\_ratio 0.0206#> 8 Big 5 1 past\_g\_neutral\_ratio future\_g\_ratio 0.166 #> 9 Big 5 1 past\_xg\_neutral\_ratio future\_g\_ratio 0.183 #> 10 Big 5 1 past\_xgot\_neutral\_ratio future\_g\_ratio 0.155 #> # ℹ 2,294 more rows#> # ℹ Use `print(n = ...)` to see more rows Plotting the R-squared values TAG\_LABEL <- htmltools::tagList( htmltools::tags$span(htmltools::HTML(enc2utf8("")), style = 'font-family:fb'), htmltools::tags$span("@TonyElHabr"),)CAPTION\_LABEL <- '**Data**: Opta via fbref. 2018 - 2024 seasons, excluding 2020.<br/>**Definitions**: Ratio = team value / (team value + opponent value).<br/>**Inspiration**: 11tegen, American Soccer Analysis'PLOT\_RESOLUTION <- 300WHITISH\_FOREGROUND\_COLOR <- 'white'COMPLEMENTARY\_FOREGROUND\_COLOR <- '#cbcbcb' # '#f1f1f1'BLACKISH\_BACKGROUND\_COLOR <- '#1c1c1c'COMPLEMENTARY\_BACKGROUND\_COLOR <- '#4d4d4d'FONT <- 'Titillium Web'sysfonts::font\_add\_google(FONT, FONT)## https://github.com/tashapiro/tanya-data-viz/blob/main/chatgpt-lensa/chatgpt-lensa.R for twitter logosysfonts::font\_add('fb', 'Font Awesome 6 Brands-Regular-400.otf')showtext::showtext\_auto()showtext::showtext\_opts(dpi = PLOT\_RESOLUTION)ggplot2::theme\_set(ggplot2::theme\_minimal())ggplot2::theme\_update( text = ggplot2::element\_text(family = FONT), title = ggplot2::element\_text(size = 16, color = WHITISH\_FOREGROUND\_COLOR), plot.title = ggtext::element\_markdown(face = 'bold', size = 16, color = WHITISH\_FOREGROUND\_COLOR), plot.title.position = 'plot', plot.subtitle = ggtext::element\_markdown(size = 16, color = COMPLEMENTARY\_FOREGROUND\_COLOR), axis.text = ggplot2::element\_text(color = WHITISH\_FOREGROUND\_COLOR, size = 11), legend.text = ggplot2::element\_text(size = 12, color = WHITISH\_FOREGROUND\_COLOR, face = 'plain'), legend.title = ggplot2::element\_text(size = 12, color = WHITISH\_FOREGROUND\_COLOR, face = 'bold'), axis.title.x = ggtext::element\_markdown(size = 14, color = WHITISH\_FOREGROUND\_COLOR, face = 'bold', hjust = 0.99), axis.title.y = ggtext::element\_markdown(size = 14, color = WHITISH\_FOREGROUND\_COLOR, face = 'bold', hjust = 0.99), axis.ticks = ggplot2::element\_line(color = WHITISH\_FOREGROUND\_COLOR), axis.line = ggplot2::element\_blank(), strip.text = ggplot2::element\_text(size = 14, color = WHITISH\_FOREGROUND\_COLOR, face = 'bold', hjust = 0), panel.grid.major = ggplot2::element\_line(color = COMPLEMENTARY\_BACKGROUND\_COLOR), panel.grid.minor = ggplot2::element\_line(color = COMPLEMENTARY\_BACKGROUND\_COLOR), panel.grid.minor.x = ggplot2::element\_blank(), panel.grid.minor.y = ggplot2::element\_blank(), plot.margin = ggplot2::margin(10, 20, 10, 20), plot.background = ggplot2::element\_rect(fill = BLACKISH\_BACKGROUND\_COLOR, color = BLACKISH\_BACKGROUND\_COLOR), plot.caption = ggtext::element\_markdown(color = WHITISH\_FOREGROUND\_COLOR, hjust = 0, size = 10, face = 'plain', lineheight = 1.1), plot.caption.position = 'plot', plot.tag = ggtext::element\_markdown(size = 10, color = WHITISH\_FOREGROUND\_COLOR, hjust = 1), plot.tag.position = c(0.99, 0.01), panel.spacing.x = grid::unit(2, 'lines'), panel.spacing.y = grid::unit(1, 'lines'), # panel.background = ggplot2::element\_rect(fill = BLACKISH\_BACKGROUND\_COLOR, color = BLACKISH\_BACKGROUND\_COLOR) panel.background = ggplot2::element\_rect(fill = BLACKISH\_BACKGROUND\_COLOR, color = WHITISH\_FOREGROUND\_COLOR))BASE\_PRETTY\_PREDICTOR\_NAMES <- c( 'past\_ppg' = 'Points per Game', 'past\_g\_ratio' = 'Goal Ratio', 'past\_shot\_ratio' = 'Total Shots Ratio', 'past\_sot\_ratio' = 'Shots on Target Ratio', 'past\_xg\_ratio' = 'xG Ratio')APPENDIX\_PRETTY\_PREDICTOR\_NAMES <- c( 'past\_ppa\_ratio' = 'Passes into Penalty Area Ratio', 'past\_xgot\_ratio' = 'xGOT Ratio', # 'past\_xag\_ratio' = 'xAG Ratio', 'past\_xg\_xag\_ratio' = 'xG+xAG Ratio', 'past\_xg\_ratio' = 'xG Ratio')PRETTY\_TARGET\_NAMES <- c( 'future\_ppg' = 'Future Points per Game', 'future\_g\_ratio' = 'Future Goals Ratio')BASE\_PREDICTOR\_PALETTE <- c( 'Points per Game' = '#cad2c5', 'Goal Ratio' = '#8bc34a', 'Total Shots Ratio' = '#ffc107', 'Shots on Target Ratio' = '#448aff', 'xG Ratio' = '#f44336')APPENDIX\_PREDICTOR\_PALETTE <- c( 'Passes into Penalty Area Ratio' = '#dbf679', 'xGOT Ratio' = '#2ec4b6', # 'xAG Ratio' = '#e0aaff', # '#7b2cbf' 'xG+xAG Ratio' = '#e0aaff', 'xG Ratio' = '#f44336')CROSSHAIR\_LABEL <- '<i>Crosshairs mark first match week where R<sup>2</sup>>0.5.</i>'plotting\_rolling\_r2s <- function(rolling\_r2s, pretty\_predictor\_names, predictor\_palette) { prettified\_rolling\_r2s <- rolling\_r2s |> dplyr::filter( predictor %in% names(pretty\_predictor\_names) ) |> dplyr::mutate( pretty\_predictor = factor(pretty\_predictor\_names[predictor], rev(pretty\_predictor\_names)), pretty\_target = PRETTY\_TARGET\_NAMES[target] ) |> ## put xG on top dplyr::arrange(league\_group, pretty\_target, dplyr::desc(pretty\_predictor)) min\_prettified\_rolling\_r2s <- prettified\_rolling\_r2s |> dplyr::filter(r2 > 0.5) |> dplyr::group\_by(league\_group, pretty\_target, pretty\_predictor) |> dplyr::slice\_min(game\_idx, n = 1) |> dplyr::ungroup() prettified\_rolling\_r2s |> ggplot2::ggplot() + ggplot2::aes( x = game\_idx, y = r2 ) + ggplot2::geom\_segment( data = min\_prettified\_rolling\_r2s, ggplot2::aes( color = pretty\_predictor, x = 0, xend = game\_idx, y = r2, yend = r2 ), linetype = 2, linewidth = 0.5 ) + ggplot2::geom\_segment( data = min\_prettified\_rolling\_r2s, ggplot2::aes( color = pretty\_predictor, x = game\_idx, xend = game\_idx, y = 0, yend = r2 ), linetype = 2, linewidth = 0.5 ) + ggplot2::geom\_line( linewidth = 0.75, ggplot2::aes(color = pretty\_predictor) ) + ggplot2::scale\_color\_manual( values = predictor\_palette ) + ggplot2::guides( color = ggplot2::guide\_legend( title = NULL, position = 'inside', label.theme = ggplot2::element\_text(color = WHITISH\_FOREGROUND\_COLOR, size = 11, FONT), override.aes = list(linewidth = 2) ) ) + ggplot2::scale\_y\_continuous( expand = c(0, 0), limits = c(0, 1), breaks = seq(0, 1, by = 0.2), labels = scales::number\_format(accuracy = 0.1) ) + ggplot2::scale\_x\_continuous( expand = c(0, 0), limits = c(0, 38), breaks = seq.int(0, 35, by = 5), labels = seq.int(0, 35, by = 5) ) + ggtext::geom\_richtext( data = tibble::tibble( league\_group = 'Big 5', pretty\_target = sort(unname(PRETTY\_TARGET\_NAMES))[1] ), ggplot2::aes( x = 1, y = 0.7, label = CROSSHAIR\_LABEL ), fill = NA, label.color = NA, label.padding = grid::unit(rep(0, 4), 'pt'), color = WHITISH\_FOREGROUND\_COLOR, family = FONT, size = 10 / .pt, hjust = 0, vjust = 0.5 ) + ggplot2::theme( legend.position.inside = c(0.82, 0.35), legend.key.spacing.y = ggplot2::unit(-4, 'pt'), panel.grid.major = ggplot2::element\_blank() ) + ggplot2::facet\_grid(league\_group~pretty\_target) + ggplot2::labs( title = 'Predictiveness of Season-to-Date Metrics on Rest-of-Season Performance', x = 'Match Day', y = 'R<sup>2</sup>', caption = CAPTION\_LABEL, tag = TAG\_LABEL )}## Can pick either rolling\_r2s\_gd0 or rolling\_r2s\_abs\_gd1 to plot since we're looking at just## non-gamestate features hererolling\_r2s\_plot <- rolling\_r2s\_abs\_gd1 |> plotting\_rolling\_r2s( pretty\_predictor\_names = BASE\_PRETTY\_PREDICTOR\_NAMES, predictor\_palette = BASE\_PREDICTOR\_PALETTE )ggplot2::ggsave( rolling\_r2s\_plot, filename = file.path(PROJ\_DIR, 'rolling-r2s.png'), width = 9, height = 9 / 1.5) There’s a bit of visual noise here that can make it hard to differentiate traces. This can be smoothed out by resampling.

Specifically, if we randomly reorder matchweeks before calculating cumulative in-season measures, and do that 100 times, we end up with a plot that looks like this.

Calculating bootstrapped R-squared values calculate\_resampled\_rolling\_r2s <- function( combined\_df, resamples = 10, seed = 42, parallel\_seed = 117, cores\_prop = 0.5 ) { match\_idx\_grid <- combined\_df |> dplyr::distinct(season, country, gender, tier, game\_idx) n\_cores <- parallel::detectCores() cores\_for\_parallel <- ceiling(n\_cores * cores\_prop) future::plan( future::multisession, workers = cores\_for\_parallel ) withr::local\_seed(seed) resampled\_rolling\_df <- furrr::future\_map( 1:resamples, .progress = TRUE, .options = furrr::furrr\_options( seed = parallel\_seed ), \(.i) { resampled\_match\_idx\_grid <- match\_idx\_grid |> dplyr::rename(orig\_game\_idx = game\_idx) |> dplyr::slice\_sample(n = nrow(match\_idx\_grid), replace = FALSE) |> dplyr::group\_by(country, gender, tier, season) |> dplyr::mutate( game\_idx = dplyr::row\_number(), ) |> dplyr::ungroup() combined\_df |> dplyr::rename(orig\_game\_idx = game\_idx) |> dplyr::left\_join( resampled\_match\_idx\_grid, by = dplyr::join\_by(season, country, gender, tier, orig\_game\_idx) ) |> calculate\_rolling\_r2s() } ) |> purrr::list\_rbind() future::plan(future::sequential) resampled\_rolling\_df |> dplyr::summarize( .by = c(league\_group, game\_idx), dplyr::across( dplyr::matches('\_\_'), \(.x) mean(.x, na.rm = TRUE) ) )}do\_calculate\_resampled\_rolling\_r2s <- purrr::compose( calculate\_resampled\_rolling\_r2s, pivot\_rolling\_r2s, .dir = 'forward')N\_RESAMPLES <- 100resampled\_rolling\_r2s\_gd0 <- combined\_df\_gd0 |> do\_calculate\_resampled\_rolling\_r2s(resamples = N\_RESAMPLES)resampled\_rolling\_r2s\_abs\_gd1 <- combined\_df\_abs\_gd1 |> do\_calculate\_resampled\_rolling\_r2s(resamples = N\_RESAMPLES) Plotting the bootstrapped R-squared values BOOTSTRAPPED\_CAPTION\_LABEL <- stringr::str\_replace( CAPTION\_LABEL, '<br\\/>', glue::glue(' Data resampled by match day {N\_RESAMPLES} times.<br/>'))relabel\_for\_bootstrap\_plot <- function(...) { list( ..., ggplot2::labs( y = 'Average Bootstrapped R<sup>2</sup>', caption = BOOTSTRAPPED\_CAPTION\_LABEL, x = 'Randomized Match Day' ) )}resampled\_rolling\_r2s\_plot <- resampled\_rolling\_r2s\_gd0 |> plotting\_rolling\_r2s( pretty\_predictor\_names = BASE\_PRETTY\_PREDICTOR\_NAMES, predictor\_palette = BASE\_PREDICTOR\_PALETTE ) + relabel\_for\_bootstrap\_plot()ggplot2::ggsave( resampled\_rolling\_r2s\_plot, filename = file.path(PROJ\_DIR, 'bootstrapped-rolling-r2s.png'), width = 9, height = 9 / 1.5) Indeed, this looks like the bootstrapped plot from ASA.5 Note that, for a given measure, the bootstrapped R-squared values are slightly smaller across the board compared to the non-bootstrapped values.6 This is perhaps not surprising, as resampling tends to have a “shrinking” effect on figures. Intuitively, this noise could be associated with scheduling bias, injuries, etc.

Extending prior artNow we incorporate gamestate-aware measures.7 In this case, we’re mostly interested game time where the score is close.

Most commonly, “close” is defined as a tied score, whether it’s 0-0, 1-1, etc. Let’s call this the gd = 0 approach. The indirect assumption is that teams start to play more conservatively when leading (and more aggressively when trailing), thereby distorting the ratio of shots, goals, and xG that we might otherwise expect given the relative quality of the teams. By excluding events when the game is not tied, we might achieve more “signal” in our measures of performance.

We might also define “close” as periods when the absolute difference in goals is just 1, so 0-1, 1-2, 3-2, 5-4, etc. Let’s call this the abs(gd) = 1 approach. This definition indirectly assumes that teams don’t start to play more complacently when ahead until they have a 2 goal lead. This approach would capture more game time (and have higher tallies of goals, shots, etc.) compared to the gd = 0 approach, although less time than just using the whole match.

In the plot below, I’ve split out each of the measures of past performance into its own facet, paired with the neutral gamestate version using the gd = 0 definition. This makes it easy to see which pairs of past and future performance measures are improved by subsetting the past performance measure to neutral gamestates. To reduce the scope and enhance the focus of the work, I’ve chosen to look at just points per game as our measure of future performance, and just the goals and xG ratios as our measures of past performance.

Plotting the bootstrapped R-squared values, including the neutral game state metrics BASE\_AND\_NEUTRAL\_PRETTY\_PREDICTOR\_LOOKUP <- list( 'xG Ratio' = c( 'past\_xg\_ratio', 'past\_xg\_neutral\_ratio' ), 'Goal Ratio' = c( 'past\_g\_ratio', 'past\_g\_neutral\_ratio' ) #, # 'Total Shots Ratio' = c( # 'past\_shot\_ratio', # 'past\_shot\_neutral\_ratio' # ), # 'Shots on Target Ratio' = c( # 'past\_sot\_ratio', # 'past\_sot\_neutral\_ratio' # ))BASE\_AND\_NEUTRAL\_TARGET\_PREDICTOR <- 'future\_ppg'unlist\_and\_invert <- function(x) { unlisted <- unlist(x) names(unlisted) <- rep(names(x), lengths(x)) setNames(names(unlisted), unname(unlisted))}BASE\_AND\_NEUTRAL\_PRETTY\_PREDICTOR\_GROUP\_NAMES <- unlist\_and\_invert( BASE\_AND\_NEUTRAL\_PRETTY\_PREDICTOR\_LOOKUP)IS\_NEUTRAL\_PALETTE <- c( `TRUE` = '#00b4d8', `FALSE` = '#caf0f8')plotting\_rolling\_r2s\_with\_neutral <- function(rolling\_r2s, gamestate\_description) { prettified\_rolling\_r2s <- rolling\_r2s |> dplyr::filter( predictor %in% names(BASE\_AND\_NEUTRAL\_PRETTY\_PREDICTOR\_GROUP\_NAMES), target == BASE\_AND\_NEUTRAL\_TARGET\_PREDICTOR ) |> dplyr::mutate( is\_neutral = stringr::str\_detect(predictor, 'neutral'), pretty\_predictor\_group = factor( BASE\_AND\_NEUTRAL\_PRETTY\_PREDICTOR\_GROUP\_NAMES[predictor], levels = names(BASE\_AND\_NEUTRAL\_PRETTY\_PREDICTOR\_LOOKUP) ) ) |> ## put xG on top dplyr::arrange(league\_group, pretty\_predictor\_group, is\_neutral) min\_rolling\_r2s <- prettified\_rolling\_r2s |> dplyr::filter(r2 > 0.5) |> dplyr::group\_by(league\_group, predictor) |> dplyr::slice\_min(game\_idx, n = 1) |> dplyr::ungroup() prettified\_rolling\_r2s |> ggplot2::ggplot() + ggplot2::aes( x = game\_idx, y = r2, group = predictor ) + ggplot2::geom\_segment( data = min\_rolling\_r2s, ggplot2::aes( color = is\_neutral, x = 0, xend = game\_idx, y = r2, yend = r2 ), linetype = 2, linewidth = 0.5 ) + ggplot2::geom\_segment( data = min\_rolling\_r2s, ggplot2::aes( color = is\_neutral, x = game\_idx, xend = game\_idx, y = 0, yend = r2 ), linetype = 2, linewidth = 0.5 ) + ggplot2::geom\_line( linewidth = 1, ggplot2::aes(color = is\_neutral) ) + ggplot2::scale\_color\_manual( values = IS\_NEUTRAL\_PALETTE ) + ggplot2::guides( color = 'none' ) + ggplot2::scale\_y\_continuous( limits = c(0, 1), breaks = seq(0, 1, by = 0.2), labels = scales::number\_format(accuracy = 0.1) ) + ggplot2::scale\_x\_continuous( breaks = seq.int(0, 35, by = 5), labels = seq.int(0, 35, by = 5) ) + ggplot2::theme( panel.grid.major = ggplot2::element\_blank(), plot.subtitle = ggtext::element\_markdown(size = 12) ) + ggplot2::facet\_grid(league\_group~pretty\_predictor\_group) + ggplot2::labs( title = 'Predictiveness of Season-to-Date Metrics on Rest-of-Season Points Per Game', subtitle = glue::glue('Are <b><span style="color:{IS\_NEUTRAL\_PALETTE[["FALSE"]]}">full match</span></b> metrics more predictive when subset to <b><span style="color:{IS\_NEUTRAL\_PALETTE[["TRUE"]]}">{gamestate\_description}</span></b>?'), x = 'Randomized Match Day', y = 'Average Bootstrapped R<sup>2</sup>', caption = BOOTSTRAPPED\_CAPTION\_LABEL, tag = TAG\_LABEL )}rolling\_r2s\_with\_neutral\_gd0\_plot <- resampled\_rolling\_r2s\_gd0 |> plotting\_rolling\_r2s\_with\_neutral('tied gamestates')rolling\_r2s\_with\_neutral\_abs\_gd1\_plot <- resampled\_rolling\_r2s\_abs\_gd1 |> plotting\_rolling\_r2s\_with\_neutral('gamestates with absolute goal difference <= 1')ggplot2::ggsave( rolling\_r2s\_with\_neutral\_gd0\_plot, filename = file.path(PROJ\_DIR, 'bootstrapped-rolling-r2s-with-neutral-gd-0.png'), width = 8, height = 8 / 1.5)ggplot2::ggsave( rolling\_r2s\_with\_neutral\_abs\_gd1\_plot, filename = file.path(PROJ\_DIR, 'bootstrapped-rolling-r2s-with-neutral-abs-gd-1.png'), width = 8, height = 8 / 1.5) So we see that subsetting xG and goals to tied gamestates reduces how predictive they are of future points per game, both for the Big 5 leagues and the MLS. Perhaps this is not surprising, as we’re disregarding a lot of data (over 50%!) that is included in full match sums.

Let’s see if the same plot using the abs(gd) = 1 neutral gamestate definition looks any different.

Ah, so using this alternative definition of “neutral” gamestate, the predictiveness of the season-to-date xG and goal ratios is much closer to the full match analogues, so perhaps the abs(gd) = 1 is superior. Nonetheless, it’s evident that neutral gamestate xG and goal ratios are no better than the full match ratios, indicating that there is no incremental value to be had with focusing on neutral gamestates. This falls in line with what Ijtsma hypothesized–that the effect of within-match complacency is minimal, and accounting for it (by reducing tallies to neutral gamestates) is more than likely to reduce predictiveness.

Discussion and ConclusionOk, so we didn’t find a ground-breaking result with neutral gamestates, although the value of a “null result” should not be understated.

To offer some kind of insight that might be useful, I present the game index (i.e. match day) at which my data shows that the season-to-date measures start to provide a reliable signal for forecasting future performance.8 I define the reliability “threshold” as the first week in which the running R-squared value exceeds 0.5. For those reading the footnotes or broadly aware of the literature on xG predictiveness, this is effectively my look at the common adage that xG becomes a reliable indicator of future performance somewhere between the 5 and 15 game mark.9

Calculating when past performance measures become reliable indicators of future performance rolling\_r2s\_abs\_gd1 |> dplyr::filter( league\_group == 'Big 5', target == 'future\_g\_ratio', predictor %in% names(BASE\_PRETTY\_PREDICTOR\_NAMES) ) |> dplyr::mutate( pretty\_predictor = BASE\_PRETTY\_PREDICTOR\_NAMES[predictor] ) |> dplyr::filter(r2 > 0.5) |> dplyr::group\_by(pretty\_predictor) |> dplyr::slice\_min(game\_idx, n = 1) |> dplyr::ungroup() |> tidyr::complete( pretty\_predictor = BASE\_PRETTY\_PREDICTOR\_NAMES ) |> dplyr::arrange(game\_idx) |> dplyr::transmute( `Predictor` = pretty\_predictor, `Match Day` = tidyr::replace\_na(as.character(game\_idx), '-') ) |> knitr::kable( align = 'lr' ) |> clipr::write\_clip()rolling\_r2s\_abs\_gd1 |> dplyr::filter( league\_group == 'Big 5', target == 'future\_ppg', predictor %in% names(BASE\_PRETTY\_PREDICTOR\_NAMES) ) |> dplyr::mutate( pretty\_predictor = BASE\_PRETTY\_PREDICTOR\_NAMES[predictor] ) |> dplyr::filter(r2 > 0.5) |> dplyr::group\_by(pretty\_predictor) |> dplyr::slice\_min(game\_idx, n = 1) |> dplyr::ungroup() |> tidyr::complete( pretty\_predictor = BASE\_PRETTY\_PREDICTOR\_NAMES ) |> dplyr::arrange(game\_idx) |> dplyr::transmute( `Predictor` = pretty\_predictor, `Match Day` = tidyr::replace\_na(as.character(game\_idx), '-') ) |> knitr::kable( align = 'lr' ) |> clipr::write\_clip()

Earliest match day in which the past metric becomes a reliable measure of future goals ratio in the Big 5 leagues.| Predictor | Match Day | | --- | --- | | xG Ratio | 9 | | Goal Ratio | 11 | | Shots on Target Ratio | 12 | | Points per Game | 13 | | Total Shots Ratio | - |

Earliest match day in which the past metric becomes a reliable measure of future points per game in the Big 5 leagues.| Predictor | Match Day | | --- | --- | | xG Ratio | 13 | | Goal Ratio | 15 | | Points per Game | 16 | | Shots on Target Ratio | 16 | | Total Shots Ratio | - |

AppendixI was interested in evaluating running in-season R-squared values for a few more measures of past performance:

passes into the penalty area ratio
expected goals on target (xGOT) ratio
xG + expected assisted goals10 (xG+xAG) ratio

Below is the same running R-squared plot shown before, just with these new measures, along side our tried and true xG ratio measure.

Aha! It seems that we have identified a measure–xG+xAG ratio (purple above)–that becomes reliable for forecasting rest-of-season performance even earlier than xG ratio. Specifically, it exceeds the reliability threshold at the 7 and 10 game marks for future goal ratio and future points per game respectively. That’s a non-trivial improvement compared to the 9 and 13 match day marks for xG ratio.

Alas, xG+xAG metaphorically stands on the shoulders of xG, adding in the component of expected assists (xA) on shots. Assuming that the underlying xA model is well calibrated, perhaps we should not be surprised to see that the composite xG+xAG measure outperforms all others.

On the other hand, I suppose I am a bit surprised that xGOT ratio does not seem to do nearly as well as xG ratio in terms of forecasting rest-of-season performancet. The implication is that there is value in the xG of off-target shots for projecting the future. By comparison, shots on target ratio tend to be more predictive than just the shots ratio, meaning that including off-target shots introduces noise that reduces predictiveness. That’s an interesting difference in shots and xG!

No matching itemsFootnotes1. If you know anything about the broader soccer analytics discourse, you’ve more than likely heard about one of the Ijtsma’s findings regarding the predictiveness of xG. As Eliot puts it: “If you’ve ever heard or read someone say that it takes 5-10 (4 in the article) games for xG to be predictive, they probably read this article or got it second hand from someone who has.”↩︎ 2. Note that there are some critics of the choice to rely on R-squared as a measure of predictiveness. For the purpose of this post, we’ll take that criticism on the chin.↩︎ 3. A “ratio” here is broadly defined in pseudo-code as team's value / (team's value + opponent's value). Ijtsma’s notes in a reply to a comment that one might arguably use a difference formula, i.e. team's value - opponent's value, as ratios are susceptible to noise when values are themselves fractional. This is most relevant for xG, and not so relevant for shots and goals, which are inherently discrete.↩︎ 4. Notably, I exclude the COVID-19 influenced 2020 season, so as to reduce a bit of noise in the data. Although bootstrapping games within-season should reduce much of this noise, in practice, FBref is missing some game-level xG data for 2020 matches, so this choice is sort of just a pragmatic one.↩︎ 5. I only perform 100 bootstraps instead of 1,000 like Eliot did since I found 100 sufficient for creating visual clarity.↩︎ 6. This may not be easy to see directly since the bootstrapped and non-bootstrapped curves are not plotted altogether. Nonetheless, one can compare the peaks of the curves against the y-axis in this plot compared to the prior one to verify the suppressing effect of the resampling.↩︎ 7. If you’ve been following the code, these measures have already been calculated.↩︎ 8. The tables that follow show non-bootstrapped R-squared values. The bootstrapped R-squared values only exceed 0.5 for a handful of measures, typically in the 15-20 game range.↩︎ 9. Note that Ijtsma says that xG ratio becomes useful as early as the fourth match week. However, his statement is based purely on a visual assessment of when the R-squared curve generally “settles down” after the expected early season up-tick. Looking at his graph, the R-squared values for xG don’t actually exceed the 0.5 threshold until some point between the 10th and 15th match days.↩︎ 10. FBref’s explanation regarding how xAG differs from expected assists (xA): “xA, or expected assists, is the likelihood that a given completed pass will become a goal assist… Players receive xA for every completed pass regardless of whether a shot occurred or not… [Expected Assisted Goals (xAG)] indicates a player’s ability to set up scoring chances… Players receive xAG only when a shot is taken after a completed pass.”↩︎ To leave a comment for the author, please follow the link and comment on their blog: Tony's Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Expected goals, gamestate, and predictiveness

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhen working with Excel files, you may need to extract data from one workbook and use it in another. This can be done manually by copying and pasting the data, but it can be time-consuming and error-prone, especially when dealing with large datasets. One way to automate this process is by using Visual Basic for Applications (VBA) to extract the data from one workbook and execute the VBA code from R.

In this blog post, I’ll walk you through the process of extracting data from another workbook using VBA and how to execute this from R. We’ll use the data in Sheet1 from an example workbook.

Extracting Data from Another Workbook Using VBAStep 1: Setting Up the VBA CodeFirst, we need to write a VBA script that will open another workbook, extract data from Sheet1, and return this data. Here’s a simple VBA code to accomplish this:

Open the VBA editor by pressing Alt + F11.
Insert a new module by right-clicking on any existing module or the workbook name, then selecting Insert > Module.
Copy and paste the following VBA code into the module:

Sub ExtractData() Dim sourceWorkbook As Workbook Dim targetWorkbook As Workbook Dim sourceSheet As Worksheet Dim targetSheet As Worksheet Dim sourceRange As Range Dim targetRange As Range ' Define the path to the source workbook Dim sourceFilePath As String sourceFilePath = "C:\Users\ssanders\Documents\GitHub\steveondata\posts\2024-06-19\random\_data.xlsx" ' Change this to your actual file path ' Open the source workbook Set sourceWorkbook = Workbooks.Open(sourceFilePath) Set sourceSheet = sourceWorkbook.Sheets("Sheet1") Set sourceRange = sourceSheet.Range("A1:B30") ' Adjust the range as needed ' Open the target workbook Set targetWorkbook = ThisWorkbook Set targetSheet = targetWorkbook.Sheets("Sheet1") Set targetRange = targetSheet.Range("A1:B30") ' Adjust the range as needed ' Clear the target range before pasting targetRange.Clear ' Copy the data from source to target sourceRange.Copy Destination:=targetRange ' Close the source workbook without saving sourceWorkbook.Close SaveChanges:=False ' Save and close the target workbook targetWorkbook.Save targetWorkbook.Close SaveChanges:=True ' Quit Excel Application.QuitEnd Sub This script opens another workbook, copies the data from Sheet1, and pastes it into the current workbook’s Sheet1. Modify the sourceFilePath to the location of your source workbook and adjust the ranges as necessary. The data was already in a workbook and thus we knew the dimensions of the data.

Step 2: Executing the VBA Code from RNow that we have the VBA code ready, let’s write some R code to execute this VBA macro. We’ll use the RDCOMClient package to interact with Excel from R.

Install the RDCOMClient package if you haven’t already:

install.packages("RDCOMClient", repos = "http://www.omegahat.net/R") 2. Load the package and write the R code to run the VBA macro:

library(RDCOMClient)# Path to your Excel workbook containing the VBA macroexcelFilePath <- "C:/Users/ssanders/Documents/GitHub/steveondata/posts/2024-06-19/get\_data\_from\_another\_workbook.xlsm"# Create a COM object to interact with ExcelexcelApp <- COMCreate("Excel.Application")# Open the workbookworkbook <- excelApp$Workbooks()$Open(excelFilePath)# Make Excel visible (optional)excelApp[["Visible"]] <- FALSE# Run the VBA macroexcelApp$Run("ExtractData")

NULL

```

Close the workbook without savingworkbook$Close(FALSE)

```

[1] TRUE

```

Quit ExcelexcelApp$Quit()

```

NULL

```

Release COM objectrm(excelApp, workbook)

`` This R script creates a COM object to interact with Excel, opens the workbook containing our VBA macro, runs the macro, and then quits Excel. Make sure to modify theexcelFilePath` to point to your actual workbook.

Now let’s see if it actually worked:

library(readxl)f\_path <- "C:/Users/ssanders/Documents/GitHub/steveondata/posts/2024-06-19/random\_data.xlsx"read\_excel(f\_path, sheet = "Sheet1", col\_names = FALSE)

New names:• `` -> `...1`

```

A tibble: 30 × 1 ...1 1 -0.371 2 -1.00 3 0.226 4 -0.323 5 -0.142 6 1.19 7 -0.827 8 0.715 9 -0.10510 -1.06 # ℹ 20 more rows

`` ConclusionIn this post, we’ve covered how to set up a VBA macro to extract data from another workbook and how to execute this macro from R using theRDCOMClient` package. This approach allows you to leverage the power of VBA for Excel automation while controlling the process from R, providing a seamless integration between the two environments.

Feel free to adjust the VBA code and R script to suit your specific needs. Happy coding!

For more information on integrating R with other tools, check out my other posts at www.spsanderson.com/steveondata/.

If you have any questions or run into issues, don’t hesitate to reach out on LinkedIn or Mastodon. Let’s keep the conversation going!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extracting Data from Another Workbook Using VBA and Executing It from R

View Details

[This article was first published on R on R & Data Analysis - Eric Stemmler, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Annualization is when results that are with respect to a time period smaller then one year are scaled up so that it becomes with respect to one year. As a general principle this can also be called projection, grossing up, up-scaling, expansion or extrapolation.

In this article I want to implement and compare different methods for annualization (to stick with one term), in order to consolidate my own understanding of the topic. The method of choice is particularly easy to implement in R in a few lines of code and frames the problem of annualization in terms of missing data.

Example: Annual RidershipOne usecase for annualization can appear in survey analysis. The working example here will be from public transport: Imagine 3 riderhsip surveys are conducted in a city in 2023 to find out the total number of passengers travelling by bus per day. The ultimate goal is to estimate the annual ridership of this city. It is known that, due to seasonality, ridership varies throughout the year – e.g., it can be lower during summer. Hence the need to survey different periods of a year. Conducting surveys is an expensive and time-consuming endeavor, so that usually only a limited number of surveys can be conducted. In addition, another level of sampling comes into effect since usually it is impractical to survey every unit – in this case buses. Therefore, a number of certain buses are chosen among all the bus lines.

In total the procedure of analysing those surveys together can be summarized asin 2 steps:

For each survey, calculate an estimate of daily ridership that isrepresentative for the period that the respective survey covers
Combine results and annualize to yield an annual total ridership DataWe will create first a dataset to work with. For this, we will create three random periods of a year, each of length 4 weeks for which we will assume we have estimates of daily ridership – essentially having results ready from step number 1.

Corresponding to step number one, we will create what we can call a base data set which we will regard as a reference of true daily ridership.Simulating the conduncting of surveys this regard means that we get 3 averages of periods of 4 weeks.

Below is the code to create our base data. For the point of this article, we will also create a correlating factor for daily ridership, which in this case will be the number of performed kilometers, but more on will follow below.

library(data.table)set.seed(20240528)dt <- data.table(date = seq.Date(from = as.Date("2023-01-01"), to = as.Date("2023-12-31"), by = "day"))dt[, kms := rnorm(n = 365, mean = 10 * (month(date) - 6)^2 + 500, sd = 25)]dt[, ppd := rnorm(n = 365, mean = kms * 4, sd = 50)]plot(dt$kms, dt$ppd, xlab = "performed kilometers [km]", ylab = "passengers per day [1/day]")

par(mfrow = c(1, 2))plot(dt$date, dt$kms, xlab = "", ylab = "performed kilometers [km]")abline(h = mean(dt$kms))plot(dt$date, dt$ppd, xlab = "", ylab = "passengers per day")abline(h = mean(dt$ppd)) Figure 1: Seasonality of performed kilometers (left) and daily ridership (right). Horizontal line marks the average. Notice that the most representative months (closest to average) are April and October.

As we can see, we now have a dataset of daily ridership (ppd), which correlates with the number of performed kilometers (kms) and both are subsect to a seasonal trend (by month).

Now we will create survey results based on this - which effectively means observing only 3 periods of the dataset and averaging the observed ridership to a monthly mean.To simulate this, we will randomly create three periods. The tricky part is that we need those periods to be non-overlapping. For this, we’ll simply choose three different starting dates, test if they are at least 4 weeks (120 days) apart and if not try again.

In the next section we will calculate the annual ridership from those estimates and compare it to the true value.

repeat { # substract 31 days in December to ensure surveys are conducted within one # year we also want to make sure to get one survey in summer s <- sort(ceiling(c(runif(n = 1, min = 0, max = 179), runif(n = 1, min = 180, max = 180 + 2 * 31), runif(n = 1, min = (180 + 2 * 31) + 1, max = 365 - 31)))) if (all(diff(s) > 40)) break}# s <- as.Date('2023-01-01') + ssdt <- data.table(survey\_begin = as.Date("2023-01-01") + s)sdt[, survey\_end := survey\_begin + (4 * 7)]sdt <- dt[sdt, on = c("date>=survey\_begin", "date<=survey\_end")][, .(ppd = mean(ppd)), by = c(survey\_begin = "date", survey\_end = "date.1")]print(sdt)## survey\_begin survey\_end ppd## <Date> <Date> <num>## 1: 2023-01-07 2023-02-04 2966.356## 2: 2023-08-22 2023-09-19 2279.425## 3: 2023-10-10 2023-11-07 2705.054 Method 1: Simple MeanOne way to annualize is to multiply a result by the number of instances it will occure in one year.In this case this could be 12, if we were to consider a result for a single month.If we had zero understanding of the generative process of the data, we could simply annualize all 3 surveys and take the average as combined result.

truth <- dt[, sum(ppd)]error <- (sdt[, mean(ppd * 365)] - truth)/truth The estimatation error is 6.2%.

Method 2: Weighted MeanMethod 1 underestimates the true number, since we are giving too much weight to summer periods that we know have a lower ridership.Probably a more informed way would be weight results according to our understanding of the estimates representativeness.Vaguely knowing that summer is different from the rest of the year, this could be expressed with two different weights for summer and off-summer.One way to choose weights is to use the inverse of the sampling probability. In this case this means the inverse of sampling a summer month vs. an off-summer month.In our case, we randomly chose three random 4 week periods. For simplicity, we will “round” the period to the more dominant month.The chance of picking a summer month (July or August) is (\frac{2}{12}) and to pick an off-summer month (\frac{10}{12}).So a summer month survey result will be weighted by a factor of (\frac{12}{2} = 6) and an off-summer survey results by (\frac{12}{10} = 1.2).This could also be justified by looking at the how much service level e.g., bus kilometers are performed during summer compared to off-summer for instance, and to realize that just because this it is much more likely to sample less buses in summer.

In R, we can use function weighted.mean for exactly this purpose.

sdt[, weight := mean(month(seq.Date(from = survey\_begin, to = survey\_end, by = "day"))), by = .(survey\_begin, survey\_end)]sdt[, weight := fifelse(month(survey\_begin) %in% c(7, 8), 12/2, 12/10)]print(sdt)## survey\_begin survey\_end ppd weight## <Date> <Date> <num> <num>## 1: 2023-01-07 2023-02-04 2966.356 1.2## 2: 2023-08-22 2023-09-19 2279.425 6.0## 3: 2023-10-10 2023-11-07 2705.054 1.2error <- (sdt[, weighted.mean(x = ppd, w = weight) * 365] - truth)/truth The estimatation error is -2.29%.

Method 3: Linear RegressionIndeeed, method two improves the estimation error.Another way to look at the problem of annualization is to think of it as a dataset with missing data.In these terms, annualization is simply the sum of a value that is with respect to a single day.Having survey results for only parts of a year, another approach can be to estimate values for the missing days.In the context of missing data, if we where sure that the data is missing completely at ranom, we could simply re-sample data from the existing data as a replacement.However, in most cases (including this one), data is not missing completely randomly and it is usually better to use other known correlating factors.For this approach, the problem then is shifted to estimating surrogate values.

Missing value imputation can be done in R by using the package mice, which implementes so-called Fully Conditional Specification.However, the basic idea of this is to use as much information as possible to estimate plausible imputations for missing data using regression.

As a demonstration, we will run a simple linear regression using our correlate of performed kilometers.In order for this to work, we need to “expand” our survey data set to all days to get a dataset with missing entries in column ppd.We can then run a simple base R regression using the lm() function, which by default will discard any rows for which there are missing entries in any of the outcome or predictor variables.The resulting regression object can then be used in conjunction with base R’s predict() function to replace all missing occurrenences of ppd with our estimates.

```

add kilometers performed during the observed survey periodssdt <- sdt[dt, on = c("survey_begin<=date", "survey_end>=date")][, .(survey_begin, survey_end, weight, kms, ppd)]reg <- lm(ppd ~ kms, data = sdt)sdt[, ppd_est := predict(reg, newdata = sdt)]error <- (sdt[, sum(ppd_est)] - truth)/truth

``` With this we get an estimatation error 1.23%.

To leave a comment for the author, please follow the link and comment on their blog: R on R & Data Analysis - Eric Stemmler.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Annualization

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#191–192

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #191Usually on Power Query Challenges we are getting some table transformations, but not this time. We have some pretty hardcore text manipulation here. What we have to do? Get from text only “words” that consists of:

digits mixed with special characters plus capital letters mixed with special characters or
capital letters mixed with special characters plus digits mixed with special characters.

In both cases special characters are optional. But can be mixed as densely as author would only imagine, even after every letter or digit.

But it is not the end, after finding proper strings we need to clear all special characters, and concatenate in order letters > digits, separated with underscore.

I can confess that hardest thing here was to compose regular expression to find those “words”. It took me about 90% of all time consumed for this task. Check this out.

Loading libraries and data library(tidyverse)library(readxl)library(rebus)path = "Power Query/PQ\_Challenge\_191.xlsx"input = read\_excel(path, range = "A1:A11")test = read\_excel(path, range = "A1:B11") Transformation pattern1 = "\\b[A-Z]+(?:[!@#$%^&*\_+=]*[A-Z]*)*[!@#$%^&*\_+=]*[0-9]+(?:[!@#$%^&*\_+=]*[0-9]*)*\\b"pattern2 = "\\b[0-9]+(?:[!@#$%^&*\_+=]*[0-9]*)*[!@#$%^&*\_+=]*[A-Z]+(?:[!@#$%^&*\_+=]*[A-Z]*)*\\b"order\_chars = function(text) { text = str\_replace\_all(text, "[^[:alnum:]]", "") letters = str\_extract\_all(text, "[A-Z]")[[1]] %>% paste0(collapse = "") numbers = str\_extract\_all(text, "[0-9]")[[1]] %>% paste0(collapse = "") result = paste0(letters, "\_", numbers) return(result)}result = input %>% mutate(pat1 = str\_extract\_all(Text, pattern1), pat2 = str\_extract\_all(Text, pattern2)) %>% mutate(ext = map2(pat1, pat2, ~c(.x, .y))) %>% select(-c(pat1, pat2)) %>% unnest(ext, keep\_empty = T) %>% mutate(result = map\_chr(ext, order\_chars)) %>% group\_by(Text) %>% summarise(`Answer Expected` = paste0(result, collapse = ", ")) %>% mutate(`Answer Expected` = if\_else(`Answer Expected` == "NA\_NA", NA\_character\_, `Answer Expected`)) Validation — “by eye” res = left\_join(test, result, by = c("Text" = "Text"))# A tibble: 10 × 3 Text `Answer Expected.x` `Answer Expected.y` <chr> <chr> <chr> 1 Life is beautiful LA$340 LA\_340 LA\_340 2 Q#AR\_8 Dream big, work hard QAR\_8 QAR\_8 3 Actions 55 speak 83\_LDR louder than words LDR\_83 LDR\_83 4 Every Q9#02 MOMENT 89abc matters Q\_902 Q\_902 5 Kindness costs 45A6Q nothing 83 ABC NA NA 6 Believe88 45 you 2\_3*ABC can, and you're halfway there ABC\_23 ABC\_23 7 Happiness Q@56, TY#787 is a choice Q\_56, TY\_787 Q\_56, TY\_787 8 Time 99+RT heals all wounds GHOPQ*45 RT\_99, GHOPQ\_45 GHOPQ\_45, RT\_99 9 Knowledge is ABc\_5726 power 23#PQR PQR\_23 PQR\_23 10 EMBRACE THE A$B$C$2$3$8$8$0 JOURNEY ABC\_23880 ABC\_23880 ExtraI need to explain Regex for those who are not really familiar with it.

\\b[A-Z]+(?:[!@#$%^&*\_+=]*[A-Z]*)*[!@#$%^&*\_+=]*[0-9]+(?:[!@#$%^&*\_+=]*[0-9]*)*\\b- \\b: at the beginning and end are responsible for applying code to "word" which means fragment of string separated by whitespaces.- [A-Z]+: fragment consisting of one or more capital letters- (?:....): is called non-capturing group, I am using it to made group for purpose of checking its multi-occurence and that is why there is * after, which means zero or more occurences.- [!@#$%^&*\_+=]*[A-Z]*: inside non-capturing group means that we are looking for group in which there can be special sign or capital letter. I use it to check if after first segment of letters there are more mixed with special characters.-[!@#$%^&*\_+=]*: we have optional special characters in middle of expression as well- [0-9]+: fragment consisting of one or more digit- (?:[!@#$%^&*\_+=]*[0-9]*)*: and again we have non-capturing group for mixing digits and special characters I hope I don’t need to explain second one, because it is only reversed: digits first than capital letters.

Puzzle #192And we have some table manipulation as well. Today we have project management issue to solve. We have pretty nice data about planned and actual performance of some projects. We have dates for each scenario, and we need to calculate if projects are late, on time or are done faster than planned, and of course if they consume planned time, or maybe more or less. It is not very hard task, but needs a lot of transformations and conditional expressions. Find out yourself.

Loading libraries and data library(tidyverse)library(readxl)path = 'Power Query/PQ\_Challenge\_192.xlsx'input = read\_excel(path, range = "A1:E14")test = read\_excel(path, range = "G1:J6") Transformation count\_workdays <- function(from, to) { map2(from, to, seq, by = "days") %>% map(~ tibble(timeperiod = .x)) %>% map(~ mutate(.x, weekday = wday(timeperiod, week\_start = 1))) %>% map(~ filter(.x, weekday %in% 1:5)) %>% map\_int(~ nrow(.x))}result = input %>% filter\_all(any\_vars(!is.na(.))) %>% fill(everything(), .direction = "down") %>% rename("scenario" = 3) %>% pivot\_wider(names\_from = scenario, values\_from = c(4, 5)) %>% mutate(`Schedule Performance` = case\_when( `To Date\_Actual` > `To Date\_Plan` ~ "Overrun", `To Date\_Actual` < `To Date\_Plan` ~ "Underrun", TRUE ~ "On Time" ), `Actual Dates` = map2\_int(`From Date\_Actual`, `To Date\_Actual`, count\_workdays) , `Plan Dates` = map2\_int(`From Date\_Plan`, `To Date\_Plan`, count\_workdays), `Cost Performance` = case\_when( `Actual Dates` > `Plan Dates` ~ "Overrun", `Actual Dates` < `Plan Dates` ~ "Underrun", TRUE ~ "At Cost" )) %>% mutate(nr = row\_number(), .by = Project) %>% select(Project, Phase, nr, `Schedule Performance`, `Cost Performance`) %>% mutate(Project = if\_else(nr == 1, Project, NA\_character\_)) %>% select(-nr) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Academic and Personal Website Creation: A Quarto Tutorial, which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Academic and Personal Website Creation: A Quarto Tutorial

Date: Thursday, July 18th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Brier Gallihugh is an incoming fifth year doctoral candidate in social psychology at Ohio University. Broadly speaking, research interests involve prejudice and discrimination of minoritized groups. An advocate for all things open science and statistical, Brier spends countless time inside of RStudio and using R for statistical analyses and document/manuscript generation. Post PhD (likely Spring 2025), Brier hopes to gain employment as either a data analyst or data scientist.

Description: In the digital world having an online presence is at worst a strong suggestion and at best a firm requirement for anyone who wishes to advertise what they do. This is true both in academic circles (i.e., lab websites) and industry circles (i.e., portfolios) alike. However, creating websites can often require a vast knowledge in CSS and HTML coding in order to put together a professional product. Thankfully this is where Quarto comes in handy. This workshop will show participants how to get going quickly on creating and hosting a website for professional or personal use tailored to each participants individual needs using Quarto. Participants will need to have the latest versions of both R and RStudio installed prior to the workshop. Further, a GitHub and Netlify account (used to host the website) is also advised.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Academic and Personal Website Creation: A Quarto Tutorial workshop was first posted on June 18, 2024 at 2:03 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Academic and Personal Website Creation: A Quarto Tutorial workshop

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The main criticism people have when it comes to ggplot2 is the static nature of the charts it has to offer. Truth be told, it will never be an interactive visualization king like Highcharts, but it doesn’t mean animation is out of the picture.

Meet R gganimate – a natural extension of ggplot2 that allows you to visualize your data change through time or some other variable, and then render and export the chart as a set of PNGs, or a single GIF/MP4.

You’ll learn all about it today and will end up with some neat looking visualization towards the end of the article. Let’s start small by installing the package first and making your first animated graph.

Is R a good programming language for analyzing huge datasets? It certainly is, but not without some gotchas.

Table of contents:

R gganimate – Your First Animated Chart in R
Rendering Options in R gganimate
How to Make Your R gganimate Charts Stand Out
Summing up R gganimate

R Gganimate – Your First Animated Chart In RThe gganimate package is available on CRAN, which means you can install it by running the install.packages() command through the R console:

install.packages("gganimate") As soon as you load it, you’ll get a warning message like the one below:

Image 1 – gganimate warning noteIt’s nothing concerning, the package is just informing you that you don’t have the necessary packages to render visualizations as GIFs and MP4s, so they’ll be saved as a sequence of individual images instead.

We’ll cross that bridge when we get there.

For now, focus on the snippet below. It calculates the average life expectancy from the Gapminder dataset represented by continent and year. This sort of aggregation is convenient, as it’ll allow you to display an animated representation of average life expectancy through time for every continent.

Down below, the only new function (assuming you’re familiar with ggplot2) is transition\_time(). It’s a function from the gganimate package, and it will instruct ggplot2 to render the visualization for each distinct value of year. It will also interpolate the values that are missing since population data isn’t available at a single year time step.

The animate() function then renders the chart:

library(dplyr)library(gganimate)library(gapminder)avg\_life\_exp\_cont\_year <- gapminder %>% group\_by(continent, year) %>% summarise( avgLifeExp = mean(lifeExp) )p <- ggplot(avg\_life\_exp\_cont\_year, aes(x = continent, y = avgLifeExp)) + geom\_col() + transition\_time(year)animate(p) Since you don’t have the necessary packages (yet) to render the chart as GIF or MP4, a sequence of PNG images is stored on disk:

Image 2 – gganimate chart rendered as imagesEverything works, but doing animation this way is a nightmare, and something you’ll rarely do.

Let’s focus on more convenient formats next.

Rendering Options In R GganimateRendering a gganimate visualization as a GIF or MP4 boils down to calling a slightly different function. For that reason, we’ll show most of the work with GIFs (resizing, changing frame rate) step by step, and only show final implementation with MP4s.

GIFExporting GIFs through gganimate requires you to install the gifski package first:

install.packages("gifski") Once installed, you can call the gifski\_renderer() function to save your animated visualization to a file:

animate( p, renderer = gifski\_renderer(file = "avg\_life\_exp\_cont\_year.gif")) Image 3 – First gganimate chart rendered as a GIFIt exports a low-dpi 480×480 GIF by default, which isn’t ideal for sharing and publication purposes. Luckily, you can change that.

The following code snippet shows how you can change the total number of frames, frames displayed per second, and the total clip duration:

animate( p, renderer = gifski\_renderer(file = "avg\_life\_exp\_cont\_year.gif"), nframes = 600, fps = 60, duration = 10) Image 4 – Increasing frames and durationIt’s better, but the final output is still a bit small. The animate() function also expects values for width and height parameters, which is just what we’re looking for:

animate( p, width = 1024, height = 768, nframes = 600, fps = 60, duration = 10, renderer = gifski\_renderer(file = "avg\_life\_exp\_cont\_year.gif")) Image 5 – Customizing chart height and widthNow the chart looks presentable. Not visually appealing, but has enough resolution for you to share it with your peers.

MP4When it comes to rendering MP4 files, you’ll have to install the av package first:

install.packages("av") Once installed, simply replace gifski\_renderer() with av\_renderer(). All the other parameters passed to animate() remain the same:

animate( p, width = 1024, height = 768, res = 300, nframes = 600, fps = 60, renderer = av\_renderer(file = "avg\_life\_exp\_cont\_year.mp4")) Now you know the basics of rendering an animated visualization in R. The only problem is – they look awful! In the following section, you’ll learn how to animate and style various types of ggplot2 visualizations, starting with the one you already have.

How To Make Your R Gganimate Charts Stand OutThis section will show you how to build and animate bar charts, line charts, and treemaps with gganimate. If you need a refresher on basic data visualization with R, or are looking for your next chart animation idea, these articles will serve as a great starting point:

Bar charts in R and ggplot2
Line charts in R and ggplot2
Scatter plots in R and ggplot2
Boxplots with R and ggplot2
Histograms with R and ggplot2

Let’s proceed by tweaking the chart you already have.

Bar ChartsTo start, let’s slightly modify the data aggregation process. The average life expectancy will now be rounded to 1 decimal place, and an additional column will be created representing the average life expectancy as a string – you’ll see why in a bit.

Then in the plotting section, you’ll need to add a couple more layers to change the color and add text labels at the top of bars. Here, you can see we’re using frame\_time reference in the chart title. It comes from gganimate and is available to you any time you’re using transition\_time(). In other words, it will dynamically change the year in the chart title.

The animation and rendering code remains unchanged:

avg\_life\_exp\_cont\_year <- gapminder %>% group\_by(continent, year) %>% summarise( avgLifeExp = round(mean(lifeExp), 1) ) %>% mutate( avgLifeExpText = as.character(avgLifeExp) )p <- ggplot(avg\_life\_exp\_cont\_year, aes(x = continent, y = avgLifeExp)) + geom\_col(fill = "#0099f9") + geom\_text(aes(label = avgLifeExpText), vjust = -0.5, size = 3) + labs( title = "Average life expectancy by continent for the year {frame\_time}", x = "Continent", y = "Average life expectancy" ) + theme\_minimal(base\_size = 7) + transition\_time(year)animate( p, width = 1024, height = 768, res = 300, nframes = 600, fps = 60, renderer = gifski\_renderer(file = "avg\_life\_exp\_cont\_year.gif")) Image 6 – Fully styled bar chartYou can see how the title year value is updated for every year, even for the years not available in the dataset. The individual bars update every 5 years since that’s the frequency in the underlying data.

Line ChartsThe good thing about gganimate is that you can use it to animate any ggplot2 visualization, provided the data is grouped and organized correctly. We’ll demonstrate that by rendering a line chart showing the average GDP per capita through time, rendered by continent.

There are no significant changes to the code – you’re still aggregating the dataset, creating a chart object, and then rendering it. The main difference is the use of the transition\_states() function. This one is almost always used with categorical variables. Parameters transiton\_length and state\_length control the length of each transition and the length of pause between each transition:

avg\_gdppercap\_cont\_year <- gapminder %>% group\_by(continent, year) %>% summarise( avgGdpPercap = round(mean(gdpPercap)) )p <- ggplot(avg\_gdppercap\_cont\_year, aes(x = year, y = avgGdpPercap)) + geom\_line(color = "#0099f9", linewidth = 1) + geom\_point(color = "#0099f9", size = 3) + labs( title = "Average GDP Per Capita for {closest\_state}", x = "Year", y = "Average GDP Per Capita" ) + theme\_minimal(base\_size = 7) + transition\_states(states = continent, transition\_length = 2, state\_length = 1)animate( p, width = 1024, height = 768, res = 300, renderer = gifski\_renderer(file = "avg\_gdppercap\_cont\_year.gif")) Image 7 – Fully styled line chartIt’s not the best-suited visualization for this dataset, but you get the idea – gganimate is versatile.

TreemapsAnd finally, let’s take a look at treemaps. You’ll need an additional package to work with treemaps in ggplot2 and gganimate. It’s called treemapify, and you can install it through the install.packages() command.

The code snippet below shows you how to create an animated treemap of the European countries’ population through time. The size of each box corresponds to its share in the overall population, and we’ll also add text labels for a bit more context.

Just like with line charts, you’ll also use the transition\_states() transition here:

library(treemapify)eu\_population\_by\_country <- gapminder %>% filter(continent == "Europe") %>% mutate(pop = pop / 1000000) %>% select(-c(lifeExp, gdpPercap))p <- ggplot(eu\_population\_by\_country, aes(area = pop, fill = pop, label = paste(country, round(pop, 2), sep = "\n"))) + geom\_treemap() + geom\_treemap\_text(colour = "white", place = "centre", size = 8) + theme(legend.position = "none") + scale\_fill\_viridis\_c() + labs( title = "Europe population by country in {closest\_state}" ) + transition\_states(states = year, transition\_length = 2, state\_length = 1)animate( p, width = 1920, height = 1080, res = 300, renderer = gifski\_renderer(file = "eu\_population\_by\_country.gif")) Image 8 – Fully styled treemap chartLooks like Germany is in a significant lead from day one! Feel free to play around with the rendered chart resolution, as some of the less populated country names don’t fit into the boxes.

Summing Up R GganimateAnd there you have it – how to visualize ggplot2 charts. It mostly boils down to organizing your data in the correct way and calling the appropriate transition function at the end of the chart code – depending on the type of variable you’re dealing with. Rendering is quick, easy, and highly customizable, both in the realm of file formats and key characteristics, such as height, width, and framerate settings.

This sort of data visualization may not be applicable to web applications, but it sure has its place in online articles and magazines. Remember, if a picture is worth a thousand words, then animation is worth much, much more.

DuckDB makes R crazy fast. Here’s how it compares to dplyr.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Gganimate: How To Make Stunning Chart Animations With Ggplot2

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. AboutThe goal of these solutions is to provide a reference for those who come from Stata and want to learn R. I prioritized readability and simplicity over performance and elegance. There were parts of the code were it was challenging to stick to a literal code translation, and I had to use R idioms to make the code more readable.

You can access the website here, and the repository here.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Advanced International Trade in R

View Details

[This article was first published on Albert Rapp, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Guten Tag!

As data scientists we are judged by how quickly we can generate meaningful insights. But we’re often stuck with cleaning our data first. That’s bad. So that’s why I’m super excited to tell you that my new “Data Cleaning with R Master Class” just launched.

This video course will teach you how to stop wasting time and clean your messy data fast. In fact, I will give you an in-depth curriculum for all your data cleaning needs.

Count me in!

What you can get from the courseOnce you complete the course, I am confident that you can leave that data cleaning mess behind you much quicker and get to your insights faster. Now, how do I plan to teach you how to do that?

I’ve condensed years of experience into five parts with each of them focusing on one specific aspect of your data cleaning efforts. All of these parts first show you the mechanics of useful data functions and then tie everything together with real-world examples.

Here’s the breakdown of the course:

Part 1: Get a deep understanding of the basics of computing and summarizing columns. Improve your productivity with advanced tricks.
Part 2: Learn how to read any file and get it from a messy state into a usable format. This will be particularly useful for Excel files.
Part 3: Knowing how to work with text data (and regex in particular) unlocks so many doors for you. I’ll show you how that works.
Part 4: Working with times and dates is always painful. But your live becomes easier when you use powerful functions from {lubridate}.
Part 5: Unlocking the hard but insanely useful functional programming paradigm will make you so much more productive.

Sign me up!

Where is the special offer ?I know what you’re thinking. A course release is great. But there needs to be some promo offer, right? And you’re right. I’ve got a deal in store for you.

First, this course is currently in pre-sale mode. This means that you get full access to

Part 1 of the course now and
Parts 2, 3, 4 & 5 as they become available in the next couple of months.

And the best part: The course is currently priced at 50% off compared to the final price when the course is completed.

Take me to the course!

This course has a jam-packed & ambitious learning path for you. Completing it will make you so much more efficient at data cleaning. So, if you want to get to your data insights quicker, join the course today.

To leave a comment for the author, please follow the link and comment on their blog: Albert Rapp.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Master Data Cleaning & Get To Insights Faster

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Umar Isah Adam, the founder and organizer of the R User Group Kano, Nigeria, spoke with the R Consortium during the pandemic about his efforts to engage the next generation of students in the R community. Recently, the R Consortium followed up with Umar to discuss the group’s progress over the past few years. He discussed the increasing acceptance and interest in R within academia. The user group is working with various colleges in Kano state to introduce R to students and teach them the fundamentals. Umar also shared his experience using R for managerial tasks related to student data. He hopes to persuade college management to use R for data handling instead of the current manual processes.

Please share your background and involvement with the RUGS group.

My name is Umar Isah Adam, and I’m from Kano State, Nigeria. I studied mathematics at the Federal University Dutse, Jigawa State. During my studies, I became interested in statistics and technology. One of my lecturers mentioned R as a statistical analysis tool, which piqued my interest. I learned it by researching online and watching videos. Later, a friend introduced me to R User Groups. I found that I was interested in R and noticed there wasn’t a group in Kano State, so I applied to start a chapter there, and it was approved.

Can you share what the R community is like in Kano, Nigeria?

The use of R is relatively new in Kano State. Most academics in the area use SPSS in their work. It makes it challenging for R to gain traction in this environment. Despite the challenges, we have been making progress with the support of our user group. Currently, I work as an assistant lecturer at a college in Kano State. I recently organized a well-attended seminar for lecturers and students at the Kano State College of Education and Preliminary Studies. I also posted a video of the workshop on YouTube and have received requests for more information.

There’s room for improvement. We’ve received requests from academic institutions to host events or provide information about the power of R. However, we cannot do so now due to the nature of my work and inadequate funding. However, we plan to start a 10-week training session soon. It will likely be free, as we are collaborating with the Kano State College of Education and Preliminary Studies to organize it. R isn’t very popular here, and more than 70% of academicians need help understanding what it is and how to use it effectively. However, those introduced to it have shown a high interest in learning and utilizing it.

We aim to introduce R to the academic community, and after this, we plan to move on to another college and launch a new program. In summary, R is not widely known in our society, but we are progressing. There has been an increase in the acceptance of R and a growing interest from different people in academia, particularly in R. Many are interested. Still, there needs to be more awareness about it. Most people need to learn what R is and how to use it. Therefore, most of our upcoming programs will focus on introducing the R language.

Additionally, there is an issue with student access. Most of our students don’t have personal computers and can only access them on campus, usually at the ICT department. This lack of access also affects student engagement. However, among academics and lecturers in our colleges, there is growing interest in R.

Do you host in-person or online events? How do you make your events inclusive?

It’s important to remember that online events became essential during the pandemic. However, due to internet connectivity issues, we avoid online meetings or events most of the time. As a result, our sessions are usually held offline. We have been hosting events within colleges and other institutions to make them easily accessible to students and academics. It is also more cost-effective and popular than hosting in private locations. Advertising these events has proven effective, as interested individuals are usually willing to attend when they see the advertisement.

We attempted to transfer between colleges, such as those owned by the state government. The majority of the data and processes are research-based. Therefore, we strive to incorporate more R programming aligned with academic requirements. We aim to limit topics to the use of R in academia to ensure that attendees feel more connected and can see the practical applications of using R. For instance, compared to using SPSS, where one often needs to use code or convert data into another format, with R, one can easily import data into the working environment and manipulate it as needed.

Please share about a project you are currently working on or have worked on using the R language. What is the goal/reason, result, or anything interesting, especially related to your industry?

I usually demonstrate to people around me, including the school management, how easy it is to use R. For example, we need help with the examination office potentially losing some of their data. However, they have a backup on an external drive. I am importing the data from the old template to the new one in Excel format. I am also working on calculating the student results and offloading them into the new portal we have developed. Doing this job manually might take a month, but if I successfully create this program, it will complete the job in two to three days. It will demonstrate to the school management the importance and impact of using R.

I am proposing to the college management to introduce a certified course of study on “Introduction to R” within the ICT department. Showcasing how this programming language can impact the working environment will help them understand the need for this course. Many students rely on fundamental analyses using questionnaires, frequency, and percentage without exploring visualization techniques. As a supervisor, I encourage using R for data analysis in student projects, as it provides a more comprehensive approach. However, many students need access to computers. Therefore, by offering this course, we can equip them with valuable skills and knowledge to benefit their future careers.

Learn moreThe post Bridging the Digital Divide: Umar Isah Adam on Expanding R Access for Kano, Nigeria Students appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Bridging the Digital Divide: Umar Isah Adam on Expanding R Access for Kano, Nigeria Students

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 474–478

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #478Waves of digits in style of Kanagawa wave, created by Dall-e3Today we get some numbers and our task was to detect which one of them are wavy. What does it mean? Digits in those numbers are going up and down, one after another. But how to do it nice and with easy readable code. I choose to use very interesting technique. I did two things. First I checked differences between each pair of consecutive digits, and then because I wouldn’t like to write condition for each possible result, I used function sign(), which gives me one of only 3 values -1 for negatives, 0 for equals, and 1 for positive difference. So all what I have to check if every difference is either +2 or -2 from previous. Check it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/474 Wavy Numbers.xlsx", range = "A1:A10")test = read\_excel("Excel/474 Wavy Numbers.xlsx", range = "B1:B6") Transformation is\_wavy <- function(number) { digits <- str\_split(as.character(number), "")[[1]] %>% as.numeric() differences <- diff(digits) signs <- sign(differences) if (length(signs) < 2) { return(FALSE) } all(abs(diff(signs)) == 2)}result = input %>% mutate(wavy = map\_lgl(Numbers, is\_wavy)) %>% filter(wavy) %>% select(`Answer Expected` = Numbers) Validation identical(result, test)# [1] TRUE Puzzle #475Today we can feel like official guest of some ceremony, because we need to stand in line and cut ribbon with veeeery long word in specific, given places. Is it easy, yes. Is it tricky, oh yes. Look to find out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/475 Split by Positions.xlsx", range = "A2:B12")test = read\_excel("Excel/475 Split by Positions.xlsx", range = "C2:H12") Transformation split\_string\_by\_pos <- function(string, positions\_str) { positions <- str\_split(positions\_str, "\\s*,\\s*") %>% unlist() %>% as.numeric() starts <- c(1, positions) ends <- c(positions - 1, nchar(string)) map2(starts, ends, ~ substr(string, .x, .y))}result = input %>% mutate(split = map2(Names, Position, split\_string\_by\_pos)) %>% unnest\_wider(split, names\_sep = "\_") %>% select(Text1 = split\_1, Text2 = split\_2, Text3 = split\_3, Text4 = split\_4, Text5 = split\_5, Text6 = split\_6) Validation identical(result, test)# [1] TRUE Puzzle #476Today’s is pretty decent data manipulation task. We have stores with sales, but also departments that we need to assign with sales proportionally. Nice, short and simple job. Look closer.

Loading libraries and data library(tidyverse)library(readxl)input1 = read\_excel("Excel/476 Assigning Sales.xlsx", range = "A2:B5")input2 = read\_excel("Excel/476 Assigning Sales.xlsx", range = "D2:E11")test = read\_excel("Excel/476 Assigning Sales.xlsx", range = "G2:I11") Transformation result = input1 %>% left\_join(input2, by = "Store") %>% mutate(n = n(), .by = Store) %>% mutate(Sales = Sales / n) %>% select(Store, Branch, Sales) Validation identical(result, test)# [1] TRUE Puzzle #477So, some challenges are really easy, when you read it, but worse when you start to do something. We have list of people with some values, and we need them to stand in rows, but every consecutive row has to had one person more. O, it was tricky.

Loadingzlibraries and data library(readxl)library(tidyverse)input = read\_excel("Excel/477 Records Split and Alignment.xlsx", range = "A2:B15")test = read\_excel("Excel/477 Records Split and Alignment.xlsx", range = "D2:M6") names(test) = gsub("\\.+\\d+", "", names(test)) Transformation nr = nrow(input)seq = 1i = 1while(sum(seq) <= nr){ seq = c(seq, i) i = i + 1}seq = seq[-1]slice\_dataframe <- function(df, seq) { indices <- map2(c(0, cumsum(seq)[-length(seq)]), cumsum(seq), ~(.x + 1):.y) map(indices, ~df[.x, ])}indexed\_input = slice\_dataframe(input, seq)pad\_and\_bind\_dataframes <- function(dfs) { max\_length <- max(map\_int(dfs, nrow)) pad\_df <- function(df, length) { if (nrow(df) < length) { additional\_rows <- tibble(x = rep(NA, length - nrow(df))) df <- bind\_rows(df, additional\_rows) } df } padded\_dfs <- map(dfs, pad\_df, length = max\_length) bound\_df <- bind\_cols(padded\_dfs) %>% select(-starts\_with("x")) bound\_df <- bound\_df %>% filter\_all(any\_vars(!is.na(.))) bound\_df}result = pad\_and\_bind\_dataframes(indexed\_input)names(result) = gsub("\\.+\\d+", "", names(result)) Validation all.equal(result, test)# [1] TRUE Puzzle #478And again we have two reports with different stores and we need to merge them. If you know dplyr, this task is a piece of cake. Look up for solution.

Loading data and libraries library(tidyverse)library(readxl)path = "Excel/478 Merge Tables.xlsx"input1 = read\_excel(path, range = "A2:C9")input2 = read\_excel(path, range = "E2:H10")test = read\_excel(path, range = "J2:M14") Transformation result = input1 %>% full\_join(input2, by = c("Org", "Year")) %>% arrange(Org, Year) %>% mutate(Sales = map2\_dbl(Sales.x, Sales.y, ~ sum(c(.x, .y), na.rm = TRUE))) %>% select(Org, Year, Prime, Sales) Validation identical(result, test)#> [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
PS. Couple weeks ago, I started uploading on Github not only R, but also in Python. Come and check it.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. AboutLast week I wrote that the 2nd edition of The Hitchhiker’s Guide to Linear Models can be downloaded for free (or for a suggested price of 10 USD) from Leanpub.

The 2nd edition now also features a sale featuring an extra with all the datasets and codes used in the book. These are presented as RStudio projects with R scripts to enhance the hands-on book experience. This extra is sold for 20 USD (the normal price is 29 USD) and the promotion will last until June 21th, 2024.

For every exercise I did my best to connect the specific statistical concepts with R code, and every time I use linear algebra I connect it with a concrete R example. In this book you will not find something such as “this is left as an exercise to the reader”.

This books contains no proofs. I tried to replace those with multiple examples consisting in analyzing my own experiments, such as throwing a tennis ball and measuring the time it takes to hit the ground from different heights, and another where I got two thermometers and measured the temperature outside a building at the same time of the day on different days.

Table of contents1. Preface 2. R Setup 1. R and RStudio 1. Windows and Mac 2. Linux 2. Installing R 1. Windows and Mac 2. Linux 3. Installing RStudio 1. Windows and Mac 2. Linux 4. Installing R Packages 1. Windows and Mac 2. Linux 5. Changing RStudio colors and font 1. Windows and Mac 2. Linux 6. Installing Quarto 1. Windows and Mac 2. Linux 3. Linear algebra review 1. Using R as a calculator 2. System of linear equations 3. Matrix 4. Transpose matrix 5. Matrix multiplication 6. Matrix representation of a system of linear equations 7. Identity matrix 8. Inverse matrix 9. Solving systems of linear equations 4. Statistics review 1. Using R as a calculator 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 2. Data and dataset 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 3. Summation 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 4. Probability 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 5. Descriptive statistics 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 6. Distributions 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 7. Sample size 1. Mean 2. Variance 3. Standard deviation 4. Covariance 5. Correlation 6. Normal distribution 7. Poisson distribution 8. Student’s t-distribution 9. Computing probabilities with the normal distribution 10. Computing probabilities with the Poisson distribution 11. Computing probabilities with the t-distribution 5. Recommended workflow 1. Creating projects 2. Creating scripts 3. Creating notebooks 4. Organizing code sections 5. Customizing notebooks’ output 6. Read, Manipulate, and Plot Data 1. The datasauRus dataset in R format 2. The Quality of Government dataset in CSV format 3. The Quality of Government dataset in SAV (SPSS) format 4. The Quality of Government dataset in DTA (Stata) format 5. The Freedom House dataset in XLSX (Excel) format 7. Linear Model with One Explanatory Variable 1. Model specification 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 2. The Galton dataset 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 3. A word of caution about Galton’s work 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 4. Loading the Galton dataset 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 5. Estimating linear models’ coefficients 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 6. Logarithmic transformations 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 7. Plotting model results 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 8. Linear model does not equal straight line 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 9. Transforming variables 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 10. Regression with weights 1. Linear model as correlation 2. Linear model as matrix multiplication 3. Relation between correlation and matrix multiplication 4. Computational note 8. Linear Model with Multiple Explanatory Variables 1. Model specification 1. Root Mean Squared Error and Mean Absolute Error 2. RMSE and MAE interpretation 3. Coefficient’s standard error 4. Coefficient’s t-statistic 5. Coefficient’s p-value 6. Residual standard error 7. Model’s multiple R-squared (or unadjusted R-squared) 8. Model’s adjusted R-squared 9. Model’s F-statistic 10. Error’s normality 11. Error’s homoscedasticity (homogeneous variance) 2. Life expectancy, GDP and well-being in the Quality of Government dataset 1. Root Mean Squared Error and Mean Absolute Error 2. RMSE and MAE interpretation 3. Coefficient’s standard error 4. Coefficient’s t-statistic 5. Coefficient’s p-value 6. Residual standard error 7. Model’s multiple R-squared (or unadjusted R-squared) 8. Model’s adjusted R-squared 9. Model’s F-statistic 10. Error’s normality 11. Error’s homoscedasticity (homogeneous variance) 3. Estimating linear models’ coefficients 1. Root Mean Squared Error and Mean Absolute Error 2. RMSE and MAE interpretation 3. Coefficient’s standard error 4. Coefficient’s t-statistic 5. Coefficient’s p-value 6. Residual standard error 7. Model’s multiple R-squared (or unadjusted R-squared) 8. Model’s adjusted R-squared 9. Model’s F-statistic 10. Error’s normality 11. Error’s homoscedasticity (homogeneous variance) 4. Model accuracy 1. Root Mean Squared Error and Mean Absolute Error 2. RMSE and MAE interpretation 3. Coefficient’s standard error 4. Coefficient’s t-statistic 5. Coefficient’s p-value 6. Residual standard error 7. Model’s multiple R-squared (or unadjusted R-squared) 8. Model’s adjusted R-squared 9. Model’s F-statistic 10. Error’s normality 11. Error’s homoscedasticity (homogeneous variance) 5. Model summary 1. Root Mean Squared Error and Mean Absolute Error 2. RMSE and MAE interpretation 3. Coefficient’s standard error 4. Coefficient’s t-statistic 5. Coefficient’s p-value 6. Residual standard error 7. Model’s multiple R-squared (or unadjusted R-squared) 8. Model’s adjusted R-squared 9. Model’s F-statistic 10. Error’s normality 11. Error’s homoscedasticity (homogeneous variance) 6. Error’s assumptions 1. Root Mean Squared Error and Mean Absolute Error 2. RMSE and MAE interpretation 3. Coefficient’s standard error 4. Coefficient’s t-statistic 5. Coefficient’s p-value 6. Residual standard error 7. Model’s multiple R-squared (or unadjusted R-squared) 8. Model’s adjusted R-squared 9. Model’s F-statistic 10. Error’s normality 11. Error’s homoscedasticity (homogeneous variance) 9. Linear Model with Binary and Categorical Explanatory Variables 1. Model specification with binary variables 1. ANOVA is a particular case of a linear model with binary variables 2. Corruption and popular vote in the Quality of Government dataset 3. Estimating a linear model and ANOVA with one predictor and two categories 4. Corruption and regime type in the Quality of Government dataset 5. Estimating a linear model and ANOVA with one predictor and multiple categories 6. Estimating a linear model with continuous and categorical predictors 7. Corruption and interaction variables in the Quality of Government dataset 8. Estimating a linear model with binary interactions 9. Confidence intervals with binary interactions 10. Estimating a linear model with categorical interactions 11. Confidence intervals with categorical interactions 2. Model specification with binary interactions 1. ANOVA is a particular case of a linear model with binary variables 2. Corruption and popular vote in the Quality of Government dataset 3. Estimating a linear model and ANOVA with one predictor and two categories 4. Corruption and regime type in the Quality of Government dataset 5. Estimating a linear model and ANOVA with one predictor and multiple categories 6. Estimating a linear model with continuous and categorical predictors 7. Corruption and interaction variables in the Quality of Government dataset 8. Estimating a linear model with binary interactions 9. Confidence intervals with binary interactions 10. Estimating a linear model with categorical interactions 11. Confidence intervals with categorical interactions 3. Model specification with categorical interactions 1. ANOVA is a particular case of a linear model with binary variables 2. Corruption and popular vote in the Quality of Government dataset 3. Estimating a linear model and ANOVA with one predictor and two categories 4. Corruption and regime type in the Quality of Government dataset 5. Estimating a linear model and ANOVA with one predictor and multiple categories 6. Estimating a linear model with continuous and categorical predictors 7. Corruption and interaction variables in the Quality of Government dataset 8. Estimating a linear model with binary interactions 9. Confidence intervals with binary interactions 10. Estimating a linear model with categorical interactions 11. Confidence intervals with categorical interactions 10. Linear Model with Fixed Effects 1. Year fixed effects 1. Model specification 2. Corruption and popular vote in the Quality of Government dataset 3. Estimating year fixed effects’ coefficients 4. Estimating country-time fixed effects’ coefficients 2. Country fixed effects 1. Model specification 2. Corruption and popular vote in the Quality of Government dataset 3. Estimating year fixed effects’ coefficients 4. Estimating country-time fixed effects’ coefficients 3. Country-year fixed effects 1. Model specification 2. Corruption and popular vote in the Quality of Government dataset 3. Estimating year fixed effects’ coefficients 4. Estimating country-time fixed effects’ coefficients 11. Generalized Linear Model with One Explanatory Variable 1. Model specification 1. Gaussian model 2. Poisson model 3. Quasi-Poisson model 4. Binomial model (or logit model) 2. Model families 1. Gaussian model 2. Poisson model 3. Quasi-Poisson model 4. Binomial model (or logit model) 12. Generalized Linear Model with Multiple Explanatory Variables 1. Obtaining the original codes and data 2. Loading the original data 3. Ordinary Least Squares 4. Poisson Pseudo Maximum Likelihood 5. Tobit 6. Reporting multiple models

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Hitchhiker’s Guide to Linear Models with codes and datasets sale

View Details

[This article was first published on R/exams, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Ideas and experiences from an award-winning bridging course in mathematics at Universität Innsbruck, whose teaching and examination culture is guided by learning outcomes and extensively uses R/exams.

Guest post by Pia Tscholl & Lisa Schlosser (Universität Innsbruck).

MotivationUniversities in German-speaking countries face above-average dropout rates in STEM (Science, Technology, Engineering, Mathematics) subjects. These high dropout rates are worrying, as the demand for qualified workers in these fields exceeds the available supply. Especially deficient mathematical knowledge plays a major role during the dropout process in STEM fields. For this reason, the majority of the German-speaking universities offer mathematical remedial or bridging courses at the beginning of STEM degree programs. However, researchers criticize that many remedial courses do not offer audience-appropriate assessments to efficiently diagnose mathematical deficits. Echoing this critique, a team at the University of Innsbruck has developed a mathematical self-assessment, which is implemented via R/exams and made available to participants of the mathematical remedial course via the university’s learning management system OpenOlat. The implementation with R/exams offers the advantage of large test sets with (potentially) randomized tasks which can be easily modified and evaluated in an automated manner. Due to the automated evaluation, the self-assessment can be used in courses with a large number of participants. Moreover, the created assessment can be easily exported to other formats (pdf, moodle, …).

Creating task lists by content areaThe self-assessment covers the following content areas (with abbreviations based on the German titles):

AK: General mathematical competencies.
ZF: Numbers and functions.
GU: Equations and inequalities.
KV: Coordinates and vector geometry.
GT: Geometry and trigonometry.
DR: Differential calculus.
IR: Integral calculus.
SW: Statistics and probability.

A separate test set with a certain number of R/exams tasks is created for each content area. All tasks are saved in the same folder and named as follows: “ContentArea_AscendingNumber.Rmd”, e.g. the first task of the content area ZF is named ZF_001.Rmd, the second ZF_002.Rmd and so on. Consistent naming makes it possible to quickly create task lists by content area:

testset\_ZF <- list.files(path = "wd/task", pattern = "^ZF\_[0-9]+\\.Rmd$") where wd is your current working directory or the path where your task folder is located and task is the name of the folder where the tasks are stored. The argument pattern is a regular expression starting with ZF (or another content area) followed by _, some digits and ending with .Rmd. Subsequently, testset_ZF contains the ZF task list to be exported to OpenOlat.

Example: Defining a taskAs an example, the code for a specific randomized task is presented below. It asks the participants to compare two randomly selected fractions to check which one is greater or whether they are equal. In addition to the code being shown below (in R/Markdown format) it can also be downloaded in both R/Markdown and R/LaTeX format as ZF_071.Rmd or ZF_071.Rnw, respectively.

{r, include = FALSE}## Generate two random fractions based on numbers from 1 to 10f1 <- sample(1:10, 2, replace = FALSE)f2 <- sample(1:10, 2, replace = FALSE)a <- f1[1] b <- f1[2] c <- f2[1] d <- f2[2]## Save random fractions a/b and c/d as character## (don't forget escaping \ within the math mode)fr1 <- paste0("$\frac{", a, "}{", b, "}$")fr2 <- paste0("$\frac{", c, "}{", d, "}$")## Possible answersanswers <- c( paste(fr1, "is greater than", fr2), paste(fr2, "is greater than", fr1), "Both fractions are equal")## Correct solutionsol <- c(0, 0, 0)if(a/b > c/d) { sol[1] <- 1} else if(c/d > a/b) { sol[2] <- 1} else { sol[3] <- 1}## Explanationk <- answers[as.logical(sol)]eq <- c(" > ", " < ", " = ")[as.logical(sol)]explanation <- paste0(k, " since $", a, " \cdot ", d, eq, b, " \cdot ", c, "$.")Question========Which of these numbers is greater: `r fr1` or `r fr2`?{r, echo = FALSE, results = "asis"}answerlist(answers, markup = "markdown")Solution========`r explanation`Meta-information================exname: Comparing fractionsextype: schoiceexsolution: `r paste(sol, collapse = "")` One random version of the exercise is shown below as rendered in an OpenOlat test after entering an incorrect answer:

Further meta-information could be added to the exercise via the exextra tags, if needed. For num questions, the tag extol defines the tolerance range for numerical solutions. Exercise templates for different task types (num, mchoice, schoice, cloze, …) are provided on the R/exams web page at https://www.R-exams.org/templates/.

The elegant aspect of randomized tasks is that students can perform the same test multiple times for the purpose of practice or improved self-assessment. If implemented foresighted like in our example, solutions and explanations are automatically adapted - so there is no additional work for teachers despite many task variations.

The next section will explain how to export such randomized tasks from R/exams to OpenOlat.

Randomized export to OpenOlatAs explained in the previous section, consistent file naming makes it easy to create a list of test files, hereinafter referred to as testset_ZF. The following code exemplifies how such a test set, consisting of several randomized tasks, can be exported from R/exams to OpenOlat.

library("exams") # Load R/exams packageseed <- 6020 # Select seed, so randomization can be repeatedset.seed(seed) # Set seedexport\_ZF <- exams2openolat( # Function for exporting tasks to OpenOlat edir = "wd/tasks", # Directory where task files are stored file = testset\_ZF, # File names of the tasks n = 20, # Number of randomized versions of the test set name = paste0("ZF\_Testset\_seed", seed), # Remember seed also in file name stitle = "ZF", # Section title ititle = "Aufgaben", # Item title solutionswitch = TRUE # Display explanation right after saving the answer) Further arguments, such as cutvalue (threshold for passing), navigation (disabling switching between tasks), and duration (maximum processing time) can individualize the test implementation regarding the needs of the course (see this blog post for some practical illustration).

The exams2openolat function creates a zip folder with the chosen name in the current working directory.

Now, you can insert a new Test element in your OpenOlat course. In the editor mode, it is possible to import the created zip folder to the Test-Konfiguration element. At this point, the work is already done - yay! Each time the test is called up, one of the created randomized test sets is selected by OpenOlat. If desired, further settings can be done using the editor mode in OpenOlat, such as displaying the test results after the test has been completed (Testkonfiguration > Report), a winner’s podium for the three best (anonymous) participants (HighScore > Siegertreppchen), or an automated submission confirmation by E-mail (Email Bestätigung).

It is noteworthy - since this question occurs often among our students - that decimal input for num questions can be entered in the OpenOlat export with dot or comma.

Results and feedbackOnce a class or group has finished the self-assessment, various tools provide information on the results. For example, an anonymous ranking showing the achieved number of points of the three best performing participants and a histogram over all point results can provide a first overview on the results.

Moreover, each participant is informed about his/her performance after finishing the assessment for one of the content areas. In particular, the total number of achieved points, the time needed to finish and the number of answered questions is listed. Additionally, each question, the given answer, and the expected answer can be accessed by the participant.

Experience and outlookA problem that still arises with num tasks is the incorporation of fractions. While they can be properly in the tasks using mathematical notation, there is still no convenient way to insert fractions as solutions when completing the test set. As of now, fractions can only be transmitted as decimal numbers, which can be tedious and prone to errors. One possibility would be to implement the numerator and denominator as two num solution-elements of a cloze task in R/exams. However, this would have the disadvantage that reduced or expanded variants of the respective fractions are not recognized as solutions. In this case, the most reduced fraction would have to be requested as a solution, which in turn is again prone to errors. Furthermore, if only the numerator or only the denominator is inserted correctly (and the solution is therefore wrong overall), 50% of the points are still awarded.

Additionally, it would be desirable to provide students with the solutions, including implemented explanations for the tasks, after completing the test. It is currently only possible to display explanations immediately after saving the answer, but not as final feedback.

However, we received primarily positive feedback from students and colleagues regarding the implementation of our mathematics self-assessment using R/exams in OpenOlat.

The concept for this Bridging Course Mathematics even won the Austrian teaching award Ars Docendi State Prize 2023. The photo below shows the entire team Tobias Hell, Elisabeth Hell, Pia Tscholl, and Lisa Schlosser (left to right) being presented with the award by Secretary of Education Martin Polaschek (Photo: Martin Lusser/BMBWF).

One advantage of this digitalized self-assessment - in addition to those already mentioned - is the ability to collect data in larger quantities and without further effort. For example, a full PhD position has been realized to evaluate the data collected through the presented self-assessment.

To leave a comment for the author, please follow the link and comment on their blog: R/exams.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mind the Gap: Using R/exams to Ease the Transition into STEM Studies

View Details

[This article was first published on R Archives - Dan Oehm | Gradient Descending, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Alone Australia season 2 has finished and is now available in the package and ready for analysis. As per usual install via Git or CRAN.

devtools::install\_github("doehm/alone")install.packages("alone") Any issues please raise them via Git.

Survival AnalysisIt was another great season. Season 1 started off rough with a few early taps, but those in season 2 hung around for a little longer which shows in the survival chart. While the average days lasted is longer for season 2, Gina still holds the record for lasting 67 days.

Over the next few weeks, I’ll update my analysis comparing the US and AU versions.

Two key pieces of information missing from the data are the full names of the contestants and the loadouts for AU. I haven’t been able to find this data anyway. If you do come across it, let me know and I can add it to the package.

Alone Season 11Alone season 11 has just started and will be available after the season. In the meantime, you can see the results and grab what data is available from Google Sheets.

The post {alone} v0.4 is now available appeared first on Dan Oehm | Gradient Descending.

To leave a comment for the author, please follow the link and comment on their blog: R Archives - Dan Oehm | Gradient Descending.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {alone} v0.4 is now available

View Details

[This article was first published on Albert Rapp, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. In this blog post I’m going to show you the six most fundamental functions for your data cleaning journey. And as always, you can watch the video version of this blog post on YouTube:

Getting startedAll of these functions come from the tidyverse. So this is why we are going to load that package.

library(tidyverse) And of course, we’re going to need some data, and this is where the palmerpenguins package comes in. We can either load the palmerpenguins package like we did with the tidyverse, or we can just access the penguins data set via its full name, namely the palmerpenguins package name followed by :: followed by the data set name.

penguins <- palmerpenguins::penguinspenguins## # A tibble: 344 × 8## species island bill\_length\_mm bill\_depth\_mm flipper\_length\_mm body\_mass\_g## <fct> <fct> <dbl> <dbl> <int> <int>## 1 Adelie Torgersen 39.1 18.7 181 3750## 2 Adelie Torgersen 39.5 17.4 186 3800## 3 Adelie Torgersen 40.3 18 195 3250## 4 Adelie Torgersen NA NA NA NA## 5 Adelie Torgersen 36.7 19.3 193 3450## 6 Adelie Torgersen 39.3 20.6 190 3650## 7 Adelie Torgersen 38.9 17.8 181 3625## 8 Adelie Torgersen 39.2 19.6 195 4675## 9 Adelie Torgersen 34.1 18.1 193 3475## 10 Adelie Torgersen 42 20.2 190 4250## # ℹ 334 more rows## # ℹ 2 more variables: sex <fct>, year <int> Count all the things with count()Once we have that data set, we can pass it to the count() function. Once we do that, we see in the console that we get a count.

penguins |> count()## # A tibble: 1 × 1## n## <int>## 1 344 But this output right now isn’t particularly useful because it just shows us how many things are in this data set. You see, count() becomes way more useful when you also add a column name from the dataset inside of count(). That way, we could count how many different species are in the data set.

penguins |> count(species)## # A tibble: 3 × 2## species n## <fct> <int>## 1 Adelie 152## 2 Chinstrap 68## 3 Gentoo 124 So that’s how we know how many Adelie, Gentoo and Chinstrap penguins are there. Furthermore, we can also sort the rows in this data set by also adding the sort = TRUE argument inside of count(). This is particularly helpful if you have a data set that has a whole bunch of different things inside one column and you want to get the most frequent ones at the top.

penguins |> count(species, sort = TRUE)## # A tibble: 3 × 2## species n## <fct> <int>## 1 Adelie 152## 2 Gentoo 124## 3 Chinstrap 68 And we could even count combinations of different columns.

penguins |> count(species, island, sort = TRUE)## # A tibble: 5 × 3## species island n## <fct> <fct> <int>## 1 Gentoo Biscoe 124## 2 Chinstrap Dream 68## 3 Adelie Dream 56## 4 Adelie Torgersen 52## 5 Adelie Biscoe 44 Let me note that count() is super useful because obviously you can count things, but also I use it all of the time as intermediate steps when I filter or rearrange some part of the data. This helps me to check that the count of what is now inside of a data set matches what I expect after some data operation.

Get the right columns with select()Next let us talk about the select() function. This function helps you select columns from your data set. Say, you have a data set with a whole bunch of columns and you want to focus on a subset of that. Then you can just

take your data,
pass it to the select() function, and
list all of the things that you want to select.

penguins |> select(flipper\_length\_mm, bill\_length\_mm, species)## # A tibble: 344 × 3## flipper\_length\_mm bill\_length\_mm species## <int> <dbl> <fct> ## 1 181 39.1 Adelie ## 2 186 39.5 Adelie ## 3 195 40.3 Adelie ## 4 NA NA Adelie ## 5 193 36.7 Adelie ## 6 190 39.3 Adelie ## 7 181 38.9 Adelie ## 8 195 39.2 Adelie ## 9 193 34.1 Adelie ## 10 190 42 Adelie ## # ℹ 334 more rows That’s was pretty easy, right? Here, I kind of want to leave it at this stage. But be aware that there are a whole bunch of advanced tricks that you can use in select() to not have to list all of the things.

Instead, you can just describe the things that you want. For example, you can get all the columns

that have a particular text in their name (like _mm in the penguins dataset) or
that are of specific type (e.g. numeric)

In these cases, you can use a couple of tricks in combination with select() that make your life much easier. But here I really want to focus on the fundamentals. So let’s ignore those advanced tricks for now.

Reduce the rows to specific observations with filter()The next essential function is filter(). It is like select() but for rows. Using that function you can filter your data according to specific criteria. That way, you get all of the columns of the data that you pass to filter(), but you get only the rows that match a specific criteria.

For example, we can

take our penguins data,
pass it to the filter() function, and
state that we want to get only the penguins whose bill length is larger than 55 millimeters.

penguins |> filter(bill\_length\_mm > 55)## # A tibble: 5 × 8## species island bill\_length\_mm bill\_depth\_mm flipper\_length\_mm body\_mass\_g## <fct> <fct> <dbl> <dbl> <int> <int>## 1 Gentoo Biscoe 59.6 17 230 6050## 2 Gentoo Biscoe 55.9 17 228 5600## 3 Gentoo Biscoe 55.1 16 230 5850## 4 Chinstrap Dream 58 17.8 181 3700## 5 Chinstrap Dream 55.8 19.8 207 4000## # ℹ 2 more variables: sex <fct>, year <int> I’ll explain how to read that code in a second (in case you don’t find it intuitive.) Let me add one more example first, though. For instance, we could also use the between() function in combination with filter() to find the penguins that have a bill length larger than 55 but smaller than 58.

penguins |> filter(between(bill\_length\_mm, 55, 58))## # A tibble: 4 × 8## species island bill\_length\_mm bill\_depth\_mm flipper\_length\_mm body\_mass\_g## <fct> <fct> <dbl> <dbl> <int> <int>## 1 Gentoo Biscoe 55.9 17 228 5600## 2 Gentoo Biscoe 55.1 16 230 5850## 3 Chinstrap Dream 58 17.8 181 3700## 4 Chinstrap Dream 55.8 19.8 207 4000## # ℹ 2 more variables: sex <fct>, year <int> Okay, here you might wonder how to understand this thing here. So, let me give you an insight of how filter() function does things.

Basically, whenever you describe a condition inside of filter(), then what happens is that it takes the column that you used and applies the command to get a vector full of TRUE and FALSE. Let’s try this out manually to see what happens. Let’s grab the bill lengths.

bill\_length\_mm## Error in eval(expr, envir, enclos): object 'bill\_length\_mm' not found As you can see, we can’t just access the data column via its name outside of the filter() function. R simply doesn’t know where this column name is coming from. Inside of the filter() function R knows to look at the data set. But outside of filter() you will have to tell R that this thing comes from our penguins data set. You do that by via the data set name and the $ operator.

penguins$bill\_length\_mm## [1] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42.0 37.8 37.8 41.1 38.6 34.6## [16] 36.6 38.7 42.5 34.4 46.0 37.8 37.7 35.9 38.2 38.8 35.3 40.6 40.5 37.9 40.5## [31] 39.5 37.2 39.5 40.9 36.4 39.2 38.8 42.2 37.6 39.8 36.5 40.8 36.0 44.1 37.0## [46] 39.6 41.1 37.5 36.0 42.3 39.6 40.1 35.0 42.0 34.5 41.4 39.0 40.6 36.5 37.6## [61] 35.7 41.3 37.6 41.1 36.4 41.6 35.5 41.1 35.9 41.8 33.5 39.7 39.6 45.8 35.5## [76] 42.8 40.9 37.2 36.2 42.1 34.6 42.9 36.7 35.1 37.3 41.3 36.3 36.9 38.3 38.9## [91] 35.7 41.1 34.0 39.6 36.2 40.8 38.1 40.3 33.1 43.2 35.0 41.0 37.7 37.8 37.9## [106] 39.7 38.6 38.2 38.1 43.2 38.1 45.6 39.7 42.2 39.6 42.7 38.6 37.3 35.7 41.1## [121] 36.2 37.7 40.2 41.4 35.2 40.6 38.8 41.5 39.0 44.1 38.5 43.1 36.8 37.5 38.1## [136] 41.1 35.6 40.2 37.0 39.7 40.2 40.6 32.1 40.7 37.3 39.0 39.2 36.6 36.0 37.8## [151] 36.0 41.5 46.1 50.0 48.7 50.0 47.6 46.5 45.4 46.7 43.3 46.8 40.9 49.0 45.5## [166] 48.4 45.8 49.3 42.0 49.2 46.2 48.7 50.2 45.1 46.5 46.3 42.9 46.1 44.5 47.8## [181] 48.2 50.0 47.3 42.8 45.1 59.6 49.1 48.4 42.6 44.4 44.0 48.7 42.7 49.6 45.3## [196] 49.6 50.5 43.6 45.5 50.5 44.9 45.2 46.6 48.5 45.1 50.1 46.5 45.0 43.8 45.5## [211] 43.2 50.4 45.3 46.2 45.7 54.3 45.8 49.8 46.2 49.5 43.5 50.7 47.7 46.4 48.2## [226] 46.5 46.4 48.6 47.5 51.1 45.2 45.2 49.1 52.5 47.4 50.0 44.9 50.8 43.4 51.3## [241] 47.5 52.1 47.5 52.2 45.5 49.5 44.5 50.8 49.4 46.9 48.4 51.1 48.5 55.9 47.2## [256] 49.1 47.3 46.8 41.7 53.4 43.3 48.1 50.5 49.8 43.5 51.5 46.2 55.1 44.5 48.8## [271] 47.2 NA 46.8 50.4 45.2 49.9 46.5 50.0 51.3 45.4 52.7 45.2 46.1 51.3 46.0## [286] 51.3 46.6 51.7 47.0 52.0 45.9 50.5 50.3 58.0 46.4 49.2 42.4 48.5 43.2 50.6## [301] 46.7 52.0 50.5 49.5 46.4 52.8 40.9 54.2 42.5 51.0 49.7 47.5 47.6 52.0 46.9## [316] 53.5 49.0 46.2 50.9 45.5 50.9 50.8 50.1 49.0 51.5 49.8 48.1 51.4 45.7 50.7## [331] 42.5 52.2 45.2 49.3 50.2 45.6 51.9 46.8 45.7 55.8 43.5 49.6 50.8 50.2 So now you see that we get a vector full of the length that we want to look at. Now, what happens inside of filter() is that this vector is applied according to what you specified. In the first example, we just checked is this thing larger than 55. Here’s what that results in.

penguins$bill\_length\_mm > 55## [1] FALSE FALSE FALSE NA FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [37] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [49] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [61] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [73] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [85] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [97] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [109] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [121] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [133] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [145] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [157] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [169] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [181] FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE## [193] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [205] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [217] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [229] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [241] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [253] FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [265] FALSE FALSE FALSE TRUE FALSE FALSE FALSE NA FALSE FALSE FALSE FALSE## [277] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [289] FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE## [301] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [313] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [325] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE## [337] FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE Now you see that what you get is a vector full of TRUEs and FALSEs. And as you can probably guess:

TRUE corresponds to the occasions where the bill length was indeed larger than 55, and
FALSE corresponds to the cases where this wasn’t the case.

Really, what you put inside of filter() are operations that at the end of the day deliver TRUEs and FALSEs. Then, filter() just takes the rows that correspond to TRUE. You could simulate this by taking these TRUEs and FALSEs that we’ve just captured manually and sticking that into the which() function.

which(penguins$bill\_length\_mm > 55)## [1] 186 254 268 294 340 This gives you all of the row numbers for which you had a TRUE, and these are the rows that filter() will return.

Now why did I explain this in so much detail when it was really a simple operation? The thing is, I want you to understand this as things become more complicated. For example, with our second example using the between() function, you might wonder why this was valid syntax. Basically, you have to understand what between() does. So here’s an example.

between(c(54, 55, 56, 57, 58), 55, 57)## [1] FALSE TRUE TRUE TRUE FALSE As you can see, between() is a function that does exactly what we just talked about, i.e. it returns TRUEs and FALSEs. So really, this was just a different operation that has some slightly other syntax. But the syntax doesn’t matter to us as long as at the end of the day this operation that we use inside of filter() will return TRUEs and FALSEs so that filter() can do its thing.

Calculating and transforming columns with mutate()Now let’s compute new things from our data. For example, we can compute a new column that takes the bill lengths that we’ve just looked at and scales them. Scaling is a technique used to transform the values of a variable so that they are centered around zero. This transformation allows you to get a better understanding of whether a value is high or low relative to the other values in the dataset (even if you are not familiar with the specific units of measurement.)

For example, if you scale the bill length variable, a scaled value of

0 would indicate an average bill length,
while a positive value would indicate an above-average bill length, and
a negative value would indicate a below-average bill length.

And to do this scaling, we need the mutate() function. It allows you to modify existing columns or create new columns in a dataset. Here’s how it works:

Pass your data set to the mutate() function.
Specify the column name and set it equal to the desired transformation.

penguins |> mutate(bill\_length\_mm = scale(bill\_length\_mm)[,1]) |> select(bill\_length\_mm)## # A tibble: 344 × 1## bill\_length\_mm## <dbl>## 1 -0.883## 2 -0.810## 3 -0.663## 4 NA ## 5 -1.32 ## 6 -0.847## 7 -0.920## 8 -0.865## 9 -1.80 ## 10 -0.352## # ℹ 334 more rows And via the same strategy, we can also create a new column. We just have to assign a new column name.

penguins |> mutate(scaled\_bill\_length = scale(bill\_length\_mm)[,1]) |> select(bill\_length\_mm, scaled\_bill\_length)## # A tibble: 344 × 2## bill\_length\_mm scaled\_bill\_length## <dbl> <dbl>## 1 39.1 -0.883## 2 39.5 -0.810## 3 40.3 -0.663## 4 NA NA ## 5 36.7 -1.32 ## 6 39.3 -0.847## 7 38.9 -0.920## 8 39.2 -0.865## 9 34.1 -1.80 ## 10 42 -0.352## # ℹ 334 more rows Behind the scenes, mutate() takes the specified column…

scale(penguins$bill\_length\_mm)[,1]## [1] -0.88320467 -0.80993901 -0.66340769 NA -1.32279862 -0.84657184## [7] -0.91983750 -0.86488825 -1.79902541 -0.35202864 -1.12131806 -1.12131806## [13] -0.51687637 -0.97478674 -1.70744334 -1.34111504 -0.95647033 -0.26044656## [19] -1.74407616 0.38062795 -1.12131806 -1.13963448 -1.46932994 -1.04805240## [25] -0.93815391 -1.57922843 -0.60845845 -0.62677486 -1.10300165 -0.62677486## [31] -0.80993901 -1.23121655 -0.80993901 -0.55350920 -1.37774787 -0.86488825## [37] -0.93815391 -0.31539581 -1.15795089 -0.75498976 -1.35943145 -0.57182562## [43] -1.45101353 0.03261607 -1.26784938 -0.79162259 -0.51687637 -1.17626731## [49] -1.45101353 -0.29707939 -0.79162259 -0.70004052 -1.63417768 -0.35202864## [55] -1.72575975 -0.46192713 -0.90152108 -0.60845845 -1.35943145 -1.15795089## [61] -1.50596277 -0.48024354 -1.15795089 -0.51687637 -1.37774787 -0.42529430## [67] -1.54259560 -0.51687637 -1.46932994 -0.38866147 -1.90892390 -0.77330618## [73] -0.79162259 0.34399512 -1.54259560 -0.20549732 -0.55350920 -1.23121655## [79] -1.41438070 -0.33371222 -1.70744334 -0.18718091 -1.32279862 -1.61586126## [85] -1.21290014 -0.48024354 -1.39606428 -1.28616579 -1.02973599 -0.91983750## [91] -1.50596277 -0.51687637 -1.81734182 -0.79162259 -1.41438070 -0.57182562## [97] -1.06636882 -0.66340769 -1.98218956 -0.13223166 -1.63417768 -0.53519279## [103] -1.13963448 -1.12131806 -1.10300165 -0.77330618 -0.97478674 -1.04805240## [109] -1.06636882 -0.13223166 -1.06636882 0.30736229 -0.77330618 -0.31539581## [115] -0.79162259 -0.22381374 -0.97478674 -1.21290014 -1.50596277 -0.51687637## [121] -1.41438070 -1.13963448 -0.68172411 -0.46192713 -1.59754485 -0.60845845## [127] -0.93815391 -0.44361071 -0.90152108 0.03261607 -0.99310316 -0.15054808## [133] -1.30448221 -1.17626731 -1.06636882 -0.51687637 -1.52427919 -0.68172411## [139] -1.26784938 -0.77330618 -0.68172411 -0.60845845 -2.16535371 -0.59014203## [145] -1.21290014 -0.90152108 -0.86488825 -1.34111504 -1.45101353 -1.12131806## [151] -1.45101353 -0.44361071 0.39894437 1.11328455 0.87517115 1.11328455## [157] 0.67369059 0.47221003 0.27072946 0.50884286 -0.11391525 0.52715927## [163] -0.55350920 0.93012040 0.28904588 0.82022191 0.34399512 0.98506964## [169] -0.35202864 0.96675323 0.41726078 0.87517115 1.14991738 0.21578022## [175] 0.47221003 0.43557720 -0.18718091 0.39894437 0.10588173 0.71032342## [181] 0.78358908 1.11328455 0.61874135 -0.20549732 0.21578022 2.87166037## [187] 0.94843681 0.82022191 -0.24213015 0.08756532 0.01429966 0.87517115## [193] -0.22381374 1.04001889 0.25241305 1.04001889 1.20486662 -0.05896600## [199] 0.28904588 1.20486662 0.17914739 0.23409663 0.49052644 0.83853832## [205] 0.21578022 1.13160096 0.47221003 0.19746381 -0.02233317 0.28904588## [211] -0.13223166 1.18655021 0.25241305 0.41726078 0.32567871 1.90089038## [217] 0.34399512 1.07665172 0.41726078 1.02170247 -0.07728242 1.24149945## [223] 0.69200701 0.45389361 0.78358908 0.47221003 0.45389361 0.85685474## [229] 0.65537418 1.31476511 0.23409663 0.23409663 0.94843681 1.57119492## [235] 0.63705776 1.11328455 0.17914739 1.25981586 -0.09559883 1.35139794## [241] 0.65537418 1.49792926 0.65537418 1.51624567 0.28904588 1.02170247## [247] 0.10588173 1.25981586 1.00338606 0.54547569 0.82022191 1.31476511## [253] 0.83853832 2.19395302 0.60042493 0.94843681 0.61874135 0.52715927## [259] -0.40697788 1.73604265 -0.11391525 0.76527266 1.20486662 1.07665172## [265] -0.07728242 1.38803077 0.41726078 2.04742170 0.10588173 0.89348757## [271] 0.60042493 NA 0.52715927 1.18655021 0.23409663 1.09496813## [277] 0.47221003 1.11328455 1.35139794 0.27072946 1.60782775 0.23409663## [283] 0.39894437 1.35139794 0.38062795 1.35139794 0.49052644 1.42466360## [289] 0.56379210 1.47961284 0.36231154 1.20486662 1.16823379 2.57859773## [295] 0.45389361 0.96675323 -0.27876298 0.83853832 -0.13223166 1.22318303## [301] 0.50884286 1.47961284 1.20486662 1.02170247 0.45389361 1.62614416## [307] -0.55350920 1.88257397 -0.26044656 1.29644869 1.05833530 0.65537418## [313] 0.67369059 1.47961284 0.54547569 1.75435906 0.93012040 0.41726078## [319] 1.27813228 0.28904588 1.27813228 1.25981586 1.13160096 0.93012040## [325] 1.38803077 1.07665172 0.76527266 1.36971435 0.32567871 1.24149945## [331] -0.26044656 1.51624567 0.23409663 0.98506964 1.14991738 0.30736229## [337] 1.46129643 0.52715927 0.32567871 2.17563660 -0.07728242 1.04001889## [343] 1.25981586 1.14991738 and then assigns that vector do the desired column name.

Reduce your data to key quantities with summarize()The summarize() function in R allows you to summarize your data into single values. It works similarly to mutate(), but instead of adding new columns, it reduces the data set to a single row. For example, you can use summarize() to compute the mean of the bill_length_mm and flipper_length_mm columns

penguins |> summarize( mean\_bill\_length = mean(bill\_length\_mm, na.rm = TRUE), mean\_flipper\_length = mean(flipper\_length\_mm, na.rm = TRUE) )## # A tibble: 1 × 2## mean\_bill\_length mean\_flipper\_length## <dbl> <dbl>## 1 43.9 201. Note that we use na.rm = TRUE to remove any missing values before computing the means. And if you want to repeat the calculations for different subsets of your data, such as different species of penguins, you can use the .by argument:

penguins |> summarize( mean\_bill\_length = mean(bill\_length\_mm, na.rm = TRUE), mean\_flipper\_length = mean(flipper\_length\_mm, na.rm = TRUE), .by = species )## # A tibble: 3 × 3## species mean\_bill\_length mean\_flipper\_length## <fct> <dbl> <dbl>## 1 Adelie 38.8 190.## 2 Gentoo 47.5 217.## 3 Chinstrap 48.8 196. This will repeat the calculations separately for each species of penguin.

Sort your data with arrange()The arrange() function in R allows you to sort the rows of your dataset based on one or more columns. By default, it sorts in ascending order.

penguins |> summarize( mean\_bill\_length = mean(bill\_length\_mm, na.rm = TRUE), mean\_flipper\_length = mean(flipper\_length\_mm, na.rm = TRUE), .by = species ) |> arrange(mean\_flipper\_length)## # A tibble: 3 × 3## species mean\_bill\_length mean\_flipper\_length## <fct> <dbl> <dbl>## 1 Adelie 38.8 190.## 2 Chinstrap 48.8 196.## 3 Gentoo 47.5 217. But you can use the desc() function to sort in descending order:

penguins |> summarize( mean\_bill\_length = mean(bill\_length\_mm, na.rm = TRUE), mean\_flipper\_length = mean(flipper\_length\_mm, na.rm = TRUE), .by = species ) |> arrange(desc(mean\_flipper\_length))## # A tibble: 3 × 3## species mean\_bill\_length mean\_flipper\_length## <fct> <dbl> <dbl>## 1 Gentoo 47.5 217.## 2 Chinstrap 48.8 196.## 3 Adelie 38.8 190. ConclusionIn this blog post/video, we covered the 6 most fundamental functions for data cleaning with R. These functions provide a solid foundation for transforming and manipulating data in R.

If you enjoyed this content and want to learn more advanced data cleaning techniques, be sure to check out my Data Cleaning with R Master Class. In this master class, we

dive deeper into these functions,
learn to clean specific data formats (such as Excel files),
handling text data, and
deal with date formats.

Be sure to check out the course. And if you found this helpful, here are some other ways I can help you:

3 Minute Wednesdays: A weekly newsletter with bite-sized tips and tricks for R users
Insightful Data Visualizations for “Uncreative” R Users: A course that teaches you how to leverage {ggplot2} to make charts that communicate effectively without being a design expert.

To leave a comment for the author, please follow the link and comment on their blog: Albert Rapp.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The 6 Most Fundamental Functions for Data Cleaning with R

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Descriptive Statistics in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Descriptive Statistics in R: A Step-by-Step Guide

Descriptive statistics are a crucial part of data analysis, as they provide a snapshot of the central tendency and variability of a dataset.

In R, there are two primary functions that can be used to calculate descriptive statistics: summary() and sapply().

In this article, we will explore how to use these functions to gain a deeper understanding of our data.

Replace first match in R » Data Science Tutorials

Method 1: Using the summary() Function

The summary() function is a simple and efficient way to calculate various descriptive statistics for each variable in a data frame. To use this function, simply call it on your data frame, like so:

summary(my\_data) The summary() function will return a variety of values for each variable, including the minimum, first quartile, median, mean, third quartile, and maximum.

For example, let’s say we have the following data frame:

df <- data.frame(x=c(1, 4, 4, 5, 6, 7, 10, 12), y=c(2, 2, 3, 3, 4, 5, 11, 11), z=c(8, 9, 9, 9, 10, 13, 15, 17)) We can use the summary() function to calculate descriptive statistics for each variable:

summary(df) This will output:

x y z Min. :1.000 Min. :2.000 Min. :8.00 1st Qu.:4.000 1st Qu.:2.750 1st Qu.:9.00 Median :5.500 Median :3.500 Median :9.50 Mean :6.125 Mean :5.125 Mean :11.25 3rd Qu.:7.750 3rd Qu.:6.500 3rd Qu.:13.50 Max. :12.000 Max. :11.000 Max. :17.00 Method 2: Using the sapply() Function

The sapply() function is a more versatile option for calculating descriptive statistics. It allows us to specify a custom function to apply to each variable in the data frame.

For example, we can use the sapply() function to calculate the standard deviation of each variable:

sapply(df, sd, na.rm=TRUE) This will output:

x y z 3.522884 3.758324 3.327376 We can also use the sapply() function to calculate more complex descriptive statistics by defining a custom function within it.

For example, let’s say we want to calculate the range of each variable:

sapply(df, function(df) max(df)-min(df), na.rm=TRUE) This will output:

x y z 11 9 9 Conclusion

In this article, we have explored two methods for calculating descriptive statistics in R: the summary() function and the sapply() function.

The summary() function provides a quick and easy way to calculate common descriptive statistics for each variable in a data frame.

The sapply() function offers more flexibility and allows us to define custom functions to calculate more complex descriptive statistics.

By using these functions effectively, we can gain a deeper understanding of our data and make more informed decisions about our analysis and visualization strategies.

Major Components of Time Series Analysis
Sample Size Calculation and Power Clinical Trials
Biases in Statistics Common Pitfalls
Area Under Curve in R (AUC)
Filtering Data in R 10 Tips -tidyverse package
How to Perform Tukey HSD Test in R
Statistical Hypothesis Testing-A Step by Step Guide
How to Create Frequency Tables in R
PCA for Categorical Variables in R
sweep function in R

The post Descriptive Statistics in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Descriptive Statistics in R

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Multiple Plots to PDF in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Multiple Plots to PDF in R: A Step-by-Step Guide. When working with data analysis and visualization, it’s often necessary to save multiple plots to a single PDF file.

This can be a convenient way to organize and share results, especially when working with multiple datasets or multiple models.

In R, saving multiple plots to a PDF is a straightforward process that can be achieved using the pdf() function and the par() function.

The basic syntax for saving multiple plots to a PDF in R is as follows:

destination = 'C:\\Users\\Stat\\Documents\\plots.pdf'pdf(file=destination)par(mfrow = c(2,2))for (i in 1:4) { x=rnorm(i) y=rnorm(i) plot(x, y) }dev.off() In this example, we first specify the path to the destination file using the destination variable. We then open the PDF file using the pdf() function, specifying the file path and name.

Next, we use the par() function to specify the layout of the plots on the page. In this case, we’re using a 2×2 grid, which means that four plots will be arranged in two rows and two columns.

The for loop is used to generate and plot four random datasets using the rnorm() function.

Each dataset is plotted using the plot() function, and the resulting plots are saved to the PDF file. Finally, we use the dev.off() function to turn off PDF plotting.

When you run this code, you’ll find that a single PDF file is created with four plots arranged in a 2×2 grid.

This is a great way to save multiple plots to a single file and can be useful for sharing results or including multiple plots in a report.

However, what if you want to save multiple plots to different pages in the same PDF file? In this case, you can simply remove the par() function from the code. This will cause each plot to be saved to its own page in the PDF file.

Here’s an updated version of the code:

destination = 'C:\\Users\\Stats\\Documents\\plots.pdf'pdf(file=destination)for (i in 1:4) { x=rnorm(i) y=rnorm(i) plot(x, y) }dev.off() In this example, each plot is saved to its own page in the PDF file. This can be useful when working with large datasets or complex models that require multiple plots to effectively communicate results.

How to Find Unmatched Records in R » Data Science Tutorials

In conclusion, saving multiple plots to a PDF in R is a straightforward process that can be achieved using the pdf() function and the par() function.

By specifying the layout of the plots on the page using par(), you can arrange multiple plots in a single PDF file. Alternatively, you can remove par() and save each plot to its own page in the PDF file.

With these techniques, you can effectively organize and share your results with ease.

How to Set Axis Limits in ggplot2?
Cluster Sampling in R With Examples
How to find the best regression models in R-Mallows’ Cp
Create groups based on the lowest and highest values in R?
What Data Science Is and What You Can Do With It
How to find z score in R-Easy Calculation-Quick Guide
R transform function with Example
Kruskal Wallis test in R-One-way ANOVA Alternative
How to Overlay Plots in R-Quick Guide with Example

The post Multiple Plots to PDF in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Multiple Plots to PDF in R

View Details

[This article was first published on R – NIMBLE, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We’ve released the newest version of NIMBLE on CRAN and on our website. NIMBLE is a system for building and sharing analysis methods for statistical models, especially for hierarchical models and computationally-intensive methods (such as MCMC, Laplace approximation, and SMC).

This release provides provides extensive new functionality, including:

A Pólya-gamma sampler, sampler\_polyagamma, for conjugate sampling of linear predictor parameters in logistic regression model specifications, including handling zero inflation and stochastic design matrices. This sampler must be added to an MCMC configuration manually.
A new sampler, sampler\_noncentered, which samples the mean or standard deviation of a set of random effect values in a transformed space such that the random effects are deterministically shifted or scaled given new values of their hyperparameters. For random effects written in a centered parameterization, sampling is performed as if they had been written in a noncentered parameterization, thereby enabling a variant on the Yu and Meng (2011) interweaving sampling strategy of sampling in both parameterizations.This sampler must be added to an MCMC configuration manually.
Adaptive Gauss-Hermite quadrature (AGHQ) for integrating over continuous latent effects, as an extension of NIMBLE’s Laplace approximation functionality. We also add user-friendly R functions, runLaplace and runAGHQ, for using Laplace and AGHQ approximation for maximum likelihood estimation.
A more flexible optimization system via nimOptim, with support for nlminb built in as well as the capability for users to provide potentially arbitrary optimization functions in R.
Allowing the use of nimbleFunctions with setup code in models, either for user-defined functions via <- or for user-defined distributions via ~. This supports holding large objects outside of model nodes for use in models.
A completely revamped MCEM algorithm, using automatic derivatives in the maximization when possible, fixing a bug so that any parts of the model not connected to the latent states are included in MLE calculations, giving greater control and adding minor extensions to the ascent-based MCEM approach, and converting buildMCEM to be a nimbleFunction rather than an R function.

In addition to the new functionality above, other enhancements and bug fixes include:

Improving the speed of MCMC and MCMC building in certain cases.
Adding an argument to buildMCMC controlling whether to initialize values in the model.
Providing the ability to control the number of digits printed in C++ output.
Allowing use of a categorical MCMC sampler with user-specified dcat-like distributions.
Warning of use of backward indexing in models.
Improve documentation of the LKJ distribution and of advanced aspects of writing code for derivative tracking using the AD system.
Fixing an insufficient check for conjugacy in stick-breaking specifications of Bayesian nonparametric distributions.
Fixing compilation failures occurring on Red Hat Linux.
Reenabling functionality for user-provided Eigen library and related updates to the autoconf configuration used in package building.
Enhancing functionality to support model macros, which will be fully released and documented in the future.
Removing deprecated is.na.vec and is.nan.vec functions.
Improving some warnings and error messages.

Please see the release notes on our website for more details.

To leave a comment for the author, please follow the link and comment on their blog: R – NIMBLE.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Version 1.2.0 of NIMBLE released

View Details

[This article was first published on business-science.io, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Hey guys, welcome back to my R-tips newsletter. Lately it’s felt like Python, which has dozens of tools for Large Language Models (LLMs), is moving faster into LLMs than the R community. Thankfully, I stumbled upon a new R package that integrates Large Language Models (LLMs) from OpenAI, Mistral AI, and more all inside of R. It’s called tidychatmodels, and I’m sharing how to get started in under 5 minutes. Let’s go!

Table of ContentsHere’s what you’re learning today:

What is tidychatmodels? You’ll discover what tidychatmodels is and how it brings LLMs to RStudio
Benefits of using Tidy Chat Models How tidychatmodels uses the familiar tidyverse-style syntax for working with LLMs
How to use LLMs inside of R I have prepared a full tutorial for making an R Code Chatbot.
Make a quick Shiny App with tidychatmodels I’ll explain how to make your first shiny app to upload and explore a dataset stored as a CSV file.

Get the Code (In the R-Tip 081 Folder)

SPECIAL ANNOUNCEMENT: ChatGPT for Data Scientists Workshop on July 17thInside the workshop I’ll share how I built a Machine Learning Powered Production Shiny App with ChatGPT (extends this data analysis to an insane production app):

What: ChatGPT for Data Scientists

When: Wednesday July 17th, 2pm EST

How It Will Help You: Whether you are new to data science or are an expert, ChatGPT is changing the game. There’s a ton of hype. But how can ChatGPT actually help you become a better data scientist and help you stand out in your career? I’ll show you inside my free chatgpt for data scientists workshop.

Price: Does Free sound good?

How To Join: Register Here

R-Tips WeeklyThis article is part of R-Tips Weekly, a weekly video tutorial that shows you step-by-step how to do common R coding tasks. Pretty cool, right?

Here are the links to get set up.

Sign up for our R-Tips Newsletter and get the code.
YouTube Tutorial

This Tutorial is Available in Video (6-minutes)I have a 6-minute video that walks you through setting up tidychatmodels in R and running your first LLM.

What is tidychatmodels?tidychatmodels is a tidyverse-style interface to LLMs (Large Language Models) developed by Albert Rapp. According to it’s documentation:

[tidychatmodels] provides a simple interface to chat with your favorite AI chatbot from R. It is inspired by the modular nature of {tidymodels} where you can easily swap out any ML model for another one but keep the other parts of the workflow the same.

Supported LLM Model VendorsCurrently model vendors are supported:

OpenAI (maker of the super-popular ChatGPT)
Mistral AI Open source AI

Benefits of using TidychatmodelsOne of the main benefits is the unified interface. This makes it easy to swap in and out different LLMs by simply changing the API key and LLM provider.

It’s literally that easy to swap out an LLM provider for another LLM model.

Tutorial: How to use LLMs inside of R with tidychatmodelsIt takes about 1 minute to get tidychatmodels set up so you can start using LLMs inside of R. All the tutorial code shown is available in the R-Tips Newsletter folder for R-Tip 081.

Step 1 – Set Up Tidy Chat Models:The first step is to set up tidychatmodels. You’ll need to install the software and set up your LLM provider’s API key. Follow these 4 steps:

Install: tidychatmodels is not on CRAN as of this article. But you can install from GitHub.
Load: Load tidyverse and tidychatmodels
API Key: Set up your OpenAI API Key in your R environment.
Get Your API Key: Use Sys.getenv().

Step 2 – Create a Chat ModelOnce you have Tidy Chat Models set up, you’re ready to make your first chat model. Let’s create a chat model that is specifically designed to write R code. This is how:

Use OpenAI model vendor
Select “gpt-4o” for the LLM specification
Add parameters like temperature and max tokens
Set the role for the chatbot via prompt engineering

Step 3 – Run the LLMThe next step is to get a user request, and to process the request using the OpenAI gpt-4o model.

The user request: Our user is requesting to create a shiny app that uploads a dataset provided as a CSV and makes a visualization with the interactive plotly library.
The LLM is then run using the perform_chat() function

Step 4 – Extract the resultsThe last step is to extract the results.

To extract the results, use extract_chat(silent = TRUE), which returns a data frame (tibble)
To print the code in human readable text, use pluck() and cat().
This returns the Shiny App code from our AI R Coder Copilot. (We’ll see if it runs next)

Does the Shiny App work from our R Coder AI CopilotThe shiny app works! I uploaded the CSV file stored in the 081 Folder, and I can quickly view a scatter plot of numeric columns.

Reminder: The code is available free inside R-tipsAll of the code you saw today is available in R-Tips Newsletter folder for R-Tip 081

Conclusions:This is exciting! tidychatmodels is making it easier and more productive for me to use LLM’s in my R workflow. I look forward to seeing how tidychatmodels progresses as LLM’s become a bigger part of my data science process.

There you have it. How to get LLMs in R. But, the next problem is that you’ll need to solve business problems with data science and R.

If you would like to grow your Business Data Science skills, then please read on…

Need to advance your business data science skills?I’ve helped 6,107+ students learn data science for business from an elite business consultant’s perspective.

I’ve worked with Fortune 500 companies like S&P Global, Apple, MRM McCann, and more.

And I built a training program that gets my students life-changing data science careers (don’t believe me? see my testimonials here):

6-Figure Data Science Job at CVS Health ($125K)Senior VP Of Analytics At JP Morgan ($200K)50%+ Raises & Promotions ($150K)Lead Data Scientist at Northwestern Mutual ($175K)2X-ed Salary (From $60K to $120K)2 Competing ML Job Offers ($150K)Promotion to Lead Data Scientist ($175K)Data Scientist Job at Verizon ($125K+)Data Scientist Job at CitiBank ($100K + Bonus)Whenever you are ready, here’s the system they are taking:Here’s the system that has gotten aspiring data scientists, career transitioners, and life long learners data science jobs and promotions…

Join My 5-Course R-Track Program Now!
(And Become The Data Scientist You Were Meant To Be…)

P.S. – Samantha landed her NEW Data Science R Developer job at CVS Health (Fortune 500). This could be you.

To leave a comment for the author, please follow the link and comment on their blog: business-science.io.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Large Language Models (LLMs) in R with tidychatmodels

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIf you often work with Excel files and are looking to streamline your data import and export processes, R and Python offer some powerful packages to help you. Here, I’ll introduce you to some essential tools in both R and Python that will make handling Excel files a breeze.

R Packages for Excel FilesreadxlThe readxl package is one of the most straightforward options for reading Excel files into R. It supports both .xls and .xlsx formats and is particularly appreciated for its simplicity and speed.

Here’s a quick example:

```

Load the readxl packagelibrary(readxl)# Read the Excel filedata <- read_excel("path_to_your_file.xlsx")# View the first few rows of the datahead(data)

``openxlsxIf you need to do more than just read Excel files,openxlsx` is a fantastic choice. This package allows you to read, write, and format Excel files, providing greater flexibility for data manipulation and presentation.

Example:

```

Load the openxlsx packagelibrary(openxlsx)# Read the Excel filedata <- read.xlsx("path_to_your_file.xlsx")# Write data to a new Excel filewrite.xlsx(data, "path_to_new_file.xlsx")

``xlsxThexlsxpackage is another versatile tool for handling Excel files in R. It supports reading, writing, and formatting Excel files, and works well for both.xlsand.xlsx` formats.

Example:

```

Load the xlsx packagelibrary(xlsx)# Read the Excel filedata <- read.xlsx("path_to_your_file.xlsx", sheetIndex = 1)# Write data to a new Excel filewrite.xlsx(data, "path_to_new_file.xlsx")

`` Python Packages for Excel FilespandasThepandaslibrary is a cornerstone of data analysis in Python, and it includes theread_excel()` function for reading Excel files. This function is highly versatile and integrates seamlessly with other pandas functionalities.

Example:

```

Import the pandas packageimport pandas as pd# Read the Excel filedata = pd.read_excel("path_to_your_file.xlsx", sheet_name="Sheet1")# Display the first few rows of the dataprint(data.head())

``openpyxlFor more advanced Excel operations in Python,openpyxl` is an excellent choice. It allows you to read and write Excel 2010 xlsx/xlsm/xltx/xltm files and offers extensive formatting capabilities.

Example:

```

Import the openpyxl packagefrom openpyxl import load_workbookimport pandas as pd# Load the workbookwb = load_workbook("path_to_your_file.xlsx")# Select a sheet by namesheet = wb['Sheet1']# Print the value of cell A1print(sheet['A1'].value)

``` Learn MoreFor a deeper dive into working with Excel files using R and Python, check out my book Extending Excel with Python and R. It’s packed with practical examples and tips to enhance your data processing workflows.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Working with Excel Files in R and Python

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Posit Connect is often the de facto platform to deploy R/Shiny applications, Python applications, and Quarto-Shiny documents. And when something goes wrong, we immediately go and visit the logs. But often the logs are walls of text that we have to scour through and when you have been coding and debugging for hours, it is often that you miss the crucial line that could have pointed you right at the error.

We faced a similar problem earlier this year in our collaboration at Elkem, one of the world’s leading silicon manufacturers, where we have multiple apps per project deployed for multiple locations with varying levels of access. In about a week, thanks to an already documented idea, we were able to set up a rudimentary version of LogAnalyzer.

As we became heavy users of the app, we decided its value was too much to keep it to ourselves. Thanks to the openness of the team at Elkem, we are now able to present an open-source version of the same app.

To read more about our overall collaboration with them, you can visit the case study.

While there are other very capable log-monitoring tools like Grafana, setting them up is somewhat complicated and requires you setting up a whole infrastructure, perhaps, on a cloud provider. Often, there is a huge amount of collaboration and process management involved and not everyone has that kind of time or resources.

‍LogAnalyzer’s value is in how easy it is to set up and use, and how it leverages the existing Posit Connect infrastructure. The simplicity and ease of use and setup is often all you need. That’s where LogAnalyzer comes in. It’s ready to use out of the box, is set up with a simple deployment that can be made by anyone who has access to the Posit Connect server.

How LogAnalyzer WorksThe LogAnalyzer app has three panes, as you can see in the GIF above. The first pane is simply the list of Deployments available on the Posit Connect server it is deployed on. The second pane lets you pickup a job (deployment or run) based on the Job Id.

There is also the timestamp placed conveniently for you to check the deployment you want to explore. And then, you get semantically coloured logs for the Deployment and the Job in question.

Reds signify all stderr statements (and some nifty keywords such as ‘halt’ or ‘terminate’).

Yellows are the stdout statements.

Greens are, well, just simple prints and other harmless lines. A simple glance can now tell you what went wrong.

‍What’s more? If you prefer to look at the text version, there is a download button available for you to download it. And since the app is powered by reactable under the hood, every part of it is searchable, making it even easier for you to find that pesky bug.

How to Use LogAnalyzerHere comes the beautiful part. You don’t have to do anything. Deploying your own copy of LogAnalyzer is as simple as 1, 2, 3.

Under the hood, LogAnalyzer uses Appsilon’s {rhino}, which enables an industry-best practices approach to R/Shiny app development so you can be rest assured it’s built with an opinionated approach in mind. Also, the views leverage the popular {reactable} package so everything is searchable. The three panes are, in essence, three heavily customised reactable tables. And the app uses Posit Connect’s official API so all your data passes through secure channels, seamlessly.

Explore the capabilities of the Rhino package and start building your first app today by following our step-by-step tutorial.

Clone the AppYou can check out the GitHub repository. If you prefer to use the terminal, it’s as simple as:

git clone git@github.com:Appsilon/LogAnalyzer.git 2. Deploy the AppBefore you do this step, we recommend restore the {renv} environment using:

renv::restore() Alternatively, if you want to test the app locally first, you may have to add some necessary environment variables to your .Renviron file on your own (you can use the usethis::edit_r_environ() from the {usethis} package to do this as well).

Here, you can set the CONNECT_SERVER and CONNECT_API_KEY environment variables and don’t forget to restart your session.

The CONNECT_SERVER is the address to your Posit Connect instance. If you access your apps using connect.yourorganisation.com/app1, the CONNECT_SERVER value is connect.yourorganisation.com.
The CONNECT_API_KEY can be generated from the Posit Connect dashboard and grants the user programmatic access to all deployments. The documentation for the Posit Connect API can be found here.

Note that these environment variables are available to default to any deployed app on Posit Connect, and you only need to set these up locally. To learn more about them and to generate an API key, you can check the official Posit Connect documentation.

Tip: You can also change the logo in app/static and modify the mod_header.R to make it truly your own.

Now, you can simply deploy the application to Posit Connect either with the {rsconnect} package or with the RStudio GUI

Set CONNECT_API_KEY (or not)There are two ways to go about this: either let Posit Connect set the CONNECT_API_KEY for you automatically or set the environment variable in the Vars section of the Posit Connect GUI for the application yourself.

What’s the difference between these two approaches? It boils down to who is using LogAnalyzer and the intended usage.

If you are deploying LogAnalyzer for personal use and want to analyse logs only for your own apps—that you deploy yourself—then use the first approach and let it all work on its own. You don’t need to set up an environment variable at all.
If you want LogAnalyzer to be accessed team-wide with access to all logs for all deployments to all users, you could manually set up a key for a user with Admin rights or who belongs to the group with access to all apps. This way, the LogAnalyzer instance would be able to fetch logs regardless of who deployed the app.

‍

Either way, the CONNECT_SERVER will be set for you and defaults to the server the application is deployed on.

And voila ! That really is it. You have your own instance of LogAnalyzer deployed and ready to use. No more sifting through long chunks of text!

An Ending NoteWe invite you to find issues with the application as well as think of potential improvements. If and when you see a way you can improve this application, feel free to raise an issue or a pull request. We look forward to your contributions!

Frequently Asked QuestionsQ: What types of applications does LogAnalyzer support?

A: LogAnalyzer currently supports log monitoring for R/Shiny applications, Python applications, and Quarto-Shiny documents deployed on Posit Connect.

Q: Can I use LogAnalyzer for applications not deployed on Posit Connect?

A: No, LogAnalyzer is designed to work with the Posit Connect infrastructure and leverages the Posit Connect API to fetch log data.

Q: How does LogAnalyzer handle large log files?

A: LogAnalyzer uses the {reactable} package under the hood, which allows for efficient rendering and searching of large datasets, including log files.

Q: Can I customize the color scheme for log entries?

A: Yes, the color scheme for log entries (red, yellow, green) can be customized by modifying the appropriate CSS styles in the app’s code.

Q: Is LogAnalyzer open-source?

A: Yes, LogAnalyzer is an open-source project, and you can find the repository on GitHub.

Got questions or feedback? Join our community and start a conversation!

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: LogAnalyzer: An Easy-to-Use Log Monitoring Tool for R/Shiny Applications

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Our courses for the second half of 2024 have now been released. We have everything from the very basics of R and Python for data science, to advanced statistical modelling and machine learning. Interested in dashboards and reporting? We have courses on reporting with Quarto, as well as both introductory and advanced Shiny. Already know the basics but want to hone your skills? We have plenty of intermediate courses for you, as well as a course to take a look at some best practices in R and Python.

Whether you want to start from scratch, or improve your skills, Jumping Rivers has a training course for you.

RIntroduction to RCourse Level: Foundation

Upcoming course dates: 3rd July, 7th October

R is a versatile language for statistical computing and graphics. In this course you will learn the advantages of using R and how to get started. You will gain familiarity with the RStudio interface and learn the R basics. Also included is an introduction to the Tidyverse and how to use various packages for data storage, visualisation and manipulation. This course provides a great foundation to begin your R journey!

Programming with RCourse Level: Intermediate

Upcoming course dates: 15th July, 21st October

The benefit of using a programming language such as R is that we can automate repetitive tasks. This course covers the fundamental techniques such as functions, for loops and conditional expressions. By the end of this course, you will understand what these techniques are and when to use them. This is a one-day intensive course on R.

Data Wrangling in the TidyverseCourse Level: Foundation

Upcoming course dates: 10th July, 16th October

If you work with data, you probably spend a lot of time cleaning it and wrangling it into the correct shape. This course will show you how you can use R to efficiently clean and wrangle your data into a format that’s ready for analysis. You will learn about the Tidyverse, what tidy data really is, and how to practically achieve it with packages such as {dplyr}, {tidyr}, {lubridate} and {forcats}.

Data Visualisation with ggplot2Course Level: Intermediate

Upcoming course dates: 22nd July, 4th November

Want to learn how to effectively visualise your data in R using the elegant {ggplot2} package? With {ggplot2} it’s easy to customise everything from plot layouts and themes to scales, colours, and more! This course will comprehensively take you through basic plot types such as bar and line charts as well as cover more advanced topics such as interactive graphics with {plotly}.

R Best PracticesCourse Level: Intermediate

Upcoming course dates: 22nd July

So you can write code? Great. But can you write code which is easy to read, simple to maintain, and reproducible? Under the pressure of deadlines even the best of us can fall victim to bad-practices. In this course we motivate the importance of good-practices, and show how we can make best practices second nature by incorporating them into our normal workflow.

Object Oriented Programming in RCourse Level: Advanced

Upcoming course dates: 15th July

The training course will cover R object-oriented programming techniques. We’ll discuss what OOP is and the different varieties within R. Beginning with the popular S3 and S4 OOP frameworks, we’ll finish with the new {R6} package that is used extensively in Shiny applications. By the end of the course, participants will be able to use OOP within their own code.

ShinyIntroduction to ShinyCourse Level: Intermediate

Upcoming course dates: 10th July, 7th October

Do you want to provide interactive visualisation and data exploration features for users who do not have R and data science skills? Discover how easy it can be to use R and {shiny} to create your own apps and dashboards for exploring data without relying on web development or external BI tools. We will show you various examples of input widgets and outputs to display tables and visualisations.

Advanced Concepts in ShinyCourse Level: Advanced

Upcoming course dates: 23rd September, 14th October

Take your interactive {shiny} skills to the next level by creating more robust, responsive and maintainable applications. In this course, we’ll visit more advanced topics that can be used to improve the experience for both those producing the apps and those using them. Subjects will cover: additional ways to react to and validate user inputs; restructuring your app with modules; and an introduction to testing your {shiny} apps.

PythonIntroduction to PythonCourse Level: Foundation

Upcoming course dates: 9th September, 14th October

Python is a general-purpose programming language popular among data scientists and statisticians. In this one-day introductory course, participants will learn to import, summarise and visualise their data. At each step, we avoid using “magic code”, and stress the importance of understanding what Python is doing.

Programming with PythonCourse Level: Intermediate

Upcoming course dates: 16th September, 23rd October

The benefit of using a programming language such as Python is that we can automate repetitive tasks. This course covers the fundamental techniques such as functions, for loops and conditional expressions. By the end of this course, you will understand what these techniques are and how they can be applied to solve real-world data wrangling tasks.

Data Visualisation with PythonCourse Level: Intermediate

Upcoming course dates: 17th June, 23rd September, 11th November

Python has a number of packages for the effective creation of graphics to communicate your data insights. This course will examine two popular libraries for creating static 2D plots: Matplotlib and Seaborn. During the training session, we’ll cover plotting basics and customisation of figures with Matplotlib, before moving onto complex statistical visualisations with Seaborn.

Python Best PracticesCourse Level: Intermediate

Upcoming course dates: 22nd July

ReportingReporting with QuartoCourse Level: Intermediate

Upcoming course dates: 24th June, 23rd September, 18th November

Do you create interactive documents that always need to be updated when the data changes? Then this course is for you. In this course you will learn how to use Quarto to create high quality, dynamic, fully reproducible documents. Quarto is a multi-language open source publishing tool that allows for the creation of dynamic content with Python, R, Julia and Observable.

Machine LearningMachine Learning with TidymodelsCourse Level: Intermediate

Upcoming course dates: 16th September, 11th November

Machine learning is the process of applying statistical techniques to gain systematic information about a quantity of interest. We will be specifically focusing on how we can use the {tidymodels} suite of packages to implement these techniques. We cover key reasons for model fitting, such as prediction and inference, on quantitative and qualitative responses.

Advanced Machine Learning with TidymodelsCourse Level: Advanced

Upcoming course dates: 23rd September, 18th November

A course that builds on the material covered in our Machine Learning with Tidymodels course. We take a look at how we can fit linear discriminant analysis (LDA) models using {discrim}, assessing model reliability using V-fold cross validation, pre-processing, tree-based models & more. If you wish to explore the abundance of model fitting techniques {tidymodels} has to offer, then this course is certainly for you!

SQLAn Introduction to SQL with RCourse Level: Intermediate

Upcoming course dates: 2nd October

Using databases is a fundamental part of a data scientist’s role. The main focus of this training course is to introduce SQL databases, write your first SQL queries, and show how R can be used to retrieve and manipulate data stored in a relational database. The course uses both the {DBI} and {dbplyr} packages.

We use the PostgreSQL database as an example for public courses. For in-house training, we are happy to adapt the course to match your database requirements.

Introduction to SQL with PythonCourse Level: Intermediate

Upcoming course dates: 2nd October

Using databases is a fundamental part of a data scientist’s role. This training course introduces SQL databases and the SQL command syntax, and shows how Python can be used to retrieve and manipulate data held in a relational database. The course also discusses how SQLAlchemy can be used to define and interact with databases using object-oriented Python code.

We use a PostgreSQL database as an example, and communicate with this using a psycopg2 connection.

StatisticsStatistical Modelling with RCourse Level: Intermediate

Upcoming course dates: 9th September, 23rd October

From the very beginning, R was designed for statistical modelling. Out of the box, R makes standard statistical techniques easy. This course covers the fundamental modelling techniques. We begin the day by revising hypotheses tests, before moving onto ANOVA tables and regression analysis. The class ends by looking at more sophisticated methods such as clustering and principal components analysis (PCA).

Introduction to Bayesian Inference using RStanCourse Level: Intermediate

Upcoming course dates: 1st July, 14th October

Despite the promise of big data, inferences are often limited by its systematic structure. Only by carefully modelling this structure can we take full advantage of the data. Stan is a platform for facilitating this modelling, providing an expressive modelling language to implement state-of-the-art algorithms, to draw subsequent Bayesian inferences. This course will teach participants how to interface with Stan through R!

Introduction to Bayesian Inference using PyStanCourse Level: Intermediate

Upcoming course dates: 15th July, 21st October

The course will teach participants how to interface with Stan through Python!

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: June 2024 Training Update

View Details

[This article was first published on R on Publishable Stuff, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.There are tons of well-known global indicators. We’ve all heard of grossdomestic product, life expectancy, rate of literacy, etc. But, eversince I discovered pinballmap.com, possiblythe world’s most comprehensive database of public pinball locations,I’ve been thinking about a potential new global indicator: PublicPinball Machines per Capita. Thanks to Pinball Map’s well-documentedpublic API, this indicatoris now a reality!

Here’s how this wasput together (and just scroll to the bottom for a CSV file with thisindicator for all countries).

Pulling public pinball locations from Pinball MapPinball Map is, from what I can discern, the most popular app forfinding out where there are arcades and bars with pinball machines. It’sopen for anyone to register new pinball locations, but not only that, the app itself is open source, andthe data it collects is available through a public API under apermissive licence! Using this API, we will pull essential data for ourPublic Pinball Machines per Capita indicator: all registered pinballlocations and their respective machine counts.

Loading packages library(httr2) # To interact with the Pinball Map APIlibrary(jsonlite) # To parse the JSON responseslibrary(tidyverse) # To munch, crunch, and plot the datalibrary(ggrepel) # For less crowed labels on plotslibrary(WDI) # To pull in other country-level datalibrary(maps) # For plotting maps Code for pulling pinball stats from the Pinball MapAPI ```

We're going to pull a lot of data here, possibly abusing the Pinball Map API,# a bit. But I'm an active patreon sponsor, so hopefully that's OK...# Did I mention that they are on patreon? https://www.patreon.com/pinballmap# Pulls and parses JSON from the given URLget_req_json <- (url) { request(url) |> req_perform() |> resp_body_json(simplifyVector = TRUE)}# Pulling all regions defined by the Pinball Map APIregions <- get_req_json("https://pinballmap.com/api/v1/regions.json")$regions# Now looping over the region names and for each name pull down all locationsregion_locations <- lapply(regions$name, (name) { url <- paste0("https://pinballmap.com/api/v1/region/", name, "/locations.json") get_req_json(url)$locations})# Pull down all "regionless" locations. Actually, most locations are regionless.regionless_locations <- get_req_json( "https://pinballmap.com/api/v1/locations.json?regionless_only=true")$locations# Finally, combine it all...locations <- bind_rows(region_locations, regionless_locations) |> select(name, country, city, lat, lon, num_machines) |> mutate(lat = as.numeric(lat), lon = as.numeric(lon)) |> # ... and order locations from north to south arrange(desc(lat))sample_n(locations, size = 5)

```

name country city lat lon num\_machines1 Arena Lanes Bowling Center US Oak Lawn 41.7 -87.7 32 Pete's Treats US Union Springs 42.9 -76.7 13 The Summit Windsor US Loveland 40.4 -105.0 64 Skylark Lounge US Denver 39.7 -105.0 25 The Escape Gamebar US Atlanta 33.9 -84.3 5 The above shows a sample of five out of the 10,330 locations where youcan play pinball, as of June 2024. As we have the longitude and latitudewe can also figure out that the northernmost place to play pinball is inRovaniemi, Finland, and the southernmost place is in Woolston, NewZealand.

Code locations[c(1, nrow(locations)),]

name country city lat lon num\_machines1 Kauppayhtiö FI Rovaniemi 66.5 25.7 210330 Fish & Chips On Ferry NZ Woolston -43.5 172.7 1 Or, why not just plot all pinball locations on a world map?

Plot code extreme\_locations <- locations |> filter(lat %in% range(lat)) |> mutate(display\_label = paste(city, country, sep = ", "))ggplot() + geom\_polygon(data = map\_data("world"), aes(x = long, y = lat, group = group), fill = "lightblue", color = "lightblue3") + geom\_point(data = locations, aes(x = lon, y = lat), color = "magenta4", size = 1, alpha = 0.50) + geom\_point(data = extreme\_locations, aes(x = lon, y = lat), color = "red2", size = 2) + geom\_text(data = extreme\_locations, aes(x = lon, y = lat, label = display\_label), nudge\_x = -25) + theme\_void() + ggtitle("Pinball Locations Worldwide (according to pinballmap.com)") Finally, we can now sum up how many public pinball machines there are ineach country, where the USA, unsurprisingly, takes the lead.

Code pinball\_stats <- locations |> group\_by(country) |> summarise( n\_locations = n(), n\_machines = sum(num\_machines)) |> arrange(desc(n\_machines))pinball\_stats

```

A tibble: 65 × 3 country n_locations n_machines 1 US 7831 32287 2 CA 511 1765 3 AU 427 1247 4 DE 129 1099 5 FR 247 707 6 SE 79 692 7 GB 160 500 8 FI 98 496 9 NL 69 46110 JP 86 351# ℹ 55 more rows

`` Calculating Public Pinball Machines *per Capita*Knowing how many public pinball machines there are in each country isn’tenough, we also need to consider the size of the population. Thanks to theWDI` package it’s easy topull this, and any other indicators you fancy, from the World Bank OpenData and to calculate the number of PublicPinball Machines per Capita (here per million people).

Code for pulling World Development Indicators country\_stats\_by\_year = WDI( indicator = c( "NY.GDP.PCAP.CD", "SP.POP.TOTL", "SP.DYN.LE00.IN", "SP.DYN.TFRT.IN", "IT.NET.USER.ZS", "AG.LND.FRST.ZS" ), extra = TRUE, latest = 1)country\_stats <- country\_stats\_by\_year |> arrange(country, year) |> group\_by(country) |> # Keep the latest indicator for each country summarize(across(everything(), \(x) last(na.omit(x)))) |> select( country\_name = country, country\_code = iso2c, gdp\_per\_capita = NY.GDP.PCAP.CD, population = SP.POP.TOTL, life\_expectancy = SP.DYN.LE00.IN, births\_per\_woman = SP.DYN.TFRT.IN, internet\_usage\_perc = IT.NET.USER.ZS, forest\_coverage\_perc = AG.LND.FRST.ZS ) Code for calculating Public Pinball Machines perCapita pinball\_country\_stats <- country\_stats |> # Let's keep only larger countries filter(population > 500000) |> inner\_join(pinball\_stats, by = join\_by(country\_code == country)) |> mutate( n\_locations\_per\_million\_capita = round(n\_locations / population * 1000000, 3), n\_machines\_per\_million\_capita = round(n\_machines / population * 1000000, 3)) |> arrange(desc(n\_machines\_per\_million\_capita))select(pinball\_country\_stats, country\_name, population, n\_machines, n\_machines\_per\_million\_capita)

```

A tibble: 58 × 4 country_name population n_machines n_machines_per_million_capita 1 United States 333287557 32287 96.9 2 Finland 5556106 496 89.3 3 Sweden 10486941 692 66.0 4 Denmark 5903037 323 54.7 5 Norway 5457127 266 48.7 6 Australia 26005540 1247 48.0 7 Canada 38929902 1765 45.3 8 New Zealand 5124100 171 33.4 9 Switzerland 8775760 267 30.410 Netherlands 17700982 461 26.0# ℹ 48 more rows

``` Now, there’s out new global indicator! Looks like the USA is still inthe lead, but now the Nordic countries have bubbled up as some of thecountries with the highest pinball density.

Plot code pinball\_country\_stats |> head(10) |> mutate( country\_name = forcats::fct\_reorder(country\_name, n\_machines\_per\_million\_capita), n\_machines\_per\_million\_capita = round(n\_machines\_per\_million\_capita, 1) ) |> ggplot(aes(x = n\_machines\_per\_million\_capita, y = country\_name)) + geom\_col(fill = "lightgreen") + geom\_text(aes(label = n\_machines\_per\_million\_capita), hjust = 1.2) + labs( x = "Number of machines per million capita", y = "Country", title = "Top 10 countries by number of public pinball machines per million capita" ) Public Pinball Machines per Capita VS other indicatorsLet’s have a look at how Public Pinball Machines per Capita compares tosome other indicators. How about Life Expectancy?

Plot code ggplot(pinball\_country\_stats, aes(x = life\_expectancy, y = n\_machines\_per\_million\_capita)) + geom\_label\_repel(aes(label = country\_name), fill = "lightblue", max.overlaps = Inf, box.padding = -0.2) + scale\_x\_log10(labels = scales::label\_comma(), limits = c(67, NA)) + scale\_y\_log10(labels = scales::label\_comma()) + labs( x = "Life expectancy at birth (years)", y = "Number of machines per million capita", title = "Number of Public Pinball Machines per Capita vs life expectancy" ) So maybe playing pinball actually makes you live longer! What’s thatthing they say about correlation, now again… Or what about the fertilityrate (the average number of births per woman)?

Plot code ggplot(pinball\_country\_stats, aes(x = births\_per\_woman, y = n\_machines\_per\_million\_capita)) + geom\_label\_repel(aes(label = country\_name), fill = "lightcoral", max.overlaps = Inf, box.padding = -0.2) + scale\_x\_log10(labels = scales::label\_comma()) + scale\_y\_log10(labels = scales::label\_comma()) + labs( x = "Fertility rate (no. births per woman)", y = "Number of machines per million capita", title = "Number of Public Pinball Machines per Capita vs fertility rate" ) Nope, no clear relationship there. Actually, out of all the indicators Ilooked through, the one with the highest correlation to Public PinballMachines per Capita was…

Plot code ggplot(pinball\_country\_stats, aes(x = gdp\_per\_capita, y = n\_machines\_per\_million\_capita)) + geom\_label\_repel(aes(label = country\_name), fill = "lightgreen", max.overlaps = Inf, box.padding = -0.2) + geom\_smooth(method = "lm", se = FALSE, color = "#d03030aa") + scale\_x\_log10(labels = scales::label\_comma()) + scale\_y\_log10(labels = scales::label\_comma()) + labs( x = "GDP per capita (in USD)", y = "Number of machines per million capita", title = "Number of Public Pinball Machines per Capita vs GDP per Capita" ) … GDP per Capita. This shouldn’t surprise anyone who’s ever looked intobuying a pinball machine and walked away in shock having learned thata new machine would set you back $8000, at least. Still, the correlationbetween these two indicators is strikingly high:

Code cor( log(pinball\_country\_stats$n\_machines\_per\_million\_capita), log(pinball\_country\_stats$gdp\_per\_capita))

[1] 0.815 With such a strong correlation with GDP per Capita, it can beinteresting to look at the residuals of the linear regression lineabove. That is, what’s left after the influence of GDP per Capita hasbeen “accounted” for (and I can’t stress the quotes enough here, aswe’re not really accounting for anything).

Plot code lm\_model <- lm(log(n\_machines\_per\_million\_capita) ~ log(gdp\_per\_capita), data = pinball\_country\_stats)pinball\_country\_stats$residual <- residuals(lm\_model)ggplot(pinball\_country\_stats, aes(x = gdp\_per\_capita, y = residual)) + geom\_label\_repel(aes(label = country\_name), fill = "lightgreen", max.overlaps = Inf, box.padding = -0.2) + geom\_smooth(method = "lm", se = FALSE, color = "#d03030aa") + scale\_x\_log10(labels = scales::label\_comma()) + labs( x = "GDP per capita (in USD)", y = "Residual", title = "Residual after accounting for GDP per capita" ) Here Hungary and Croatia show up as being relative pinball fanatics,considering their GDP per Capita. While Singapore and Luxembourgcouldn’t care less for the silver ball. If you want to take a lookyourself, here’s a CSV file with the full Public Pinball Machines perCapita dataset:

Code pinball\_country\_stats |> select(country\_name, country\_code, population, n\_locations, n\_machines, n\_machines\_per\_million\_capita, gdp\_per\_capita) |> write\_csv("public\_pinball\_machines\_per\_capita\_2024.csv") public_pinball_machines_per_capita_2024.csv

Caveats: This indicator is mostly a joke, 100% depends on thecompleteness of Pinball Map, and countries without a single registeredpinball machine are excluded.

To leave a comment for the author, please follow the link and comment on their blog: R on Publishable Stuff.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Public Pinball Machines per Capita: A new global indicator

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Joins Are No Mystery Anymore: Hands-On Tutorial — Part 2Welcome back to the second part of our series, “Joins Are No Mystery Anymore: Hands-On Tutorial.” In the first part, we explored the foundational types of joins, including Inner Join, Left Join, Right Join, Full Join, and Semi Join. Through practical, real-life scenarios and step-by-step code examples, we learned how to effectively combine datasets and uncover valuable insights.

In this second part, we’ll delve into more advanced join techniques. We’ll start with Anti Joins, which help identify unmatched rows between datasets. Following that, we’ll explore Cross Joins, Natural Joins, Self Joins, and Equi Joins. Each join type will be demonstrated with real-life scenarios to enhance your understanding and practical application. Get ready to take your data analysis skills to the next level!

Anti JoinAn Anti Join returns all rows from the left table where there are no matching values in the right table. It is useful for identifying rows in the left table that do not have corresponding rows in the right table.

Explanation of the ScenarioIn this scenario, we have subscription information and payment records. We want to find subscriptions that have not been paid for. This helps in identifying outstanding payments and managing accounts receivable.

Data file:
https://github.com/kgryczan/medium_publishing/blob/main/anti_join_data.RData

Description of the DatasetsWe will use two datasets:

subscriptions: Contains information about subscriptions.
Columns: subscription_id, customer_id, start_date
payments: Contains information about payments made for subscriptions.
Columns: payment_id, subscription_id, amount, payment_date

Step-by-Step Code ExamplesLoading the datasets:

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("anti_join_data.RData")# Display the datasetsprint(subscriptions)# A tibble: 30 × 3 subscription_id customer_id start_date 1 1 108 2024-01-01 2 2 107 2024-01-02 3 3 110 2024-01-03 4 4 113 2024-01-04 5 5 110 2024-01-05 6 6 111 2024-01-06 7 7 108 2024-01-07 8 8 107 2024-01-08 9 9 107 2024-01-0910 10 112 2024-01-10# ℹ 20 more rowsprint(payments) payment_id subscription_id amount payment_date 1 201 4 154. 2024-01-05 2 202 29 134. 2024-01-06 3 203 25 158. 2024-01-07 4 204 20 169. 2024-01-08 5 205 18 170. 2024-01-09 6 206 27 91.4 2024-01-10 7 207 23 52.4 2024-01-11 8 208 21 151. 2024-01-12 9 209 12 76.4 2024-01-13 10 210 25 133. 2024-01-14 # ℹ 20 more rows

``` Performing the Anti Join

```

Perform the anti joinunpaid_subscriptions <- anti_join(subscriptions, payments, by = "subscription_id")# Display the resultprint(unpaid_subscriptions)# A tibble: 13 × 3 subscription_id customer_id start_date 1 1 108 2024-01-01 2 2 107 2024-01-02 3 5 110 2024-01-05 4 6 111 2024-01-06 5 8 107 2024-01-08 6 10 112 2024-01-10 7 15 113 2024-01-15 8 16 115 2024-01-16 9 19 115 2024-01-1910 22 111 2024-01-2211 24 108 2024-01-2412 28 115 2024-01-2813 30 102 2024-01-30

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the anti_join function from the dplyr package to filter the subscriptions dataset to include only those subscriptions that do not have matching entries in the payments dataset, based on the subscription_id column.
Finally, we display the result to see which subscriptions have not been paid for.

Interpretation of ResultsThe resulting dataset unpaid_subscriptions contains only the rows from the subscriptions dataset where there is no matching row in the payments dataset. This means that only unpaid subscriptions are included.

Homework for ReadersIn the same anti_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

courses: Contains information about courses.
Columns: course_id, course_name, instructor
enrollments: Contains information about student enrollments.
Columns: enrollment_id, course_id, student_id, enrollment_date

Your task is to perform an anti join on these datasets to identify courses that have no enrollments. Use the course_id column for joining.

Cross JoinA Cross Join returns the Cartesian product of two tables, combining all rows from the left table with all rows from the right table. This join type is useful when you want to create all possible combinations of the rows in two tables.

Explanation of the ScenarioIn this scenario, we have menu items and days of the week. We want to create a schedule of menu items for each day of the week. This helps in planning and organizing the weekly menu offerings.

Data file:
https://github.com/kgryczan/medium_publishing/blob/main/cross_join_data.RData

Description of the DatasetsWe will use two datasets:

menu_items: Contains information about the menu items.
Columns: item_id, item_name, category
days_of_week: Contains information about the days of the week.
Columns: day_id, day_name

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("cross_join_data.RData")# Display the datasetsprint(menu_items)# A tibble: 10 × 3 item_id item_name category 1 1 Pancakes Breakfast 2 2 Sandwich Lunch 3 3 Salad Lunch 4 4 Burger Lunch 5 5 Soup Dinner 6 6 Pizza Dinner 7 7 Spaghetti Dinner 8 8 Tacos Dinner 9 9 Sushi Dinner 10 10 Steak Dinner print(days_of_week)# A tibble: 7 × 2 day_id day_name 1 1 Monday 2 2 Tuesday 3 3 Wednesday4 4 Thursday 5 5 Friday 6 6 Saturday 7 7 Sunday

``` Performing the Cross Join

```

Perform the cross joinmenu_schedule <- tidyr::crossing(menu_items, days_of_week)# Display the resultprint(menu_schedule)# A tibble: 70 × 5 item_id item_name category day_id day_name 1 1 Pancakes Breakfast 1 Monday 2 1 Pancakes Breakfast 2 Tuesday 3 1 Pancakes Breakfast 3 Wednesday 4 1 Pancakes Breakfast 4 Thursday 5 1 Pancakes Breakfast 5 Friday 6 1 Pancakes Breakfast 6 Saturday 7 1 Pancakes Breakfast 7 Sunday 8 2 Sandwich Lunch 1 Monday 9 2 Sandwich Lunch 2 Tuesday 10 2 Sandwich Lunch 3 Wednesday# ℹ 60 more rows

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the crossing function from the tidyr package to perform the cross join between the menu_items and days_of_week datasets. This function creates all possible combinations of the rows in the two datasets.
Finally, we display the result to see the complete schedule of menu items for each day of the week.

Interpretation of ResultsThe resulting dataset menu_schedule contains all possible combinations of the rows from the menu_items and days_of_week datasets. Each row represents a menu item scheduled for a particular day of the week.

Homework for ReadersIn the same cross_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

shirts: Contains information about shirts.
Columns: shirt_id, color, size
pants: Contains information about pants.
Columns: pants_id, color, size

Your task is to perform a cross join on these datasets to generate all possible outfits by combining shirts and pants. Use the shirt_id and pants_id columns for joining.

Natural JoinA Natural Join joins two tables based on columns with the same name and type in both tables. It automatically matches rows with equal values in the common columns, removing the need to specify the joining column.

Explanation of the ScenarioIn this scenario, we have authors and books. We want to find authors and their corresponding books based on a common column. This helps in linking authors with the books they have written.

Data file:
https://github.com/kgryczan/medium_publishing/blob/main/natural_join_data.RData

Description of the DatasetsWe will use two datasets:

authors: Contains information about authors.
Columns: author_id, name, nationality
books: Contains information about books.
Columns: book_id, author_id, title, genre

Step-by-Step Code ExamplesLoading the datasets:

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("natural_join_data.RData")# Display the datasetsprint(authors)# A tibble: 20 × 3 author_id name nationality 1 1 Author A Australia 2 2 Author B Australia 3 3 Author C Canada 4 4 Author D USA 5 5 Author E UK 6 6 Author F USA 7 7 Author G Australia 8 8 Author H Australia 9 9 Author I UK 10 10 Author J Australia 11 11 Author K Australia 12 12 Author L USA 13 13 Author M USA 14 14 Author N Canada 15 15 Author O Canada 16 16 Author P Canada 17 17 Author Q USA 18 18 Author R Australia 19 19 Author S USA 20 20 Author T USA print(books)# A tibble: 20 × 4 book_id author_id title genre 1 101 1 Book A Fantasy 2 102 2 Book B Fiction 3 103 3 Book C Sci-Fi 4 104 4 Book D Fiction 5 105 5 Book E Sci-Fi 6 106 6 Book F Fantasy 7 107 7 Book G Fantasy 8 108 8 Book H Mystery 9 109 9 Book I Mystery10 110 10 Book J Mystery11 111 11 Book K Fantasy12 112 12 Book L Fiction13 113 13 Book M Fiction14 114 14 Book N Mystery15 115 15 Book O Sci-Fi 16 116 16 Book P Sci-Fi 17 117 17 Book Q Fantasy18 118 18 Book R Sci-Fi 19 119 19 Book S Mystery20 120 20 Book T Mystery

``` Performing the Natural Join:

```

Perform the natural joinauthors_books <- authors %>% inner_join(books, by = "author_id")# Display the resultprint(authors_books)# A tibble: 20 × 6 author_id name nationality book_id title genre 1 1 Author A Australia 101 Book A Fantasy 2 2 Author B Australia 102 Book B Fiction 3 3 Author C Canada 103 Book C Sci-Fi 4 4 Author D USA 104 Book D Fiction 5 5 Author E UK 105 Book E Sci-Fi 6 6 Author F USA 106 Book F Fantasy 7 7 Author G Australia 107 Book G Fantasy 8 8 Author H Australia 108 Book H Mystery 9 9 Author I UK 109 Book I Mystery10 10 Author J Australia 110 Book J Mystery11 11 Author K Australia 111 Book K Fantasy12 12 Author L USA 112 Book L Fiction13 13 Author M USA 113 Book M Fiction14 14 Author N Canada 114 Book N Mystery15 15 Author O Canada 115 Book O Sci-Fi 16 16 Author P Canada 116 Book P Sci-Fi 17 17 Author Q USA 117 Book Q Fantasy18 18 Author R Australia 118 Book R Sci-Fi 19 19 Author S USA 119 Book S Mystery20 20 Author T USA 120 Book T Mystery

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the inner_join function from the dplyr package to perform the natural join between the authors and books datasets on the author_id column.
Finally, we display the result to see which authors have written which books.

Interpretation of ResultsThe resulting dataset authors_books contains all rows from both the authors and books datasets where there is a matching value in the author_id column. This means that only authors who have written books are included, along with the details of those books. (Yes, it is a kind of inner join.)

Homework for ReadersIn the same natural_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

staff: Contains information about staff members.
Columns: staff_id, name, role
assignments: Contains information about project assignments.
Columns: assignment_id, staff_id, project_name

Your task is to perform a natural join on these datasets to combine staff details with their project assignments. Use the staff_id column for joining.

Self JoinA Self Join is a join of a table to itself. It is used to compare rows within the same table. This can be particularly useful for hierarchical data, such as organizational structures, where you need to find relationships between rows within the same table.

Explanation of the ScenarioIn this scenario, we have employee information, and each employee has a manager, who is also an employee. We want to find the relationship between employees and their managers using the same table. This helps in understanding the organizational hierarchy.

Data file:
https://github.com/kgryczan/medium_publishing/blob/main/self_join_data.RData

Description of the DatasetsWe will use one dataset:

employees: Contains information about employees and their managers.
Columns: employee_id, name, manager_id

Step-by-Step Code ExamplesLoading the dataset

```

Load the necessary librarieslibrary(dplyr)# Load the datasetload("self_join_data.RData")# Display the datasetprint(employees)# A tibble: 20 × 3 employee_id name manager_id 1 1 Alice E 12 2 2 Bob C 6 3 3 Carol L 8 4 4 Zoe K 15 5 5 Alice G 10 6 6 Bob D 7 7 7 Carol S 19 8 8 Zoe T 1 9 9 Alice O 1310 10 Bob O 211 11 Carol G 1612 12 Zoe P 813 13 Alice H 414 14 Bob K 1415 15 Carol B 1816 16 Zoe E 417 17 Alice O 718 18 Bob T 1419 19 Carol R 820 20 Zoe G 16

``` Performing the Self Join

```

Perform the self joinemployees_managers <- employees %>% inner_join(employees, by = c("manager_id" = "employee_id"), suffix = c("_employee", "_manager"))# Display the resultprint(employees_managers)# A tibble: 20 × 5 employee_id name_employee manager_id name_manager manager_id_manager 1 1 Alice E 12 Zoe P 8 2 2 Bob C 6 Bob D 7 3 3 Carol L 8 Zoe T 1 4 4 Zoe K 15 Carol B 18 5 5 Alice G 10 Bob O 2 6 6 Bob D 7 Carol S 19 7 7 Carol S 19 Carol R 8 8 8 Zoe T 1 Alice E 12 9 9 Alice O 13 Alice H 410 10 Bob O 2 Bob C 611 11 Carol G 16 Zoe E 412 12 Zoe P 8 Zoe T 113 13 Alice H 4 Zoe K 1514 14 Bob K 14 Bob K 1415 15 Carol B 18 Bob T 1416 16 Zoe E 4 Zoe K 1517 17 Alice O 7 Carol S 1918 18 Bob T 14 Bob K 1419 19 Carol R 8 Zoe T 120 20 Zoe G 16 Zoe E 4

``` Explanation of the Code:

We first load the dataset using the load function.
We then use the inner_join function from the dplyr package to join the employees table to itself. The join condition matches the manager_id of one row with the employee_id of another row. The suffix argument is used to distinguish between the employee and manager columns.
Finally, we display the result to see the relationship between employees and their managers.

Interpretation of ResultsThe resulting dataset employees_managers contains pairs of employees and their managers. Each row shows an employee along with their corresponding manager, including details such as names and IDs.

Homework for ReadersIn the same self_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

friends: Contains information about friendships within a social network.
Columns: person_id, friend_id

Your task is to perform a self join on this dataset to analyze the friendships and find mutual friends. Use the person_id and friend_id columns for joining.

Equi JoinAn Equi Join is a type of join that combines rows from two tables based on equality conditions between specified columns. It is one of the most common types of joins used in SQL and relational database management. Take into a consideration that all types we already talked about are equi joins. It is just wider definition, beacuse we are looking for equality of keys.

Explanation of the ScenarioIn this scenario, we have orders and order details. We want to join these tables to get a comprehensive view of each order along with its details. This helps in understanding the full scope of each transaction.

Data file:
https://github.com/kgryczan/medium_publishing/blob/main/equi_join_data.RData

Description of the DatasetsWe will use two datasets:

orders: Contains information about customer orders.
Columns: order_id, customer_id, order_date
order_details: Contains detailed information about each order.
Columns: order_detail_id, order_id, product_id, quantity

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("equi_join_data.RData")# Display the datasetsprint(orders)# A tibble: 20 × 3 order_id customer_id order_date 1 1 113 2024-01-01 2 2 113 2024-01-02 3 3 112 2024-01-03 4 4 105 2024-01-04 5 5 117 2024-01-05 6 6 115 2024-01-06 7 7 112 2024-01-07 8 8 107 2024-01-08 9 9 117 2024-01-0910 10 106 2024-01-1011 11 105 2024-01-1112 12 109 2024-01-1213 13 117 2024-01-1314 14 113 2024-01-1415 15 101 2024-01-1516 16 107 2024-01-1617 17 115 2024-01-1718 18 103 2024-01-1819 19 108 2024-01-1920 20 115 2024-01-20print(order_details)# A tibble: 40 × 4 order_detail_id order_id product_id quantity 1 1 17 206 10 2 2 18 215 4 3 3 13 218 8 4 4 16 217 3 5 5 2 217 5 6 6 2 203 4 7 7 20 219 9 8 8 18 214 4 9 9 12 218 410 10 14 214 4# ℹ 30 more rows

``` Performing the Equi Join

```

Perform the equi joinorders_with_details <- inner_join(orders, order_details, by = "order_id")# Display the resultprint(orders_with_details)# A tibble: 40 × 6 order_id customer_id order_date order_detail_id product_id quantity 1 1 113 2024-01-01 39 210 5 2 2 113 2024-01-02 5 217 5 3 2 113 2024-01-02 6 203 4 4 2 113 2024-01-02 22 219 1 5 2 113 2024-01-02 25 220 8 6 2 113 2024-01-02 36 213 3 7 2 113 2024-01-02 38 202 9 8 4 105 2024-01-04 12 215 4 9 4 105 2024-01-04 24 201 510 4 105 2024-01-04 30 219 3# ℹ 30 more rows

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the inner_join function from the dplyr package to perform the equi join between the orders and order_details datasets on the order_id column.
Finally, we display the result to see the full details of each order.

Interpretation of ResultsThe resulting dataset orders_with_details contains all rows from the orders dataset with the matching rows from the order_details dataset based on the order_id column. This means that each order is enriched with its detailed information.

Homework for ReadersIn the same equi_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

athletes: Contains information about athletes.
Columns: athlete_id, name, sport
performance_records: Contains information about performance records of athletes.
Columns: record_id, athlete_id, event, score

Your task is to perform an equi join on these datasets to match athletes with their performance records. Use the athlete_id column for joining.

In this second part of our series, “Joins Are No Mystery Anymore: Hands-On Tutorial,” we delved into more advanced join techniques that are essential for comprehensive data analysis. We explored:

Anti Join: Identifying rows in one table that do not have corresponding rows in another, helping to spot outstanding payments.
Cross Join: Creating all possible combinations of rows from two tables, useful for planning and organizing.
Natural Join: Automatically joining tables based on columns with the same names and types, simplifying the linking process.
Self Join: Comparing rows within the same table to uncover relationships, such as employee-manager hierarchies.
Equi Join: Combining rows from two tables based on equality conditions, providing detailed insights into orders and their specifics.

Through practical scenarios and step-by-step code examples, we enhanced our understanding of these joins and their applications. Each join type was demonstrated with real-life datasets, allowing you to see how these techniques can be applied to solve everyday data problems.

Next week, we will conclude our series with even more specialized join techniques. We’ll cover Non-Equi Joins, Rolling Joins, Overlap Joins, and Fuzzy Joins. These advanced joins will help you handle more complex data scenarios, such as matching based on non-equality conditions, finding the nearest matches, and dealing with approximate or fuzzy data. Stay tuned as we unlock the full potential of joins in R and take your data analysis skills to the ultimate level.

Thank you for sticking with us through the second part of our “Joins Are No Mystery Anymore: Hands-On Tutorial” series! Your dedication to mastering data joins in R is commendable.

As a special treat for our patient readers, next week we’ll dive into “Anatomy of a Basic Joining Function.” This extra content will break down the different arguments in joining functions, explaining what each one does and how it changes the output. It’s a deep dive into the mechanics of joins that will enhance your understanding and give you even greater control over your data analysis.

Stay tuned and happy coding!

Joins Are No Mystery Anymore: Hands-On Tutorial — Part 2 was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Joins Are No Mystery Anymore: Hands-On Tutorial — Part 2

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Note: Thank you Kamil Żyła for providing guidance and expertise in writing R code for this article.

R, being single-threaded in nature, isn’t the fastest programming language out there. You have options when it comes to parallelism, but these often don’t reduce the runtime as much as you’d want.

So when the official 1 Billion Row Challenge popped up, our interest went through the roof! The challenge doesn’t allow for third-party packages. We’ll start the article by obeying the rules, but later, we’ll see what sort of performance benefits DataTable and DuckDB offer. For reference, the fastest Java implementation takes 1.535 seconds, and pure Python is around 10-15 times slower than that.

‍Without much ado, let’s see how R stacks up!

Can you parallelize data frame operations in R? You most definitely can with the R doParallel package!

Table of contents:* 1 Billion Row Challenge Introduction * Base R Implementation – No Third-Party Libraries * R 1 Billion Row Challenge – Using Third-Party Packages * ‍Benchmark Results – One Billion Row Challenge in R * Summing up R One Billion Row Challenge

1 Billion Row Challenge IntroductionThe idea of the challenge is simple – Go through a 1 billion row (13.8GB) text file that contains arbitrary temperature measurements and calculate summary statistics for each station (min, mean, and max). The data is generated by a script and is not downloadable from the web.

Once processed, you should end up with a text output similar to the following:

Image 1 – Sample 1BRC outputThe official challenge author runs the submissions on a Hetzner Cloud CCX33 instance (8 dedicated vCPU, 32 GB RAM). We don’t have access to it, so we’ll use a powerful 16” M3 Pro Macbook Pro with 12 CPU cores and 36 GB of RAM. Your results will be different if using different hardware.

You now know what the challenge is. Up next, we’ll continue with the base R solution that uses no external packages.

Base R Implementation – No Third-Party LibrariesAs you know by now, 1BRC doesn’t allow third-party packages. That’s a problem because you’ll have to reinvent the wheel. It’s not a realistic scenario by any means, since the whole industry relies on these packages when writing data pipelines.

Still, let’s see what R has to offer. We’ll write a couple of completely manual solutions first, and then opt for a couple of convenient built-in R functions.

Single-Threaded ImplementationThis one takes some time to write, but is simple to read and understand. There are no fancy R-specific functions in it, and if you’re familiar with any programming language, this implementation should feel right at home.

We start by creating a new environment and then iterating over the text file one line at a time. Summary statistics are computed and updated on each iteration. For the mean, we’re keeping track of the sum and count, which are then divided to get the average value:

neutral\_values <- c(100, -100, 0, 0)process\_file <- function(file\_name) { result <- new.env() con <- file(file\_name, "r") while (length(line <- readLines(con, n = 1, warn = FALSE)) > 0) { parts <- strsplit(line, split = ";", fixed = TRUE)[[1]] station\_name <- parts[1] measurement <- as.numeric(parts[2]) values <- get0(station\_name, result, ifnotfound = neutral\_values) result[[station\_name]] <- c( min(values[[1]], measurement), max(values[[2]], measurement), values[[3]] + measurement, values[[4]] + 1 ) } close(con) lines <- lapply(sort(names(result)), function(station\_name) { values <- result[[station\_name]] min\_value <- values[[1]] max\_value <- values[[2]] mean\_value <- values[[3]] / values[[4]] sprintf("%s=%.1f/%.1f/%.1f", station\_name, min\_value, mean\_value, max\_value) }) output <- paste0("{", paste(lines, collapse = ", "), "}") print(output)}process\_file("measurements.txt") Multi-Threaded ImplementationThe doParallel package in R allows you to parallelize an R function across multiple CPU cores. The challenge states no third-party packages are allowed, but this one has nothing to do with data processing, so we’ll consider it fair.

The core idea is the same as in the single-threaded implementation, but you now need to split the text file into equally sized chunks and process the chunks in parallel. You also need to calculate summary statistics for each chunk, and then combine the statistics for the final output:

library(foreach)library(doParallel)n\_workers <- detectCores() - 1cluster <- makeCluster(n\_workers)registerDoParallel(cluster)get\_file\_chunks <- function(file\_name) { file\_size <- file.info(file\_name)$size chunk\_size <- ceiling(file\_size / n\_workers) con <- file(file\_name, "r") boundaries <- lapply(seq\_len(n\_workers - 1), function(i) { boundary <- i * chunk\_size seek(con, boundary) line <- readLines(con, n = 1) boundary + nchar(line) + 1 }) close(con) boundaries <- c(0, boundaries, file\_size) lapply(seq\_len(n\_workers), function(i) { list( start\_pos = boundaries[[i]], end\_pos = boundaries[[i + 1]] ) })}process\_file <- function(file\_name, max\_cpu = 12) { process\_chunk <- function(file\_name, start\_pos, end\_pos) { neutral\_values <- c(100, -100, 0, 0) result <- new.env() con <- file(file\_name, "r") seek(con, start\_pos) while (seek(con) < end\_pos) { line <- readLines(con, n = 1, warn = FALSE) parts <- strsplit(line, split = ";", fixed = TRUE)[[1]] station\_name <- parts[1] measurement <- as.numeric(parts[2]) values <- get0(station\_name, result, ifnotfound = neutral\_values) result[[station\_name]] <- c( min(values[[1]], measurement), max(values[[2]], measurement), values[[3]] + measurement, values[[4]] + 1 ) } close(con) return(result) } chunks <- get\_file\_chunks(file\_name) result <- foreach(i = chunks) %dopar% { process\_chunk( file\_name = file\_name, start\_pos = i$start\_pos, end\_pos = i$end\_pos ) } result\_fmt <- list() for (chunk\_res in result) { for (station\_name in names(chunk\_res)) { if (!(station\_name %in% names(result\_fmt))) { result\_fmt[[station\_name]] <- chunk\_res[[station\_name]] } else { result\_fmt[[station\_name]][1] <- min(result\_fmt[[station\_name]][1], chunk\_res[[station\_name]][1]) result\_fmt[[station\_name]][3] <- result\_fmt[[station\_name]][3] + chunk\_res[[station\_name]][3] result\_fmt[[station\_name]][2] <- max(result\_fmt[[station\_name]][2], chunk\_res[[station\_name]][2]) result\_fmt[[station\_name]][4] <- result\_fmt[[station\_name]][4] + chunk\_res[[station\_name]][4] } } } for (station\_name in names(result\_fmt)) { result\_fmt[[station\_name]][5] <- result\_fmt[[station\_name]][3] / result\_fmt[[station\_name]][4] } result\_fmt <- result\_fmt[order(names(result\_fmt))] output <- "{" for (station\_name in names(result\_fmt)) { min\_value <- result\_fmt[[station\_name]][1] mean\_value <- result\_fmt[[station\_name]][5] max\_value <- result\_fmt[[station\_name]][2] output <- paste0(output, station\_name, "=", sprintf("%.1f/%.1f/%.1f, ", min\_value, mean\_value, max\_value)) } output <- substr(output, 1, nchar(output) - 2) output <- paste0(output, "}") print(output) stopCluster(cluster)}process\_file("measurements.txt") More Elegant Base R ImplementationsBoth of the previous implementations are long and don’t leverage any built-in functions R is famous for. In this section, we’ll explore two R-specific implementations, one using aggregate() and the other using tapply(). Both will be called on a data frame that has been read into memory by using read.delim().

Let’s take a look into aggregate() first:

process\_file <- function(file\_name) { df <- read.delim( file = file\_name, header = FALSE, sep = ";", col.names = c("station\_name", "measurement") ) summarize <- function(x) { c( min = min(x), mean = mean(x), max = max(x) ) } res <- aggregate(measurement ~ station\_name, data = df, FUN = summarize) res <- do.call(data.frame, res) output <- "{" for (i in 1:nrow(res)) { row <- res[i, ] output <- paste0(output, row$station\_name, "=", sprintf("%.1f/%.1f/%.1f, ", row$measurement.min, row$measurement.mean, row$measurement.max)) } output <- substr(output, 1, nchar(output) - 2) output <- paste0(output, "}") print(output)}process\_file("measurements.txt") The aggregate() function returns two columns – station name and the matrix of measurements. This matrix needs to be expanded into multiple columns before printing (do.call()).

Up next, let’s take a look at the tapply() example:

process\_file <- function(file\_name) { df <- read.delim( file = file\_name, header = FALSE, sep = ";", col.names = c("station\_name", "measurement") ) summarize <- function(x) { c( min = min(x), mean = mean(x), max = max(x) ) } res <- do.call("rbind", tapply(df$measurement, df$station\_name, summarize)) res <- data.frame(res) res$station\_name <- row.names(res) row.names(res) <- NULL output <- "{" for (i in 1:nrow(res)) { row <- res[i, ] output <- paste0(output, row$station\_name, "=", sprintf("%.1f/%.1f/%.1f, ", row$min, row$mean, row$max)) } output <- substr(output, 1, nchar(output) - 2) output <- paste0(output, "}") print(output)}process\_file("measurements.txt") The tapply() function calculates statistical summaries by a group based on the levels of one or several factors, making it an ideal candidate here.

R 1 Billion Row Challenge – Using Third-Party PackagesOnto the third-party R packages now! These are typically used in R pipelines to increase code readability, reduce runtime, and minimize the amount of code you need to write.

DplyrThe dplyr package is used and loved by many in R community. It has an intuitive and beginner-friendly syntax, which means any newcomer will feel comfortable in a matter of days. Still, it’s a tool built for convenience, not for performance.

That being said, it does allow you to read and aggregate the data in a single command, due to the pipe operator:

library(dplyr)process\_file <- function(file\_name) { df <- read.csv( file = file\_name, header = FALSE, sep = ";", col.names = c("station\_name", "measurement") ) %>% group\_by(station\_name) %>% summarise( min\_measurement = min(measurement), mean\_measurement = mean(measurement), max\_measurement = max(measurement) ) %>% arrange(station\_name) output <- "{" for (i in 1:nrow(df)) { row <- df[i, ] output <- paste0(output, row$station\_name, "=", sprintf("%.1f/%.1f/%.1f, ", row$min\_measurement, row$mean\_measurement, row$max\_measurement)) } output <- substr(output, 1, nchar(output) - 2) output <- paste0(output, "}") print(output)}process\_file("measurements.txt") Data.TableIn R, most developers turn to the data.table package when dplyr becomes too slow. There’s even a dtplyr package that combines the best of both worlds, but it’s out of the scope for today’s article.

In plain English, data.table is an enhanced version of data.frame and allows for faster data lookups, analysis, and aggregation. It does have a steeper learning curve when compared to dplyr, and that’s the reason why most developers opt for dtplyr.

But for a simple data aggregation the syntax isn’t all that different:

library(data.table)process\_file <- function(file\_name) { dt <- fread(file\_name, sep = ";", col.names = c("station\_name", "measurement")) dt\_summary <- dt[, .( min\_measurement = min(measurement), mean\_measurement = mean(measurement), max\_measurement = max(measurement) ), by = station\_name ] setkey(dt\_summary, station\_name) output <- "{" for (i in 1:nrow(dt\_summary)) { row <- dt\_summary[i, ] output <- paste0( output, row$station\_name, "=", sprintf("%.1f/%.1f/%.1f, ", row$min\_measurement, row$mean\_measurement, row$max\_measurement) ) } output <- substr(output, 1, nchar(output) - 2) output <- paste0(output, "}") print(output)}process\_file("measurements.txt") DuckDBAnd finally, let’s discuss DuckDB. It’s an in-memory relational OLAP DBMS designed to go through huge amounts of data blazing fast. Our recent comparison of DuckDB and dplyr concluded that the prior is almost 20 times faster while requiring minimal code changes.

DuckDB allows you to use dplyr-like syntax or SQL. We’ll opt for the latter today, just to mix things up:

library(duckdb)library(stringr)process\_file <- function(file\_name) { df <- dbGetQuery( conn = dbConnect(duckdb()), statement = str\_interp(" select station\_name, min(measurement) as min\_measurement, cast(avg(measurement) as decimal(8, 1)) as mean\_measurement, max(measurement) as max\_measurement from read\_csv( '${file\_name}', header=false, columns={'station\_name': 'varchar', 'measurement': 'decimal(8, 1)'}, delim=';', parallel=true ) group by station\_name order by station\_name ") ) output <- "{" for (i in 1:nrow(df)) { row <- df[i, ] output <- paste0(output, row$station\_name, "=", sprintf("%.1f/%.1f/%.1f, ", row$min\_measurement, row$mean\_measurement, row$max\_measurement)) } output <- substr(output, 1, nchar(output) - 2) output <- paste0(output, "}") print(output)}process\_file("measurements.txt") Congratulations – you’ve made it to the end of the package comparison and coding part of the article! Up next, we’ll compare the results.

Benchmark Results – One Billion Row Challenge in RAfter navigating through countless lines of code, it’s time to take a look at the average runtime results. For reference, the results you’re about to see are an average of 3 runs for each data processing method. The hardware of choice was a 16” M3 Pro MacBook Pro with 12 CPU cores and 36 GB of RAM:

Image 2 – Average runtime results for R 1BRCThere’s a clear reason why everyone and their mothers use third-party packages. It requires you to write less code and reduces runtime by orders of magnitude.

Vanilla (single-threaded) R implementation was by far the slowest – averaging around 50 minutes. Distributing the workload over 11 CPU cores reduced the average runtime to about 17 minutes.

No matter what we tried to do, the aggregate() method approach always ran out of memory.

The real winner here is DuckDB, providing on average ~ 335x faster runtime when compared to single-threaded R implementation, or ~ 115x faster runtime when compared to multi-threaded implementation.

Summing up R One Billion Row ChallengeTo conclude, R isn’t the fastest programming language out there, nor was it ever designed to be one. However, the sheer amount of R-specific functions (apply family) and third-party packages make it a viable solution for processing huge amounts of data. Today you’ve seen this claim proven, as we managed to process 1 billion rows of data in under 10 seconds!

The biggest bottleneck here is the data itself. No one would ever (we hope) store such a huge amount of records in an uncompressed plain text file. Alternative file formats, such as Apache Parquet, would reduce the file size appropriately 5 times, which would then have the effect of further reducing the data processing time.

‍What are your thoughts on the 1 Billion Row Challenge in R? Have you managed to reduce the runtime even further? Make sure to let us know in our Shiny community on Slack.

Dealing with slow and frustrating data.frame computations? We explain how R and doParallel allow you to make it faster.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R One Billion Row Challenge: Is R Viable Option for Analyzing Huge Datasets?

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, everyone!

I’m excited to introduce a new content series that will be shared across multiple platforms, including my blog, LinkedIn, Mastodon, and Telegram. This series is designed to provide you with valuable insights, practical tips, and deep dives into the tools and techniques I’ve developed and co-authored. Whether you’re a data scientist, analyst, or someone looking to enhance your data manipulation skills, there will be something for everyone.

Here’s what you can expect each week:

Monday: Introduction to a PackageOn Mondays, I’ll introduce one of the packages I have written. These introductions will cover the package’s purpose, its main features, and how it can help you in your data analysis and visualization tasks. Whether you’re familiar with my work or new to it, these posts will provide a comprehensive overview of each package’s capabilities.

Tuesday: An R FunctionTuesdays will be dedicated to exploring a specific R function. I’ll provide a detailed explanation of how the function works, its applications, and some examples to help you understand how to use it effectively in your projects. This will be a great way to expand your R programming skills and learn new techniques.

Wednesday: VBA and ROn Wednesdays, we’ll explore the integration of VBA and R. These posts will show you how to leverage the power of both tools to automate tasks, enhance your Excel capabilities, and streamline your workflows. If you’re looking to bridge the gap between Excel and R, these sessions will be invaluable.

Thursday: Practical ExampleThursdays will feature practical examples from the package introduced on Monday. I’ll walk you through real-world scenarios and show you how to apply the package to solve specific problems. These examples will help you see the practical applications of the tools and give you ideas for your own projects.

Friday: Insights from “Extending Excel with Python and R”Finally, Fridays will be dedicated to insights and snippets from my book, co-authored with David Kun, titled “Extending Excel with Python and R.” We’ll cover various topics from the book, providing you with a sneak peek into its contents and practical tips for extending Excel’s functionality using Python and R.

I am looking forward to sharing this journey with you and hearing your feedback. Make sure to follow along, and don’t hesitate to ask questions or share your thoughts in the comments. Let’s learn and grow together!

Stay tuned for the first post of this series coming next Monday!

Best regards,
Steve

Connect with me:

Website: www.spsanderson.com
Blog: www.spsanderson.com/steveondata/
LinkedIn: www.linkedin.com/in/spsanderson
Mastodon: mstdn.social/@stevensanderson
Telegram: t.me/steveondata

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introduction of My Content Series

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Developing dendroNetwork as a package was not a goal from the beginning, but looking back, I think that it should have been. I wish someone had suggested making a package to me much earlier. Why? Because of many things, but mostly: reproducibility and transparency. This enables others to also use the method and software. In addition, I really enjoyed the open peer-review process after submitting to rOpenSci, because the reviewers showed me various improvements and how people from outside my discipline viewed the software and documentation. Meeting new people is always nice and a great bonus of the process.

A few years ago I published a method developed to explore patterns in the similarity between tree-ring series using network analysis.1 In dendrochronology the pair-wise comparison of tree-ring series is an often used method to look for patterns in, for example, the provenance of the wood, especially for archaeological wood, since we often know where the wooden objects came from, but not where the trees were felled. This type of study often results in a complex set of relations and I think that network analysis is a great method for visualising and analysing these relations without losing the complexity. I developed this method and shared the code with my publication.2 Part of the code had been written by others and adapted for the analyses. The fact that Angelo Salatino (The Open University) shared his code openly and Andy Bunn (Western Washington University) did the same, shows how beneficial openly sharing code is to others and for the progress of research. Further research was performed, refining the method and exploring other avenues, such as looking at wood use within a Roman ship and the provenance of wood used in these ships.3 Later, I read a blog post about packaging and how easy it could be. Unfortunately, I do not remember which blog it was, but it inspired me to convert my code into a package, for the benefit of others, but also because I wanted to learn how to do this.

For me, creating a package from my existing scripts started with these three excellent resources (in random order):

https://devguide.ropensci.org/
https://r-pkgs.org/
https://carpentries-incubator.github.io/lesson-R-packaging/

The easy bit was converting scripts into functions, since this is pretty straight forward programming. However, I never realized how much work writing (good) documentation is. This took me much longer than I expected, also because I was (and am) doing this in my spare time. However, it was also worthwhile to do this and explain what the functions and variables are for. I also learned a lot about the potential for continuous integration, automated testing and checking, especially from the rOpenSci-documentation. A lot of information can be found there, which is really helpful. It takes an effort to read everything, but in the end it can save you a lot of time if you read it before you start developing a package!

I really enjoyed the open peer-review process. After reading all the documentation and making sure that dendroNetwork complied to the standards of rOpenSci, the process of submission was very easy. I enjoyed the feedback I got from the reviewers Kaija Gahm and Zachary Gajewski and their comments really improved the software in a whole.

I would recommend any research who develops a new method with code to do the following things:

Be nice, and share and document your code!
Develop functions not scripts.
Develop a package from the start, especially when devising a new method(ology).
Read the documentation (or: RTFM)
Submit to rOpenSci and enjoy!

Visser, R. M. (2021). Dendrochronological Provenance Patterns. Network Analysis of Tree-Ring Material Reveals Spatial and Economic Relations of Roman Timber in the Continental North-Western Provinces. Journal of Computer Applications in Archaeology, 4(1), 230. https://doi.org/10.5334/jcaa.79 ︎
Visser, R. M. (2022). Dendrochronological Provenance Patterns. Code and Data of Network Analysis of Tree-Ring Material [dataset]. https://doi.org/10.5281/zenodo.10200361 ︎
Visser, R. M., & Vorst, Y. (2022). Connecting Ships: Using Dendrochronological Network Analysis to Determine the Wood Provenance of Roman-Period River Barges Found in the Lower Rhine Region and Visualise Wood Use Patterns. International Journal of Wood Culture, 3(1-3), 123-151. https://doi.org/10.1163/27723194-bja10014

Visser, R., & Vorst, Y. (2022). Analyses, data and figures related to: ‘Connecting ships: using dendrochronological network analysis to determine the wood provenance of Roman-period river barges found in the Lower Rhine region and to visualise patterns of wood use’ [dataset]. https://doi.org/10.5281/zenodo.7243539 ︎ To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: From scripts to package. Developing dendroNetwork and learning with rOpenSci

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The ability to write JavaScript code is essential to creating fast, custom, highly interactive Shiny applications. Whenever you want near-instant user interaction, use browser features, or add custom server-client communication, you would reach out to JavaScript.

Despite all that, the majority of JS code in Shiny apps looks… sad. As if it’s a throw-away one-time copy-pasted code doing one little JS trick – nobody wants to go back to this code and seemingly no one wants to maintain and care about this code.

Make JavaScript your friend for building amazing Shiny apps! Read this guide to learn easy tips and tricks for writing clean, maintainable JavaScript code.

The situation gets out of hand when the number of lines of JavaScript code in the project grows. This is where Rhino helps: developers can easily split their code across multiple files, treat those files as modules (use import statements), and expose exported functions to the R code (via App.myFunction()). Rhino even gives you a pre-configured webpack that produces a minimized JS bundle.

But today we are going to go even further, and learn how to treat JS code with the same (if not greater!) care as we treat our R code. To keep up with this tutorial you will need to have VS Code, Node.js and Rhino (at the moment of writing versions are 1.86.0, 18.16.0 and 1.7.0 respectively).

Part 0. Setup and IntroductionCreate a Rhino app and run one of its Node.js-dependant functions to initialize the .rhino folder. We also need to make .rhino indexable by git – the tweaks we are about to make should be available to CI/CD pipelines, as well as other developers on the team. In the R console, run these two commands:

rhino::init()rhino::build\_js() ‍

After the installation is complete, replace .rhino/.gitignore file contents with just the following:

node\_modules ‍

It ensures that we don’t track node_modules (JavaScript dependencies) in git, but other config files are available in version control.

Finally, to always have properly formatted JavaScript code, I like to enable the Prettier extension, enable formatOnSave setting in VS Code, and add .prettierrc file to the root directory with the following content:

{ "singleQuote": true} Now we can start writing some code. Out of the box we have an app/js/index.js file available to us, where we can write some JavaScript. Let’s try this:

function doSomething(a, b) { console.log('hello');} ‍

This function doesn’t look good, and if we run rhino::lint_js() we will get a few errors:

/Users/pavel/tmp/app/js/index.js 1:10 error 'doSomething' is defined but never used no-unused-vars 1:22 error 'a' is defined but never used no-unused-vars 1:25 error 'b' is defined but never used no-unused-vars 3 problems (3 errors, 0 warnings) ‍

Okay, we can fix those by changing our function body and exporting the function itself (exported functions become available on App object that is created by Rhino):

export function doSomething(a, b) { const result = a * b; return result;} ‍

Now, it would be great to document the parameters of this function as it will produce unexpected results if a and b are not numbers. This is the very first step to start caring about our code: similar to roxygen2 in R, we have JSDoc in JavaScript that enables us to write inline documentation:

/** * My function that does something. * @param {number} a - first argument * @param {number} b - second argument * @returns {number} Result of the operation. */export function doSomething(a, b) { const result = a * b; return result;} ‍

Not only it looks better, the documentation is immediately available to us when we use this function in VS Code:

Part 1. Simple TypingWhat we’ve seen so far was only the beginning and honestly, nothing fancy yet. Documenting the code is a common best practice, but even so it’s still worth mentioning, because we are going to write maintainable JavaScript.

Anyway, this is where things take off a little bit. While roxygen comments in R are mere suggestions, JSDoc comments are much more powerful. By leveraging the TypeScript compiler & server built into VS Code, we can get a TypeScript-like experience only with the comments.

To prove it, let’s first change our function signature:

/** * My function that does something. * @param {string} a - first argument * @param {string} b - second argument * @returns {void} Result of the operation. */export function doSomething(a, b) { const result = a * b; return result;} ‍

Notice how, according to the documentation, it now expects string arguments and doesn’t return any value. But we don’t see any errors – we will only get them at runtime.

To fix it, let’s create a jsconfig.json file at the root of our project – this file will give some instructions to VS Code and we will update this file a few times in the future:

{ "compilerOptions": { "checkJs": true, "strict": true }} Now, we just need to reload VS Code by opening the command palette (Ctrl+Shift+P or Cmd+Shift+P for MacOS) and calling Developer: Reload Window.

‍

Voila:

A more common scenario is depicted on the next screenshot: correct function signature, but incorrect argument types passed to the function:

You may think: this example is not very exciting, it’s clear what arguments should be passed to it. Well, let’s consider another function:

/** * @param {HTMLElement} element * @param {string} margin * @returns {void} */function addMargin(element, margin) { element.style.margin = margin;} ‍

If you have ever worked with a UI framework in React, you might expect margin to be a number (because it will be processed and scaled based on other theme properties to make sure that the overall design system remains consistent).

But in vanilla JS it should be a string, e.g. “12px” or “1rem”, etc. And this is what we have defined in the JSDoc.

Moreover, thanks to the TypeScript server running, we have documentation and type checks not only for our code, but also for the built-in object – if we try to assign a number to element.style.margin we will immediately see a TypeScript error.

Part 2. Advanced TypingAside from the primitive values, we often use Objects in JavaScript. Let’s consider a function that takes an Object representing a person, transforms one of its properties and returns the result:

function getDateOfBirth(person) { return person.dob;} ‍

I get zero autocompletion on my person Object – but it’s fine (is it?), because I remember that it should have a dob property which is stored as Date.

So I will take this property, convert it to locale-aware date string and return it, right? Well, I wasn’t sure if I remember the method name correctly, so I had to create a new variable of type Date and check the available methods on it. Turns out I was wrong, and would get an error at runtime in my browser.

It would help to document the type of the person parameter, but this time it is an Object, not a primitive value. We can define complex object types with JSDoc the following way:

/** * @typedef Person * @property {string} name - Full name of a person * @property {number} age - Number of full years * @property {Date} dob - Date of birth */ ‍

And now we can use Person as a custom type when defining parameters of our function, which will give us nice code completion.

‍

Defining types in this way is OK, but there is a more convenient way. Remember that VS Code is running a TypeScript server in the background, which means that we can leverage not only JSDoc, but global type declarations as well using“d.ts” files.

Types from such files will not affect our code and they won’t be included in the final JS bundle – they will be used only by VS Code itself to run type checking and to give us code suggestions.

Let’s extract our type definition into a separate data.d.ts file in app/js/types folder and make it a little bit more complex:

// app/js/types/data.d.tsinterface Person { name: string; age: number; dob: Date; address: { street: string; city: string; zip: number; };} ‍

Now we can use this interface in our JavaScript file:

‍

And just like that we wrote a little bit of TypeScript. We globally declared an interface Person that can be used throughout our codebase.

Combining R and Javascript for the first? Check out this blog post to learn wh it’s a good idea and how you can get started.

Part 3. Type Definitions From 3rd PartiesGlobal type declarations are easier to read, easier to use and easier to maintain. But the best feature of d.ts files is that many 3rd party libraries written in JS ship these declaration files to provide developers with code completion and type safety.

‍

I suggest that we try that out with our most beloved library: Shiny. This command will install Shiny type definitions, and the process might take a few minutes at first.

cd .rhinonpm i -D https://github.com/rstudio/shiny ‍

You can verify that the installation is successful by checking for the presence of .rhino/node_modules/@types/rstudio-shiny folder.

‍

Usually when developing a JavaScript project one would have node_modules at the root of the project, so VS Code knows where to look for installed packages and types.

In the case of a Rhino app all JS-related stuff is hidden in the .rhino folder, so we need to explicitly “point” VS Code to the correct place. We can do it by updating the jsconfig.json and reloading the window as before.

{ "compilerOptions": { "checkJs": true, "strict": true, "typeRoots": [ ".rhino/node\_modules/@types/" ] }} Let’s go back to the index.js file and write some typical JS code that you can see in a Shiny app:

‍

I hope you enjoyed that sweet autocomplete just as I did! The only thing I had to remember is the shiny:connected event name.

This example was a little bit artificial, so let’s come up with something more useful. Along the way I will demonstrate another powerful feature of types – ability to catch potential errors when writing the code, not when running the app.

A fairly common situation in web development – and Shiny apps are no exception – is when developers know that a certain feature only works properly in certain browsers.

Let’s use an example from MDN to detect which web browser is used to access the app. In the scope of this article we only want to show the browser name on the screen, but in a real world scenario you may want to show a warning, or even disable some functionalities.

Here’s the JavaScript function to get the browser name:

/** * @param {string} userAgent * @returns */function getBrowserName(userAgent) { // The order matters here, and this may report false positives for unlisted browsers. if (userAgent.includes('Firefox')) { return 'Mozilla Firefox'; } else if (userAgent.includes('SamsungBrowser')) { return 'Samsung Internet'; } else if (userAgent.includes('Opera') || userAgent.includes('OPR')) { return 'Opera'; } else if (userAgent.includes('Edge')) { return 'Microsoft Edge (Legacy)'; } else if (userAgent.includes('Edg')) { return 'Microsoft Edge (Chromium)'; } else if (userAgent.includes('Chrome')) { return 'Google Chrome or Chromium'; } else if (userAgent.includes('Safari')) { return 'Apple Safari'; } else { return 'unknown'; }} ‍

And here’s a function that will set the corresponding input value in Shiny:

function setBrowserName() { const browserName = getBrowserName(window.navigator.userAgent); Shiny.setInputValue('app-browser\_name', browserName);}$(() => { setBrowserName();}); ‍

If you paste this code to your local VS Code, you will notice a TypeScript error: Cannot invoke an object which is possibly ‘undefined’. What does it even mean and how is it possible? We will find out in a moment, but first let’s update our R code. Find output$message code block and replace it with the following:

```

don't forget to add req to box::use(shiny[...])output$message <- renderUI({ req(input$browser_name) tags$h1(input$browser_name)})

``` ‍

Rebuild JS bundle and start the app (pro tip: you can open another terminal and run rhino::build_js(watch = TRUE) to keep it always up to date):

rhino::build\_js()shiny::runApp() Open the app in the browser, and notice that… we don’t have anything on the screen .

If you open developer tools in the browser, you’d see something like Uncaught TypeError: Shiny.setInputValue is not a function. We know that Shiny has the setInputValue method – the problem is that at the moment of invocation, setInputValue was undefined. The reason lies in the fact that Shiny initialization finishes with a small delay after the document is ready.

‍

Anyway, guess what? This is exactly what our TypeScript error was trying to convey (maybe not in the best terms, but still) – there is a chance that when calling this function, it is not available yet. We can fix this issue by doing two things:

check that setInputValue is available,
invoke the whole thing only after Shiny has emitted “connected” event:

function setBrowserName() { if (Shiny.setInputValue === undefined) { throw new Error('This function must be called only after shiny:connected'); } const browserName = getBrowserName(window.navigator.userAgent); Shiny.setInputValue('app-browser\_name', browserName);}$(document).on('shiny:connected', () => { setBrowserName();}); ‍

Build & run again and you will see the element. Try to open the app in Google Chrome and in another browser (e.g. Safari) to confirm that the text indeed changes.

Part 4. Rhino React ComponentsRhino 1.4 introduced the ability to write React components and easily add them to the application. Let’s put together all the knowledge we’ve gathered so far and write a typed, documented React component. You can follow the OG Use React in Rhino tutorial to get familiar with the process.

‍

Let’s make all necessary preparations: delete contents of index.js file, install shiny.react, install react types.

Notice that we need to install type definitions for the React version used by shiny.react:

Rscript -e 'rhino::pkg\_install("shiny.react")'version=$(Rscript -e 'cat(shiny.react::reactDependency()$version)') && npm i -D --prefix .rhino "@types/react@$version" Writing React components means that we are going to write some JSX, so we need to explicitly enable it in jsconfig.json:

{ "compilerOptions": { "checkJs": true, "strict": true, "jsx": "react-jsx", "typeRoots": [ ".rhino/node\_modules/@types/" ] }} ‍

Finally, let’s create a Reveal.jsx file in the app/js folder and copy-paste the component code from the Rhino tutorial:

‍

const { useState } = React;export default function Reveal({ id, children }) { const [visible, setVisible] = useState(false); return ( <div id={id}> <button type="button" onClick={() => setVisible(!visible)}> {visible ? 'Hide' : 'Show'} </button> {visible && children} </div> );} ‍

We will be immediately greeted with a few type errors:

The first error suggests we should import React. However, in this case it is not necessary, as shiny.react ensures there is a global React object available. To silence the error we can simply replace React with window.React.

The next problem is implicit props type. A React component is just a function, so we can use JSDoc to specify the types of its arguments. But first, let’s create a react.d.ts declaration file in the app/js directory. Working with 3rd party types in JSDoc is a bit clunky: often times VS Code won’t recognize types from npm packages, so we can “re-export” them in a local declaration file like this:

// app/js/types/react.d.tstype ReactElement = import('react').ReactElement; ‍

And now we can prepend our component function with a JSDoc comment:

The component looks good, so now let’s register it with Rhino in the index.js file:

‍

/*** A component that allows to toggle content visibility with a button click** @param {{id: string; children: ReactElement}} props** @example* <Reveal id={"test"}>* <div>I'm hidden</div>* </Reveal>** @returns {ReactElement}*/export default function Reveal({ id, children }) {...} ‍

import Reveal from './Reveal';window.Rhino.registerReactComponents({ Reveal }); ‍

TypeScript will now complain that the window doesn’t have a Rhino property. We can fix it by adding a local Window type declaration (why it works):

// app/js/types/window.d.tsinterface Window { Rhino: { registerReactComponents: (components) => void, };} ‍

Now everything should look fine. You can follow the rest of the tutorial from Rhino documentation and make sure that everything works as expected.

Bonus: CypressRhino comes with Cypress end-to-end testing framework which allows you to simulate user behavior by writing test cases in JavaScript. For those not familiar with Cypress it might be a little difficult to write those tests without code completion.

There is a way around this problem – a reference meta tag in the beginning of a JavaScript file:

// app.cy.js/// <reference types="../../../.rhino/node\_modules/cypress/" />describe('app', () => { beforeEach(() => { cy.visit('/'); }); it('starts', () => {});}); Notice how we explicitly tell VS Code where to look for type definitions. With this single line, autocompletion is already there. Very nice indeed! However, this path looks somewhat cumbersome, and what’s more, it will be different in different scripts if you want to structure your test cases in folders.

There is another way: the JSConfig way. All we need to do is update the jsconfig.json file by adding .rhino/node_modules to the typeRoots property. However if you do it, VS Code will basically drown in type errors – because some node_modules don’t have type definitions and those that do might not work with this non-standard approach.

To avoid errors we need to explicitly define for which packages we require types in the types property. Notice that we now also need to include rstudio-shiny – previously it was implicitly imported from @types, but as soon as compilerOptions.types property is defined nothing is imported implicitly.

{ "compilerOptions": { "checkJs": true, "strict": true, "jsx": "react-jsx", "typeRoots": [".rhino/node\_modules/@types/", ".rhino/node\_modules"], "types": ["cypress", "rstudio-shiny"] }} {

Now we should have code suggestions not only for functions and methods but also for values – which is extremely useful in methods like should.

Final ThoughtsI hope this material was helpful and more developers will treat JS in Shiny more seriously. Moreover, I find that adding “cool bits” of JavaScript functionality to a Shiny app is a nice way to start your web development journey and dive deeper into the technologies of the web. Next step – learn how to write code in TypeScript, transpile it into JavaScript and bundle it with a Shiny app.

On a final note, I would like to add a friendly reminder to use global declaration types with discretion. After all, these are only declarations – they may not necessarily coincide with what you get at runtime.

Did you find this blog post useful? Join our community and sign up for our weekly newsletter to get more content like this delivered to your inbox.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Develop Robust and Maintainable JavaScript Code Within a Shiny Application

View Details

[This article was first published on R Archives - Dan Oehm | Gradient Descending, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I wanted a space to throw all my tables and charts made using the {survivoR} R package into, so I started The Sanctuary (built with Quarto, of course). It has interactive tables about the castaways, challenges, voting history, confessionals, episode details, and a bunch more.

While there is a lot of data out there about Survivor it’s rarely all in one place. This provides a view of castaways across seasons and various stats. There won’t be a lot of explanation or in-depth analysis, just a truckload of data, tables, and charts to explore. Longer form posts will remain on the blog.

The Sanctuary is updated regularly during seasons and whenever new data hits Git. It’s the companion for the {survivoR} package.

ExamplesI won’t share too much here, I’ll let you explore for yourself, but here is an idea of what you can find.

Tribal Council Success: Top 10 CastawaysThe score is a measure of how many Tribal Councils they survived, the difficulty of surviving the vote (e.g surviving a Tribal with 4 people is harder than surviving one with 12), and how many votes they copped. Denise, Sandra, and Stephanie take out the top 3 spots.

Full table

Individual Immunity Wins: Top 10 CastawaysThe challenge score is a measure of challenge success. For individual immunity challenges Ozzy takes out the top spot followed by Brad Culpepper in season 34 Games Changers and Mike Holloway in Season 30 Worlds Apart.

Full table

Season IMDb ratings: Top 10 highest rated seasonsThe highest rated season based on IMDb ratings is season 31 Cambodia, the second chance season, followed by season 40 Winners at War and Season 20 Heroes vs Villains. The top 3 are all returnee seasons. The 4th highest rated season is season 7 Pearl Islands, which is also the highest rated all newbie season.

Full table

Confessionals: Top 10 castaways with the most confessionalsRussell Hantz still holds the most number of coveralls in a season, which is going to be hard to beat. The next two are Rob Cesternino and Colby Donaldson to round out the top 3.

Full table

Anyway, expect more things as time goes on.

The post The Sanctuary: Stats and data from {survivoR} appeared first on Dan Oehm | Gradient Descending.

To leave a comment for the author, please follow the link and comment on their blog: R Archives - Dan Oehm | Gradient Descending.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Sanctuary: Stats and data from {survivoR}

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, R community!

Today, we’re jumping into a common yet powerful task in data manipulation: splitting character strings and extracting the first element. We’ll explore how to accomplish this in base R, as well as using the stringi and stringr packages.

Let’s get started!

ExamplesUsing strsplit() in Base RBase R provides the strsplit() function for splitting strings. Here’s a quick look at the syntax:

strsplit(x, split, fixed = FALSE, perl = FALSE, useBytes = FALSE) * x: Character vector to be split. * split: The delimiter (separator) to use for splitting. * fixed: If TRUE, split is interpreted as a string, not a regular expression. * perl: If TRUE, perl-compatible regular expressions can be used. * useBytes: If TRUE, the operation is performed byte-wise rather than character-wise.

Example 1: Splitting a single string string <- "apple,orange,banana"split\_result <- strsplit(string, ",")first\_element <- sapply(split\_result, `[`, 1)print(first\_element)

[1] "apple" Example 2: Splitting a vector of strings strings <- c("apple,orange,banana", "cat,dog,mouse")split\_results <- strsplit(strings, ",")first\_elements <- sapply(split\_results, `[`, 1)print(first\_elements)

[1] "apple" "cat"

Using stringi PackageThe stringi package offers a powerful function stri_split_fixed() for splitting strings. Let’s look at its syntax:

stri\_split\_fixed(str, pattern, n = -1, simplify = FALSE) * str: Character vector to be split. * pattern: The delimiter for splitting. * n: Maximum number of pieces to return. * simplify: If TRUE, returns a matrix.

Example 1: Splitting a single string library(stringi)string <- "apple,orange,banana"split\_result <- stri\_split\_fixed(string, ",")first\_element <- sapply(split\_result, `[`, 1)print(first\_element)

[1] "apple" Example 2: Splitting a vector of strings strings <- c("apple,orange,banana", "cat,dog,mouse")split\_results <- stri\_split\_fixed(strings, ",")first\_elements <- sapply(split\_results, `[`, 1)print(first\_elements)

[1] "apple" "cat"

Using stringr PackageThe stringr package provides str_split_fixed() and str_split() functions. Here’s the syntax for str_split():

str\_split(string, pattern, n = Inf, simplify = FALSE) * string: Character vector to be split. * pattern: The delimiter for splitting. * n: Maximum number of pieces to return. * simplify: If TRUE, returns a matrix.

Example 1: Splitting a single string library(stringr)string <- "apple,orange,banana"split\_result <- str\_split(string, ",")first\_element <- sapply(split\_result, `[`, 1)print(first\_element)

[1] "apple" Example 2: Splitting a vector of strings strings <- c("apple,orange,banana", "cat,dog,mouse")split\_results <- str\_split(strings, ",")first\_elements <- sapply(split\_results, `[`, 1)print(first\_elements)

[1] "apple" "cat"

Your Turn!Now it’s your turn to practice! Try splitting different strings and extracting the first element using base R, stringi, and stringr. Experiment with various delimiters and see how each function handles them.

I look forward to hearing about your experiences with string manipulation in R!

Until next time, happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Split a Character String and Get the First Element in R

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. MotivationI was ready to copy a table with a geometry column from R to PostgreSQL, and I saw this error that wasn’t there yesterday:

Error: Failed to fetch row : ERROR: type "geometry" does not exist at character 108 I ran sudo -i -u postgres; psql -d mydatabase -c "CREATE EXTENSION postgis;", which means to re-activate PostGIS for my particular database, but it said that the extension was not installed, as in the following error message:

ERROR: could not open extension control file "/usr/share/postgresql/10/extension/postgis.control": No such file or directory I tried to re-install PostGIS as I wrote in my notes:

sudo dnf install postgis 25\_12 But I got more errors:

Error: Unable to find a match: postgis25\_12 A further search with sudo dnf list postgis returned No matching Packages to list. It happened that somebody updated the system and some packages were removed in the process in order to avoid conflicts.

Here is how I solved it.

Add PostgreSQL 12 repositoryWhy PostgresSQL 12? Because of the following dependency problems:

RHEL 8 offers PostgreSQL 10, which works with PostGIS 2.4, which in turn depends on Armadillo 9.x.
RHEL 8 offers Armadillo 12.x, which is incompatible with PostGIS 2.4.
The official PostgreSQL repositories for newer PostgreSQL versions (such as 15) lead to more unmet dependencies.
The oldest PostgreSQL available in the PostgreSQL repository is 12, which is compatible with PostGIS 3.4, that in turn is compatible with Armadillo 12.x.

I typed cat /etc/redhat-release to check the version of RHEL I was using, and it was 8.10. In my previous notes I had 8.4. Because of this I typed sudo nano /etc/yum.repos.d/pgdg.repo and added the following lines:

[pgdg12]name=PostgreSQL 12 for RHEL8baseurl=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8.10-x86\_64enabled=1gpgcheck=0[pgdgextras12]name=PostgreSQL Extras for RHEL8baseurl=https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-8.10-x86\_64/enabled=1gpgcheck=0 Then I updated the repositories with:

sudo dnf clean allsudo dnf makecache Then I ran:

sudo dnf remove postgres*sudo dnf install postgis34\_12 The last step installed the following packages without conflicts:

Installed: CGAL-4.14-1.rhel8.x86\_64 SFCGAL-1.4.1-13.rhel8.x86\_64 SFCGAL-libs-1.4.1-13.rhel8.x86\_64 gdal38-libs-3.8.5-3PGDG.rhel8.x86\_64 gmp-c++-1:6.1.2-11.el8.x86\_64 gpsbabel-1.6.0-3.el8.x86\_64 libarrow-8.0.1-2.el8.x86\_64 libdeflate-1.9-3.el8.x86\_64 libgeotiff17-1.7.1-6PGDG.rhel8.x86\_64 libgeotiff17-devel-1.7.1-6PGDG.rhel8.x86\_64 libqhull\_r-2015.2-5.el8.x86\_64 librttopo-1.1.0-2.rhel8.x86\_64 libspatialite50-5.1.0-5PGDG.rhel8.x86\_64 libspatialite50-devel-5.1.0-5PGDG.rhel8.x86\_64 libusb-1:0.1.5-12.el8.x86\_64 postgis34\_12-3.4.2-3PGDG.rhel8.x86\_64 postgresql12-12.19-3PGDG.rhel8.x86\_64 postgresql12-contrib-12.19-3PGDG.rhel8.x86\_64 postgresql12-libs-12.19-3PGDG.rhel8.x86\_64 postgresql12-server-12.19-3PGDG.rhel8.x86\_64 proj94-9.4.0-1PGDG.rhel8.x86\_64 qt5-qtsvg-5.15.3-2.el8.x86\_64 re2-20190801-1.el8.x86\_64 shapelib-1.5.0-12.el8.x86\_64 I checked with psql --version that returned psql (PostgreSQL) 12.19.

Activating PostgresSQLAfter running sudo -i -u postgres; psql -d mydatabase I got the following message:

psql: error: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? I typed the following commands to start the server:

sudo postgresql-12-setup initdbsudo systemctl start postgresql-12 That made sudo -i -u postgres; psql work.

Creating a new databaseI created a new database with the following commands:

CREATE DATABASE mydatabase;\q Then after typing psql -d mydatabase I ran:

CREATE ROLE student;ALTER ROLE student WITH LOGIN;ALTER ROLE student WITH ENCRYPTED PASSWORD 'SomePassword';GRANT CONNECT ON DATABASE mydatabase TO student;GRANT USAGE ON SCHEMA public TO student;GRANT SELECT ON ALL TABLES IN SCHEMA public TO student;GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO student;ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO student;GRANT CREATE ON SCHEMA public TO student; Connecting from RI ran:

con <- dbConnect( drv = Postgres(), dbname = Sys.getenv("POSTGRES\_DB"), user = Sys.getenv("POSTGRES\_USR"), password = Sys.getenv("POSTGRES\_PWD"), host = Sys.getenv("POSTGRES\_HOST")) That gave me the following error:

Error: connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "student" One way to fix it is to type sudo nano /var/lib/pgsql/12/data/pg_hba.conf and change these lines:

```

"local" is for Unix domain socket connections onlylocal all all peer# IPv4 local connections:host all all 127.0.0.1/32 md5 # IPv6 local connections:host all all ::1/128 md5

`` Where the change is to replaceident/peerwithmd5` in the IPv4/6 lines.

Then I ran sudo systemctl restart postgresql-12 and the connection worked.

Installing PostGISAfter running sudo -i -u postgres; psql -d mydatabase I ran:

CREATE EXTENSION postgis; PostGIS also required sudo systemctl restart postgresql-12.service or then R pretends to write properly, but when you read the table back from SQL, the geometry column is of type pq_NA and of binary type instead of sfc_geometry.

Finally, I was able to write the table with the geometry column from R to PostgreSQL and read it back with the proper geometry column type.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Install PostGIS on Red Hat Enterprise Linux (RHEL) 8

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium recently interviewed Victor Lee, organizer of the Korea R User Group, about his role establishing and expanding the Korean R community. Victor shared his journey, beginning with an introduction to R and open source programming languages while working at the Hyundai Motor Company, and later, his efforts in establishing the tidyverse community in Korea. He highlighted his extensive experience with R, including writing blog posts, publishing Quarto books, and building websites for the Korea R User Group. Victor will be a Software Carpentry instructor at the Software Carpentry Workshops at Sejong University.

Please share about your background and your involvement in the R Community.

My first introduction to our community was about 10 years ago, and it wasn’t a good experience. I used to work at the Hyundai Motor Company at that time and was intrigued by the software carpentry led by Greg Wilson. I also delved into statistics and open-source programming languages, particularly S and R programming. I was heavily involved in posting about tidyverse, which was my entry point into the community environment. In Korea, I sought out the Korean community, which mainly focused on the basics. This made me realize the need for a community in Korea based on tidyverse principles, and that’s why I started the tidyverse community in Korea 10 years ago.

I was first introduced to S-PLUS during my undergraduate years as a statistics major, and I was fascinated by its superior graphics compared to SAS/SPSS. After majoring in computer engineering and working at Hyundai Motor Company for 10 years, I obtained a Software Carpentry Instructor certification and translated “Python for Informatics” into a Korean book. I became captivated by the Hadleyverse, and Since 2016, I have been co-organizing the Seoul R Meetup, sponsored by Kyobo DPLANEX (a continuous sponsor and is currently the largest sponsor of the Seoul R Meetup, representing one of South Korea’s leading insurance companies) alongside Choonghyun Ryu, the founder of the Korea R User Group. In 2021, we hosted the Korea R Conference, and in 2021, we established the Korea R User Group as a non-profit organization, transitioning from a community to an official organization.

What is your level of experience with the R language?

With the support of the R community, ChatGPT, and Copilot AI, I now confidently tackle any data science problem using R. For about 10 years, I’ve consistently written blog posts using R Markdown and now Quarto. Upgrading my e-books with Bookdown led to the publication of five Quarto books on data science. Using the Quarto framework, I also built the Korea R User Group and R Conference websites. As a civic data journalist, I’ve written around 100 articles utilizing R’s visualization capabilities. Reflecting on my journey, I see how effectively I’ve applied the R language in various fields.

What industry are you currently in? How do you use R in your work?

I originally set up the Korean R community 10 years ago and am a founding member of the nonprofit Korea R User Group, established three years ago. I left KPMG to dedicate my time to running the Korea R User Group. This year, I have been fully involved in managing the organization and leading several projects, including two major abandoned projects, focusing on them for the past few months.

Currently, I am focusing on publishing and developing open statistical packages at a non-profit public interest corporation. In 2020, with good intentions, I started the “Open Statistical Package” project to independently develop statistical packages like SAS, SPSS, and Minitab. However, some Shiny developers without a strong background in statistics took the project in their direction, causing it to lose steam. It felt as though they had hijacked the project and the hard work the Korea R User Group put in, leaving us frustrated and disappointed.

To prevent this kind of thing from happening again, we’re beefing up our license policy, including trademark registration for BitStat[1]. We’re also switching up our development engines to webr and shinylive and are in the process of creating BitStat2[2].

[1]: https://github.com/bit2r/BitStat[2]: https://github.com/bit2r/BitStat2We also established a publishing company named “BitStat” as the Korea R User Group promoted Quarto digital writing as a new open source project. Recently, we have published and released five data science books, expanding the base of R users. While writing the sixth book on probability and statistics, I restarted the development of open statistical packages using Web-R and Shinylive.

R has evolved from a simple data analysis and statistical language to a tool that can replace office software. I now use Quarto to create almost all documents, and R is the first language I use in developing the open statistical package that I am currently working on.

Why do industry professionals come to your user group? What is the benefit for attending?

In Korea, about 20 to 30 years ago, R was the number one programming language for data science and statistics, particularly in areas like machine learning. However, with the rise of Python, many R users transitioned to Python due to its increasing popularity. Despite this shift, R remains significant in Korea, with many people continuing to use both R and Python.

For my day-to-day work, I find R quite convenient and easy to use, especially for therapeutic data and open-source case studies. This year, I’ve noticed that users who join the Korea R User Group come from diverse backgrounds, including drug discovery, regulatory agency, and real estate.

Over the past decade, many users joined the group to determine whether Python or R was better suited for their work. However, the recent trend clearly leans towards artificial intelligence development, such as LLM (Large Language Model) development. Participants from various industries with an interest in quantitative analysis are now attending the user group.

Their motivation for attending, apart from AI fields represented by LLM, is to acquire the latest technology in other data science areas and to gain knowledge from diverse, in-depth analysis experiences and model development. Additionally, many people come to obtain information about Quarto, ggplot, gt, and shiny, seeking business opportunities related to these tools.

What trends do you currently see in R language and your industry? Any trends you see developing in the near future?

This year, our community in Korea is focusing on Quarto due to upcoming government policy changes. Analog methods are expected to disappear within five years, so the government is funding the development of AI digital textbooks. I believe Quarto technology, the next generation of R Markdown, is perfect for this purpose.

As generative artificial intelligence (AI) has gained significant attention in Korea, there is growing interest in using R and Python together with generative AI to solve data science problems and increase productivity, rather than focusing on the languages themselves. When using generative AI with languages such as R, Python, and SQL, it becomes necessary to find tools that can automate and store the outputs, inevitably leading to increased interest in Quarto.

This perspective has been reinforced by my experience using Quarto in various ways, starting from R Markdown. I have come to realize that Quarto is truly well-suited for generative AI and data science. If other countries are developing AI texts using Quarto or R Markdown, we could introduce this technology to the Korean market and the Korean government.

Having written five books – plus a sixth on probability and statistics – I’ve experimented with various features of Quarto books. I’ve realized we no longer need older statistical packages like SAS and SPSS. My current project involves implementing statistical software using WebAssembly (WASM) technology.

We would like to get to know you more personally. Can you please tell me about yourself? For example, your hobbies/interests or anything else you want to share.

Initially, I wasn’t sure if I would succeed, but I became involved in election campaigns and grew passionate about analyzing political and election data. My interest lies in using data to uncover trends and insights from various social datasets.

Next month, we will have a data journalism meetup, and I have friends who will join because of the articles I wrote. They will showcase some of their analyses on TV, including summaries of data related to election campaigns.

I first developed a connection with data while majoring in statistics and then pursued computer engineering in graduate school. Although this combination of backgrounds is common now, it was unusual in Korea at the time, giving me a unique career path. My passion for open-source software and faith in the community have driven me to where I am today.

I enjoy analyzing data, and whenever I come across interesting datasets, I analyze them and document my experiences on my blog. This hobby, along with the copyright-free nature of data, led me to develop an interest in predicting election winners using data from annual elections in South Korea. Since 2016, I have experienced three general elections, presidential elections, and local elections. Although there won’t be an election next year, I am very much looking forward to the next one.

Learn moreThe post Full-time Korea R User Group Founder Victor Lee Sees AI Future for R and Quarto Textbooks appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Full-time Korea R User Group Founder Victor Lee Sees AI Future for R and Quarto Textbooks

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#187–188

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #187There are two ways of showing data and it really depends on what is the purpose of certain presentation. Sometimes we only need to show those data point that have any data in it, but sometimes we need to make something like empty data point, which means that we need to include in our report all needed dimension values, all labels, but still have no value or value replaced by 0 for this row.

And that is what the task today is about. We have sales values for different continents in different years, but we don’t have cases where sales were 0. But we need to restructure report to have it. With some twists because we need summary rows and empty rows, and… North America is abbreviated to NA, and you know what could it cause in data. Check my solution.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_187.xlsx", range = "A1:C12")test = read\_excel("Power Query/PQ\_Challenge\_187.xlsx", range = "E1:G30") Transformation all <- expand\_grid(Continent = unique(sort(input$Continent)), Year = unique(sort(input$Year)))result1 <- all %>% left\_join(input, by = c("Continent", "Year")) %>% mutate(Sales = replace\_na(Sales, 0), Year = as.character(Year))years <- unique(sort(result1$Year))empty\_row <- tibble(Continent = NA, Year = NA, Sales = NA\_real\_)totals <- map\_dfr(years, ~ { yearly\_data <- result1 %>% filter(Year == .x) total\_row <- summarise(yearly\_data, Continent = "TOTAL", Year = .x, Sales = sum(Sales)) bind\_rows(yearly\_data, total\_row, empty\_row)})grand\_total <- summarise(result1, Continent = "GRAND TOTAL", Year = "2010-2013", Sales = sum(Sales))result <- bind\_rows(totals, grand\_total) Validation identical(result, test)# [1] TRUE Puzzle #188Sometimes we are missing some dimension of data, because they are just aggregated into bigger sets. And we have it here. Sales were summarized by period, very irregular period. And we need that data per quarter. As we don’t know how exactly sales went, we need to calculate how many days of each quarters there was sales and assign money proportionally. Let dig in it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_188.xlsx", range = "A1:D4")test = read\_excel("Power Query/PQ\_Challenge\_188.xlsx", range = "F1:H11") Transformation result = input %>% mutate(date = map2(`From Date`, `To Date`, seq, by = "day"), days = map\_int(date, length), daily = Amount / days) %>% unnest(date) %>% mutate(quarter = quarter(date), year = year(date) %>% as.character() %>% str\_sub(3, 4), Quarter = paste0("Q",quarter,"-",year)) %>% summarise(Amount = sum(daily) %>% round(0), .by = c(Store, Quarter)) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Technology stocks have risen sharply from the second half of the previous year. Perhaps the rumors of the FED probable rate cuts affected that, but I want to look closer at it. I will examine this situation with a fund based mostly on NASDAQ-100 which is known heavily for technology stocks, and S&P Global 1200 Information Technology index.

library(tidyverse)library(timetk)library(tidyquant)#İş Asset Technology Mixed Fund (ITP)#(https://www.investing.com/etfs/tryispo00621)df\_itp <- read\_csv("https://raw.githubusercontent.com/mesdi/blog/main/itp.csv")#Tibble objectdf\_itp\_tbl <- df\_itp %>% mutate(Date = parse\_date(Date, "%m/%d/%Y")) %>% janitor::clean\_names() %>% select(date, itp = price) %>% arrange(date)#Federal Funds Effective Ratedf\_fedfunds <- tq\_get("FEDFUNDS", get = "economic.data") %>% #Low to high frequency pad\_by\_time(date, .by = "day") %>% tidyr::fill(price, .direction = "up") %>% select(date, fedfunds = price) %>% #adding days to match the df\_itp mutate(date = date + days(57)) #Merging all the data setsdf\_merged <- df\_itp\_tbl %>% left\_join(df\_fedfunds) I’ve added the FED effective rates to my model to improve the accuracy. Now, I will construct a causal inference model using a Bayesian time series structure. I will do that with CausalImpact package.

To do that, I set an intervention that probably affected the response variable (fund price). On March 30, 2023, the Federal Trade Commission (FTC) took action against several companies that used AI to create and distribute misleading advertisements. I will use this date as a pre and post-treatment period.

```

Zoo object for the CausalImpact functiondf_merged_zoo <- with(df_merged, zoo(cbind(itp, fedfunds), date))#The pre-period and the post-period data setspre.period <- as.Date(c("2021-04-07", "2023-03-30"))post.period <- as.Date(c("2023-03-31", "2024-05-28"))#Causal effectlibrary(CausalImpact)impact <- CausalImpact(df_merged_zoo, pre.period, post.period)#Summarized resultssummary(impact)

``` As seen in the above results, the relative effect shows that 66% increase which means the observed values are 66% higher than the counterfactual values which denotes how the response variable would have evolved if the intervention had never occurred. The result is significant because 95% confidence intervals(CI) exclude 0.

Finally, we will draw a plot showing the results we mentioned above.

```

Plotting the effectslibrary(ggtext)#Text box data frame for the plotdf_textbox <- data.frame( x = c(as.Date("2023-03-30"), as.Date("2023-11-01")), y = c(5,3), label = c("Actions against deceptive AI marketing", "Actions against unfair credit scoring algorithms"), family = "Bricolage Grotesque")plot(impact, c("original")) + geom_vline(xintercept = as.Date("2023-11-01"), linetype = "dashed", color = "#a9a9a9", linewidth = 0.8)+ geom_textbox(data = df_textbox, aes(x = x, y = y, label = label, family = family), width = unit(1.2, "inch"))+ labs(title = "Increment of +66% in the ITP due to the actions taken by the FTC") + theme(plot.title = ggtext::element_markdown(face = "bold", hjust = 0.5, size = 15), text = element_text(family = "Bricolage Grotesque"))

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Technology Stocks Surge: Causal Impact of FTC Actions

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionGreetings, fellow data enthusiasts! Today, we’re thrilled to unveil a fresh wave of functionalities in the ever-evolving TidyDensity package. Buckle up, as we delve into the realm of distribution statistics!

This update brings a bounty of new functions that streamline the process of extracting key parameters from various probability distributions. These functions adhere to the familiar naming convention util_distribution_name_stats_tbl(), making them easily discoverable within your R workflow.

Let’s meet the newcomers:

util_zero_truncated_negative_binomial_stats_tbl(): Uncovers the secrets of the zero-truncated negative binomial distribution.
util_zero_truncated_poisson_stats_tbl(): Demystifies the zero-truncated Poisson distribution.
util_zero_truncated_geometric_stats_tbl(): Unveils the hidden characteristics of the zero-truncated geometric distribution.
util_pareto1_stats_tbl(): Extracts the essence of the Pareto Type I distribution.
util_paralogistic_stats_tbl(): Unlocks the mysteries of the paralogistic distribution.
util_inverse_weibull_stats_tbl(): Illuminates the parameters of the inverse Weibull distribution.
util_inverse_pareto_stats_tbl(): Provides insights into the inverse Pareto distribution.
util_inverse_burr_stats_tbl(): Offers a glimpse into the world of the inverse Burr distribution.
util_generalized_pareto_stats_tbl(): Simplifies extracting parameters from the generalized Pareto distribution.

Now, you might be wondering, “How do I put these new functions to use?” Fear not, for the answer is as easy as pie!

ExamplesLet’s explore the zero-truncated binomial distribution. Suppose we’re simulating the number of successes in 10 trials with a success probability of 0.1 (but hey, successes of zero aren’t possible in this scenario!).

library(dplyr)library(TidyDensity) # Assuming you've installed TidyDensityset.seed(123)tidy\_zero\_truncated\_binomial(.size = 10, .prob = 0.1) |> util\_zero\_truncated\_binomial\_stats\_tbl() |> glimpse()

Rows: 1Columns: 15$ tidy\_function <chr> "tidy\_zero\_truncated\_binomial"$ function\_call <chr> "Zero Truncated Binomial c(10, 0.1)"$ distribution <chr> "Zero Truncated Binomial"$ distribution\_type <chr> "discrete"$ points <dbl> 50$ simulations <dbl> 1$ mean <dbl> 1.58$ mode <dbl> 1$ range <chr> "1 to 4"$ std\_dv <dbl> 0.8103917$ coeff\_var <dbl> 0.5129061$ computed\_std\_skew <dbl> 1.133051$ computed\_std\_kurt <dbl> 3.212143$ ci\_lo <dbl> 1$ ci\_hi <dbl> 3 This code snippet generates a dataset of zero-truncated binomial values and then utilizes the util_zero_truncated_binomial_stats_tbl() function to extract a summary table containing key parameters like the mean, variance, and quantiles.

Your Turn to Explore!We encourage you to jump in and experiment with these new additions. Explore the documentation for each function (accessible through ?util_distribution_name_stats_tbl) to discover their specific functionalities and supported distributions.

With these new tools at your disposal, you’ll be well-equipped to gain deeper insights into your data and unlock the power of various probability distributions in your R adventures!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Unveiling New Tools in the TidyDensity Arsenal: Distribution Parameter Wrangling

View Details

[This article was first published on Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Two roads diverged in a wood and I, I took the one less traveled by, and that has made all the difference.

A little-known historical tidbit is that Robert Frost’s The Road Less Traveled – so often cited as a celebration of individuality and difficult choices – was in fact meant as a joke to tease an indecisive friend. Frost’s intent was to be ironic; to make fun of someone who is overly dramatic looking back on their choices. Come on buddy, he says, just pick a path – they are both beautiful and will get you somewhere interesting.

Bro, just pick one, they probably both end up at a Starbucks anyways.I bring this up because, like in the poem, I believe we in the R community often overdramatize moments of divergence between different R dialects, packages, and syntaxes.

And I, like poor Robert Frost, also feel there may be some misunderstanding around the my intent here.

So to make sure my opinions are loud and clear, we’ll get some help from this cartoon lady to shout out the things I most want y’all to hear from me:

I have a lot of opinions about R!

This blog post is my attempt to dispel the myths surrounding the relationship between data.table and the tidyverse, and to explain why I believe deeply in both.

“Two roads diverged in a yellow wood, and sorry I could not travel both…”A bit of history to kick us off. (I know, I know, I’ll keep it short.)

The first official 1.0 version of R was released February 29, 2000, and if I had to guess, I’d bet the first add-on package was created the next day. The beating heart of R is the base language, that the R Core Team has lovingly and diligently maintained for over 4 decades – but its soul, if you will, is the incredible collection of packages that expand and adapt this core.

data.table was released in 2008 by Matt Dowle. (See this video for a very cool recap of the inspiration and process from Matt himself!) Since then, it has grown enormously in scope, contributors, user base, and dependencies.

In 2014, dplyr was released by Hadley Wickham, the birth of what we now know as the tidyverse.

This meant that users now had several options to pick from if they wanted to, say, calculate means by group:

```

Base Raggregate(bill_length_mm ~ species, data = penguins, mean)

```

data.tablepenguins_dt <- data.table(penguins)penguins_dt[, .(mean_bill=mean(bill_length_mm)), by=species]

```

dplyrpenguins %>% group_by(species) %>% summarize(mean_bill = mean(bill_length_mm))

`` **data.table` is not technically Base R!**

It is true that the data.table syntax most closely mimics that of Base R data frames, and deliberately so. However, data.table is an open-source package like any other. Nobody - and I mean nobody - uses only Base R in their work. What a silly culture that would be, if we have all the beautiful multiverse of an open-source language, and we limit ourselves only to the core functionality!

I don’t even remember what this ad was for.“… looked down one as far as I could … then took the other, just as fair…”So: We have multiple dialects. How to choose which one to use?

There is no single right answer to that question; it’s all a matter of individual preference and use case. What do you, as the programmer, value most? Brevity of code? Readability of code? Speed? Familiarity? Consistency with collaborators? Availability of learning resources?

I could go on - there are infinitely many reasons, from the personal to the professional to the practical, to choose one path or the other. Sometimes, the answer is as simple as, “This is the way that I know how to do it.”

I can’t tell you how to pick what works for you.

“Dialect” or syntax choices in R are contextual and case-by-case, not lifetime commitments!

The idea of “loyalty” to a package is nonsense. A package is a tool. You might have admiration, respect, or even loyalty to a package developer; you might even therefore trust that its worth your time and energy to follow their recommendations.

But if you start feeling bad when you sprinkle a little Base R into your tidy workflow… if you are ashamed for piping a data.table object into ggplot… well that’s getting us nowhere, is it? We are blessed with an overabundance of useful tools and we shouldn’t be limiting ourselves!

```

Great news! This is not illegal!penguins %>% data.table() %>% .[, .(mean_bill=mean(bill_length_mm)), by=species]

``` We all use our own favorite collection of packages, in the combinations that work for us. It might be fun to discuss and learn about new options or new preferences, but no more purity culture, please!

It’s okay to use different syntaxes and package styles all in one workflow!

“…I shall be telling this with a sigh, somewhere ages and ages hence…”It has come to the point where I can’t avoid mentioning what mainly motivated this blog post: The Great Twitter War of 2018. (Please read that sentence with every ounce of irony you have in you).

Briefly for those who weren’t “lucky” enough to be in the tweetstorm: Sometime around 2018, the #rstats Twitter community exploded into a debate about the relative merits of the tidyverse, data.table, and Base R.

It was basically a lot of this. (Source: XKCD#386)It’s sad to me that the community seems to remember this time as a fight, because so much of that conversation was productive and interesting. Educators shared their experiences teaching with different dialects. Developers talked about the speed trade-offs of the various options. New users were excited to be exposed to information about their options.

But - as seems to be the norm on the internet - a vocal subset of this conversation took the form of an “us vs. them” debate, and weird lines were drawn between data.table/Base R and the tidyverse.

RabblerabblerabbleIt’s important to note that the primary developers themselves - Hadley Wickham and Matt Dowle - were not the cause of the drama. In fact, this good conversations from this Twitter whirlwind lead to the creation of on of my favorite packages, dtplyr!

So why am I partially digging up a buried hatchet?

Because sadly, even today, I sometimes run into vitriol when I post on social media about data.table or the tidyverse, and I know I’m not alone in this.

Even today, I have my college students asking me about the rift in the R world, and if they have to “choose a side” to learn R.

And most relevant to this blog - I have gotten a lot of questions about why I am involved in a data.table project, since I’m “supposed” to be Team Tidyverse.

Therefore, to be ultra clear:

This grant is NOT about helping data.table “beat” dplyr.

This could not be further from the truth! I’m a tidyverse girlie - from my dplyr earrings to my hex fabric shirts - and I am also a data.table girlie. I, personally, would not be working on this project if I thought anyone involved viewed it as anti-tidyverse in any way.

What we want is the same thing any open-source fan wants:

We want users to be aware of the many fantastic tools, including data.table, that exist in the R world.
We want developers to be inspired to build new and exciting packages, that stand on the shoulders of giants like data.table, the tidyverse, and so many others.
We want beloved packages like data.table to stick around long term, and to grow and evolve with R and the R community.

“… and that has made all the difference.”So, where are we going from here, as a community? Only good places, I think, no matter which path we take in the yellow wood!

I am so excited about this project and about the NSF-POSE grant - both for the longevity of data.table, and for everything we are learning about open-source ecosystems and how to sustain them.

I love the #rstats community!!! Let’s do cool stuff together.

AddendumWant to hear me rant more about the R community, multiple dialects/languages, and this grant project? I’ll be speaking on these topics at UseR!2024, JSM, and Posit::conf - or you can always find me on BlueSky or Fosstodon!

Testing infrastructure for data.tableMar 10, 2024Toby HockingCommunity interviews about {data.table}Mar 6, 2024Anirban ChetiaResults of the 2023 surveyFeb 25, 2024Aljaž SlugaColumn assignment and reference semantics in {data.table}Feb 18, 2024Toby HockingThe Benefits of data.table SyntaxFeb 5, 2024Tyson BarrettNew governance, release with new featuresJan 30, 2024Toby Dylan HockingPiping data.tablesJan 28, 2024Elio CampitelliAnnouncement: Jan Gorecki, data.table AmbassadorJan 14, 2024Kelly BodwinAnnouncement: Jan Gorecki, data.table AmbassadorJan 14, 2024Kelly BodwinSummary of LatinR conferenceNov 19, 2023Toby Dylan HockingAnnouncement: The data.table Ambassadors Travel GrantNov 1, 2023Kelly BodwinAnnouncement: data.table translation projectsOct 17, 2023Welcome to the {data.table} ecosystem project!An NSF-POSE funded venture.Oct 15, 2023Toby HockingNo matching items To leave a comment for the author, please follow the link and comment on their blog: Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Two Roads Diverged

View Details

[This article was first published on ouR data generation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.One of my goals for the simstudy package is to make it as easy as possible to generate data from a wide range of data distributions. The recent update created the possibility of generating data from a customized distribution specified in a user-defined function. Last week, I added two functions, genDataDist and addDataDist, that allow data generation from an empirical distribution defined by a vector of integers. (See here for how to download latest development version.) This post provides a simple illustration of the new functionality.

Here are the libraries needed, in case you want to follow along:

library(simstudy)library(data.table)library(ggplot2)set.seed(1234) The target density is simply defined by specifying a vector that is intended to loosely represent a data distribution. We start by specifying the vector (which can be of any length):

base\_data <- c(1, 2, 2, 2, 2, 2, 2, 3, 3, 4, 4, 5, 6, 6, 7, 8, 9, 9, 9, 10, 10, 10, 10, 10) We can look at the density to make sure this is the distribution we are interested in drawing our data from:

emp\_density <- density(base\_data, n = 10000)den\_curve <- data.table(x = emp\_density$x, y = emp\_density$y)ggplot(data = den\_curve, aes(x = x, y = y)) + geom\_line(linewidth = 1) + scale\_y\_continuous(name = "density\n", limits = c(0, 0.11), breaks = seq(0, .10, by = .02)) + scale\_color\_manual(values = colors) + theme(panel.grid = element\_blank()) Actually drawing samples from this distribution is a simple call to genDataDensity. The key argument is the data distribution as as represented by the vector of integers:

dx <- genDataDensity(10000, dataDist = base\_data, varname = "x1") Here’s a look at the sampled data and their relationship to the target density:

ggplot(data = dx, aes(x=x1)) + geom\_histogram(aes(y = after\_stat(count / sum(count))), binwidth = 1, fill = "grey", color = "black", alpha = .2) + geom\_line(data = den\_curve, aes(x = x, y = y), color = "black", linewidth = 2) + scale\_y\_continuous(name = "density\n", limits = c(0, 0.11), breaks = seq(0, .10, by = .02)) + scale\_x\_continuous(limits = c(-6, 15), breaks = seq(-5, 10, by = 5)) + theme(panel.grid = element\_blank(), plot.title = element\_text(face = "bold", size = 10)) Just to show that this was not a fluke, here are three additional target distributions, specified with three different vectors:

base\_data <- list( c(1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 9, 10), c(1, 2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 5, 6, 6, 7, 7, 7, 8, 9, 10, 10, 10, 10, 10), c(1, 2, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 10)) We can generate data from each of the distributions and then confirm that each one adequately fits its target distribution:

dx1 <- genDataDensity(10000, dataDist = base\_data[[1]], varname = "x1")dx2 <- genDataDensity(10000, dataDist = base\_data[[2]], varname = "x1")dx3 <- genDataDensity(10000, dataDist = base\_data[[3]], varname = "x1") Addendum: code to generate multiple distribution plotHere is a little more detail in case someone might find it useful to have the code that generates the “facet” plot. In the plot with the single distribution, I specified the histogram with this command:

geom\_histogram(aes(y = after\_stat(count / sum(count))), ...) When I tried to apply this to the “facet” plot, the denominator of that plot (sum(count)) was not calculated for each subgroup (i.e., dataset), but was the total across all datasets. As a result, the dataset-specific proportions were underestimated; we can see that here:

dx <- rbindlist(list(dx1, dx2, dx3), idcol = TRUE)ggplot(data = dx, aes(x=x1)) + geom\_histogram( aes(y = after\_stat(count / sum(count)), fill = factor(.id), color = factor(.id)), binwidth = 1, alpha = .2) + geom\_line(data = dens, aes(x = x, y = y, color = factor(.id)), linewidth = 2) + xlab("\nx1") + ylab("density\n") + scale\_fill\_manual(values = colors) + scale\_color\_manual(values = colors) + theme(panel.grid = element\_blank(), plot.title = element\_text(face = "bold", size = 10), legend.position = "none") + facet\_grid(~ .id) I looked around for a way to address this, but couldn’t find anything that obviously addressed this shortcoming (though I am convinced it must be possible, and I just couldn’t locate the solution). I considered using ggarrangeor something similar, but was not satisfied with the results. Instead, it turned out to be faster just to calculate the proportions myself. This is the process I used:

First, I created a dataset with the bins (using a bin size of 1):

cuts <- seq(dx[,floor(min(x1))], dx[,ceiling(max(x1))], by = 1)dcuts <- data.table(bin = 1:length(cuts), binlab = cuts)dcuts## bin binlab## <int> <num>## 1: 1 -3## 2: 2 -2## 3: 3 -1## 4: 4 0## 5: 5 1## 6: 6 2## 7: 7 3## 8: 8 4## 9: 9 5## 10: 10 6## 11: 11 7## 12: 12 8## 13: 13 9## 14: 14 10## 15: 15 11## 16: 16 12## 17: 17 13## 18: 18 14 Then, I allocated each observation to a bin using the cut function:

dx[, bin := cut(x1, breaks = cuts, labels = FALSE)]dx <- merge(dx, dcuts, by = "bin")dx## Key: <bin>## bin .id id x1 binlab## <int> <int> <int> <num> <num>## 1: 1 1 1251 -2.097413 -3## 2: 1 1 2215 -2.580587 -3## 3: 1 1 2404 -2.042049 -3## 4: 1 1 3228 -2.078958 -3## 5: 1 1 5039 -2.055471 -3## --- ## 29996: 17 3 7690 13.290347 13## 29997: 17 3 8360 13.083991 13## 29998: 17 3 8860 13.149421 13## 29999: 17 3 9214 13.043727 13## 30000: 17 3 9743 13.199752 13 Finally, I calculated the distribution-specific proportions (showing only the second distribution):

dp <- dx[, .N, keyby = .(.id, binlab)]dp[, p := N/sum(N), keyby = .id]dp[.id == 2]## Key: <.id>## .id binlab N p## <int> <num> <int> <num>## 1: 2 -3 3 0.0003## 2: 2 -2 38 0.0038## 3: 2 -1 130 0.0130## 4: 2 0 340 0.0340## 5: 2 1 619 0.0619## 6: 2 2 938 0.0938## 7: 2 3 1161 0.1161## 8: 2 4 1155 0.1155## 9: 2 5 1035 0.1035## 10: 2 6 882 0.0882## 11: 2 7 828 0.0828## 12: 2 8 861 0.0861## 13: 2 9 822 0.0822## 14: 2 10 641 0.0641## 15: 2 11 384 0.0384## 16: 2 12 140 0.0140## 17: 2 13 23 0.0023 And now the facet plot will work just fine. Here is the code and the plot (again).

ggplot(data = dp, aes(x = binlab, y = p)) + geom\_bar(aes(fill = factor(.id), color = factor(.id)), stat = "identity", alpha = .4) + geom\_line(data = dens, aes(x = x, y = y, color = factor(.id)), linewidth = 2) + xlab("\nx1") + ylab("density\n") + scale\_fill\_manual(values = colors) + scale\_color\_manual(values = colors) + theme(panel.grid = element\_blank(), plot.title = element\_text(face = "bold", size = 10), legend.position = "none") + facet\_grid(~ .id) To leave a comment for the author, please follow the link and comment on their blog: ouR data generation.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: simstudy: another way to generate data from a non-standard density

View Details

[This article was first published on R on Nicola Rennie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.When you use colour to denote the values of a variable in a visualisation, it’s very common to add a legend showing how the colours map to different values. If you create your charts using {ggplot2}, a legend is added automatically when you add colour or fill within the aesthetic mapping.

One problem with these legends is that they take up a lot of space – space where we could be plotting data instead! An alternative to using a traditional legend, is using coloured text within a subtitle or annotation.

Note: this approach works well when the colours are used for categorical data, but not so well for continuous colour scales.

As an example, let’s go back to our favourite animal dataset. No, not penguins – though they are a close second! We’re looking at lemur data from Duke Lemur Center. The data was used as a#TidyTuesday dataset back in 2021 so we can load it in from the CSV file on GitHub:

| 1 | lemurs <- readr::read\_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-08-24/lemur\_data.csv") |

Alternatively, you could load it using the{tidytuesdayR} R package.

I’ve used this data several times for teaching, and in aprevious blog post about using both R and Python in Quarto documents.

Let’s do a quick bit of data wrangling using {dplyr}, by filtering to look at only adult,collared brown lemurs, keeping only the columns we need (weight, age, and sex), renaming thos columns with slightly nicer looking variable names, and recoding the "M" and "F" values in Sex as "male" and "female":

| 1 2 3 4 5 6 7 8 9101112131415161718 | library(dplyr)lemur\_data <- lemurs |> filter( taxon == "ECOL", age\_category == "adult" ) |> select(c(age\_at\_wt\_mo, weight\_g, sex)) |> rename( Age = age\_at\_wt\_mo, Weight = weight\_g, Sex = sex ) |> mutate( Sex = case\_when( Sex == "M" ~ "male", Sex == "F" ~ "female", ) ) |

Click here for a bonus lemur photo!
Image:unsplash.com/photos/gray-and-white-cat-on-brown-floor-6OFUMVoRdq4

Now we can create a simple scatter plot using geom_point() in {ggplot2}, by mapping Age onto the x axis, Weight onto the y axis, and colouring the points based on Sex:

| 1234 | library(ggplot2)base\_plot <- ggplot(data = lemur\_data) + geom\_point( mapping = aes(x = Age, y = Weight, colour = Sex)) |

Let’s also add a title and subtitle:

| 12345 | base\_plot + labs( title = "How much does a collared brown lemur weigh?", subtitle = "This scatter plot shows how the weights of male and female lemurs change with age." ) |

This looks fine. But let’s use some nicer colours, and use coloured text instead of the legend.

Coloured text with {ggtext}I mentioned in a recentR for the Rest of Us podacst how I used to use R for the first 95% of my plots but then added annotations and edits using other tools because it was easier. I also mentioned how that’s no longer the case - I now make (almost) 100% of my R visualations in R, and that’s largely down to the {ggtext} package. The{ggtext} package provides simple Markdown and HTML rendering for text in {ggplot2}.

Let’s start by defining a named vector of colours for male and female lemurs:

| 1234 | col\_vec <- c( "male" = "#0E7C7B", "female" = "#922D50") |

For more complicated examples you can, of course, use something like case_when() to map colours to values to save you from typing out the values.

Now we need to pass these colours into the subtitle text. Let’s create a variable that contains the text, called ggtext_st. We use HTML code (and a tiny bit of CSS) to format the text with colours. If you’re not familiar with HTML code, an example might look like this:

| 1 | "<span style='color:red'>Text to appear goes here</span>" |

We wrap all of the text we want to appear inside <span></span> tags. Inside the style option, we pass in CSS options such as color.

We could manually pass in the text we want to appear in the subtitle:

| 1234 | ggtext\_st <- "This scatter plot shows how the weights of <span style='color:#0E7C7B'>male</span> and <span style='color:#922D50'>female</span> lemurs change with age.") |

However, to make sure that the colours and variables are correctly mapped in the text, we can use the{glue} package to pass in the variables from col_vec:

| 12345 | ggtext\_st <- glue::glue( "This scatter plot shows how the weights of <span style='color:{col\_vec[[1]]}'>{names(col\_vec)[[1]]}</span> and <span style='color:{col\_vec[[2]]}'>{names(col\_vec)[[2]]}</span> lemurs change with age.") |

We can add the colours to our plot using scale_colour_manual(), and pass our new subtitle into the labs() function:

| 123456789 | new\_plot <- base\_plot + scale\_colour\_manual( values = col\_vec ) + labs( title = "How much does a collared brown lemur weigh?", subtitle = ggtext\_st )new\_plot |

That doesn’t look quite right…

That’s because we haven’t actually used the {ggtext} package yet! We need to tell {ggplot2} that we’re using HTML code in the subtitle. We do this using the theme() function and setting the plot.subtitle argument to element_textbox_simple(). You can also use element_textbox if you prefer. Another nice feature of element_textbox_simple() is that long text is automatically wrapped to the width of the plot rather than running off the end! We can also remove the existing legend at the same time:

| 123456 | library(ggtext)new\_plot + theme( legend.position = "none", plot.subtitle = element\_textbox\_simple() ) |

I hope you’ll agree that this looks much cleaner and tidier!

See also geom_richtext() or geom_textbox() as alternatives to geom_text().

Coloured text with {marquee}The recently released{marquee} package provides a markdown parser and renderer for the R graphics engine - meaning it can be used to render more complex markdown text in {ggplot2} graphics.

{marquee} is built for the future so you do need the lastest version of R Graphics engines for it to work.

To add coloured text with Markdown format, we wrap the text to appear in curly brackets. After the opening curly bracket, add a . and the colour you want to the text to appear:

| 1 | "{.red Text to appear goes here}" |

As in the previous example with {ggtext}, we can use glue() to bring the colours and the labels into the subtitle text string. Since the markdown formatting is written inside curly brackets { and }, the glue() function might get confused about which brackets it’s supposed to be using and complain. To avoid this, we’ll tell glue() to use [ and ] instead.

| 12345 | marquee\_st <- glue::glue( "This scatter plot shows how the weights of {.[col\_vec[[1]]] [names(col\_vec)[[1]]]} and {.[col\_vec[[2]]] [names(col\_vec)[[2]]]} lemurs change with age.", .open = "[", .close = "]") |

Much like {ggtext}, we need to tell {ggplot2} that we’re using {marquee} to format text, by passing element_marquee() to the plot.subtitle argument in theme():

| 1 2 3 4 5 6 7 8 910111213 | library(marquee)base\_plot + scale\_colour\_manual( values = col\_vec ) + labs( title = "How much does a collared brown lemur weigh?", subtitle = marquee\_st ) + theme( legend.position = "none", plot.subtitle = element\_marquee(width = 1) ) |

The text wrapping using {marquee} isn’t automatic, but you can set width = 1 to wrap the text.

See also geom_marquee() as an alternative to geom_text().

Comparing {ggtext} and {marquee}You can see that though {ggtext} and {marquee} use HTML vs Markdown, respectively, the way you add coloured text isn’t too different - the process is similar. If all you’re doing is adding coloured text, there isn’t much of a reason to choose one over the other - unless you have a strong preference for HTML or Markdown syntax. One benefit of {marquee} is that it doesn’t just work with {ggplot2} - you can also use it with other graphics packages in R. You can read more about {marquee} in theblog post by Thomas Lin Pedersen.

I, for one, am very excited about all of the new text features coming to R through {marquee} - it brings a lot more than just coloured text in subtitles. But {ggtext} still has a special place for me as the package that changed how easy it is to go from the start to finish of a visualisation using only R.

To leave a comment for the author, please follow the link and comment on their blog: R on Nicola Rennie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Coloured text in {ggplot2}: {ggtext} vs {marquee}

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Polytomous Latent Class Analysis and Regression in R which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Polytomous Latent Class Analysis and Regression in R

Date: Wednesday, July 3rd, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Lana Bojanić is a research associate and PhD candidate at the University of Manchester. With over 7 years of experience using R, she is also a co-founder of the R user group at the University of Manchester and R Ladies Zagreb, Croatia. Lana is passionate about introducing people to R and supporting them during their transition to full-time R users.

Description: Polytomous (multi-category) data is common in many fields that utilise surveys, tests or- assessments. This workshop will deal with latent class analysis and latent class regression analysis of this data type, using PoLCA package. Furthermore, we will cover the necessary data preparation for this analysis, specifying the model, and calculating/extracting fit values. Finally, we will look into different ways of plotting results for this analysis.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Polytomous Latent Class Analysis and Regression in R workshop was first posted on June 3, 2024 at 3:08 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Polytomous Latent Class Analysis and Regression in R workshop

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 464–468

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #464As you already now well, Vijay A. Verma aka ExcelBI love finding some unique sequences of numbers and then combine them, and this time is something like this again. Today we have evil numbers that are palindromic sa well. Probably just like me you haven’t heard about evil ones before. They are numbers that if you transform it to binary, will have even number of ones. Palindromic ones are easy. They need to be like in mirror, read from start and end looks exactly the same. Today special is function intToBin() thanks to which we bring binary representation of number painlessly.

Loading libraries and data library(tidyverse)library(readxl)library(stringi)library(R.utils)test = read\_excel('Excel/464 Palindromic Evil Numbers.xlsx', range = "A1:A1001") Transformation is\_palindromic = function(x) { x\_str <- as.character(x) x\_str == stri\_reverse(x\_str)}is\_evil = function(x) { str\_count(intToBin(x),"1") %% 2 == 0}range = tibble(numbers = 1:1000000) %>% mutate(palindromic = is\_palindromic(numbers), evil = is\_evil(numbers)) %>% filter(palindromic & evil) %>% filter(numbers >= 10) %>% head(1000) Validation all.equal(range$numbers, test$`Answer Expected`)# [1] TRUE Puzzle #465Today we have (for me personally), one of harder challenges, because it is about optimisation of resource allocation. I am not really familiar with it on professional ground, so I only tried to achieve a goal. I hope you will find it interesting and insightful.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/465 Task Assignment.xlsx", range = "A1:E10") Transformation task\_candidates <- input %>% select(Task\_ID = `Task ID`, P1, P2, P3, P4) %>% pivot\_longer(cols = P1:P4, names\_to = "Person", values\_to = "Candidate") %>% filter(!is.na(Candidate))assignments <- tibble(Task\_ID = integer(), Person = character())assign\_tasks <- function(candidates) { task\_count <- tibble(Person = unique(candidates$Person), Count = 0) for (task in unique(candidates$Task\_ID)) { possible\_people <- candidates %>% filter(Task\_ID == task) %>% arrange(task\_count$Count[match(Person, task\_count$Person)]) chosen\_person <- possible\_people$Person[1] assignments <<- assignments %>% add\_row(Task\_ID = task, Person = chosen\_person) task\_count$Count[task\_count$Person == chosen\_person] <- task\_count$Count[task\_count$Person == chosen\_person] + 1 }}assign\_tasks(task\_candidates)while (any(assignments %>% count(Person) %>% pull(n) > 3)) { assignments <- tibble(Task\_ID = integer(), Person = character()) assign\_tasks(task\_candidates)}assignments %>% arrange(Task\_ID) %>% mutate(Person = case\_when( Person == "P1" ~ "A", Person == "P2" ~ "B", Person == "P3" ~ "C", Person == "P4" ~ "D" )) Result (because can be different than shown)Puzzle #466We had evil numbers and now we have another weird ones. Bouncy like on trampoline. What does it mean? That order of digits in number is not decreasing, not increasing and not being flat. We need to cut them up and check how digits behave in each single number. And surprise… We need them in the number of 10k. Quite a big number so I used technique called memory allocation at the beginning. Function doesn’t need copy previous list and append new value (because it is time and memory consuming). I add vector if size 10k at the very begining, so memory is already having storage and we are just putting in the place, like book on shelves.

Loading libraries and data library(tidyverse)library(readxl)test = read\_excel("Excel/466 Bouncy Numbers.xlsx", range = "A1:A10001") Transformation is\_bouncy = function(n) { digits = str\_split(as.character(n), "")[[1]] %>% as.integer() is\_decreasing = all(digits == cummax(digits)) is\_increasing = all(digits == cummin(digits)) return(!is\_decreasing & !is\_increasing)}find\_bouncy\_numbers = function(limit) { bouncy\_numbers = integer(limit) count = 0 num = 100 while (count < limit) { if (is\_bouncy(num)) { count = count + 1 bouncy\_numbers[count] = num } num = num + 1 } bouncy\_numbers}bouncy\_numbers = find\_bouncy\_numbers(10000) Validation all.equal(as.numeric(test$`Answer Expected`), bouncy\_numbers) # TRUE Puzzle #467Little bit less numbers, litlle bit more cleaning. Some people were planning meetings not discussing about time availability. So as usual, we have a role of “Hey Dude, could you fix it?”. Of course we could. We need to find out which meeting is overlapping another. To find it I used lubridate package and its objects intervals. However one of co-competitors use another very creative solution using non-equi overlapping joins. Check it up.

Open libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/467 Overlapping Times.xlsx", range = "A1:C8")test = read\_excel("Excel/467 Overlapping Times.xlsx", range = "E2:L9") %>% select(`Task ID` = ...1, everything()) Transformation r1 = input %>% mutate(interval = interval(`From Time`, `To Time`)) %>% select(-`From Time`, -`To Time`) combinations = expand\_grid(r1, r1, .name\_repair = "unique") %>% filter(`Task ID...1` != `Task ID...3`) %>% mutate(overlap = ifelse(int\_overlaps(interval...2, interval...4), "Y", NA\_character\_)) %>% select(`Task ID` = `Task ID...1`, `Second Task ID` = `Task ID...3`, overlap) %>% pivot\_wider(names\_from = `Second Task ID`, v Validation identical(test, combinations)# [1] TRUE Puzzle # 468Last puzzle for today was pretty tricky, and we needed to find rows that has lowest value of column C1, etc. So my approach was to find indices of rows that fulfill our conditions, extract them and bind them together. Check how I use map_int, map_dfr and which.min/which.max.

Loading data and libraries library(tidyverse)library(readxl)input = read\_excel("Excel/467 Generate Min and Max Rows.xlsx", range = "A2:F20") test = read\_excel("Excel/467 Generate Min and Max Rows.xlsx", range = "I2:N10")inst = read\_excel("excel/467 Generate Min and Max Rows.xlsx", range = "H3:H10", col\_names = "Inst") Transformation r1 = inst %>% mutate(Inst = str\_sub(Inst,1,6)) %>% separate(Inst, into = c("fun", "column"), sep = " ", remove = F) %>% mutate(fun = str\_to\_lower(fun))r2 = r1 %>% mutate(index = ifelse(fun == "min", map\_int(column, ~which.min(input[[.x]])), map\_int(column, ~which.max(input[[.x]]))))result = map\_dfr(r2$index, ~input[.x,]) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
PS. Couple weeks ago, I started uploading on Github not only R, but also in Python. Come and check it.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, R enthusiasts! I’m excited to share some fantastic updates to the TidyDensity package. These updates introduce a suite of parameter estimate functions designed to make your data analysis more efficient and insightful. Whether you’re dealing with common distributions or more specialized ones, these functions have got you covered.

Why Parameter Estimation?Parameter estimation is crucial when working with statistical distributions. It allows you to infer the parameters of a distribution from your data, providing insights into its underlying structure. This is particularly useful when you want to model real-world phenomena accurately.

New Parameter Estimate FunctionsHere’s a quick rundown of the newly introduced functions in TidyDensity:

util_zero_truncated_negative_binomial_param_estimate()
util_zero_truncated_poisson_param_estimate()
util_f_param_estimate()
util_zero_truncated_geometric_param_estimate()
util_t_param_estimate()
util_pareto1_param_estimate()
util_paralogistic_param_estimate()
util_inverse_weibull_param_estimate()
util_inverse_pareto_param_estimate()
util_inverse_burr_param_estimate()
util_generalized_pareto_param_estimate()
util_generalized_beta_param_estimate()
util_zero_truncated_binomial_param_estimate()

Each function is tailored to a specific distribution, providing a streamlined way to estimate its parameters.

Example: Estimating Parameters of a t DistributionLet’s dive into an example using the util_t_param_estimate() function. Suppose you have data that you believe follows a t distribution. Here’s how you can estimate its parameters:

library(dplyr)library(ggplot2)library(TidyDensity)set.seed(123)x <- rt(100, df = 10, ncp = 0.5)output <- util\_t\_param\_estimate(x)# Display the estimated parametersprint(output$parameter\_tbl)

```

A tibble: 2 × 7 dist_type samp_size mean variance method df_est ncp_est 1 T Distribution 100 0.612 0.949 MME 0.959 0.6122 T Distribution 100 0.612 0.949 MLE 8.32 0.571

`` In this example, we generated some data from a t distribution with degrees of freedom (df) of 10 and a non-centrality parameter (ncp) of 0.5. Using theutil_t_param_estimate()` function, we estimated these parameters from the data.

The parameter_tbl in the output contains the estimated values, while combined_data_tbl can be used for visualization.

Visualizing the ResultsHere’s what the output might look like:

```

Visualize the combined dataoutput$combined_data_tbl |> tidy_combined_autoplot(.interactive = TRUE)

`` In the above plot, we visualize the output of theutil_t_param_estimate()` function from the TidyDensity package. The visualization shows how well the estimated t distribution fits our sample data. The x-axis represents the data values, while the y-axis shows the density. The different colors represent the data and the estimated density functions.

How to Use the New FunctionsEach of the new parameter estimate functions follows a similar approach. Here’s a step-by-step guide to get you started:

Load your data: Ensure your data is properly formatted and loaded into R.
Select the appropriate function: Choose the function that matches the distribution you believe your data follows.
Estimate the parameters: Use the selected function to estimate the parameters.
Analyze and visualize: Review the estimated parameters and use the visualization functions to see how well the estimated distribution fits your data.

Your Turn!I highly encourage you to try these new functions on your own datasets. Whether you’re working with common distributions or tackling more specialized ones, these tools can help you gain deeper insights into your data.

Feel free to experiment and see how these functions perform with different types of data. The more you explore, the better you’ll understand the strengths and applications of each distribution.

ConclusionThe new parameter estimate functions in TidyDensity open up exciting possibilities for data analysis. By simplifying the process of parameter estimation, they allow you to focus more on interpreting results and making informed decisions.

Give these functions a try and see how they can enhance your analysis workflow. Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: An Overview of the New Parameter Estimate Functions in the TidyDensity Package

View Details

[This article was first published on sweissblaug, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Have you ever…

…spent hours, maybe even days, tackling a tricky coding problem at work, only to discover a better, cleaner, and faster open-source alternative? UGH! There’s got to be a better way!

You’re in luck!

Inspired by the populist policies of recent US administrations and a global resurgence in protectionist trade policies, we present to you the pRotectionism package for R. This package aims to level the playing field by extending the runtime of more efficient code.

Key Features

taRiff(): Want to delay the execution of an efficient function? Simply call taRiff() on the function along with the desired delay time in seconds. Voila! You’re back in the game, competing against more efficient code on a more even playing field.
autaRky(): If you are Jones-ing for that extra edge, autaRky() is for you. This function removes all imported libraries, providing you with a self-sufficient coding environment that just might help you keep your job.

Get Started

Download the package from GitHub: pRotectionism

Contribute

Interested in adding functionality? You’re more than welcome to contribute! Just be aware that any new functionality might incur an additional taRiff() at my discretion.

To leave a comment for the author, please follow the link and comment on their blog: sweissblaug.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: pRotectionism: The way to compete in 2024

View Details

[This article was first published on Maëlle's R blog on Maëlle Salmon's personal website, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.“Make small Git commits with informative messages” is a piece of advice we hear a lot when learning Git. That’s why we might want to sometimes rewrite history in a branch. In this post, I’d like to underline three main () reasons why you’ll be happy you, or someone else, made small and informative Git commits in a codebase.

A disclaimer: these three reasons are only valid if you do not write perfectly working and readable code all the time. If you do, you won’t need to use your Git log for debugging and undoing purposes, so who cares about your Git commits?

A mysterious line of code (Git blame)Imagine you encounter a mysterious line of code in a script that either someone else or yourself wrote a while ago.

x <- x - 1 As luck will have it, there is no comment explaining the line around it. How can you guess the intent of that line?

A Git command might help you: Git blame! It’d better called Git explain, as blaming is a bad idea. Anyway, it’s a tool for seeing the code with more context. For each line, you get to see when it was added or last modified, by whom, and when.

For instance, I added or modified the 2 lines at the top and the the two lines in the middle, that are highlighted in yellow, a few years ago. The line that is interesting to me, ‘x <- x - 1’, was added by ropenscibot in 2017.

Now, I can try and display more information about the Git commit that last modified, or added that mysterious line. The person who last touch the line chose our adventure…

If we’re unlucky, we get a Git commit whose message is ‘“Commit a bunch of files before lunch ”’ and whose diff shows 145 changed files with 2,624 additions and 2,209 deletions. Not exactly helpful, we’re still stuck.
If we’re lucky, we get a Git commit whose message is ‘“fix: adapt code to tool’s 0-indexing”’ and whose diff shows 2 changed files with 3 additions and 2 deletions. Now we know why x was modified this way! We can add an explaining comment to make the code more readable for later, and continue with our work.

Git blame is a built-in Git thing, but for me, it’s first and foremost a button in the Github interface for any file. Use Git blame as you prefer.

A bad idea 7 commits ago (Git revert)Oh no, that idea from 7 commits ago is bad! Do we

Manually remove the change;
Simply revert the commit that added the change?

Once again, the quality and granularity of the commits will decide for you. If the snapshot from 7 commits ago included many different unrelated changes, it will be easier to manually undo the changes yourself by going into the files and deleting or amending stuff. If the snapshot from 7 commits ago was small enough, you can use Git revert with that commit ID, that will create a new commit undoing that commit.

You can try out Git revert in an exercise of the saperlipopette package.

All was well 3 days ago, now my thing is brokenImagine you were doing something with your somewhat complex R project 3 days ago, and that all was well.

Now, after a bunch of changes to your still complex R project, you set out to do the same thing, and… it no longer works.

To add insult to injury, your usual debugging tools aren’t helping you for some reasons. Is all hope lost? No!

Git has a tool to help you go over the Git history between now and then (3 days ago, 100 commits ago) in an optimal way: Git bisect. Git bisect will make you try some commits to help pinpoint which change, which commit exactly, introduced the bug you’re now observing.

Your Git history looks something like this:

a commit at which you know the thing worked,
a commit at which it no longer works,
with commits in between about which you have no information because you were not doing or testing the thing, so at some point you introduced a bug.

You tell Git bisect which commit was for sure good, and which commit is bad, and it starts putting your project in the state it was at a given commit, let’s say the third commit:

You have to tell Git bisect whether that commit is good or bad (no complex vocabulary for once). Let’s say your thing seems to work at that point.

Git bisect will assume all commits up to that commit were good.

It now makes you explore another commit.

You try your thing, it does not work. Therefore this commit, and the following ones, are bad.

You’re in luck, after only a few steps you get to the last commit to try out.

Again a bad one, your thing does not work for your project in that state.

It’s the first bad commit, so it’s the one that introduced the breaking change!

Now, again, the person who made that commit chose our adventure:

If we’re unlucky, we get a Git commit whose message is ‘“Commit a bunch of files before workout ”’ and whose diff shows 145 changed files with 2,624 additions and 2,209 deletions. Nooo, that’s still a ton of code to read.
If we’re lucky, we get a Git commit whose message is ‘“refactor: start using YAML”’ and whose diff shows 2 changed files with 3 additions and 2 deletions. Ah, YAML, what an idea! We can now search where we forgot to add quotes in a YAML file.

You can practice Git bisect in an exercise of the saperlipopette package.

Two bonus reasonsEasier reviewSmall commits will be easier to review by another human. Ideally, it means the code that ends up in the codebase is better, so that there’ll be less future need of debugging and undoing.

Better automatic changelog generationIf you use a tool like the R package fledge for generating your changelog or changelog draft, that uses Git commits as input data… you’ll get a better starting changelog if your commits are good.

ConclusionIn this post I showed three main reasons for making small and informative Git commits, all related to debugging or undoing some bad changes. So, creating small, informative Git commits will help someone in the future in trying times. I hope it can inspire myself in particular to pay attention to the Git history I create.

The saperlipopette package contains a few exercises that can help write a better Git history, including: “Oh shit, I committed and immediately realized I need to make one small change!", “Hey I’d like to split these changes to the same file into several commits!", “Hey I’d like to make my commits in a branch look informative and smart!"

To leave a comment for the author, please follow the link and comment on their blog: Maëlle's R blog on Maëlle Salmon's personal website.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Why you need small, informative Git commits

View Details

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A simple, yet effective way to set your colour palette in R using ggplot library.

library(ggplot2)set.seed(2908)my\_palette <- c("red", "limegreen", "#3357FF", "goldenrod1", "#33FFFF", "brown")data <- data.frame( x = 1:25, y = rnorm(25), group = rep(c("A", "B", "C", "D", "E"), each = 5) ) After that, we can start “chaining” ggplot graphs.

Scatter plot ggplot(data, aes(x = x, y = y, color = group)) + geom\_point(size = 3) + scale\_color\_manual(values = my\_palette, na.value = "grey45") + theme\_minimal() Barchart / Histogram ggplot(data, aes(x = x, y = y, color = group, fill=group)) + geom\_bar(stat = "identity") + scale\_color\_manual(values = my\_palette, na.value = "grey45") + theme\_minimal() Boxplot ggplot(data, aes(x = x, y = y, fill=group)) + geom\_boxplot() + scale\_color\_manual(values = my\_palette, na.value = "grey45") + theme\_minimal() 3.14 chart data2 <- data.frame ( y = c(2,15,24,9,17,2), group = LETTERS[1:6]) )ggplot(data2, aes(x='', y=y, fill=group)) + geom\_bar(stat="identity", width=1, colour="white") + scale\_color\_manual(values = my\_palette, na.value = "grey45") + coord\_polar("y", start=0) + theme\_void() Finally custom colours! As always, the complete code is available on GitHub in Useless_R_function repository.

Enjoy R-scripting and stay healthy!

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simple custom colour palettes with R ggplot graphs

View Details

[This article was first published on geocompx, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The ESRI Shapefile format is possibly the most popular data format for storing spatial vector data. It is well-established, widely supported, and can be used in most GIS software. At the same time, it has many shortcomings and limitations: it is not a single file, but a collection of files; it has a limit of 2GB per file; it does not support more than one geometry type per file, and more.1 In the last few years, a few improved alternatives to the Shapefile format have been developed, such as the GeoPackage and the FlatGeobuf format.2

In 2017, together with Roger Bivand and Robin Lovelace, we created the spData R package – a collection of spatial data both as R objects and external file formats. The package has two main goals: to provide easy access to spatial data for teaching purposes and to provide datasets that could be used as examples in the documentation of other packages. Many of the datasets in the package are stored in the ESRI Shapefile format, which is not ideal.

Thus, after some discussions, we decided to replace the Shapefile format with the GeoPackage format in the spData package (version number: 2.3.1). We hope that this change will serve as a good example and encourage others to use modern spatial data formats. At the time of writing (June 2024), we added GeoPackage versions of all the datasets in the package, but we also plan to remove the ESRI Shapefile versions in the near future. Thus, if you use the spData package, be aware of this change and update your code accordingly. Also, if you have any comments or suggestions, please let us know in the issue on the package’s GitHub page at https://github.com/Nowosad/spData/issues/62.

Footnotes1. More about its limitations can be found at the Switch from Shapefile website and in the Geocomputation with R book.︎ 2. And more alternatives are being developed, such as GeoParquet.︎

ReuseCC BY 4.0CitationBibTeX citation: @online{nowosad2024, author = {Nowosad, Jakub}, title = {Shapefile Must... Be Replaced}, date = {2024-06-02}, url = {https://geocompx.org//post/2024/spData-bp1}, langid = {en}} For attribution, please cite this work as:Nowosad, Jakub. 2024. “Shapefile Must… Be Replaced.” June2, 2024. https://geocompx.org//post/2024/spData-bp1. To leave a comment for the author, please follow the link and comment on their blog: geocompx.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Shapefile must… be replaced

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionThe latest update the the TidyDensity package introduces several new functions that make it easier to work with data in R. In this article, we’ll take a look at the new AIC functions and how they work.

New FunctionsThe set of functions that we will go over are the util_dist_aic() functions, where dist is the distribution in question, for example util_negative_binomial_aic(). These functions calculate the Akaike Information Criterion (AIC) for a given distribution and data. The AIC is a measure of the relative quality of a statistical model for a given set of data. The lower the AIC value, the better the model fits the data. Here is a bit about the functions.

Usage util\_negative\_binomial\_aic() Arguments* .x: A numeric vector of data values.

ValueA numeric value representing the AIC for the given data and distribution.

DetailsThis function calculates the Akaike Information Criterion (AIC) for a distribution fitted to the provided data.

This function fits a distribution to the provided data. It estimates the parameters of the distribution from the data. Then, it calculates the AIC value based on the fitted distribution.

Initial parameter estimates: The function uses the param estimate family of functions in order to estimate the starting point of the parameters. For example util_negative_binomial_param_estimate().

Optimization method: Since the parameters are directly calculated from the data, no optimization is needed.

Goodness-of-fit: While AIC is a useful metric for model comparison, it’s recommended to also assess the goodness-of-fit of the chosen model using visualization and other statistical tests.

Examples library(TidyDensity)set.seed(123)# Generate some datax <- rnorm(100)# Calculate the AIC for a negative binomial distributioncat( " AIC of rnorm() using TidyDensity: ", util\_normal\_aic(x), "\n", "AIC of rnorm() using fitdistrplus: ", fitdistrplus::fitdist(x, "norm")$aic)

AIC of rnorm() using TidyDensity: 268.5385 AIC of rnorm() using fitdistrplus: 268.5385 New AIC FunctionsHere is a listing of all of the new AIC functions:

util_negative_binomial_aic()
util_zero_truncated_negative_binomial_aic()
util_zero_truncated_poisson_aic()
util_f_aic()
util_zero_truncated_geometric_aic()
util_t_aic()
util_pareto1_aic()
util_paralogistic_aic()
util_inverse_weibull_aic()
util_pareto_aic()
util_inverse_burr_aic()
util_generalized_pareto_aic()
util_generalized_beta_aic()
util_zero_truncated_binomial_aic()

ConclusionThanks for reading. I hope you find these new functions useful in your work. If you have any questions or feedback, please feel free to reach out. I worked hard to ensure where I could that results would come back identical to what would be calculated from the amazing fitdistrplus package.

Happy Coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: An Overview of the New AIC Functions in the TidyDensity Package

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Call for abstracts now openWe are excited to announce the Call for Abstracts for Shiny in Production 2024, to be held on 9th-10th October 2024 in Newcastle upon Tyne, UK. This event brings together industry experts, data scientists, and developers to explore the latest advancements and best practices in deploying Shiny applications in production settings.

About the ConferenceAs Shiny continues to revolutionise data visualisation and interactive web applications, the need for robust, scalable, and efficient production environments is more critical than ever. This conference aims to address these needs by providing a platform for knowledge sharing, collaboration, and innovation.

Whether you’re a seasoned {shiny} user who wants to network and share knowledge, someone who’s just getting started and wants to learn from the experts, or anybody in between, if you’re interested in {shiny}, this conference is for you.

Topics of InterestWe invite abstracts on a wide range of topics, including but not limited to:

Scalable Architectures: Techniques for scaling Shiny applications to handle large datasets and high user loads.
Security Best Practices: Ensuring the security and privacy of data within Shiny applications.
Performance Optimisation: Strategies for improving the speed and responsiveness of Shiny apps.
Integration with Other Technologies: Combining Shiny with other tools and platforms for enhanced functionality.
Python: Developing Python Shiny apps
Case Studies: Real-world examples of successful Shiny deployments in various industries.
Automated Testing and Continuous Deployment: Best practices for maintaining high-quality applications through automated workflows.

To get an idea of past topics, check out our YouTube channel, where we have playlists of talks from Shiny in Production 2022 and 2023.

Submission GuidelinesTo submit your abstract, please follow these guidelines:

Abstract Length: Up to 250 words.
Deadline: Submissions must be received by 11:59 on 30th June 2024.
Submission Portal: Submit your abstract here.

Important Dates* Abstract Submission Deadline: 30th June 2024 * Notification of Acceptance: 1st August 2024 * Conference Dates: 9th-10th October 2024

For more information, visit our conference website.

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Shiny in Production 2024: Call for Abstracts

View Details

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Calculating cumulative percentage or percentage per group for each time can sometimes be a task with a slight twist. Let’s check this with ggplot2 and tidyverse.

library(ggplot2)library(tidyverse)data <- data.frame( sector = rep(1:20, each = 5), item = rep(1:5, times = 20), value = rpois(100, 10) ) Three (out of many more) ways to show how this can be achieved.

using simple calculation ```

using simple calculationggplot(data, aes(x = factor(sector), y = value / sum(value) * 100, fill = factor(item))) + geom_bar(stat = "identity", position = "fill") + scale_y_continuous(labels = scales::percent_format()) + labs(x = "Sector", y = "Percentage", title = "Stacked 100% Bar Plot by Sector") + coord_flip()

replacing percent with tapply

replacing percent with tapplyggplot(data, aes(x = factor(sector), y = value / tapply(value, sector, sum)[as.character(sector)] * 100, fill = factor(item))) + geom_bar(stat = "identity", position = "fill") + scale_y_continuous(labels = scales::percent_format()) + labs(x = "Sector", y = "Percentage", title = "Stacked 100% Bar Plot by Sector") + coord_flip()

without any complications ggplot(data, aes(x = factor(sector), y = value, fill = factor(item))) + geom_bar(stat = "identity", position = "fill") + scale_y_continuous(labels = scales::percent_format(), name = "Percentage") + labs(x = "Sector", title = "Stacked 100% Bar Plot by Sector") + coord_flip() ``` And by all means, the diagram is in all cases the same. Just the examples can be slightly more over-engineered

Happy R-coding and stay healthy!

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Calculating data for visualization on stacked 100% bar

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Joins Are No Mystery Anymore: Hands-On Tutorial — Part 1Welcome! In this tutorial, I’ll be your guide as we unravel the mysteries of data joins in R. Whether you’re working with customer records, inventory lists, or historical documents, mastering data joins is essential for any data analyst or scientist. Together, we’ll explore a variety of join types through real-life examples and datasets, making complex concepts easy to understand and apply. By the end of this tutorial, you’ll be equipped with the knowledge and skills to confidently join data and uncover the valuable insights hidden within. Let’s get started and make joins a breeze!

At the very beginning… All datasets I am working on and getting using load() function are prepared for you and uploaded to Github.

Inner JoinAn Inner Join is used to combine rows from two tables based on a related column between them. It returns only the rows where there is a match in both tables. If there are no matches, the result set will not include those rows.

Explanation of the ScenarioIn our scenario, we have customer orders and payments. We want to find orders that have been paid. This will help us understand which customers have completed their payments and which orders are still pending.

Data file: https://github.com/kgryczan/medium_publishing/blob/main/inner_join_data.RData

Description of the DatasetsWe will use two datasets:

orders: Contains information about customer orders. Columns: order_id, customer_id, order_date
payments: Contains information about payments made for orders. Columns: payment_id, order_id, amount, payment_date

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("inner_join_data.RData")# Display the datasetsprint(orders, n=5)# A tibble: 30 × 3 order_id customer_id order_date 1 1 102 2024-01-012 2 113 2024-01-023 3 108 2024-01-034 4 111 2024-01-045 5 106 2024-01-05# ℹ 25 more rowsprint(payments, n=5)# A tibble: 20 × 4 payment_id order_id amount payment_date 1 201 27 167. 2024-01-05 2 202 30 80.2 2024-01-06 3 204 28 110. 2024-01-08 4 206 24 159. 2024-01-10 5 207 4 173. 2024-01-11 # ℹ 15 more rows

``` Performing the Inner Join

```

Perform the inner joinorders_paid <- inner_join(orders, payments, by = "order_id")# Display the resultprint(orders_paid)# A tibble: 20 × 6 order_id customer_id order_date payment_id amount payment_date 1 2 113 2024-01-02 209 90.6 2024-01-13 2 2 113 2024-01-02 229 129. 2024-02-02 3 4 111 2024-01-04 207 173. 2024-01-11 4 6 104 2024-01-06 228 181. 2024-02-01 5 7 111 2024-01-07 217 80.8 2024-01-21 6 11 109 2024-01-11 212 108. 2024-01-16 7 12 109 2024-01-12 224 183. 2024-01-28 8 13 105 2024-01-13 216 194. 2024-01-20 9 13 105 2024-01-13 226 117. 2024-01-30 10 16 112 2024-01-16 223 176. 2024-01-27 11 20 104 2024-01-20 208 131. 2024-01-12 12 20 104 2024-01-20 225 66.2 2024-01-29 13 21 115 2024-01-21 227 130. 2024-01-31 14 23 104 2024-01-23 230 103. 2024-02-03 15 24 111 2024-01-24 206 159. 2024-01-10 16 26 111 2024-01-26 213 89.4 2024-01-17 17 27 110 2024-01-27 201 167. 2024-01-05 18 28 101 2024-01-28 204 110. 2024-01-08 19 28 101 2024-01-28 221 156. 2024-01-25 20 30 114 2024-01-30 202 80.2 2024-01-06

``` Explanation of the Code

We first load the datasets using the load function.
We then use the inner_join function from the dplyr package to join the orders and payments datasets on the order_id column.
Finally, we display the result to see which orders have been paid.

Interpretation of Results

The resulting dataset orders_paid contains only the rows where there is a match in both orders and payments datasets. This means that only the orders that have been paid are included in the result. Each row in the result represents an order that has been matched with a corresponding payment, showing details from both the orders and payments tables.

Homework for ReadersIn the same inner_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

enrollments: Contains information about student enrollments.
Columns: student_id, course_id, enrollment_date
exam_results: Contains information about exam results.
Columns: student_id, course_id, exam_score, exam_date

Your task is to perform an inner join on these datasets to find students who have both enrolled and taken exams. Use the student_id and course_id columns for joining.

Left Join (Left Outer Join)A Left Join returns all rows from the left table, and the matched rows from the right table. If there is no match, the result is NULL on the side of the right table.

Explanation of the ScenarioIn this scenario, we have product information and sales records. We want to find all products, including those that haven’t been sold. This helps in understanding which products are in stock and which are moving in the market.

Data file: https://github.com/kgryczan/medium_publishing/blob/main/left_join_data.RData

Description of the DatasetsWe will use two datasets:

products: Contains information about the products.
Columns: product_id, product_name, category
sales: Contains information about the sales made.
Columns: sale_id, product_id, quantity_sold, sale_date

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("left_join_data.RData")# Display the datasetsprint(products, n = 5)# A tibble: 30 × 3 product_id product_name category 1 1 Product A Category 12 2 Product B Category 33 3 Product C Category 34 4 Product D Category 35 5 Product E Category 3# ℹ 25 more rowsprint(sales, n = 5)# A tibble: 30 × 4 sale_id product_id quantity_sold sale_date 1 101 2 10 2024-02-012 102 29 10 2024-02-023 103 16 6 2024-02-034 104 30 5 2024-02-045 105 25 4 2024-02-05# ℹ 25 more rows

``` Performing the Left Join

```

Perform the left joinproducts_sales <- left_join(products, sales, by = "product_id")# Display the resultprint(products_sales)# A tibble: 41 × 6 product_id product_name category sale_id quantity_sold sale_date 1 1 Product A Category 1 106 7 2024-02-06 2 2 Product B Category 3 101 10 2024-02-01 3 2 Product B Category 3 118 8 2024-02-18 4 3 Product C Category 3 NA NA NA 5 4 Product D Category 3 107 4 2024-02-07 6 4 Product D Category 3 127 2 2024-02-27 7 5 Product E Category 3 113 9 2024-02-13 8 6 Product F Category 3 NA NA NA 9 7 Product G Category 1 NA NA NA 10 8 Product H Category 2 NA NA NA # ℹ 31 more rows

``` Explanation of the Code

We first load the datasets using the load function.
We then use the left_join function from the dplyr package to join the products and sales datasets on the product_id column.
Finally, we display the result to see all products, including those that haven’t been sold.

Interpretation of ResultsThe resulting dataset products_sales contains all rows from the products dataset, with matched rows from the sales dataset. If a product hasn’t been sold, the columns from the sales dataset will have NULL values.

Homework for ReadersIn the same left_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

employees: Contains information about employees.
Columns: employee_id, name, department
parking_permits: Contains information about parking permits issued.
Columns: permit_id, employee_id, permit_date

Your task is to perform a left join on these datasets to find all employees, including those without a parking permit. Use the employee_id column for joining.

Right Join (Right Outer Join)A Right Join returns all rows from the right table, and the matched rows from the left table. If there is no match, the result is NULL on the side of the left table.

Explanation of the ScenarioIn this scenario, we have marketing campaigns and responses to those campaigns. We want to find all responses, including those that did not belong to a campaign. This helps in understanding the effectiveness of marketing campaigns and identifying responses that might be related to other activities.

Data file: https://github.com/kgryczan/medium_publishing/blob/main/right_join_data.RData

Description of the DatasetsWe will use two datasets:

campaigns: Contains information about marketing campaigns.
Columns: campaign_id, campaign_name, start_date
responses: Contains information about responses to campaigns.
Columns: response_id, campaign_id, response_date

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("right_join_data.RData")# Display the datasetsprint(campaigns, n = 5)# A tibble: 20 × 3 campaign_id campaign_name start_date 1 2 Campaign B 2024-01-022 4 Campaign D 2024-01-043 5 Campaign E 2024-01-054 7 Campaign G 2024-01-075 8 Campaign H 2024-01-08# ℹ 15 more rowsprint(responses, n = 5)# A tibble: 30 × 3 response_id campaign_id response_date 1 101 11 2024-01-05 2 102 27 2024-01-06 3 103 2 2024-01-07 4 104 16 2024-01-08 5 105 22 2024-01-09 # ℹ 25 more rows

``` Performing the Right Join

```

Perform the right joinresponses_campaigns <- right_join(campaigns, responses, by = "campaign_id")# Display the resultprint(responses_campaigns, n = 30)# A tibble: 30 × 5 campaign_id campaign_name start_date response_id response_date 1 2 Campaign B 2024-01-02 103 2024-01-07 2 4 Campaign D 2024-01-04 112 2024-01-16 3 4 Campaign D 2024-01-04 121 2024-01-25 4 5 Campaign E 2024-01-05 127 2024-01-31 5 8 Campaign H 2024-01-08 130 2024-02-03 6 15 Campaign O 2024-01-15 119 2024-01-23 7 15 Campaign O 2024-01-15 129 2024-02-02 8 16 Campaign P 2024-01-16 104 2024-01-08 9 16 Campaign P 2024-01-16 106 2024-01-10 10 16 Campaign P 2024-01-16 110 2024-01-14 11 16 Campaign P 2024-01-16 116 2024-01-20 12 16 Campaign P 2024-01-16 124 2024-01-28 13 17 Campaign Q 2024-01-17 126 2024-01-30 14 18 Campaign R 2024-01-18 123 2024-01-27 15 27 Campaign NA 2024-01-27 102 2024-01-06 16 28 Campaign NA 2024-01-28 108 2024-01-12 17 28 Campaign NA 2024-01-28 109 2024-01-13 18 28 Campaign NA 2024-01-28 117 2024-01-21 19 30 Campaign NA 2024-01-30 113 2024-01-17 20 11 NA NA 101 2024-01-05 21 22 NA NA 105 2024-01-09 22 19 NA NA 107 2024-01-11 23 6 NA NA 111 2024-01-15 24 14 NA NA 114 2024-01-18 25 3 NA NA 115 2024-01-19 26 9 NA NA 118 2024-01-22 27 9 NA NA 120 2024-01-24 28 11 NA NA 122 2024-01-26 29 9 NA NA 125 2024-01-29 30 11 NA NA 128 2024-02-01

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the right_join function from the dplyr package to join the campaigns and responses datasets on the campaign_id column.
Finally, we display the result to see all responses, including those that did not belong to a campaign.

Homework for ReadersIn the same right_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

online_courses: Contains information about online courses.
Columns: course_id, course_name, launch_date
completions: Contains information about course completions.
Columns: completion_id, course_id, student_id, completion_date

Your task is to perform a right join on these datasets to find all completions, including those for courses that may have been removed. Use the course_id column for joining.

Full Join (Full Outer Join)A Full Join returns all rows when there is a match in either the left or right table. If there is no match, the result is NULL on the side where there is no match.

Explanation of the ScenarioIn this scenario, we have inventory records from two warehouses. We want to get a complete list of all products and quantities, whether they are in one warehouse or the other. This helps in having a comprehensive view of inventory across multiple locations.

Data file: https://github.com/kgryczan/medium_publishing/blob/main/full_join_data.RData

Description of the DatasetsWe will use two datasets:

warehouse1: Contains inventory information from warehouse 1.
Columns: product_id, product_name, quantity
warehouse2: Contains inventory information from warehouse 2.
Columns: product_id, product_name, quantity

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("full_join_data.RData")# Display the datasetsprint(warehouse1, n = 5)# A tibble: 20 × 3 product_id product_name quantity 1 1 Product A 1532 2 Product B 2003 3 Product C 1114 4 Product D 1085 5 Product E 177# ℹ 15 more rowsprint(warehouse2, n = 5)# A tibble: 16 × 3 product_id product_name quantity 1 15 Product O 1612 16 Product P 943 17 Product Q 634 18 Product R 945 19 Product S 111# ℹ 11 more rows

``` Performing the Full Join

```

Perform the full joininventory_full <- full_join(warehouse1, warehouse2, by = "product_id", suffix = c("_wh1", "_wh2"))# Display the resultprint.AsIs(inventory_full) product_id product_name_wh1 quantity_wh1 product_name_wh2 quantity_wh21 1 Product A 153 NA2 2 Product B 200 NA3 3 Product C 111 NA4 4 Product D 108 NA5 5 Product E 177 NA6 6 Product F 161 NA7 7 Product G 175 NA8 8 Product H 70 NA9 9 Product I 72 NA10 10 Product J 89 NA11 11 Product K 189 NA12 12 Product L 109 NA13 13 Product M 177 NA14 14 Product N 124 NA15 15 Product O 123 Product O 16116 16 Product P 188 Product P 9417 17 Product Q 119 Product Q 6318 18 Product R 188 Product R 9419 19 Product S 169 Product S 11120 20 Product T 124 Product T 19721 21 NA Product U 8122 22 NA Product V 9323 23 NA Product W 19924 24 NA Product X 8025 25 NA Product Y 10426 26 NA Product Z 6527 27 NA Product NA 11228 28 NA Product NA 11629 29 NA Product NA 5830 30 NA Product NA 167

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the full_join function from the dplyr package to join the warehouse1 and warehouse2 datasets on the product_id column. The suffix argument is used to distinguish between columns from the two warehouses.
Finally, we display the result to see a comprehensive inventory list.

Interpretation of ResultsThe resulting dataset inventory_full contains all rows from both the warehouse1 and warehouse2 datasets. If a product is only in one warehouse, the columns from the other warehouse will have NULL values. As we see in our result products O to T, are available in both warehouses.

Homework for ReadersIn the same full_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

companyA_employees: Contains information about employees from company A.
Columns: employee_id, name, department
companyB_employees: Contains information about employees from company B.
Columns: employee_id, name, department

Your task is to perform a full join on these datasets to ensure all employees are accounted for from both companies and who is working for both. Use the employee_id column for joining.

Semi JoinIntroduction to Semi JoinA Semi Join returns all rows from the left table where there are matching values in the right table, but does not duplicate columns from the right table. It is useful for filtering the left table based on the presence of matching rows in the right table.

Explanation of the ScenarioIn this scenario, we have customer information and order records. We want to find all customers who have made orders. This helps in identifying active customers.

Data file: https://github.com/kgryczan/medium_publishing/blob/main/semi_join_data.RData

Description of the DatasetsWe will use two datasets:

customers: Contains information about customers.
Columns: customer_id, name, address
orders: Contains information about customer orders.
Columns: order_id, customer_id, order_date

Step-by-Step Code ExamplesLoading the datasets

```

Load the necessary librarieslibrary(dplyr)# Load the datasetsload("semi_join_data.RData")# Display the datasetsprint(customers, n=5)# A tibble: 30 × 3 customer_id name address 1 1 Alice F 423 Pine St2 2 Bob NA 779 Elm St 3 3 Carol B 257 Oak St 4 4 Zoe O 452 Elm St 5 5 Alice F 73 Pine St # ℹ 25 more rowsprint(orders, n=5)# A tibble: 30 × 3 order_id customer_id order_date 1 101 11 2024-01-012 102 3 2024-01-023 103 18 2024-01-034 104 29 2024-01-045 105 9 2024-01-05# ℹ 25 more rows

``` Performing the Semi Join

```

Perform the semi joincustomers_with_orders <- semi_join(customers, orders, by = "customer_id")# Display the resultprint.AsIs(customers_with_orders) customer_id name address1 1 Alice F 423 Pine St2 3 Carol B 257 Oak St3 5 Alice F 73 Pine St4 6 Bob NA 587 Pine St5 8 Zoe V 475 Elm St6 9 Alice P 397 Oak St7 10 Bob P 804 Pine St8 11 Carol O 961 Pine St9 12 Zoe I 14 Pine St10 13 Alice X 104 Pine St11 14 Bob I 981 Elm St12 17 Alice R 295 Elm St13 18 Bob NA 393 Maple St14 20 Zoe NA 845 Maple St15 21 Alice X 145 Elm St16 22 Bob I 179 Maple St17 23 Carol W 140 Oak St18 24 Zoe Y 431 Elm St19 25 Alice M 261 Oak St20 26 Bob E 4 Maple St21 29 Alice Z 609 Pine St

``` Explanation of the Code:

We first load the datasets using the load function.
We then use the semi_join function from the dplyr package to filter the customers dataset to include only those customers who have matching entries in the orders dataset, based on the customer_id column.
Finally, we display the result to see which customers have made orders.

Interpretation of ResultsThe resulting dataset customers_with_orders contains only the rows from the customers dataset where there is a matching row in the orders dataset. This means that only customers who have made at least one order are included.

Homework for ReadersIn the same semi_join_data.RData file, there is another set of datasets for a more creative scenario. You will find:

products: Contains information about products.
Columns: product_id, product_name, category
reviews: Contains information about product reviews.
Columns: review_id, product_id, review_date, rating

Your task is to perform a semi join on these datasets to identify products that have been reviewed by customers. Use the product_id column for joining.

In this first part of our series, we’ve embarked on a journey to demystify data joins in R. We’ve covered the foundational types of joins that are essential for any data analyst: Inner Join, Left Join, Right Join, Full Join, and Semi Join. Through practical, real-life scenarios and step-by-step code examples, we explored how to combine datasets to gain valuable insights.

We’ve seen how Inner Joins help us find orders that have been paid, Left Joins reveal products that haven’t been sold, Right Joins show responses that didn’t belong to any campaign, Full Joins provide a comprehensive view of inventory across warehouses, and Semi Joins filter customers who have made orders. Each of these joins plays a critical role in data analysis, enabling us to connect disparate pieces of information in meaningful ways.

Next week, we’ll continue our exploration by diving into more advanced join techniques. We’ll cover Anti Joins, Cross Joins, Natural Joins, Self Joins, and Equi Joins, each with their own unique applications and benefits. Additionally, we’ll set some challenging exercises to reinforce your learning and build confidence in applying these joins to your own data projects.

Stay tuned for the next installment, where we continue to unlock the power of data joins in R and take your data analysis skills to the next level. Happy coding!

Joins Are No Mystery Anymore: Hands-On Tutorial — Part 1 was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Joins Are No Mystery Anymore: Hands-On Tutorial — Part 1

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, fellow R enthusiasts! I’m thrilled to share some fantastic updates to the TidyDensity package. These updates bring a wealth of new features, functions, and enhancements, making distribution analysis more comprehensive and efficient. Let’s dive into the details!

New FeaturesNegative Binomial Distribution util_negative_binomial_aic()*: Calculate the Akaike Information Criterion (AIC) for the negative binomial distribution. This function aids in model selection, helping you determine the best-fitting model for your data.

Zero-Truncated Negative Binomial Distribution util_zero_truncated_negative_binomial_param_estimate(): Estimate the parameters of the zero-truncated negative binomial distribution. * util_zero_truncated_negative_binomial_aic(): Calculate the AIC for the zero-truncated negative binomial distribution. * util_zero_truncated_negative_binomial_stats_tbl()*: Create a summary table for the zero-truncated negative binomial distribution.

Zero-Truncated Poisson Distribution util_zero_truncated_poisson_param_estimate(): Estimate the parameters of the zero-truncated Poisson distribution. * util_zero_truncated_poisson_aic(): Calculate the AIC for the zero-truncated Poisson distribution. * util_zero_truncated_poisson_stats_tbl()*: Create a summary table for the zero-truncated Poisson distribution.

F Distribution util_f_param_estimate(): Estimate the parameters for the F distribution. * util_f_aic()*: Calculate the AIC for the F distribution.

Zero-Truncated Geometric Distribution util_zero_truncated_geometric_param_estimate(): Estimate the parameters of the zero-truncated geometric distribution. * util_zero_truncated_geometric_aic(): Calculate the AIC for the zero-truncated geometric distribution. * util_zero_truncated_geometric_stats_tbl()*: Create a summary table for the zero-truncated geometric distribution.

Triangular Distribution util_triangular_aic()*: Calculate the AIC for the triangular distribution.

T Distribution util_t_param_estimate(): Estimate the parameters of the T distribution. * util_t_aic()*: Calculate the AIC for the T distribution.

Pareto Type I Distribution util_pareto1_param_estimate(): Estimate the parameters of the Pareto Type I distribution. * util_pareto1_aic(): Calculate the AIC for the Pareto Type I distribution. * util_pareto1_stats_tbl()*: Create a summary table for the Pareto Type I distribution.

Paralogistic Distribution util_paralogistic_param_estimate(): Estimate the parameters of the paralogistic distribution. * util_paralogistic_aic(): Calculate the AIC for the paralogistic distribution. * util_paralogistic_stats_tbl()*: Create a summary table for the paralogistic distribution.

Inverse Weibull Distribution util_inverse_weibull_param_estimate(): Estimate the parameters of the Inverse Weibull distribution. * util_inverse_weibull_aic(): Calculate the AIC for the Inverse Weibull distribution. * util_inverse_weibull_stats_tbl()*: Create a summary table for the Inverse Weibull distribution.

Inverse Pareto Distribution util_inverse_pareto_param_estimate(): Estimate the parameters of the Inverse Pareto distribution. * util_inverse_pareto_aic(): Calculate the AIC for the Inverse Pareto distribution. * util_inverse_pareto_stats_tbl()*: Create a summary table for the Inverse Pareto distribution.

Inverse Gamma Distribution util_inverse_burr_param_estimate(): Estimate the parameters of the Inverse Gamma distribution. * util_inverse_burr_aic(): Calculate the AIC for the Inverse Gamma distribution. * util_inverse_burr_stats_tbl()*: Create a summary table for the Inverse Gamma distribution.

Generalized Pareto Distribution util_generalized_pareto_param_estimate(): Estimate the parameters of the Generalized Pareto distribution. * util_generalized_pareto_aic(): Calculate the AIC for the Generalized Pareto distribution. * util_generalized_pareto_stats_tbl()*: Create a summary table for the Generalized Pareto distribution.

Generalized Gamma Distribution util_generalized_beta_param_estimate(): Estimate the parameters of the Generalized Gamma distribution. * util_generalized_beta_aic(): Calculate the AIC for the Generalized Gamma distribution. * util_generalized_beta_stats_tbl()*: Create a summary table for the Generalized Gamma distribution.

Zero-Truncated Binomial Distribution util_zero_truncated_binomial_stats_tbl(): Create a summary table for the Zero Truncated binomial distribution. * util_zero_truncated_binomial_param_estimate(): Estimate the parameters of the Zero Truncated binomial distribution. * util_zero_truncated_binomial_aic()*: Calculate the AIC for the Zero Truncated binomial distribution.

Minor Improvements and Fixes util_negative_binomial_param_estimate(): Updated to use optim() for parameter estimation, enhancing accuracy and efficiency. * quantile_normalize()*: Added names to columns when .return_tibble = TRUE for better readability and usability.

ConclusionThese updates significantly expand the functionality of TidyDensity, providing more tools for robust distribution analysis. Whether you’re working with standard or specialized distributions, these new functions and improvements will streamline your workflow and enhance your analytical capabilities.

I encourage you to explore these new features and see how they can benefit your projects. As always, your feedback is invaluable, so please share your thoughts and experiences with these updates. Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exciting New Updates to TidyDensity: Enhancing Distribution Analysis!

View Details

[This article was first published on R Archives - Dan Oehm | Gradient Descending, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Wrapping up season 46 and time for another release of survivoR. A few new things in this release including two new datasets.

Install from Git or CRAN:

install.packages("survivoR")devtools::install\_github("doehm/survivoR") As usual, if you find any issues, raise an issue on Git – survivoR issues

For non-R users, it’s free to download from Google Sheets.

News* New seasons: + US46 * New datasets added: + episode_summary – the summary of the episode from Wikipedia + challenge_summary – a summarised version of challenge_results for easy analysis * New fields added: + team on challenge_results – identifying the team that the castaways were on during the challenge

Episode SummaryI have included the episode summary extracts from Wikipedia that detail the events of the episode. It usually includes pre-challenge events and discussions of strategy, challenge description and results, strategy discussions amongst the tribe heading to Tribal Council, and the result. It may be interesting for NLP type applications.

```

episode_summary# A tibble: 647 × 4 version version_season episode episode_summary 1 US US01 1 "The two tribes paddled their way to their respective beaches on a raft with meager supplies. Upon ar… 2 US US01 2 "Following their Tribal Council, Tagi found its fish traps still empty. Disappointed that Rudy was no… 3 US US01 3 "At the Tagi tribe, Stacey still wanted to get rid of Rudy and tried to create a girl alliance to do … 4 US US01 4 "At Pagong, Ramona started to feel better after having been sick and tried to begin pulling her weigh… 5 US US01 5 "At Tagi, Dirk and Sean were still trying to fish instead of helping around camp, but to no avail. Su… 6 US US01 6 "Both tribes were wondering what the merge was going to be like. Tagi was afraid due to their numeric… 7 US US01 7 "The day after Pagong voted Joel out, one person from each tribe went to the opposite tribe's camp an… 8 US US01 8 "At camp, the remaining members of the former Pagong tribe felt vulnerable because the Tagi tribe had… 9 US US01 9 "While Richard was catching fish, the other players began to realize that nobody voted him out becaus…10 US US01 10 "Some people were happy that Jenna was voted out because she was getting on everyone's nerves. Everyo…# 637 more rows# Use print(n = ...) to see more rows `` Challenge SummaryWhen I was making some charts for The Sanctuary and specifically the challenge score I realised it was quite difficult to summarise thechallenge_results` table to the different types of challenges e.g. individual immunity. There are a few edge cases where there are combined challenges e.g. Team / Individual Immunity and Reward challenges where a team will win reward and the last person standing on each team wins immunity. So there are 3 winning outcomes – reward only, immunity only, and immunity and reward for the last person standing.

To make it easier to summarise I created challenge_summary. It looks like this…

```

challenge_summary# A tibble: 50,428 × 12 category version_season challenge_id challenge_type outcome_type tribe castaway_id castaway n_entities n_winners n_in_team won 1 All US01 1 Immunity and Reward Tribal Pagong US0002 B.B. 2 1 8 1 2 All US01 1 Immunity and Reward Tribal Pagong US0004 Ramona 2 1 8 1 3 All US01 1 Immunity and Reward Tribal Pagong US0006 Joel 2 1 8 1 4 All US01 1 Immunity and Reward Tribal Pagong US0007 Gretchen 2 1 8 1 5 All US01 1 Immunity and Reward Tribal Pagong US0008 Greg 2 1 8 1 6 All US01 1 Immunity and Reward Tribal Pagong US0009 Jenna 2 1 8 1 7 All US01 1 Immunity and Reward Tribal Pagong US0010 Gervase 2 1 8 1 8 All US01 1 Immunity and Reward Tribal Pagong US0011 Colleen 2 1 8 1 9 All US01 1 Immunity and Reward Tribal Tagi US0001 Sonja 2 1 8 010 All US01 1 Immunity and Reward Tribal Tagi US0003 Stacey 2 1 8 0# 50,418 more rows# Use print(n = ...) to see more rows `` The other challenge datasets can be easily joined to this table.challenge_summary` is not MECE, for example, the category contains ‘All’, ‘Individual’, ‘Individual Reward’, and ‘Individual Immunity’, to name a few. The results are counted separately for each category. You will need to filter for the right category before using the table.

Not every castaway is counted in every category. If they didn’t make it to the merge they didn’t compete in an individual challenge (except in some edge cases). Rather than their record being 0, they are not featured in that category. See Github for more details.

The post {survivoR} 2.3.3 is now available appeared first on Dan Oehm | Gradient Descending.

To leave a comment for the author, please follow the link and comment on their blog: R Archives - Dan Oehm | Gradient Descending.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {survivoR} 2.3.3 is now available

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This post was authored by Gregory Chen, Biostatistics and Research Decision Sciences (BARDS), MSD, Switzerland, and Ning Leng, Product Development Data Sciences (PDD), F. Hoffmann-La Roche, USA

On January 30, 2024, the R Consortium Submission Working Group made a presentation to Swissmedic in Bern, Switzerland, with 10 attendees in person and 50 online. It started with a motivation as to why to consider using open source and specifically R for regulatory submissions. The group then proceeded to show cases of the pilot 1 and 2 submission to FDA.

The conclusion was an insightful discussion for about 20 minutes with the participants on the lessons learned, key factors to sort in line for broader adoption of R and Shiny for regulatory submissions, and what would be most added value for a regulatory shiny app, namely

How to deploy the submitted R package and Shiny App to guarantee the clinical outcomes can be smoothly reproduced on the regulatory side
What would be the ultimate purpose of a regulatory shiny app, and what are value-added features? Should the app only focus on offering interactivities to facilitate the review of tables, figures, and listings in CSR, or should it also include designed features to enable exploratory, descriptive analysis (e.g., for subgroups) to certain degrees, which may greatly shorten back-and-forth inquiry between regulator and drug developer?
Validation and version traceability of dependent open source R packages used in the submission package
How to leverage existing and emerging cross-industrial initiatives (e.g. R consortium) in the open source space to support and ease the potential technical issue during the adoption of R for submission

Accompanying this post, the full presentation slide deck is made publicly available here, inviting further exploration and discussion.

The R Consortium’s presentation at Swissmedic represents a hopeful step toward more interactive, efficient, and transparent regulatory submissions. As the conversation between the R Consortium and regulatory bodies continues, our future collection of pilot projects hopefully will offer richer examples and templates to our growing R community within the pharmaceutical sector, spanning both regulatory and drug developer sides.

To find out more about the R Consortium Submission Working Group, please see: https://rconsortium.github.io/submissions-wg/

The post One More Step Forward: The R Consortium Submission Working Group’s Presentation to Swissmedic on Regulatory Submission using R and Shiny appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: One More Step Forward: The R Consortium Submission Working Group’s Presentation to Swissmedic on Regulatory Submission using R and Shiny

View Details

[This article was first published on R on kieranhealy.org, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’m about to start work on a second edition of my Data Visualization book. As a result I continue to mess around with stuff I’m considering including in a new edition. The other day I pulled some block-level Census data and drew a map of the distribution of people of color in New York City, which is to say the share of the population that reports being something other than Non-Hispanic White. I’ve polished that map a little more and drawn some additional ones. As before, the main substantive issues to bear in mind are (a) how the Census measures and classifies race and ethnicity, and (b) the new and I am inclined to think unwise practice of “differential privacy”. The main thing to know about the former is that in the present US Census Bureau schema people of Hispanic or Latino origin may be of any race. (I follow the Census’s nomenclature here, by the way, as it’s their data.) This is the reason that “Non-Hispanic White” is a category for example. The main thing to know about the latter is that it deliberately introduces noise into counts within units where the observed N is small.

The other thing to remember is every choropleth maker’s oldest friend, the Modifiable Areal Unit Problem. Census Blocks are the smallest spatial unit we can make a choropleth map of, but they’re not “real”, so to speak.

As usual, the tools used to make these maps are R, ggplot, and the tidycensus and sf packages. Together, of course, with the really phenomenal range of data made available by the Census Bureau API.

For these new versions I decided to add a little more context by sketching in some of the coastline, particularly the outline of New Jersey to the west, along with a reminder that Long Island continues to exist past the Queens border, etc These coastal outlines come from the NOAA CUSP maps.

New York City’s POC population.

New York City population, percent reporting ‘Black Alone’ to race question.

New York City population, percent Hispanic/Latino origin.

New York City population, precent reporting ‘Asian Alone’ to race question.

PDFs of these maps:

Percent POC
Percent Black Alone
Percent Hispanic/Latino
Percent Asian Alone

To leave a comment for the author, please follow the link and comment on their blog: R on kieranhealy.org.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Race and Ethnicity in New York City

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, R enthusiasts!

Today, I’m excited to introduce a new function in the healthyR.data package: get_provider_meta_data(). This function is excellent for anyone working with healthcare datasets, making it easy to fetch and filter metadata from the Centers for Medicare & Medicaid Services (CMS) repository.

OverviewThe get_provider_meta_data() function simplifies the process of retrieving and managing metadata for healthcare datasets. By allowing users to filter data based on various criteria, it streamlines data management and enhances analytical capabilities.

Syntax and ArgumentsThe function syntax is straightforward and highly customizable:

.identifier: A dataset identifier to filter the data.
.title: A title to filter the data.
.description: A description to filter the data.
.keyword: A keyword to filter the data.
.issued: A date when the dataset was issued to filter the data.
.modified: A date when the dataset was modified to filter the data.
.released: A date when the dataset was released to filter the data.
.theme: A theme to filter the data.
.media_type: A media type to filter the data.

What It ReturnsThe function returns a tidy tibble containing metadata about the datasets. This tibble includes the following columns:

identifier
title
description
keyword
issued
modified
released
theme
media_type
download_url
contact_fn
contact_email
publisher_name

DetailsWhen you call get_provider_meta_data(), it fetches JSON data from the CMS metadata URL. The function then processes this data by: 1. Selecting relevant columns. 2. Unnesting nested lists. 3. Cleaning column names. 4. Processing dates and media types for enhanced usability.

Practical ExampleLet’s walk through an example to see how get_provider_meta_data() works in action.

Suppose we want to retrieve metadata for a dataset based upong a specific data identifier? Here’s how we can do it:

library(healthyR.data)library(dplyr)library(tidyr)# Retrieve metadata for a datset with identifier "3614-1eef"get\_provider\_meta\_data(.identifier = "3614-1eef") |> glimpse()

Rows: 1Columns: 16$ identifier <chr> "3614-1eef"$ title <chr> "Addiction Medicine Office Visit Costs"$ description <chr> "Returns addiction medicine office visit costs per zip…$ keyword <list> "Addiction Medicine"$ issued <date> 2022-07-11$ modified <date> 2022-07-11$ released <date> 2023-09-28$ landing\_page <chr> "https://data.medicare.gov/provider-data/dataset/3614-…$ theme <list> "Physician office visit costs"$ access\_level <chr> "public"$ archive\_exclude <lgl> NA$ contact\_fn <chr> "PPL Dataset"$ contact\_email <chr> "PPL\_Dataset@cms.hhs.gov"$ publisher\_name <chr> "Centers for Medicare & Medicaid Services (CMS)"$ download\_url <chr> "https://data.cms.gov/provider-data/sites/default/file…$ media\_type <chr> "text/csv" In this example, we are filtering the metadata based on the dataset identifier “3614-1eef”. The glimpse() function allows us to view the structure of the resulting tibble.

Now, what if we want to filter data that meets a certain keyword? Here’s how we can do that:

provider\_data\_tbl <- get\_provider\_meta\_data(.keyword = "medic")# Let's see all the titles that contain the keyword "medic"provider\_data\_tbl[["title"]]

[1] "Addiction Medicine Office Visit Costs" [2] "Emergency Medicine Office Visit Costs" [3] "Geriatric Medicine Office Visit Costs" [4] "Internal Medicine Office Visit Costs" [5] "Medical Genetics and Genomics Office Visit Costs" [6] "Medical Oncology Office Visit Costs" [7] "Medical Toxicology Office Visit Costs" [8] "Nuclear Medicine Office Visit Costs" [9] "Osteopathic Manipulative Medicine Office Visit Costs" [10] "Pediatric Medicine Office Visit Costs" [11] "Physical Medicine and Rehabilitation Office Visit Costs" [12] "Preventive Medicine Office Visit Costs" [13] "Sleep Medicine Office Visit Costs" [14] "Sports Medicine Office Visit Costs" [15] "Undersea and Hyperbaric Medicine Office Visit Costs" [16] "Medical Equipment Suppliers" [17] "Home Health Care - Patient Survey (HHCAHPS) 2022Q4 to 2023Q3" [18] "Home Health Care - Patient Survey (HHCAHPS) National Data 2022Q4 to 2023Q3"[19] "Home Health Care - Patient Survey (HHCAHPS) State Data 2022Q4 to 2023Q3" [20] "Home Health Care - Patient Survey (HHCAHPS) Measure Dates 2022Q4 to 2023Q3"[21] "Medicare Spending Per Beneficiary - Hospital Additional Decimal Places" [22] "Hospital Value-Based Purchasing (HVBP) - Efficiency Scores" [23] "Medicare Hospital Spending by Claim" [24] "Medicare Spending Per Beneficiary - Hospital" [25] "Medicare Spending Per Beneficiary - National" [26] "Medicare Spending Per Beneficiary - State"

```

Now let's group them by themeprovider_data_tbl |> count(theme, sort = TRUE) |> unnest(cols = c(theme))

```

A tibble: 4 × 2 theme n 1 Physician office visit costs 152 Hospitals 63 Home health services 44 Supplier directory 1

``` In this example, the metadata is filtered based on the keyword “medic”. We then extract the titles containing the keyword and group them by theme to see the distribution of themes in the filtered data. Notice that we filtered the keyword not on a full word but on a partial match, which can be useful for broad searches.

Benefits of Using get_provider_meta_data()This function is particularly useful for:

Data Scientists and Analysts: Quickly finding relevant datasets without manually searching through large repositories.
Healthcare Researchers: Accessing comprehensive metadata to support research and analysis.
Developers: Integrating CMS metadata retrieval into applications or workflows with minimal effort.

ConclusionThe get_provider_meta_data() function is a robust tool for anyone working with healthcare data. It not only saves time but also provides a cleaner, more efficient way to manage and analyze dataset metadata.

Give it a try and see how it can enhance your data workflows. Happy coding!

Feel free to share your experiences and any creative ways you’re using this function in the comments below. Until next time, keep exploring and innovating with R!

Steve

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing get_provider_meta_data() in healthyR.data

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Run a specific code block in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Run a specific code block in R, We will cover the basic syntax and definitions of the switch function, as well as two examples of how to apply it.

Definition and Basic R Syntax

The switch function in R runs a specific code block and returns its result. The basic syntax is:

switch(select\_block, "code\_block\_1", "code\_block\_2", "code\_block\_3") Example 1: Basic Application of switch() Function

In this example, we will demonstrate a simple application of the switch function.

We have three code blocks that return the character strings “first”, “second”, and “third”.

We will use the switch function to select one of these code blocks and return its result.

Let’s assume that we want to select the second code block. We can do this using the switch function:

Predictive Modeling and Data Science » Data Science Tutorials

switch(2, "first", "second", "third")# "second" Example 2: Using switch within a for-Loop

We can also use the switch function within a for-loop. Let’s assume that we have a vector that specifies several different code blocks that we want to run:

my\_input <- c(3, 1, 2, 4, 1) We can then use this vector to return the output of different code blocks:

for(i in 1:length(my\_input)) { my\_output <- switch(my\_input[i], "first", "second", "third") print(my\_output)}# [1] "third"# [1] "first"# [1] "second"# NULL# [1] "first" Note that our input vector contained the value 4, but we have defined only three code blocks within the switch function.

For that reason, the switch statement returned NULL at the index position of the value 4.

ConclusionIn this tutorial, we have demonstrated how to apply the switch statement in R.

We have covered two examples of how to use the switch function, including a basic application and using it within a for-loop.

By using the switch function, you can easily choose between different code blocks in your R programming code.

Select Columns by Index Using dplyr
Summary statistics in R
Ogive curve in R
How to perform the Sobel test in R
Apply Function into two input vectors in R
How to read or export large datasets in R
How to Calculate a Bootstrap Standard Error in R
Cluster Sampling in R With Examples

The post Run a specific code block in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Run a specific code block in R

View Details

[This article was first published on rstats on Irregularly Scheduled Programming, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I saw a toot celebrating a short, clean implementation of a square rootfinding algorithm and wanted to dig a bit deeper into how it works, with adiversion into some APL.

This was the toot from JimGardner

Doubly pleased with myself.

Been doing the Tour of Go. Got to the section where you make a square root function, which should return once the calculated value stops changing. Struggled for ages. Trimmed and trimmed. Until finally… this!

The calculation for z was given, and I don’t understand it at all. But I don’t care. It was a total mess when I started and has turned out quite neat. I’m very satisfied.

But why “doubly pleased”? Because I’ve been solely using Neovim so far for Go!!

package main import ( "fmt" ) func Sqrt(x float64) float64 {z := 1.0 for { y := z z -= (z*z - x) / (2 * z) if z == y { return z } }} func main() { fmt.Println(Sqrt(16)) } It’s a nice, not-too-complicated algorithm to play with, and I agree it’s hard tosee why it works for this application, so I thought it would be neat to walkthrough that.

What we’re trying to solve here is the function (y = x^2) which we could write as(f(x) = x^2 – y) for which we want the value (x) where (f(x) = 0).

Newton’s Method is an iterativemethod for solving equations of this type (not all equations, mind you – I have anentire chapter of my PhD thesis discussing exactly why it can’t be used tosolve the equations I was solving that my supervisor insisted it could). It worksby using the slope (derivative) at a point to guide towards a solution. The formulafor the updated value (x_{n+1}) given some guess (x_n) is

[x_{n+1} = x_n – \frac{f(x_n)}{f'(x_n)}]where (f'(x)) is the derivative of the function (f) at the point (x). For (f(x) = x^2 – y)the derivative is (f'(x) = 2x) so we can substitute this and (f(x)) into the above formula

[x_{n+1}=x_n−\frac{x_n^2-y}{2x_n}]This is what the Go code calculates; given an initial guess of (x_n = 1) it calculates the next value as

x = x - (x*x - y) / (2 * x) where, here, y is the value we’re finding the square root of.

In R this could be written as

SQRT <- function(x) { z <- 1 while (TRUE) { y <- z z <- z - (z*z - x)/(2*z) if (abs(y -z) < 0.0001) return(z) }} (since base::sqrt is already defined) where I’ve used a tolerance rather thanrelying on exact numerical equality. The while(TRUE) construct is equivalentto Go’s for {} syntax; an infinite loop.

R actually has another way to write that which is even closer; repeat {}

SQRT <- function(x) { z <- 1 repeat { y <- z z <- z - (z*z - x)/(2*z) if (abs(y -z) < 0.0001) return(z) }} One might notice that this approach requires essentially squaring a value, whichis hardly expensive, but we can simplify and cancel out (x_n), so

[x_{n+1}=\frac{x_n-\frac{y}{x_n}}{2}]in which case we have

SQRT <- function(x) { z <- 1 repeat { y <- z z <- (z + x/z)/2 if (abs(y -z) < 0.0001) return(z) }} One of the reasons I wanted to dig into this was the fact that it’s a convergence…

In APL the power operator (⍣ aplwi)applies a function some specified number of times, so

f ⍣n x applies f to x n times, i.e. (f⍣3)x produces f(f(f(x))).

It can also be used as ⍣= where it will continue to apply the function untilthe output no longer changes (is equal). A classic example is thegolden ratio; take the reciprocalthen add 1 until it converges, i.e.

[x_{n+1} = 1+\frac{1}{x_n}]

which you can try for yourselfhere

1+∘÷⍣=11.618033989 In this, +∘÷ is the (tacit) function created bycomposing (∘) ‘addition of 1’ (1+, a partial application of a function) and‘reciprocal’ (÷), which is iterated until it no longer changes (withinmachine precision).

Iterating until convergence is exactly what we want, since we’re looking for thevalue satisfying

[x_n = x_{n+1}=\frac{x_n-\frac{y}{x_n}}{2}]APL uses ⍵ as the right argument placeholder and ⍺ as the left, so thefunction we want to apply repeatedly to the right argument is

{⍵-(((⍵×⍵)-⍺)÷(2×⍵))} If we provide 1 as the right argument (the start value) and 16 as the leftargument, we get

16{⍵-(((⍵×⍵)-⍺)÷(2×⍵))}⍣=14 You can try this out yourself at tryapl.org(link should load that expression).

We can turn that into a function, once again using the argument placeholder

sqrt←{⍵{⍵-(((⍵×⍵)-⍺)÷(2×⍵))}⍣=1} sqrt 255 sqrt 819 Taking the simplification above, we can write this a bit shorter as

sqrt←{⍵{(⍵+(⍺÷⍵))÷2}⍣=1} sqrt 14412 As clean as the Go code looks, I think there’s a certain beauty to being ableto write this in just 20 characters. It’s not for everyone, I get that.

I love these opportunities to learn a bit more about languages.

If you have comments, suggestions, or improvements, as always, feel free to usethe comment section below, or hit me up onMastodon.

devtools::session_info() ```

─ Session info ───────────────────────────────────────────────────────────────## setting value## version R version 4.3.3 (2024-02-29)## os Pop!_OS 22.04 LTS## system x86_64, linux-gnu## ui X11## language (EN)## collate en_AU.UTF-8## ctype en_AU.UTF-8## tz Australia/Adelaide## date 2024-05-29## pandoc 3.1.11 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)## ## ─ Packages ───────────────────────────────────────────────────────────────────## package * version date (UTC) lib source## blogdown 1.18 2023-06-19 [1] CRAN (R 4.3.2)## bookdown 0.36 2023-10-16 [1] CRAN (R 4.3.2)## bslib 0.6.1 2023-11-28 [3] CRAN (R 4.3.2)## cachem 1.0.8 2023-05-01 [3] CRAN (R 4.3.0)## callr 3.7.3 2022-11-02 [3] CRAN (R 4.2.2)## cli 3.6.2 2023-12-11 [3] CRAN (R 4.3.2)## crayon 1.5.2 2022-09-29 [3] CRAN (R 4.2.1)## devtools 2.4.5 2022-10-11 [1] CRAN (R 4.3.2)## digest 0.6.34 2024-01-11 [3] CRAN (R 4.3.2)## ellipsis 0.3.2 2021-04-29 [3] CRAN (R 4.1.1)## evaluate 0.23 2023-11-01 [3] CRAN (R 4.3.2)## fastmap 1.1.1 2023-02-24 [3] CRAN (R 4.2.2)## fs 1.6.3 2023-07-20 [3] CRAN (R 4.3.1)## glue 1.7.0 2024-01-09 [3] CRAN (R 4.3.2)## htmltools 0.5.7 2023-11-03 [3] CRAN (R 4.3.2)## htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.3.2)## httpuv 1.6.12 2023-10-23 [1] CRAN (R 4.3.2)## icecream 0.2.1 2023-09-27 [1] CRAN (R 4.3.2)## jquerylib 0.1.4 2021-04-26 [3] CRAN (R 4.1.2)## jsonlite 1.8.8 2023-12-04 [3] CRAN (R 4.3.2)## knitr 1.45 2023-10-30 [3] CRAN (R 4.3.2)## later 1.3.1 2023-05-02 [1] CRAN (R 4.3.2)## lifecycle 1.0.4 2023-11-07 [3] CRAN (R 4.3.2)## magrittr 2.0.3 2022-03-30 [3] CRAN (R 4.2.0)## memoise 2.0.1 2021-11-26 [3] CRAN (R 4.2.0)## mime 0.12 2021-09-28 [3] CRAN (R 4.2.0)## miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.3.2)## pkgbuild 1.4.2 2023-06-26 [1] CRAN (R 4.3.2)## pkgload 1.3.3 2023-09-22 [1] CRAN (R 4.3.2)## prettyunits 1.2.0 2023-09-24 [3] CRAN (R 4.3.1)## processx 3.8.3 2023-12-10 [3] CRAN (R 4.3.2)## profvis 0.3.8 2023-05-02 [1] CRAN (R 4.3.2)## promises 1.2.1 2023-08-10 [1] CRAN (R 4.3.2)## ps 1.7.6 2024-01-18 [3] CRAN (R 4.3.2)## purrr 1.0.2 2023-08-10 [3] CRAN (R 4.3.1)## R6 2.5.1 2021-08-19 [3] CRAN (R 4.2.0)## Rcpp 1.0.11 2023-07-06 [1] CRAN (R 4.3.2)## remotes 2.4.2.1 2023-07-18 [1] CRAN (R 4.3.2)## rlang 1.1.3 2024-01-10 [3] CRAN (R 4.3.2)## rmarkdown 2.25 2023-09-18 [3] CRAN (R 4.3.1)## rstudioapi 0.15.0 2023-07-07 [3] CRAN (R 4.3.1)## sass 0.4.8 2023-12-06 [3] CRAN (R 4.3.2)## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.2)## shiny 1.7.5.1 2023-10-14 [1] CRAN (R 4.3.2)## stringi 1.8.3 2023-12-11 [3] CRAN (R 4.3.2)## stringr 1.5.1 2023-11-14 [3] CRAN (R 4.3.2)## urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.3.2)## usethis 2.2.2 2023-07-06 [1] CRAN (R 4.3.2)## vctrs 0.6.5 2023-12-01 [3] CRAN (R 4.3.2)## xfun 0.41 2023-11-01 [3] CRAN (R 4.3.2)## xtable 1.8-4 2019-04-21 [1] CRAN (R 4.3.2)## yaml 2.3.8 2023-12-11 [3] CRAN (R 4.3.2)## ## [1] /home/jono/R/x86_64-pc-linux-gnu-library/4.3## [2] /usr/local/lib/R/site-library## [3] /usr/lib/R/site-library## [4] /usr/lib/R/library## ## ──────────────────────────────────────────────────────────────────────────────

``` To leave a comment for the author, please follow the link and comment on their blog: rstats on Irregularly Scheduled Programming.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Iterative Square Root

View Details

[This article was first published on pharmaverse blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Cytel is the first CRO involved in the {admiral} open-source project. Having the ambition to demonstrate our skills in the open-source projects, it was with lots of excitement that we accepted the challenge to enter the {admiral} development team family. We are thus thrilled to share with you this new & challenging experience.

What experience we had when we startedAs a starting point, we went through the {admiral} documentation, and how to create a package websites to get familiar with the work and processes. There is no need to memorize everything, it’s only to get an overview and know that the documentation is available and can be reviewed at any time. Then, we were encouraged to take on and resolve our first dummy issue which was to create an initial and easy function. This was a concrete example to proceed with the full process of creation, documentation and building site. What was amazing is that we have received so much support from the {admiral} Community on how to get started with this dummy issue.

Moreover, weekly meetings were set up for the first few weeks between ourselves and one of the {admiral} team members to support our integration in the team and to address any queries or issues we may have. This helped us a lot to familiarize ourselves with the processes and requirements of this open-source project.

In addition to these meetings, the full core development team meets bi-weekly to discuss and dispatch the tasks within the team, as well as issues newcomers are facing during their on-boarding. Before a new version is released, these bi-weekly meetings becomes a weekly meeting.

All this support made us feel good and gave us the confidence in tackling and fixing the issues, as well as with the entire {admiral} process.

How we upskilledDuring this ongoing open-source experience, we have had the opportunity to dive into a couple of existing issues that have expanded our technical knowledge. We have worked with various packages such as {devtools} and {pkgdown} that are themselves used for the creation and maintenance of R packages. We also sharpened our problem-solving abilities. We received guidance from {admiral} team’s expertise on how to address the issues and improve our implementation; these helped us improve the quality of our code and broaden our understanding of best practices in development with their insights and suggestions.

{admiral} is not only developing code. It is also the development of the corresponding documentation and vignettes, as well as maintaining the online user guideline. This is also a forum where colleagues and users share opinions, what they think, how they would resolve an issue or how it is handled in their own company. There’s also a lot of communication around individual topics.

While completing tasks or resolving bugs which were identified in existing functions, it requires digging into the corresponding functions and learning how exactly they work. By becoming an {admiral} developer, we also become an {admiral} user.

The tasks are numerous and can correspond to any programming level.

What have been the challenges/learningsIn terms of challenges, setting up the environment was time consuming. Of course, the main environment is company dependent, but following the first setup, there can be challenges to make the {admiral} package work as expected.

Some packages are required to run single steps. As the {renv} package has been deprecated for the use of {admiral} earlier this year, a single step is now necessary: devtools::install_dev_deps(dependencies = TRUE). You just need to know it. Once all necessary packages are well installed and launched, it can be very easy to start developing some code or resolve some issues.

Also, at the beginning, Github (where {admiral} is developed and discussed) can be scary and may seem complicated due to the potential for mistakes. It’s pretty hard to make a mistake that might impact the whole project as you can’t push to main directly. While going through the processes and learning how it exactly works, it’s most of the time a simple process, and while working on the right parallel branch, nothing serious can happen. This is the most important point of working with Github: work on the right parallel branch, never/rarely on the main branch directly!

Another challenge is to understand the automatic CI/CD (Continuous Integration / Continuous Delivery) checks and their dependencies with other packages such as {admiraldev}. The CI/CD checks are used to ensure any code updates we make won’t break anything else in the codebase. These checks are implemented within the {admiralci} package and used in {admiral} when you create a Pull request. Those updates can thus have some impacts on the {admiral} package itself.

ReflectionOn reflection, we didn’t see a substantial difference when coming from a CRO rather than coming from a Pharma company. But it might actually be the exposure to R which could differ. Indeed, while coming from a CRO, the tasks and projects depend on the clients we work for. We have been lucky to work on a project using R and Github for the past two years before joining the {admiral} team. We have to admit that this previous experience helped us a lot during the on-boarding.

What have we enjoyedBecoming a co-developer and active contributor to {admiral} was an excellent opportunity for us to learn and to grow, and a challenge which we accepted with enthusiasm. By utilizing our backend expertise, we demonstrated our ability to adapt to new programming languages, frameworks, and tools. By working on this open-source package, we were able to expand our technical horizons and strengthen our problem-solving abilities.

Overall, our experience as a developer working on open-source projects has equipped us with essential skills in teamwork, communication, conflict resolution, and leadership.

What we enjoyed the most in this cross-company project is the team spirit and sharing of knowledge. You learn that there is no stupid question, and in addition to the mentoring that you receive at the beginning you are free to ask any questions and can be ensured to receive various valuable answers and suggestions. The process is also quite meticulous and painstakingly documented. You can’t really make any mistakes, and if something goes wrong, there won’t be any impact on the project: before merging your own updates to the main branch, a Pull Request with a reviewer and an approval is necessary. Some updates can be requested before being approved and merged as needed, which makes us feel more comfortable and confident.
As there is no such thing as zero risk, if such a situation occurs, it will most likely be discovered and can be fixed for the next release.

Comments are always welcome, whatever it is, but our favorite one in the process is the “Well done!”: Pat yourself on the back for a job well done! Much love to your accomplishment!

We are looking forward to praising ourselves over and over again.

Last updated2024-05-29 13:09:13.550351

DetailsSource, Session info

ReuseCC BY 4.0CitationBibTeX citation: @online{gautier2024, author = {Gautier, Fanny and Patil, Lina}, title = {Our Experience as New Admiral Developers, Coming from a {CRO}}, date = {2024-05-29}, url = {https://pharmaverse.github.io/blog/posts/2024-05-21\_our\_experience\_a.../our\_experience\_as\_new\_admiral\_developers.html}, langid = {en}} For attribution, please cite this work as:Gautier, Fanny, and Lina Patil. 2024. “Our Experience as NewAdmiral Developers, Coming from a CRO.” May 29, 2024. https://pharmaverse.github.io/blog/posts/2024-05-21_our_experience_a…/our_experience_as_new_admiral_developers.html. To leave a comment for the author, please follow the link and comment on their blog: pharmaverse blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Our experience as new admiral developers, coming from a CRO

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#185–186

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Today we need to add index for employees. De facto, we need to assign them their number in order of their occurance, not looking at fact how many times their appear in document. I can tell you that at the beginning it was looking hard, but I finished presenting 3 approaches, all of them really short in matter of code. Check it out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_185.xlsx", range = "A1:B13")test = read\_excel("Power Query/PQ\_Challenge\_185.xlsx", range = "D1:F13") Transformation — approach 1 — position in vector result = input %>% mutate(Index = map\_dbl(Emp, ~ which(unique(Emp) == .x)[1]), .by = Group) Transformation — approach 2 — ranking with factorization result2 = input %>% mutate(Index = dense\_rank(factor(Emp, levels = unique(Emp))), .by = Group) Transformation — approach 3 — simple sorting of factors result3 = input %>% mutate(Index = as.integer(factor(Emp, levels = unique(Emp))), .by = Group) Validation all.equal(result, test)# [1] TRUEall.equal(result2, test)# [1] TRUEall.equal(result3, test)# [1] TRUE Puzzle #186Sometimes in logistics there are unexpected events, and probably someone decided to be aware of delivery not only on exact day, but also one day earlier and one day later. And now we need to apply those notes into calendar, but we can put only one delivery for one day. So exact date is for us more important. Check how I managed to completed this task.

Loading libraries and data library(tidyverse)library(readxl)library(janitor)input1 = read\_excel("Power Query/PQ\_Challenge\_186.xlsx", range = "A1:A30") %>% clean\_names()input2 = read\_excel("Power Query/PQ\_Challenge\_186.xlsx", range = "C1:D7") %>% clean\_names()test = read\_excel("Power Query/PQ\_Challenge\_186.xlsx", range = "F1:H30") %>% clean\_names() Transformation marked\_dates <- input2 %>% mutate( preceding\_date = delivery\_date - days(1), following\_date = delivery\_date + days(1) ) %>% pivot\_longer( cols = c(preceding\_date, delivery\_date, following\_date), names\_to = "type", values\_to = "marked\_date" ) %>% mutate(type = factor(type, levels = c("preceding\_date", "following\_date","delivery\_date"), ordered = TRUE))calendar\_with\_markings <- input1 %>% left\_join(marked\_dates, by = c("calendar\_date" = "marked\_date")) %>% mutate(marked = !is.na(vendor)) %>% group\_by(calendar\_date) %>% mutate(proper\_type = max(type, na.rm = TRUE)) %>% ungroup() %>% filter(proper\_type == type | is.na(proper\_type)) %>% mutate(delivery\_date = case\_when( type == "delivery\_date" ~ calendar\_date, type == "preceding\_date" ~ calendar\_date + days(1), type == "following\_date" ~ calendar\_date - days(1) )) %>% select(calendar\_date, delivery\_date, vendor) Validation identical(test, calendar\_with\_markings)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHey, R users! Today, we’re going to look at a great new addition to the healthyR.data package—the get_cms_meta_data() function! This function is a helpful tool for retrieving and analyzing metadata from CMS (Centers for Medicare & Medicaid Services) datasets. Whether you’re a healthcare analyst, data scientist, or R programming fan, you’ll find this function very useful. Let’s break it down and explore how it works.

Overview of get_cms_meta_data()The get_cms_meta_data() function lets you retrieve metadata from CMS datasets easily. You can customize your search using various parameters, ensuring you get precisely the data you need. Here’s the syntax:

get\_cms\_meta\_data( .title = NULL, .modified\_date = NULL, .keyword = NULL, .identifier = NULL, .data\_version = "current", .media\_type = "all") Arguments:* .title: Search by title. * .modified_date: Search by modified date (format: “YYYY-MM-DD”). * .keyword: Search by keyword. * .identifier: Search by identifier. * .data_version: Choose between “current”, “archive”, or “all”. Default is “current”. * .media_type: Filter by media type (“all”, “csv”, “API”, “other”). Default is “all”.

Return Value:A tibble containing data links and relevant metadata about the datasets.

Details:The function fetches JSON data from the CMS data URL and extracts relevant fields to create a tidy tibble. It selects specific columns, handles nested lists by unnesting them, cleans column names, and processes dates and media types to make the data more useful for analysis. The columns in the returned tibble include:

title
description
landing_page
modified
keyword
described_by
fn
has_email
identifier
start
end
references
distribution_description
distribution_title
distribution_modified
distribution_start
distribution_end
media_type
data_link

Practical ExamplesLet’s see the get_cms_meta_data() function in action with a couple of examples.

Example 1: Basic UsageFirst, we’ll load the necessary libraries and fetch some metadata:

```

Library Loadslibrary(healthyR.data)library(dplyr)# Get datacms_data <- get_cms_meta_data()glimpse(cms_data)

```

Rows: 107Columns: 19$ title <chr> "Accountable Care Organization Participants",…$ description <chr> "The Accountable Care Organization Participan…$ landing\_page <chr> "https://data.cms.gov/medicare-shared-savings…$ modified <date> 2024-01-29, 2024-04-23, 2024-01-12, 2024-01-…$ keyword <list> <"Medicare", "Value-Based Care", "Coordinate…$ described\_by <chr> "https://data.cms.gov/resources/accountable-c…$ fn <chr> "Shared Savings Program - CM", "Shared Saving…$ has\_email <chr> "SharedSavingsProgram@cms.hhs.gov", "SharedSa…$ identifier <chr> "https://data.cms.gov/data-api/v1/dataset/976…$ start <date> 2014-01-01, 2017-01-01, 2021-01-01, 2021-01-…$ end <date> 2024-12-31, 2024-12-31, 2021-12-31, 2021-12-…$ references <chr> "https://data.cms.gov/resources/acos-aco-part…$ distribution\_description <chr> "latest", "latest", "latest", "latest", "late…$ distribution\_title <chr> "Accountable Care Organization Participants",…$ distribution\_modified <date> 2024-01-29, 2024-04-23, 2024-01-12, 2024-01-…$ distribution\_start <date> 2024-01-01, 2024-01-01, 2021-01-01, 2021-01-…$ distribution\_end <date> 2024-12-31, 2024-12-31, 2021-12-31, 2021-12-…$ media\_type <chr> "API", "API", "API", "API", "API", "API", "AP…$ data\_link <chr> "https://data.cms.gov/data-api/v1/dataset/976…

```

Attributesatb <- attributes(cms_data)atb$names

```

[1] "title" "description" [3] "landing\_page" "modified" [5] "keyword" "described\_by" [7] "fn" "has\_email" [9] "identifier" "start" [11] "end" "references" [13] "distribution\_description" "distribution\_title" [15] "distribution\_modified" "distribution\_start" [17] "distribution\_end" "media\_type" [19] "data\_link"

atb$class

[1] "cms\_meta\_data" "tbl\_df" "tbl" "data.frame"

atb$url

[1] "https://data.cms.gov/data.json"

atb$date\_retrieved

[1] "2024-05-28 10:20:18 EDT"

atb$parameters

$.data\_version[1] "current"$.media\_type[1] "all"$.titleNULL$.modified\_dateNULL$.keywordNULL$.identifierNULL In this example, we’re simply calling get_cms_meta_data() without any parameters. This fetches the default dataset metadata. The glimpse() function from the dplyr package provides a quick overview of the data structure.

Example 2: Custom Search by Keyword and TitleNow, let’s refine our search by specifying a keyword and title:

get\_cms\_meta\_data( .keyword = "nation", .title = "Market Saturation & Utilization State-County") |> glimpse()

Rows: 1Columns: 19$ title <chr> "Market Saturation & Utilization State-County"$ description <chr> "The Market Saturation and Utilization State-…$ landing\_page <chr> "https://data.cms.gov/summary-statistics-on-u…$ modified <date> 2024-04-02$ keyword <list> <"National", "States & Territories", "Countie…$ described\_by <chr> "https://data.cms.gov/resources/market-satur…$ fn <chr> "Market Saturation - CPI"$ has\_email <chr> "MarketSaturation@cms.hhs.gov"$ identifier <chr> "https://data.cms.gov/data-api/v1/dataset/89…$ start <date> 2023-10-01$ end <date> 2023-12-31$ references <chr> "https://data.cms.gov/resources/market-satura…$ distribution\_description <chr> "latest"$ distribution\_title <chr> "Market Saturation & Utilization StateCounty"$ distribution\_modified <date> 2024-04-02$ distribution\_start <date> 2023-10-01$ distribution\_end <date> 2023-12-31$ media\_type <chr> "API"$ data\_link <chr> "https://data.cms.gov/data-api/v1/dataset/890… In this example, we filter the metadata by the keyword “nation” and the title “Market Saturation & Utilization State-County”. The pipe operator (|>) is used to pass the result directly into the glimpse() function for a quick preview.

Breaking Down the CodeLet’s break down the code blocks to understand what they’re doing:

Basic Usage1. Load Libraries:

library(healthyR.data)library(dplyr) We load the healthyR.data package to access the get_cms_meta_data() function and the dplyr package for data manipulation. 2. Fetch Metadata:

cms\_data <- get\_cms\_meta\_data() We call get_cms_meta_data() without any parameters to get the default dataset metadata. 3. Preview Data:

glimpse(cms\_data) The glimpse() function gives us a quick look at the structure and contents of the fetched metadata.

Custom Search1. Custom Search Call:

get\_cms\_meta\_data( .keyword = "nation", .title = "Market Saturation & Utilization State-County") |>glimpse() Here, we call get_cms_meta_data() with specific parameters for keyword and title to narrow down our search. The result is passed to glimpse() using the pipe operator for an immediate preview.

ConclusionThe get_cms_meta_data() function is a versatile and flexible tool for accessing CMS metadata, making your data analysis tasks more efficient and effective. Whether you’re looking for specific datasets or just exploring the available metadata, this function has got you covered.

Try out get_cms_meta_data() in your next R project and explore the potential of CMS data with ease! Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Unveiling the Power of get_cms_meta_data() in healthyR.data

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Replace first match in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Replace first match in R, This article explains how to replace patterns in characters in R using the sub() and gsub() functions.

We will cover the basic syntax and definitions of the two functions, as well as two examples of how to apply them.

Basic R Syntax:Replace first match in R

The basic syntax for sub() and gsub() is as follows:

sub("old", "new", x)gsub("old", "new", x) The sub() function replaces the first match in a character string with new characters, while the gsub() function replaces all matches in a character string with new characters.

Example 1: Comparing sub() and gsub()

Before we can apply sub() and gsub(), we need to create an example character string in R:

x <- "aaabbb" Our example character string contains the letters “a” and “b” (each three times). We will replace the character pattern “a” with the new character “c”.

How to create a heatmap in R » Data Science Tutorials

Using sub():

sub("a", "c", x)# "caabbb" The sub() function replaces only the first match with our new character (i.e. the first “a” is replaced by “c”).

Using gsub():

gsub("a", "c", x)# "cccbbb" The gsub() function replaces all matches with “c” (i.e. all “a” of our example character string).

Example 2: Replacing Multiple Patterns

In this example, we will replace multiple patterns with the same new character.

We can do this by using the | operator between the different patterns that we want to match. For example:

sub("a|b", "c", x)# "caabbb" Using sub() with multiple patterns does not change the result, because the first match is still the first “a” of our example character string.

Using gsub() with multiple patterns:

gsub("a|b", "c", x)# "cccccc" The gsub() function replaces all characters with “c”, since each of the characters in our example character string matches “a” or “b”.

ConclusionIn this article, we have demonstrated how to use sub() and gsub() to replace patterns in characters in R.

We have covered two examples of how to apply these functions, including replacing a single pattern and replacing multiple patterns.

By using these functions, you can quickly and easily replace patterns in your character strings in R.

Reorder Boxplots in R with Examples
How to Create Pareto Chart in R
How to Calculate the Standard Error of the Mean in R
Point Biserial Correlation in R-Quick Guide
Best ML Project with Dataset and Source Code
One-Dimensional Optimization in R

The post Replace first match in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Replace first match in R

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 459–463

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #459This episode first puzzle is based on pure math. We need to find next perfect square number to one given. Given number doesn’t have to be square itself. But it is really easy if you think about it. We need to find out what is square root of given number, then round down to closest integer, add 1 and square it back. That is so easy, that function can be written as oneliner. Check it out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/459 Next Perfect Square.xlsx", range = "A1:A10")test = read\_excel("Excel/459 Next Perfect Square.xlsx", range = "B1:B10") Transformation find\_next\_perf\_square = function(n) (floor(sqrt(n)) + 1) ** 2result = input %>% mutate(`Answer Expected` = map\_dbl(Number, find\_next\_perf\_square)) %>% select(-Number) Validation identical(result, test)# [1] TRUE Puzzle #460Yes, R can be used as scissors for words, sentences and so on. Today we have to cut given string every N places, but somehow counting from the end. Because if there is not enough characters shorter group need to be at the beginning, not at the end. And groups need to be separated with dash (vel hyphen). Looking at the difficulty, not the easiest, but nice to have tasks like this from time to time. Look how I did it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/460 Insert Dash Splitter.xlsx", range = "A1:B10")test = read\_excel("Excel/460 Insert Dash Splitter.xlsx", range = "C1:C10") Transformation split\_by\_dash = function(word, n) { str\_split(word, "", simplify = TRUE) %>% rev() %>% split(rep(1:ceiling(length(.) / n), each = n, length.out = length(.))) %>% map(~paste0(rev(.), collapse = "")) %>% rev() %>% paste0(collapse = "-")}result = input %>% mutate(`Answer Expected` = map2\_chr(String, N, split\_by\_dash)) %>% select(3) Validation identical(result, test)# [1] TRUE Puzzle #461Looks like someone mixed chapters for new book and we need to order them for publisher. If we sort it right away, it will not work, because it is string not a number, and will try to put 12 at the beginning because 1 is earlier in alphabetical order. So what we are gonna do? Cut, sort, bring back. Nothing easier.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/461 Sort the Numbers.xlsx", range = "A1:A10")test = read\_excel("Excel/461 Sort the Numbers.xlsx", range = "B1:B10") Transformation result = input %>% separate(String, into = c("A", "B", "C", "D"), sep = "\\.", remove = FALSE) %>% mutate(across(A:D, as.numeric)) %>% arrange(A, B, C, D) %>% select(String) Validation identical(result$String, test$`Answer Expected`)# [1] TRUE Puzzle #462Usually I use purrr package for any tasks that are repeatable, iterable, but this time I realised that maybe not the shortest, but most readable option will be to use old-fashioned loops. We need to find empty cells in matrix, then fill it with maximum value from neighbouring cells. Let’s get looping.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/462 Fill in the Grid.xlsx", range = "A2:J11", col\_names = F) %>% as.matrix()test = read\_excel("Excel/462 Fill in the Grid.xlsx", range = "A14:J23", col\_names = F) %>% as.matrix() Transformation na\_coords = which(is.na(input), arr.ind = T) get\_surrounding\_values = function(x, y, matrix){ values = c() for (i in -1:1) { for (j in -1:1) { if (x + i > 0 & x + i <= nrow(matrix) & y + j > 0 & y + j <= ncol(matrix)) { values = c(values, matrix[x + i, y + j]) } } } return(max(values, na.rm = T))}for (i in 1:nrow(na\_coords)) { input[na\_coords[i, 1], na\_coords[i, 2]] = get\_surrounding\_values(na\_coords[i, 1], na\_coords[i, 2], input)} Validation identical(input, test)#> [1] TRUE Puzzle #463Last task this week was refering to inventory management. Having only partial information, based only on changes of inventory level, we need to prepare levels for each month, even if there were no changes. Tricky, but I hired purrr::accumulate2() for this job. Check it out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/463 Inventory Calculation.xlsx", range = "A1:C6") %>% janitor::clean\_names()test = read\_excel("Excel/463 Inventory Calculation.xlsx", range = "E2:F14") %>% janitor::clean\_names() Transformation months = tibble(abbs = month.abb, month = 1:12)result = months %>% left\_join(input, by = c("abbs" = "month")) %>% replace\_na(list(incoming\_qty = 0, outgoing\_qty = 0)) %>% mutate(inventory = accumulate2(incoming\_qty, outgoing\_qty, .init = 0, .f = ~ ..1 + ..2 - ..3)[-1]) %>% select(month = abbs, inventory) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
PS. Couple weeks ago, I started uploading on Github not only R, but also in Python. Come and check it.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Display the structure in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Display the structure in R, we will demonstrate how to use the str() function in R to print the structure of a data object.

We will cover three examples: displaying the structure of a data frame, a list, and a vector.

Example 1: Displaying the Structure of a Data Frame

To begin, we need to create an example data frame in R:

data <- data.frame(x1 = 1:5, x2 = letters[1:5], x3 = factor(c("yes", "no", "yes", "yes", "no"))) Next, we can apply the str() function to this data frame:

str(data) The output will show the structure of the data frame, including the number of observations, the number of variables, and the type of each variable. For example:

'data.frame': 5 obs. of 3 variables: $ x1: int 1 2 3 4 5 $ x2: chr "a" "b" "c" "d" ... $ x3: Factor w/ 2 levels "no","yes": 2 1 2 2 1 This output shows that our data frame has 5 observations and 3 variables.

The variables x1 and x2 are numeric and character, respectively, while x3 is a factor with two levels.

Example 2: Displaying the Structure of a List

Next, we will create an example list and apply the str() function to it:

my\_list <- list(letters[3:1], 555, c(1, 3, 5)) The output will show the structure of the list, including the number of elements and the type of each element. For example:

How to create a ggalluvial plot in R? » Data Science Tutorials

List of 3 $ : chr [1:3] "c" "b" "a" $ : num 555 $ : num [1:3] 1 3 5 This output shows that our list has three elements.

The first element is a character vector, the second element is a numeric value, and the third element is a numeric vector.

Example 3: Displaying the Structure of a Vector

Finally, we will create an example vector and apply the str() function to it:

vec <- c(21, 5, 3, 17, 8, 7, 7, 13) The output will show the structure of the vector, including its type and length. For example:

num [1:8] 21 5 3 17 8 7 7 13 This output shows that our vector is numeric and has a length of 8.

ConclusionIn this article, we have demonstrated how to use the str() function in R to print the structure of a data object.

We have covered three examples: displaying the structure of a data frame, a list, and a vector.

By using the str() function, you can quickly and easily evaluate the structure of your data objects in R.

Random Number Generator in R
Select Unique Rows in a Data Frame in R
How to Calculate Partial Correlation coefficient in R-Quick Guide
Kruskal Wallis test in R-One-way ANOVA Alternative
Import CSV Files into R Step-by-Step Guide
Cluster Analysis in R
How to Calculate Mean Absolute Error in R
Conditional Mean in R with examples

The post Display the structure in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Display the structure in R

View Details

[This article was first published on Blog on Credibly Curious, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’m very happy to announce {geotargets} version 0.1.0! The {geotargets} package extends {targets} to work with geospatial data formats. Version 0.1.0 supports terra::vect(), terra::rast() and terra::sprc() formats. This R package is only possible due to the great work by Eric Scott and Andrew Brown. While this blog post is on my website, I want to emphasise that this project is very much a team effort.

You can download {geotargets} from the R universe like so:

install.packages("geotargets", repos = c("https://njtierney.r-universe.dev", "https://cran.r-project.org")) What is targets? Why do I need geotargets?The targets package is an R package for managing analytic pipelines. It means that you can write out an analysis in a specific manner, and then as you update code, it will only rerun the necessary parts. Essentially it helps you avoid running large pieces of analysis when you don’t need to. To learn more about targets, I’d highly recommend reading the {targets} manual.

Let’s show an example. Let’s say we want to get an example raster file from {terra}, we can do the following:

terra\_rast\_example <- system.file( "ex/elev.tif", package = "terra" ) |> terra::rast()terra\_rast\_example#> class : SpatRaster #> dimensions : 90, 95, 1 (nrow, ncol, nlyr)#> resolution : 0.008333333, 0.008333333 (x, y)#> extent : 5.741667, 6.533333, 49.44167, 50.19167 (xmin, xmax, ymin, ymax)#> coord. ref. : lon/lat WGS 84 (EPSG:4326) #> source : elev.tif #> name : elevation #> min value : 141 #> max value : 547 Here is the equivalent code in a targets pipeline – the reason we want to use {targets} here is we save the results so we don’t need to run them again. In this case the example code doesn’t take long to run. But imagine reading in the raster was hugely time and computer expensive and we didn’t want to do it again. The {targets} package stores the information so we can just read it back in later, and if we try and run the code again it will not update the code unless the data input has changed. Neat, right?

library(targets)tar\_dir({ # tar\_dir() runs code from a temporary directory. tar\_script({ library(targets) list( tar\_target( terra\_rast\_example, system.file("ex/elev.tif", package = "terra") |> terra::rast() ) ) }) tar\_make() x <- tar\_read(terra\_rast\_example) x})#> ▶ dispatched target terra\_rast\_example#> ● completed target terra\_rast\_example [1.196 seconds]#> ▶ ended pipeline [1.825 seconds]#> #> class : SpatRaster#> Error: external pointer is not valid We get an error!

Error: external pointer is not valid This is a relatively common gotcha moment when using libraries like {terra}. This is due to limitations with its underlying C++ implementation. There are specific ways to write and read these objects. See ?terra for details.

But how do we use {geotargets} to help with this? It helps handle these write and read steps, so you don’t have to worry about them and can use targets as you are used to.

So instead of tar_target(), you use tar_terra_rast() to save a {terra} raster:

library(targets)tar\_dir({ # tar\_dir() runs code from a temporary directory. tar\_script({ library(targets) library(geotargets) list( tar\_terra\_rast( terra\_rast\_example, system.file("ex/elev.tif", package = "terra") |> terra::rast() ) ) }) tar\_make() x <- tar\_read(terra\_rast\_example) x})#> ▶ dispatched target terra\_rast\_example#> ● completed target terra\_rast\_example [0.006 seconds]#> ▶ ended pipeline [0.061 seconds]#> #> class : SpatRaster #> dimensions : 90, 95, 1 (nrow, ncol, nlyr)#> resolution : 0.008333333, 0.008333333 (x, y)#> extent : 5.741667, 6.533333, 49.44167, 50.19167 (xmin, xmax, ymin, ymax)#> coord. ref. : lon/lat WGS 84 (EPSG:4326) #> source : terra\_rast\_example #> name : elevation #> min value : 141 #> max value : 547 Similarly, there are tar_terra_vect() and tar_terra_sprc() for dealing with vector (shapefile) and sprc (collections of rasters). See the README example for more information.

If you’d like to see these functions being used in a more practical context, see the demo-geotargets repository.

What’s next?We are actively developing {geotargets}, and the next release will focus on adding support for splitting rasters into tiles, preserving SpatRaster metadata, and adding support for {stars}. You can see the full list of issues for more detail on what we are working on.

ThanksWe have recently generously received support from the R Consortium for our project, “{geotargets}: Enabling geospatial workflow management with {targets}”, and so we would like to thank them for their support.

I’d also like to thank Michael Sumner, Anthony North, and Miles McBain for their helpful discussions, as well as Will Landau for writing targets, and being incredibly responsive and helpful to the issues and questions we have asked as we wrote {geotargets}.

To leave a comment for the author, please follow the link and comment on their blog: Blog on Credibly Curious.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {geotargets} 0.1.0

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Keep in mind that there’s no hyperparameter tuning in these examples. Hyperparameter tuning must be used in practice. Looking for reticulate and rpy2 experts to discuss speedups for this R package (port from the stable Python version) installation and loading. There’s still room for improvement in this R port, especially in terms of data structure (a data structure that would handle time series not as matrices) choices.

%load\_ext rpy2.ipythonThe rpy2.ipython extension is already loaded. To reload it, use: %reload\_ext rpy2.ipython!pip install nnetsauce%%R# 0 - packages -----utils::install.packages(c("reticulate", "remotes", "forecast", "fpp2"))remotes::install\_github("Techtonique/nnetsauce\_r") # slow herelibrary("reticulate")library("nnetsauce")library("fpp2")%%R# 1 - data -----set.seed(123)X <- fpp2::uschangeidx\_train <- 1:floor(0.8*nrow(X))X\_train <- X[idx\_train, ]X\_test <- X[-idx\_train, ]%%R# 2 - model fitting ---obj\_MTS <- nnetsauce::MTS(sklearn$linear\_model$BayesianRidge(), lags = 1L) # use a Bayesian model for uncertainty quantificationobj\_DeepMTS <- nnetsauce::DeepMTS(sklearn$linear\_model$ElasticNet(), lags = 1L, replications=100L, kernel='gaussian') # use Kernel density for uncertainty quantificationobj\_MTS$fit(X\_train)obj\_DeepMTS$fit(X\_train)%%R# 3 - model predictions ---preds\_MTS <- obj\_MTS$predict(h = nrow(X\_test), level = 95, return\_std = TRUE)preds\_DeepMTS <- obj\_DeepMTS$predict(h=nrow(X\_test), level = 95)100%|██████████| 100/100 [00:00<00:00, 3510.91it/s]100%|██████████| 100/100 [00:00<00:00, 5638.11it/s]%%R# 4 - Graph ---par(mfrow=c(2, 4))for (series\_id in c(2, 3, 4, 5)){ plot(1:nrow(X\_test), X\_test[, series\_id], main = paste0("MTS (Bayesian) -- \n", colnames(fpp2::uschange)[series\_id]), type='l', ylim = c(min(preds\_MTS$lower[, series\_id]), max(preds\_MTS$upper[, series\_id]))) lines(preds\_MTS$lower[, series\_id], col="blue", lwd=2) lines(preds\_MTS$upper[, series\_id], col="blue", lwd=2) lines(preds\_MTS$mean[, series\_id], col="red", lwd=2)}for (series\_id in c(2, 3, 4, 5)){ plot(1:nrow(X\_test), X\_test[, series\_id], main = paste0("DeepMTS (KDE) -- \n", colnames(fpp2::uschange)[series\_id]), type='l', ylim = c(min(preds\_DeepMTS$lower[, series\_id]), max(preds\_DeepMTS$upper[, series\_id]))) lines(preds\_DeepMTS$lower[, series\_id], col="blue", lwd=2) lines(preds\_DeepMTS$upper[, series\_id], col="blue", lwd=2) lines(preds\_DeepMTS$mean[, series\_id], col="red", lwd=2)} In this figure, KDE stands for Kernel Density Estimation. Prediction intervals are depicted as a blue line, and mean forecast as a red line. The true value is depicted as a black line. Again, keep in mind that every model is used with its default hyperparameters, and hyperparameters’ tuning will give a different result.

Visualizing predictive simulations for DeepMTS

%%Rpar(mfrow=c(2, 2))matplot(preds\_DeepMTS$sims[[1]], type='l', col=1:4, lwd=2, lty=1, ylim=c(-40, 40))matplot(preds\_DeepMTS$sims[[25]], type='l', col=1:4, lwd=2, lty=1, ylim=c(-40, 40))matplot(preds\_DeepMTS$sims[[50]], type='l', col=1:4, lwd=2, lty=1, ylim=c(-40, 40))matplot(preds\_DeepMTS$sims[[100]], type='l', col=1:4, lwd=2, lty=1, ylim=c(-40, 40)) To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Forecasting the Economy

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Convert characters to time in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Convert characters to time in R, we will explore how to convert characters to time objects and vice versa using the strptime and strftime functions in R.

These functions are part of the base package and provide a convenient way to work with dates and times.

Basic R Syntax: Convert characters to time in R

The basic syntax for the strptime and strftime functions is as follows:

strptime(character, format = "%Y-%m-%d")strftime(time, format = "%Y-%m-%d") The strptime function converts a character string to a time object, while the strftime function converts a time object to a character string.

Example 1: Convert Character to Time ObjectWe can use the strptime function to convert a character string to a time object. For example, let’s create an example character string:

time\_1 <- "2020-06-01" We can then use the strptime function to convert this character string to a time object:

time\_1a <- strptime(time\_1, format = "%Y-%m-%d") The output will be a time object with a timezone:

time\_1a# [1] "2020-06-01 CEST" Example 2: Time Object with Time ZoneWe can also specify the time zone when converting a character string to a time object. For example:

time\_1b <- strptime(time\_1, format = "%Y-%m-%d", tz = "EST") The output will be a time object with the specified time zone:

Replace NA with Zero in R » Data Science Tutorials

time\_1b# [1] "2020-06-01 EST" Example 3: Time Object with Hour, Minute, and SecondWe can also specify the hour, minute, and second when converting a character string to a time object. For example:

time\_2 <- "2020-06-01 16:15:10"time\_2a <- strptime(time\_2, format = "%Y-%m-%d %H:%M:%S", tz = "EST") The output will be a time object with the specified hour, minute, and second:

time\_2a# [1] "2020-06-01 16:15:10 EST" Example 4: Time Object with MillisecondsWe can also specify milliseconds when converting a character string to a time object. For example:

time\_3 <- "2020-06-01 16:11:00.255"time\_3a <- strptime(time\_3, format = "%Y-%m-%d %H:%M:%S", tz = "EST") The output will be a time object with the specified milliseconds:

time\_3a# [1] "2020-06-01 16:15:10.255 EST" Example 5: Convert Time Object to CharacterWe can use the strftime function to convert a time object to a character string. For example:

time\_3b <- strftime(time\_3a) The output will be a character string:

time\_3b# [1] "2020-06-01 16:11:00" By using these functions, we can easily convert between characters and time objects in R.

Model Selection in Machine Learning
Introduction to Hadoop Data Processing Applications
Adding text labels to ggplot2 Bar Chart
Stringr in r 10 data manipulation Tips and Tricks
XGBoost’s assumptions
X bar R Chart Relation in Quality Control
Curve fitting in R Quick Guide

The post Convert characters to time in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Convert characters to time in R

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Identify positions in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Identify positions in R, we will explore how to use the str_subset and str_which functions in R to filter and find patterns in character strings.

These functions are part of the stringr package, which provides a variety of functions for working with strings.

Identify positions in RTo demonstrate the usage of str_subset and str_which, we will create a vector of character strings x and install/load the stringr package:

install.packages("stringr")library("stringr")x <- c("aaa", "bbb", "abc") Example 1: Using str_subset to Filter Patterns

We can use the str_subset function to filter a character string vector and keep only the strings that match a pattern.

How to Calculate Ratios in R » Data Science Tutorials

For example, we can use it to extract all character strings that contain the letter “a”:

str\_subset(x, "a") This will output a vector of character strings that contain the letter “a”:

```

"aaa" "abc"

`` As you can see,str_subset` has filtered our original vector and returned only the strings that match the pattern.

Example 2: Using str_which to Find Positions

We can also use the str_which function to find the positions of character strings that match a pattern.

For example, we can use it to find the positions of all character strings that contain the letter “a”:

str\_which(x, "a") This will output a vector of indices that indicate the positions of the character strings that match the pattern:

```

1 3

``` The RStudio console output shows that the character strings at position 1 and 3 of our vector are containing the letter “a”.

ConclusionIn this article, we have learned how to use the str_subset and str_which functions in R to filter and find patterns in character strings.

By using these functions, you can easily extract specific patterns from your data and identify their positions.

Add mean value in Boxplots in R with examples
Find confidence intervals in R
How to Use the scale() Function in R
Stock Market Secrets -Stocks Unknown Facts
How to perform Rolling Average in R
Machine Learning Impact on your day-to-day life!
Simple Linear Regression in R

The post Identify positions in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Identify positions in R

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Last year, Edson Kambeu, founder and organizer of the Botswana R User Group, shared his plans with the R Consortium to implement data into local businesses in the New R Community in Botswana Wants to Implement Data Into Local Businesses. He recently updated the R Consortium about the group’s growth and recent activities. The group has attracted a global audience through its online events and actively collaborates with R User Groups in the region.

The Botswana R User Group is seeking speakers for their upcoming online events. If you are an R expert interested in sharing your experience with R users in Botswana, please contact Edson at botswanarusers@gmail.com.

Please share about your background and involvement with the RUGS group.

My educational background is in finance. I pursued finance and investments for my master’s degree but also studied economics during my undergraduate years. Mathematics has been my strongest subject since primary school, and I’m passionate about it. This passion led me to develop an interest in Statistics and statistical software.

In the past, I mainly used SPSS, Stata, and EViews for my statistical analysis projects. Then, someone introduced me to data science. During my research on data science, I discovered that two popular programming languages are used for it. I installed Python for the first time, but I could not use it as I didn’t have a computer science background. So, I switched to R and started watching a few YouTube videos. From there, I continued to learn and improve my skills in R.

R was my first language of choice for Data Science. Currently, I use both R and Python for my work.

I am pursuing a Master’s in Computer Science with Data Science from the University of Sunderland. Our different modules use R and Python, and knowing both languages is helping me in my studies.

As I was learning R around 2019 and beginning to follow several R Users on Twitter, I discovered that small R communities gathered together to learn and share knowledge about it. R Ladies Johannesburg in South Africa inspired me the most, as they held events more frequently during that time. I then became interested in starting an R User community in Botswana.

In February 2020, I reached out to Heather Turner, who was scheduled to visit Botswana and other Southern African countries to conduct Introduction to R workshops. During our conversation, Heather provided me with all the information needed to start a community. As a result, in March 2020, Botswana R Users was established during Heather Turner’s Introduction to R workshop.

How has your group been doing since we last talked?

Our meetup group had about 100 members when we last talked to you. We now have almost 400 members. However, I have observed that people from different countries are joining us. We are now a global meetup group rather than a Botswana User group. This is because we mostly hold online meetups, which allow people from other countries to join.

Participants attending an online meetup hosted by Botswana R users in collaboration with Estwatini R Users and Bulawayo R

We are, however, still committed to growing the local community. We want to see more local participation in our meetup group. Last year, we collaborated with R Ladies Gaborone to organize an introduction to R workshop to increase our local membership. We are pleased to announce that this year, we plan to hold another workshop as a pre-conference event in the upcoming Botswana Deep Learning Indaba conference in July 2024. This workshop will help us to increase our local membership further and create more awareness about our group.

Participant at the Introduction to R Workshop held in collaboration between Botswana R users and R Ladies Gaborone

We value collaborations with our partner R User meetup groups in Southern Africa. In recent years, we have had regular meetups involving collaborative efforts with the Bulawayo R User Group from Zimbabwe, the Eswatini R User Group from Swaziland, and the Namibia R User Group from Namibia. We have established a routine of holding joint meetups almost every two months, depending on the availability of speakers. The idea is to grow our communities by increasing the frequency of activities.

Vebash Naidoo of RLadies Jozi presenting in an online meetup for Botswana R Users You have a Meetup titled “GIS and Creating Dashboards in R. A case study of conflicts events in Kenya,” can you share more on the topic covered? Why this topic?

I had an opportunity to attend a series of workshops and webinars organized by the United Nations for their Datathon. I realized the importance of GIS in advancing sustainable development. In January 2024, I invited Godwin Murithi, a GIS specialist, to present a topic on GIS. The topic was “GIS and Creating Dashboards in R. A case study of conflict events in Kenya.” We wanted to expose our members to the rising field of GIS and show them how the R language and various packages can help solve GIS problems. It was a fascinating topic for our participants, and they loved it.

How has the use of R evolved in the industry since we last talked?

We are observing an increasing acceptance of the R programming language, particularly in universities. Some universities have adopted R as their primary language for statistics and quantitative programs. This trend indicates academic institutions’ growing preference for open source programming languages.

Organizing is one of the most challenging tasks. To get speakers, I have primarily used Twitter (now called X) and LinkedIn to communicate with people who might want to speak at our meetup groups. Lately, there has been a problem with sending direct messages on Twitter. The reason is that Twitter has changed its messaging system. Now, to send a direct message, you need to be verified. I have been affected by the fact that my usual way of talking to people has been disrupted. Therefore, I have resorted to using LinkedIn to search for people interested in R and reach out to them. Sometimes, they are too busy or cancel, which can be challenging. However, I have been successful in finding potential speakers through these platforms.

Occasionally, we use different video conferencing tools like Zoom and Google Meet. We usually rely on these two platforms. However, sometimes a speaker may prefer using Google Meet over Zoom, so we try to be flexible and accommodate their preferences.

We also use GitHub. We have our account, and if the speaker has their material on GitHub, they can share the link with us. Alternatively, they can provide us with the material directly, and we will upload it onto our own GitHub account for the community to access. Ultimately, it all depends on the speaker’s preference.

Please share about a project you are working on or have worked on using the R language. What is the goal/reason, result, or anything interesting, especially related to the industry you work in?

One of my recent school projects was to create a dashboard about UK imports and exports, which I completed towards the end of last year. I developed this project using Shiny and R packages such as Shiny Dashboard, ggplot2, and dplyr.

I’m currently working on another project that is still in its early stages. The goal of this project is to identify areas in Botswana that require greater financial inclusion. I am currently gathering data and plan to utilize R and Python to apply geospatial techniques.

What trends do you currently see in R language and your industry? Any trends you see developing in the near future?

I have observed that people find Quarto and GIS techniques interesting. The community is gaining increasing interest in these areas, and I foresee the increasing use of R in GIS applications.

Learn moreThe post Collaborative Growth: The Botswana R User Group and Regional Partnerships appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Collaborative Growth: The Botswana R User Group and Regional Partnerships

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Announcing the Latest Updates to the healthyR.data R PackageI’m excited to share the latest updates to the healthyR.data R package! This release brings new functionality and minor improvements, all aimed at making your data management tasks easier and more efficient. Here’s a breakdown of what’s new:

New Functionsget_cms_meta_data()This new function is designed to retrieve metadata from the Centers for Medicare & Medicaid Services (CMS). Whether you’re working on health research, policy analysis, or clinical studies, this function provides a straightforward way to access essential CMS data.

Learn more about get_cms_meta_data()

Syntax:

get\_cms\_meta\_data( .title = NULL, .modified\_date = NULL, .keyword = NULL, .identifier = NULL, .data\_version = "current", .media\_type = "all") get_provider_meta_data()Similarly, the get_provider_meta_data() function allows you to fetch metadata related to healthcare providers. This can be particularly useful for projects that require comprehensive information about provider attributes and characteristics.

Learn more about get_provider_meta_data()

Syntax:

get\_provider\_meta\_data( .identifier = NULL, .title = NULL, .description = NULL, .keyword = NULL, .issued = NULL, .modified = NULL, .released = NULL, .theme = NULL, .media\_type = NULL) fetch_cms_data() and fetch_provider_data()We’ve also added fetch_cms_data() and fetch_provider_data(), two powerful functions for fetching actual data from CMS and healthcare providers, respectively. These functions are perfect for those who need to integrate large datasets into their workflows seamlessly.

Learn more about fetch_cms_data()

Learn more about fetch_provider_data()

Syntax:

fetch\_cms\_data(.data\_link)fetch\_provider\_data(.data\_link) Minor Fixes and ImprovementsBug Fix in current_hosp_data()We’ve addressed a bug related to directory file paths in the current_hosp_data() function. This fix ensures smoother operation and better reliability when managing hospital data.

Learn more about current_hosp_data()

No Breaking ChangesI’m pleased to report that this update does not include any breaking changes. You can upgrade to the latest version without worrying about compatibility issues with your existing code.

ConclusionThese enhancements to the healthyR.data package are all about providing more robust tools for healthcare data analysis. I hope these new functions and fixes will improve your workflows and make your data management tasks more efficient.

As always, I welcome your feedback and suggestions. Happy coding!

Best regards, Steve Sanderson

Feel free to reach out if you have any questions or need further assistance with these new features. Your feedback is invaluable in helping us improve healthyR.data continuously.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Update to healthyR.data 1.1.0

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Wrap a character string in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Wrap a character string in R, we will explore how to use the str_wrap function in R to wrap a character string into nicely formatted paragraphs.

This function is part of the stringr package, which provides a variety of functions for working with strings.

Wrap a character string in RTo demonstrate the usage of str_wrap, we will create a character string x and install/load the stringr package:

How to Calculate Ratios in R » Data Science Tutorials

x <- "heyho I am a very long string"install.packages("stringr")library("stringr") Example 1: Wrapping a Character String with str_wrapWe can use the str_wrap function to wrap our character string x into nicely formatted paragraphs.

We can specify the maximum width of each line by providing an integer value as the second argument:

str\_wrap(x, 10) This will output a wrapped character string with each line limited to 10 characters:

```

"heyho\nI am a\nvery long\nstring"

`` As you can see, thestr_wrapfunction has inserted newline characters (\n`) into our character string, resulting in well-formatted paragraphs.

Example 2: Wrapping with Multiple Line BreaksWe can also use the str_wrap function to wrap our character string with multiple line breaks.

For example, we can specify the number of line breaks by providing an integer value as the second argument:

How to create Radar Plot in R-ggradar » Data Science Tutorials

str\_wrap(x, 3) This will output a wrapped character string with three line breaks:

```

"heyho\nI am a\nvery long\nstring\n"

`` As you can see, thestr_wrap` function has inserted multiple newline characters into our character string, resulting in well-formatted paragraphs.

ConclusionIn this article, we have learned how to use the str_wrap function in R to wrap a character string into nicely formatted paragraphs.

By using this function, you can easily format your text data and make it more readable.

Kurtosis in R-What do you understand by Kurtosis?
What is an adaptive clinical trial?
One-sample Wilcoxon test in R
Test if two objects are nearly equal in R
Regression analysis in R-Model Comparison
Reorder Boxplots in R with Examples

The post Wrap a character string in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Wrap a character string in R

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Dear rOpenSci friends, it’s time for our monthly news roundup!

You can read this post on our blog.Now let’s dive into the activity at and around rOpenSci!

rOpenSci HQrOpenSci in the Research Organization RegistryrOpenSci was added to the Research Organization Registry (ROR) in its latest release. The ROR is a global, community-led registry of open persistent identifiers for research organizations. ROR IDs help link and disambiguate metadata about organizations in the scholarly record, much like DOIs and ORCiDs do for manuscripts and researchers. Linked metadata is rOpenSci’s love language ! Find us at https://ror.org/019jywm96.

What are the rOpenSci Champions up to now?Our Champions and mentors have been carrying out various programmed activities.The first stage of the program has a very important training component. This year, we divided the training into technical and community tracks, with several one to two-hour sessions each. Here, you can find the openly available material for each session.

The Technical Track is focused on good software and package development practices:

Beautiful Code, Because We’re Worth It!: a 1-hour session on good practices for writing code.
Package Development: The Mechanics: three 2-hour sessions for beginners to demystify the creation of an R package.
Package Development: Not Rocket Science: a 2-hour hands-on workshop for advanced R package development with tips and tricks.
How rOpenSci Performs Peer Review: a 2-hour workshop to showcase and discuss how rOpenSci communicates, builds, and reviews software.

The Community Track is focused on community-building skills:

How To Contribute to Open Projects and Communities: a 2-hour session presenting community participation frameworks, examples, and recommendations for your package.
Marketing Ideas For Your Package: A blog post presenting a series of activities and tools for advertising your package.

Next month, we will complete the training with a workshop on Git and GitHub and another on Event Organization.

CoworkingRead all about coworking!

Tuesday, June 4th, 09:00 Australia Western (01:00 UTC), R in the Wild with cohosts Ernest Guevarra, Tomás Zaba, Nicholus Tint Zaw, Zython Paul Lachica and Steffi LaZerte.
- Work on some of your own wild workflows
- Learn how others use R in the Wild (at work, with or without teams)
- Chat with our cohosts, discuss the challenges they face, and wins they have through their work
- Tuesday, July 2nd, 14:00 Europe Central (12:00 UTC), Git and GitHub with cohost Zhian Kamvar and Steffi LaZerte.
- Read up on Git and GitHub and how they might serve you
- Start your first Git/GitHub project
- Chat with our cohost about the pros and cons of Git & GitHub, resources for getting started and tips and tricks.

And remember, you can always cowork independently on work related to R, work on packages that tend to be neglected, or work on what ever you need to get done!

Software New packagesThe following package recently became a part of our software suite:

dendroNetwork, developed by Ronald Visser: Creating dendrochronological networks based on the similarity between tree-ring series or chronologies. The package includes various functions to compare tree-ring curves building upon the dplR package. The networks can be used to visualise and understand the relations between tree-ring curves. These networks are also very useful to estimate the provenance of wood as described in Visser (2021) DOI:10.5334/jcaa.79 or wood-use within a structure/context/site as described in Visser and Vorst (2022) DOI:10.1163/27723194-bja10014. It is available on CRAN. It has been reviewed by Kaija Gahm and Zachary Gajewski.

Discover more packages, read more about Software Peer Review.

New versionsThe following nine packages have had an update since the last newsletter: comtradr (v1.0.0), dendroNetwork (0.5.4), drake (7.13.10), fellingdater (v1.0.2), melt (v1.11.4), nasapower (v4.2.1), osmextract (v0.5.1), stplanr (v1.2.0), and traits (v0.5.1).

Software Peer ReviewThere are thirteen recently closed and active submissions and 7 submissions on hold. Issues are at different stages:

One at ‘6/approved’:
- dendroNetwork, Create networks of dendrochronological series using pairwise similarity. Submitted by RonaldVisser.
- Three at ‘5/awaiting-reviewer(s)-response’:
- karel, Learning programming with Karel the robot. Submitted by Marcos Prunello.
- rOPTRAM, Derive soil moisture using the OPTRAM algorithm. Submitted by Micha Silver.
- mregions2, Access Data from Marineregions.org: The Marine Regions Gazetteer and the Marine Regions Data Products. Submitted by salvafern.
- Two at ‘4/review(s)-in-awaiting-changes’:
- rix, Rix: Reproducible Environments with Nix. Submitted by Bruno Rodrigues.
- agromet, Índices y Estadísticos Climáticos e Hidrológicos. Submitted by Paola Corrales.
- Three at ‘3/reviewer(s)-assigned’:
- cancerprof, API Client for State Cancer Profiles. Submitted by Brian Park.
- osmapiR, OpenStreetMap API. Submitted by Joan Maspons.
- fwildclusterboot, Fast Wild Cluster Bootstrap Inference for Linear Models. Submitted by Alexander Fischer. (Stats).
- Three at ‘2/seeking-reviewer(s)’:
- chopin, CHOPIN: Computation for Climate and Health research On Parallelized INfrastructure. Submitted by Insang Song.
- rsi, Efficiently Retrieve and Process Satellite Imagery. Submitted by Michael Mahoney.
- sits, Satellite Image Time Series Analysis for Earth Observation Data Cubes. Submitted by Gilberto Camara.
- One at ‘1/editor-checks’:
- gigs, Assess Fetal, Newborn, and Child Growth with International Standards. Submitted by Simon Parker. (Stats).

Find out more about Software Peer Review and how to get involved.

On the blog* Communication Tips for your Open-Source Project by Maëlle Salmon.

Calls for contributionsCalls for maintainersIf you’re interested in maintaining any of the R packages below, you might enjoy reading our blog post What Does It Mean to Maintain a Package?.

historydata, datasets for historians. Issue for volunteering.
USAboundaries (and USAboundariesdata), historical and contemporary boundaries of the United States of America . Issue for volunteering.

Also refer to our help wanted page – before opening a PR, we recommend asking in the issue whether help is still needed.

Package development cornerSome useful tips for R package developers.

Token now needed for code coverage via codecov and {covr} on GitHub ActionsTest coverage reports are useful when assessing and improving tests of an R package.One can run the covr package locally, or send results to an interface like codecov.io that provides interactive exploration of the output.It is possible to compute test coverage and send the results to codecov.io on GitHub Actions, using the r-lib/actions actions.

Now, recently, workflows have started to fail if one did not set a codecov.io token as an environment variable.How to solve this

For an individual repo, add your codecov.io token as CODECOV_TOKEN to the repository secrets.
For an organization, ask the organization owner to create an organization wide codecov.io token and to store it as an organization secret. Make sure to update your GitHub Actions workflow file, for instance by running usethis::use_github_action("test-coverage").

Notes on package repositoriesLluís Revilla Sancho wrote about Packaging R: getting in repositories.We particularly note his two definitions of package repositories: the first one consists in making install.packages() work, the second in adding a layer of checks to packages in the repository.He then added “R-universe is using the first definition but could be used to generate repositories with checks that comply with the second definition.”

Standardize a DESCRIPTION file in one function callDo you know about the desc::desc_normalize() function that orders and formats DESCRIPTION fields in a standard way?Once you start using it, there’s no way back.

You can also call it indirectly via usethis::use_tidy_description() that also sets the Encoding field to UTF-8.

Automated refactoring with xmlparsedataRefactoring code can be tedious manual work, but it can also be tedious automated work. Read a post about how to replace all occurrences of a given function call with another one using an XML representation of the code.

What’s your favorite IDE?Athanasia Monica Mowinckel wrote an informative post about the IDEs she uses.

Code review resourcesBeside the tidyverse code review guidance, we can now recommend you check out the code review anxiety workbook by Carol Lee and Kristen Foster-Marks, that explains what code review anxiety is, and describes efficient methods to deal with it.

If you haven’t subscribed to our newsletter yet, you can do so via a form. Until it’s time for our next newsletter, you can keep in touch with us via our website and Mastodon account.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci News Digest, May 2024

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Welcome back to the world of purrr! Last time (about a year ago), we spun a metaphorical yarn about the wonders of purrr in R. Today, we're rolling up our sleeves and diving into a hands-on tutorial. We're going to explore how purrr makes working with lists and vectors a breeze, transforming and manipulating them like a data wizard.

With purrr, you can apply functions to each element of a list or vector, manipulate them, check conditions, and so much more. It's all about making your data dance to your commands with elegance and efficiency. Ready to unleash some functional magic?

Are map Functions Like apply Functions?You might be wondering, “Aren’t map functions just fancy versions of apply functions?” It's a fair question! Both map and apply functions help you apply a function to elements in a data structure, but purrr takes it to a whole new level.

Here’s why purrr and its map functions are worth your attention:

Consistency: purrr functions have a consistent naming scheme, making them easier to learn and remember.
Type Safety: map functions in purrr return outputs of consistent types, reducing unexpected errors.
Integration: Seamlessly integrate with other tidyverse packages, making your data wrangling pipeline smoother.

Let’s see a quick comparison:

library(tidyverse)# Using lapply (base R)numbers <- list(1, 2, 3, 4, 5)squared\_lapply <- lapply(numbers, function(x) x^2)# Using map (purrr)squared\_map <- map(numbers, ~ .x^2)print(squared\_lapply)[[1]][1] 1[[2]][1] 4[[3]][1] 9[[4]][1] 16[[5]][1] 25print(squared\_map)[[1]][1] 1[[2]][1] 4[[3]][1] 9[[4]][1] 16[[5]][1] 25 Both do the same thing, but purrr’s map function is more readable and concise, especially when paired with the tidyverse syntax.

Here’s another example with a built-in dataset:

```

Using lapply with a built-in datasetiris_split <- split(iris, iris$Species)mean_sepal_length_lapply <- lapply(iris_split, function(df) mean(df$Sepal.Length))# Using map with a built-in datasetmean_sepal_length_map <- map(iris_split, ~ mean(.x$Sepal.Length))print(mean_sepal_length_lapply)$setosa[1] 5.006$versicolor[1] 5.936$virginica[1] 6.588print(mean_sepal_length_map)$setosa[1] 5.006$versicolor[1] 5.936$virginica[1] 6.588

``` Again, the purrr version is cleaner and easier to understand at a glance.

Convinced? Let’s move on to explore simple maps and their variants to see more of purrr’s magic. Ready?

Simple Maps and Their VariantsNow that we know why purrr’s map functions are so cool, let’s dive into some practical examples. The map function family is like a Swiss Army knife for data transformation. It comes in different flavors depending on the type of output you want: logical, integer, character, or double.

Let’s start with the basic map function:

library(tidyverse)# Basic map examplenumbers <- list(1, 2, 3, 4, 5)squared\_numbers <- map(numbers, ~ .x^2)squared\_numbers Easy, right? Yes, but we have one twist here. Result is returned as list, and we don’t always need list. So now, let’s look at the type-specific variants. These functions ensure that the output is of a specific type, which can help avoid unexpected surprises in your data processing pipeline.

Logical (map_lgl):

```

Check if each number is evenis_even <- map_lgl(numbers, ~ .x %% 2 == 0)is_even[1] FALSE TRUE FALSE TRUE FALSE# it is not list anymore, it is logical vector

``` * Integer (map_int):

```

Double each number and return as integersdoubled_integers <- map_int(numbers, ~ .x * 2)doubled_integers[1] 2 4 6 8 10

``` * Character (map_chr):

```

Convert each number to a stringnumber_strings <- map_chr(numbers, ~ paste("Number", .x))number_strings[1] "Number 1" "Number 2" "Number 3" "Number 4" "Number 5"

``` * Double (map_dbl):

```

Half each number and return as doubleshalved_doubles <- map_dbl(numbers, ~ .x / 2)halved_doubles[1] 0.5 1.0 1.5 2.0 2.5

``` Let’s apply this to a built-in dataset to see it in action:

```

Using map_dbl on the iris dataset to get the mean of each numeric columniris_means <- iris %>% select(-Species) %>% map_dbl(mean)iris_meansSepal.Length Sepal.Width Petal.Length Petal.Width 5.843333 3.057333 3.758000 1.199333

``` Here, we’ve calculated the mean of each numeric column in the iris dataset, and the result is a named vector of doubles.

Pretty neat, huh? The map family makes it easy to ensure your data stays in the format you expect.

Ready to see how purrr handles multiple vectors with map2 and pmap?

Not Only One Vector: map2 and pmap + VariantsSo far, we’ve seen how map functions work with a single vector or list. But what if you have multiple vectors and want to apply a function to corresponding elements from each? Enter map2 and pmap.

map2: This function applies a function to corresponding elements of two vectors or lists.
pmap: This function applies a function to corresponding elements of multiple lists.

Let’s start with map2:

library(tidyverse)# Two vectors to work withvec1 <- c(1, 2, 3)vec2 <- c(4, 5, 6)# Adding corresponding elements of two vectorssum\_vecs <- map2(vec1, vec2, ~ .x + .y)sum\_vecs[[1]][1] 5[[2]][1] 7[[3]][1] 9 Here, map2 takes elements from vec1 and vec2 and adds them together.

Now, let’s step it up with pmap:

```

Creating a tibble for multiple listsdf <- tibble( a = 1:3, b = 4:6, c = 7:9)# Summing corresponding elements of multiple listssum_pmap <- pmap(df, ~ ..1 + ..2 + ..3)sum_pmap[[1]][1] 12[[2]][1] 15[[3]][1] 18

``` In this example, pmap takes elements from columns a, b, and c of the tibble and sums them up.

Look at syntax in those two examples. In map2, we give two vectors or lists, and then we are reffering to them as .x and .y. Further in pmap example we have data.frame, but it can be a list of lists, and we need to refer to them with numbers like ..1, ..2 and ..3 (and more if needed).

Variants of map2 and pmapJust like map, map2 and pmap have type-specific variants. Let’s see a couple of examples using data structures already defined above:

map2_dbl:

```

Multiplying corresponding elements of two vectors and returning doublesproduct_vecs <- map2_dbl(vec1, vec2, ~ .x * .y)product_vecs[1] 4 10 18

``` * pmap_chr:

```

Concatenating corresponding elements of multiple lists into stringsconcat_pmap <- pmap_chr(df, ~ paste(..1, ..2, ..3, sep = "-"))concat_pmap[1] "1-4-7" "2-5-8" "3-6-9"

``` These variants ensure that your results are of the expected type, just like the basic map variants.

With map2 and pmap, you can handle more complex data transformations involving multiple vectors or lists with ease.

Ready to move on and see what lmap and imap can do for you?

Using imap for Indexed Mapping and Conditional Maps with _if and _atLet’s combine our exploration of imap with the conditional mapping functions map_if and map_at. These functions give you more control over how and when functions are applied to your data, making your code more precise and expressive.

imap: Indexed MappingThe imap function is a handy tool when you need to include the index or names of elements in your function calls. This is particularly useful for tasks where the position or name of an element influences the operation performed on it.

Here’s a practical example with a named list:

library(tidyverse)# A named list of scoresnamed\_scores <- list(math = 90, science = 85, history = 78)# Create descriptive strings for each scorescore\_descriptions <- imap(named\_scores, ~ paste(.y, "score is", .x))score\_descriptions$math[1] "math score is 90"$science[1] "science score is 85"$history[1] "history score is 78" In this example:

We have a named list named_scores with subject scores.
We use imap to create a descriptive string for each score that includes the subject name and the score.

Conditional Maps with map_if and map_atSometimes, you don’t want to apply a function to all elements of a list or vector — only to those that meet certain conditions. This is where map_if and map_at come into play.

map_if: Conditional Mapping

Use map_if to apply a function to elements that satisfy a specific condition (predicate).

```

Mixed list of numbers and charactersmixed_list <- list(1, "a", 3, "b", 5)# Double only the numeric elementsdoubled_numbers <- map_if(mixed_list, is.numeric, ~ .x * 2)doubled_numbers[[1]][1] 2[[2]][1] "a"[[3]][1] 6[[4]][1] "b"[[5]][1] 10

``` In this example:

We have a mixed list of numbers and characters.
We use map_if to double only the numeric elements, leaving the characters unchanged.

map_at: Specific Element Mapping

Use map_at to apply a function to specific elements of a list or vector, identified by their indices or names.

```

A named list of mixed typesspecific_list <- list(a = 1, b = "hello", c = 3, d = "world")# Convert only the character elements to uppercaseuppercase_chars <- map_at(specific_list, c("b", "d"), ~ toupper(.x))uppercase_chars$a[1] 1$b[1] "HELLO"$c[1] 3$d[1] "WORLD"

``` In this example:

We have a named list with mixed types.
We use map_at to convert only the specified character elements to uppercase.

Combining imap, map_if, and map_at allows you to handle complex data transformation tasks with precision and clarity. These functions make it easy to tailor your operations to the specific needs of your data.

Shall we move on to the next chapter to explore walk and its friends for side-effect operations?

Make Something Happen Outside of Data: walk and Its FriendsSometimes, you want to perform operations that have side effects, like printing, writing to a file, or plotting, rather than returning a transformed list or vector. This is where the walk family of functions comes in handy. These functions are designed to be used for their side effects, as they return NULL.

walkThe basic walk function applies a function to each element of a list or vector and performs actions like printing or saving files.

library(tidyverse)# A list of numbersnumbers <- list(1, 2, 3, 4, 5)# Print each numberwalk(numbers, ~ print(.x))[1] 1[1] 2[1] 3[1] 4[1] 5 In this example, walk prints each element of the numbers list.

walk2When you have two lists or vectors and you want to perform side-effect operations on their corresponding elements, walk2 is your friend.

```

Two vectors to work withvec1 <- c("apple", "banana", "cherry")vec2 <- c("red", "yellow", "dark red")# Print each fruit with its colorwalk2(vec1, vec2, ~ cat(.x, "is", .y, "\n"))apple is red banana is yellow cherry is dark red

``` Here, walk2 prints each fruit with its corresponding color.

iwalkiwalk is the side-effect version of imap. It includes the index or names of the elements, which can be useful for logging or debugging.

```

A named list of scoresnamed_scores <- list(math = 90, science = 85, history = 78)# Print each subject with its scoreiwalk(named_scores, ~ cat("The score for", .y, "is", .x, "\n"))The score for math is 90 The score for science is 85 The score for history is 78

``` In this example, iwalk prints each subject name with its corresponding score.

Practical Example with Built-in DataLet’s use a built-in dataset and perform some side-effect operations. Suppose you want to save plots of each numeric column in the mtcars dataset to separate files.

```

Directory to save plotsdir.create("plots")# Save histograms of each numeric column to fileswalk(names(mtcars), ~ { if (is.numeric(mtcars[[.x]])) { plot_path <- paste0("plots/", .x, "_histogram.png") png(plot_path) hist(mtcars[[.x]], main = paste("Histogram of", .x), xlab = .x) dev.off() }})

``` In this example:

We create a directory called “plots”.
We use walk to iterate over the names of the mtcars dataset.
For each numeric column, we save a histogram to a PNG file.

This is a practical demonstration of how walk can be used for side-effect operations such as saving files.

Why Do We Need modify Then?Sometimes you need to tweak elements within a list or vector without completely transforming them. This is where modify functions come in handy. They allow you to make specific changes to elements while preserving the overall structure of your data.

modifyThe modify function applies a transformation to each element of a list or vector and returns the modified list or vector.

library(tidyverse)# A list of numbersnumbers <- list(1, 2, 3, 4, 5)# Add 10 to each numbermodified\_numbers <- modify(numbers, ~ .x + 10)modified\_numbers[[1]][1] 11[[2]][1] 12[[3]][1] 13[[4]][1] 14[[5]][1] 15 In this example, modify adds 10 to each element of the numbers list.

modify_ifmodify_if is used to conditionally modify elements that meet a specified condition (predicate).

```

Modify only the even numbers by multiplying them by 2modified_if <- modify_if(numbers, ~ .x %% 2 == 0, ~ .x * 2)modified_if[[1]][1] 1[[2]][1] 4[[3]][1] 3[[4]][1] 8[[5]][1] 5

``` Here, modify_if multiplies only the even numbers by 2.

modify_atmodify_at allows you to specify which elements to modify based on their indices or names.

```

A named list of mixed typesnamed_list <- list(a = 1, b = "hello", c = 3, d = "world")# Convert only the specified elements to uppercasemodified_at <- modify_at(named_list, c("b", "d"), ~ toupper(.x))modified_at$a[1] 1$b[1] "HELLO"$c[1] 3$d[1] "WORLD"

``` In this example, modify_at converts the specified character elements to uppercase.

modify with Built-in DatasetLet’s use the iris dataset to demonstrate how modify functions can be applied in a practical scenario. Suppose we want to normalize numeric columns by dividing each value by the maximum value in its column.

```

Normalizing numeric columns in the iris datasetnormalized_iris <- iris %>% modify_at(vars(Sepal.Length, Sepal.Width, Petal.Length, Petal.Width), ~ .x / max(.x))head(normalized_iris) Sepal.Length Sepal.Width Petal.Length Petal.Width Species1 0.6455696 0.7954545 0.2028986 0.08 setosa2 0.6202532 0.6818182 0.2028986 0.08 setosa3 0.5949367 0.7272727 0.1884058 0.08 setosa4 0.5822785 0.7045455 0.2173913 0.08 setosa5 0.6329114 0.8181818 0.2028986 0.08 setosa6 0.6835443 0.8863636 0.2463768 0.16 setosahead(iris)1 5.1 3.5 1.4 0.2 setosa2 4.9 3.0 1.4 0.2 setosa3 4.7 3.2 1.3 0.2 setosa4 4.6 3.1 1.5 0.2 setosa5 5.0 3.6 1.4 0.2 setosa6 5.4 3.9 1.7 0.4 setosa

``` In this example:

We use modify_at to specify the numeric columns of the iris dataset.
Each value in these columns is divided by the maximum value in its respective column, normalizing the data.

modify functions offer a powerful way to make targeted changes to your data, providing flexibility and control.

Predicates: Does Data Satisfy Our Assumptions? every, some, and noneWhen working with data, it’s often necessary to check if certain conditions hold across elements in a list or vector. This is where predicate functions like every, some, and none come in handy. These functions help you verify whether elements meet specified criteria, making your data validation tasks easier and more expressive.

everyThe every function checks if all elements in a list or vector satisfy a given predicate. If all elements meet the condition, it returns TRUE; otherwise, it returns FALSE.

library(tidyverse)# A list of numbersnumbers <- list(2, 4, 6, 8)# Check if all numbers are evenall\_even <- every(numbers, ~ .x %% 2 == 0)all\_even[1] TRUE In this example, every checks if all elements in the numbers list are even.

someThe some function checks if at least one element in a list or vector satisfies a given predicate. If any element meets the condition, it returns TRUE; otherwise, it returns FALSE.

```

Check if any number is greater than 5any_greater_than_five <- some(numbers, ~ .x > 5)any_greater_than_five[1] TRUE

``` Here, some checks if any element in the numbers list is greater than 5.

noneThe none function checks if no elements in a list or vector satisfy a given predicate. If no elements meet the condition, it returns TRUE; otherwise, it returns FALSE.

```

Check if no number is oddnone_odd <- none(numbers, ~ .x %% 2 != 0)none_odd[1] TRUE

``` In this example, none checks if no elements in the numbers list are odd.

Practical Example with Built-in DatasetLet’s use the mtcars dataset to demonstrate how these predicate functions can be applied in a practical scenario. Suppose we want to check various conditions on the columns of this dataset.

```

Check if all cars have more than 10 miles per gallon (mpg)all_mpg_above_10 <- mtcars %>% select(mpg) %>% map_lgl(~ every(.x, ~ .x > 10))all_mpg_above_10mpgTRUE# Check if some cars have more than 150 horsepower (hp)some_hp_above_150 <- mtcars %>% select(hp) %>% map_lgl(~ some(.x, ~ .x > 150))some_hp_above_150hpTRUE# Check if no car has more than 8 cylindersnone_cyl_above_8 <- mtcars %>% select(cyl) %>% map_lgl(~ none(.x, ~ .x > 8))none_cyl_above_8cylTRUE

``` In this example:

We check if all cars in the mtcars dataset have more than 10 mpg using every.
We check if some cars have more than 150 horsepower using some.
We check if no car has more than 8 cylinders using none.

These predicate functions provide a straightforward way to validate your data against specific conditions, making your analysis more robust.

What If Not: keep and discardWhen you’re working with lists or vectors, you often need to filter elements based on certain conditions. The keep and discard functions from purrr are designed for this purpose. They allow you to retain or remove elements that meet specified criteria, making it easy to clean and subset your data.

keepThe keep function retains elements that satisfy a given predicate. If an element meets the condition, it is kept; otherwise, it is removed.

library(tidyverse)# A list of mixed numbersnumbers <- list(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)# Keep only the even numberseven\_numbers <- keep(numbers, ~ .x %% 2 == 0)even\_numbers[[1]][1] 2[[2]][1] 4[[3]][1] 6[[4]][1] 8[[5]][1] 10 In this example, keep retains only the even numbers from the numbers list.

discardThe discard function removes elements that satisfy a given predicate. If an element meets the condition, it is discarded; otherwise, it is kept.

```

Discard the even numbersodd_numbers <- discard(numbers, ~ .x %% 2 == 0)odd_numbers[[1]][1] 1[[2]][1] 3[[3]][1] 5[[4]][1] 7[[5]][1] 9

``` Here, discard removes the even numbers, leaving only the odd numbers in the numbers list.

Practical Example with Built-in DatasetLet’s use the iris dataset to demonstrate how keep and discard can be applied in a practical scenario. Suppose we want to filter rows based on specific conditions for the Sepal.Length column.

library(tidyverse)# Keep rows where Sepal.Length is greater than 5.0iris\_keep <- iris %>% split(1:nrow(.)) %>% keep(~ .x$Sepal.Length > 5.0) %>% bind\_rows()head(iris\_keep) Sepal.Length Sepal.Width Petal.Length Petal.Width Species1 5.1 3.5 1.4 0.2 setosa2 5.4 3.9 1.7 0.4 setosa3 5.4 3.7 1.5 0.2 setosa4 5.8 4.0 1.2 0.2 setosa5 5.7 4.4 1.5 0.4 setosa6 5.4 3.9 1.3 0.4 setosa# Discard rows where Sepal.Length is less than or equal to 5.0iris\_discard <- iris %>% split(1:nrow(.)) %>% discard(~ .x$Sepal.Length <= 5.0) %>% bind\_rows()head(iris\_discard) Sepal.Length Sepal.Width Petal.Length Petal.Width Species1 5.1 3.5 1.4 0.2 setosa2 5.4 3.9 1.7 0.4 setosa3 5.4 3.7 1.5 0.2 setosa4 5.8 4.0 1.2 0.2 setosa5 5.7 4.4 1.5 0.4 setosa6 5.4 3.9 1.3 0.4 setosa In this example:

We split the iris dataset into a list of rows.
We apply keep to retain rows where Sepal.Length is greater than 5.0.
We apply discard to remove rows where Sepal.Length is less than or equal to 5.0.
Finally, we use bind_rows() to combine the list back into a data frame.

Combining keep and discard with mtcarsSimilarly, let’s fix the mtcars example:

```

Keep cars with mpg greater than 20 and discard cars with hp less than 100filtered_cars <- mtcars %>% split(1:nrow(.)) %>% keep(~ .x$mpg > 20) %>% discard(~ .x$hp < 100) %>% bind_rows()filtered_cars mpg cyl disp hp drat wt qsec vs am gear carbMazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2

``` In this combined example:

We split the mtcars dataset into a list of rows.
We use keep to retain cars with mpg greater than 20.
We use discard to remove cars with hp less than 100.
We combine the filtered list back into a data frame using bind_rows().

Do Things in Order of List/Vector: accumulate, reduceSometimes, you need to perform cumulative or sequential operations on your data. This is where accumulate and reduce come into play. These functions allow you to apply a function iteratively across elements of a list or vector, either accumulating results at each step or reducing the list to a single value.

accumulateThe accumulate function applies a function iteratively to the elements of a list or vector and returns a list of intermediate results.

Let’s start with a simple example:

library(tidyverse)# A list of numbersnumbers <- list(1, 2, 3, 4, 5)# Cumulative sum of the numberscumulative\_sum <- accumulate(numbers, `+`)cumulative\_sum[1] 1 3 6 10 15 reduceThe reduce function applies a function iteratively to reduce the elements of a list or vector to a single value.

Here’s a basic example:

```

Sum of the numberstotal_sum <- reduce(numbers, `+`)total_sum[1] 15

``` Practical Example with Built-in DatasetLet’s use the mtcars dataset to demonstrate how accumulate and reduce can be applied in a practical scenario.

Using accumulate with mtcars

Suppose we want to calculate the cumulative sum of the miles per gallon (mpg) for each car.

```

Cumulative sum of mpg valuescumulative_mpg <- mtcars %>% pull(mpg) %>% accumulate(`+`)cumulative_mpg[1] 21.0 42.0 64.8 86.2 104.9 123.0 137.3 161.7 184.5 203.7 221.5 237.9 255.2 270.4 280.8 291.2 305.9 338.3 368.7[20] 402.6 424.1 439.6 454.8 468.1 487.3 514.6 540.6 571.0 586.8 606.5 621.5 642.9

``` In this example, accumulate gives us a cumulative sum of the mpg values for the cars in the mtcars dataset.

Using reduce with mtcars

Now, let’s say we want to find the product of all mpg values:

```

Product of mpg valuesproduct_mpg <- mtcars %>% pull(mpg) %>% reduce(`*`)product_mpg[1] 1.264241e+41

``` In this example, reduce calculates the product of all mpg values in the mtcars dataset.

Do It Another Way: compose and negateCreating flexible and reusable functions is a hallmark of efficient programming. purrr provides tools like compose and negate to help you build and manipulate functions more effectively. These tools allow you to combine multiple functions into one or invert the logic of a predicate function.

composeThe compose function combines multiple functions into a single function that applies them sequentially. This can be incredibly useful for creating pipelines of operations.

Here’s a basic example:

library(tidyverse)# Define some simple functionsadd1 <- function(x) x + 1square <- function(x) x * x# Compose them into a single functionadd1\_and\_square <- compose(square, add1)# Apply the composed functionresult <- add1\_and\_square(2) # (2 + 1)^2 = 9result[1] 9 In this example:

We define two simple functions: add1 and square.
We use compose to create a new function, add1_and_square, which first adds 1 to its input and then squares the result.
We apply the composed function to the number 2, yielding 9.

Practical Example with Built-in DatasetLet’s use compose with a more practical example involving the mtcars dataset. Suppose we want to create a function that first scales the horsepower (hp) by 10 and then calculates the logarithm.

```

Define scaling and log functionsscale_by_10 <- function(x) x * 10safe_log <- safely(log, otherwise = NA)# Compose them into a single functionscale_and_log <- compose(safe_log, scale_by_10)# Apply the composed function to the hp columnmtcars <- mtcars %>% mutate(log_scaled_hp = map_dbl(hp, ~ scale_and_log(.x)$result))head(mtcars) mpg cyl disp hp drat wt qsec vs am gear carb log_scaled_hpMazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 7.003065Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 7.003065Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 6.835185Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 7.003065Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 7.467371Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 6.956545

``` In this example:

We define two functions: scale_by_10 and safe_log.
We compose these functions into scale_and_log.
We apply the composed function to the hp column of the mtcars dataset and add the results as a new column.

negateThe negate function creates a new function that returns the logical negation of a predicate function. This is useful when you want to invert the logic of a condition.

Here’s a simple example:

```

Define a simple predicate functionis_even <- function(x) x %% 2 == 0# Negate the predicate functionis_odd <- negate(is_even)# Apply the negated functionresults <- map_lgl(1:10, is_odd)results [1] TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE

``` In this example:

We define a predicate function is_even to check if a number is even.
We use negate to create a new function is_odd that returns the opposite result.
We apply is_odd to the numbers 1 through 10.

Practical Example with Built-in DatasetLet’s use negate in a practical scenario with the iris dataset. Suppose we want to filter out rows where the Sepal.Length is not greater than 5.0.

```

Define a predicate functionis_long_sepal <- function(x) x > 5.0# Negate the predicate functionis_not_long_sepal <- negate(is_long_sepal)# Filter out rows where Sepal.Length is not greater than 5.0iris_filtered <- iris %>% split(1:nrow(.)) %>% discard(~ is_not_long_sepal(.x$Sepal.Length)) %>% bind_rows()head(iris_filtered) Sepal.Length Sepal.Width Petal.Length Petal.Width Species1 5.1 3.5 1.4 0.2 setosa2 5.4 3.9 1.7 0.4 setosa3 5.4 3.7 1.5 0.2 setosa4 5.8 4.0 1.2 0.2 setosa5 5.7 4.4 1.5 0.4 setosa6 5.4 3.9 1.3 0.4 setosa

``` In this example:

We define a predicate function is_long_sepal to check if Sepal.Length is greater than 5.0.
We use negate to create a new function is_not_long_sepal that returns the opposite result.
We use discard to remove rows where Sepal.Length is not greater than 5.0, then combine the filtered list back into a data frame.

With compose and negate, you can create more flexible and powerful functions, allowing for more concise and readable code.

ConclusionCongratulations! You’ve journeyed through the world of purrr, mastering a wide array of functions and techniques to manipulate and transform your data. From basic mapping to creating powerful function compositions, purrr equips you with tools to make your data wrangling tasks more efficient and expressive.

Whether you’re applying functions conditionally, dealing with side effects, or validating your data, purrr has you covered. Keep exploring and experimenting with these functions to unlock the full potential of functional programming in R.

Gift for patient readersI decided to give you some useful, yet not trivial use cases of purrr functions.

Define list of function to apply on data apply\_funs <- function(x, ...) purrr::map\_dbl(list(...), ~ .x(x)) Want to apply multiple functions to a single vector and get a tidy result? Meet apply_funs, your new best friend! This nifty little function takes a value and a bunch of functions, then maps each function to the vector, returning the results as a neat vector.

Let’s break it down:

x: The value you want to transform.
...: A bunch of functions you want to apply to x.
purrr::map_dbl: Maps each function in the list to x and returns the results as a vector of doubles.

Suppose that you want to apply 3 summary functions on vector of numbers. Here’s how you can do it:

number <- 1:48results <- apply\_funs(number, mean, median, sd)results[1] 24.5 24.5 14.0 Using pmap as equivalent of Python’s zipSometimes you need to zip two tables or columns together. In Python there is zip function for it, but we do not have twin function in R, unless you use pmap. I will not make it longer, so check it out in one of my previous articles.

Rendering parameterized RMarkdown reportsAssuming that you have kind of report you use for each salesperson, there is possibility, that you are changing parameters manually to generate report for person X, for date range Y, for product Z. Why not prepare lists of people, time range, and list of products, and then based on them generate series of reports by one click only.

Mastering purrr: From Basic Maps to Functional Magic in R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering purrr: From Basic Maps to Functional Magic in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionGood morning, everyone!

Today, we’re going to talk about how to handle rows in your dataset that contain a specific string. This is a common task in data cleaning and can be easily accomplished using both base R and the dplyr package. We’ll go through examples for each method and break down the code so you can understand and apply it to your own data.

ExamplesUsing Base RFirst, let’s see how to select and drop rows containing a specific string using base R. We’ll use the grep() function for this.

Example DataLet’s create a simple data frame to work with:

data <- data.frame( id = 1:5, name = c("apple", "banana", "cherry", "date", "elderberry"), stringsAsFactors = FALSE)print(data)

id name1 1 apple2 2 banana3 3 cherry4 4 date5 5 elderberry Selecting Rows with a Specific StringSuppose we want to select rows where the name contains the letter “a”. We can use grep():

selected\_rows <- data[grep("a", data$name), ]print(selected\_rows)

id name1 1 apple2 2 banana4 4 date Explanation:

grep("a", data$name) searches for the letter “a” in the name column and returns the indices of the rows that match.
data[grep("a", data$name), ] uses these indices to subset the original data frame.

Dropping Rows with a Specific StringTo drop rows that contain the letter “a”, we can use the -grep() notation:

dropped\_rows <- data[-grep("a", data$name), ]print(dropped\_rows)

id name3 3 cherry5 5 elderberry Explanation:

-grep("a", data$name) returns the indices of the rows that do not match the search term.
data[-grep("a", data$name), ] subsets the original data frame by excluding these rows.

Using dplyrThe dplyr package makes these tasks even more straightforward with its intuitive functions.

Example DataWe’ll use the same data frame as before. First, make sure you have dplyr installed and loaded:

```

install.packages("dplyr")library(dplyr)

`` Selecting Rows with a Specific StringUsingdplyr, we can select rows containing “a” with thefilter()function combined withstr_detect()from thestringr` package:

library(stringr)selected\_rows\_dplyr <- data %>% filter(str\_detect(name, "a"))print(selected\_rows\_dplyr)

id name1 1 apple2 2 banana3 4 date Explanation:

%>% is the pipe operator, allowing us to chain functions together.
filter(str_detect(name, "a")) filters rows where the name column contains the letter “a”.

Dropping Rows with a Specific StringTo drop rows containing “a” using dplyr, we use filter() with the negation operator !:

dropped\_rows\_dplyr <- data %>% filter(!str\_detect(name, "a"))print(dropped\_rows\_dplyr)

id name1 3 cherry2 5 elderberry Explanation:

!str_detect(name, "a") negates the condition, filtering out rows where the name column contains the letter “a”.

SummaryBoth base R and dplyr provide powerful ways to select and drop rows based on specific strings. The grep() function in base R and the combination of filter() and str_detect() in dplyr are versatile tools for your data manipulation needs.

Give these examples a try with your own datasets! Experimenting with different strings and data structures will help reinforce these concepts and improve your data manipulation skills.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Drop or Select Rows with a Specific String in R

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Duplicate and concatenate in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Duplicate and concatenate in R, we will explore how to use the str_dup function in R to duplicate and concatenate strings within a character vector.

The str_dup function is part of the stringr package, which provides a variety of functions for working with strings.

Example Data

To demonstrate the usage of str_dup, we will create a character string x containing the letters “ABC” and three dots:

Top 10 online data science programs » Data Science Tutorials

x <- "ABC..." Example: Application of str_dup Function in R

Before we can use the str_dup function, we need to install and load the stringr package:

install.packages("stringr")library("stringr") Now, we can use the str_dup function to duplicate the character string x five times:

str\_dup(x, 5) This will output a new character string containing the original string x five times:

Descriptive statistics vs Inferential statistics: Guide » Data Science Tutorials

```

"ABC...ABC...ABC...ABC...ABC..."

`` As you can see, thestr_dupfunction duplicates the original stringx` five times and concatenates the results into a single character string.

ConclusionIn this article, we have learned how to use the str_dup function in R to duplicate and concatenate strings within a character vector.

By using the str_dup function, you can easily create repeated strings for various purposes, such as data manipulation or text analysis.

How to calculate Hamming Distance in R
aggregate Function in R- A powerful tool for data frames
McNemar’s test in R
Data Science Job Portal in India
Markov Chain Introduction in R
List of Places to Visit in Wayanad-Kerala
How to Plot Categorical Data in R-Quick Guide

The post Duplicate and concatenate in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Duplicate and concatenate in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionSplitting numbers into individual digits can be a handy trick in data analysis and manipulation. Today, we’ll explore how to achieve this using base R functions, specifically gsub() and strsplit(). Let’s walk through the process step by step, explain the syntax of each function, and provide some examples for clarity.

SyntaxUnderstanding gsub() and strsplit()First, let’s get familiar with the two main functions we’ll be using:

gsub(pattern, replacement, x):
- pattern: A regular expression describing the pattern to be matched.
- replacement: The string to replace the matched pattern.
- x: The input vector, which is usually a character string.

The gsub() function replaces all occurrences of the pattern in x with the replacement.

strsplit(x, split):
- x: The input vector, which is usually a character string.
- split: The delimiter on which to split the input string.

The strsplit() function splits the elements of a character vector x into substrings based on the delimiter specified in split.

ExamplesSplitting a Number into DigitsLet’s go through a few examples to see how we can split numbers into digits using these functions.

Example 1: Basic Splitting of a Single Number ```

Step 1: Convert the number to a character stringnumber <- 12345number_str <- as.character(number)number_str

```

[1] "12345"

```

Step 2: Use gsub() to insert a delimiter (space) between each digitnumber_with_spaces <- gsub("(.)", "\1 ", number_str)number_with_spaces

```

[1] "1 2 3 4 5 "

```

Step 3: Use strsplit() to split the string on the delimiterdigits <- strsplit(number_with_spaces, " ")[[1]]# Step 4: Convert the result back to numericdigits_numeric <- as.numeric(digits)# Print the resultprint(digits_numeric)

```

[1] 1 2 3 4 5 Explanation:

We convert the number to a character string using as.character().
We use gsub("(.)", "\\1 ", number_str) to insert a space between each digit. The pattern (.) matches any character, and \\1 refers to the matched character followed by a space.
We split the string on spaces using strsplit(number_with_spaces, " ").
Finally, we convert the resulting character vector back to numeric using as.numeric().

Example 2: Splitting Multiple Numbers in a Vector ```

Vector of numbersnumbers <- c(6789, 5432)# Function to split a single number into digitssplit_number <- function(number) { number_str <- as.character(number) number_with_spaces <- gsub("(.)", "\1 ", number_str) digits <- strsplit(number_with_spaces, " ")[[1]] as.numeric(digits)}# Apply the function to each number in the vectorsplit_digits <- lapply(numbers, split_number)# Print the resultprint(split_digits)

```

[[1]][1] 6 7 8 9[[2]][1] 5 4 3 2 Explanation:

We define a vector of numbers.
We create a function split_number that takes a number and splits it into digits using the same steps as in Example 1.
We apply this function to each number in the vector using lapply().
The result is a list where each element is a vector of digits for each number in the original vector.

Try It Yourself!Now that we’ve gone through the examples, it’s your turn to give it a try! Experiment with different numbers, vectors, and even customize the splitting function to handle special cases or additional formatting. The more you practice, the more comfortable you’ll become with these handy base R functions.

Happy Coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Split a Number into Digits in R Using gsub() and strsplit()

View Details

[This article was first published on vusaverse, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Automate Canvas LMS Tasks with vvcanvasIntroductionToday, we’re thrilled to introduce the vvcanvas package, an R library aimed at enhancing interactions with the Canvas Learning Management System (LMS). Whether you’re involved in development, teaching, or research, this package offers a robust interface to tap into the extensive capabilities of Canvas LMS via R programming.

What is vvcanvas?vvcanvasThe vvcanvas package serves as a bridge between R and the Canvas LMS API, allowing users to authenticate, retrieve course information, fetch specific details, and perform various operations within the Canvas ecosystem. It’s like having a personal assistant for your Canvas needs, right from your R console!

Getting Started with vvcanvasHere’s a quick guide to get you up and running:

Acquire an API key from your Canvas LMS instance. Your Canvas administrator or the Canvas API documentation can guide you through this process.
Load the vvcanvas library and use the canvas_authenticate function to authenticate. Here’s an example snippet:

```

Install the package from CRAN install.packages("vvcanvas")## Or install the development version from GitHub devtools::install_github("vusaverse/vvcanvas")## Load package library(vvcanvas)## Authenticate on the Tableau Server# Replace the placeholders with your API key and base URLapi_key <- "YOUR_API_KEY"base_url <- "https://your_canvas_domain.com/"# Authenticate with the Canvas LMS APIcanvas <- canvas_authenticate(api_key, base_url)## Alternatively, you can set system variables# Set the API key and base URL as environment variablesSys.setenv(CANVAS_API_KEY = "YOUR_API_KEY")Sys.setenv(CANVAS_BASE_URL = "https://your_canvas_domain.com/")# Authenticate with the Canvas LMS APIcanvas <- canvas_authenticate()## The above "canvas" object can now be passed in every Canvas LMS API method.

`` With the authentication step completed, you can now utilize the various functions provided by thevvcanvas` package to interact with the Canvas LMS.

In order to retrieve a dataframe with all courses you can use the following function:

```

Pass the canvas object to the get_courses functioncourses <- get_courses(canvas)

`` Contributing to vvcanvasvvcanvas` is an open-source project, and contributions from the community are highly encouraged. If you encounter any bugs, have feature requests, or would like to contribute code improvements, you can open an issue or submit a pull request on the GitHub repository.

Parting Note: Canvas Analytics ProjectAs you delve into vvcanvas, you might wonder about the broader implications of leveraging Canvas data. That’s where the canvas-analytics project comes into play. This repository focuses on creating a data pipeline for extracting, aggregating, and analyzing data from Canvas LMS. It’s all about turning raw data into actionable insights, facilitating data-driven decisions across education and learning analytics.

Stay tuned for Part II of this blog series, where we’ll dive deeper into the canvas-analytics project, exploring how it empowers educators, researchers, and administrators with data-driven insights.

Whether you’re new to R or a seasoned programmer, the vvcanvas package and the upcoming exploration of the canvas-analytics project promise to open up exciting possibilities for leveraging Canvas LMS data. Let’s embark on this journey together, transforming the way we approach educational technology and data analytics.

Further readingLiked this post? Check out more R-related content on r-bloggers.com.

Have a look at our other packages in the vusaverse collection:

polyglotr: R package for translation services
vvdoctor: Shiny app for statistics.
vvcanvas: R package for Canvas LMS.
vvtableau: R package for Tableau Server.
vvtermtime: R package for Semestry TermTime.
vvauditor: R package for auditing data.
vvconverter: R package for converting data.
vvfiller: R package for filling data.
vvmover: R package for reading and writing data.
vvsculptor: R package for sculpting data.
vvshiny: R package to create shiny apps.

To leave a comment for the author, please follow the link and comment on their blog: vusaverse.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introduction to vvcanvas

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#183–184

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #183This Saturday we had quite interesting case to solve. We had table with rental agreements, which need to be transformed to kind of payment schedule. We have lenght of contract, interest rate increase after first year, and so on. It is one of cases where we can use formula for compound percent. Check it out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_183.xlsx", range = "A1:F5")test = read\_excel("Power Query/PQ\_Challenge\_183.xlsx", range = "H1:K24") %>% mutate(Rental = as.integer(Rental)) Transformation result = input %>% unite("OYQ", Year, Quarter, sep = " ") %>% mutate(OYQ = yq(OYQ)) %>% rowwise() %>% mutate(quarters = list(seq.Date(from = as.Date(OYQ), by = "quarter", length.out = `Total Periods`))) %>% ungroup() %>% unnest(quarters) %>% mutate(Year = year(quarters), Quarter = paste0("Q",quarter(quarters)), rn = row\_number(), roll\_year = (rn - 1) %/% 4 , .by = Vendor) %>% mutate(Rental = round(Rental * (1 + `% Hike Yearly`/100)^roll\_year) %>% as.integer()) %>% select(Vendor, Year, Quarter, Rental) Validation identical(result, test)# [1] TRUE Puzzle #184Sunday with Regex… good mind workout. Today we have some strings. And inside them suppose to be sequence as follow: Letters followed by digits. Sometimes there are more then one of such sequences, sometimes there are not even one. So we have to take last possible sequence from given string and concatenate them together inside the group. Conditional structures need to be used as well. Lets do it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_184.xlsx", range = "A1:B10")test = read\_excel("Power Query/PQ\_Challenge\_184.xlsx", range = "D1:G4") Transformation result = input %>% mutate(group = str\_extract\_all(Text,"[A-Za-z]+\\d+")) %>% mutate(group = map\_chr(group, ~if(length(.x) > 1) tail(.x, 1) else if(length(.x) == 0) NA\_character\_ else .x)) %>% summarise( Text = paste(group[!is.na(group)], collapse = "-"), `Original Count` = n() %>% as.numeric(), `New Count` = sum(!is.na(group)) %>% as.numeric(), .by = Set ) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.At Appsilon, we are committed to supporting the R community and empowering developers. We are thrilled to introduce our Hex Logo Design Contest—a great opportunity to enhance your R package with a unique, custom-designed hex logo!

‍

If you’re the creator of an R package that needs a visually appealing logo or if your current logo is outdated and needs a refresh, this contest is for you. We understand the importance of branding and visual identity, so we’re offering free, custom hex logo designs for a selected package.

‍

Didn’t make it to ShinyConf 2024? Catch up on the highlights in our blog post.

Why Join the Hex Logo Design Contest?Here are a few compelling reasons:

Stand Out with Better Branding: A well-designed logo makes your package look more professional and attention-grabbing to new users.
Get a Tailored High-Quality Design: Winners get a brand new hex logo custom-made for their package by our skilled designer, Hubert Hałun.
Work Directly with the Designer: You’ll have 1-2 sessions with Hubert to give feedback and request changes until the logo is perfect.
Make Your Package Look Amazing: With a slick new logo, your package will look modern, polished and super appealing.

‍

To enter the contest, fill out our submission form by May 31st and provide details about your R package, including its name, purpose, functionality, and links. Don’t forget to briefly describe your desired design concept or any specific preferences for the logo.

‍

It’s important to note that only creators of R packages without an existing hex logo or those with a logo needing a makeover are eligible to apply, and each applicant can submit only one R package for consideration.

Sign Up for the Hex Contest Today!Don’t miss this opportunity to give your R package the visually appealing identity it deserves. Visit our contest page today, and let our designers work their magic!

Remember, a well-designed logo is not just an aesthetic enhancement; it’s a powerful tool that can elevate your package’s visibility, appeal, and credibility within the R community.

Visit our contest page and submit your R package now!

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Get a Free New Logo for Your R Package in Our Hex Design Contest

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Sort or Order Rank in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Sort or Order Rank in Rwith sort(), order(), and rank() Functions. We will learn how to sort data in R using the sort(), order(), and rank() functions.

These functions are essential for data manipulation and analysis in R.

Example 1: Sorting VectorsLet’s create an example vector and apply the sort(), order(), and rank() functions to it.

x <- c(4, -10, 8, 0)x# Sort the vectorsort(x)# [1] -10 0 4 8# Order the vectororder(x)# [1] 2 4 1 3# Rank the vectorrank(x)# [1] 3 1 4 2 Example 2: Ordering Data Frame RowsLet’s create an example data frame and order its rows by one of its columns.

my\_data <- data.frame(x, y = LETTERS[1:4])my\_data# Order the data frame by column xmy\_data[order(my\_data$x), ]# x y# 2 -10 B# 4 0 D# 1 4 A# 3 8 C Example 3: Sorting ListsLet’s create an example list and sort it by its names.

ggdogs on ggplot2 » Data Science Tutorials

my\_list <- list(B = 1:5, C = 6:10, A = LETTERS[15:19])my\_list# Sort the list alphabeticallymy\_list[order(names(my\_list))]# $A# [1] "O" "P" "Q" "R" "S"# # $B# [1] 1 2 3 4 5# # $C# [1] 6 7 8 9 10 Example 4: Sorting DescendinglyLet’s sort the same vector in descending order using the sort() and order() functions.

sort(x, decreasing = TRUE)# [1] 8 4 0 -10order(x, decreasing = TRUE)# [1] 3 1 4 2 Example 5: Sorting Data by GroupLet’s create an example data frame with multiple columns and sort it by group.

Triangular Distribution in R » Data Science Tutorials

my\_data2 <- data.frame(value = c(6, 2, 4, 1, 8, 5), group1 = c("B", "A", "B", "C", "A", "B"), group2 = c("E", "E", "E", "D", "D", "D"))my\_data2# Sort the data by groupmy\_data2[order(my\_data2$group1, my\_data2$group2), ]# value group1 group2# 3 8 A D# 2 2 A E# 5 5 B D# 6 6 B E# 4 4 B E# 1 1 C D Example 6: Lesser Known Sorting FunctionsLet’s explore some lesser known sorting functions in R.

is.unsorted(x)# [1] TRUEsort.list(x)# [1] -10 -10 -10 -10 -10 -10 -10 -10 -10 -10sort.int(x)# [1] -10 -10 -10 -10 -10xtfrm(x)# [1] -10 -10 -10 -10 Sort or Order Rank in R, These examples demonstrate how to use the sort(), order(), and rank() functions in R to sort data in various ways.

K Nearest Neighbor Algorithm in Machine Learning
How to Interpolate Missing Values in R With Example
Line Plots in R-Time Series Data Visualization
Python is superior to R for writing quality codes
Sample Size 30 Is it good enough?
How to Interpret a Standard Deviation of Zero?

The post Sort or Order Rank in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Sort or Order Rank in R

View Details

[This article was first published on ouR data generation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Over the past few years, a number of folks have asked if simstudy accommodates customized distributions. There’s been interest in truncated, zero-inflated, or even more standard distributions that haven’t been implemented in simstudy. While I’ve come up with approaches for some of the specific cases, I was never able to develop a general solution that could provide broader flexibility.

This shortcoming changes with the latest version of simstudy, now available on CRAN. Custom distributions can now be specified in defData and defDataAdd by setting the argument dist to “custom”. To introduce the new option, I am providing a couple of examples.

Specifying the customized distributionWhen defining a custom distribution in defData, you provide the name of the user-defined function as a string in the formula argument. The arguments of this custom function are listed in the variance argument, separated by commas and formatted as “arg_1 = val_form_1, arg_2 = val_form_2, (\dots), arg_K = val_form_K”.

The arg_k’s represent the names of the arguments passed to the customized function, where (k) ranges from (1) to (K). You can use values or formulas for each val_form_k. If formulas are used, ensure that the variables have been previously generated. Double dot notation is available in specifying value_formula_k. It is important to note that the parameter list of the actual function must include an argument”n = n”, but (n) should not be included in the definition as part of defData or defDataAdd (specified in the variance field).

Example 1Here is an example where we generate data from a zero-inflated beta distribution. (I’ve implemented something like this in the past using a mixture distribution, which is also a fine way to go). I’ve created a user-defined function zeroBeta that takes on shape parameters (a) and (b) for the beta distribution, as well as (p_0), the proportion of the sample that takes on a value of zero. Note that the function also takes an argument (n) that will not to be be specified in the data definition; (n) will represent the number of observations being generated:

zeroBeta <- function(n, a, b, p0) { betas <- rbeta(n, a, b) is.zero <- rbinom(n, 1, p0) betas*!(is.zero)} The data definition specifies that we want to create a variable (zb) from the user-defined zeroBeta function with (a) and (b) set to 0.75, and (p_0 = 0.02):

def <- defData( varname = "zb", formula = "zeroBeta", variance = "a = 0.75, b = 0.75, p0 = 0.02", dist = "custom") The data are generated with a call to genData as is typically done in simstudy:

set.seed(1234)dd <- genData(100000, def)## Key: <id>## id zb## <int> <num>## 1: 1 0.93922887## 2: 2 0.35609519## 3: 3 0.08087245## 4: 4 0.99796758## 5: 5 0.28481522## --- ## 99996: 99996 0.81740836## 99997: 99997 0.98586333## 99998: 99998 0.68770216## 99999: 99999 0.45096868## 100000: 100000 0.74101272 A plot of the data highlights an over-representation of zeroes:

Example 2In this second example, I am generating sets of truncated Gaussian distributions with means ranging from (-1) to (1). (I wrote about this a while ago - the approach implemented here is an alternative way to generate these data.) rnormt is a customized (user-defined) function that generates the truncated Gaussian data, and requires four arguments (the left truncation value, the right truncation value, the distribution average without truncation and the distribution standard deviation without truncation):

rnormt <- function(n, min, max, mu, s) { F.a <- pnorm(min, mean = mu, sd = s) F.b <- pnorm(max, mean = mu, sd = s) u <- runif(n, min = F.a, max = F.b) qnorm(u, mean = mu, sd = s) } In this example, truncation limits differ based on group membership. Initially, three groups are created (represented by the variable defined as limit), followed by the generation of truncated values (named tn). For Group 1, truncation is defined by the range of (-1) to (1); for Group 2, the range is (-2) to (2); and for Group 3, the range is (-3) to (3). We’ll generate three data sets, each with a distinct mean denoted by M, using the double-dot notation to implement the different means.

def <- defData( varname = "limit", formula = "1/4;1/2;1/4", dist = "categorical" ) |> defData( varname = "tn", formula = "rnormt", variance = "min = -limit, max = limit, mu = ..M, s = 1.5", dist = "custom" ) The data generation requires three calls to genData, one for each different mean value (\mu). I have chosen to implement this with lapply:

mu <- c(-1, 0, 1)dd <-lapply(mu, function(M) genData(100000, def)) The output is a list of three data sets; here are the first six observations from each of the three data sets:

```

[[1]]## Key: ## id limit tn## ## 1: 1 2 0.6949619## 2: 2 2 -0.3641963## 3: 3 2 -0.4721632## 4: 4 3 -2.6083796## 5: 5 2 -0.6800441## 6: 6 3 -0.5813880## ## [[2]]## Key: ## id limit tn## ## 1: 1 1 0.4853614## 2: 2 2 -0.5690811## 3: 3 2 0.5282246## 4: 4 2 0.1107778## 5: 5 2 -0.3504309## 6: 6 2 1.9439890## ## [[3]]## Key: ## id limit tn## ## 1: 1 2 1.3560628## 2: 2 2 1.4543616## 3: 3 3 1.4491010## 4: 4 2 0.7328855## 5: 5 2 -0.1254556## 6: 6 2 -0.7455908

``` A plot highlights the group differences for each of the three data sets:

To leave a comment for the author, please follow the link and comment on their blog: ouR data generation.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: simstudy 0.8.0: customized distributions

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 454–458

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #454Sombody was counting something, and as usually we have to check it and find the most insigthful info. We have to find the length of each sequence, but in some of the, there are ranges with text. That mean that we have to use some R magic. Let’s play.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/454 Extraction of number of nodes.xlsx", range = "A1:A9")test = read\_excel("Excel/454 Extraction of number of nodes.xlsx", range = "B1:B9") Transformation replace\_notation\_with\_range <- function(text\_vector) { str\_replace\_all(text\_vector, "\\d+ to \\d+", function(match) { numbers <- str\_split(match, " to ") %>% unlist() %>% as.numeric() range <- seq(from = numbers[1], to = numbers[2]) paste(range, collapse = ", ") })}count\_numbers <- function(text\_vector) { str\_count(text\_vector, "\\d+") %>% as.numeric()}result = input %>% mutate(Pronlem = str\_to\_lower(Pronlem)) %>% mutate(Pronlem = map\_chr(Pronlem, replace\_notation\_with\_range)) %>% mutate(Count = count\_numbers(Pronlem)) %>% select(Count) Validation identical(result$Count, test$`Answer Expected`)# [1] TRUE Puzzle #455Have you heard of antiperfect numbers? They are in some weird way perfect to me… perfect to play with. We need to find out which of given numbsers are antiperfect. But what does it mean? That if you take all divisors except number itself, change order of letters and add them up, they will be equal to original number. Let’s do it.

Loading libraries and data library(tidyverse)library(readxl)library(numbers)input = read\_excel("Excel/455 Anti perfect numbers.xlsx", range = "A1:A10")test = read\_excel("Excel/455 Anti perfect numbers.xlsx", range = "B1:B5") Transformation is\_antiperfect = function(number) { divisors = divisors(number) divisors = divisors[-length(divisors)] reversed\_divisors = map(divisors, ~str\_c(rev(str\_split(.x, "")[[1]]), collapse = "")) %>% as.numeric() sum\_rev\_div = sum(reversed\_divisors) return(sum\_rev\_div == number)}result = input %>% mutate(is\_antiperfect = map\_lgl(Numbers, is\_antiperfect)) %>% filter(is\_antiperfect) %>% select(`Expected Answer` = Numbers) Validation identical(result, test)# [1] TRUE Puzzle #456Today’s challenge is pretty easy. And that is why I will give you two ways to do it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/456 Extract special Characters.xlsx", range = "A1:A10")test = read\_excel("Excel/456 Extract special Characters.xlsx", range = "B1:B10") Transformation — approach 1 ```

approach 1 - remove alphanumerics result = input %>% mutate(String = str_replace_all(String, "[[:alnum:]]", "")) %>% mutate(String = ifelse(String == "", NA, String))

Transformation — approach 2

approach 2 - extract special charactersresult2 = input %>% mutate(String = str_extract_all(String, "[^[:alnum:]]") %>% map_chr(~paste(.x, collapse = ""))) %>% mutate(String = ifelse(String == "", NA, String))

Validation identical(result$String, test$Expected Answer)#> [1] TRUEidentical(result2$String, test$Expected Answer)#> [1] TRUE ``` Puzzle #457Today we have task similar, but we have more complicated case. I used Regex capacities to find all numbers that are “hugged”with any kind of parenthesis. I can say that except Regex itself pretty easy case. Regex needs to use lookbehind and lookahead in cases.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/457 Extract Numbers in Parenthesises.xlsx", range = "A1:A10")test = read\_excel("Excel/457 Extract Numbers in Parenthesises.xlsx", range = "B1:B10") Transformation library(tidyverse)library(readxl)input = read\_excel("Excel/457 Extract Numbers in Parenthesises.xlsx", range = "A1:A10")test = read\_excel("Excel/457 Extract Numbers in Parenthesises.xlsx", range = "B1:B10") Validation identical(result, test)# [1] TRUE Puzzle #458Capital letters stands out in text as too tall soldier in a row. And our host gave us chance to make special meeting for the tallest soldiers. We need to find longest sequence of capital letters in this words. If there are more than one possible, we need to concatenate them.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/458 Maximum Consecutive Uppercase Alphabets.xlsx", range = "A1:A11")test = read\_excel("Excel/458 Maximum Consecutive Uppercase Alphabets.xlsx", range = "B1:B11") Transformation get\_longest\_capital = function(string) { caps = str\_extract\_all(string, "[A-Z]+") %>% unlist() caps\_len = ifelse(length(caps) == 0, NA, max(nchar(caps))) caps = caps[nchar(caps) == caps\_len] %>% paste0(collapse = ", ") return(caps)}result = input %>% mutate(ans = map\_chr(Words, get\_longest\_capital)) %>% mutate(ans = ifelse(ans == "", NA\_character\_, ans)) Validation all.equal(result$ans, test$`Expected Answer`)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
PS. Couple weeks ago, I started uploading on Github not only R, but also in Python. Come and check it.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Effective Visual Communication with R, which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Effective Visual Communication with R

Date: Thursday, June 20th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Claus Wilke is a data scientist and computational biologist at The University of Texas at Austin. He is known for his work on popular R packages for data visualization, such as cowplot, ggridges, and ggtext, as well as his contributions to the package ggplot2. He is also the author of the book Fundamentals of Data Visualization, published in 2019, which provides a concise introduction to effectively visualizing many different types of data sets.

Description: In the first half of this workshop, Wilke will provide a high-level perspective on how to make good visualizations and how to use them effectively to communicate and reason about data. The second half will be more hands-on and will address how to use R to make interactive plots, deal with overplotting, and make compound figures.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Effective Visual Communication with R workshop was first posted on May 20, 2024 at 2:56 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Effective Visual Communication with R workshop

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.If there’s one thing with a certain downward trend, it’s got to be people’s attention span. Even Google reports that pages with a load time of 5 seconds increase their probability of bounce by 90%! And that was in 2017!

As an R Shiny developer, you must do everything in your power to stop users leaving your app and increase engagement. One way to do it is to speed up data loading and processing time. Combining R Shiny and DuckDB is an excellent way to go.

In this article, you’ll learn what happens to R when you try to load over 200 million rows of data in your Shiny app, and what are the go-to ways to make it happen in seconds, not minutes.

Looking for an alternative way to process huge amounts of data? Try R dtplyr – A dplyr-like package with a data.table backend.

Table of Contents:* DuckDB vs. Dplyr – How Much Faster is DuckDB On Real Data? * Bringing DuckDB to R Shiny – How to Speed Up Slow Shiny Apps * Summing up R Shiny and DuckDB

DuckDB vs. Dplyr – How Much Faster is DuckDB On Real Data?We’ve already written about DuckDB on our blog, so we won’t repeat ourselves here. Today it’s all about hands-on experience.

The goal of this section is to test how much faster DuckDB is when compared to dplyr – a de facto standard analysis tool in R.

Dataset ConfigurationIf you want to follow along, you’ll need to download monthly Yellow Taxi trip data from 2019 to 2023 in Parquet format. You’ll end up with 60 individual data files, all taking around 3.3 GB of disk space:

Image 1 – Taxi dataset split into multiple parquet filesThe Shiny application you’ll build shortly will load the data one year (12 Parquet files) at a time. For that reason, let’s also inspect how many rows of data you’ll be working with:

Image 2 – Row counts for each dataset yearShort answer – a lot. Over 218 million in total, with 2019 alone having more than 84 million of recorded taxi rides. If even the idea of working with this amount of data in R Shiny makes your head spin, well, you’re not alone.

Required R PackagesYou’ll need the following packages installed to follow along:

library(arrow)library(dplyr)library(lubridate)library(duckdb)library(stringr) If any of these are not installed, simply run the install.packages(“”) command from the R console to get you going.

Benchmark Function – DplyrOnto the fun stuff now!

The goal of this section is to write a function that reads one year’s worth of Parquet files from disk and performs some sort of aggregation. We’ll keep things simple and only calculate the basic statistics – number of rides, average duration, distance, price, tip amount, and total distance and price – all on a monthly basis.

If you know dplyr, the aggregation shouldn’t look foreign to you. What might look relatively new is the way you need to load the data. Put simply, the data is scattered between 12 Parquet files and you need a dynamic way to get the full file paths. Once you have that, you can use the arrow::open\_dataset() function to turn the results into an Arrow table, and then into a data frame.

Here’s the full logic for our get\_dataset\_for\_year\_dplyr() function:

get\_dataset\_for\_year\_dplyr <- function(folder\_path, file\_year) { time\_start <- Sys.time() file\_paths <- c() # Months are represented as strings in parquet file names months <- c("01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12") # Iterate and append full file path to `file\_paths` for (mnt in months) { file\_paths <- append( file\_paths, paste0(folder\_path, "/yellow\_tripdata\_", file\_year, "-", mnt, ".parquet") ) } # Open the dataset for an entire year - 12 parquet files - and aggregate res <- as.data.frame(Scanner$create(open\_dataset(file\_paths))$ToTable()) %>% mutate( period = format(as.Date(tpep\_pickup\_datetime), "%Y-%m"), trip\_duration = as.integer(tpep\_dropoff\_datetime - tpep\_pickup\_datetime) ) %>% filter(grepl(file\_year, period)) %>% group\_by(period) %>% summarise( num\_rides\_m = round(n() / 1000000, 2), avg\_trip\_duration = as.integer(mean(trip\_duration)), avg\_trip\_distance = round(mean(trip\_distance), 2), total\_trip\_distance\_m = round(sum(trip\_distance) / 1000000, 2), avg\_trip\_price = round(mean(total\_amount), 2), total\_trip\_price\_m = round(sum(total\_amount) / 1000000, 2), avg\_tip\_amount = round(mean(tip\_amount), 2) ) %>% arrange(period) time\_end <- Sys.time() # Return both the aggregated dataset and the runtime return(list( data = res, runtime = as.numeric(time\_end - time\_start) ))} The function returns both the aggregated dataset and the time it took to read and parse the data. This second piece of information will be crucial for a comparison later.

Benchmark Function – DuckDBLet’s now do the same but with DuckDB!

The get\_dataset\_for\_year\_duckdb() function accomplishes the same as the previous one, but by using DuckDB instead of dplyr. One clear benefit here is that you can use a Glob pattern to specify file locations instead of creating a list in a loop.

DuckDB allows you to write aggregations through dplyr-like functions or through SQL. We’ve opted for the latter, just to introduce some variety:

get\_dataset\_for\_year\_duckdb <- function(folder\_path, file\_year) { time\_start <- Sys.time() # DuckDB can read files from folder by using a glob pattern parquet\_path <- paste0(folder\_path, "/yellow\_tripdata\_", file\_year, "-*.parquet") # SQL statement to perform data aggregation # String interpolation is used to inject dynamic string parts into the query duck\_query <- str\_interp(" select period, round(count(*) / 1000000, 2) as num\_rides\_m, cast(avg(trip\_duration) as integer) as avg\_trip\_duration, round(avg(trip\_distance), 2) as avg\_trip\_distance, round(sum(trip\_distance) / 1000000, 2) as total\_trip\_distance\_m, round(avg(total\_amount), 2) as avg\_trip\_price, round(sum(total\_amount) / 1000000, 2) as total\_trip\_price\_m, round(avg(tip\_amount), 2) as avg\_tip\_amount from ( select date\_part('year', tpep\_pickup\_datetime) as trip\_year, strftime(tpep\_pickup\_datetime, '%Y-%m') as period, epoch(tpep\_dropoff\_datetime - tpep\_pickup\_datetime) as trip\_duration, trip\_distance, total\_amount, tip\_amount from parquet\_scan('${parquet\_path}') where trip\_year = ${file\_year} ) group by period order by period ") # Run the query with DuckDB in memory res <- dbGetQuery( conn = dbConnect(duckdb()), statement = duck\_query ) time\_end <- Sys.time() # Return both the aggregated dataset and the runtime return(list( data = res, runtime = as.numeric(time\_end - time\_start) ))} And now, let’s compare the two!

Dplyr vs. DuckDB Benchmark ResultsNote: The actual runtimes you’ll get will depend on the hardware you’re running R on. For reference, we’re using a 16” M3 Pro Macbook Pro with 12 CPU cores and 36 GB of RAM. If you can feed more cores to the underlying DuckDB engine, you’ll likely be able to process the data even faster.

Running the get\_dataset\_for\_year\_dplyr() and get\_dataset\_for\_year\_duckdb() will return two values – the aggregated data and the runtime. Let’s take a peek into the data first:

Image 3 – Aggregated dataset sampleIt’s a nice, condensed, and informative overview of the millions of rows that went into the calculation.

As for the runtimes, the results are interesting:

Image 4 – Dataset aggregation time in secondsRemember the 5-second rule from the introduction? Imagine what would happen if the user had to wait almost a minute! To make things worse, you’ll probably host your Shiny app on a significantly less powerful server.

‍

Nevertheless, DuckDB is a clear winner – being around 35x faster. Let’s now use it in R Shiny to see how that translates to application load speeds.

Bringing DuckDB to R Shiny – How to Speed Up Slow Shiny AppsYou’ll want to start by creating two files – helpers.R and app.R. In the prior, simply paste all the package imports and logic for our two data loading functions (Dplyr and DuckDB).

As for the latter, let’s get into it!

Shiny App CodeThe application you’ll write now will be quite straightforward, as it will show one table and three charts. The user will be able to select a year from a dropdown menu, and the visuals will be rerendered once the data is loaded. You’ll also see the total loading time. It’s an unnecessary component for a production app, sure, but is here just to illustrate a point.

As for the external packages, you’ll want to make sure you’re comfortable with the following ones:

DT: A package used to visualize table data in R and R Shiny. It has built-in search and pagination functionalities that will come in handy.
Highcharter: Amazing package for producing interactive and animated visualizations for R and R Shiny.

The app is split into two parts – a sidebar that shows the title, filters, and the runtime in seconds, and a main panel that shows the visuals.

In the server() function, you’re free to change the data loading function (Dplyr or DuckDB) to see the performance differences between the two. The code snippet below uses the get\_dataset\_for\_year\_duckdb() function.

Anyway, here’s the full code snippet:

```

Load the file with the helper functionssource("helpers.R", local = TRUE)library(shiny)library(DT)library(highcharter)ui <- fluidPage( sidebarLayout( sidebarPanel( tags$h3("Yellow Taxi App"), tags$hr(), # User can select a year between 2019 and 2023 selectInput(inputId = "inYear", label = "Year:", choices = c(2019, 2020, 2021, 2022, 2023), selected = 2023), tags$hr(), tags$h5("Data load time:"), textOutput(outputId = "outTextDuration") ), mainPanel( DTOutput(outputId = "outTable"), highchartOutput(outputId = "outChartNumRides", height = 500), highchartOutput(outputId = "outChartTripDistance", height = 500), highchartOutput(outputId = "outChartTripPrice", height = 500) ) ))server <- function(input, output) { # Use either `get\_dataset\_for\_year\_dplyr` or `get\_dataset\_for\_year\_duckdb` to get the dataset and the runtime data <- reactive({ get_dataset_for_year_duckdb( folder_path = "", file_year = input$inYear ) }) # Duration text output$outTextDuration <- renderText({ paste0(round(data()$runtime, 2), " seconds.") }) # Table output$outTable <- renderDT({ datatable( data = data()$data, colnames = c( "Period", "Number of rides (Million)", "Avg. Ride Duration (Seconds)", "Avg. Ride Distance", "Total Ride Distance", "Avg. Ride Price", "Total Ride Price (Million)", "Avg. Ride Tip Amount" ), caption = paste("Yellow Taxi rides summary statistics for", input$inYear, "."), filter = "top" ) }) # Chart - Number of rides output$outChartNumRides <- renderHighchart({ hchart(data()$data, "column", hcaes(x = period, y = num_rides_m), color = "#800000", name = "#Rides (million)") |> hc_title(text = "Number of rides per month (millions)", align = "left") |> hc_xAxis(title = list(text = "Time period")) |> hc_yAxis(title = list(text = "Number of rides (millions)")) }) # Chart - Trip distance output$outChartTripDistance <- renderHighchart({ hchart(data()$data, "line", hcaes(x = period, y = avg_trip_distance), color = "#0198f9", name = "Distance") |> hc_title(text = "Average ride distance in km", align = "left") |> hc_xAxis(title = list(text = "Time period")) |> hc_yAxis(title = list(text = "Distance (km)")) }) # Chart - Trip price output$outChartTripPrice <- renderHighchart({ hchart(data()$data, "line", hcaes(x = period, y = avg_trip_price), color = "#fe6706", name = "Price") |> hc_title(text = "Average ride price in $", align = "left") |> hc_xAxis(title = list(text = "Time period")) |> hc_yAxis(title = list(text = "Price ($)")) })}shinyApp(ui = ui, server = server)

``` You can now run the app to see what it looks like:

Image 5 – Shiny application (1)It seems like the data for 2023 loaded in around 0.5 seconds with DuckDB!

You can see the monthly statistics in a table form, or through three different charts:

Image 6 – Shiny application (2)Overall, it’s a simple R Shiny application that could benefit from adding more content and tweaking the visuals, but we’ll leave that up to you.

Our task is to compare the runtime differences, so let’s start with dplyr.

Runtime Examination – DplyrUsing the application version that runs the get\_dataset\_for\_year\_dplyr() function is a painful experience, to say the least. Cold starting the application took a painfully long 22 seconds:

Image 7 – Dplyr application version (1)But, things get worse as the data grows in size. For example, loading the 2019 data (84.5 million rows) took almost a minute:

Image 8 – Dplyr application version (2)That’s a full minute that your user can’t interact with the application. Maybe the wait time is worth it, but we have our doubts.

Runtime Examination – DuckDBSwitching data processing backends from dplyr to DuckDB makes a world of difference. For example, loading the largest subset (2019) takes only 1.5 seconds:

Image 9 – DuckDB application version (1)2022, which has just under 40 million rows is done loading and parsing in under 1 second:

Image 10 – DuckDB application version (2)We don’t have much else to say, the numbers speak for themselves.

Runtime ComparisonTo remove any chance of “luck” from our runtime comparison, we’ve run the data loading process in R Shiny 5 times for both Dplyr and DuckDB and for every year. Here are the average load time numbers:

Dplyr:

2019 – 56.54 seconds
2020 – 17.22 seconds
2021 – 19.03 seconds
2022 – 26.24 seconds
2023 – 25.69 seconds

DuckDB:

2019 – 1.51 seconds
2020 – 0.98 seconds
2021 – 0.66 seconds
2022 – 0.73 seconds
2023 – 0.61 seconds

In other words, Dplyr takes 0.61 to 0.7 seconds on average to read and parse 1 million rows of Parquet data, while DuckDB takes only 0.016 to 0.04 seconds for the same task.

Summing up R Shiny and DuckDBTo conclude, the technology you choose to read and process data for your Shiny app matters. It’s easy to point fingers at the web framework, but in reality, that’s almost never a bottleneck. You’ve seen just which sorts of performance improvements you can get by switching from Dplyr to DuckDB. It’s not even close!

Now, to get this level of performance from DuckDB you’ll need a powerful CPU with as many cores as you can get. On the other hand, R dplyr is single-threaded, so going for a 16-core CPU configuration in deployment scenarios doesn’t make much sense.

It’s a tradeoff between a fast-running application and a happy wallet. Optimize for what you need.

Your Shiny app works, but what’s next? It’s time to make it beautiful. Read our latest blog post to find out how.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Shiny and DuckDB: How to Speed Up Your Shiny Apps When Working With Large Datasets

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Divide data into groups in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Divide data into groups in R, we will learn how to use the split and unsplit functions in R to divide and reassemble vectors into groups.

These functions are useful when you need to separate a large dataset into smaller groups based on specific criteria and then reassemble the data back into a single vector.

Definitions and Basic R Syntaxes

The split function divides data into groups, while the unsplit function reverses the output of the split function. The basic R syntaxes for these functions are:

split(values, groups)unsplit(split\_values, groups) Creation of Example Data

We will create an example vector and a grouping vector to demonstrate the use of the split and unsplit functions.

What is the best way to filter by row number in R? » Data Science Tutorials

vec <- 1:10vec# 1 2 3 4 5 6 7 8 9 10groups <- c(rep("A", 3), rep("B", 5), rep("C", 2))groups# "A" "A" "A" "B" "B" "B" "B" "B" "C" "C" Example 1: Using split() Function in R

In this example, we will use the split function to divide our example data into three groups based on the grouping vector.

my\_split <- split(vec, groups)my\_split# $A# [1] 1 2 3# # $B# [1] 4 5 6 7 8# # $C# [1] 9 10 As you can see, the split function created a list called my_split, which contains three list elements, each representing a group.

Stacked Barplot in R » Data Science Tutorials

Example 2: Using unsplit() Function in R

In this example, we will use the unsplit function to reassemble the data back into a single vector.

my\_unsplit <- unsplit(my\_split, groups)my\_unsplit# [1] 1 2 3 4 5 6 7 8 9 10 As you can see, the unsplit function successfully reassembled the data back into a single vector.

ConclusionIn this tutorial, we have learned how to use the split and unsplit functions in R to divide and reassemble vectors into groups.

We have demonstrated how to use these functions to separate a large dataset into smaller groups based on specific criteria and then reassemble the data back into a single vector.

With these functions, you can easily manipulate and analyze large datasets in R.

Reinforcement Learning in Machine Learning
How to do Data Format in R
How to Join Objects in R
How to select mutual funds?
Introduction to Hadoop Data Processing Applications
How to add straight lines to a plot in R
Minimum number of units in an Experimental Design
What is the future of data analytics?

The post Divide data into groups in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Divide data into groups in R

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Extract certain rows of data set in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Extract certain rows of data set in R, we will learn how to extract specific rows of a data set using the slice function of the dplyr package in R.

This function is useful when you need to extract specific rows of a large data set and perform further analysis on those rows.

Creation of Example Data

We will use the following data frame as an example:

data <- data.frame(x1 = 1:5, x2 = LETTERS[1:5], x3 = 5)data# x1 x2 x3# 1 1 A 5# 2 2 B 5# 3 3 C 5# 4 4 D 5# 5 5 E 5 This data frame contains five rows and three columns. We will use this data frame to demonstrate how to use the slice function.

How to Remove Columns from a data frame in R » Data Science Tutorials

Example: Application of slice Function

The slice function can be used to extract specific rows of a data frame.

To use the slice function, we need to specify the name of our input data and the row index of all rows we want to retain.

For example, we can extract the first, third, and fifth row of the example data as follows:

slice(data, c(1, 3, 5))# x1 x2 x3# 1 1 A 5# 2 3 C 5# 3 5 E 5 In this example, we extracted the first, third, and fifth row of the example data.

Example: Extracting Specific Rows with a Condition

We can also use the slice function to extract specific rows based on a condition.

Data Science Challenges in R Programming Language (datasciencetut.com)

For example, we can extract all rows where the value in column x1 is greater than or equal to 3 as follows:

slice(data, x1 >= 3)# x1 x2 x3# 2 2 B 5# 3 3 C 5# 4 4 D 5# 5 5 E 5 In this example, we extracted all rows where the value in column x1 is greater than or equal to 3.

ConclusionIn this tutorial, we have learned how to use the slice function of the dplyr package in R to extract specific rows of a data set.

We have demonstrated how to use the slice function to extract specific rows based on a condition and how to extract specific rows by specifying the row index.

With these examples, you can easily extract specific rows of a large data set and perform further analysis on those rows.

Self Organizing Maps in R- Supervised Vs Unsupervised
Restricted Boltzmann Machine (RBM)
TBATS Time Series Modelling in R
Concatenate inputs in R
A Comprehensive Guide To Python For Image-Based Data Mining
How to interpret Margin of Error Results?
Group wise Correlation in R

The post Extract certain rows of data set in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extract certain rows of data set in R

View Details

[This article was first published on business-science.io, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Hey guys, I have a special announcement. I’m gearing up for the launch of my new course: Generative AI for Data Scientists! In just 11 days, my new course will be live. Here’s everything you need to know.

Special Announcement: Generative AI Course Launch on May 30thInside the live launch event, I’ll share 6 months of research on AI, LLMs, and where it fits for Data Scientists who want to accelerate their careers.

What: Generative AI for Data Scientists

When: Thursday May 30th, 2pm EST

How It Will Help You: Whether you are new to data science or are an expert, AI is changing the game. There’s a ton of hype. But how can AI actually help you become a better data scientist and help you avoid layoffs and stand out in your career? I’ll show you inside my live course launch event.

Price: Does Free sound good?

How To Join: Register Here

Table of Contents 4-Minute Video: Building an AI Business Intelligence Copilot: Get a sneak peek at one of the course projects * Why learn Generative AI if you’re a Data Professional? This is what I’ve discovered about AI. * Next Steps* How to attend the live event for free.

4-Minute Video: Building an AI Business Intelligence Copilot Watch this 4-minute video to get a sneak peek at 1 of the projects from the course launch event.

Why should you learn Generative AI if your a data professional?Over the past 6 months, I’ve been working with Python, Generative AI, and LLMs daily. And on May 30th, I’m excited to share what I’ve learned.

The demand for AI talent is skyrocketing.According to the Silicon Valley Business Journal, the typical AI engineer at OpenAI makes $925,000 a year.

That’s almost a million dollars. And it’s 3X the industry median for software engineers. Here’s why this is happening everywhere (and how you can be part of the great AI shift).

AI is changing the way businesses work, people get information, and processes happen. And the people who can build AI have unlimited career potential.

New Role: The AI Business ScientistWhile AI is replacing some jobs, it’s creating a new role: the AI Business Scientist.

And that’s why I’m excited to announce a new course is coming to my Business Science portfolio.

I’m showcasing how to take your data science skills to become the AI expert for your organization…

… using Python’s strengths:

OpenAI
LangChain
Streamlit
Scikit Learn
H2O
FastAPI

Complete 4 AI ProjectsThis course is like no other where I will teach you how to complete 4 AI projects that progressively build your AI and LLM skills:

Project 1: A custom streamlit web app to make summary PDF business reports from 100’s of pages of call transcripts Project 2: An AI-powered Marketing Strategy Assistant trained from a Marketing Strategy knowledge database (RAG Project) Project 3: A Customer Analytics AI Agent that combines Machine Learning, SQL Database, and Pandas skills and reports visualizations and tables based on your questions Project 4: A Multi-Agent App with a Supervisor Bot that controls your AI Marketing and BI botsExactly what companies have told me they need.

Next Step: Join me on May 30thIf you are ready to take the leap in your career, then join me and 3,500+ data scientists that have registered for my AI course waitlist + live launch event.

The Live Launch Event takes place Thursday, May 30th at 2pm EST.

There are a ton of new goodies I’m giving away and you’ll learn a ton about the Python AI & LLM ecosystem in the training.

Course Waitlist + Live Event Registration https://learn.business-science.io/python-generative-ai-apps

See you Thursday, May 30th at 2PM EST!

Your Friendly Neighborhood Data Scientist,

– Matt Dancho

Join the AI Course Waitlist + Live Event Registration

To leave a comment for the author, please follow the link and comment on their blog: business-science.io.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Course Launch: Generative AI for Data Scientists in 11 Days!

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A few weeks ago in #112 and #120, I presented a few Python examples of Deep Quasi-Randomized ‘neural’ networks (QRNs). In this post, I will provide a detailed introduction to this new family of models, with examples in Python and R, and a preprint.

Link to the preprint
Link to a Jupyter Python notebook with a benchmark on 14 data sets: https://github.com/Techtonique/nnetsauce/blob/master/nnetsauce/demo/thierrymoudiki_20240519_deep_qrns.ipynb

At the basis of Deep QRNs are QRNs; nnetsauce’s CustomClassifier class objects, which, in turn, depend on a base Machine Learning model. This base learner could be any classifier, and in particular any scikit-learn classifier, or xgboost, or else. Here is how a CustomClassifier works, with the information flowing from left to right (forward pass only)

Deep QRNs arise from the QRN presented in this figure: in the case where we’d like to have a 3-layered deep QRN, the base Machine Learning model depicted in the figure can in turn be a QRN, and the obtained QRN can in turn be a QRN. Here are R examples:

Install nnetsauce

install.packages("nnetsauce", repos = c("https://techtonique.r-universe.dev", "https://cran.r-project.org")) Load nnetsauce

library("nnetsauce") iris data

library(datasets)set.seed(123)X <- as.matrix(iris[, 1:4])y <- as.integer(iris$Species) - 1L# split data into training and test sets(index\_train <- base::sample.int(n = nrow(X), size = floor(0.8*nrow(X)), replace = FALSE))X\_train <- X[index\_train, ]y\_train <- y[index\_train]X\_test <- X[-index\_train, ]y\_test <- y[-index\_train]# base model is a Logistic Regressionobj2 <- sklearn$linear\_model$LogisticRegressionCV()# there are 3 layers in the deep modelobj <- DeepClassifier(obj2, n\_layers = 3L)# adjust the model res <- obj$fit(X\_train, y\_train)# accuracy, must be 1print(mean(obj$predict(X\_test)==y\_test)) palmer penguins data

library(palmerpenguins)data(penguins)penguins\_ <- as.data.frame(palmerpenguins::penguins)replacement <- median(penguins$bill\_length\_mm, na.rm = TRUE)penguins\_$bill\_length\_mm[is.na(penguins$bill\_length\_mm)] <- replacementreplacement <- median(penguins$bill\_depth\_mm, na.rm = TRUE)penguins\_$bill\_depth\_mm[is.na(penguins$bill\_depth\_mm)] <- replacementreplacement <- median(penguins$flipper\_length\_mm, na.rm = TRUE)penguins\_$flipper\_length\_mm[is.na(penguins$flipper\_length\_mm)] <- replacementreplacement <- median(penguins$body\_mass\_g, na.rm = TRUE)penguins\_$body\_mass\_g[is.na(penguins$body\_mass\_g)] <- replacement# replacing NA's by the most frequent occurencepenguins\_$sex[is.na(penguins$sex)] <- "male" # most frequent# one-hot encoding for covariatespenguins\_mat <- model.matrix(species ~., data=penguins\_)[,-1]penguins\_mat <- cbind.data.frame(penguins\_$species, penguins\_mat)penguins\_mat <- as.data.frame(penguins\_mat)colnames(penguins\_mat)[1] <- "species"y <- penguins\_mat$speciesX <- as.matrix(penguins\_mat[,2:ncol(penguins\_mat)])n <- nrow(X)p <- ncol(X)set.seed(1234)index\_train <- sample(1:n, size=floor(0.8*n))X\_train <- X[index\_train, ]y\_train <- factor(y[index\_train])X\_test <- X[-index\_train, ][1:5, ]y\_test <- factor(y[-index\_train][1:5])# base model is a Logistic Regressionobj2 <- nnetsauce::sklearn$linear\_model$LogisticRegressionCV()# there are 3 layers in the deep modelobj <- DeepClassifier(obj2, n\_layers = 3L)# adjust the modelres <- obj$fit(X\_train, y\_train)# accuracy, must be 1print(mean(obj$predict(X\_test) == y\_test)) It’s worth mentioning that the R version is a bit less stable than the Python version. Maybe because I’m not a reticulate superstar. I’m open to any suggestion/pull requests regarding this R port from the Python package.

To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A detailed introduction to Deep Quasi-Randomized ‘neural’ networks

View Details

[This article was first published on Albert Rapp, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. library(tidyverse)library(giscoR)library(ggiraph)germany\_districts <- gisco\_get\_nuts( year = "2021", nuts\_level = 3, epsg = 3035, country = 'Germany') |> as\_tibble() |> janitor::clean\_names()germany\_states <- gisco\_get\_nuts( year = "2021", nuts\_level = 1, epsg = 3035, country = 'Germany') |> as\_tibble() |> janitor::clean\_names()ggplt <- germany\_districts |> ggplot(aes(geometry = geometry)) + geom\_sf( data = germany\_states, aes(fill = nuts\_name), color = 'black', linewidth = 0.5 ) + geom\_sf\_interactive( fill = NA, aes( data\_id = nuts\_id, tooltip = glue::glue('{nuts\_name}') ), linewidth = 0.1 ) + theme\_void() + theme( legend.position = 'none' )girafe(ggobj = ggplt)

library(sf)state\_nmbrs <- map\_dbl( germany\_districts$geometry, \(x) { map\_lgl( germany\_states$geometry, \(y) st\_within(x, y) |> as.logical() ) |> which() }) germany\_districts\_w\_state <- germany\_districts |> mutate( state = germany\_states$nuts\_name[state\_nmbrs] )ggplt <- germany\_districts\_w\_state |> ggplot(aes(geometry = geometry)) + geom\_sf( data = germany\_states, aes(fill = nuts\_name), color = 'black', linewidth = 0.5 ) + geom\_sf\_interactive( fill = NA, aes( data\_id = nuts\_id, tooltip = glue::glue('{nuts\_name}<br>{state}') ), linewidth = 0.1 ) + theme\_void() + theme( legend.position = 'none' )girafe(ggobj = ggplt) Nicer tooltip and nicer colors

make\_nice\_label <- function(nuts\_name, state) { nuts\_name\_label <- htmltools::span( nuts\_name, style = htmltools::css( fontweight = 600, font\_family = 'Source Sans Pro', font\_size = '32px' ) ) state\_label <- htmltools::span( state, style = htmltools::css( font\_family = 'Source Sans Pro', font\_size = '20px' ) ) glue::glue('{nuts\_name\_label}<br>{state\_label}')}ggplt <- germany\_districts\_w\_state |> mutate( nice\_label = map2\_chr( nuts\_name, state, make\_nice\_label ) ) |> ggplot(aes(geometry = geometry)) + geom\_sf( data = germany\_states, aes(fill = nuts\_name), color = 'black', linewidth = 0.5 ) + geom\_sf\_interactive( fill = NA, aes( data\_id = nuts\_id, tooltip = nice\_label ), linewidth = 0.1 ) + theme\_void() + theme( legend.position = 'none' ) + scale\_fill\_manual( values = c("#A0CBE8FF", "#F28E2BFF", "#FFBE7DFF", "#59A14FFF", "#8CD17DFF", "#B6992DFF", "#F1CE63FF", "#499894FF", "#86BCB6FF", "#E15759FF", "#FF9D9AFF", "#79706EFF", "#BAB0ACFF", "#D37295FF", "#FABFD2FF", "#B07AA1FF", "#D4A6C8FF", "#9D7660FF", "#D7B5A6FF") )girafe(ggobj = ggplt) To leave a comment for the author, please follow the link and comment on their blog: Albert Rapp.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Three Ways to Include Images in Your ggplots

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Select variables of data frame in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Select variables of data frame in R, we will learn how to use the select and rename functions of the dplyr package to select and rename variables in R.

We will use the dplyr package to manipulate a data frame, which is a fundamental data structure in R.

The tutorial consists of two examples that demonstrate how to use the select and rename functions to select and rename variables.

Creation of Example Data

We will use the following data frame for the examples of this tutorial:

data <- data.frame(x1 = 1:3, x2 = LETTERS[1:3], x3 = 5)data# x1 x2 x3# 1 1 A 5# 2 2 B 5# 3 3 C 5 This data frame contains three rows and three columns. We will use this data frame to demonstrate how to use the select and rename functions.

Example 1: Extract Variables with select Function

The select function is used to extract specific variables from a data frame. We can use the select function to extract the variables x1 and x3 from our data frame:

How to copy files in R » Data Science Tutorials

select(data, c(x1, x3))# x1 x3# 1 1 5# 2 2 5# 3 3 5 This will return a subset of our original data frame containing only the two selected columns.

select(data, c(x1, -x3))

x1

1 1

2 2

3 3

Example 2: Change Variable Name with rename Function

The rename function is used to change the column names of specific variables. We can use the rename function to change the name of the first column of our data frame from x1 to x1_new:

Extract columns of data frame in R » Data Science Tutorials

dplyr::rename(data, x1\_new = x1)# x1\_new x2 x3# 1 1 A 5# 2 2 B 5# 3 3 C 5 This will return a new data frame with the modified column names.

Conclusion

In this tutorial, we have learned how to use the select and rename functions of the dplyr package to select and rename variables in R.

We have demonstrated how to use these functions to extract specific variables from a data frame and change the column names of specific variables. With these functions, you can easily manipulate and analyze your data in R.

How to Calculate Mean Absolute Percentage Error (MAPE) in R
Change Legend Position in R Plots with examples
R Programming For Data Science
Unequal Variance t-test in R:- Welch’s t-Test
What is neural network in machine learning?
Data Normalization in R
Application of Bayes Theorem in R

The post Select variables of data frame in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Select variables of data frame in R

View Details

[This article was first published on R – Statistical Odds & Ends, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I just came across a really interesting and simple algorithm for estimating the number of distinct elements in a stream of data. The paper (Chakraborty et al. 2023) is available on arXiv; see this Quanta article (Reference 2) for a layman’s explanation.

Problem statement

Let’s state the problem formally. Let’s say we are given a stream where each , and let denote the number of distinct elements of . For any input parameters , we want an -approximation of , i.e. a number such that

A naive solution would be to maintain a buffer which keeps track of all distinct elements seen so far. As we inspect each new item, we check if the item is in : if it’s there we discard the new item, and if it’s not we add it to . After going through the items, the size of is exactly the number of distinct items. However, it has bad worst case memory and checking whether an element is in can be costly if is large (e.g. think of counting the number of distinct members visiting google.com today).

Algorithm

There is a long line of work seeking to do better than the naive solution (see Bibliographic Remarks in Reference 1); Chakraborty et al. claims to be the first state-of-the-art algorithm whose proof requires just basic probability theory. The algorithm is easy enough to be reproduced here; see Reference 2 for an excellent walkthrough of an example run of the algorithm.

Algorithm 1 from Chakraborty et al. (2023), annotations are mine.At a high level, the algorithm maintains a buffer of seen items. The algorithm proceeds in rounds. In each round, we check whether each incoming item is in or not. If it is not in , add it with probability . If it is in , keep it in with probability . is made smaller and smaller as we go through more rounds. Whenever the buffer reaches its threshold number of items (threshold fixed throughout the algorithm), each element is removed from it with probability . When we run out of elements, our estimate of the number of distinct items is .

Here is the theorem for this algorithm:

Theorem. For any data stream and any , the algorithm above outputs an -approximation of . It uses space in the worst case.

The proof is not too long (about 3 pages) and while requiring some restatement of the problem via alternate algorithms, the most advanced tool it uses is Chernoff’s bound.

To obtain an -approximation for items, the algorithm requires a buffer of size . This grows logarithmically with . If we want our approximation to be twice as accurate, we need the buffer to be 4 times as large (). In practice the buffer can probably be much smaller: the authors are only interested in big-O type dependencies and so made the constants bigger wherever it made the proof simpler. (It’s worth noting that the algorithm gives the exact number of distinct items if the buffer happens to be larger than the true number of distinct items.)

Simulations

This algorithm is really easy to implement. Here is a quick implementation in R (definitely can be optimized further, just coded it up quickly for prototyping purposes):

```

threshold: number of elements buffer can hold# trajectory: if TRUE, return the estimate after every item in stream (default FALSE)estimate_distinct_elements <- function(data_stream, threshold, trajectory = FALSE) { p <- 1 m <- length(data_stream) buffer <- c() hit_fail_event <- FALSE estimates <- rep(NA, m) for (i in 1:m) { buffer <- setdiff(buffer, data_stream[i]) if (runif(1) < p) buffer <- c(buffer, data_stream[i]) if (length(buffer) == threshold) { buffer <- buffer[runif(threshold) < 0.5] p <- p / 2 if (length(buffer) == threshold) { # fail event hit_fail_event <- TRUE } } estimates[i] <- ifelse(hit_fail_event, -1, length(buffer) / p) } if (trajectory) { return(estimates) } else { return(estimates[m]) }}

``` Let’s take this for a run with some simulations. All simulation code can be found here. For a start, let’s generate a stream of length 100, where each item is an integer from 1 to 100 (inclusive). For the particular data stream we generated, there were 60 distinct elements. This means that for the naive solution, we would have buffer of length 60.

First, we run the algorithm 1000 times with buffer length 30, half of what the naive solution needed. The algorithm never hit the “fail” event, and here are some summary statistics of the distinct count estimate:

```

Min. 1st Qu. Median Mean 3rd Qu. Max. # 34.00 52.00 56.00 60.02 68.00 104.00

``` The mean and median are pretty close to the true value! Here is a histogram of the estimates:

It’s worth noting that while the estimate is generally pretty good (50% chance of being within 8 of the true number), it can be wildly off as well!

Let’s rerun the simulation for the same data stream but with buffer length 15 (a quarter of the true number of distinct items). Again the algorithm never hit the “fail” event. The summary statistics and histogram are below and as one might expect, while the estimates are still centered at the right place, there is more variance in the estimate. (The histograms might look similar due to different scales on the x-axis.)

```

Min. 1st Qu. Median Mean 3rd Qu. Max. # 8.00 48.00 56.00 59.81 72.00 144.00

``` Since this is a streaming algorithm, it provides an estimate of the number of distinct items after inspecting each item. Let’s look at how good the estimate is over time. Here are the plots for 10 estimate trajectories (true count in blue) for buffer length = 30 and buffer length = 15. Notice that the algorithm gives an exact estimate until the buffer is first filled. As expected, larger buffer length tends to give better estimates.

Let’s do one final simulation: Let’s look at a stream of length 10,000 with each element being a random integer from 1 to 4,000 (inclusive). For our data, the true number of distinct elements is 3,686.

Here is the plot of 10 trajectory estimates with buffer length 500 (about 14% of the true number of uniques) as well as the relative error of the estimate:

That’s not too bad! The estimates are pretty much within 10% of the true values. How about a much smaller buffer length of 50 (1.4% of the true number of uniques):

There’s much wider variation in the estimates now, probably too much to be useful for any real-world application.

References:

Chakraborty, S., et al. (2023). Distinct Elements in Streams: An Algorithm for the (Text) Book.
Nadis, S. (2024). Computer Scientists Invent an Efficient New Way to Count.

To leave a comment for the author, please follow the link and comment on their blog: R – Statistical Odds & Ends.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A simple probabilistic algorithm for estimating the number of distinct elements in a data stream

View Details

[This article was first published on RStudioDataLab, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Data analysis is all about turning raw data into actionable insights. I was working on a research project analyzing survey data from thousands of respondents. The clock was ticking, and I needed to summarize responses to hundreds of questions quickly. Manually counting each response would have taken days, if not weeks.

Then, I discovered the magic of the count function in R. In a matter of minutes, I transformed a messy dataset into a neatly summarized table, revealing patterns and trends that were previously hidden. That’s the power of the count function – it’s a game-changer for data analysts of all levels. You can read more about count function.

Table of ContentsKey points1. The count function in R’s dplyr package summarises the frequency of values within a dataset. Forget manual counting; count does the heavy lifting for you. 2. Count effortlessly adapts to your data’s structure when dealing with categorical factors like car models or numeric variables like horsepower. 3. Count seamlessly integrates with other dplyr functions, allowing you to filter, group, and transform your data fluently and intuitively. 4. Don’t let incorrect data types or missing values trip you up. With some know-how, you can easily troubleshoot common issues and ensure accurate results. 5. Beyond simple counting, the count function is your gateway to uncovering patterns, trends, and relationships hidden within your data.

What is the Count Function in R?The count function in R from the dplyr package empowers you to swiftly summarize and tabulate the frequency or number of values that occur within a dataset. It’s just like a magnifying glass that zooms in on the distribution of values, allowing you to answer critical questions effortlessly:

How many cars in the mtcars dataset have 4 cylinders?
What’s the distribution of transmission types (am vs. automatic)?
How many observations fall into each combination of cylinder count and transmission type?

This function operates like a frequency calculator, adeptly identifying and quantifying unique values within your data. It generates a new data frame that lists each distinct value and its corresponding count, offering a clear and concise distribution summary.

The dplyr package also provides a family of related functions that complement count:

Required Libraries for count function in RIn this tutorial, we will use the following libraries and data set.

library(dplyr)data(mtcars) Read more how to install libraries in R

tally() in RA function that directly prints the count results to your console, perfect for quick checks.

mtcars %>% count(cyl) %>% tally() In the R code above, we use tally() in conjunction with count() to count the number of cars based on the number of cylinders and print the result directly to the console.

add_count() in RInstead of creating a separate data frame, it seamlessly adds a new column to your existing dataset, recording the count for each group or value.

mtcars\_with\_counts <- mtcars %>% add\_count(cyl)head(mtcars\_with\_counts) Here, add_count() adds a new column named “n” to the mtcars dataset, showing the number of cars with a given number of cylinders.

add_tally() in RSpecifically designed for grouped data, this function works in harmony with group_by(), enabling you to add counts within pre-defined groups effortlessly.

mtcars %>% group\_by(vs) %>% add\_tally() We first use the group_by() function in the R code to group the data set by the number of cylinders, and then we count the number of cars in each group using add_tally().

Whether you’re exploring categorical variables, analyzing grouped data, or delving into weighted counts, the count function and its counterparts offer a flexible and efficient toolkit for uncovering the hidden patterns within your data.

Related Posts Why Choose the Count Function? 1. Efficiency and Speed: Compared to manual counting or writing custom loops, counting significantly streamlines the process of summarizing frequencies. It’s designed to operate efficiently on large datasets, saving you valuable time and effort. 2. Simplicity and Readability: The syntax of count is concise and intuitive, making your code easier to read and understand. This clarity is especially beneficial when collaborating with others or revisiting your analysis later. 3. Versatility Across Data Types: Whether your data includes categorical factors (e.g., car models, survey responses), numeric variables (e.g., ages, sales figures), or a combination of both, count handles it all with ease. This adaptability makes it a versatile tool for various data analysis tasks. 4. Seamless Integration with Tidyverse: If you’re already familiar with the dplyr package and the tidyverse philosophy, count fits right into your existing workflow. You can seamlessly combine it with other dplyr functions like filter, group_by, and mutate to create powerful data manipulation pipelines. 5. Clear and Informative Output: The count function generates a tidy data frame as its output, making it easy to visualize, interpret, and further analyze the summarized results. You can readily create bar charts, tables, or other visualizations to communicate your findings. 6. Handling Missing Values: By default, count excludes missing values (NA) from its calculations, ensuring that your summaries are accurate and relevant. However, you can also include missing values if they are meaningful in your analysis.

How Does the Count Function Work?Think of the count function as a helpful tally counter. It looks at your data and counts how often each unique item appears.

Step 1: Load the dplyr PackageBefore using count, you must have the dplyr package loaded into R. It gives you access to a whole set of tools for working with data.

```

install.packages("dplyr")library(dplyr)data(mtcars)

```

Before We start, Make sure you Have the following: Comprehensive Guide: How to install RStudio * How to Import and Install Packages in R: A Comprehensive Guide * No interest in Coding? Try our Shiny Apps Step 2: Use count function to Count Unique Values*Let’s say you want to know how many cars in the mtcars dataset have different numbers of cylinders. Here’s the code:

mtcars %>% count(cyl) This code does three things:1. It takes your mtcars dataset. 2. The %>% symbol (called a “pipe”) sends the data into the count function. 3. The count(cyl) part tells the count to look at the cyl column (number of cylinders) and count how many times each unique value appears.

This means there are 11 cars with four cylinders, seven with six cylinders, and 14 with eight cylinders.

Step 3: Counting with GroupsWant to get learn some advance?

You can count within groups. For example, how many cars have automatic or manual transmissions (am) for each number of cylinders (cyl).

mtcars %>% count(cyl, am) Now you get a table that shows the count for each combination of cylinders and transmission type:Step 4: Handling Missing Values (NA)By default, the count doesn’t include any rows where the value you’re counting is missing (NA). If you want to include those missing values in your count, add na.rm = FALSE to the count function.

```

This will not include NAsmtcars %>% count(cyl)# This will include NAsmtcars %>% count(cyl, na.rm = FALSE)

``` Key Points to Remember* The count function makes counting things in your data super easy. * You can count how often different values appear in one column or across multiple columns. * By default, count ignores missing values, but you can change that if necessary.

Overview of Count Functions in R

| Function | Library | Pros | Cons | | --- | --- | --- | --- | | table() | Base R | – Simple and built-in function. | – Limited to categorical data. | | – No need for additional libraries. | – Not as flexible for complex counting operations. | | count() | dplyr | – Flexible and intuitive syntax. | – Requires installation of the dplyr package. | | – Works well with the tidyverse ecosystem. | – May be slower for very large datasets. | | summarise(n = n()) | dplyr | – Can be combined with other dplyr functions for complex summaries. | – Slightly more complex syntax. | | – Efficient for grouped counting. | – Requires understanding of dplyr syntax. | | tally() | dplyr | – Simple for quick counts. | – Less flexible than count(). | | – Integrates with dplyr pipelines. | – Not as widely used or known. | | table1::tbl() | table1 | – Designed for descriptive statistics and counts. | – Requires installation of the table1 package. | | – Useful for creating summary tables in reports. | – May be overkill for simple counts. | | aggregate() | Base R | – Powerful for grouped counts and aggregations. | – More complex and less intuitive syntax. | | – No additional libraries needed. | – Can be less efficient than dplyr for very large datasets. | | tableone::CreateTableOne() | tableone | – Excellent for medical and clinical data summaries. | – Requires installation of the tableone package. | | – Provides comprehensive tables for research. | – May be complex for simple counting needs. | | data.table::dcast() | data.table | – Highly efficient and fast for large datasets. | – Requires installation of the data.table package. | | – Flexible for various counting and aggregation tasks. | – Slightly steeper learning curve for syntax. | | janitor::tabyl() | janitor | – Easy to use for frequency tables and proportions. | – Requires installation of the janitor package. | | – Integrates well with tidyverse. | – Limited to categorical data. | | plyr::count() | plyr | – Simple and intuitive for counting. | – The plyr package is older and less efficient than dplyr. | | – Good for basic counting tasks. | – plyr is being phased out in favor of dplyr. |

Common Errors and Solutions with the Count Function in REven with its user-friendly design, the count function can sometimes throw a curveball. Let’s tackle some common hiccups you might encounter and provide solutions to get you back on track:

Incorrect Data TypesImagine counting unique values in a column that’s not a factor or character variable. You might get an error or unexpected results.

Always double-check your data types. Use functions like class() or str() to verify that the column you’re working with is suitable for counting unique values. If needed, convert the column to a factor using as.factor().

Example:Let’s say we want to count the unique values in the hp (horsepower) column of the mtcars dataset. Before proceeding, we check the data type:

class(mtcars$hp)#If it's not a factor, we can convert it:mtcars$hp <- as.factor(mtcars$hp)mtcars %>% count(hp)

Missing Values (NA):By default, count excludes rows with missing (NA) values in the column you’re counting. This can lead to undercounting if those missing values are meaningful.

To include missing values in your count, add the argument na.rm = FALSE to the count function.

```

Count unique values in the `cyl` column, including NAsmtcars %>% count(cyl, na.rm = FALSE)

``` Unexpected Results (Grouping Gone Wrong)Sometimes, you might get results that don’t match your expectations, especially when working with grouped data.

Carefully review your group_by() statement. Ensure you’re grouping by the correct variables and in the desired order. Double-check for typos or incorrect variable names.

```

Correct groupingmtcars %>% group_by(cyl, am) %>% count()

``` General Troubleshooting Tips Read the Error Messages: Error messages are your friends! They often provide valuable clues about what went wrong. * Consult the Documentation: The official dplyr documentation is a treasure trove of information. Look up the count function to clarify its usage and arguments. * Seek Help Online:* If you’re still stuck, don’t hesitate to ask for help on online forums like Stack Overflow. The R community is known for its helpfulness and expertise.

By being mindful of these common errors and following the suggested solutions, you’ll be well on your way to mastering the count function and confidently summarizing your data in R.

ConclusionThe count function in R is an adaptable and necessary tool for any data analyst. Its ability to quickly and efficiently summarize the frequency of values within datasets makes it a true workhorse in data wrangling. The count function seamlessly adapts to various data types and scenarios, from counting unique values to analysing grouped data.

We’ve delved into the inner workings of the count, highlighting its simplicity and integration with the powerful dplyr package. By understanding its core purpose, you’re equipped to easily tackle a wide range of data summarization tasks. We’ve also explored common pitfalls and provided practical solutions, ensuring you can navigate potential challenges confidently.

Remember, the count function isn’t just about numbers; it’s about extracting meaning and insights from your data. Whether exploring the characteristics of cars in the mtcars dataset or analyzing complex survey responses, count enables you to uncover patterns, trends, and relationships that might otherwise remain hidden.

So, the next time you’re faced with a dataset waiting to be deciphered, don’t hesitate to reach for the count function. Its efficiency, versatility, and intuitive syntax make it your trusted ally in the quest for data-driven discoveries.

Frequently Asked Questions (FAQs)Is there a counting function in R?Yes, R offers several counting functions. The most versatile and commonly used is the count function, which is part of the dplyr package. It efficiently summarizes the frequency of values within a dataset.

What is count() used for?The count() function is used to tally the occurrences of unique values within a variable or combination of variables. It’s your go-to tool for quickly understanding the distribution of data.

What package is count in R?The count function is in the dplyr package, a core component of the tidyverse, a collection of R packages designed for data science.

How to count rows in R?To count the total number of rows in a data frame (like the mtcars dataset), you can use the nrow() function:

nrow(mtcars) # This will show us there are 32 rows in mtcars dataset

How do you count characters in R?The nchar() function counts the number of characters in a string:

nchar(“Hello, R!”) # Returns 8

How to use count if?The count function doesn’t have an “if” condition built in. However, you can combine it with a filter from dplyr to achieve conditional counting:

mtcars %>% filter(cyl == 4) %>% count()

This will count only the rows where there are four cylinders (cyl = 4)

What’s an n()?Within dplyr, n() is a special function used to count the number of observations (rows) in a group or the entire dataset when used with summarize. It’s often paired with group_by to count observations per group.

Example Counting with Grouped Data

mtcars %>%

group_by(cyl) %>%

summarize(Count = n())

What is the use of count() and count_? count(): As discussed earlier, count() creates a new data frame with the unique values and corresponding counts. * The count_* family (add_count, add_tally) adds a new column to your existing data frame, showing the count for each group or value. This is useful to keep the original data structure while adding count information.

What is the count method?In R, “count” typically refers to functions like count(), table(), or length() rather than a specific “method.” These functions provide different ways to count elements within data structures.

What is %>% in R?The %>% symbol, called the “pipe” operator, is a handy tool from the magrittr package (included in the tidyverse). It allows you to chain functions together, passing the output of one function as the input to the next. This makes your code more readable and easier to follow.

Which function in RStudio?RStudio is an integrated development environment (IDE) for R, not a function itself. The functions we’ve discussed (like count, n(), nrow(), nchar()) are all part of R and can be used within RStudio.

What is the sum() function in R?The sum() function adds up numeric values. You can use it to calculate the total of a column in a data frame:

sum(mtcars$mpg) # Calculates the total miles per gallon across all cars

How to count observations in R?* For the total number of observations (rows) in a data frame, use nrow(). * To count observations within groups, use group_by() followed by summarize(n = n()) (or tally()).

Which function gives the count of levels in R?The nlevels() function tells you how many unique levels (categories) a factor variable has:

nlevels(mtcars$cyl)

This will return the number of unique levels in the cyl variable.

How do I count the number of values in a list in R?Use the length() function to find the number of elements in a list.

What is the difference between N and count in R?* N: Within dplyr, N is a special symbol representing the total number of rows in a data frame. It’s often used within summarize() for calculations based on the total count. * count: The count function is a specialized tool from dplyr designed to efficiently count unique values and create summary tables.

Transform your raw data into actionable insights. Let my expertise in R and advanced data analysis techniques unlock the power of your information. Get a personalized consultation and see how I can streamline your projects, saving you time and driving better decision-making. Contact me today at info@rstudiodatalab.com or visit to schedule your discovery call.

Join Our Community Book a free call. To leave a comment for the author, please follow the link and comment on their blog: RStudioDataLab.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Count Function in R I dplyr::count()

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Export output as text in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Export output as text in R, The sink function in R is used to redirect the output of the console to a file or a connection.

This allows you to capture the output of your R code and save it to a file or send it to a network connection.

In this article, we will show you how to use the sink function to export text to a txt file, export data as a txt file, and export data as a csv file.

Example 1: Export Character String as txt File

We can use the sink function to export a character string to a txt file. Here is an example:

sink("example\_1.txt") # Create empty txt file"some output" # Write text to filesink() # Close connection to file This will create a new txt file called “example_1.txt” in the current working directory and write the character string “some output” to it.

How to Display Percentages on Histogram in R » Data Science Tutorials

Example 2: Export Data Frame as txt File

We can also use the sink function to export an entire data frame to a txt file. Here is an example:

sink("example\_2.txt") # Create empty txt fileChickWeight # Print ChickWeight datasink() # Close connection to file This will create a new txt file called “example_2.txt” in the current working directory and write the entire ChickWeight data frame to it.

Example 3: Export Data Frame as csv File

We can also use the sink function to export data in other formats than txt. For example, we can export data as a csv file. Here is an example:

sink("example\_3.csv") # Create empty csv fileChickWeight # Print ChickWeight datasink() # Close connection to file This will create a new csv file called “example_3.csv” in the current working directory and write the ChickWeight data frame to it.

How to perform One-Sample Wilcoxon Signed Rank Test in R? » Data Science Tutorials

Conclusion

In this article, we have shown you how to use the sink function in R to export text to a txt file, export data as a txt file, and export data as a csv file.

We hope that this tutorial has been helpful in showing you how you can use the sink function in your own R code.

How to add calculated variables in R
How to calculate proportion in a table
Simple Linear Regression in r
Projects for Data Science Beginners
Best Books For Deep Learning
How to Remove Outliers in R

The post Export output as text in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Export output as text in R

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Positive or Negative in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Positive or Negative in R, we will learn how to use the sign function in R to determine whether a numeric value is positive or negative.

The sign function returns the signs of numeric elements, with a value of 1 for positive numbers, 0 for zero, and -1 for negative numbers.

Basic R Syntax

sign(5) # Basic R syntax of sign function Example: Return whether Number is Positive or Negative

We can use the sign function to check whether a number is positive or negative.

For example, let’s create a vector of numbers from -2 to 5 and apply the sign function:

sign(-2:5) # Apply sign function# -1 -1 0 1 1 1 1 1 This returns the signs of the elements in the vector.

How to Display Percentages on Histogram in R » Data Science Tutorials

The first two elements are negative (indicated by -1), the third element is zero (indicated by 0), and the remaining elements are positive (indicated by +1).

Example: Checking Multiple Numbers

We can also use the sign function to check multiple numbers at once.

For example, let’s create a vector of numbers and apply the sign function:

numbers <- c(-3, 0, 2, -5, 7)sign(numbers) # Apply sign function to multiple numbers# -1 0 1 -1 1 This returns the signs of the elements in the vector.

The first element is negative (indicated by -1), the second element is zero (indicated by 0), the third element is positive (indicated by +1), and the fourth element is negative (indicated by -1), and the fifth element is positive (indicated by +1).

Online Course R Statistics: Statistics with R
How to Arrange the Bars in ggplot2
What is the best way to filter by row number in R? » Data Science Tutorials
Chi Square for Independence-Mantel–Haenszel test in R
Self Organizing Maps in R- Supervised Vs Unsupervised
How to find the Mean Deviation? MD Vs MAD-Quick Guide

The post Positive or Negative in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Positive or Negative in R

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.ChatGPT was only the beginning. Generative AI is now revolutionizing every industry.

As the race to adopting AI tools intensifies, the decisions we make today about AI will likely shape our success for decades to come.

To learn how the world’s data and AI leaders are adapting to this new era, join DataCamp on June 26–27 for RADAR: AI—a two-day digital event exploring how businesses and individuals can unlock their full potential with AI.

From scaling AI within your organization and driving innovation to navigating the evolving job market, uncover practical steps to implement in your career or organization to maximize the power of generative AI.

A star-studded agenda:
Including voices from GitHub, Dropbox, PwC, Hugging Face, Accenture, and many more, unpack how AI is evolving and how you can take advantage.

A peak at what to you can expect:

June 27, 2024 (all times EDT):
9:00–9:45
Welcome to RADAR!Jonathan Cornelissen, Co-Founder & CEO, DataCamp

9:50–10:35

Generative AI in Practice: 100+ Amazing Ways Generative AI is Changing Business and Society

Bernard Marr, World-renowned Futurist, AI Advisor, and Author of “Generative AI in Practice”, Bernard Marr & Co

10:50–11:35

From Learning to Earning: Navigating the AI Job Landscape

Sadie St Lawrence, Founder & CEO at Women in Data, and Megan Finck,

Global Head of Talent Acquisition—Engineering, IT and Data Analytics, Boeing

11:40–12:25

Generative AI: Trends, Impact, and Practical Applications for 2024

Principal, Insight Partners, and Sandhya Venkatachalam, Partner, Axiom Partners

12:40–1:25

The High Cost of AI Hype

Eric Siegel, Founder of Machine Learning Week, Prediction Impact

1:30–2:15
Building Tomorrow’s Workforce, Today: Scaling Internal AI AcademiesMike Baylor, CDAO, Lockheed Martin, Carolann Diskin, Senior Technical Program Manager, Dropbox, Matthew Graviss, and CDAIO, U.S. Department of State

2:20–3:05

Scaling Data Quality in the Age of Generative AI

Barr Moses, CEO, Monte Carlo Data, Prukalpa Sankar, Co-founder, Atlan, and George Fraser CEO, Fivetran

June 27, 2024
9:00–9:45
Building Trust in AI: Scaling Responsible AI Within Your Organization

Haniyeh Mahmoudian, Chief AI Ethicist at Datarobot, Eske Montoya Martinez van Egerschot, Chief AI Governance and Ethics at DigiDiplomacy, Associate Partner at Meines Holla & Partners, and Alexandra Ebert, Chief Trust Officer at MOSTLY AI

9:50–10:35

Leading with AI: Leadership Insights on Driving Successful AI Transformation

Chandra Donelson, Chief Data and Analytics Officer at United States Air Force, Semih Kumluk, Head of AI and Digital at PwC, and Giorleny Altamirano Rayo

10:50–11:35

The Future of Programming: Accelerating Coding Workflows with LLMs

Michele Catasta, VP of AI at Replit, Jordan Tigani, CEO at Mother Duck, and Ryan J. Salva, VP of Product at GitHub

11:40–12:25

Hugging Face and the Future of the Open Source AI Ecosystem

Julien Simon, Chief Evangelist at Hugging Face

12:40–13:25

Building an AI Strategy: Key Steps for Aligning AI with Business Goals

Vin Vashishta, AI Advisor and Founder of V Squared, Cindi Howson, Chief Data Strategy Officer at ThoughtSpot, and Sonali Bhavsar, Managing Director at Accenture

13:30–14:15

Charting the Path: What the Future Holds for Generative AIEdo Liberty, CEO at Pinecone, Tomasz Tunguz, General Partner at Theory Ventures, and Nick Elprin, CEO at Domino Data Lab

14:20–15:05

Closing Session & AMA

DataCamp’s Co-founders, Jonathan Cornelissen, CEO, and, Martijn Theuwissen, COO

Once again, RADAR AI is free for everyoneRadar: AI is more than just another AI conference, but a place to connect with like-minded individuals and organizations. Whether you’re a data scientist, business leader, or just AI-curious, Radar: AI Edition offers something for everyone.

Don’t let the data and AI revolution pass you by.

Free DataCamp Online Conference: Radar: AI Edition, June 26–27, 2024 was first posted on May 17, 2024 at 1:29 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Free DataCamp Online Conference: Radar: AI Edition, June 26–27, 2024

View Details

[This article was first published on Rtask, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You can read the original post in its original format on Rtask website by ThinkR here: gitlabr V2.1.0 – gitlab-ci with ‘pak’ and functions to deal with groups on GitLab

A lot of cleanups, new features, and bug fixes for this new version of ‘gitlabr’. You will find brand-new CI/CD templates and functions to manage GitLab groups from R.

Brand new gitlab-ci with ‘pak’, ready to use* The template for checking R packages uses {pak} for dependency management. You will no longer have issues with missing system dependencies. * You should see better use of caching, especially cache sharing between jobs. You should spend less time waiting for things to process… * The templates for bookdown have also been revised. * You can choose whether to update R dependencies by installing them or not with upgrade = TRUE. * And a brand-new vignette to introduce how to use gitlabr::use_gitlab_ci()

Managing GitLab groups* You can now create, edit, and delete groups from R on your GitLab. * You can also manage subgroups. * You can retrieve lists of project and group members.

gitlabr v2.1.0 – Complete NEWSBreaking changes* use_gitlab_ci() does not use repo_name anymore as “rocker” images fix CRAN to a specific date. * Functions deprecated since version 0.7 are removed * Transfer ownership of the project to ThinkR-open

New features* use_gitlab_ci() allows to decide whether to update R packages during the CI pipeline * Add vignette to explain how to use ‘gitlabr’ with GitLab CI * gl_new_group(), gl_edit_group(), gl_delete_group(), gl_list_groups(), gl_list_sub_groups() to deal with groups on a GitLab instance (@mpolano) * gl_new_subgroup() to create a subgroup in a group (@margotbrd) * gl_delete_file() to delete a file in a repository * gl_list_project_members() and gl_list_group_members to retrieve members of a project or a group (#61, @datawookie) * gitlab() queries allow for a vector of parameters to be passed to the API. This is needed when the API asks for an array (@klmr)

Minor changes* multilist_to_tibble() transforms a deep structured nested list from an API into a tibble (#86, @statnmap, @ymansiaux)

Bug fixes* gl_file_exists(), gl_list_files(), gl_push_file(), gl_delete_file() now work with files in subdirectories * Fix handling file path in sub-directories in gl_push_file() (#73, #111) * Fix combination of page and max_page for infinite pagination (#90) * Fix auto_format=FALSE (#82)

This post is better presented on its original ThinkR website here: gitlabr V2.1.0 – gitlab-ci with ‘pak’ and functions to deal with groups on GitLab

To leave a comment for the author, please follow the link and comment on their blog: Rtask.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: gitlabr V2.1.0 – gitlab-ci with ‘pak’ and functions to deal with groups on GitLab

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIf you’ve ever worked with text data in R, you know how important it is to have powerful tools for pattern matching. One such tool is the gregexpr() function. This function is incredibly useful when you need to find all occurrences of a pattern within a string. Today, we’ll go into how gregexpr() works, explore its syntax, and go through several examples to make things clear.

Understanding gregexpr() SyntaxThe gregexpr() function stands for “global regular expression,” and it’s designed to locate all matches of a pattern within a text string. Here’s the basic syntax:

gregexpr( pattern, text, ignore.case = FALSE, perl = FALSE, fixed = FALSE, useBytes = FALSE ) * pattern: The regular expression pattern you want to search for. * text: The text string or vector of text strings to be searched. * ignore.case: A logical value indicating whether to ignore case. Default is FALSE. * perl: A logical value indicating whether to use Perl-compatible regex. Default is FALSE. * fixed: A logical value indicating whether the pattern is a fixed string. Default is FALSE. * useBytes: A logical value indicating whether to perform byte-by-byte matching. Default is FALSE.

ExamplesExample 1: Basic UsageLet’s start with a simple example. Suppose we want to find all occurrences of the letter “a” in the string “banana”.

text <- "banana"pattern <- "a"matches <- gregexpr(pattern, text)print(matches)

[[1]][1] 2 4 6attr(,"match.length")[1] 1 1 1attr(,"index.type")[1] "chars"attr(,"useBytes")[1] TRUE This will return a list with the starting positions of each match. Here, the numbers 2, 4, and 6 indicate the positions of “a” in the string “banana”.

Example 2: Ignoring CaseWhat if we want to search for the pattern without considering case? We can set ignore.case = TRUE.

text <- "BaNaNa"pattern <- "a"matches <- gregexpr(pattern, text, ignore.case = TRUE)print(matches)

[[1]][1] 2 4 6attr(,"match.length")[1] 1 1 1attr(,"index.type")[1] "chars"attr(,"useBytes")[1] TRUE Even though our string has uppercase “A” and lowercase “a”, the function treats them the same because we set ignore.case = TRUE.

Example 3: Using Perl-Compatible RegexSometimes, we need more advanced pattern matching. By setting perl = TRUE, we can use Perl-compatible regular expressions.

text <- "cat, bat, rat"pattern <- "[bcr]at"matches <- gregexpr(pattern, text, perl = TRUE)print(matches)

[[1]][1] 1 6 11attr(,"match.length")[1] 3 3 3attr(,"index.type")[1] "chars"attr(,"useBytes")[1] TRUE This will find all occurrences of “bat”, “cat”, and “rat”. The positions 1, 6, and 11 correspond to the starting positions of “cat”, “bat”, and “rat” respectively.

Example 4: Fixed String MatchingIf you want to search for a fixed substring rather than a regex pattern, set fixed = TRUE.

text <- "batman and catwoman"pattern <- "man"matches <- gregexpr(pattern, text, fixed = TRUE)print(matches)

[[1]][1] 4 17attr(,"match.length")[1] 3 3attr(,"index.type")[1] "chars"attr(,"useBytes")[1] TRUE This will match the substring “man” exactly. The output will show the starting positions of each match along with the length of the match.

Example 5: Extracting MatchesYou can extract the matched substrings using the regmatches() function.

text <- "apple, banana, cherry"pattern <- "[a-z]{5}"matches <- gregexpr(pattern, text)extracted <- regmatches(text, matches)print(extracted)

[[1]][1] "apple" "banan" "cherr" This will extract all substrings of length 5 from the text. The output will be a list of the matched substrings.

Wrapping UpThe gregexpr() function is a powerful tool for pattern matching in R. With its flexibility and various options, you can tailor it to fit your needs perfectly. Try using it in your own projects and see how it can simplify your text processing tasks.

Feel free to experiment with different patterns and options. The best way to get comfortable with gregexpr() is by practicing.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering gregexpr() in R: A Comprehensive Guide

View Details

[This article was first published on R Stories, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Why avoid real names?Using real names (or other personal identities) in research or related activities such as publishing on the web or in journals, presentations, and/or demonstrations can be problematic due to privacy concerns, existing data protection laws and regulations, and ethical obligations. Examples of highly sensitive data include student academic records, health records, and financial records.

To ensure the safety and privacy of all human subjects, many universities and other institutions establish oversight committees known as Institutional Review Boards (IRBs). The board is tasked with minimizing risk to participants. When using data collected from institutional participants, users are required to sign their proposals on the usage of the data, which, among other things, is required to include a declaration to use anonymous or pseudonymous names to avoid the use of real names.

The ‘randomNames’ R-PackageOne of my favorite R Packages is the ‘randomNames’ package which is actually simple and easy to use. It has a single function allowing users to generate random first and last names.

Here are a few simple usage examples.

Generate 6 random names as “Last, First”

library(randomNames)randomNames(6) ## [1] "Montoya, Jesus" "Gonzales, Seidy" "Galvan, Clint" "el-Baig, Misbaah"## [5] "Juniel, Jasmine" "Martel, Katelyn" Generate 6 random female names as “Last, First”

randomNames(6, gender = 1) ## [1] "el-Mohammed, Ruqayya" "Fischer, Taylor" "al-Shariff, Afeefa"## [4] "Wall, Patricia" "Guha, Judy" "al-Murad, Ruqayya" Generate 6 random names with half of them male names as “Last, First”

randomNames(6, gender = c(0,0,0,1,1,1)) ## [1] "Cooper, William" "Leewaye, Dustin" "Miller, Nathan" "Lee, Janna"## [5] "Hawley, Aijah" "Duncan, Alyssa" Generate 6 random names with three of them African Americans and the other three Whites (not Hispanic) as “Last, First”

randomNames(6, ethnicity = c(3,3,3,5,5,5)) ## [1] "Kirkman, Damien" "Salas, Sierra" "Lawson, Jasmine" "Harris, Joshua"## [5] "Hoffman, Logan" "Potter, Tyler" More exampleBelow is an example of a file which is publicly available by X-University. It is a schedule of classes at a fall semester generated by a software by a company called Ellucian. The data has variable names such as enrollment number (ENRLD), instructor name (INSTRUCTOR), credit hours (HRS), etc, for schedule of classes. Many higher education institutions use the same software to generate class schedules hence have similar outputs.Here you can find some examples of similar outputs: Savannah State University Spring 2024 Class Schedule, Fort Valley State University Class Schedule, Benedict College Class Schedule Fall 2022.Some of them allow users to download the schedules as an excel or csv file. One can also use cut/paste to use the data.

For the purpose of this demonstration, below is a schedule of spring 2024 classes by X-University; and we want to change instructor names in the schedule by random names.

library(dplyr)schedule<-read.csv("X-College\_Spring\_2024\_Classes\_Schedule.csv")schedule<-subset(schedule,INSTRUCTOR !="") #skip classes with no instructorschedule<-filter(schedule, P.of.T==1) #Session 1 Classes onlyhead(schedule)## P.of.T SUBJ NUMB TITLE HRS ENRLD MAXENRL TIMES## 1 1 ACCT 2101 PRINCIPLES OF ACCOUNTING I 3 28 30## 2 1 ACCT 2101 PRINCIPLES OF ACCOUNTING I 3 30 30 11:00-11:50## 3 1 ACCT 2102 PRINCIPLES OF ACCOUNTING II 3 16 30## 4 1 ACCT 2102 PRINCIPLES OF ACCOUNTING II 3 15 30 12:30-01:45## 5 1 ACCT 3103 INTERM ACCOUNTING I 3 7 30 03:30-04:45## 6 1 ACCT 4123 COST ACCOUNTING 3 9 30 05:15-06:30## DAYS INSTRUCTOR## 1 Haile, Brandie## 2 M W F Lopez, Elizabeth## 3 Haile, Brandie## 4 T R Lopez, Elizabeth## 5 T R Haile, Brandie## 6 M W el-Hassan, Nawaar Instructor names show under the variable ‘INSTRUCTOR’ and we want to change them by random names while keeping everything else in the data unchanged.

original\_names<-unique(schedule$INSTRUCTOR)random\_names<-randomNames(length(original\_names)) #Random names to replace original namesfor(i in 1:length(original\_names)){schedule[schedule == schedule$INSTRUCTOR[i]]<-random\_names[i]}head(schedule)## P.of.T SUBJ NUMB TITLE HRS ENRLD MAXENRL TIMES## 1 1 ACCT 2101 PRINCIPLES OF ACCOUNTING I 3 28 30## 2 1 ACCT 2101 PRINCIPLES OF ACCOUNTING I 3 30 30 11:00-11:50## 3 1 ACCT 2102 PRINCIPLES OF ACCOUNTING II 3 16 30## 4 1 ACCT 2102 PRINCIPLES OF ACCOUNTING II 3 15 30 12:30-01:45## 5 1 ACCT 3103 INTERM ACCOUNTING I 3 7 30 03:30-04:45## 6 1 ACCT 4123 COST ACCOUNTING 3 9 30 05:15-06:30## DAYS INSTRUCTOR## 1 Morgan, Hershel## 2 M W F Perkins, Kara## 3 Morgan, Hershel## 4 T R Perkins, Kara## 5 T R Morgan, Hershel## 6 M W Hong, Prachi There we have it, all instructor names under “INSTRUCTOR” in the data are replaced by random names!

References[1] randomNames package was created by Damian Betebenner and the repository is available in GitHub at randomNames (version 1.6-0.0).

To leave a comment for the author, please follow the link and comment on their blog: R Stories.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Applications of randomNames (an R-Package): avoid real names!

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Do you maintain an open-source project like an R package or a collection thereof, and wonder how to best use various communication channels to inform and engage with your community of users?We’ve consolidated this list of tips.Some of them are required in our opinion, others are simply nice to have.

Required: Having good release notesSince you’re developing a product, the first act of communication is to write informative release notes.Release notes generally describe updates and changes, commonly in a file called NEWS.md. These files generally have one header per release, with sub-headers used to group changes in meaningful categories.

Resources to get started with release notes include:

usethis::use_news_md() to create the NEWS.md file.
the chapter about NEWS files in the tidyverse style guide

You can partially automate release notes from commit messages using for instance the fledge package (quite powerful if paired with the Conventional commits convention).

Release notes can directly inform users, who can read them

from GitHub through watching release events for the repository;
from the documentation website, on which pkgdown transforms NEWS.md files into a page called changelog.

They will also be useful as raw material for further communication efforts like blog posts about releases.

Required: Issue trackersNot only should your repositories have issue/ticket trackers but you should also make sure enough team members watch them or will go through new tickets or comments at least once in a while.Maintaining and responding to issues is an important part of maintaining a community of users.

Your project’s use of issues might be advertised through a pinned issue; you could even temporarily limit interactions.These terms and links are for GitHub, but similar ideas and features exist for other code hosting platforms.

Required: Polish project profilesAll open-source software has a profile, potentially spread across many places such as GitHub organizations or Mastodon accounts. A logo can be a key identifier of your profile, and should appear consistently throughout your profile(s). It is also important to include informative descriptions, and to verify all URLs (docs for GitHub, docs for Mastodon).

For a GitHub organization you can ask your members whether they’d like to make their GitHub organization membership public, which might give a more collaborative image even before anyone dives into commit activity.

Profile polishing need not take a lot of time, and can only improve your project’s image.

Required: Make sure you handle access smartlyFor any platform requiring log-ins or some sort of access rights, make sure everyone who needs access has it, and that access is removed from anybody who no longer needs it.

You might want to look into password management systems such as 1Password.

As the makeup of a development team changes over time, it might be wise to review access regularly, and to make that review part of some sort of onboarding/offboarding checklist.

Required: have a space for private discussionsAlthough open-source development demands that a lot happens in the open, it is also important to cultivate a safe space where team members can let off steam, discuss sensitive matters, or share pets’ pictures.This could take such forms as a Slack workspace, Discord, Matrix, or Element server, or cutting edge options like flat or CQ2.

Ideally, you should own the space, unless you can rely on an external partner (a funder? a larger cohalition of projects?) to keep providing it to you.

Having a forumFor a small project, issue trackers might be all you need to handle bug reports, feature requests, and general Q&A.Bigger projects might nevertheless benefit from the creation and curation of a dedicated discussion forum.

You could use Discourse, or GitHub Discussions.

Having a blog with an RSS feedCompared to release notes, blog posts about new versions provide more of a narrative so can be easier to read.They can still point users to the release notes for more information.

The blog of an open-source project could also contain other types of posts such as a deep-dive on a feature, announcement of funding or call for contributions, or monetary support.

When choosing a website builder, try to choose one that is free and either familiar for your project’s team or easy enough to get acquainted with.Blog posts that are Markdown based are easier to write from release notes.Also make sure publishing a new blog post is not a complicated 100-step process or no one will want to write one.You could choose to use GitHub for a review and preview process of blog posts.

If you do create a blog, make sure to also create an RSS feed for it.With most static website generators this is either the default or available by turning on an option (docs for Quarto).

Once your blog has an RSS feed, register it in relevant aggregators such as R Weekly in the R world.

Having comments on blog postsIf you choose to open comments on your blog posts, make sure to integrate comments within your project’s forum.

This is very easy with Discourse (which we use on this very blog), and GitHub Discussions via Giscus (which are also easy to integrate with Quarto among others).

Integrating comments with your forum means you only need to watch one space, and also help connect readers of your blog posts to the forum.

Having social media profilesSocial media can be useful to spread the word about your project and its updates, and to engage with users.You might choose to make your social media “read-only” by stating clearly you do not have the resources to answer questions there.

Ideally, concentrate your social media usage on nice platforms, and platforms where your projects’ users and community are likely to congregate.

ConclusionIn this post, we shared some tips for communication for your open-source project.Use communication channels in accordance with your project’s goals and resources.You might also be interested in our past community call Set Up Your Package to Foster a Community.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Communication Tips for your Open-Source Project

View Details

[This article was first published on Blog on Credibly Curious, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.My venetian blinds, in black and white

I recently watched Josiah Parry’s wonderful video, “Making R 300x times faster!” It’s a great demonstration of how to rewrite code to be faster, and it’s worth your time. He rewrites some R code to be faster, then improves the speed again by writing some Rust code, which is called from R. He gets a 300 times speedup, which is really awesome.

Then someone writes in with an example of some code that is even faster than that, just using R code. It ends up being about 6 times faster than his Rust code. So a (300×6) 2000 times speed up. The main thing that helped with that was ensuring to vectorise your R code. Essentially, not working on the rows, but instead working on the columns.

Throughout the video Josiah makes good use of the bench R package to evaluate how much faster your code is. This idea is called “microbenchmarking”, and it involves running your code many times to evaluate how much faster it is than some other option. The reason you want to run your code many times is there is often variation around the runtimes in your code, so you don’t just want to base your improvements around a single measurement. It’s a general standard approach to attempt tp truly compare your approach to another.

All this being said, you should be wary of trying to make your code fast first without good reason. You want to make sure your code does the right thing first. Don’t just start trying to write performant code. Or as Donald Knuth says:

“The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.”.

If you want to learn more about how to speed up your code, I think it’s worthwhile reading up on the measuring performance chapter in Advanced R.

An example microbenchmarkLet’s take an example from the naniar package. I’ll give more detail of this story of this optimisation at the end of this section. For the moment, let’s say we want to get the number of missing values in a row of a data frame. We can do something like this:

library(dplyr)#> #> Attaching package: 'dplyr'#> The following objects are masked from 'package:stats':#> #> filter, lag#> The following objects are masked from 'package:base':#> #> intersect, setdiff, setequal, unionmy\_n\_miss <- function(data){ data |> rowwise() |> mutate( n\_miss = sum(is.na(c\_across(everything()))) ) |> ungroup()}my\_n\_miss(airquality)#> # A tibble: 153 × 7#> Ozone Solar.R Wind Temp Month Day n\_miss#> <int> <int> <dbl> <int> <int> <int> <int>#> 1 41 190 7.4 67 5 1 0#> 2 36 118 8 72 5 2 0#> 3 12 149 12.6 74 5 3 0#> 4 18 313 11.5 62 5 4 0#> 5 NA NA 14.3 56 5 5 2#> 6 28 NA 14.9 66 5 6 1#> 7 23 299 8.6 65 5 7 0#> 8 19 99 13.8 59 5 8 0#> 9 8 19 20.1 61 5 9 0#> 10 NA 194 8.6 69 5 10 1#> # ℹ 143 more rows But we can speed this up using rowSums() instead:

new\_n\_miss <- function(data){ n\_misses <- rowSums(is.na(data)) data |> mutate( n\_miss = n\_misses ) |> as\_tibble()}new\_n\_miss(airquality)#> # A tibble: 153 × 7#> Ozone Solar.R Wind Temp Month Day n\_miss#> <int> <int> <dbl> <int> <int> <int> <dbl>#> 1 41 190 7.4 67 5 1 0#> 2 36 118 8 72 5 2 0#> 3 12 149 12.6 74 5 3 0#> 4 18 313 11.5 62 5 4 0#> 5 NA NA 14.3 56 5 5 2#> 6 28 NA 14.9 66 5 6 1#> 7 23 299 8.6 65 5 7 0#> 8 19 99 13.8 59 5 8 0#> 9 8 19 20.1 61 5 9 0#> 10 NA 194 8.6 69 5 10 1#> # ℹ 143 more rowsmy\_n\_miss(airquality)#> # A tibble: 153 × 7#> Ozone Solar.R Wind Temp Month Day n\_miss#> <int> <int> <dbl> <int> <int> <int> <int>#> 1 41 190 7.4 67 5 1 0#> 2 36 118 8 72 5 2 0#> 3 12 149 12.6 74 5 3 0#> 4 18 313 11.5 62 5 4 0#> 5 NA NA 14.3 56 5 5 2#> 6 28 NA 14.9 66 5 6 1#> 7 23 299 8.6 65 5 7 0#> 8 19 99 13.8 59 5 8 0#> 9 8 19 20.1 61 5 9 0#> 10 NA 194 8.6 69 5 10 1#> # ℹ 143 more rows We can measure the speed using bench::mark():

library(bench)bm <- mark( old = my\_n\_miss(airquality), new = new\_n\_miss(airquality))bm#> # A tibble: 2 × 6#> expression min median `itr/sec` mem\_alloc `gc/sec`#> <bch:expr> <bch:tm> <bch:tm> <dbl> <bch:byt> <dbl>#> 1 old 23.9ms 23.9ms 41.8 355.5KB 334. #> 2 new 347.5µs 384.5µs 2385. 81.8KB 34.8 This runs the code at least twice, and prints out the amount of time it takes to run the code provided on the right hand side of “old” and “new”. But you can name them whatever you want.

Now, it can be kind of hard to see just how much faster this is, if you just look at comparing the times, as the times are given here in…well, actually I’m not sure why our friend from the greek alphabet mu, µ, from the greek alphabet is here, actually? If, like me, you needed to double check the standard measures of order of magnitude wiki page, you might not know that “ms” means milli – or one thousandth, and µ means “micro”, or one millionth. The point is that the new one is many times faster than the old one.

We can do a plot to help see this:

plot(bm)#> Loading required namespace: tidyr So we can see that the new one really is a lot faster.

But if I just want to be able to say something like:

It is XX times faster

then we can use the (somewhat unknown?) relative = TRUE option of bench’s S3 method for summary method:

summary(bm, relative = TRUE)#> # A tibble: 2 × 6#> expression min median `itr/sec` mem\_alloc `gc/sec`#> <bch:expr> <dbl> <dbl> <dbl> <dbl> <dbl>#> 1 old 68.7 62.2 1 4.34 9.62#> 2 new 1 1 57.0 1 1 And this is great, from this we can see it is about 60 times faster. And that the old approach uses 15 times more memory.

The story behind this speedup in naniar.Now, I didn’t just come up with a speedup for missing values on the fly. The story here is that there was going to be some (very generous) improvements to the naniar package from Romain François in the form of C++ code. However, Jim Hester suggested some changes, I think twitter (which I can’t find anymore), and he then kindly submitted a pull request showing that rowSums in R ends up being plenty fast.

This is a similar story to Josiah’s, where he used Rust code to get it faster, but then there was a faster way just staying within R.

Sometimes, you don’t need extra C or Fortran or Rust. R is enough!

And if you want to be able to compare the speeds of things, don’t forget the relative = TRUE argument in summary when using bench::mark.

Other packages for microbenchmarkingbench isn’t the only way to measure things! Other ones I’ve enjoyed using in the past are microbenchmark and tictoc. I’ve particularly enjoyed tictoc because you get to do this:

library(tictoc)tic()new\_n\_miss(airquality)#> # A tibble: 153 × 7#> Ozone Solar.R Wind Temp Month Day n\_miss#> <int> <int> <dbl> <int> <int> <int> <dbl>#> 1 41 190 7.4 67 5 1 0#> 2 36 118 8 72 5 2 0#> 3 12 149 12.6 74 5 3 0#> 4 18 313 11.5 62 5 4 0#> 5 NA NA 14.3 56 5 5 2#> 6 28 NA 14.9 66 5 6 1#> 7 23 299 8.6 65 5 7 0#> 8 19 99 13.8 59 5 8 0#> 9 8 19 20.1 61 5 9 0#> 10 NA 194 8.6 69 5 10 1#> # ℹ 143 more rowstoc()#> 0.01 sec elapsed Which feels a bit nicer than using system.time():

system.time({ new\_n\_miss(airquality)})#> user system elapsed #> 0.001 0.000 0.002 Also, notice that those two times are different? This is why we use benchmarking, to run those checks many times!

EndAnd that’s it, that’s the blog post. The relative = TRUE option in mark is super neat, and I don’t think many people know about it. Thanks again to Jim Hester for originally creating the bench package.

To leave a comment for the author, please follow the link and comment on their blog: Blog on Credibly Curious.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Find Out How many Times Faster your Code is

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Introduction to Shiny and Interactive Web ApplicationsIn today’s data-driven world, the ability to create dynamic, interactive web applications is a highly valuable skill. Shiny, a package developed by RStudio, provides an elegant framework for building such applications using R. It enables data scientists and analysts to transform their analyses into interactive experiences, making data insights accessible and engaging. This article series will guide you through mastering Shiny, starting with the basics and gradually introducing more advanced concepts and tools, including powerful packages from Appsilon that enhance Shiny’s capabilities.

Purpose and Benefits of ShinyShiny allows you to turn your R scripts into interactive web applications effortlessly. Whether you’re looking to create simple data visualizations or complex, multi-page applications, Shiny offers the flexibility and power needed to meet your objectives. Some key benefits include:

Ease of Use: Shiny’s syntax is intuitive, and if you are familiar with R, you can quickly start building applications.
Interactive Data Exploration: Users can interact with data visualizations, filtering and modifying parameters in real-time to uncover insights.
Rapid Prototyping: Shiny allows for quick development and iteration, making it perfect for prototyping data products.
Integration with R: Leverage the full power of R, including its extensive library of packages for data manipulation, visualization, and analysis.

Getting Started with ShinyBefore diving into creating your first Shiny application, ensure you have R and RStudio installed. Additionally, you’ll need to install the Shiny package if you haven’t already. Here’s how to set up your environment:

install.packages("shiny") Basic Structure of a Shiny AppA Shiny application consists of two main components:

UI (User Interface): Defines the layout and appearance of your app.
Server: Contains the logic that runs behind the scenes, processing inputs and generating outputs.

Let’s create a simple Shiny app to demonstrate these components. The following code defines a basic app that allows users to interact with a dataset and visualize its contents.

Your First Simple AppWe’ll create an app that displays the famous mtcars dataset. Users can select variables to plot and see the relationship between them.

library(shiny)# Define the UIui <- fluidPage( titlePanel("Mtcars Dataset Explorer"), sidebarLayout( sidebarPanel( selectInput("xvar", "X-axis variable", choices = names(mtcars)), selectInput("yvar", "Y-axis variable", choices = names(mtcars), selected = "mpg") ), mainPanel( plotOutput("scatterPlot") ) ))# Define the server logicserver <- function(input, output) { output$scatterPlot <- renderPlot({ ggplot(mtcars, aes\_string(x = input$xvar, y = input$yvar)) + geom\_point() + labs(title = paste("Scatter plot of", input$xvar, "vs", input$yvar)) })}# Run the applicationshinyApp(ui = ui, server = server) In this app:

The UI is defined using fluidPage, which includes a title panel and a layout with a sidebar and main panel.
The sidebar contains two selectInput elements for choosing the variables to plot.
The main panel displays a plotOutput for the scatter plot.
The server logic uses renderPlot to generate the scatter plot based on user inputs.

This simple example demonstrates the basic structure of a Shiny app, showcasing how user inputs can dynamically influence the output. With this foundation, we are ready to explore more advanced features and customizations in the next chapters, including leveraging powerful Appsilon packages to enhance our Shiny applications.

Exploring the Capabilities of “vanilla” ShinyBefore we dive into the powerful enhancements offered by Appsilon packages, it’s essential to thoroughly understand the capabilities of “vanilla” Shiny. This chapter will explore what Shiny can do out of the box, including its core features, customization options, and how it facilitates interactive data exploration. By mastering these foundational aspects, you will be well-prepared to leverage additional tools to create even more sophisticated applications.

Core Features of Vanilla ShinyVanilla Shiny provides a robust framework for building interactive web applications directly from R. Its key features include:

Interactive Widgets: Shiny offers a variety of input controls like sliders, dropdowns, text inputs, and date selectors. These widgets allow users to interact with your data and analyses dynamically.
Reactive Programming: At the heart of Shiny is its reactivity system, which ensures that the output updates automatically whenever the inputs change. This reactive model simplifies the development of interactive applications.
Dynamic User Interfaces: Shiny allows you to create UIs that change dynamically in response to user inputs. This enables the development of more interactive and responsive applications.
Seamless Integration with R: Since Shiny is built for R, you can use any R package within your Shiny apps. This includes popular packages for data manipulation (dplyr), visualization (ggplot2), and machine learning (caret).
Extensibility: Shiny applications can be extended with custom HTML, CSS, and JavaScript, allowing for more advanced customization and functionality.

Exploring Interactive WidgetsShiny provides a rich set of input controls that you can use to create interactive applications. Here are some commonly used widgets:

Slider Input: Allows users to select a range of values.

sliderInput("obs", "Number of observations:", min = 1, max = 1000, value = 500) * Select Input: Provides a dropdown menu for users to select from a list of options.

selectInput("var", "Variable:", choices = names(mtcars)) * Text Input: Allows users to enter text.

textInput("caption", "Caption:", "Data Summary") * Date Input: Allows users to select a date.

dateInput("date", "Date:", value = Sys.Date()) These widgets can be combined to create a rich user interface for your applications.

Understanding ReactivityReactivity is a core concept in Shiny that makes it easy to build interactive applications. Reactive expressions and observers automatically update outputs when their inputs change.

Reactive Expressions: Functions that return a value and automatically re-execute when their dependencies change.

reactiveExpression <- reactive({ input$sliderValue * 2}) * Observers: Functions that perform actions rather than returning values, and automatically re-execute when their dependencies change.

observe({ print(input$sliderValue)}) Here’s an example demonstrating reactivity:

library(shiny)# Define the UIui <- fluidPage( titlePanel("Reactive Example"), sidebarLayout( sidebarPanel( sliderInput("num", "Number of observations:", 1, 100, 50) ), mainPanel( textOutput("value"), plotOutput("histPlot") ) ))# Define the server logicserver <- function(input, output) { output$value <- renderText({ paste("You selected", input$num, "observations") }) output$histPlot <- renderPlot({ hist(rnorm(input$num)) })}# Run the applicationshinyApp(ui = ui, server = server) In this example:

The text output (output$value) and the plot output (output$histPlot) are both reactive, updating automatically when the slider input (input$num) changes.

Customizing the UI with HTML and CSSWhile Shiny’s built-in functions are powerful, you may sometimes need more control over the UI’s appearance and behavior. Shiny allows you to use custom HTML and CSS for further customization.

Here’s an example of incorporating custom HTML and CSS:

library(shiny)# Define the UIui <- fluidPage( tags$head( tags$style(HTML(" body { background-color: #f7f7f7; } h1 { color: #2c3e50; } .well { background-color: #ecf0f1; } ")) ), titlePanel("Custom Styled App"), sidebarLayout( sidebarPanel( sliderInput("num", "Number of observations:", 1, 100, 50) ), mainPanel( plotOutput("histPlot") ) ))# Define the server logicserver <- function(input, output) { output$histPlot <- renderPlot({ hist(rnorm(input$num)) })}# Run the applicationshinyApp(ui = ui, server = server) In this example:

We used tags$head and tags$style to include custom CSS directly in the Shiny app.
The background color, header color, and well panel color have been customized using CSS.

Extending Shiny with JavaScriptFor even more advanced interactivity and functionality, you can extend Shiny applications with custom JavaScript. Shiny provides hooks for integrating JavaScript code, allowing you to add custom behavior to your apps.

Here’s an example of adding a custom JavaScript alert when a button is clicked:

library(shiny)# Define the UIui <- fluidPage( titlePanel("JavaScript Integration"), sidebarLayout( sidebarPanel( actionButton("alertButton", "Show Alert") ), mainPanel( plotOutput("histPlot") ) ), tags$script(HTML(" $(document).on('click', '#alertButton', function() { alert('Button clicked!'); }); ")))# Define the server logicserver <- function(input, output) { output$histPlot <- renderPlot({ hist(rnorm(100)) })}# Run the applicationshinyApp(ui = ui, server = server) In this example:

We used tags$script to include custom JavaScript directly in the Shiny app.
A JavaScript alert is displayed when the button is clicked.

By mastering these core features and customization options, you can create powerful and engaging Shiny applications. In the next chapter, we will explore how to enhance these applications further with Appsilon’s styling packages, adding even more capabilities and visual appeal to your Shiny projects.

UI Design with Appsilon’s Styling PackagesThe user interface (UI) is a critical aspect of any web application, as it determines how users interact with your app and how accessible and engaging it is. In Shiny, the default UI components are functional but can sometimes look plain and lack the polish needed for professional applications. This is where Appsilon’s styling packages come in. By using shiny.semantic, shiny.fluent, and semantic.dashboard, you can create visually appealing and highly interactive UIs that stand out.

Using shiny.semantic for Elegant UIsshiny.semantic allows you to use Semantic UI, a front-end framework that provides a wide range of theming options and UI components, within your Shiny applications. This integration helps you create modern, responsive, and user-friendly interfaces without needing extensive knowledge of HTML or CSS.

To start using shiny.semantic, you'll first need to install and load the package:

install.packages("shiny.semantic")library(shiny.semantic) Let’s enhance our previous mtcars app with shiny.semantic to give it a more modern look:

library(shiny)library(shiny.semantic)library(ggplot2)# Define the UI with shiny.semanticui <- semanticPage( title = "Mtcars Dataset Explorer", segment( title = "Mtcars Dataset Explorer", sidebar\_layout( sidebar\_panel( selectInput("xvar", "X-axis variable", choices = names(mtcars)), selectInput("yvar", "Y-axis variable", choices = names(mtcars), selected = "mpg") ), main\_panel( plotOutput("scatterPlot") ) ) ))# Define the server logicserver <- function(input, output) { output$scatterPlot <- renderPlot({ ggplot(mtcars, aes\_string(x = input$xvar, y = input$yvar)) + geom\_point() + labs(title = paste("Scatter plot of", input$xvar, "vs", input$yvar)) })}# Run the applicationshinyApp(ui = ui, server = server) In this enhanced version:

We replaced fluidPage with semanticPage to utilize Semantic UI.
We used segment and sidebar_layout to structure the UI components.
The overall look is more modern and visually appealing compared to the default Shiny components.

Building Dashboards with semantic.dashboardFor more complex applications that require a dashboard layout, semantic.dashboard offers powerful tools to create sophisticated dashboards with ease. It extends shiny.semantic and adds pre-styled dashboard components.

Here’s an example of a dashboard layout for our mtcars app:

library(shiny)library(semantic.dashboard)library(ggplot2)# Define the UI with semantic.dashboardui <- dashboardPage( dashboardHeader(title = "Mtcars Dashboard"), dashboardSidebar( sidebarMenu( menuItem("Dashboard", tabName = "dashboard", icon = icon("dashboard")), menuItem("Data Explorer", tabName = "dataexplorer", icon = icon("table")) ) ), dashboardBody( tabItems( tabItem(tabName = "dashboard", fluidRow( box(title = "Controls", width = 4, selectInput("xvar", "X-axis variable", choices = names(mtcars)), selectInput("yvar", "Y-axis variable", choices = names(mtcars), selected = "mpg") ), box(title = "Scatter Plot", width = 8, plotOutput("scatterPlot")) ) ), tabItem(tabName = "dataexplorer", dataTableOutput("dataTable") ) ) ))# Define the server logicserver <- function(input, output) { output$scatterPlot <- renderPlot({ ggplot(mtcars, aes\_string(x = input$xvar, y = input$yvar)) + geom\_point() + labs(title = paste("Scatter plot of", input$xvar, "vs", input$yvar)) }) output$dataTable <- renderDataTable({ mtcars })}# Run the applicationshinyApp(ui = ui, server = server) In this dashboard version:

We used dashboardPage, dashboardHeader, dashboardSidebar, and dashboardBody to create a structured layout.
The sidebar contains a menu for navigation.
The body is divided into two tabs: one for the scatter plot and one for exploring the data table.

Creating Fluent UIs with shiny.fluentshiny.fluent integrates Microsoft’s Fluent UI into Shiny applications, providing a rich set of controls and styles. It is particularly useful for creating applications with a Microsoft Office-like feel.

Here’s how you can use shiny.fluent to enhance the mtcars app:

library(shiny)library(shiny.fluent)library(ggplot2)# Define the UI with shiny.fluentui <- fluentPage( Text(variant = "xxLarge", content = "Mtcars Dataset Explorer"), Stack( tokens = list(childrenGap = 10), Dropdown.shinyInput("xvar", label = "X-axis variable", options = lapply(names(mtcars), function(x) list(key = x, text = x)), value = "mpg"), Dropdown.shinyInput("yvar", label = "Y-axis variable", options = lapply(names(mtcars), function(x) list(key = x, text = x)), value = "hp"), plotOutput("scatterPlot") ))# Define the server logicserver <- function(input, output, session) { output$scatterPlot <- renderPlot({ ggplot(mtcars, aes\_string(x = input$xvar, y = input$yvar)) + geom\_point() + labs(title = paste("Scatter plot of", input$xvar, "vs", input$yvar)) })}# Run the applicationshinyApp(ui = ui, server = server) In this example:

Dropdown.shinyInput is used to create dropdowns for the x-axis and y-axis variables.
The Dropdown component's options argument is correctly set up with key and text fields.
plotOutput is used to display the scatter plot.
The server logic captures the input selections and updates the plot accordingly.

Accessibility and Usability TipsEnsuring that your applications are accessible and user-friendly is crucial. Here are some tips:

Use shiny.i18n for Internationalization: shiny.i18n makes it easy to translate your Shiny apps into multiple languages, ensuring they are accessible to a broader audience.
Consistent Styling: Maintain consistent styles across your application for a professional look and feel.
Responsive Design: Ensure your app works well on different devices and screen sizes.

By leveraging these Appsilon packages, you can create visually appealing, user-friendly, and highly interactive Shiny applications. In the next chapter, we will delve into advanced reactivity and routing, further enhancing the interactivity and user experience of your applications.

Advanced Reactivity and RoutingWith a solid understanding of Shiny’s core capabilities and how to enhance the UI using Appsilon’s styling packages, it’s time to delve into more advanced features. This chapter focuses on leveraging advanced reactivity with shiny.react and implementing efficient navigation using shiny.router. These tools will help you create more dynamic, responsive, and user-friendly applications.

Advanced Reactivity with shiny.reactshiny.react is a package that brings the power of React.js, a popular JavaScript library for building user interfaces, into Shiny. By using shiny.react, you can create highly responsive and interactive components that enhance the user experience.

Let’s enhance our previous mtcars app with shiny.react to add more responsive components:

library(shiny)library(shiny.react)library(shiny.fluent)library(ggplot2)# Define the UI with shiny.react and shiny.fluentui <- fluentPage( Text(variant = "xxLarge", content = "Mtcars Dataset Explorer"), Stack( tokens = list(childrenGap = 10), Dropdown.shinyInput("xvar", label = "X-axis variable", options = lapply(names(mtcars), function(x) list(key = x, text = x)), value = "mpg"), Dropdown.shinyInput("yvar", label = "Y-axis variable", options = lapply(names(mtcars), function(x) list(key = x, text = x)), value = "hp"), plotOutput("scatterPlot") ))# Define the server logicserver <- function(input, output, session) { output$scatterPlot <- renderPlot({ ggplot(mtcars, aes\_string(x = input$xvar, y = input$yvar)) + geom\_point() + labs(title = paste("Scatter plot of", input$xvar, "vs", input$yvar)) })}# Run the applicationshinyApp(ui = ui, server = server) In this code:

Dropdown.shinyInput is used to create dropdown inputs, integrating Fluent UI with Shiny reactivity.
The Dropdown component's options argument is correctly set up with key and text fields.
The fluentPage function is used to structure the UI.

Implementing Routing with shiny.routerAs your Shiny applications grow in complexity, managing navigation and routing becomes crucial. shiny.router is a package that provides a simple way to add routing to your Shiny apps, allowing you to create single-page applications (SPAs) with multiple views.

Integrating Data Science and VisualizationWith the basics of Shiny and enhanced UI elements covered, it’s time to delve into the core functionality that makes Shiny a powerful tool for data science and visualization. In this chapter, we will explore how to handle data within Shiny applications, create dynamic reports, and integrate advanced visualization libraries to provide insightful and interactive data presentations.

Data Handling in ShinyEfficient data handling is crucial for any Shiny application, especially when dealing with large datasets or complex analyses. Shiny provides several mechanisms to manage data effectively, including reactive expressions and data caching.

Reactive Data HandlingReactivity is at the heart of Shiny, allowing applications to respond to user inputs dynamically. Here’s an example of how to use reactive expressions to handle data in Shiny:

library(shiny)library(ggplot2)# Define UIui <- fluidPage( titlePanel("Reactive Data Example"), sidebarLayout( sidebarPanel( numericInput("obs", "Number of observations:", 1000, min = 1, max = 10000) ), mainPanel( plotOutput("distPlot") ) ))# Define server logicserver <- function(input, output) { # Reactive expression to generate random data data <- reactive({ rnorm(input$obs) }) # Render plot output$distPlot <- renderPlot({ ggplot(data.frame(x = data()), aes(x)) + geom\_histogram(binwidth = 0.2) + labs(title = "Histogram of Randomly Generated Data") })}# Run the applicationshinyApp(ui = ui, server = server) In this example:

A numericInput allows the user to specify the number of observations.
A reactive expression data() generates random data based on the user input.
The renderPlot function uses this reactive data to generate and display a histogram.

Dynamic Reporting with ShinyShiny can be combined with rmarkdown and knitr to create dynamic reports that update based on user inputs. This is particularly useful for generating customized reports on the fly.

Here’s an example of a simple Shiny app that generates a report using rmarkdown:

library(shiny)library(rmarkdown)# Define UIui <- fluidPage( titlePanel("Dynamic Report Example"), sidebarLayout( sidebarPanel( numericInput("obs", "Number of observations:", 1000, min = 1, max = 10000), downloadButton("report", "Generate Report") ), mainPanel( plotOutput("distPlot") ) ))# Define server logicserver <- function(input, output) { # Reactive expression to generate random data data <- reactive({ rnorm(input$obs) }) # Render plot output$distPlot <- renderPlot({ ggplot(data.frame(x = data()), aes(x)) + geom\_histogram(binwidth = 0.2) + labs(title = "Histogram of Randomly Generated Data") }) # Generate report output$report <- downloadHandler( filename = function() { paste("report-", Sys.Date(), ".html", sep = "") }, content = function(file) { tempReport <- file.path(tempdir(), "report.Rmd") file.copy("report.Rmd", tempReport, overwrite = TRUE) params <- list(obs = input$obs) rmarkdown::render(tempReport, output\_file = file, params = params, envir = new.env(parent = globalenv())) } )}# Run the applicationshinyApp(ui = ui, server = server) For this example to work, you’ll need a report.Rmd file in your working directory with the following content:

---title: "Dynamic Report"output: html\_documentparams: obs: NA---{r setup, include=FALSE}knitr::opts_chunk$set(echo = TRUE)## ReportThis report was generated dynamically using rmarkdown.The number of observations selected was r params$obs.data <- rnorm(params$obs)hist(data, main = "Histogram of Randomly Generated Data") Enhancing Shiny with Appsilon’s ExtensionsEnhancing your Shiny applications with Appsilon’s powerful extensions can significantly improve functionality, usability, and visual appeal. This chapter provides an overview of key Appsilon packages, such as shiny.semantic, shiny.fluent, semantic.dashboard, shiny.i18n, shiny.router, and shiny.react.

Key Extensionsshiny.semantic:

Integrates Semantic UI for modern, responsive designs.
Offers a wide range of UI components and theming options.

shiny.fluent:

Uses Microsoft’s Fluent UI framework for styling.
Provides consistent and visually appealing UI elements.

semantic.dashboard:

Extends shiny.semantic to create sophisticated dashboards.
Includes pre-styled components for interactive and appealing dashboards.

shiny.i18n:

Facilitates internationalization and localization.
Enables translation of Shiny apps into multiple languages, improving accessibility.

shiny.router:

Implements routing for single-page applications.
Manages navigation and structure of large applications efficiently.

shiny.react:

Integrates React.js components into Shiny.
Enhances interactivity and responsiveness of Shiny applications.

Summary of Examples UI Enhancement with shiny.semantic and shiny.fluent: Transforming basic Shiny apps into modern, responsive applications using Semantic UI and Fluent UI frameworks. * Creating Dashboards with semantic.dashboard: Building interactive and visually appealing dashboards using pre-styled components. * Internationalization with shiny.i18n: Translating Shiny applications to make them accessible to a global audience. * Routing with shiny.router: Adding navigation and structuring large applications as single-page apps. * Advanced Reactivity with shiny.react*: Incorporating React.js for highly interactive and responsive UI components.

Using these Appsilon extensions, you can significantly enhance the capabilities of your Shiny applications. These tools enable you to create more robust, user-friendly, and visually appealing applications, tailored to meet the needs of diverse users and complex projects.

ConclusionIn this article, we have explored how to harness the power of Shiny for building interactive web applications in R, leveraging advanced UI frameworks, modular development, and data visualization techniques. By integrating Appsilon’s extensions, you can significantly enhance the functionality, usability, and visual appeal of your Shiny applications.

While this guide covers various aspects of Shiny development, it’s important to note that deploying Shiny applications online is a crucial step that we haven’t delved into in detail. As I’m not an expert in deployment, I recommend the following resources for learning how to deploy Shiny applications:

Getting Started with ShinyApps.io
Introduction to Shiny Server
R Shiny Docker: How To Run Shiny Apps in a Docker Container
The Ultimate Guide to Deploying a Shiny App on AWS
How To Set Up Shiny Server on Ubuntu 20.04

By exploring these resources, you can learn how to make your Shiny applications accessible to users worldwide, ensuring they are robust, scalable, and secure.

Thank you for following along with chapters on mastering Shiny and its extensions. I hope you found the information valuable and that it helps you in your journey to creating powerful, interactive web applications with R.

Shiny and Beyond: Mastering Interactive Web Applications with R and Appsilon Packages was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Shiny and Beyond: Mastering Interactive Web Applications with R and Appsilon Packages

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Gergely Daróczi, the founder and organizer of the Budapest Users of R Network, updated the R Consortium about the group’s recent activities. Last year, Gergely discussed the group’s inception, and the challenges faced by the group during the pandemic. The group has now resumed in-person meetings, followed by networking sessions. The recent events organized by the group have focused on bioinformatics, large language models, and mathematical modeling.

Gergely Daróczi is an enthusiast R user and package developer, Ph.D. in Sociology; former assistant professor and founder of an R-based web reporting application at rapporter.net; ex Lead R Developer, then Director of Analytics at CARD.com; later Senior Director of Data Operations at System1; currently balancing the CTO role of Rx Studio, part-time lecturer at CEU along with a few open source side projects. He has contributed to a number of scientific journal articles (mainly in social sciences but in medical sciences as well), maintains a dozen CRAN packages, and wrote a book on “Mastering Data Analysis with R“.

Please share about your background and involvement with the RUGS group.

I have a background in social sciences, and it was during one of my university classes 20 years ago that I was introduced to the R language. We had to use R to run simulations related to the chaotic behavior of the Hungarian potato market. I found R more enjoyable and versatile than other GUI tools like IBM’s SPSS and started using it for other projects as well. Later, I even developed some additional packages for R.

I have been working with R for almost 20 years now. Despite my academic background in social sciences, I have worked in various industries, such as ad tech, fintech, and health tech, for the past 10 years.

In 2013, I attended my first useR! conference in Albacete, Spain, and it was a great experience to meet fellow R users from around the world. At the conference, I met Szilard Pafka, a Hungarian living in LA and organizer of the Los Angeles R User group. He suggested that I start an R User group in Hungary. After returning home, I decided to give it a shot, and we held our first meeting at the end of the summer of 2013. In a university room, it felt like there were only a dozen R users from academia. However, a lot has changed since then, as we now have almost 2,000 members in the local R User group, which exceeded my original expectations for such a small country like Hungary. It has been an interesting and great experience.

In Hungary, the community’s growth began slowly, with only 20 to 30 members in the first few years. However, it gradually increased over time. The community also hosted some famous personalities such as Romain Francois, Matt Dowle, and Hadley Wickham, which further accelerated its growth. Additionally, the community organized the first satRday and second ERUM conference, which provided a platform for networking and knowledge sharing, further strengthening the community.

How has the group been doing since our last conversation?

After COVID, restarting the meetups was very challenging. We didn’t organize any virtual events because the main benefit of meetups was meeting in person, having face-to-face conversations, and getting to know each other. Therefore, we waited until the quarantine was over and it was safe to meet in person. We started slowly, organizing only two events per year with around 30 to 70 attendees, which was much lower than before COVID-19. However, it has been great to reconnect with old friends and make new ones.

Recently, we have been focusing on bioinformatics and I was introduced to a local company that offered help with reaching out to speakers. Speakers drive these community meetings by bringing in a topic for discussion and talk, which we continue to discuss later on. Our past few events have focused on life sciences and have followed a lightning talk format, where we had around five 15-minute talks at each event. The topics were diverse, covering life sciences, some with LLMs involved, others focused on highly advanced math for modeling. We also had shiny applications that showed the biodiversity of forests in Hungary and some open-source tools besides R.

Any techniques you recommend using for planning for or during the event?

I can only offer subjective experiences on the matter, but I have witnessed the success of both virtual and in-person communities. However, our focus is on providing an exceptional in-person experience. To achieve this, we search for a central venue that is easily accessible for most of our members. This can be challenging, even in Hungary, a small country, as it can be difficult for members from other cities to travel to the capital for meetups. Nevertheless, we do our best to find a central venue, such as a university or an industry partner who can offer a space for talks and a networking opportunity afterward.

It is important to have a room with plenty of chairs and a larger area for people to gather after the talks. We can provide soft drinks, beer, or wine along with some pizzas and have a chat for an hour or two after the talk. The venue is a crucial factor. It’s also important to have speakers who are interested in the community so that they will come to learn as well. It’s great to have speakers with interesting topics, but the most important thing for me is networking. After the talks, coming together and getting to know others, learning about their struggles, and maybe sharing some tips in person with each other, becoming friends, or learning about opportunities in other industries. Networking and facilitating connections are crucial tasks for R user group organizers.

What trends do you currently see in R language?

Five years ago, machine learning models were a hot topic, and everyone discussed different implementations of GBM. However, things have changed, and nowadays, large language models (LLM) rule over all the topics. LLMs are often implemented in languages other than R, making it difficult to train them from R. Despite this, there are still many use cases for LLMs, even in life sciences and health tech. However, caution must be taken when using AI and LLMs in these fields. Recently, at two bioinformatics events, some nice use cases of LLMs were shared with the audience. This has attracted new members interested in learning how to use AI or LLMs, which can be as simple as doing some API integrations in R, such as calling the chatGPT API to generate text or images.

I’m excited that COVID restrictions are easing up and meetups are returning to normal. I can’t wait for the first in-person useR! conference in Salzburg in a few months. I highly recommend that anyone who can travel to Salzburg in July join us. The city has excellent train connections to European cities, so I hope many people from Europe can make it. I’m looking forward to attending an in-person useR! conference again.

Currently, I’m focusing on the ETL pipeline of the Spare Cores project, collecting information on cloud compute resources, which will soon have the R bindings as well. In the past, I’ve been working on R packages related to reporting (e.g. “pander”) and using R in production (e.g. “logger,” “dbr” or “boto3”). Recently, I enjoyed integrating APIs and frameworks from other programming languages, such as Python (kudos to the reticulate team!), in R.

Learn moreThe post Gergely Daróczi’s Journey: Empowering R Users in Hungary appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Gergely Daróczi’s Journey: Empowering R Users in Hungary

View Details

[This article was first published on R on kieranhealy.org, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I was messing around with some Census data this morning. I had two main thoughts. One was to show the utility of old-fashioned grayscale when it comes to mapping data (or displaying it in general). The goal of most carefully thought-through dataviz color palettes is to make them legible to viewers, which mostly means making them as linear as possible in the luminance (or brightness) channel. Which is, at root, making them more like grayscale. I also wanted to work up an example based on block-level Census counts from 2020. This was a little trickier than I initially anticipated because as far as I could tell the Tigris map data returned via the fantastic Tidycensus package was not clipped to shorelines at the block level. (It’ll do it for tract and county level, etc. But apparently not blocks right now.) I messed around with the erase_water() function in tigris which nearly did what I wanted but seemed to leave some artifacts no matter how I tuned its sensitivity. In the end I got a borough boundary shapefile from NYC Open Data and clipped the Census block data to it with st_intersection(). From some previous work I had some shapefiles for NTAs that are parks and other non-residential zones. I got the 2020 Census population counts and race/ethnicity by block for the city and, having calculated the proportion of Non-Hispanic Whites, subtracted that from one to get the POC population. It’s just a rough and ready esimate. Recall that in the US Census there’s a (historcially interesting and practically vexatious) way of classifying race and ethnicity. There’s a “race” question that yields categories like “Black”, “White”, “Asian”, and so on. And there’s an “ethnicity” question about Hispanic or Latino origin. In the Census schema, Hispanics can be of any race. (This whole situation develops over decades and contines to change.) So “Non-Hispanic White” means “People who chose White alone as their race on the Census and also did not say they were of Hispanic origin”. And the map shows the complement of that, by Census Block.

NYC POC map.

A PDF is available as well if you want to look at it in a higher resolution. All the usual caveats apply about not reifying the block areas, but they’re the most fine-grained unit data that are publicly available from the 2020 Census. One thing that’s of interest are the fairly common “empty” areas where the Census does not record any residents. These include Parks and Cemeteries, LaGuardia and JFK Airports, and a variety of places like Brooklyn Navy Yards or the U.N., etc,where no-one lives. (It also includes Rikers Island, where people do live. You can see the special blocks designating the buildings.) There are more such places than you might think, especially in Midtown Manhattan and on either side of Newtown Creek, part of the border between Brooklyn and Queens.

To leave a comment for the author, please follow the link and comment on their blog: R on kieranhealy.org.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: New York City’s POC Population

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Writing Shiny for Python applications is fun and games, but what about deploying them? Well, it’s the same story. Shiny for Python Shinylive saves the day by offering a beginner-friendly ways (plural) to share Shiny apps. Today you’ll learn a bunch of them. By the end, you’ll have your app running on Shinylive through the editor, GitHub, and also deployed locally.

Keep in mind that Shinylive is still an experimental feature, and some features might change in the future. We recommend following the official Shinylive documentation to stay up to date.

We recently launched a Shiny for Python framework for production-ready applications, Tapyr. Learn more about Tapyr and how you can set it up in this blog post.

Table of contents:

Introduction to Shiny for Python Shinylive
How to Use Shinylive Editor for Sharing Shiny for Python Apps
How to Share Shiny for Python Applications with Shinylive and GitHub
Deploy Shiny for Python Apps Anywhere – Experimental
Summing up Shiny for Python Shinylive

Introduction to Shiny for Python ShinyliveSo, what exactly is Shinylive? It’s an experimental feature that allows your Shiny for Python applications to run in a web browser. This means you don’t need a separate server for running Python.

With Shinylive, Python and Shiny run in the web browser, which stands for both the client and the server of the application.

This was all made possible because of two technologies:

WebAssembly – This is a binary format for compiled programs that runs in a web browser at near-native speeds.
Pyodide – A port of Python and many packages compiled into WebAssembly.

The main advantage of a Shinylive deployment is that the computations are done on the client. In many cases, that’s beneficial to you as a Shiny developer, since the client browser typically has more power than the server. That’s especially the case when you scale the app to dozens or hundreds of concurrent users.

On the other side, you can also consider this to be a disadvantage. For example, sometimes you need to perform intensive computations that require a powerful server. In this case, it’s not the best idea to shift the computations to the client.

We’ll explore some additional advantages and disadvantages below.

Pros of Using ShinyliveHere’s a list of clear, no-brainer advantages that come with Shinylive deployment:

No installation: You don’t need to install Python or Shiny on a computer.
Super easy sharing: Share your applications by generating a shareable URL.
Simple deployment: You can use Shinylive to deploy your apps to any static web hosting service (more on this at the end of the article).
Scalability: Apps are essentially just static files, so it’s easy to scale them.
Bullet-proof security: The app isn’t running on a server, and hence eliminates almost all security risks.

Cons of Using ShinyliveThere’s a disadvantage for every advantage, and Shinylive is no exception. Let’s go over a couple:

Python package limitation: Not all Python packages are available on Pyodide. Here’s a list of currently available ones.
Download size: Packages required to run Shiny apps via Python and Pyodide take around 13MB of data. Other packages also have a tool (Numpy = 7.5MB, Pandas = 13MB, Matplolib = 11.5MB). Keep in mind that this data will be cached, so only the first run will be slower.
No secrets: Application data and code are sent to the browser, which means the user can inspect them with ease.

You now know some advantages and drawbacks of using Shinylive. Up next, we’ll get practical!

Shiny for Python works wonders with databases – Here’s how to get started with MySQL and Postgres.

How to Use Shinylive Editor for Sharing Shiny for Python AppsThis section will walk you through using and sharing your Shiny for Python apps via the Shinylive editor.

Writing the Shiny for Python ApplicationFirst things first – we need an app. We’ll leverage the one you’ve seen in our previous Shiny for Python articles. It essentially plots a histogram of values drawn from a normal distribution. The user can change the number of bins, and the chart updates automatically.

Here’s the code for the entire application:

from shiny import ui, render, Appimport numpy as npimport matplotlib.pyplot as pltapp\_ui = ui.page\_fluid( # Sidebar layout ui.layout\_sidebar( # Sidebar with the input controls ui.panel\_sidebar( ui.h2("Page Charts"), ui.hr(), ui.input\_slider(id="slider", label="Number of bins:", min=5, max=25, value=15) ), # Main area with the chart ui.panel\_main( ui.output\_plot(id="histogram") ) ))def server(input, output, session): @output @render.plot def histogram(): # 500 samples from a normal distribution centered around 100 x = 100 + np.random.randn(500) plt.title("A histogram", size=20) plt.hist(x=x, bins=input.slider(), color="gray", ec="black")app = App(ui=app\_ui, server=server) You can paste this code into the Shinylive editor. Simply replace the already existing code and hit the run button.

This is what you’ll see on the right-hand side:

Image 1 – Shiny for Python editor (1)

As promised, you can change the number of bins by playing around with the slider in the sidebar:

Image 2 – Shiny for Python editor (2)

And that’s our app. Nothing fancy, but it’ll do for the rest of the article.

How to Share Shiny for Python AppsHave you noticed the “Share” button in the top right corner of the Shinylive editor? If not, here’s what it looks like:

Image 3 – Creating a shareable link

Clicking on it will open up a modal window, similar to this one:

Image 4 – Editor and application URLs

This is where all the magic happens. You can share the Editor URL, meaning the user will have access to the code and will be able to change it, or you can share the Application URL, meaning the user will only see the rendered application.

We recommend you click on the “Hide header” checkbox when sharing the application URL, so that the blue Shiny for Python header isn’t shown to the user.

You can copy either of these URLs and open them up in a new browser window. Here’s what you’ll get for the Application URL:

Image 5 – Shared Shiny for Python application

Impressive, isn’t it? Keep in mind that these URLs differ only in one thing – the first one has the /editor/ part, while the latter has /app/. You can guess where each one leads to.

Up next, let’s explore the option of Shinylive deployment from GitHub.

How to Share Shiny for Python Applications with Shinylive and GitHubThe answer is simple – GitHub gists. We’ll now walk you through the process of creating one and using it in Shinylive.

The main advantage of using Gists is that you can change the code at any time, but your sharing URL will stay the same.

Creating a GitHub GistIf you have a GitHub account, you can create a new Gist by clicking on the plus button in the top right corner. Here’s how yours should look like once created:

Image 6 – Shiny for Python app GitHub gist

The only thing we’ve changed is the histogram bar color – from gray to red. It’s a non-essential change but will prove to us the GitHub gist connection is actually working.

Sharing Shiny for Python App Through a GitHub GistTake a look at the gist URL. You only need the ID part, or the part after your username:

Image 7 – GitHub gist URL

You can copy this ID part and paste it at the end of the following URL:

https://shinylive.io/py/app/#gist= In our case, the complete URL would be:

https://shinylive.io/py/app/#gist=6316216a6150963a1553d55cee93b758 You can now open this URL to verify the connection works:

Image 8 – Shared Shiny for Python application (2)

And it does – we now have a Shiny for Python application with red histogram bars.

Once again, the main advantage of using gists with Shinylive is that you can update the code and reuse the Shinylive application URL. Feel free to test it out.

Deploy Shiny for Python Apps Anywhere – ExperimentalShinylive also allows you to deploy your Shiny for Python applications to static web hosts. This feature is currently experimental, so the details might change in the future. Refer to the official documentation if you have any trouble running the code.

This deployment option will essentially allow you to take your app and host it on services such as Netlify or GitHub pages. There’s also a Posit Connect option, but these services require an entire article to cover in detail.

Configuring a Shinylive Local DistributionThe first step is to install the shinylive Python package. You can do so by running the following command from the Terminal:

pip install shinylive This is the output you will see:

Image 9 – Installing Shinylive Python package

Once installed, you can create a new Shiny for Python application with the shiny create command. We’ll name ours histogram_app

shiny create histogram\_app You should see the following output:

Image 10 – Creating a Shiny for Python app

And finally, you can now use shinylive to export the Shiny for Python application:

shinylive export histogram\_app site This will create a new site directory with the static app contents:

Image 11 – Exporting Shiny for Python app for static serving

We have the app exported now, so the only thing left to do is to run it via the web server.

Serving the Local ApplicationProvided your exported Shiny for Python application is stored in a site folder, you can run it by executing the following shell command:

python -m http.server --directory site 8008 If everything went well, you’ll see a message that the app is running locally on port 8008:

Image 12 – Serving the Shiny for Python app

You can now open localhost:8008 and see your application running locally:

Image 13 – Served local Shiny for Python app

You can take things a step further by opening a local Shinylive editor. To do so, just visit localhost:8008/edit, and you’ll see a familiar-looking window:

Image 14 – Local Shinylive editor

And that’s the basic of Shinylive! Let’s make a brief recap next.

Summing up Shiny for Python ShinyliveShinylive makes sharing Shiny for Python apps easier than ever. You can do everything from the online editor, or opt for GitHub gists for more control. Also, you can serve the app locally or on any static site hosting service, such as GitHub pages.

And that’s exactly what we’ll cover next – taking local Shiny for Python applications online through GitHub pages, Netlify, Cloudflare, and Posit Connect. Make sure to stay tuned to the Appsilon blog to learn more.

What are your thoughts on Shinylive? Does it simplify your workflow for sharing Shiny for Python applications? Let us know in our Shiny community.

Is Shiny for Python better than R Shiny? We did the comparison, you do the reading.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Shiny for Python Shinylive: How to Run Shiny for Python Apps Without a Python Server

View Details

[This article was first published on Dave On R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The dplyr coalesce function is incredibly useful and similar to the SQL COALESCE function. Given a set of vectors, it finds and keeps the first non-NA value at each position. For example, the following code returns the vector c(1, 2, 3, 4).

library(dplyr)coalesce(c(1, NA, 3, NA), c(2, 2, 4, 4))

[1] 1 2 3 4 * At position 1, the first non-NA value is 1 (from the first vector). * At position 2, the first non-NA value is 2 (from the second vector, because the value at position 2 in the first vector is NA). * At position 3, the first non-NA value is 3 (from the first vector). * Finally, at position 4, the first non-NA value is 4 (from the second vector, because the value at position 4 in the first vector is NA).

The coalesce function is not limited to two vectors. You can use as many vectors as you’d like.

coalesce(c(1, 2, NA, NA), c(3, 3, 3, NA), c(4, 4, 4, 4))

[1] 1 2 3 4 The vectors must be of equal length or length 1. Vectors of length 1 will be recycled. The following:

coalesce(c(1, NA, NA, 5), 3)

[1] 1 3 3 5 Is equivalent to:

dplyr::coalesce(c(1, NA, NA, 5), c(3, 3, 3, 3))

[1] 1 3 3 5 I most often use coalesce to replace all NA values in a vector with a single value. For example, the following code replaces all NA values with 0:

coalesce(c(1, NA, NA, 5, 6), 0)

[1] 1 0 0 5 6 Here are two alternative ways to do the same thing.

library(tidyr)replace\_na(c(1, NA, NA, 5, 6), 0) # this function is in the tidyr package

[1] 1 0 0 5 6

x <- c(1, NA, NA, 5, 6)ifelse(is.na(x), 0, x)

[1] 1 0 0 5 6 Of the three options I have shown so far, I prefer coalesce and replace_na. (coalesce is a more general version of replace_na; replace_na takes a vector and a single value to replace the NA values in that vector with). If you want to go outside of base R and the tidyverse, data.table::fcoalesce is a much faster version of dplyr::coalesce.

Let’s compare the speeds!

library(microbenchmark)library(data.table)set.seed(11)x <- 1:10^7x[sample(1:10^7, size = 10^6, replace = FALSE)] <- NAmicrobenchmark( fcoalesce(x, 0L), coalesce(x, 0L), replace\_na(x, 0L), ifelse(is.na(x), 0L, x), times = 5 )

Unit: milliseconds expr min lq mean median uq max fcoalesce(x, 0L) 14.4582 16.1549 18.47224 18.5650 19.1356 24.0475 coalesce(x, 0L) 285.4586 289.1238 318.36890 292.7322 297.1620 427.3679 replace\_na(x, 0L) 37.2726 41.0626 51.48104 41.4943 52.0923 85.4834 ifelse(is.na(x), 0L, x) 168.9397 178.8357 191.00360 180.9929 186.2458 240.0039 neval 5 5 5 5 data.table:fcoalesce is the winner in terms of speed, followed by replace_na. Of the four methods above, coalesce and fcoalesce are the most general, since they are not limited to replacing all NA values with a single value.

Is there a base R equivalent to dplyr::coalesce? No. But we can easily create one using just base R code.

To think about how we would do that, let’s start with two vectors:

x <- c(1, 2, NA, NA)y <- c(2, 2, 3, NA)coalesce(x, y)

[1] 1 2 3 NA How could we get the same results using the ifelse function? It’s simple. We return the value in y when the value in x is NA.

ifelse(is.na(x), y, x)

[1] 1 2 3 NA That’s simple enough. But what if we want to use three vectors?

z <- c(4, 4, 4, 4)coalesce(x, y, z)

[1] 1 2 3 4 We can start with our code from the case where we used two vectors.

output\_step\_1 <- ifelse(is.na(x), y, x) When both x and y are NA (when ifelse(is.na(x), y, x) gives us NA), we want to use what is in z. Otherwise, we want to keep the results from step 1 above.

ifelse(is.na(output\_step\_1), z, output\_step\_1)

[1] 1 2 3 4 That worked! But what if we want to generalize this to any number of input vectors? We can use the base R function Reduce. For our case where we used x, y, and z, we could do:

Reduce(function(x, y) ifelse(is.na(x), y, x), list(x, y, z))

[1] 1 2 3 4 Reduce(f, list(x, y, z)), where f is a function of two variables, is the equivalent of f(f(x, y), z). And Reduce(f, list(x, y, z, a)) is equivalent to f(f(f(x, y), z), a). The Reduce function is used to iteratively apply a function to elements of a list, reducing it to a single value. It takes a function with two parameters and applies it to the first two elements of the list, then applies the same function to the result and the next element, and so on, until all elements are combined into a single value.

To use Reduce to mimic coalesce(x, y, z, ...), we need to apply the logic f <- function(x, y) { ifelse(is.na(x), y, x)} over and over starting from left to right. In other words, for three vectors x, y, and z, we need to do:

f <- function(x, y) { ifelse(is.na(x), y, x)}f(f(x, y), z)

[1] 1 2 3 4 Which is equivalent to

Reduce(f, list(x, y, z))

[1] 1 2 3 4 So a very simple base R function equivalent to the coalesce function is:

coalesce\_base\_r <- function(...) { args <- list(...) Reduce(function(x, y) ifelse(is.na(x), y, x), args)} Let’s see if it produces identical results to dplyr::coalesce:

set.seed(11)x <- 1:10^7y <- 1:10^7x[sample(1:10^7, size = 10^6, replace = FALSE)] <- NAy[sample(1:10^7, size = 10^6, replace = FALSE)] <- NAz <- 1Ldplyr\_result <- coalesce(x, y, z)base\_r\_result <- coalesce\_base\_r(x, y, z)identical(dplyr\_result, base\_r\_result)

[1] TRUE We get identical results!

Let’s compare the speed:

microbenchmark(coalesce(x, y, z), coalesce\_base\_r(x, y, z), times = 5)

Unit: milliseconds expr min lq mean median uq max coalesce(x, y, z) 299.6434 321.6311 355.9083 357.0767 385.4815 415.709 coalesce\_base\_r(x, y, z) 349.0094 406.5152 400.0490 412.2628 413.9706 418.487 neval 5 5 Our base R version of coalesce is almost identical in speed dplyr’s!

But our function contains some flaws. dplyr’s coalesce function forces the vectors passed to it to either be of the same length or be of length 1. If we try:

coalesce(c(1, 2, 3, NA, 6), c(4, 5))

Error in `coalesce()`:! Can't recycle `..1` (size 5) to match `..2` (size 2). We get an error, since the first vector has length 5 and the second has length 2.

Here’s a better base R version of coalesce:

coalesce\_base <- function(...) { args <- list(...) # Check for NULL, zero-length vectors, and collect lengths lengths <- sapply(args, function(x) { if (is.null(x) || length(x) == 0) { stop("Arguments must not be NULL or zero-length vectors") } length(x) }) # Determine the maximum length max\_length <- max(lengths) # Check if lengths are consistent. # Only allow vectors of length equal to max length or length of 1 if (any(lengths != max\_length & lengths != 1)) { stop("All arguments must have the same length, except for vectors of length 1 which can be recycled") } # Use Reduce with ifelse to coalesce Reduce(function(x, y) ifelse(is.na(x), y, x), args)}# Example usage:v1 <- c(NA, 2, NA, 4, NA)v2 <- c(1, NA, 3, NA, NA)v3 <- 0coalesce\_base(v1, v2, v3)

[1] 1 2 3 4 0 Again, let’s compare the speed of our function to dplyr::coalesce and data.table::fcoalesce.

microbenchmark(fcoalesce(x, y, z), coalesce(x, y, z), coalesce\_base\_r(x, y, z), times = 5)

Unit: milliseconds expr min lq mean median uq fcoalesce(x, y, z) 16.9233 17.5952 19.12082 17.6570 19.5384 coalesce(x, y, z) 304.2171 321.0251 343.79622 355.3758 358.6747 coalesce\_base\_r(x, y, z) 344.0853 357.0263 383.64736 386.5437 388.1294 max neval 23.8902 5 379.6884 5 442.4521 5 data.table::fcoalesce is the clear winner when it comes to speed! Our function is almost identical in speed to dplyr::coalesce!

To leave a comment for the author, please follow the link and comment on their blog: Dave On R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Base R Equivalents of dplyr Functions Part 1 – coalesce

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionCounting words in a string is a common task in data manipulation and text analysis. Whether you’re parsing tweets, analyzing survey responses, or processing any textual data, knowing how to count words is crucial. In this post, we’ll explore three ways to achieve this in R: using base R’s strsplit(), the stringr package, and the stringi package. We’ll provide clear examples and explanations to help you get started.

ExamplesCounting Words Using Base R’s strsplit()Base R provides a straightforward way to split strings and count words using the strsplit() function. Here’s a simple example:

```

Define a stringtext <- "R is a powerful language for data analysis."# Split the string into wordswords <- strsplit(text, "\s+")[[1]]# Count the wordsword_count <- length(words)# Print the resultword_count

```

[1] 8 Explanation:

Define a String: We start with a string, text.
Split the String: The strsplit() function splits the string into words based on whitespace (\\s+).
Count the Words: We use length() to count the elements in the resulting vector, which represents the words.

Syntax:

strsplit(x, split, fixed = FALSE, perl = FALSE, useBytes = FALSE) * x: Character vector or string to be split. * split: Regular expression or string to split by. * fixed: Logical, if TRUE, split is a fixed string, not a regular expression. * perl: Logical, if TRUE, perl = TRUE enables Perl-compatible regexps. * useBytes: Logical, if TRUE, use byte-wise splitting.

Try modifying the text variable to see how the word count changes!

Counting Words Using stringrThe stringr package provides a more readable and convenient approach to string manipulation. To use stringr, you’ll need to install and load the package:

```

Install stringr if you haven't already# install.packages("stringr")# Load the stringr packagelibrary(stringr)# Define a stringtext <- "R makes text manipulation easy and fun."# Split the string into wordswords <- str_split(text, "\s+")[[1]]# Count the wordsword_count <- length(words)# Print the resultword_count

```

[1] 7 Explanation:

Load the Package: After installing and loading stringr, we define our string, text.
Split the String: We use str_split() to split the string into words.
Count the Words: The length() function counts the number of words.

Syntax:

str\_split(string, pattern, n = Inf, simplify = FALSE) * string: Input character vector. * pattern: Pattern to split by (regular expression). * n: Maximum number of pieces to return. * simplify: Logical, if TRUE, return a matrix with elements.

The stringr package makes the code more intuitive and easier to read. Experiment with different strings to get comfortable with str_split().

Counting Words Using stringiThe stringi package is known for its powerful and efficient string manipulation functions. Here’s how to use it to count words:

```

Install stringi if you haven't already# install.packages("stringi")# Load the stringi packagelibrary(stringi)# Define a stringtext <- "Learning R can be a rewarding experience."# Split the string into wordswords <- stri_split_regex(text, "\s+")[[1]]# Count the wordsword_count <- length(words)# Print the resultword_count

```

[1] 7 Explanation:

Load the Package: Install and load the stringi package.
Split the String: Use stri_split_regex() to split the string based on whitespace.
Count the Words: Count the words using length().

Syntax:

stri\_split\_regex(str, pattern, n = -1, omit\_empty = FALSE, tokens\_only = FALSE, simplify = FALSE) * str: Input character vector. * pattern: Regular expression pattern. * n: Maximum number of pieces. * omit_empty: Logical, if TRUE, remove empty strings from the output. * tokens_only: Logical, if TRUE, return tokens. * simplify: Logical, if TRUE, return a matrix with elements.

The stringi package offers high performance and is great for handling large datasets or complex text manipulations. Give it a try with different text inputs to see its efficiency in action.

ConclusionCounting words in a string is a fundamental task in text analysis, and R provides multiple ways to accomplish this. We’ve explored three methods: base R’s strsplit(), stringr, and stringi. Each method has its strengths, and you can choose the one that best fits your needs.

Feel free to experiment with these examples and try counting words in your own strings. By practicing, you’ll become more comfortable with string manipulation in R, opening the door to more advanced text analysis techniques.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Counting Words in a String in R: A Comprehensive Guide

View Details

[This article was first published on vgherard, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.TheoryAICConsider the AIC for the usual linear model (Y = X\beta + \varepsilon):

[\text{AIC} = \frac{1}{2}\ln(2\pi e\hat \sigma^2)+\frac{p+1}{N}(#eq:AIC)]

where (p) is the dimension of thecovariate vector (X) and (\hat \sigma ^2) is the ML estimate of the(Y\vert X) conditional variance. Theexpectation of @ref(eq:AIC) under model assumptions can be found byusing the fact that, for a (\chi^2)random variable with (\nu) degrees offreedom1:

[\mathbb E(\ln\chi ^2 )=\ln2+ \psi(\frac{\nu}{2})(#eq:ElogX2)] where: [\psi(x)\equiv\frac{\text d}{\text d x}\ln \Gamma(x) \approx \lnx-\frac{1}{2x}(#eq:PsiFun)] and the second equality results from the Stirlingapproximation (\Gamma(x) =\sqrt{2\pi}x^{x-\frac{1}{2}}e^{-x}). We obtain:

[\mathbb E(\text{AIC}) = \frac{\ln \left[2\pi e\mathbb V(Y\vert X)\right]}{2}+\frac{1}{2}\ln\left(\frac{N-p}{2}\right)-\frac{1}{2}\frac{1}{N-p}+\frac{p+1}{N},(#eq:EAIC)] where, according to standard assumptions, (\mathbb V(Y \vert X)) is assumed to beconstant in (X).

Now consider two such models, with different covariate vectors (X_1) and (X_2), of dimension (p_1) and (p_2) respectively, both assumed to be wellspecified. Denote, as before:

[\text{AIC}_i =\frac{1}{2}\ln(2\pi e\hat \sigma^2_i)+\frac{p_i+1}{N}(#eq:AICi)] for (i = 1,\,2). Equation@ref(eq:EAIC) gives the unconditional expectation of (\text{AIC}) for both models2, so that:

[\mathbb E(\text{AIC}_1 – \text{AIC}_2) =\frac{1}{2}\ln\left(\frac{\mathbb V(Y\vert X_1)}{\mathbb V(Y\vertX_2)}\right)+\frac{p_1-p_2}{2N}+\mathcal O(N^{-2})(#eq:DeltaEAIC).]

Assuming, without loss of generality, that (p_1 \leq p_2), we have:

[\mathbb E(\text{AIC}_1 – \text{AIC}_2) < 0 \iff N <\frac{p_2-p_1}{\ln\left(\frac{\mathbb V(Y\vert X_1)}{\mathbb V(Y\vertX_2)}\right)}.(#eq:AICCondition)] To gain some intuition, suppose that the set of variablescontained in (X_1) is a subset ofthose contained in (X_2), so that thetwo corresponding models are nested. Eq. @ref(eq:AICCondition) tells usthat, for (N) below a certainthreshold, AIC will prefer the more “parsimonious” model involving (X_1) only. In particular, if (\mathbb V(Y\vert X_1)\approx \mathbb V(Y\vertX_2)), we can make a first-order approximation in the RHS of Eq.@ref(eq:AICCondition), that yields:

[N \lesssim \frac{\mathbb V(Y\vert X_2)}{\mathbb V(Y\vert X_1)-\mathbbV(Y\vert X_2)}(p_2-p_1).(#eq:AICConditionApprox)]

Cross-entropyIn parallel to AIC, we can consider the exact “information criterion”provided by the model in-sample cross-entropy under the true datagenerating process. For a single linear model, the in-samplecross-entropy is:

[\text{CE}_{\text {in}} = \frac{1}{2}\ln(2\pi e \hat \sigma ^2)+\frac{1}{2}\frac{\sigma ^2-\hat \sigma^2+\frac{1}{N}(\beta-\hat{\beta})^{T}\mathbf{X}^{T}\mathbf{X}(\beta-\hat{\beta})}{\hat\sigma ^2}.(#eq:InSampleCrossEntropy)] (“in-sample” refers to the fact that we fix, i.e.condition, on the covariate vector of the training sample, (\mathbf X).) The (\mathbf X) conditional expectation of(\text{CE}_{\text {in}}), again undermodel assumptions, can be computed by noticing two facts:

The numerator and denominator are conditionally independent (\chi^2) variables with (p) and (N-p) degrees of freedom respectively. Thiscan be seen by rewriting these as (\boldsymbol \epsilon ^T \mathbf H \boldsymbol\epsilon), and (\boldsymbol \epsilon^T (1-\mathbf H) \boldsymbol \epsilon), respectively, where(\mathbf H = \mathbf X (\mathbf X ^T \mathbfX)^{-1} \mathbf X ^T) as usual.
For a (\chi ^2) random variablewith (\nu) degrees of freedom we have(\mathbb E(\frac{1}{\chi ^2})=\frac{1}{\nu –2}).

Using these results, we can show that:

[\mathbb E(\text{CE}_{\text {in}}\vert \mathbf X)=\mathbbE(\text{AIC}\vert \mathbf X)+\mathcal O(N^{-2})(#eq:AICvsCE)] (an equation which is true by design of AIC).

Before rushing to the (wrong) conclusion that (\text{AIC}_1 – \text{AIC}_2) willcorrespondingly estimate a difference of expected cross-entropies, letus notice that the relevant in-sample cross-entropy to be considered formodel evaluation is Eq. @ref(eq:InSampleCrossEntropy) with (\mathbf X) corresponding to the fullcovariate vector: this is the target we should try to estimate (atleast to the extent that our goal is predicting (Y) given (X)). For this reason, strictly speaking,Eq. @ref(eq:AICvsCE) is exact only if our model is well specified as amodel of (Y \vert X). Otherwise, inorder to estimate consistently (\mathbbE(\text{CE}_{\text {in}}\vert \mathbf X)), we should useTakeuchi’s Information Criterion (TIC) rather than AIC.

A bit more pragmatically, in the real world we could assume theremainder of @ref(eq:AICvsCE) to be (\mathcalO (N^{-1})) (rather than (\mathcal O(N^{-2}))), but generally small with respect the leading orderAIC correction ((\frac{p+1}{N})).This will be the case if the models being compared are approximatelywell specified.

SimulationSetupWe take the data generating process to be:

[Y = m X + q + \varepsilon, (#eq:DGPSim)] with:

[X \sim \mathcal N (0,\,1),\quad \varepsilon \sim \mathcal N(0,\,1),\quad\varepsilon \perp X. (#eq:DGPSim2)]

m <- 0.1q <- 0rxy <- function(n) { tibble( x = rnorm(n, sd = 1), y = m * x + q + rnorm(n, sd = 1) )} We compare the model with vs. without slope term ((m = 0) vs. (m \neq 0)), which we will denote bysuffixes (1) and (1\oplus X), respectively. The functionsbelow compute AIC and in-sample cross-entropy from the correspondinglm objects. We also define a “Naive Information Criterion”(\text{NIC} \equiv \log(\hat\sigma)).

nic <- function(fit) { p <- length(coef(fit)) n <- nobs(fit) sigma\_hat <- sigma(fit) * sqrt((n - p) / n) log(sigma\_hat)}aic <- function(fit) { p <- length(coef(fit)) n <- nobs(fit) sigma\_hat <- sigma(fit) * sqrt((n - p) / n) log(sigma\_hat) + (p + 1) / n + 0.5 *(1 + log(2*pi))}ce <- function(fit, data) { p <- length(coef(fit)) n <- nobs(fit) sigma\_hat <- sigma(fit) * sqrt((n - p) / n) y\_hat <- fitted(fit) mu <- data$x * m + q res <- 0 res <- res + 0.5 / (sigma\_hat^2) res <- res + log(sigma\_hat) res <- res + mean(0.5 * (y\_hat - mu)^2 / (sigma\_hat^2)) res <- res + 0.5 * log(2 * pi) return(res)} From our results above, we expect:

[\mathbb E(\text{AIC}_{1\oplus X}-\text{AIC}_{1} )<0 \iff N \geq\frac{1}{\ln(1+m^2)}\left(1+\mathcal O(m^2 )\right)(#eq:DeltaEAICSim)] The expected in-sample cross-entropies cannot be computedexplicitly, but for relatively small (m^2) we expect (cf. Eq.@ref(eq:AICvsCE)):

[\mathbb E((\text{CE}_{\text {in}})_i)=\mathbb E(\text{AIC}_i)+\mathcalO(N^{-2},\,m^2N^{-1}),(#eq:AICvsCESim)]

I will use tidyverse for plotting results.

library(dplyr)library(ggplot2) In order to make results reproducible let’s:

set.seed(840) ResultsWe simulate fitting models (1) and(1\oplus X) at different sample sizesfrom the data generating process described above.

fits <- tidyr::expand\_grid( n = 10 ^ seq(from = 1, to = 3, by = 0.5), b = 1:1e3 ) |> mutate(data = lapply(n, rxy)) |> group\_by(n, b, data) |> tidyr::expand(model = c(y ~ 1, y ~ x)) |> ungroup() |> mutate( fit = lapply(row\_number(), \(i) lm(model[[i]], data = data[[i]])), ce = sapply(row\_number(), \(i) ce(fit[[i]], data[[i]])), aic = sapply(fit, aic), nic = sapply(fit, nic), model = format(model) ) |> select(-c(fit, data)) The plots below show the dependence from sample size of (\mathbb E(\Delta\text{AIC})) and (\mathbb E(\Delta\text{CE}_\text{in})), aswell as AIC selection frequencies. Notice that for (N = \frac{1}{m^2}), even though (\mathbb E(\Delta\text{AIC}) = 0), theselection frequency of the “complex” model (1\oplus X) is still below (\text{50 %}). This is because thedistribution of (\Delta\text{AIC}) isasymmetric, as seen in the second plot, and (\mathbb E(\Delta\text{AIC}) < \text{median}(\Delta\text{AIC})).

fits |> mutate( is\_baseline = model == "y ~ 1", delta\_ce = ce - ce[is\_baseline], delta\_aic = aic - aic[is\_baseline], delta\_nic = nic - nic[is\_baseline], .by = c(n, b), ) |> filter(!is\_baseline) |> summarise( `E( ΔCE )` = mean(delta\_ce), `E( ΔAIC )` = mean(delta\_aic), `E( ΔNIC )` = mean(delta\_nic), .by = n ) |> tidyr::pivot\_longer( -n, names\_to = "metric", values\_to = "value" ) |> ggplot(aes(x = n, y = value, color = metric)) + geom\_point() + geom\_line() + geom\_hline(yintercept = 0, linetype = "dashed") + geom\_vline(aes(xintercept = 1 / m^2), linetype = "dotted") + scale\_x\_log10("Sample Size") + coord\_cartesian(ylim = c(-0.025, 0.025)) + ylab(expression(IC)) + theme(legend.position = "bottom", legend.title = element\_blank()) + ggtitle("AIC vs. in-sample cross-entropy", "Expected values") + NULL

fits |> filter(aic == min(aic), .by = c(n, b)) |> summarise(count = n(), .by = c(n, model)) |> ggplot(aes(fill = model, x = n, y = count)) + geom\_col() + scale\_x\_log10("Sample Size") + ylab("Count") + theme(legend.position = "bottom") + ggtitle("AIC model selection frequencies")

fits |> filter(n %in% c(10, 100, 1000)) |> mutate(delta\_aic = aic - aic[model == "y ~ 1"], .by = c(n, b)) |> filter(model != "y ~ 1") |> mutate(expec = -0.5 * log(1 + m^2) + 0.5 / n) |> ggplot(aes(x = delta\_aic, color = as.factor(n))) + geom\_density() + coord\_cartesian(xlim = c(-0.1, NA)) + labs(x = "ΔAIC", y = "Density", color = "Sample Size") + ggtitle("ΔAIC probability density") Finally, here is something I have no idea where it comes from. Theplot below shows the scatterplot of in-sample cross-entropy differencesvs. the AIC differences. It is well known that AIC onlyestimates the expectation of these differences, averaged over potentialtraining samples. One may ask whether AIC has anything to say about theactual cross-entropy difference for the estimated models, conditional onthe realized training sample.

Assuming I have made no errors here, the tilted-U shape of thisscatterplot is a clear negative answer. What’s especially interesting isthat, apparently, these differences have a negative correlation. I failto see where do the negative correlation and the U-shape come from.

fits |> filter(n == 100) |> mutate( is\_baseline = model == "y ~ 1", delta\_ce = ce - ce[is\_baseline], delta\_aic = aic - aic[is\_baseline], .by = c(n, b), ) |> filter(!is\_baseline) |> ggplot(aes(x = delta\_aic, y = delta\_ce)) + geom\_point(size = 1, alpha = 0.2) + lims(x = c(-0.02, 0.01), y = c(-0.01, 0.03)) + labs(x = "ΔAIC", y = "ΔCE") + ggtitle("AIC vs. in-sample cross-entropy", "Point values for N = 100") + NULL

See e.g. 1503.06266︎
The same equation actually gives the expectation of(\text{AIC}) conditional to thein-sample covariate vector (\mathbbX). Since this conditioning differs for the two different modelsinvolving (X_1) and (X_2), in our comparison of expected valueswe must interpret this as unconditional expectations, in general.︎ To leave a comment for the author, please follow the link and comment on their blog: vgherard.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: AIC in the well-specified linear model: theory and simulation

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium recently interviewed Jan Vitek, a professor at Northeastern University’s Khoury College of Computer Sciences. He specializes in programming languages, compilers, and systems. Notably, he developed one of the first real-time Java virtual machines in collaboration with Boeing, which involved writing the navigation software of a ScanEagle UAV in Java and demonstrating that it out-performed the legacy version of the system written in C++. Vitek is actively involved in the programming language community and has held multiple leadership roles, including chairing SIGPLAN. In his spare time Vitek is a cinephile with a presence on Letterboxd and is the human of a dog named Olaf.

Vitek has been working on R for a decade. He is currently working on the MaintainR 2021 project, which aims to support and update the key components of the R ecosystem. The R Consortium is funding this project.

Can you provide an overview of the MaintainR 2021 project and its main objectives?

“When does a programming language die?” is the wrong question. Languages do not die, they slowly fade into irrelevance. A language fades away when no longer deemed useful enough for people to learn it and convince their colleagues to adopt it in their work and to maintain software projects written in it. Why does this happen? It comes about when newer languages that are better or appear cooler, start to emerge. The rise of Python has shifted many machine learning users from R to Python. The success of Julia has pushed performance-sensitive users to develop new mathematical libraries in this new language. Is R fading?

The programming landscape is evolving, and R, which has been around since 1995, isn’t the newest option available. To remain relevant any complex language depends on a large ecosystem of software elements that must be maintained and fixed regularly. R is certainly complex and it has many dependencies. It relies on a core group of developers who are allowed to make changes in the key parts of the language. These developers, while, on average, being significantly younger than Joe Biden, are not getting younger.

My work focuses on trying to modernize R. I’ve been examining R from a computer science perspective for about a decade, focusing on software components such as just-in-time compilers. My group is currently in the midst of writing our third attempt at writing a compiler for R. This effort led me to bring my collaborator, Tomas Kalibera, into the R community through our projects, which sparked his desire to assist the community. This was all part of a natural extension of our research. The goal of the MaintainR project is to maintain key parts of the R environment, which are challenging for volunteers to sustain. We have not found companies willing to contribute top-notch software engineers for this maintenance effort for their own reasons—perhaps they don’t have the resources, or they’re occupied with other tasks. Thus, our effort is focused on providing the necessary maintenance to prolong R’s usefulness.

The R ecosystem is dependent on the R interpreter, the core libraries, and CRAN. Which takes the most effort to maintain and why?

Everything in this project is challenging because the components vary greatly in size and heterogeneity. The interpreter is the smallest part, which everyone relies on. Then, there’s the core library, which is about ten times larger than the interpreter and is a mix of R, C, and Fortran. Fortran isn’t as popular as it used to be, and we encounter issues when compiling it with modern compilers like LLVM. Ensuring Fortran compiles across all desired architectures and operating systems has been a persistent challenge.

We’ve also had difficulties integrating patches into LLVM and GCC for this purpose. Changes in these compilers can lead to breakages in our environment. The crown packages contain vast code—potentially 100 times more than the core library. This creates an inverted pyramid scenario where the amount of code increases as you move up the structure.

Maintaining these packages is not our direct responsibility, but we can’t ignore them. Some are crucial for the users’ satisfaction. Some packages inevitably break as the language evolves and new versions are released. Tomas often has to approach maintainers to inform them of these issues. Sometimes, they respond and agree to implement fixes, but not always. Even when a technical fix might take just half a day, it can require a full week of negotiation with a developer to accept the patch.

This social aspect of software maintenance is significant and often the most challenging part. Developers have their own priorities, and a patch that doesn’t align with their goals can be seen as disruptive. Sometimes, the delay is simply because they are slow to respond. This complex interplay of technical and social challenges is a constant part of our efforts to keep the project moving forward.

Tomas Kalibera, a member of the core R team and supported as part of the MaintainR project, has implemented CheckR, a software tool for verification of the C code linked against the R interpreter. Can you explain how CheckR improves the R ecosystem and the overall quality of packages available to R users?

My team developed a tool called CheckR, which addresses issues arising from libraries written with a substantial amount of C code. The aim is to identify potentially misbehaving C that could cause unpredictable crashes, leading end users to mistakenly believe that R itself is faulty when, in fact, the issue may stem from a poorly written library or careless usage.

CheckR processes the C code, transforms it, and an analyzer identifies points where things might go wrong. A common issue it detects involves what we call “Protect bugs.” This happens when the R code sends a value down to C, and C must “protect” this value to prevent it from being reclaimed by the garbage collector. Sometimes, developers handle this in a hasty and imprecise manner. If they make an error, the value given to C can be reclaimed and reused, leading to memory corruption—this could result in security flaws or crashes.

CheckR is a static analyzer that flags potential issues but is not always definitive. It identifies possible problems, and we return to the developers to discuss whether these should be fixed. Often, developers are skeptical about the identified issues, which can lead to extended discussions. Sometimes, these issues might never occur, but often they do, and since CheckR is used daily across our entire codebase, it automatically generates reports that help us address these vulnerabilities.

The next steps with this tool aren’t always clear-cut because we can’t predict all potential issues. For example, one persistent challenge has been how the Windows operating system encodes Unicode characters, requiring months of troubleshooting. Could we have foreseen this particular issue? Not really. It’s part of the unpredictable nature of software development, where new problems can emerge at any time.

Have you been successful in extending the life of R by having CheckR run daily and helping with the interpreter, libraries, and CRAN?

Thousands of changes have been made to the R environment over time, and while I’d like to say that these changes have definitely improved things, as a scientist, I feel the need to provide concrete evidence, which I can’t always do. However, I can confidently say that each time we identify and fix a bug, the system has one less problem. The challenge, though, is that the potential for bugs can be virtually unlimited because new code is continually being added. It’s an ongoing process, and realistically, there might never be a point where we can declare it completely done.

How has it been working with the R Consortium? Would you recommend applying for an ISC grant to other R developers?

In our case, a lot of the work we do isn’t glamorous, and most volunteers are drawn to projects where they can attach their names to something flashy. Yet, there’s a continuous stream of necessary tasks that aren’t as appealing but are essential. Without a steady source of funding, sustaining efforts like ours would be impossible. The process we follow is streamlined, the community is welcoming, and your contributions can significantly impact a large user base.

The key message here is that funding is incredibly beneficial, especially for supporting those who contribute more gradually; the return on this investment is significant. For instance, in our project, without the funding, we couldn’t have supported the work of Tomas Kalibera, and nothing would have progressed. No company was willing to employ someone full-time for this, despite it being a crucial component of the ecosystem. Being able to provide funding allows us to engage someone who might otherwise have to spend their time on other activities and only contribute to this project in their spare time. Having someone fully dedicated for even a limited period is a tremendous advantage.

Learn moreThe post Enhancing R: The Vision and Impact of Jan Vitek’s MaintainR Initiative appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Enhancing R: The Vision and Impact of Jan Vitek’s MaintainR Initiative

View Details

[This article was first published on pacha.dev/blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. MotivationI’ve been playing with the idea of running Doom on Shiny for a while. If Doom runs “everywhere”, it must run on Shiny, right?

Doom was released in 1993 and since then it has been ported to many different platforms including:

A Kodak digital camera from 1998
An ATM
A pregnancy test

The resultI was able to run Doom on Shiny!

I have uploaded a short demo to YouTube. In some countries the video is only for +18 audiences because of the original game parental advisory.

Here’s a picture of the game running on Shiny:

How it worksI created what I think is a simple Shiny app that I released on GitHub.

Here is the code:

library(shiny)library(httpuv)# Define the path to the directory containing your Doom game filesdoom\_directory <- "./doom-wasm/src"# Start a server to serve Doom filesstartServer("127.0.0.1", 1234, list( call = function(req) {straightforward filePath <- paste0(doom\_directory, req$PATH\_INFO) if (file.exists(filePath)) { return(list( status = 200, headers = list( "Content-Type" = "text/html" ), body = readBin(filePath, "raw", file.info(filePath)$size) )) } else { return(list( status = 404, headers = list( "Content-Type" = "text/html" ), body = "File not found" )) } }))ui <- fluidPage( tags$h1("Doom in Shiny"), tags$iframe(style = "width:850px; height:640px;", src = "http://127.0.0.1:1234/index.html"))server <- function(input, output, session) { }# Run the Shiny appshinyApp(ui, server, options = list(port = 4321)) To run the app, you need to build Doom from source. The same repository contains the instructions.

If you like what I do as a content creator, you can donate on Buy Me a Coffee.

To leave a comment for the author, please follow the link and comment on their blog: pacha.dev/blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: If Doom runs everywhere, it must run on Shiny

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhether you’re doing some data cleaning or exploring your dataset, checking if a column contains a specific string can be a crucial task. Today, I’ll show you how to do this using both str_detect() from the stringr package and base R methods. We’ll also tackle finding partial strings and counting occurrences. Let’s dive right in!

Using str_detect from stringrFirst, we’ll use the str_detect function. The stringr package is part of the tidyverse collection, which brings a set of user-friendly functions to text manipulation. We’ll start by ensuring it’s installed and loaded:

install.packages("stringr") Now, let’s create a sample dataset:

library(stringr)# Sample datadata <- data.frame( name = c("Alice", "Bob", "Carol", "Dave", "Eve"), description = c("Software developer", "Data analyst", "UX designer", "Project manager", "Data scientist"))data

name description1 Alice Software developer2 Bob Data analyst3 Carol UX designer4 Dave Project manager5 Eve Data scientist ExamplesUsing stringrCheck for Full StringSuppose we want to check if any of the description column contains “Data analyst”:

```

Detect if 'description' contains 'Data analyst'data$has_data_analyst <- str_detect(data$description, "Data analyst")print(data)

```

name description has\_data\_analyst1 Alice Software developer FALSE2 Bob Data analyst TRUE3 Carol UX designer FALSE4 Dave Project manager FALSE5 Eve Data scientist FALSE In the output, the has_data_analyst column will be TRUE for “Bob” and FALSE for others.

Check for Partial StringLet’s expand our search to any string containing “Data”:

```

Detect if 'description' contains any word with 'Data'data$has_data <- str_detect(data$description, "Data")print(data)

```

name description has\_data\_analyst has\_data1 Alice Software developer FALSE FALSE2 Bob Data analyst TRUE TRUE3 Carol UX designer FALSE FALSE4 Dave Project manager FALSE FALSE5 Eve Data scientist FALSE TRUE This will show TRUE for “Bob” and “Eve,” where both “Data analyst” and “Data scientist” are detected.

Count OccurrencesIf you need to count how many times “Data” appears, use str_count:

```

Count occurrences of 'Data'data$data_count <- str_count(data$description, "Data")print(data)

```

name description has\_data\_analyst has\_data data\_count1 Alice Software developer FALSE FALSE 02 Bob Data analyst TRUE TRUE 13 Carol UX designer FALSE FALSE 04 Dave Project manager FALSE FALSE 05 Eve Data scientist FALSE TRUE 1 This will add a column data_count with the exact count of occurrences per row.

Using Base RFor those who prefer base R, the grepl and gregexpr functions can help.

Check for Full or Partial Stringgrepl is ideal for checking if a string is present:

```

Using grepl for full/partial string detectiondata$has_data_grepl <- grepl("Data", data$description)print(data)

```

name description has\_data\_analyst has\_data data\_count has\_data\_grepl1 Alice Software developer FALSE FALSE 0 FALSE2 Bob Data analyst TRUE TRUE 1 TRUE3 Carol UX designer FALSE FALSE 0 FALSE4 Dave Project manager FALSE FALSE 0 FALSE5 Eve Data scientist FALSE TRUE 1 TRUE This will yield the same output as str_detect.

Count OccurrencesFor counting occurrences, gregexpr is helpful:

```

Count occurrences using gregexprmatches <- gregexpr("Data", data$description)data$data_count_base <- sapply( matches, function(x) ifelse(x[1] == -1, 0, length(x)) )print(data)

```

name description has\_data\_analyst has\_data data\_count has\_data\_grepl1 Alice Software developer FALSE FALSE 0 FALSE2 Bob Data analyst TRUE TRUE 1 TRUE3 Carol UX designer FALSE FALSE 0 FALSE4 Dave Project manager FALSE FALSE 0 FALSE5 Eve Data scientist FALSE TRUE 1 TRUE data\_count\_base1 02 13 04 05 1 This will add a new data_count_base column containing the count of “Data” in each row.

Give It a Try!The best way to master string detection in R is to experiment with different patterns and datasets. Whether you use str_detect, grepl, or any other approach, you’ll find plenty of ways to customize the search. Try it out with your own datasets, and soon you’ll be searching like a pro!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Check if a Column Contains a String in R

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We wanted to say a huge thank you to everybody who attended SatRdays London 2024! It was brilliant to see you all there, and we hope you enjoyed the day as much as we did. Thank you to all of our speakers for your contributions, it was great to see such a range of talks and hear about the different ways you can use R in your fields.

Of course the day wouldn’t have been the same without our generous sponsors, so we want to say a huge thank you to CUSP London for providing the excellent venue, as well as R Consortium for your generous support.

Couldn’t make it on the day? Keep your eyes peeled on our blog and social media, as we’ll be releasing recordings of the talks on our YouTube channel in the coming months. Can’t wait that long? Check out last year’s SatRdays London recordings as well as those from our Shiny in Production conference from 2022 and 2023.

What’s next?Registration is now open for Shiny in Production 2024! This event consists of an afternoon of Shiny based workshops including:

Level Up Your Plots with Cara Thompson
Building Responsive Shiny Apps
Asynchronous Shiny
Building Apps for Humans

Followed by a day of talks from Shiny experts across a variety of industries. If you’re interested in submitting an abstract, head over to the conference website now. Submissions are open until 30th June!

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: SatRdays London 2024: Thanks for coming!

View Details

[This article was first published on Dr. Atakan Ekiz, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Image by Joshua Woroniecki (Pixabay)Many statistical analysis packages in R utilize design matrices for setting up comparisons between data subsets. The two Bioconductor packages most commonly used for transcriptomics data analysis, DESeq2 and limma, are no exception. Although the design matrices and contrasts are intuitive to understand for simple cases, things can get confusing when more complex multi-factorial studies are involved. You can find dozens of questions asked on Biostars and Bioconductor forums seeking for help in this matter.

To be fair, the DESeq2 and limma vignettes have dedicated sections explaining designs and contrasts, but I found these not very easy to follow the first time I saw them. In this note-to-self (and to-my-students) post, I intend to explain how to construct designs in various study contexts and access specific comparisons of interest using DESeq2. I benefited greatly from the DESeq2 vignette itself and the tutorial written by Hugo Tavares. The visuals embedded in this post are adapted from the accompanying slide show to Hugo Tavarez’s tutorial.

I will start with the simplest case where there are only two groups to be compared (i.e. one factor, two levels). Here, I will discuss different ways of constructing the design matrix and accessing the results. Then, I will exemplify a scenario with three groups (i.e. one factor, three levels) apply the approaches we discussed in the first case. Lastly, we will see an example involving 4 groups (i.e. two factors, two levels each). So, here goes nothin’…

1 One experimental condition, two groupsThis is a classic case where you might have a treatment group and a control group, and you are interested in figuring out how the treatment affects the gene expression. This type of data is referred to in vignettes as “one factor two levels”.

1.1 Simulate dataCode ```

for reproducibilityset.seed(1)suppressPackageStartupMessages(library(DESeq2))# simulate datadds1 <- makeExampleDESeqDataSet(n = 1000, m = 6, betaSD = 2)dds1$condition <- factor(rep(c("treatment", "control"), each = 3))# observe the samples and groupscolData(dds1)

```

DataFrame with 6 rows and 1 column condition <factor>sample1 treatmentsample2 treatmentsample3 treatmentsample4 control sample5 control sample6 control 1.2 Contrasts from design with interceptWhen we use the formula ~ condition the model matrix is set up with an intercept, that is, the first level of the condition is considered as the reference group. Since R orders factor levels based on the alphabetical order, in our case, control will be the reference. The order can be explicitly changed using relevel() or factor(..., levels=c(...)), if needed.

Here, the gene expression is modeled as seen below where is the intercept and is the coefficient for the treatment’s effect:

We can directly get the results we are interested in using the contrasts argument as seen below. contrasts argument can accept a number of inputs:

A character vector with exactly three elements: the name of a factor in design formula (eg. “condition” in our case), the name of the numerator (ie. group of interest, “treatment” in our case), and the name of the denominator (ie. reference group, “control” in our case). This would look like this for our example: c("condition", "treatment", "control"))
A list of two character vectors: the names of the fold changes for the numerator, and the names of fold changes for the denominator. This approach allows you to group comparisons together in the numerator and denominator for extracting specific comparisons (more below). In our case we only have one comparison, which is condition_treatment_vs_control, so we will provide it alone as list(condition_treatment_vs_control). Note that these names came from the colData in dds object automatically.

Code ```

design with intercept (control group is the intercept, or the reference)design(dds1) <- ~ condition# note the zeros for under treatment for samples 4-6model.matrix(~condition, colData(dds1))

```

(Intercept) conditiontreatmentsample1 1 1sample2 1 1sample3 1 1sample4 1 0sample5 1 0sample6 1 0attr(,"assign")[1] 0 1attr(,"contrasts")attr(,"contrasts")$condition[1] "contr.treatment" Code ```

differential expression analysisdds1 <- DESeq(dds1)# see the comparisonsresultsNames(dds1)

```

[1] "Intercept" "condition\_treatment\_vs\_control" Code res <- results(dds1, contrast = list("condition\_treatment\_vs\_control"))# res <- results(dds1, contrast = c("condition", "treatment", "control")) # alternatively# res 1.3 Contrasts from design without interceptSometimes it is easier to work with no-reference designs. You can tell the algorithm not to have an intercept (ie. control group) by using ~ 0 + condition formula. The way you specify the contrast argument will change, but the results will be identical with the first case. You may not see the benefit of using this approach in this simple example, but when multiple factors and factor levels are involved, it may be easier to construct no-intercept designs.

This is how a zero-intercept model looks like:

NOTE: When you have two factors in your data and use no-intercept design, you will have no reference group for the first factor (as seen here), but for the second factor a reference group is still defined. Although this is sensible, I find it somewhat confusing when it comes to specifying contrasts for specific pairwise comparisons. But the next section will help out with this.

Code dds2 <- dds1# design without intercept (no reference)design(dds2) <- ~ 0 + condition# note the zeros for under treatment for samples 4-6model.matrix(~0+condition, colData(dds2))

conditioncontrol conditiontreatmentsample1 0 1sample2 0 1sample3 0 1sample4 1 0sample5 1 0sample6 1 0attr(,"assign")[1] 1 1attr(,"contrasts")attr(,"contrasts")$condition[1] "contr.treatment" Code ```

differential expression analysisdds2 <- DESeq(dds2)# see the comparisonsresultsNames(dds2)

```

[1] "conditioncontrol" "conditiontreatment" Code res <- results(dds2, contrast = list("conditiontreatment", "conditioncontrol"))# res Above you see that we are providing list("conditiontreatment", "conditioncontrol") to contrasts argument. This way, we are telling the algorithm to take fold changes of differentially expressed genes in the conditiontreatment (differential expression is measured against the hypothetical zero) and divide that by the log fold changes of conditioncontrol. Thus you get the treatment vs control:

The results should be equivalent between the first the second design. Conceptually, the zero intercept model in DESeq2 does not compare conditions against each other, instead, each condition’s coefficient represents its own log2-fold expression estimate relative to zero.

1.4 A new approach for defining contrastsIt was easy to specify the contrasts in the examples above, but Mike Love and Hugo Tavares shares an interesting way of generalizing this concept, which is especially useful for more complex designs. This approach works for designs with or without an intercept. This approach involves three steps:

Getting the model matrix
Subsetting the matrix for each group of interest and calculate its column means - this results in a vector of coefficients for each group
Subtracting the group vectors from each other according to the comparison we’re interested in

Code ```

get the model matrixmod_mat <- model.matrix(design(dds1), colData(dds1))mod_mat

```

(Intercept) conditiontreatmentsample1 1 1sample2 1 1sample3 1 1sample4 1 0sample5 1 0sample6 1 0attr(,"assign")[1] 0 1attr(,"contrasts")attr(,"contrasts")$condition[1] "contr.treatment" Code control <- colMeans(mod\_mat[dds1$condition == "control", ])control

(Intercept) conditiontreatment 1 0 Code treatment <- colMeans(mod\_mat[dds1$condition == "treatment", ])treatment

(Intercept) conditiontreatment 1 1 Code ```

The comparison of interesttreatment - control

```

(Intercept) conditiontreatment 0 1 Code res <- results(dds1, contrast = treatment - control) The code may look overly complex to get a simple numeric vector to specify groups to compare. In this example, it is maybe so. But when there is a more complex experimental setup and you are comparing various subgroups, this can be helpful. Hugo Tavares created a function in a GitHub Gist to facilitate contrasts. But here I wanted to improve on that to avoid creation of unneccessary objects in the global environment and allow more complex contrasts.

The contraster() function below allows the user to extract contrasts from the DESeq object after specifying custom groups. group1 and group2 arguments can take lists of character vectors each having 2 or more items. The first item of the character vector should point to one of the columns in the colData(dds), that is the grouping variable. The second item of the character vector (and onwards) should be the sample subgroups to be included in the analysis. The differences are calculating subtracting group2 from group1. Read on for a discussion on the weighted argument.

Code contraster <- function(dds, # should contain colData and design group1, # list of character vectors each with 2 or more items group2, # list of character vectors each with 2 or more items weighted = F){ mod\_mat <- model.matrix(design(dds), colData(dds)) grp1\_rows <- list() grp2\_rows <- list() for(i in 1:length(group1)){ grp1\_rows[[i]] <- colData(dds)[[group1[[i]][1]]] %in% group1[[i]][2:length(group1[[i]])] } for(i in 1:length(group2)){ grp2\_rows[[i]] <- colData(dds)[[group2[[i]][1]]] %in% group2[[i]][2:length(group2[[i]])] } grp1\_rows <- Reduce(function(x, y) x & y, grp1\_rows) grp2\_rows <- Reduce(function(x, y) x & y, grp2\_rows) mod\_mat1 <- mod\_mat[grp1\_rows, ,drop=F] mod\_mat2 <- mod\_mat[grp2\_rows, ,drop=F] if(!weighted){ mod\_mat1 <- mod\_mat1[!duplicated(mod\_mat1),,drop=F] mod\_mat2 <- mod\_mat2[!duplicated(mod\_mat2),,drop=F] } return(colMeans(mod\_mat1)-colMeans(mod\_mat2)) } We can use this new function to get the exact results from before:

Code res <- results(dds1, contrast = contraster(dds1, group1 = list(c("condition", "treatment")), group2 = list(c("condition", "control")))) So far we have learned a simple two-group analysis. We will next use these in more complex experimental contexts.

2 One experimental condition, three levelsThis is a classic case when you have a control group and two treatment groups (e.g. treatmentA and treatmentB). From now on, I won’t be Let’s see how things work.

2.1 Simulate dataIn this design the gene expression is modeled as follows where is the intercept, is the coefficient (effect) of treatmentA, and the is the coefficient of treatmentB:

Code ```

simulate datadds3 <- makeExampleDESeqDataSet(n = 1000, m = 9, betaSD = 2)dds3$condition <- factor(rep(c("control", "treatmentA", "treatmentB"), each = 3))design(dds3) <- ~ conditiondds3 <- DESeq(dds3)# check the coefficients estimated by DEseqresultsNames(dds3)

```

[1] "Intercept" "condition\_treatmentA\_vs\_control"[3] "condition\_treatmentB\_vs\_control" 2.2 Getting contrastsNow we have three coefficients: 1) control group as intercept, 2) TreatmentA vs control, 3) TreatmentB vs control. We can get different results as follows:

Code ```

treatmentA vs controlres1 <- results(dds3, contrast = list("condition_treatmentA_vs_control"))# treatmentB vs controlres2 <- results(dds3, contrast = list("condition_treatmentB_vs_control"))# treatmentA vs treatmentBres3 <- results(dds3, contrast= list("condition_treatmentA_vs_control", "condition_treatmentB_vs_control"))

``` Or we can alternatively use our new function as well:

Code ```

Alternatively we can use our new functionres1 <- results(dds3, contrast = contraster(dds3, group1 = list(c("condition","treatmentA")), group2 = list(c("condition", "control"))))res2 <- results(dds3, contrast = contraster(dds3, group1 = list(c("condition","treatmentB")), group2 = list(c("condition", "control"))))res3 <- results(dds3, contrast = contraster(dds3, group1 = list(c("condition","treatmentA")), group2 = list(c("condition", "treatmentB"))))

`` This function also allows us to perform more customized analyses. For instance, if you are trying to compare treatment groups as a whole (treatmentA and treatmentB combined) against reference, you would need to writegroup1 = list(c("condition", "treatmentA", "treatmentB"))andgroup2 = list(c("condition", "control")`. The analysis will be performed by subtracting group2 from group1, that is you are comparing the average of the treatment groups to the control.

Code res4 <- results(dds3, contrast = contraster(dds3, group1 = list(c("condition", "treatmentA", "treatmentB")), group2 = list(c("condition", "control")))) Let’s see how the contrast is set up in this approach. As you can see below, each treatment group has a weight of 0.5, that is, their average is being compared against the control group.

Code contraster(dds3, group1 = list(c("condition", "treatmentA", "treatmentB")), group2 = list(c("condition", "control")))

(Intercept) conditiontreatmentA conditiontreatmentB 0.0 0.5 0.5 If the groups were unbalanced, by setting weighted = TRUE, you can assign weights according to group sizes. However, there are GitHub issues on the matter (1, 2) pointing out that this approach may be flawed where replicates don’t occur in all subgroups and numbers of replicates are different. You can see the behavior of contraster function by creating an unbalanced data set:

Code ```

Convert the last control to treatmentAdds3$condition[3] <- "treatmentA"contraster(dds3, group1 = list(c("condition", "treatmentA", "treatmentB")), group2 = list(c("condition", "control")), weighted = TRUE)

```

(Intercept) conditiontreatmentA conditiontreatmentB 0.0000000 0.5714286 0.4285714 3 Two factors with interactionIn some experiments grouping variables may interact with one another. An example for this can be that a treatment affects males and females differently. In this case we are looking at something like this:

3.1 Simulate dataCode dds4 <- makeExampleDESeqDataSet(n = 1000, m = 12, betaSD = 2)dds4$sex <- factor(rep(c("female", "male"), each = 6))dds4$condition <- factor(rep(c("treatment", "control"), 6))dds4 <- dds4[, order(dds4$sex, dds4$condition)]colnames(dds4) <- paste0("sample", 1:ncol(dds4))design(dds4) <- ~ sex + condition + sex:conditiondds4 <- DESeq(dds4)resultsNames(dds4)

[1] "Intercept" "sex\_male\_vs\_female" [3] "condition\_treatment\_vs\_control" "sexmale.conditiontreatment" 3.2 Getting contrasts3.2.1 Male vs female (in the control):Code res1 <- results(dds4, contrast = contraster(dds4, group1 = list(c("sex", "male"), c("condition", "control")), group2 = list(c("sex", "female"), c("condition", "control"))))# or equivalentlyres2 <- results(dds4, contrast = list("sex\_male\_vs\_female"))# head(res1,3); head(res2, 3) 3.2.2 Male vs female (in the treatment):Code res1 <- results(dds4, contrast = contraster(dds4, group1 = list(c("sex", "male"), c("condition", "treatment")), group2 = list(c("sex", "female"), c("condition", "treatment"))))# or equivalentlyres2 <- results(dds4, contrast = list(c("sex\_male\_vs\_female", "sexmale.conditiontreatment")))# head(res1,3); head(res2, 3) 3.2.3 Treatment vs control (for females):Code res1 <- results(dds4, contrast = contraster(dds4, group1 = list(c("sex", "female"), c("condition", "treatment")), group2 = list(c("sex", "female"), c("condition", "control"))))# or equivalentlyres2 <- results(dds4, contrast = list(c("condition\_treatment\_vs\_control")))# head(res1,3); head(res2, 3) 3.2.4 Treatment vs control (for males):Code res1 <- results(dds4, contrast = contraster(dds4, group1 = list(c("sex", "male"), c("condition", "treatment")), group2 = list(c("sex", "male"), c("condition", "control"))))# or equivalentlyres2 <- results(dds4, contrast = list(c("condition\_treatment\_vs\_control", "sexmale.conditiontreatment")))head(res1,3); head(res2, 3)

log2 fold change (MLE): 0,0,+1,+1 Wald test p-value: 0,0,+1,+1 DataFrame with 3 rows and 6 columns baseMean log2FoldChange lfcSE stat pvalue padj <numeric> <numeric> <numeric> <numeric> <numeric> <numeric>gene1 0.381213 -1.489653 4.130413 -0.360655 0.718358 0.999452gene2 17.028473 -0.271556 0.771766 -0.351863 0.724941 0.999452gene3 84.349943 0.205398 0.685823 0.299491 0.764566 0.999452

log2 fold change (MLE): condition\_treatment\_vs\_control+sexmale.conditiontreatment effect Wald test p-value: condition\_treatment\_vs\_control+sexmale.conditiontreatment effect DataFrame with 3 rows and 6 columns baseMean log2FoldChange lfcSE stat pvalue padj <numeric> <numeric> <numeric> <numeric> <numeric> <numeric>gene1 0.381213 -1.489653 4.130413 -0.360655 0.718358 0.999452gene2 17.028473 -0.271556 0.771766 -0.351863 0.724941 0.999452gene3 84.349943 0.205398 0.685823 0.299491 0.764566 0.999452 3.2.5 Interaction between sex and condition (i.e. do males and females respond differently to the treatment?)This will be a little more wordy, because we are trying to get the “difference of the differences”. For this analysis, we are interested in genes that are differentially expressed in males in treatment condition, but not in the control condition (ie. genes uniquely changing in males due to treatment). In other words, we are isolating by doing the following:

Code ```

male_vs_female in the treatment conditiondiff1 <- contraster(dds4, group1 = list(c("sex", "male"), c("condition", "treatment")), group2 = list(c("sex", "female"), c("condition", "treatment")))# male_vs_female in the control conditiondiff2 <- contraster(dds4, group1 = list(c("sex", "male"), c("condition", "control")), group2 = list(c("sex", "female"), c("condition", "control"))) res1 <- results(dds4, contrast = diff1-diff2)# or equivalentlyres2 <- results(dds4, contrast = list("sexmale.conditiontreatment"))# head(res1,3)# head(res2, 3)

``` 4 Multiple factors multiple levelsBelow is a more complex example involving an experimental condition (control, treatmentA, treatmentB), two sexes, and two batches.

4.1 Simulate dataCode ```

simulate datadds <- makeExampleDESeqDataSet(n = 1000, m = 24, betaSD = 2)dds$condition <- factor(rep(c("control", "treatmentA", "treatmentB"), each = 8))dds$sex <- factor(rep(c("male","male","male", "female", "female", "female"), 4))dds$batch <- factor(rep(c(1,2), 12))design(dds) <- ~ condition + sex + batchdds <- DESeq(dds)

```

estimating size factors

estimating dispersions

gene-wise dispersion estimates

mean-dispersion relationship

final dispersion estimates

fitting model and testing Code ```

check the coefficients estimated by DEseqresultsNames(dds)

```

[1] "Intercept" "condition\_treatmentA\_vs\_control"[3] "condition\_treatmentB\_vs\_control" "sex\_male\_vs\_female" [5] "batch\_2\_vs\_1" Code contraster(dds, group1=list(c("batch", 2), c("sex", "male"), c("condition", "treatmentB")), group2=list(c("batch", 1), c("sex", "female"), c("condition", "treatmentA")))

(Intercept) conditiontreatmentA conditiontreatmentB sexmale 0 -1 1 1 batch2 1 Code contraster(dds, group1=list(c("batch", 1), c("sex", "male"), c("condition", "treatmentB", "treatmentA")), group2=list(c("batch", 1), c("sex", "male"), c("condition", "control")))

(Intercept) conditiontreatmentA conditiontreatmentB sexmale 0.0 0.5 0.5 0.0 batch2 0.0 Code contraster(dds, group1=list(c("batch", 1), c("sex", "male"), c("condition", "treatmentB", "treatmentA")), group2=list(c("batch", 1), c("sex", "male"), c("condition", "control")), weighted = T)

(Intercept) conditiontreatmentA conditiontreatmentB sexmale 0.0 0.6 0.4 0.0 batch2 0.0 5 Further reading* https://www.biostars.org/p/395926/ * https://genomicsclass.github.io/book/pages/expressing_design_formula.html * https://genomicsclass.github.io/book/pages/interactions_and_contrasts.html * https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7873980/ * https://bioconductor.org/packages/release/workflows/vignettes/RNAseq123/inst/doc/designmatrices.html

Back to top To leave a comment for the author, please follow the link and comment on their blog: Dr. Atakan Ekiz.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A guide to designs and contrasts in DESeq2

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In the intricate world of data analysis, the task of text pattern recognition and extraction is akin to unlocking a secret cipher hidden within ancient manuscripts. This is the realm of regular expressions (regex), a powerful yet often underappreciated tool in the data scientist’s toolkit. Much like the cryptex from Dan Brown’s “The Da Vinci Code,” which holds the key to unraveling historical and cryptic puzzles, regular expressions unlock the patterns embedded in strings of text data.

However, the power of regex comes at a cost — its syntax is notoriously complex and can be as enigmatic as the riddles solved by Robert Langdon in his thrilling adventures. For those not versed in its arcane symbols, crafting regex patterns can feel like deciphering a code without a Rosetta Stone. This is where the rebus package in R provides a lifeline. It simplifies the creation of regex expressions, transforming them from a cryptic sequence of characters into a readable and manageable code, akin to translating a hidden message in an old relic.

In this tutorial, we embark on a journey akin to that of Langdon’s through Paris and London, but instead of ancient symbols hidden in art, we’ll navigate through the complexities of text data. We will explore the fundamental principles of regex that form the backbone of text manipulation tasks. From basic pattern matching to crafting intricate regex expressions with the rebus package, this guide will illuminate the path towards mastering regex in R, making the process as engaging as uncovering a secret passage in an ancient temple.

Just as Langdon used his knowledge of symbolism to solve mysteries, we will use rebus to demystify regex in R, making this powerful tool accessible and practical for everyday data tasks. Whether you're a seasoned data scientist or a novice in the field, understanding how to effectively use regex is like discovering a hidden map that leads to buried treasure, providing you with the insights necessary to make informed decisions based on your data.

With our thematic setting now established, let us delve deeper into the world of regular expressions and reveal how the rebus package can transform your approach to data analysis, turning a daunting task into an intriguing puzzle-solving adventure.

In the quest for understanding regular expressions, akin to decoding a series of cryptic messages left in Leonardo Da Vinci’s artworks, we start with the very basics — the symbols and syntax that are the foundational tools of this powerful scripting language. Just as symbols held profound meanings in ancient scripts, each character in a regex pattern holds specific and significant implications.

Unveiling the SymbolsRegular expressions operate through special characters that, when combined, form patterns capable of matching and extracting text with incredible precision. Here are a few fundamental symbols to understand:

Dot (.): Like the omnipresent eye in a Da Vinci painting, the dot matches any single character, except newline characters. It sees all but the end of a line.
Asterisk (*): Mirroring the endless loops in a Fibonacci spiral, the asterisk matches the preceding element zero or more times, extending its reach across the string.
Plus (+): This symbol requires the preceding element to appear at least once, much like insisting on the presence of a key motif in an artwork.
Question Mark (?): It makes the preceding element optional, introducing ambiguity into the pattern, akin to an unclear symbol whose meaning might vary.
Caret (^): Matching the start of a string, the caret sets the stage much like the opening scene in a historical mystery.
Dollar Sign ($): This symbol matches the end of a string, providing closure and ensuring that the pattern adheres strictly to the end of the text.

Example: Simple Patterns in ActionUsing the stringr library enhances readability and flexibility in handling regular expressions. Let’s apply this to find specific patterns:

library(stringr)text\_vector <- c("Secrets are hidden within.", "The key is under the mat.", "Look inside, find the truth.", "Bridge is damaged by the storm")str\_detect(text\_vector, "\\bis\\b")str\_detect(text\_vector, "\\bis\\b"[1] FALSE TRUE FALSE TRUE This code chunk check if word “is” is anywhere in given sentenece.

Crafting Your First RegexTo identify any word that ends with ‘ed’, signaling past actions, akin to uncovering traces of events long gone:

```

Match words ending with 'ed'str_extract(text_vector, "\b[A-Za-z]+ed\b")[1] NA NA NA "damaged"

``` This expression uses \b to ensure that 'ed' is at the end of the word, capturing complete words and not fragments—critical when every detail in a coded message matters.

Deciphering a Complex RegexLet’s consider a more intricate regex pattern:

date\_pattern <- "\\b(0[1-9]|[12][0-9]|3[01])[- /.](0[1-9]|1[012])[- /.](19|20)\\d\\d\\b"# first check if pattern is presentstr\_detect("She was born on 12/08/1993, and he on 04/07/1989.", date\_pattern)[1] TRUE# second extract the patternstr\_extract\_all("She was born on 12/08/1993, and he on 04/07/1989.", date\_pattern)[[1]][1] "12/08/1993" "04/07/1989" This regex looks extremely unfriendly at first glance, resembling an arcane code more than a helpful tool. It uses capturing groups, ranges, and alternations to accurately match dates in a specific format. Here’s the breakdown:

\b: Word boundary, ensuring we match whole dates.
(0[1–9]|[12][0–9]|3[01]): Matches days from 01 to 31.
[- /.]: Matches separators which can be a dash, space, dot, or slash.
(0[1–9]|1[012]): Matches months from 01 to 12.
(19|20)\d\d: Matches years from 1900 to 2099.

This example shows how raw regex can quickly become complex and hard to follow, much like a cryptic puzzle waiting to be solved. The rebus package can help simplify these expressions, making them more accessible and easier to manage.

Building Blocks of RebusJust as Robert Langdon in “The Da Vinci Code” used his knowledge of symbology to decode complex historical puzzles, the rebus package in R enables us to build regular expressions from understandable components, transforming arcane syntax into legible code. This approach not only simplifies regex creation but also enhances readability and maintenance, making regex patterns as approachable as reading a museum guidebook.

Assembling the CodexRebus operates on the principle of constructing regex patterns piece by piece using function calls, which represent different regex components. This method aligns with piecing together clues from a scattered array of symbols to form a coherent understanding. Here are some of the building blocks provided by rebus:

digit(): Matches any number, simplifying digit recognition.
or(): Specifies a set of characters to match, allowing customization akin to selecting specific tools for a dig site.

Example: Email Pattern Construction with RebusCrafting an email validation pattern with rebus is akin to assembling a puzzle where each piece must fit precisely:

library(rebus)# Define the pattern for a standard emailemail\_pattern <- START %R% one\_or\_more(WRD) %R% "@" %R% one\_or\_more(WRD) %R% DOT %R% or("com", "org", "net")# Use the pattern to find valid emailssample\_text <- c("contact@example.com", "hello@world.net", "not-an-email")str\_detect(sample\_text, email\_pattern)# short dictionary%R% - special rebus operator for concatenation of elementsSTART - equivalent of ^END - equivalent of $WRD - treats characters as words similar to \b in RegexDOT - is a dot, I think nobody was confused here :D This pattern, built with rebus functions, makes it easy to understand at a glance which components form the email structure, demystifying the regex pattern much like Langdon revealing the secrets behind a hidden inscription.

Deciphering Complex Text Patterns with RebusConsider a more complicated scenario where you need to validate date formats within a text. Using basic regex might involve a lengthy and cryptic pattern, but with rebus, we can construct it step-by-step:

```

Define a pattern for dates in the format DD/MM/YYYYdate_pattern <- digit(2) %R% "/" %R% digit(2) %R% "/" %R% digit(4) # Sample text for pattern matchingdates_text <- "Important dates are 01/01/2020 and 31/12/2020."# First check if pattern can be found in text.str_detect(dates_text, date_pattern)[1] TRUE# Then what it extracts.str_extract_all(dates_text, date_pattern)[[1]][1] "01/01/2020" "31/12/2020"# But it will find also not proper dates like 32/13/2210 etc.

``` This example shows how rebus simplifies complex regex tasks, turning them into a series of logical steps, much like solving a riddle in an ancient tome.

But wait a minute… It is always a good idea to dig in documentation, and check out what can be found there.

dmy\_pattern = DMYstr\_detect(dates\_text, dmy\_pattern)[1] TRUEstr\_extract\_all(dates\_text, dmy\_pattern)[[1]][1] "01/01/2020" "31/12/2020"# even shorter, even nicer, even faster Tips for Crafting Expressions with RebusWhile rebus makes it easier to create and understand regex patterns, there are tips to further enhance your mastery:

Start Simple: Begin with basic components and gradually add complexity.
Test Often: Use sample data to test and refine your patterns frequently.
Comment Your Code: Annotate your rebus expressions to explain the purpose of each component, especially in complex patterns.

Extracting Complex Medical Data from Clinical NotesIn the vein of a detective novel, akin to “The Da Vinci Code,” where each clue unravels a part of a larger mystery, this scenario involves deciphering clinical notes to extract specific medical information. This requires a keen understanding of the text’s structure and content, mirroring the precision needed to solve a cryptic puzzle left in an ancient artifact.

Setting the Scene: Medical Data Extraction ChallengeClinical notes are packed with crucial medical details in a format that is often not standardized, making the extraction of specific information like medication prescriptions and patient diagnoses a complex task. Our goal is to develop regex patterns that can accurately identify and extract this information from varied text formats.

Step-by-Step Pattern Construction Using RebusDefine Complex Patterns:* Medications often mentioned with dosages and frequencies. * Diagnoses that may include medical terms and conditions.

library(rebus)library(stringr)# Pattern for medication prescriptions# Example format: [Medication Name] [Dosage in mg] [Frequency]medication\_pattern <- one\_or\_more(WRD) %R% SPACE %R% one\_or\_more(DGT) %R% "mg" %R% SPACE %R% one\_or\_more(WRD)# Pattern for diagnoses# Example format: Diagnosed with [Condition]diagnosis\_pattern <- "Diagnosed with " %R% one\_or\_more(WRD %R% optional(SPACE %R% WRD))clinical\_notes <- c("Patient was prescribed Metformin 500mg twice daily for type 2 diabetes.", "Diagnosed with Chronic Heart Failure and hypertension.", "Amlodipine 10mg once daily was recommended.", "Review scheduled without any new prescriptions.") Sample Clinical Notes: clinical\_notes <- c("Patient was prescribed Metformin 500mg twice daily for type 2 diabetes.", "Diagnosed with Chronic Heart Failure and hypertension.", "Amlodipine 10mg once daily was recommended.", "Review scheduled without any new prescriptions.") Extract and Validate Medical Data: ```

Extracting medication detailsmedication_details <- str_extract_all(clinical_notes, medication_pattern)# Extracting diagnosesdiagnoses_found <- str_extract_all(clinical_notes, diagnosis_pattern)

``` Example: Advanced Code WalkthroughBy running the above patterns against the clinical notes, we extract structured information about medications and diagnoses:

print(medication\_details)[[1]][1] "Metformin 500mg twice"[[2]]character(0)[[3]][1] "Amlodipine 10mg once"[[4]]character(0)print(diagnoses\_found)[[1]]character(0)[[2]][1] "Diagnosed with Chronic Heart Failure and hypertension"[[3]]character(0)[[4]]character(0) This code extracts arrays containing detailed medication prescriptions and diagnosed conditions from each note, if available.

Handling Edge Cases and VariabilityMedical terms and prescriptions can vary greatly:

Expand Vocabulary in Rebus: Include variations and synonyms of medical conditions and medication names.
Adjust for Complex Dosage Instructions: Medications might have dosages described in different units or intervals.

Mastering Medical Data ExtractionJust as each puzzle piece in “The Da Vinci Code” led to deeper historical insights, each regex pattern crafted with rebus reveals vital medical information from clinical notes, enabling better patient management and data-driven decision-making in healthcare.

Mastering Regex with Rebus for Complex Data ExtractionNavigating through complex data with regex and the rebus package is akin to deciphering hidden codes and symbols in a Dan Brown novel. Just as Robert Langdon uses his knowledge of symbology to unravel mysteries in "The Da Vinci Code," data scientists and analysts use regex patterns crafted with rebus to unlock the mysteries within their data sets. This guide has shown how rebus transforms an intimidating script into a manageable and understandable set of building blocks, enabling precise data extraction across various domains, from legal documents to medical records.

Final Thoughts: The Art of Regex Crafting1. Iterative Development: Like solving a cryptic puzzle, developing effective regex patterns often requires an iterative approach. Start with a basic pattern, test it, refine it based on the outcomes, and gradually incorporate complexity as needed. 2. Comprehensive Testing: Ensure your regex patterns perform as expected across all possible scenarios. This includes testing with diverse data samples to cover all potential variations and edge cases, mirroring the meticulous verification of clues in a historical investigation. 3. Documentation and Comments: Regex patterns, especially complex ones, can quickly become inscrutable. Document your patterns and use comments within your rebus expressions to explain their purpose and structure. This practice ensures that your code remains accessible not just to you but to others who may work on it later, much like leaving a detailed map for those who follow in your footsteps. 4. Stay Updated: Just as new archaeological discoveries can change historical understandings, advancements in programming and new versions of packages like rebus can introduce more efficient ways to handle data. Keeping your skills and knowledge up to date is crucial. 5. Share Knowledge: Just as scholars share their discoveries and insights, sharing your challenges and solutions in regex with the community can help others. Participate in forums, write blogs, or give talks on your regex strategies and how you’ve used rebus to solve complex data extraction problems.

Strategies for Employing rebus Effectively Utilize rebus Libraries: Leverage the full suite of rebus functionalities by familiarizing yourself with all its helper functions and modules. Each function is designed to simplify a specific aspect of regex pattern creation, which can drastically reduce the complexity of your code. * Pattern Modularity: Build your regex patterns in modular chunks using rebus, similar to constructing a narrative or solving a multi-part puzzle. This approach not only simplifies the development and testing of regex patterns but also enhances readability and maintenance. * Advanced Matching Techniques:* For highly variable data, consider advanced regex features like lookaheads, lookbehinds, and conditional statements, which can be integrated into your rebus patterns. These features allow for more dynamic and flexible pattern matching, akin to adapting your hypothesis in light of new evidence.

Epilogue: The Power of Clarity in Data ParsingIn conclusion, mastering rebus and regex is like becoming fluent in a secret language that opens up vast archives of data, ready to be explored and understood. This guide has equipped you with the tools to start this journey, providing the means to reveal the stories hidden within complex datasets, enhance analytical accuracy, and drive insightful decisions.

Just as every clue solved brings Langdon closer to the truth in “The Da Vinci Code,” each pattern you decipher with rebus brings you closer to mastering the art of data. The path is laid out before you—begin your adventure, solve the puzzles, and unlock the potential of your data with confidence.

Appendix: The Regex Rosetta Stone — A Comprehensive Reference GuideThis appendix is designed as a quick yet comprehensive reference guide to using the rebus package for crafting regex expressions in R. Here you will find a brief description of some of the most pivotal functions, character classes, ready-made patterns, and interesting trivia on less commonly used regex features.

Most Common Functions in rebusLet’s explore some of the essential rebus functions that you can use to construct regex patterns more intuitively:
or(): Combines multiple patterns and matches any of them. Useful for alternatives in a pattern.
exactly(): Specifies that the preceding element should occur an exact number of times.
literal(): Treats the following string as literal text, escaping any special regex characters.
optional(): Indicates that the preceding element is optional, matching it zero or one time.
zero_or_more(): Matches zero or more occurrences of the preceding element.
one_or_more(): Matches one or more occurrences of the preceding element.
lookahead(): Checks for a match ahead of the current position without consuming characters.
lookbehind(): Asserts something to be true behind the current position in the text.
repeated(): Matches a specified number of repetitions of the preceding element.
whole_word(): Ensures that the pattern matches a complete word.
Most Common Character ClassesCharacter classes simplify the specification of a set of characters to match:
DGT (Digit): Matches any digit, shorthand for digit().
ALNUM (Alphanumeric): Matches any alphanumeric character.
LOWER: Matches any lowercase letter.
UPPER: Matches any uppercase letter.
SPECIALS: Matches any special characters typically found on a keyboard.
ROMAN: Matches Roman numerals.
PUNCT (Punctuation): Matches any punctuation character.
NOT_DGT (Not Digit): Matches any character that is not a digit.
HEX_DIGIT (Hexadecimal Digit): Matches hexadecimal digits (0-9, A-F).
KATAKANA, HIRAGANA: Matches characters from the Japanese Katakana and Hiragana scripts.
HEBREW, CYRILLIC, ARABIC: Matches characters from the Hebrew, Cyrillic, and Arabic scripts.
Ready Patternsrebus also includes functions for common pattern templates:
YMD: Matches dates in Year-Month-Day format.
TIME: Matches time in HH:MM:SS format.
AM_PM: Matches time qualifiers AM or PM.
CURRENCY_SYMBOLS: Matches common currency symbols.
HOUR12: Matches hour in 12-hour format.
Interesting But Less Used Character Classes (Trivia)Explore some unique and less commonly used character classes:
DOMINO_TILES: Matches Unicode representations of domino tiles.
PLAYING_CARDS: Matches Unicode characters representing playing cards.

These unique character classes add a fun and often surprising depth to regex capabilities, allowing for creative data parsing and matching scenarios, much like uncovering an unexpected twist in a puzzle or story.

By familiarizing yourself with these tools, you can significantly enhance your ability to analyze and manipulate data effectively, transforming complex text into structured and insightful information. Keep this guide handy as a reference to navigate the vast landscape of regex with confidence and precision.

Final Tip:If you haven’t already noted it, there is one small trick that will help you make step from using rebus to use “vanilla” regular expressions. When you place pattern in variable in your environment it is storing it as real RegExp, so if you would like to see it, and maybe use it directly in code, just print it to console.

```

Imagine that there is some official number that consists of following parts# Date in format YYYYMMDD, then letter T, then time in format HHMMSS and indicator AM or PM# Looks pretty simple, and indeed is using rebuspattern = YMD %R% "T" %R% HMS %R% AM_PM# Now look to raw RegExp version.print(pattern)[0-9]{1,4}[-/.:,\ ]?(?:0[1-9]|1[0-2])[-/.:,\ ]?(?:0[1-9]|[12][0-9]|3[01])T(?:[01][0-9]|2[0-3])[-/.:,\ ]?[0-5][0-9][-/.:,\ ]?(?:[0-5][0-9]|6[01])(?:am|AM|pm|PM)valid = "20180101T120000AM"str_detect(valid, pattern)#[1] TRUE

```

The Rebus Code: Unveiling the Secrets of Regex in R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Rebus Code: Unveiling the Secrets of Regex in R

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Appsilon’s core purpose is to advance technology to preserve and improve human life, and it translates into a unified set of values shared by the whole organization.

We realize this by creating and enhancing tools to solve the problems our clients are faced with, focusing on impact. We work with many leaders in the pharmaceutical and biotechnology sectors and take pride in helping them accelerate the discovery and provision of medicines.

With this in mind, we are all proud to announce that we are joining forces with the Pharmaverse Council, an influential organization dedicated to fostering collaborative, open-source software development for clinical reporting in the pharmaceutical industry.

Pharmaverse‘s mission is to streamline the drug development process by promoting open standards and interoperability between regulatory systems. By bringing together leading companies, academics, and regulatory bodies, the council facilitates collaboration and knowledge sharing to drive innovation in clinical data reporting.

A Shared Commitment to Open-Source SolutionsAt the heart of Pharmaverse is developing a curated open-source suite of R packages to enable the entire clinical reporting data pipeline, from case report forms through to regulatory submissions. This approach can reduce duplicated efforts across the pharmaceutical industry and provide a smoother experience for regulators reviewing submissions.

At Appsilon, we have long championed the open-source philosophy, believing that collective knowledge, diverse opinions, and transparency are essential for meaningful progress. Joining the Pharmaverse Council strengthens our commitment to creating accessible tools that benefit the entire pharmaceutical ecosystem.

We actively create, maintain, and contribute to numerous open-source projects, such as our package, Rhino (a part of Pharmaverse), which is useful for streamlining FDA submissions.

ConclusionWe’re proud and excited to be part of the Pharmaverse Council. In embracing the spirit of open source and collaboration, we are committed to adopting shared solutions that accelerate the development of technology for improved clinical reporting and drug discovery.

We invite the community to engage with us as we continue to share our progress and insights through our blog posts and open-source contributions. ‍

Subscribe to our newsletter for updates on this exciting journey, and check out Pharmaverse’s new newsletter as well!

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Appsilon Joins the Pharmaverse Council to Advance Open-Source Clinical Reporting

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhen working with data frames in R, you may often encounter scenarios where you need to collapse or concatenate text values based on groups within your dataset. This could involve combining text from multiple rows into a single row per group, which can be useful for summarizing data or preparing it for further analysis. In this post, we’ll explore how to achieve this task using different methods in R—specifically using base R, the dplyr package, and the data.table package.

Example DataLet’s start with an example dataset. Suppose we have a data frame df containing information about sales transactions:

```

Example data framedf <- data.frame( CustomerID = c(1, 1, 2, 2, 3), Product = c("Apple", "Orange", "Banana", "Peach", "Grapes"), Quantity = c(2, 3, 1, 2, 1), stringsAsFactors = FALSE)# Print the data frameprint(df)

```

CustomerID Product Quantity1 1 Apple 22 1 Orange 33 2 Banana 14 2 Peach 25 3 Grapes 1 ExamplesUsing Base RIn base R, you can use aggregate() to collapse text values by group. Let’s say we want to collapse the Product column by CustomerID:

```

Collapse text by CustomerID using base Rcollapsed_df <- aggregate(Product ~ CustomerID, data = df, FUN = function(x) paste(x, collapse = ", "))# Print the resultprint(collapsed_df)

```

CustomerID Product1 1 Apple, Orange2 2 Banana, Peach3 3 Grapes Here, we used aggregate() to group the Product column by CustomerID and applied a custom function to concatenate the text values separated by commas.

Using dplyrThe dplyr package provides a concise way to manipulate data frames. We can achieve the same result using dplyr’s group_by() and summarise() functions:

```

Load the dplyr packagelibrary(dplyr)# Collapse text by CustomerID using dplyrcollapsed_df <- df %>% group_by(CustomerID) %>% summarise(Product = paste(Product, collapse = ", "))# Print the resultprint(collapsed_df)

```

A tibble: 3 × 2 CustomerID Product 1 1 Apple, Orange2 2 Banana, Peach3 3 Grapes

`` Using data.tableFor larger datasets, thedata.tablepackage can offer efficient solutions. Here’s how you can collapse text by group usingdata.table`:

```

Load the data.table packagelibrary(data.table)# Convert data.frame to data.tablesetDT(df)# Collapse text by CustomerID using data.tablecollapsed_df <- df[, .(Product = paste(Product, collapse = ", ")), by = CustomerID]# Print the resultprint(collapsed_df)

```

CustomerID Product <num> <char>1: 1 Apple, Orange2: 2 Banana, Peach3: 3 Grapes ConclusionIn this blog post, we explored different methods to collapse text by group in a data frame using R. Whether you prefer the simplicity of base R, the readability of dplyr, or the efficiency of data.table, each approach allows you to perform this task effectively based on your preference and the size of your dataset.

I encourage you to try these examples with your own datasets and explore further customizations based on your specific needs. Manipulating data in R can be both powerful and intuitive, and mastering these techniques will enhance your data analysis capabilities.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Collapse Text by Group in a Data Frame Using R

View Details

[This article was first published on coding-the-past, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.1. What is a treemap?A treemap consists of a set of rectangles which represent different categories in your data and whose size is defined by a numeric value associated with the respective category. For example, a treemap could illustrate the continents on Earth, sized according to their population. For a deeper analysis, treemaps can include nested rectangles, that is, categories within categories. In our example, within each continent rectangle, new rectangles could represent countries and their populations.

When should you use a treemap?One of the main advantages of a treemap is that it allows for the interpretation of a large amount of data at a single glance. It is well-suited to show part-to-whole relationships and to highlight the hierarchies in your data. Do not use treemaps when the variable defining the size of rectangles presents little variation.

How to plot a treemap in R?To exemplify a treemap in R, we will use the Cholera dataset, which contains that on the mortality caused by cholera in England in the years 1848-1849. This data comes from the histdata R package. Moreover, you will need to install the treemap package, one of the alternatives to plot a treemap in R. We will also use RColorBrewer package for a color palette and dplyr to transform the data.

After you install the packages, load them and explore the structure of the Cholera data frame.

content_copy Copy

library(HistData)library(treemap)library(dplyr)library(RColorBrewer)# Load the data to your R environmentdata("Cholera")# Check the dataframe structurestr(Cholera) We would like to create a treemap in which we have bigger rectangles representing the regions of London and smaller rectangles representing the districts within their respective region. The size of the rectangles will inform us about the mortality caused by cholera in a given region and district. For us, the following variables are important:

region will define our outer rectangles (higher hierarchy) and will represent regions of London (West, North, Central, South, Kent);
district will define our inner rectangles (lower hierarchy), representing the districts of London;
cholera_drate represents deaths caused by cholera per 10,000 inhabitants in 1849 and will define the size of rectangles

tips_and_updates To learn more about the data, please see the package documentation here. The treemap function is used to plot the treemap in R. The main arguments necessary are:

the first argument is the dataframe;
index defines the two levels of hierarchy in our plot: region and district;
vSize specifies the death rate to define the size of our rectangles;
vColor specifies the region to define the color of our higher hierarchy rectangles;
type informs the function that vColor is a categorical variable;
the remaining parameters are used to adjust format options like color palette and position of elements.

tips_and_updates To further format your treemap, check more options in the package documentation.
content_copy Copy

treemap(Cholera, index=c("region","district"), vSize="cholera\_deaths", vColor = "region", type = "categorical", # formatting options: palette = brewer.pal(n = 5, name = "Accent"), align.labels=list( c("left", "top"), c("right", "bottom") ), border.col = "white", bg.labels = 255, position.legend = "none") Note that Kent is the region with the largest death rate, followed by Southern London. Moreover, districts like Lambeth and Bethnal Green were especially affected by the disease. This treemap is a powerful tool to give you a general picture of the data at first glance.

If you have any questions, please feel free to comment below!

Conclusions* A treemap is very useful to represent hierarchical relations in your data and provide a quick overall picture of your data;
Plotting a treemap in R can be easily accomplished with the treemap package.

To leave a comment for the author, please follow the link and comment on their blog: coding-the-past.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Treemaps In R

View Details

[This article was first published on R Programming Archives - Mark Niemann-Ross, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

R is a programming language for statistical computing and data visualization. It has been adopted in the fields of data mining, bioinformatics, and data analysis.

https://en.wikipedia.org/wiki/R_%28programming_language%29

Data mining, bioinformatics, data analysis…

…But not hardware.

We assume R isn’t useful for flipping switches and reading sensors. But that’s an assumption, not a fact.

Let’s be honest. Computer languages are just an abstraction layer on top of the metal. Languages provide constructs to make the expression of logic easier, tailored for the different ways people approach logic and data. But all languages eventually drive the behavior of transistors, and the logic gates and microcircuits built on top of those transistors.

We assume R doesn’t care about transistors. Possibly true, but I don’t believe that’s set in stone. In fact, I’ve proved it’s not true.

R is a language just like C or Python. Unlike C or MicroPython, R requires an operating system. It is a high-level language; memory management or assembler isn’t something we talk about at the R conventions. As written, it’s not well suited for base hardware such as Arduino, espy, or RP2040.

I’ve done assembler on a 6502; partly as a lark, partly because I only had 64k of memory. That’s no longer an issue with any computer in my office. Now I can focus on abstract issues like correlation and causation. I still need to interact with the real world and sometimes datasets need to be collected from buttons and sensors.

I could wire up a data acquisition tool using sensors and microprocessors then store data to some media. Or… I can use R to control the GPIO pins on a Raspberry Pi. The latter is cleaner, but I had to do two things to make it happen: Install R on a Raspberry Pi, and develop a package for R to communicate with the Raspberry Pi GPIO.

R for PiInstalling R on a Raspberry Pi is easy if you use R4Pi. These fine folks have compiled all the popular bits you’ll need. Download it, run the installer, and you’re done.

Alternatively… You can do it the hard way and install a Linux version of R from cran. Be prepared for a long wait while everything is downloaded and compiled. Snore…

rpigpiorRunning R on a Raspberry Pi is one thing. But the reason Raspberry Pi exists is the GPIO – General Purpose Input Output pins. These are wires – connections to servos, pushbuttons, lights and other electronic circuits. Real world stuff. Things that do stuff like control flow, lift weights, watch lights, feel for pushbuttons.

rpigpior is a package for R to interface with the Raspberry Pi GPIO.

Read a PushbuttonHere’s an example of using R to read a pushbutton on GPIO board pin 40. It’s pretty simple and doesn’t use any advanced R functions:

library(rpigpior)while (TRUE) { if (rpi\_get(40)) { print("Button pushed") } else { print("Button not pushed") }} Control a RelayThe Raspberry Pi is narrow-minded about voltages. It supplies either 3.3 volts d.c. or 5 volts d.c. Not 12 VDC. Certainly not a.c and especially not household 120 VAC.

I’m building an irrigation system with off-the-shelf water valves. I need 24 VAC to drive these valves and I need to control that voltage with the 3.3 VDC signal voltage supplied by the Raspberry Pi. Adafruit sells a relay for $6.95 that does exactly that. I ordered two, and in a week I had them connected and wrote R code to open and close valves.

```

running in Rlibrary(rpigpior)# turns on relay connected to board pin 11, GPIO17rpi_set(pin_number = 11, onOff = 1) # wait a bit...rpi_set(pin_number = 11, onOff = 0)

``` R for ReportingThe sprinkler system I’ve built in R checks the weather report, then decides how much to water the garden. It’s not excessively complex code, but it was simple to get the weather reports I needed and then do the calculations on how much water to provide. I’ve given it control of two irrigation valves which control the amount of water sent to the front and back yards. I also wanted to keep track of how the system was performing, so I wrote a dashboard using shiny.

There are more examples at the github repository.

Why Use RMaybe a better question is “Why Not Use R.” Granted, you can accomplish everything I’ve shown here in any other language. But I’m comfortable with R and if I can use it for the full cycle, that keeps me from having to invest extra time in learning an alternative. And I’m all about saving time!

What about you? Any thoughts on using R to control Hardware?

The post R Meets Hardware appeared first on Mark Niemann-Ross.

To leave a comment for the author, please follow the link and comment on their blog: R Programming Archives - Mark Niemann-Ross.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Meets Hardware

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#179–180

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #179I must admit that I am not a big fan of baseball, but it doesn’t stop me before this challenge. As usual we have to transform table little bit, and make some kind of summary. So let bat swings.

Loading data and libraries library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_179.xlsx", range = "A1:C10")test = read\_excel("Power Query/PQ\_Challenge\_179.xlsx", range = "E1:K4") Transformation r1 = input %>% select(-`Runs Scored`) %>% mutate(player = paste0("Player",row\_number()), .by = Team) %>% pivot\_wider(names\_from = player, values\_from = Player)r2 = input %>% mutate(max = max(`Runs Scored`), .by = Team) %>% filter(`Runs Scored` == max) %>% summarise(`Highest Scoring Player` = paste0(Player, collapse = ", "), `Highest Score` = unique(`Runs Scored`), .by = Team)result = r1 %>% left\_join(r2, by = "Team") Validation all.equal(result, test, check.attributes = FALSE)# [1] TRUE Puzzle #180This time we need to find out who and when has the largest Month over Month difference in sales (both negative or positive). So we need to use some lags here. Find out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_180.xlsx", range = "A1:B28")test = read\_excel("Power Query/PQ\_Challenge\_180.xlsx", range = "D1:G4") Transformation result = input %>% mutate(Emp = ifelse(is.na(Sales), `Emp-Month`, NA\_character\_)) %>% fill(Emp) %>% filter(!is.na(Sales)) %>% mutate(lag\_sales = lag(Sales, 1, default = 0), lag\_month = lag(`Emp-Month`, 1, default = ""), total = sum(Sales), change = abs(lag\_sales - Sales), max\_change = max(change), .by = Emp) %>% filter(change == max\_change) %>% select(Emp, `Total Sales` = total, `Max Sales Change` = max\_change, lag\_month, `Emp-Month`) %>% unite("From - To Months", lag\_month, `Emp-Month`, sep = " - ") Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The Turkish Banking sector has always been considered the decisive indicator of the Turkish economy. After the transition to the Orthodox monetary policies, the Central Bank’s interest rate has been rising to 50% in the short term. This has instituted confidence, especially for foreign investors.

This confidence has lifted the banking index (XBANK) year to date. I wonder about the remainder of the year. So, we will predict the next 8 months of the index in this article. The explanatory variable we are going to use is one-week repo rates in Turkey.

library(tidyverse)library(tidymodels)library(tidyquant)library(timetk)library(readxl)#XBANK index datadf\_xbank <- tq\_get("XBANK.IS", to = "2024-05-01") %>% tq\_transmute(select = close, mutate\_fun = to.monthly, col\_rename = "xbank") %>% mutate(date = as.Date(date)) #TCMB(Central Bank of the Republic of Turkey) One-Week Repo Ratedf\_tcmb\_rates <- read\_csv("https://raw.githubusercontent.com/mesdi/blog/main/tcmb\_repo.csv") %>% janitor::clean\_names() %>% select(date = release\_date, tcmb\_rates = actual) %>% mutate(date = #removing parentheses and the text within case\_when(str\_detect(date," \$.*\$") ~ str\_remove(date," \$.*\$"), TRUE ~ date) %>% parse\_date(date, format = "%b %d, %Y") %>% #adding one month floor\_date("month") %m+% months(1), tcmb\_rates = str\_remove(tcmb\_rates, "%") %>% as.numeric()) %>% #makes regular time series by filling the time gaps pad\_by\_time(date, .by = "month") %>% tidyr::fill(tcmb\_rates, .direction = "up") %>% #mutate(across(tcmb\_rates, .fns = \(x) ts\_impute\_vec(x, period = 1))) %>% drop\_na()#Merging all the datasetsdf\_merged <- df\_tcmb\_rates %>% left\_join(df\_xbank) %>% drop\_na() Once we build our data set we can pass the modeling phase. We will use many models to find the best suit for the data. To do that we will tune and evaluate all the models to their related parameters and rank the accuracy results in a table.

```

Modelinglibrary(tidymodels)library(modeltime)#Split Data library(timetk)split <- df_merged %>% time_series_split(assess = "1 year", cumulative = TRUE)df_train <- training(split)df_test <- testing(split)#Bootstraping for tuningset.seed(12345)df_folds <- bootstraps(df_train, strata = xbank, times = 100)#Model 1: auto_arima #(https://business-science.github.io/modeltime/reference/arima_reg.html)model_arima_reg <- arima_reg() %>% set_engine(engine = "auto_arima")#Model 2: exp_smoothing via ets#(https://business-science.github.io/modeltime/reference/exp_smoothing.html)model_ets <- exp_smoothing() %>% set_engine(engine = "ets") #Model 3: prophet_reg#(https://business-science.github.io/modeltime/reference/prophet_reg.html)model_prophet <- prophet_reg(changepoint_num = tune()) %>% set_engine(engine = "prophet")#Model 4: arima_boost #(https://business-science.github.io/modeltime/reference/arima_boost.html)model_arima_boost <- arima_boost( min_n = tune(), learn_rate = tune(), trees = tune() ) %>% set_engine(engine = "auto_arima_xgboost")#Model 5: earth ----#(https://parsnip.tidymodels.org/reference/mars.html)model_spec_mars <- mars(mode = "regression", num_terms = tune(), prod_degree = tune(), prune_method = tune()) %>% set_engine("earth") #Building workflow setslibrary(workflowsets)#Workflow set for ARIMA, ETS and Prophetrec_formula <- recipe(xbank ~ ., data = df_train) wflw_formula <- workflow_set( preproc = list(formula = rec_formula), models = list(ARIMA_reg = model_arima_reg, ETS = model_ets, Prophet = model_prophet) )#Workflow set for Boosted ARIMArec_boost <- rec_formula %>% step_timeseries_signature(date) %>% step_rm(contains("iso"), contains("minute"), contains("hour"), contains("am.pm"), contains("xts")) %>% step_normalize(contains("index.num"), date_year) %>% step_zv(all_predictors()) %>% step_dummy(contains("lbl"), one_hot = TRUE) wflw_boost <- workflow_set( preproc = list(rec_boost = rec_boost), models = list(ARIMA_boost = model_arima_boost) )#Workflow for Marsrec_mars <- rec_formula %>% step_date(date, features = "month", ordinal = FALSE) %>% step_mutate(date_num = as.numeric(date)) %>% step_normalize(date_num) %>% step_rm(date)wflw_mars <- workflow_set( preproc = list(rec_mars = rec_mars), models = list(Mars = model_spec_mars) )#Combining all the workflowsall_wflws <- bind_rows(wflw_formula, wflw_boost, wflw_mars) %>% # Make the workflow ID's a little more simple: mutate(wflow_id = gsub("(rec_boost_)|(formula_)|(rec_mars_)", "", wflow_id))#Tuning and evaluating all the modelsgrid_ctrl <- control_grid( save_pred = TRUE, parallel_over = "everything", save_workflow = TRUE )grid_results <- all_wflws %>% workflow_map( seed = 98765, resamples = df_folds, grid = 10, control = grid_ctrl )#Accuracy table of the grid resultslibrary(gt)grid_results %>% rank_results(select_best = TRUE, rank_metric = "rsq") %>% select(Models = wflow_id, .metric, mean) %>% pivot_wider(names_from = .metric, values_from = mean) %>% mutate(across(2:3, round, 2)) %>% gt(id = "my_table") %>% data_color( method = "numeric", palette = c("red", "green") ) %>% cols_align(align = "center", columns = -Models) %>% #separating alignment of column names from cells-alignment tab_style( style = cell_text(align = "left"), locations = cells_body( columns = Models )) %>% #separating cell body from each other tab_style( style = cell_borders(sides = "all", color = "white", weight = px(12), style = "solid"), locations = cells_body(columns = everything())) %>% #Rounded corners for the entire table opt_css(css = " #my_table .gt_row { border-radius: 30px; } ") %>% #Changing the column names cols_label( Models = md("Models"), rmse = md("RMSE"), rsq = md("R-Squared") ) %>% tab_header(title = "Accuracy ranking of the models") %>% opt_table_font(font = "Bricolage Grotesque")

``` According to the above table, we will choose the MARS model to predict the index. To do that, we will first finalize the model with the best parameters, according to the above accuracy results. The new accuracy results for our testing data look like dropped a little bit, but still fine.

```

Finalizing the model with the best parametersmars_best_param <- grid_results %>% extract_workflow_set_result("Mars") %>% select_best(metric = "rsq")mars_fit_wflw <- grid_results %>% extract_workflow("Mars") %>% finalize_workflow(mars_best_param) %>% fit(df_train)#Calibrate the model to the testing setcalibration_tbl_mars <- mars_fit_wflw %>% modeltime_calibrate(new_data = df_test)#Accuracy of the finalized modelcalibration_tbl_mars %>% modeltime_accuracy(metric_set = metric_set(rmse,rsq))# A tibble: 1 × 5# .model_id .model_desc .type rmse rsq# #1 1 EARTH Test 53963. 0.849

``` Now, we can make forecasts with our finalized model. As I built unseen data for forecasting, I chose %50 for one-week repo rates in Turkey intuitively.

```

Forecasting#Refit to full dataset for forcastingrefit_tbl_mars <- calibration_tbl_mars %>% modeltime_refit(data = df_merged)#Future(unseen) data framelibrary(tsibble)library(fable)date <- df_merged %>% mutate(date = yearmonth(date)) %>% as_tsibble() %>% new_data(8) %>% as_tibble() %>% mutate(date = as.Date(date))df_future <- date %>% mutate(tcmb_rates = c(rep(50,8)))xbank_fc <- refit_tbl_mars %>% modeltime_forecast(new_data = df_future) %>% select(Date = .index, XBANK = .value) %>% mutate(Price_Index = XBANK / last(df_merged$xbank)100 %>% round(0),#making 2024 Apr value = 100 to see the changes(%) XBANK = round(XBANK, 3), Date = yearmonth(Date))#Making a forecasting tablexbank_fc %>% gt(id = "my_table") %>% data_color( method = "numeric", palette = c("steelblue", "orange") ) %>% cols_align(align = "center", columns = everything()) %>% #separating alignment of column names from cells-alignment tab_style( style = cell_text(align = "left"), locations = cells_body( columns = Date )) %>% #separating cell body from each other tab_style( style = cell_borders(sides = "all", color = "white", weight = px(12), style = "solid"), locations = cells_body(columns = everything())) %>% #Rounded corners for the entire table opt_css(css = " #my_table .gt_row { border-radius: 30px; } ") %>% #Changing the column names cols_label( Date = md("*Date"), XBANK = md("BIST Banks Index"), Price_Index = md("XBANK (2024 Apr = 100)**") ) %>% tab_header(title = "Forecasting by the end of the year") %>% opt_table_font(font = "Bricolage Grotesque")

``` According to the above table, the index will continue to lift the trend by the end of the year; especially in May.

Disclaimer: This content is prepared for purely educational purposes, and cannot be considered as investment advice.

To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Time Series Forecasting: The BIST Banks Index

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 444–448

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #444This time we were spelling numbers. Yes, you heard it right. We were taking multidigit number, spell it one by one, counting how many of them were present in digit. Like: there are two ones, one three and so on. Of course after we wrote it back exactly like above and that activity four times per number. I don’t know if it has any real-life equivalent, but it is surely great challenge.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/443 Look and Say Sequence.xlsx", range = "A1:A10")test = read\_excel("Excel/443 Look and Say Sequence.xlsx", range = "B1:B10") Transfromation generate\_next = function(number) { number\_str = as.character(number) digits = str\_split(number\_str, "")[[1]] unique\_digits = unique(digits) result = map\_chr(unique\_digits, function(digit) { count = sum(digits == digit) paste0(count, digit) }) %>% paste0(collapse = "") as.numeric(result)}generate\_sequence = function(start\_digit, iter = 4) { result = start\_digit for (i in 1:iter) { next\_number = generate\_next(result[length(result)]) result = c(result, next\_number) } all = result %>% setdiff(., start\_digit) %>% paste0(collapse = ", ") return(all)}result = input %>% mutate(`Answer Expected` = map\_chr(Numbers, generate\_sequence)) Validation identical(result$`Answer Expected`, test$`Answer Expected`)# [1] TRUE Puzzle #445We’ve already drawn flags, triangles and other objects, but what? Eiffel Tower? I met some problems with doing it exactly like in task, but I did my best. (Excel has nice text centering in cells, while R doesn’t). Let see my tower.

Loading libraries and data library(tidyverse)library(gt) Transformation df = data.frame(Eiffel = c("|", "/\\", "00", "XX","XX","XX", "XXXX","XXXX","XXXX", "XXXXXXX","XXXXXXX","XXXXXXX", "XXXXXXXXX","XXXXXXXXX","XXXXXXXXX", paste0(strrep("X", 4), strrep("\_", 6), strrep("X", 4)), paste0(strrep("X", 4), strrep("\_", 8), strrep("X", 4)), strrep("X", 18), paste0(strrep("X", 5), strrep("\_", 12), strrep("X", 5)), paste0(strrep("X", 6), strrep("\_", 16), strrep("X", 6)) ))df\_gt = df %>% gt() %>% cols\_align(align = "center") df\_gt Puzzle #446We get crosstable with many airports and distances between them. It is nice version to present it, but we need 3 longest distances from city to city. And do not understand me wrong, not 3 flights, but distances, so if we have a tie, it can be 4 or more flights. Find them…

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/446 Top 3 Min Distance.xlsx", range = "A1:H8")test = read\_excel("Excel/446 Top 3 Min Distance.xlsx", range = "J2:M6") Transformation result = input %>% pivot\_longer(-Cities, names\_to = "City 2", values\_to = "Distance") %>% filter(Distance != 0) %>% unite("Cities", Cities, `City 2`, sep = " - ") %>% mutate(Cities = str\_split(Cities, " - ")) %>% mutate(Cities = map(Cities, sort)) %>% distinct() %>% mutate(rank = dense\_rank(Distance) %>% as.numeric()) %>% filter(rank <= 3) %>% arrange(rank) %>% mutate(`From City` = map\_chr(Cities, ~ .x[1]), `To City` = map\_chr(Cities, ~ .x[2])) %>% select(Rank = rank, `From City`, `To City`, Distance) Validation identical(result, test) # [1] TRUE Puzzle #447Weird, unique, special, nice etc. are one of the easiest and the simplest names for numbers. Today we have to find some more complex numbers called penholodigital. What does it mean?
They are the numbers that consist of all digits except 0’s exactly once, but at the same time are perfect squares, which means that root of the square is integer. Lets find them.

Loading data and libraries library(gtools)library(tictoc)library(tidyverse)library(readxl)test = read\_excel("Excel/447 Penholodigital Squares.xlsx", range = "A1:A31")test$`Answer Expected` = as.numeric(test$`Answer Expected`) Transformation — Approach #1 penholodigital\_numbers <- apply(permutations(9, 9, 1:9, set = FALSE), 1, function(x) { num <- as.numeric(paste0(x, collapse = "")) root <- sqrt(num) if (root == floor(root)) num else NA})penholodigital\_numbers <- na.omit(penholodigital\_numbers)toc() # 3.59 secp1 = penholodigital\_numbers %>% tibble(`Answer Expected` = .)attributes(p1$`Answer Expected`) <- NULL Transformation — Approach #2 tic()penholodigital\_numbers2 = permutations(9,9,1:9) %>% as\_tibble() %>% unite(num, V1:V9, sep = "") %>% mutate(num = as.numeric(num)) %>% filter(sqrt(num) == floor(sqrt(num)))toc() # 3.3 sec Validation identical(p1$`Answer Expected`, test$`Answer Expected`)# [1] TRUEidentical(penholodigital\_numbers2$num, test$`Answer Expected`)# [1] TRUE Puzzle #448Do we see pyramids on image? Upside down? Do not worry, we just need to construct upside down triangles filled with mirrored numbers. Easy peasy, I will just juggle one way or another, and it will be easy..

Loading libraries and data library(tidyverse)library(readxl)test2 = read\_excel("Excel/448 Draw Inverted Triangle.xlsx", range = "B2:D3", col\_names = FALSE) %>% as.matrix()test3 = read\_excel("Excel/448 Draw Inverted Triangle.xlsx", range = "B5:F7", col\_names = FALSE) %>% as.matrix()test4 = read\_excel("Excel/448 Draw Inverted Triangle.xlsx", range = "B9:H12", col\_names = FALSE) %>% as.matrix()test7 = read\_excel("Excel/448 Draw Inverted Triangle.xlsx", range = "B14:N20", col\_names = FALSE) %>% as.matrix() Transformation create\_sequence\_matrix <- function(n) { total\_elements <- n * (n + 1) / 2 max\_elements\_in\_row <- n values <- seq(total\_elements) mat <- matrix(NA, nrow = n, ncol = max\_elements\_in\_row) start\_index <- 1 for (i in 1:n) { end\_index <- start\_index + i - 1 mat[i, 1:i] <- values[start\_index:end\_index] start\_index <- end\_index + 1 } mat}flip\_horizontal <- function(mat) { mat[, ncol(mat):1, drop = FALSE]}flip\_vertical <- function(mat) { mat[nrow(mat):1, , drop = FALSE]}generate\_upsidedown\_triangle = function(n) {mat\_or = create\_sequence\_matrix(n)mat\_fh = flip\_horizontal(mat\_or)mat\_fv1 = flip\_vertical(mat\_or)mat\_fv2 = flip\_vertical(mat\_fh)mat\_fv2 <- mat\_fv2[, -ncol(mat\_fv2)]mat\_fin <- cbind(mat\_fv2, mat\_fv1)mat\_fin} Validation all.equal(generate\_upsidedown\_triangle(2), test2, check.attributes = FALSE) # TRUEall.equal(generate\_upsidedown\_triangle(3), test3, check.attributes = FALSE) # TRUEall.equal(generate\_upsidedown\_triangle(4), test4, check.attributes = FALSE) # TRUEall.equal(generate\_upsidedown\_triangle(7), test7, check.attributes = FALSE) # TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.
PS. Couple weeks ago, I started uploading on Github not only R, but also in Python. Come and check it.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on R - Hosting Data Apps, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.ShinyProxy is one the most popular open-source Shiny hosting options. You can use it to host multiple applications, apps other than Shiny, and it comes with built-in enterprise features, like authentication and authorization.

Open Analytics, the Belgian company behindShinyProxy, just announced the 3.1.0 release. Head over to the ShinyProxy release notes for all the details. I go over my favourite highlights here.

Part of the release notes for ShinyProxy 3.1.0.Pre-initialization and container sharingFrom its inception, ShinyProxy provided exactly 1 app instance per user. Later it introduced >1 replicas. Such a setup provides complete isolation for the user sessions, making it very hard to leak sensitive information. Containers started on-demand, making it unnecessary for all apps to run constantly.

Sharing an app among multiple users was not possible in ShinyProxy. This is very similar to how Shinyapps or Shiny Server operates, a single instance can allow many user sessions. Sharing the same instance also reduces wait time, i.e. if a user is already using the app, the next user can join without waiting.

With the recent change, this is now possible with ShinyProxy:

ShinyProxy 3.1.0 introduces the concept of *container pre-initialization and container sharing. For every app, you can specify a number of seats ShinyProxy has to pre-initialize (e.g. X seats). When ShinyProxy starts up, it will start X containers. As soon as a user wants to use an application, ShinyProxy assigns a container (seat) to the user. Therefore, the user has almost no waiting time for the container to start up. As soon as a seat gets claimed*, ShinyProxy will scale up the number of containers, such that there are again X seats ready to be used by new users. Similarly, when a user stops using an app, the seat is released and ShinyProxy will scale-down the pool of available seats (and thus the number of containers). In addition, you can specify how many seats can be run on a single container, therefore allowing to share a single container among multiple users.

See the announcement blog post for a screen capture that shows the reduced wait time due to container sharing.

This new feature is easy to turn on, but of course, there are some gotchas. For example, some environment variables cannot be passed to the container as before and the container cannot be customized for a specific user, because we don't know yet who the new user might be. Instead, we need to use HTTP headers to get the user name and user group variables. See the docs for the details.

New backend: AWS ECSThis new backend is a very exciting and potentially transformative addition. It allows scalability without having to manage servers or dive into the complexities of Kubernetes. As explained in the announcement blog post:

Up to now ShinyProxy supported three container backends: Docker, Docker Swarm and Kubernetes. The Docker backend is very easy to set up, but does not provide the same scalability and flexibility as the Kubernetes backend. ShinyProxy 3.1.0 now supports AWS ECS as a new backend, allowing to deploy ShinyProxy without managing servers or complex setups, but with greater scalability. The backend uses AWS Fargate and supports most of the ECS features: integration with CloudWatch, Task role support (e.g. to access S3) and EFS volumes.

A downside of AWS ECS is that in most cases it takes more time to start the container compared to the other backends. But of course it can be used together with the pre-initialized container feature, reducing the startup time to less than a second.

There is a starter example here.

Improvements for OpenID providersSome providers that use the OpenID Connect (OIDC) standard for user authentication, provide important information (so-called claims) as part of the access token.

Decoding the access token was previously possible inside the apps, but it was not available as part of the Spring Expressions (SpEL) used by ShinyProxy. Therefore, authorization was not possible immediately after login, i.e. the app is listed but the user might not be authorized to use that. Read more about the background in this GitHub issue.

With the recent change, we can now use the decoded OpenID access token in SpEL. This will open doors for some OIDC providers where group-based authorization was not possible before.

Smaller improvements* Every backend now uses the ID of the app as part of the name, making it easier to find which container corresponds to which app. * Limit the total number of apps running on a server or cloud infrastructure: set limits at a global and per-app level. * An option to show notification (a.k.a message of the day) to all users

1-click app updatesThe DigitalOcean 1-click app is being updated to pack ShinyProxy 3.1.0. Apart from adding the new version of ShinyProxy and testing the machine image, the new version of the ShinyProxy 1-click app on DigitalOcean ships with a setup script on the 1st SSH login: you will be prompted to set up TLS certificate via Let's Encrypt to serve the ShinyProxy over HTTPS. You'll need a domain name with a DNS A record pointing to the Droplet IP address, and an email address.

ShinyProxy 1-click AppShinyProxy is a great way to deploy containerized Shiny apps to production and it is only 1 click away.Hosting Data AppsPeter SolymosShinyProxy – Hosting Data AppsShinyProxy is the quickest way to deploy containerized apps in an enterprise context with built-in authentication and authorization, securing, and without limits on concurrent usageHosting Data AppsPeter SolymosTo leave a comment for the author, please follow the link and comment on their blog: R - Hosting Data Apps.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: What Is New in ShinyProxy 3.1.0

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhen working with data frames in R, it’s common to need to select specific columns based on their index positions. This task is straightforward in R, especially with base functions. In this article, we’ll explore how to select columns by their index using simple and effective techniques in base R.

Understanding Column IndexingIn R, data frames are structured with rows and columns. Columns can be referred to by their names or their numerical indices. The index of a column in a data frame represents its position from left to right, starting with 1.

Selecting Columns by IndexTo select columns by their indices, we can use the square bracket [ ] notation. This notation allows us to specify which columns we want to extract from a data frame based on their index positions.

Let’s dive into some examples.

ExamplesExample 1: Selecting Single Column by IndexSuppose we have a data frame df with several columns, and we want to select the second column. Here’s how you can do it:

```

Create a sample data framedf <- data.frame( Name = c("Alice", "Bob", "Charlie"), Age = c(25, 30, 28), Score = c(88, 92, 75))# Select the second column by index (Age)selected_column <- df[, 2]print(selected_column)

```

[1] 25 30 28 In this code snippet:

df[, 2] specifies that we want to select all rows ([,]) from the second column (2) of the data frame df.
The result (selected_column) will be a vector containing the values from the “Age” column.

Example 2: Selecting Multiple Columns by IndicesTo select multiple columns simultaneously, you can provide a vector of column indices within the square brackets. For instance, if we want to select the first and third columns from df:

```

Select the first and third columns by indices (Name and Score)selected_columns <- df[, c(1, 3)]print(selected_columns)

```

Name Score1 Alice 882 Bob 923 Charlie 75 In this example:

df[, c(1, 3)] selects all rows ([,]) from the first and third columns (c(1, 3)) of the data frame df.
The result (selected_columns) will be a subset of df containing only the “Name” and “Score” columns.

Example 3: Selecting All Columns Except OneIf you want to exclude specific columns while selecting all others, you can use negative indexing. For instance, to select all columns except the second one:

```

Select all columns except the second one (Age)selected_columns <- df[, -2]print(selected_columns)

```

Name Score1 Alice 882 Bob 923 Charlie 75 Here:

df[, -2] selects all rows ([,]) from df, excluding the second column (-2).
The result (selected_columns) will be a data frame containing columns “Name” and “Score”, excluding “Age”.

Conclusion and ChallengeSelecting columns by index is a fundamental operation in data manipulation with R. By understanding how to use basic indexing techniques, you can efficiently extract and work with specific subsets of your data frames.

I encourage you to experiment with these examples using your own data frames. Try selecting different combinations of columns or excluding specific ones to see how it affects your data subset. This hands-on approach will deepen your understanding and confidence in working with R’s data structures.

Keep exploring, and happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Select Columns by Index in R (Using Base R)

View Details

[This article was first published on pharmaverse blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. TLG Catalog website

Introducing WebR to TLG Catalog: A Game Changer for Interactive LearningI’m thrilled to announce a major update to the TLG Catalog. We have integrated WebR to the website transforming the way you interact with the R code. You don’t have to copy and run code separately, open containers and/or install packages. With WebR, you can now interact with the TLG source R code directly in your browser, making learning more interactive and enjoyable than ever before!

What is WebR?

WebR makes it possible to run R code in the browser without the need for an R server to execute the code: the R interpreter runs directly on the user’s machine.

Source: WebR documentation

In short, WebR is a project that aims to port R into WebAssembly (WASM) which then allows to run compiled code in the website. A special thanks to George Stagg from Posit for making this integration possible. While WebR is still in active development, a significant progress had been made recently increasing its robustness and efficiency.

However, it’s important to note a limitation: not all packages are compatible with WebR. A package must be compiled for WebAssembly to be used with WebR. Fortunately, there’s a dedicated WebR binary R package repository hosting close to 20,000 packages. For packages not yet available, you can utilize a dedicated GitHub Actions workflow to build them yourself, or use r-universe platform that will build it for you.

Implementation DetailsThe integration of WebR into TLG Catalog was made possible through a dedicated quarto-webr Quarto extension, which simplifies the integration process. The main challenge was to ensure a DRY (Don’t Repeat Yourself) approach with respect to the existing codebase. This was achieved through leveraging lesser-known knitr features, including knitr::knit_code$get() to reuse code chunks as well as results = "asis" to create code chunk from within another (parent) code chunk. The source code for this is open-source and available on GitHub.

Interactive teal Applications via shinyliveThe benefits of WebR extend beyond TLG outputs. It also enhances all existing teal applications. Users can now interact with applications and even live-edit their source code! Everything is inside the website itself without any additional application hosting service. This was made possible through the shinylive Quarto extension leveraging Shinylive under the hood. A huge thank you to the Shiny team for their contributions!

SummaryThe addition of interactivity via WebR marks a significant milestone for TLG Catalog. This update unlocks a myriad of possibilities previously unavailable, such as live code editing, step-by-step code execution, access to function documentation, and dynamic data exploration. This advancement brings R closer to users, especially those new to the language, fostering a more engaging and effective learning experience.

Happy learning!

Attaching package: 'dplyr'

The following objects are masked from 'package:stats': filter, lag

The following objects are masked from 'package:base': intersect, setdiff, setequal, union Last updated2024-05-08 15:37:10.73831

DetailsSource, Session info

ReuseCC BY 4.0CitationBibTeX citation: @online{rucki2024, author = {Rucki, Pawel}, title = {TLG {Catalog} 🤝 {WebR}}, date = {2024-05-08}, url = {https://pharmaverse.github.io/blog/posts/2024-05-08\_tlg\_catalog\_webr/tlg\_catalog\_webr.html}, langid = {en}} For attribution, please cite this work as:Rucki, Pawel. 2024. “TLG Catalog WebR.” May 8, 2024. https://pharmaverse.github.io/blog/posts/2024-05-08_tlg_catalog_webr/tlg_catalog_webr.html. To leave a comment for the author, please follow the link and comment on their blog: pharmaverse blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: TLG Catalog 🤝 WebR

View Details

[This article was first published on GAMbler, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Welcoming mvgam to the Comprehensive R Archive Network (CRAN) The goal of mvgam is to use a Bayesian framework to estimate parameters of Dynamic Generalized Additive Models (DGAMs) for time series with dynamic trend components. The package provides an interface to fit Bayesian DGAMs using Stan as the backend, and is particularly suited to the estimation of complex State-Space models. The formula syntax is based on that of the package mgcv to provide a familiar GAM modelling interface. There is also built-in support for the increasingly powerful marginaleffects package to make interpretation easy. The motivation for the package and some of its primary objectives are described in detail by Clark & Wells 2022 (published in Methods in Ecology and Evolution). An introduction to the package and some worked examples are also shown in the below seminar:

Ecological Forecasting with Dynamic Generalized Additive Models DGAMs)

The first release of mvgam has now been published on CRAN. While the package has featured on this blog before, for example when I explained hierarchical distributed lag models and how to model temporally autocorrelated time series with nonlinear trends, I have not thorougly introduced the it yet. So the purpose of this brief post is to cover some of the features of this stable release, and to hint at a few of the development goals for upcoming releases.

What are Generalized Additive Models (GAMs)? Generalized Additive Models (GAMs) are flexible tools that replace one or more predictors in a Generalized Linear Model (GLM) with smooth functions of predictors. These are helpful for learning arbitrarily complex, nonlinear relationships between predictors and conditional responses without needing a priori expectations about the shapes of these relationships. Rather, they are learned using penalized smoothing splines.

How do these work? The secret is a basis expansion, which in lay terms means that the covariate (time, in this example) is evaluated at a smaller set of basis functions designed to cover the range of the observed covariate values. Below is one particular type of basis, called a cubic regression basis.

Each basis function acts on its own local neighbourhood of the covariate. Once we have constructed the basis, we can estimate weights for each function. The weights allow the basis functions to have different impacts on the shape of the spline. They also give us a target for learning splines from data, as the weights act as the regression coefficients.

There are many more types of basis expansions that can be used to form penalized smooths, including multidimensional smooths, spatial smooths or even monotonic smooths. For more information on GAMs and how they can smooth through data, see my blogpost on how to interpret nonlinear effects from Generalized Additive Models.

What does mvgam do? mvgam is designed to propagate unobserved temporal processes to capture latent dynamics in the observed time series. This works in a state-space format, with the temporal trend evolving independently of the observation process. Briefly, assume $\tilde{\boldsymbol{y}}_{i,t}$ is the conditional expectation of response variable $\boldsymbol{i}$ at time $\boldsymbol{t}$. Assuming $\boldsymbol{y_i}$ is drawn from an exponential distribution with an invertible link function, the linear predictor for a multivariate Dynamic GAM can be written as:

$$for~i~in~1:N_{series}~...$$``$$for~t~in~1:N_{timepoints}~...$$

$$g^{-1}(\tilde{\boldsymbol{y}}_{i,t})=\alpha_{i}+\sum\limits_{j=1}^J\boldsymbol{s}_{i,j,t}\boldsymbol{x}_{j,t}+\boldsymbol{z}_{i,t}\,,$$

Here $\alpha$ are the unknown intercepts, the $\boldsymbol{s}$’s are unknown smooth functions of covariates $(\boldsymbol{x})$’s, which can potentially vary among the response series, and $\boldsymbol{z}$ are dynamic latent processes. Each smooth function $\boldsymbol{s_j}$ is composed of basis expansions whose coefficients, which must be estimated, control the functional relationship between $\boldsymbol{x}_{j}$ and $g^{-1}(\tilde{\boldsymbol{y}})$. But note that we can also include smooth functions of covariates in the process models $\boldsymbol{z}$.

Several advantages of GAMs are that they can model a diversity of response families, including discrete distributions (i.e. Poisson, Negative Binomial, Gamma) that accommodate common ecological features such as zero-inflation or overdispersion, and that they can be formulated to include hierarchical smoothing for multivariate responses. mvgam supports a number of different observation families, which are summarized below.

Supported observation families mvgam was originally designed to analyse and forecast time series of non-negative integer-valued data (counts). These data are traditionally challenging to analyse with existing time series analysis packages. But further development of mvgam has resulted in support for a growing number of observation families that extend to other types of data. Currently, the package can handle data for the following families:

| Distribution | Function | Support | Extra parameter(s) | | --- | --- | --- | --- | | Gaussian (identity link) | gaussian() | Real values in $(-\infty, \infty)$ | $\sigma$ | | Student’s T (identity link) | student-t() | Heavy-tailed real values in $(-\infty, \infty)$ | $\sigma$, $\nu$ | | LogNormal (identity link) | lognormal() | Positive real values in $[0, \infty)$ | $\sigma$ | | Gamma (log link) | Gamma() | Positive real values in $[0, \infty)$ | $\alpha$ | | Beta (logit link) | betar() | Real values (proportional) in $[0,1]$ | $\phi$ | | Bernoulli (logit link) | bernoulli() | Binary data in ${0,1}$ | – | | Poisson (log link) | poisson() | Non-negative integers in $(0,1,2,...)$ | – | | Negative Binomial2 (log link) | nb() | Non-negative integers in $(0,1,2,...)$ | $\phi$ | | Binomial (logit link) | binomial() | Non-negative integers in $(0,1,2,...)$ | – | | Beta-Binomial (logit link) | beta_binomial() | Non-negative integers in $(0,1,2,...)$ | $\phi$ | | Poisson Binomial N-mixture (log link) | nmix() | Non-negative integers in $(0,1,2,...)$ | – |

For all supported observation families, any extra parameters that need to be estimated (i.e. the $\sigma$ in a Gaussian model or the $\phi$ in a Negative Binomial model) are by default estimated independently for each series. However, users can opt to force all series to share extra observation parameters using share_obs_params = TRUE in mvgam(). Note that default link functions cannot currently be changed.

Supported temporal dynamic processes The dynamic processes that mvgam can fit to time series can take a wide variety of forms, some of which can be multivariate to allow the different time series to interact or be correlated. When using the mvgam() function, the user chooses between different process models with the trend_model argument. Available process models are described in ?mvgam_trends and detailed below.

Independent Random Walks Use trend_model = 'RW' or trend_model = RW() to set up a model where each series in data has independent latent temporal dynamics of the form:

$$\begin{align*} z_{i,t} & \sim \text{Normal}(z_{i,t-1}, \sigma_i) \end{align*}$$

Process error parameters $\sigma$ are modeled independently for each series. If a moving average process is required, use trend_model = RW(ma = TRUE) to set up the following:

$$\begin{align*} z_{i,t} & = z_{i,t-1} + \theta_i * error_{i,t-1} + error_{i,t} \\ error_{i,t} & \sim \text{Normal}(0, \sigma_i) \end{align*}$$

Moving average coefficients $\theta$ are independently estimated for each series and will be forced to be stationary by default $(abs(\theta)<1)$. Only moving averages of order $q=1$ are currently allowed.

Multivariate Random Walks If more than one series is included in data $(N_{series} > 1)$, a multivariate Random Walk can be set up using trend_model = RW(cor = TRUE), resulting in the following:

$$\begin{align*} z_{t} & \sim \text{MVNormal}(z_{t-1}, \Sigma) \end{align*}$$

Where the latent process estimate $z_t$ now takes the form of a vector. The covariance matrix $\Sigma$ will capture contemporaneously correlated process errors. It is parameterised using a Cholesky factorization, which requires priors on the series-level variances $\sigma$ and on the strength of correlations using Stan’s lkj_corr_cholesky distribution.

Moving average terms can also be included for multivariate random walks, in which case the moving average coefficients $\theta$ will be parameterised as an $N_{series} * N_{series}$ matrix

Autoregressive processes Autoregressive models up to $p=3$, in which the autoregressive coefficients are estimated independently for each series, can be used by specifying trend_model = 'AR1', trend_model = 'AR2', trend_model = 'AR3', or trend_model = AR(p = 1, 2, or 3). For example, a univariate AR(1) model takes the form:

$$\begin{align*} z_{i,t} & \sim \text{Normal}(ar1_i * z_{i,t-1}, \sigma_i) \end{align*}$$

All options are the same as for Random Walks, but additional options will be available for placing priors on the autoregressive coefficients. By default, these coefficients will not be forced into stationarity, but users can impose this restriction by changing the upper and lower bounds on their priors. See ?get_mvgam_priors for more details.

Vector Autoregressive processes A Vector Autoregression of order $p=1$ can be specified if $N_{series} > 1$ using trend_model = 'VAR1' or trend_model = VAR(). A VAR(1) model takes the form:

$$\begin{align*} z_{t} & \sim \text{Normal}(A * z_{t-1}, \Sigma) \end{align*}$$

Where $A$ is an $N_{series} * N_{series}$ matrix of autoregressive coefficients in which the diagonals capture lagged self-dependence (i.e. the effect of a process at time $t$ on its own estimate at time $t+1$), while off-diagonals capture lagged cross-dependence (i.e. the effect of a process at time $t$ on the process for another series at time $t+1$). By default, the covariance matrix $\Sigma$ will assume no process error covariance by fixing the off-diagonals to $0$. To allow for correlated errors, use trend_model = 'VAR1cor' or trend_model = VAR(cor = TRUE). A moving average of order $q=1$ can also be included using trend_model = VAR(ma = TRUE, cor = TRUE).

Note that for all VAR models, stationarity of the process is enforced with a structured prior distribution that is described in detail in Heaps 2022

Heaps, Sarah E. “Enforcing stationarity through the prior in vector autoregressions.” Journal of Computational and Graphical Statistics 32.1 (2023): 74-83.

Gaussian Processes The final option for modelling temporal dynamics is to use a Gaussian Process with squared exponential kernel. These are set up independently for each series (there is currently no multivariate GP option), using trend_model = 'GP'. The dynamics for each latent process are modelled as:

$$\begin{align*} z & \sim \text{MVNormal}(0, \Sigma_{error}) \\ \Sigma_{error}[t_i, t_j] & = \alpha^2 * exp(-0.5 * ((|t_i - t_j| / \rho))^2) \end{align*}$$

The latent dynamic process evolves from a complex, high-dimensional Multivariate Normal distribution which depends on $\rho$ (often called the length scale parameter) to control how quickly the correlations between the model’s errors decay as a function of time. For these models, covariance decays exponentially fast with the squared distance (in time) between the observations. The functions also depend on a parameter $\alpha$, which controls the marginal variability of the temporal function at all points; in other words it controls how much the GP term contributes to the linear predictor. mvgam capitalizes on some advances that allow GPs to be approximated using Hilbert space basis functions, which considerably speed up computation at little cost to accuracy or prediction performance.

Piecewise logistic and linear trends Modeling growth for many types of time series is often similar to modeling population growth in natural ecosystems, where there series exhibits nonlinear growth that saturates at some particular carrying capacity. The logistic trend model available in {mvgam} allows for a time-varying capacity $C(t)$ as well as a non-constant growth rate. Changes in the base growth rate $k$ are incorporated by explicitly defining changepoints throughout the training period where the growth rate is allowed to vary. The changepoint vector $a$ is represented as a vector of 1s and 0s, and the rate of growth at time $t$ is represented as $k+a(t)^T\delta$. Potential changepoints are selected uniformly across the training period, and the number of changepoints, as well as the flexibility of the potential rate changes at these changepoints, can be controlled using trend_model = PW(). The full piecewise logistic growth model is then:

$$\begin{align*} z_t & = \frac{C_t}{1 + \exp(-(k+a(t)^T\delta)(t-(m+a(t)^T\gamma)))} \end{align*}$$

For time series that do not appear to exhibit saturating growth, a piece-wise constant rate of growth can often provide a useful trend model. The piecewise linear trend is defined as:

$$\begin{align*} z_t & = (k+a(t)^T\delta)t + (m+a(t)^T\gamma) \end{align*}$$

In both trend models, $m$ is an offset parameter that controls the trend intercept. Because of this parameter, it is not recommended that you include an intercept in your observation formula because this will not be identifiable. You can read about the full description of piecewise linear and logistic trends in this paper by Taylor and Letham.

Sean J. Taylor and Benjamin Letham. “Forecasting at scale.” The American Statistician 72.1 (2018): 37-45.

Continuous time AR(1) processes Most trend models in the mvgam() function expect time to be measured in regularly-spaced, discrete intervals (i.e. one measurement per week, or one per year for example). But some time series are taken at irregular intervals and we’d like to model autoregressive properties of these. The trend_model = CAR() can be useful to set up these models, which currently only support autoregressive processes of order 1. The evolution of the latent dynamic process follows the form:

$$\begin{align*} z_{i,t} & \sim \text{Normal}(ar1_i^{distance} * z_{i,t-1}, \sigma_i) \end{align*}$$

Where $distance$ is a vector of non-negative measurements of the time differences between successive observations. See the Examples section in ?CAR for an illustration of how to set these models up.

Regression formulae mvgam supports an observation model regression formula, built off the mgcv package, as well as an optional process model regression formula. The formulae supplied to mvgam() are exactly like those supplied to glm() except that smooth terms, s(),te(), ti() and t2(), time-varying effects using dynamic(), monotonically increasing (using s(x, bs = 'moi')) or decreasing penalized splines (using s(x, bs = 'mod'); see ?smooth.construct.moi.smooth.spec for details), as well as Gaussian Process functions using gp(), can be added to the right hand side (and . is not supported in mvgam formulae). See ?mvgam_formulae for more guidance.

For setting up State-Space models, the optional process model formula can be used (see the State-Space model vignette and the shared latent states vignette for guidance on using trend formulae).

See the cheatsheet for a workflow overview Future plans I have several priorities to help improve the workflow of mvgam, as well as some features I would like to implement. These include

Converting all graphics to ggplot2 so that users can modify figures as they wish
Incorporating options for hierarchical modelling of dynamic process parameters (such as $\sigma$ parameters in AR or RW processes)
Adding support for zero-inflated observation models
Adding support for more flexible predictor effects including correlated random effects, hierarchical GPs with shared length scales $\rho$ and dynamic factor spatiotemporal effects

To leave a comment for the author, please follow the link and comment on their blog: GAMbler.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: First release of mvgam(v1.1.0) to CRAN

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Extract columns of data frame in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Extract columns of data frame in R, The pull() function in R’s dplyr package allows users to extract columns from a data frame or tibble.

Extract columns of data frame in RThis article provides two examples of how to apply the pull() function with variable names and indices.

We will also cover the necessary steps to create example data and install/load the dplyr package.

Step 1: Creating Example Data

To demonstrate the usage of the pull() function, we will use the following data frame:

data <- data.frame(x1 = 1:5, x2 = LETTERS[1:5])print(data) This data frame contains five rows and two columns (x1 and x2).

Step 2: Installing and Loading the dplyr Package

Before we can use the pull() function, we need to install and load the dplyr package:

install.packages("dplyr")library("dplyr") Example 1: Apply pull() Function with Variable Name

In the first example, we will extract the x1 column by specifying its variable name within the pull() function:

pull(data, x1) The output will be:

1 2 3 4 5 This shows that the pull() function successfully returned the x1 column as a vector.

Example 2: Apply pull() Function with Index

In the second example, we will extract the first column of the data frame by specifying its index within the pull() function:

pull(data, 1) The output will be:

1 2 3 4 5 This demonstrates that the pull() function can also extract columns using their indices.

ConclusionThe pull() function in R’s dplyr package is a convenient tool for extracting columns from data frames or tibbles.

You can use either the variable name or the index of the column you wish to extract.

Always remember to install and load the dplyr package before using the pull() function.

SharePoint R integration and analysis
How to Find Unmatched Records in R » Data Science Tutorials
Goodness of Fit Test- Jarque-Bera Test in R
Poisson Functions in R Programming
How to interpret regression results in R

The post Extract columns of data frame in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extract columns of data frame in R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.R Shiny needs no introduction, as it’s one of the go-to frameworks for building amazing applications and dashboards. But most of them lack one thing – interactivity and motion. Truth be told, R’s standard visualization packages such as ggplot2 aren’t interactive, so that’s the main reason why.

Enter R Highcharts – a free package for making interactive and animated visualizations, straight from R! We’ve already discussed in our introduction article, drilldown article , and a dedicated Shiny article, so this one will serve as icing on the cake, as you’ll learn how to add interactive maps to your applications.

After reading, you’ll know how to make choropleth and bubble maps in R and R Shiny. We have a lot of ground to cover, so let’s dig in!

Looking to add interactive Google Maps to your Shiny app? Read our guide to find out how.

Table of contents:* How to Make Choropleth R Highcharts Maps * How to Add Points to R Highcharts Maps * R Highcharts Maps in R Shiny – How to Get Started * Summing up R Highcharts Maps

How to Make Choropleth R Highcharts MapsYou can think of choropleth maps as a type of thematic map in which certain areas are shaded or patterned in proportion to the value of a variable being represented.

Let’s say you’re a restaurant chain owner in the US and operate in numerous locations across all 50 states – you can use choropleth maps to visually represent the number of restaurants in each state. The darker the shade of color, let’s say blue, the more restaurants you have running there.

We’ll showcase something similar in this section.

The DatasetTo make our choropleth map with R Highcharts maps, we’ll use the US Airports List Dataset:

Image 1 – US airports datasetIn plain English, it shows the name and geolocation of each airport in the United States and also groups them by state. Sounds like a perfect use case for choropleth maps, doesn’t it?

Download the dataset CSV file and load it into R. Simply copy the following snippet – it will also get rid of a couple of missing values:

library(highcharter)library(dplyr)airports <- read.csv("airports.csv")airports <- na.omit(airports)head(airports) Here’s what you should end up with:

Image 2 – Head of the US airports dataset in RYou now have everything needed to start building R Highcharts choropleth maps, so let’s start with that in the following section.

Base Highcharts MapThe idea now is to start small and continuously build up. You need a base map to get started and the one covering the United States sounds like a perfect candidate.

‍If you need more map options, make sure to check out the official website. Simply click on the region you’re interested in, open the example, and copy the relevant part of the URL.

For our case, countries/us/us-all sounds like a perfect candidate. Let’s wrap it into a call to hcmap to see what happens:

hcmap("countries/us/us-all", showInLegend = FALSE This is the output you’ll get:

Image 3 – Base United States mapThe foundation is there, but we have to build on it. No data is shown currently, and we have no idea what we’re representing. To solve this last issue, let’s add a title and subtitle:

hcmap("countries/us/us-all", showInLegend = FALSE) |> hc\_title(text = "United States of America") |> hc\_subtitle(text = "Airports in the USA") Image 4 – Adding title and subtitleYou now know what you should be looking at, but the data isn’t there yet. Let’s fix that next.

Choropleth Map with Individual State ColoringThe idea now is to group the data by STATE and display the number of airports in each. To be more precise, we want to color each individual state. The darker the shade, the more airports the state has.

The count() function from dplyr does both the grouping and summarization for us:

count\_by\_state <- airports %>% count(STATE, sort = TRUE)head(count\_by\_state) Here are the results:

Image 5 – Airport count by state subsetNow in hcmap(), you can provide values for data and value parameters. Pass in your data frame and specify which column represents the values that will be used for coloring. The optional name parameter is here to change the series name on a tooltip, but more on that later.

Anyhow, here’s the updated snippet:

hcmap( "countries/us/us-all", showInLegend = FALSE, data = count\_by\_state, value = "n", name = "State") |> hc\_title(text = "United States of America") |> hc\_subtitle(text = "Airports in the USA") Now we’re getting somewhere:

Image 6 – Coloring individual statesThe above results are fine, and you can even see the counts when you hover over individual states.

But let’s kick things up a notch. We now want to display state names and draw a distinct line between individual states. This will further enrich the end result and will eliminate all the guesswork from the final user.

The dataLabels,borderColor, and borderWidth parameters are what you’re looking for. The last two are intuitive to understand, but the first needs some explanation. Here, you first need to enable labels and then specify what you want to see on it. Passing in {point.name} will return the name of the state in this case:

hcmap( "countries/us/us-all", showInLegend = FALSE, data = count\_by\_state, value = "n", name = "State", dataLabels = list(enabled = TRUE, format = "{point.name}"), borderColor = "#000000", borderWidth = 0.5) |> hc\_title(text = "United States of America") |> hc\_subtitle(text = "Airports in the USA") The map is looking much better now:

Image 7 – Adding labels to statesNext, let’s work on the tooltip. The goal is to take maximum control over what’s shown to the user when they hover over individual states. To keep things simple, we’ll only change the suffix through the valueSuffix modifier. If you need to, you can also tweak the valuePrefix and valueDecimals modifiers, but these aren’t applicable in our case.

Here’s the updated snippet:

hcmap( "countries/us/us-all", showInLegend = FALSE, data = count\_by\_state, value = "n", name = "State", dataLabels = list(enabled = TRUE, format = "{point.name}"), borderColor = "#000000", borderWidth = 0.5, tooltip = list( valueSuffix = " airport(s)" )) |> hc\_title(text = "United States of America") |> hc\_subtitle(text = "Airports in the USA") You can see the code change taking effect when hovering over individual states:

Image 8 – Tooltip customizationAnd finally, let’s discuss color. There’s no easy way to change the continuous color scale, unfortunately, so you’ll have to get creative. The color\_stops() function returns a list of n colors you can pass into hc\_colorAxis(). That’s the only way to modify the color of a choropleth map in R Highcharts.

Setting n = 2 will give us a nice purple-to-yellow color palette:

hcmap( "countries/us/us-all", showInLegend = FALSE, data = count\_by\_state, value = "n", name = "State", dataLabels = list(enabled = TRUE, format = "{point.name}"), borderColor = "#000000", borderWidth = 0.5, tooltip = list( valueSuffix = " airport(s)" )) |> hc\_title(text = "United States of America") |> hc\_subtitle(text = "Airports in the USA") |> hc\_colorAxis(stops = color\_stops(n = 2)) This is our final map:

Image 9 – Color customizationAnd that’s about it when it comes to choropleth maps in R Highcharts. The other common type is a point map (bubble map), so let’s explore how to work with it next.

How to Add Points to R Highcharts MapsBubble maps allow you to add individual markers, or data points, directly to the base map. These are fantastic for our dataset if we want to display the location of individual airports, instead of the overall counts.

Dataset ModificationsFirst things first, you’ll need to modify the underlying dataset. You now need access to lat and lon attributes, since these are required to determine the map location of individual airports. Let’s also keep track of the three-letter airport abbreviations and full names:

point\_data <- data.frame( name = airports$IATA, full\_name = airports$AIRPORT, lat = airports$LATITUDE, lon = airports$LONGITUDE)head(point\_data) You should end up with the following dataset:

Image 10 – Subset for bubble mapThat’s all you need to get started with bubble maps with R Highcharts! Let’s explore the basics next.

Create and Style Bubble Maps with R HighchartsIn Highcharts, you can create a bubble map by calling the hc\_add\_series() function and specifying the map type as mappoint. Also, pass in your dataset, and the library will figure out everything else for you:

hcmap("countries/us/us-all", showInLegend = FALSE) |> hc\_add\_series( data = point\_data, type = "mappoint", name = "Airports" ) Now we’re getting somewhere:

Image 11 – Bubble map with airport locationsYou now have the data points, but hovering over them reveals a mess. It’s a collection of useless data, and you have to change it.

We’ll get more detailed in a call to tooltip. The pointFormat attribute can accept HTML tags, which means you can take ultimate control over how things look like. We’ll use it to make the first line bold and also to split the contents into multiple lines.

The modified tooltip will show the airport’s abbreviated name, full name, and geolocation rounded up to four decimal places:

hcmap("countries/us/us-all", showInLegend = FALSE) |> hc\_add\_series( data = point\_data, type = "mappoint", name = "Airports", tooltip = list( pointFormat = "{point.name} - {point.full\_name}Location: ({point.lat:,.4f}, {point.lon:,.4f})" ) ) The map still looks the same, but hovering over data points reveals a night and day difference:

Image 12 – Tooltip customizationAnd finally, let’s discuss coloring. The color attribute can accept a variety of things, but since we’re showcasing individual locations, it’s best to keep things simple and use one color only. Appsilon’s blue works like a charm:

hcmap("countries/us/us-all", showInLegend = FALSE) |> hc\_add\_series( data = point\_data, type = "mappoint", color = "#0090f9", name = "Airports", tooltip = list( pointFormat = "{point.name} - {point.full\_name}Location: ({point.lat:,.4f}, {point.lon:,.4f})" ) ) This is the final R Highcharts bubble map:

Image 13 – Bubble point color customizationYou now know how to create choropleth and bubble maps with R Highcharts, but we still haven’t discussed R Shiny implementation. Well, that’s about to change.

R Highcharts Maps in R Shiny – How to Get StartedThe R Shiny application you’re about to build is quite straightforward. It allows the user to change the marker color and add/remove states from the visualization. Not the most useful app, sure, but serves perfectly to showcase how to integrate R Highcharts maps with R Shiny.

Here are R Highcharts specifics you should know:

highchartsOutput() – Use this function in your ui to create a placeholder for your Highcharts visualization, either a chart or a map.
renderHighcharts() – Use it in server() to build a map from reactive datasets/values.

In the server() function, you’ll also want to make the dataset and marker color reactive. For the latter, we’re allowing the user to select one of three named colors from the dropdown menu, so marker\_color needs to map the English name to hex color values.

When it comes to the dataset, you’ll want to use the %in% operator to show only the states the user has selected in a multi-select input.

The rest of the code snippet is standard Shiny stuff:

library(shiny)library(dplyr)library(highcharter)airports <- read.csv("airports.csv")airports <- na.omit(airports)ui <- fluidPage( sidebarLayout( sidebarPanel( tags$h3("Highcharts Maps"), # Control the marker color selectInput(inputId = "inColor", label = "Marker color:", choices = c("Red", "Yellow", "Blue"), selected = "Blue"), # Control the states that are shown on the map selectInput(inputId = "inState", label = "States:", choices = sort(unique(airports$STATE)), selected = sort(unique(airports$STATE)), multiple = TRUE) ), mainPanel( highchartOutput(outputId = "chartUSAMap", height = 500) ) ))server <- function(input, output) { # Hold the dataset as a reactive value in which states are filtered based on user input data <- reactive({ data.frame( name = airports$IATA, full\_name = airports$AIRPORT, state = airports$STATE, lat = airports$LATITUDE, lon = airports$LONGITUDE ) %>% filter(state %in% input$inState) }) # Map string color name to its hex code marker\_color <- reactive({ switch(input$inColor, "Red" = "#f90047", "Yellow" = "#f9d400", "Blue" = "#0090f9" ) }) # Render chart output$chartUSAMap <- renderHighchart({ hcmap("countries/us/us-all", showInLegend = FALSE) |> hc\_add\_series( data = data(), type = "mappoint", color = marker\_color(), name = "Airports", tooltip = list( pointFormat = "{point.name} - {point.full\_name}Location: ({point.lat:,.4f}, {point.lon:,.4f})" ) ) })}shinyApp(ui = ui, server = server) Here’s what you’ll see after running the app:

Image 14 – R Shiny app with R Highcharts mapsIt could use some visual tweaking, but that’s a topic for another time. For now, we can safely conclude that R Highcharts maps have no trouble integrating with R Shiny, so you can have no doubt about using them in your projects.

Summing up R Highcharts MapsWhen it comes to building engaging R Shiny apps, interactivity is the key. There’s no reason to opt for static charts if animated and interactive packages exist. R Highcharts is one of these packages, and you now know everything you need to produce full-fledged data-driven dashboards showcasing maps and charts.

Stay tuned to the Appsilon blog and our newsletter, Shiny Weekly, for further guides and tips on R Shiny and Highcharts, as there sure is plenty more to come.

Another superb R package for spatial visualization is Leaflet – Read our detailed guide to get started.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Highcharts: How to Make Interactive Maps for R and R Shiny

View Details

[This article was first published on Open Analytics, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.ShinyProxy 3.1.0Last year we released a major update of ShinyProxy, bringing High Availability,Scaling, App parameters, improved integration with Kubernetes and much more.Today’s release goes even further, brining Pre-Initialization, Container sharingand even a new container backend. Along with these major features, the releaseis packed with numerous small improvements. ShinyProxy is a novel, open sourceplatform to deploy Shiny apps for the enterprise or larger organizations.Nevertheless, ShinyProxy is also an excellent choice for deploying a handful ofapps in smaller organizations. While ShinyProxy is generally used to host Shinyapplications, it has always been designed with the idea to host any applicationthat can be packaged into a Docker container. For example, ShinyProxy is usedmore and more to host complete development environments. This blog post coverssome major improvements of this release. All changes in this version can befound in the release notes.

Pre-initialization and Container SharingFor as long as ShinyProxy exists, ShinyProxy starts a new container every time auser starts an application. This container is dedicated to this user and isnever used by other users. This has worked great for many years and gives thepossibility of using ShinyProxy for hosting IDE’s and development tools, sinceeach container can be customized for this specific user and session.Furthermore, as every container is started on-demand, it allows to host a bigset of applications, without the need for all these applications to run all thetime. While this approach has many advantages, it also has the consequence thata user has to wait for both the container and application to start up and thatit requires more (CPU and memory) resources when more users are usingShinyProxy. ShinyProxy 3.1.0 introduces the concept of containerpre-initialization and container sharing. For every app, you can specify anumber of seats ShinyProxy has to pre-initialize (e.g. X seats). WhenShinyProxy starts up, it will start X containers. As soon as a user wants touse an application, ShinyProxy assigns a container (seat) to the user.Therefore, the user has almost no waiting time for the container to start up. Assoon as a seat gets claimed, ShinyProxy will scale up the number ofcontainers, such that there are again X seats ready to be used by new users.Similarly, when a user stops using an app, the seat is released and ShinyProxywill scale-down the pool of available seats (and thus the number of containers).In addition, you can specify how many seats can be run on a single container,therefore allowing to share a single container among multiple users.

Enabling pre-initialization for an app is very easy, just add theproperty minimum-seats-available: X to your app specification. With thisoption, ShinyProxy will ensure there are always X pre-initialized containers.New users will be immediately assigned to a pre-initialized container. Addingthe property seats-per-container: Y, allows a single container to be usedby Y people at the same time. ShinyProxy takes both numbers into account, inorder to determine the number of underlying containers. It is possible to usethese new features with every container backend.

In many cases, an app hosted on ShinyProxy needs to know the username (andgroups) of the current user. Therefore, ShinyProxy adds the username (andgroups) as environment variables to the container. However, this would not workwhen the container is created up-front or used by multiple users (environmentvariables cannot be updated after the container has started). Instead,ShinyProxy can now send the username and groupsas HTTP headersto the app. It is even possible tosend custom headers!

The monitoring stackcontains a new set of dashboards, providing the same level of monitoring forpre-initialized and shared containers as for regular apps.

We believe this feature is a big improvement for ShinyProxy. Together with thevarious performance improvements made in this release, this makes it easier touse ShinyProxy for public apps, which are used by a large amount of users. Checkoutthe full documentationto start using it.

AWS ECS BackendUp to now ShinyProxy supported three container backends: Docker, Docker Swarmand Kubernetes. The Docker backend is very easy to set up, but does not providethe same scalability and flexibility as the Kubernetes backend. ShinyProxy 3.1.0now supports AWS ECS as a new backend, allowing to deploy ShinyProxy withoutmanaging servers or complex setups, but with greater scalability. The backenduses AWS Fargate and supports most of the ECS features: integration withCloudWatch, Task role support (e.g. to access S3) and EFS volumes. We provide aready to use and fullydocumented Terraform deployment example.A downside of AWS ECS is that in most cases it takes more time to start thecontainer compared to the other backends. But of course it can be used togetherwith the pre-initialized container feature, reducing the startup time to lessthan a second. The architecture of ShinyProxy, allows to easily add newcontainer backends.

Custom container namesThis release streamlines the names of the containers created by ShinyProxy.Every backend now uses the id of the app as part of the name, making it easierto find which container corresponds to which app. For example, when usingDocker, the name of the container is sp-container-#{proxy.id}-0,where #{proxy.id} is replaced by the id of the app. It’s even possible tochangethe name by providing a SpEL expression.

Up to date Docker libraryAs with every release, all dependencies (including Spring Boot) have beenupdated. Furthermore, ShinyProxy now uses a better maintained Docker clientlibrary. This new library no longer relies on old dependencies and fixes thecompatibility of ShinyProxy with recent Docker releases.

Limitation on the number of usersWhen making applications available to a broad audience, you sometimes want tolimit the total number of apps running on your server or cloud infrastructure.It’s now possible to specify such a limit, both on a global and per-app level.The container sharing feature is aware of this limit and uses it to determinethe maximum number of containers it can create.

Documentation and DemosIn addition to all the improvements to the ShinyProxy code, a lot of time wasinvested to improve the ShinyProxy documentation. First of all, we now have anoverview of how to write ShinyProxy configuration (e.g. how touse listsor environment variablesin the config). The configuration of apps has been divided into smallersections, making it more clear which properties belong together. All sections inthe documentation are now numbered, and together with improvements to the ToC, thismakes it easier to browse through the always growing number of configurationproperties. The OpenID documentation has been rewritten, to better explain theprocess of configuring it, while we also extendedthe provider specific examples.Finally, we regularly addnew demos, showinghow to deploy many kinds of apps in ShinyProxy.

Full release notes can be found on the downloads page and updated documentationcan be found on https://shinyproxy.io. As always community support on this newrelease is available at

https://support.openanalytics.eu

Don’t hesitate to send in questions or suggestions and have fun with ShinyProxy!

To leave a comment for the author, please follow the link and comment on their blog: Open Analytics.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: ShinyProxy 3.1.0

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.DataCamp is launching DataLab, an AI-enabled data notebook to make it easier and faster than ever before to go from data to insight, regardless of technical ability.

The challenge DataLab SolvesDataCamp’s mission to democratize data skills has translated into its leading online data and AI learning platform.

Alongside learning, intuitive data tooling is vital for modern professionals to feel empowered to make data-driven decisions. But more often than not, tools today are getting in the way rather than helping us. Depending on your skill set, you need different tools (Excel, a SQL editor, a Python notebook).

As a result, insights are scattered across functions and collaboration becomes a drag. And, insights are often separated by technical skills or needing to go through specialist data teams to get answers.

The solution: Democratized insightsDataLab is an AI-powered chat interface specifically tailored for data analytics. The steps are simple: attach a data source, ask a question, and iterate your way to the insight you need, just like you would with a technically skilled colleague.

DataLab is an AI-powered data notebook to chat with your data

DataLab’s key features: Powered by code. The AI Assistant answers your questions by writing and running code and interpreting the outputs. With DataLab, you can seamlessly switch to a fully-featured notebook view with all of the generated code, that you can review, tweak, extend, and share. * Easy data access, wherever it lives. From CSV files and Google Sheets data to Snowflake and BigQuery: DataLab seamlessly and securely connects to all your data sources. DataLab’s AI architecture knows where to look and achieves best-in-class results by leveraging organizational knowledge, your previous activity, and industry best practices. * Built-in reporting.* Forget about copy-pasting across tools or sharing outdated screenshots of your findings. As you’re getting answers to questions in DataLab, you’re accumulating a live-updating report that you can customize before sharing with others with a single click.

DataLab’s chat interface looks similar to ChatGPT, and that’s the point! OpenAI managed to package extremely sophisticated technology into an intuitive interface.

Just because technology is complex, doesn’t mean the interface needs to be.

DataLab’s path so far
DataLab is the next iteration of DataCamp Workspace, an online data notebook with more than 50,000 monthly active users and support for SQL, Python, and R.

With the breakthrough of GPT-3.5 and many other LLMs in early 2023, DataCamp saw the opportunity to make their users even more effective and Workspace accessible to a bigger user base, both for study and work.

DataCamp added powerful AI features to write, update, fix, and explain code. The built-in Workspace AI Assistant is smarter than ChatGPT: it takes more context into account (variables, table names, and column types), leading to higher-quality suggestions and a faster workflow.

DataLab is Workspace’s notebook environment plus an intuitive AI-powered chat interface. This means you’re getting the power of AI and a fully-fledged coding environment without switching tools. Real-time collaboration, scheduling, version history, role-based access control, you name it: you’re tapping into years of work building a data notebook that is intuitive and delightful to use, whether you’re learning on DataCamp or working through your unique projects.

I have used Workspace in the past, what happens to my work?
All your workspaces (now workbooks) you created in the past will stay available and continue to function. You can continue using DataLab for free (three workbooks, 20 AI Assistant prompts, basic hardware); all you need is your DataCamp account.

Users on an existing Workspace Premium subscription can access DataLab Premium, featuring unlimited workbooks, unlimited AI Assistant prompts, powerful hardware, and more.

Try it out for free
* Visit www.datacamp.com/datalab * Pricing: DataLab is publicly available now and free to try (up to 3 projects, up to 20 AI Assistant prompts). For $99/user/year you can create unlimited projects, use the AI assistant without limitations, and access more powerful hardware. See www.datacamp.com/datalab/pricing for more details. * Interested in leveraging DataLab to drive insight in business data? Book a demo

DataCamp Launches DataLab: An AI-Enabled Data Notebook was first posted on May 6, 2024 at 7:09 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: DataCamp Launches DataLab: An AI-Enabled Data Notebook

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Cluster Analysis in R which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Cluster Analysis in R

Date: Thursday, June 6th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Sejal Davla is a neuroscientist and data scientist who works with industry and government clients on projects at the intersection of science, data, and policy. She received her PhD in neuroscience from McGill University in Canada, where her research identified new pathways in brain development and sleep. She is an advocate for open science and reproducibility and runs R programming workshops to promote best data practices.

Description: Some datasets are unlabeled without obvious classifiers. Unsupervised machine learning methods, such as clustering, allow finding patterns and homogeneous subgroups in unlabeled data. This workshop will cover the basics of cluster analysis and how to perform clustering using k-means and hierarchical clustering methods. The goal of the workshop is to help identify datasets for clustering, learn to visualize and interpret models, validate clusters, and highlight practical issues.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Cluster Analysis in R workshop was first posted on May 6, 2024 at 3:20 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cluster Analysis in R workshop

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn the world of data analysis and statistics, one of the key challenges is selecting the best model to describe and analyze your data. This decision is crucial because it impacts the accuracy and reliability of your results. Among the many tools available, the Akaike Information Criterion (AIC) stands out as a powerful method for comparing different models and choosing the most suitable one.

Today we will go through an example of model selection using the AIC, specifically focusing on its application to various statistical distributions available in the TidyDensity package. TidyDensity, a part of the healthyverse ecosystem, offers a comprehensive suite of tools for data analysis in R, including functions to compute AIC scores for different probability distributions.

What is AIC?The Akaike Information Criterion (AIC) is a mathematical tool used for model selection. It balances the goodness of fit of a model with its complexity, penalizing overly complex models to prevent overfitting. In simpler terms, AIC helps us choose the most effective model that explains our data without being too complex.

Exploring TidyDensity’s Distribution FunctionsTidyDensity provides a range of utility functions prefixed with util_ that calculate the AIC for specific probability distributions. Let’s take a closer look at some of these functions:

Beta Distribution (util_beta_aic()): Computes the AIC for a beta distribution, which is often used to model random variables constrained to the interval [0, 1].
Binomial Distribution (util_binomial_aic()): Calculates the AIC for a binomial distribution, commonly used to model the number of successes in a fixed number of independent trials.
Cauchy Distribution (util_cauchy_aic()): Computes the AIC for a Cauchy distribution, known for its symmetric bell-shaped curve.
Exponential Distribution (util_exponential_aic()): Determines the AIC for an exponential distribution, frequently used to model the time between events in a Poisson process.
Normal Distribution (util_normal_aic()): Computes the AIC for a normal distribution, which is ubiquitous in statistics due to the central limit theorem.

These are just a few examples of the distribution-specific AIC functions available in TidyDensity. Each function evaluates the goodness of fit of a particular distribution to your data and provides an AIC score, aiding in the selection of the most appropriate model.

How to Use AIC for Model SelectionUsing these functions in TidyDensity is straightforward. Simply pass your data to the desired distribution function, and it will return the AIC score. Lower AIC values indicate a better fit, so the distribution with the lowest AIC is typically chosen as the optimal model.

Here’s a simplified example of how you might use these functions:

```

Load TidyDensity librarylibrary(TidyDensity)# Generate some sample datadata <- rnorm(100, mean = 0, sd = 1)# Compute AIC for normal distributionnormal_aic <- util_normal_aic(data)# Compute AIC for exponential distributioncauchy_aic <- util_cauchy_aic(data)# Compare AIC scoresif (normal_aic < cauchy_aic) { print("Normal distribution is a better fit.")} else { print("Cauchy distribution is a better fit.")}

```

[1] "Normal distribution is a better fit."

cat("Normal AIC: ", normal\_aic, "\n")

Normal AIC: 285.9777

cat("Cauchy AIC: ", cauchy\_aic)

Cauchy AIC: 317.1025 ConclusionIn conclusion, the Akaike Information Criterion (AIC) plays a crucial role in statistical modeling and model selection. The TidyDensity package enhances this capability by providing specialized functions to compute AIC scores for various probability distributions. By leveraging these functions, data analysts and researchers can make informed decisions about which distribution best describes their data, leading to more robust and accurate statistical analyses.

If you’re interested in harnessing the power of AIC and exploring different probability distributions in R, be sure to check out TidyDensity and incorporate these tools into your data analysis toolkit. Happy modeling!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Model Selection with TidyDensity: Understanding AIC for Statistical Distributions

View Details

[This article was first published on R Stories, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Introduction to 2024For so many reasons, such as election hypes, pandemic, etc., people with trading, investing or retirement portfolios may want to know how their financial portfolios (or instruments such as stocks or ETFs in their portfolios) performed over some years, months, weeks or days. They may also want to know the average monthly, yearly, weekly or daily returns, starting from some fixed time of start in the past to the present or recent time.

Almost all portfolio managers measure performance with reference to a benchmark [3].In this short note, we will consider the historical data of the Standard and Poor’s 500 Index (S&P 500, symbol=^GSPC) from Yahoo! Finance, which is widely regarded as the best gauge of large-cap U.S. equities. Other well known benchmarks include DOW-30, NASDAQ-100, and the Russell 2000 Index for small-caps.

We will then outline a simple way to visualize or summarize monthly returns as well as average monthly returns using R. Interested readers can modify the instrument, period and length of time to their preference.

We start by installing the R packages that will be needed to produce libraries later. For more information about one of the key packages used here, the tidyquant package, see [2].

Getting and Preparing DataWe will get the data for the S&P 500 Index, symbol = ^GSPC, from Yahoo! Finance. We will then prepare the data for visualization and/or Summarization of results as needed.

To leave a comment for the author, please follow the link and comment on their blog: R Stories.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Trial Post for 2024

View Details

[This article was first published on R Stories, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.To leave a comment for the author, please follow the link and comment on their blog: R Stories.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Test title

View Details

[This article was first published on R Stories, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.manual test of posta simple way to visualize or summarize monthly returns as well as average monthly returns using R. Interested readers can modify the instrument, period and length of time to their preference.

We start by installing the R packages that will be needed to produce libraries later. For more information about one of the key packages used here, the tidyquant package, see [2].

Getting and Preparing DataWe will get the data for the S&P 500 Index, symbol = ^GSPC, from Yahoo! Finance. We will then prepare the data for visualization and/or Summarization of results as needed.

R MarkdownThis is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

summary(cars)## speed dist## Min. : 4.0 Min. : 2.00## 1st Qu.:12.0 1st Qu.: 26.00## Median :15.0 Median : 36.00## Mean :15.4 Mean : 42.98## 3rd Qu.:19.0 3rd Qu.: 56.00## Max. :25.0 Max. :120.00 Including PlotsYou can also embed plots, for example:

Note that the echo = FALSE parameter was added to the code chunk to prevent printing of the R code that generated the plot.

To leave a comment for the author, please follow the link and comment on their blog: R Stories.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: manual test

View Details

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This week’s post is about mlsauce (again), and LSBoost in particular. No new working paper (still working on it), but:

An updated R version, working at least on Linux and macOS (Windows users, if not working on your machine, give a try to the Windows Subsystem for Linux, WSL)
A new updated documentation page
My first StackOverflow question ever (still unanswered)

The examples below probably include some kind of leakage (great if you can spot it), but take it as an illustration.

0 – import packagesImporting mlsauce from GitHub remains the preferred way to install it.

```

!pip install numpy matplotlib scikit-learn!pip install git+https://github.com/Techtonique/mlsauce.git --verbose# Importing necessary librariesimport mlsauce as msimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.datasets import load_breast_cancerfrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import KernelPCA # Non-linear dimensionality reduction through the use of kernelsfrom sklearn.model_selection import cross_val_score, train_test_splitfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_score

1 – Data preprocessing

Load breast cancer datasetdata = load_breast_cancer()X = data.datay = data.targetprint(X.shape)print(y.shape)(569, 30)(569,)

1 – 1 Kernel PCA features

Standardize the featuresscaler = StandardScaler()X_scaled = scaler.fit_transform(X)# Perform Kernel PCA to extract 2 'good' features# (easier to visualize)kpca = KernelPCA(n_components=2)X_kpca = kpca.fit_transform(X_scaled)# Splitting the dataset into training and testing setsX_train_kpca, X_test_kpca, y_train, y_test = train_test_split(X_kpca, y, test_size=0.2, random_state=32)# Plotting the two principal componentsplt.figure(figsize=(8, 6))plt.scatter(X_test_kpca[:, 0], X_test_kpca[:, 1], c=y_test, cmap='viridis')plt.xlabel('Kernel Principal Component 1')plt.ylabel('Kernel Principal Component 2')plt.title('Kernel PCA of Breast Cancer Dataset')plt.colorbar(label='Malignant (0) / Benign (1)')plt.show()

1 – 2 ‘Important’ features

Training a Random Forest classifierrf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)rf_classifier.fit(X, y)# Feature importancesimportances = rf_classifier.feature_importances_print(importances)indices = np.argsort(importances)[::-1]print(indices)# Select top 2 featurestop_two_indices = indices[:2]print(data.feature_names[top_two_indices])X_rf = X[:,top_two_indices]# Splitting the dataset into training and testing setsX_train_rf, X_test_rf, y_train, y_test = train_test_split(X_rf, y, test_size=0.2, random_state=32)# Plotting the two principal componentsplt.figure(figsize=(8, 6))plt.scatter(X_test_rf[:, 0], X_test_rf[:, 1], c=y_test, cmap='viridis')plt.xlabel("Most 'important' feature 1")plt.ylabel("Most 'important' feature 2")plt.title('Response for Breast Cancer Dataset')plt.colorbar(label='Malignant (0) / Benign (1)')plt.show()[0.03484323 0.01522515 0.06799034 0.06046164 0.00795845 0.01159704 0.06691736 0.10704566 0.00342279 0.00261508 0.0142637 0.00374427 0.01008506 0.02955283 0.00472157 0.00561183 0.00581969 0.00375975 0.00354597 0.00594233 0.08284828 0.01748526 0.0808497 0.13935694 0.01223202 0.01986386 0.03733871 0.13222509 0.00817908 0.00449731][23 27 7 20 22 2 6 3 26 0 13 25 21 1 10 24 5 12 28 4 19 16 15 14 29 17 11 18 8 9]['worst area' 'worst concave points']

2 – Adjust LSBoostClassifier !pip install GPoptimport GPopt as gpimport mlsauce as msfrom sklearn.model_selection import cross_val_scoreopt_objects_lsboost = []def lsboost_cv(X_train, y_train, n_estimators=100, learning_rate=0.1, n_hidden_features=5, reg_lambda=0.1, dropout=0, tolerance=1e-4, n_clusters=2, seed=123, solver="ridge"): estimator = ms.LSBoostClassifier(n_estimators=int(n_estimators), learning_rate=learning_rate, n_hidden_features=int(n_hidden_features), reg_lambda=reg_lambda, dropout=dropout, tolerance=tolerance, n_clusters=int(n_clusters), seed=seed, solver=solver, verbose=0) return -cross_val_score(estimator, X_train, y_train, scoring='f1_macro', cv=5).mean()def optimize_lsboost(X_train, y_train, solver="ridge"): # objective function for hyperparams tuning def crossval_objective(x): return lsboost_cv( X_train=X_train, y_train=y_train, n_estimators=int(x[0]), learning_rate=x[1], n_hidden_features=int(x[2]), reg_lambda=x[3], dropout=x[4], tolerance=x[5], n_clusters=int(x[6]), solver = solver) gp_opt = gp.GPOpt(objective_func=crossval_objective, lower_bound = np.array([ 10, 0.001, 5, 1e-2, 0, 0, 0]), upper_bound = np.array([250, 0.4, 250, 1e4, 0.7, 1e-1, 4]), params_names=["n_estimators", "learning_rate", "n_hidden_features", "reg_lambda", "dropout", "tolerance", "n_clusters"], n_init=10, n_iter=190, seed=123) return {'parameters': gp_opt.optimize(verbose=2, abs_tol=1e-2), 'opt_object': gp_opt}opt_objects_lsboost.append(optimize_lsboost(X_train_kpca, y_train, solver="ridge"))opt_objects_lsboost.append(optimize_lsboost(X_train_rf, y_train, solver="ridge")) 3 – Graphs display(opt_objects_lsboost[0]['parameters'].best_params)display(opt_objects_lsboost[1]['parameters'].best_params)opt_objects_lsboost[0]['parameters'].best_params['n_estimators'] = int(opt_objects_lsboost[0]['parameters'].best_params['n_estimators'])opt_objects_lsboost[1]['parameters'].best_params['n_estimators'] = int(opt_objects_lsboost[1]['parameters'].best_params['n_estimators'])opt_objects_lsboost[0]['parameters'].best_params['n_hidden_features'] = int(opt_objects_lsboost[0]['parameters'].best_params['n_hidden_features'])opt_objects_lsboost[1]['parameters'].best_params['n_hidden_features'] = int(opt_objects_lsboost[1]['parameters'].best_params['n_hidden_features'])opt_objects_lsboost[0]['parameters'].best_params['n_clusters'] = int(opt_objects_lsboost[0]['parameters'].best_params['n_clusters'])opt_objects_lsboost[1]['parameters'].best_params['n_clusters'] = int(opt_objects_lsboost[1]['parameters'].best_params['n_clusters']){'n_estimators': 221.10595703125, 'learning_rate': 0.12772097778320313, 'n_hidden_features': 45.053253173828125, 'reg_lambda': 2496.6505697631837, 'dropout': 0.2851226806640625, 'tolerance': 0.0047698974609375, 'n_clusters': 3.1986083984375}{'n_estimators': 193.544921875, 'learning_rate': 0.3466668701171875, 'n_hidden_features': 208.9971923828125, 'reg_lambda': 1866.4632116699217, 'dropout': 0.37947998046875, 'tolerance': 0.01290283203125, 'n_clusters': 3.04443359375}import matplotlib.pyplot as pltimport numpy as npfrom matplotlib.colors import ListedColormapfrom sklearn.inspection import DecisionBoundaryDisplayfrom sklearn.pipeline import make_pipelinefrom sklearn.ensemble import GradientBoostingClassifierclassifiers = [RandomForestClassifier(), GradientBoostingClassifier(), ms.LSBoostClassifier(opt_objects_lsboost[0]['parameters'].best_params), ms.LSBoostClassifier(opt_objects_lsboost[1]['parameters'].best_params)]names = ["rf", "gb", "lsboost_pca", "lsboost_rf"]figure = plt.figure(figsize=(27, 9))i = 1datasets = [(X_kpca, y), (X_rf, y)]# iterate over datasetsfor ds_cnt, ds in enumerate(datasets): # preprocess dataset, split into training and test part X, y = ds[0], ds[1] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.4, random_state=42 ) x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5 y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5 # just plot the dataset first cm = plt.cm.RdBu cm_bright = ListedColormap(["#FF0000", "#0000FF"]) ax = plt.subplot(len(datasets), len(classifiers) + 1, i) if ds_cnt == 0: ax.set_title("Input data") # Plot the training points ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright, edgecolors="k") # Plot the testing points ax.scatter( X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, alpha=0.6, edgecolors="k" ) ax.set_xlim(x_min, x_max) ax.set_ylim(y_min, y_max) ax.set_xticks(()) ax.set_yticks(()) i += 1 # iterate over classifiers for name, clf in zip(names, classifiers): ax = plt.subplot(len(datasets), len(classifiers) + 1, i) clf = make_pipeline(StandardScaler(), clf) clf.fit(X_train, y_train) try: score = clf.score(X_test, y_test) except: # no scoring method available yet for prediction sets score = np.mean(clf.predict_proba(X_test).argmax(axis=1) == y_test) DecisionBoundaryDisplay.from_estimator( clf, X, cmap=cm, alpha=0.8, ax=ax, eps=0.5 ) # Plot the training points ax.scatter( X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright, edgecolors="k" ) # Plot the testing points ax.scatter( X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, edgecolors="k", alpha=0.6, ) ax.set_xlim(x_min, x_max) ax.set_ylim(y_min, y_max) ax.set_xticks(()) ax.set_yticks(()) if ds_cnt == 0: ax.set_title(name) ax.text( x_max - 0.3, y_min + 0.3, ("%.2f" % score).lstrip("0"), size=15, horizontalalignment="right", ) i += 1plt.tight_layout()plt.show() 43%|████▎ | 94/221 [00:00<00:00, 178.28it/s] 26%|██▋ | 51/193 [00:02<00:07, 18.66it/s] 54%|█████▍ | 51/94 [00:00<00:00, 449.07it/s]100%|██████████| 51/51 [00:00<00:00, 61.11it/s] ``` To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: mlsauce’s v0.18.2: various examples and benchmarks with dimension reduction

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post optim Function in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

optim Function in R, we will explore how to apply a general-purpose optimization using the optim function in R programming language.

We will create example data and then demonstrate the usage of the optim function to minimize the residual sum of squares.

optim Function in RFirst, let’s create the example data we will use for this tutorial:

```

Set a random seed for reproducibilityset.seed(123)# Create random datax <- rnorm(500)y <- rnorm(500) + 0.7 * x# Combine x and y into a data framedata <- data.frame(x, y)# Print the head of the datahead(data)

`` This code generates a data frame with two numeric variables,xandy`.

x y1 -0.56047565 -0.99422582 -0.23017749 -1.15482283 1.55870831 2.11788094 0.07050839 0.80041725 0.12928774 -1.41866516 1.71506499 1.1053980 Example: Applying optim Function in R

Now, let’s apply the optim function to minimize the residual sum of squares. We will manually create a function for this purpose:

```

Manually create a function for residual sum of squaresoptm_function <- function(data, par) { with(data, sum((par[1] + par[2] * x - y)^2))}

`` Next, we can use theoptim` function as shown below.

The par argument specifies the initial values for the parameters to be optimized over, the fn argument specifies our function, and the data argument specifies our data frame.

We store the output of the optim function in the optim_output object:

```

Applying optimoptim_output <- optim(par = c(0, 1), fn = optm_function, data = data)

`` Finally, we can visualize our results in a plot. We will compare the results of theoptimfunction with those of a conventional linear model provided by thelm` function:

```

Set plot parameterspar(mfrow = c(1, 2))# Plot results of the optim functionplot(data$x, data$y, main = "optim Function")abline(optim_output$par[1], optim_output$par[2], col = "red")# Plot results of the lm functionplot(data$x, data$y, main = "lm Function")abline(lm(y ~ x, data), col = "green")

`` The resulting plot (Figure 1) should show that both theoptimandlmfunctions returned the same result, indicating that our manual optimization using theoptim`.

Apply Central Limit Theorem in R » Data Science Tutorials
Gamma distribution in R » Data Science Tutorials
Surprising Things You Can Do With R »
Bias Variance Tradeoff Machine Learning Tutorial »
How to Find Unmatched Records in R » Data Science Tutorials

The post optim Function in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: optim Function in R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Welcome to the third (and final) part of our R Highcharts article series. So far, you’ve learned how to make basic interactive charts, and how to make drilldown charts. These two articles are a must-read before going over this one, since today the focus will be on code rather than explanations.

After reading today’s piece, you’ll know how to create interactive and animated dashboards with the R Shiny Highcharts module – both through basic charts and drill-downs. We’ll also go over some basics of R Shiny, such as filtering and styling. Let’s dig in!

Are your R/Shiny team skills up to date? Take them to next level by reading our ebook.

Table of contents:* Exploring R Shiny Highcharts Dashboard Elements * Building the R Shiny Highcharts Dashboard * Summing up R Shiny Highcharts

Exploring R Shiny Highcharts Dashboard ElementsThe application we’re about to create will contain a couple of elements, so it’s a good idea to explain what’s where and what does what. It will be based on the Gapminder datatset, and will show information such as life expectancy, population, and GDP per continent, year, and country.

Here’s a sketch of the application:

Image 1 – Sketch of our application‍

As you can see, the application starts with a couple of filters. These allow the user to control the continent and start/end years. Not all values and charts will be affected by the year filters, as some of them will show only the most recent values (2007).

In the Latest stats section, you’ll see four boxes showing information on the number of countries and other various statistics for a given continent. The year filters won’t do anything to these boxes, as only the most recent values will be shown.

Below, we have two Summary stats charts. These will be a bar chart showing median life expectancy by year, and a line chart showing population by year. These two are affected by both the continent and year filter values. It’s also worth noting that they show an average across all countries in a continent, so some degree of skew is to be expected.

And finally, we have the Drilldown section. It contains only one chart which shows the most recent population across the country for a given continent. When you click on an individual bar, you’ll see the population stats across years only for the given country. Neat!

Let’s start building the thing!

Building the R Shiny Highcharts DashboardThis section will be quite code-intensive and will require fundamental R Shiny knowledge. If you’re a complete beginner, we recommend checking our library of Shiny articles first.

Let’s start with the easiest part – summary stats.

Summary Stats CardsTruth be told, filters and summary stats have nothing to do with R Highcharts, but will take our dashboard to the next level. As mentioned before, we’re working with the Gapminder dataset, and the idea is to allow the user to select a continent and a year range from which the dashboard contents will get updated.

The Shiny app’s UI is divided into two parts – sidebarPanel() and mainPanel(). The first one contains all UI controls (filters), while the second one renders the contents.

Regarding filters, R’s unique() function is quite useful here, as it allows us to grab only distinct elements from a categorical variable. For the years filters, we’ll remove the highest year from the min filter and the lowest year from the max filter.

The contents of mainPanel() are organized in a way that we have one container with four containers inside it – each of which contains one summary statistic.

As for the server(), we’re simply creating a reactive dataset that calculates the summary statistics based on the selected continent and then using renderText() to display the values for each summary statistics card.

Here’s the full code snippet:

library(shiny)library(dplyr)library(purrr)library(gapminder)library(highcharter)ui <- fluidPage( sidebarLayout( sidebarPanel( titlePanel("R Shiny Highcharts"), selectInput( inputId = "inContinent", label = "Continent:", choices = unique(gapminder$continent), selected = "Europe" ), selectInput( inputId = "inYearMin", label = "Start year:", choices = unique(gapminder$year)[1:length(unique(gapminder$year)) - 1], selected = min(gapminder$year) ), selectInput( inputId = "inYearMax", label = "End year:", choices = unique(gapminder$year)[2:length(unique(gapminder$year))], selected = max(gapminder$year) ), width = 3 ), mainPanel( tags$h3("Latest stats:"), tags$div( tags$div( tags$p("#Countries"), textOutput(outputId = "outNCountries") ), tags$div( tags$p("Median life exp."), textOutput(outputId = "outMedLifeExp") ), tags$div( tags$p("Median population"), textOutput(outputId = "outMedPop") ), tags$div( tags$p("Median GDP"), textOutput(outputId = "outMedGDP") ) ), width = 9 ) ))server <- function(input, output) { data\_cards <- reactive({ gapminder %>% filter( continent == input$inContinent, year == max(year) ) %>% summarise( nCountries = n\_distinct(country), medianLifeExp = median(lifeExp), medianPopM = median(pop / 1e6), medianGDP = median(gdpPercap) ) }) output$outNCountries <- renderText({ data\_cards()$nCountries }) output$outMedLifeExp <- renderText({ paste(round(data\_cards()$medianLifeExp, 1), "years") }) output$outMedPop <- renderText({ paste0(round(data\_cards()$medianPopM, 2), "M") }) output$outMedGDP <- renderText({ paste0("$", round(data\_cards()$medianGDP, 2)) })}shinyApp(ui = ui, server = server) And here’s what the R Shiny Highcharts application looks like:

‍

Image 2 – R Shiny Highcharts dashboard (1)‍

The values are there and correct, but we’d benefit tremendously from a bit of styling. Let’s introduce CSS next.

Basic Dashboard StylingYou can use both CSS and SCSS to style your R Shiny dashboards. We’ll stick with the first option.

Create a www/styles.css file and paste the following inside it:

@import url('https://fonts.googleapis.com/css2?family=Poppins:ital,wght@0,700;1,400&display=swap');* { margin: 0; padding: 0; box-sizing: border-box;}body { font-family: 'Poppins', sans-serif; font-weight: 400;}.main-container { padding-top: 1rem;}.stat-card-container { display: flex; justify-content: space-between; column-gap: 1rem;}.stat-card { border: 2px solid #f2f2f2; border-bottom: 2px solid #0198f9; width: 100%; padding: 0.5rem 0 0.5rem 1rem;}.stat-card > p { text-transform: uppercase; color: #808080;}.stat-card > div.shiny-text-output { font-size: 3rem; font-weight: 700;} ‍

Long story short, this piece of code will change the overall font, reset a couple of styles, and make our dashboard nicer to look at.

The only problem is – the CSS file isn’t connected with R Shiny.

What you’ll need to do is to add a link tag to the head of the application and reference our styles.css file (Shiny assumes it’s located in the www folder). You’ll also want to add CSS class names to HTML attributes by piping the tagAppendAttributes(class = "class-name") function at the end of it.

Only the code in ui has changed, server() is identical as before:

‍

Image 3 – R Shiny Highcharts dashboard (2)We now have everything needed to introduce some visualizations with Highcharts.

Adding Basic R HighchartsSo far, we’ve successfully set the stage, so let’s dive into the good stuff now. This section will walk you through two basic Highcharts visualizations in R Shiny, and these will show the following:

Median life expectancy by year: For all countries in a selected continent and a selected year range.
Median GDP by year: Once again, the median is calculated on a per-country level for a continent and a selected year range.

You can work with Highcharts visualizations in R Shiny by calling the highchartOutput() function in ui(). It accepts an outputId and an optional height parameter, so you can easily tweak the basic looks straight from R.

Down in server(), it’s a familiar situation (if you’ve been following along with the series). We have a new data frame – data_charts – and it contains the filtered and aggregated life expectancy and GDP data. The renderHighchart() is used to create a Highcharts visualization, and it accepts a block of familiar functions:

library(shiny)library(dplyr)library(purrr)library(gapminder)library(highcharter)ui <- fluidPage( tags$head( tags$link(rel = "stylesheet", type = "text/css", href = "styles.css") ), sidebarLayout( sidebarPanel( titlePanel("R Shiny Highcharts"), selectInput( ... ), selectInput( ... ), selectInput( ... ), width = 3 ), mainPanel( tags$h3("Latest stats:"), tags$div( ... ) %>% tagAppendAttributes(class = "stat-card-container"), tags$div( tags$h3("Summary stats:"), tags$div( tags$div( highchartOutput(outputId = "chartLifeExpByYear", height = 500) ) %>% tagAppendAttributes(class = "chart-card"), tags$div( highchartOutput(outputId = "chartGDPByYear", height = 500) ) %>% tagAppendAttributes(class = "chart-card"), ) %>% tagAppendAttributes(class = "base-charts-container") ) %>% tagAppendAttributes(class = "card-container"), width = 9 ) %>% tagAppendAttributes(class = "main-container") ))server <- function(input, output) { data\_cards <- reactive({ ... }) data\_charts <- reactive({ gapminder %>% filter( continent == input$inContinent, between(year, as.integer(input$inYearMin), as.integer(input$inYearMax)) ) %>% group\_by(year) %>% summarise( medianLifeExp = round(median(lifeExp), 1), medianGDP = round(median(gdpPercap), 2) ) }) output$outNCountries <- renderText({ ... }) output$outMedLifeExp <- renderText({ ... }) output$outMedPop <- renderText({ ... }) output$outMedGDP <- renderText({ ... }) output$chartLifeExpByYear <- renderHighchart({ hchart(data\_charts(), "column", hcaes(x = year, y = medianLifeExp), color = "#0198f9", name = "Median life expectancy") |> hc\_title(text = "Median life expectancy by year", align = "left") |> hc\_xAxis(title = list(text = "Year")) |> hc\_yAxis(title = list(text = "Life expectancy")) }) output$chartGDPByYear <- renderHighchart({ hchart(data\_charts(), "line", hcaes(x = year, y = medianGDP), color = "#800000", name = "Median GDP") |> hc\_title(text = "Median GDP by year", align = "left") |> hc\_xAxis(title = list(text = "Year")) |> hc\_yAxis(title = list(text = "GDP")) })}shinyApp(ui = ui, server = server) We’ve also added some CSS classes to app.R, so here’s the corresponding CSS code for them:

.card-container { padding-top: 2rem;}.base-charts-container { display: flex; justify-content: space-between; column-gap: 1rem;}.chart-card { border: 2px solid #f2f2f2; width: 50%;} And this is what the application looks like now:

‍

Image 4 – R Shiny Highcharts dashboard (3)‍

Both charts look amazing, are fully interactive, and have a nice animation when first loading the dashboard or when refreshing the data.

The only thing left to do is to include a drilldown chart, so let’s go over that next.

Adding a Highcharts Drilldown ChartA drilldown chart will allow the user to click on individual chart elements to see a new, drilled-down version of the visualization. In our case, we’ll have a per-country column chart of the population for a selected continent (only for the latest year) by default. When a column is clicked, a new column chart appears showing only the population for the clicked country through time.

Seems easy enough, but remember – we’ll need two datasets. The first one is for the default visualization, and the second is for the drilled-down chart. If you’ve read the previous article you already know what you need to do.

To recap, the drilled-down dataset needs a column of type list that contains the data that’ll be visible when a single column is clicked. The hc_drilldown() function is then used to enable drilldown mode.

Here’s the code:

library(shiny)library(dplyr)library(purrr)library(gapminder)library(highcharter)ui <- fluidPage( tags$head( tags$link(rel = "stylesheet", type = "text/css", href = "styles.css") ), sidebarLayout( sidebarPanel( titlePanel("R Shiny Highcharts"), selectInput( ... ), selectInput( ... ), selectInput( ... ), width = 3 ), mainPanel( tags$h3("Latest stats:"), tags$div( ... ) %>% tagAppendAttributes(class = "stat-card-container"), tags$div( ... ) %>% tagAppendAttributes(class = "card-container"), tags$div( tags$h3("Drilldown:"), tags$div( highchartOutput(outputId = "chartDrilldown", height = 500) ) %>% tagAppendAttributes(class = "chart-card chart-card-full") ) %>% tagAppendAttributes(class = "card-container"), width = 9 ) %>% tagAppendAttributes(class = "main-container") ))server <- function(input, output) { data\_cards <- reactive({ ... }) data\_charts <- reactive({ ... }) drilldown\_chart\_base\_data <- reactive({ gapminder %>% filter( continent == input$inContinent, year == max(year) ) %>% group\_by(country) %>% summarise( pop = round(pop, 1) ) %>% arrange(desc(pop)) }) drilldown\_chart\_drilldown\_data <- reactive({ gapminder %>% filter( continent == input$inContinent, between(year, as.integer(input$inYearMin), as.integer(input$inYearMax)) ) %>% group\_nest(country) %>% mutate( id = country, type = "column", data = map(data, mutate, name = year, y = pop), data = map(data, list\_parse) ) }) output$outNCountries <- renderText({ ... }) output$outMedLifeExp <- renderText({ ... }) output$outMedPop <- renderText({ ... }) output$outMedGDP <- renderText({ ... }) output$chartLifeExpByYear <- renderHighchart({ ... }) output$chartGDPByYear <- renderHighchart({ ... }) output$chartDrilldown <- renderHighchart({ hchart( drilldown\_chart\_base\_data(), "column", hcaes(x = country, y = pop, drilldown = country), name = "Population" ) %>% hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_drilldown\_data()) ) |> hc\_colors(c("#004c5f")) |> hc\_title(text = "Population report", align = "left") |> hc\_xAxis(title = list(text = "")) |> hc\_yAxis(title = list(text = "Population")) })}shinyApp(ui = ui, server = server) ‍

There’s only a slight addition needed in styles.css to make our chart card 100% wide:

.chart-card-full { width: 100%;} And that’s it – we have a finished R Shiny Highcharts dashboard now:

‍

Image 5 – R Shiny Highcharts dashboard (4)‍

You can probably make it look better by tweaking the styles, but it does the job perfectly even in this state. Let’s make a brief recap next.

Full Source CodeIn case you’ve missed something or just want to copy/paste the code, this section is for you.

app.R library(shiny)library(dplyr)library(purrr)library(gapminder)library(highcharter)ui <- fluidPage( tags$head( tags$link(rel = "stylesheet", type = "text/css", href = "styles.css") ), sidebarLayout( sidebarPanel( titlePanel("R Shiny Highcharts"), selectInput( inputId = "inContinent", label = "Continent:", choices = unique(gapminder$continent), selected = "Europe" ), selectInput( inputId = "inYearMin", label = "Start year:", choices = unique(gapminder$year)[1:length(unique(gapminder$year)) - 1], selected = min(gapminder$year) ), selectInput( inputId = "inYearMax", label = "End year:", choices = unique(gapminder$year)[2:length(unique(gapminder$year))], selected = max(gapminder$year) ), width = 3 ), mainPanel( tags$h3("Latest stats:"), tags$div( tags$div( tags$p("# Countries:"), textOutput(outputId = "outNCountries") ) %>% tagAppendAttributes(class = "stat-card"), tags$div( tags$p("Median life exp:"), textOutput(outputId = "outMedLifeExp") ) %>% tagAppendAttributes(class = "stat-card"), tags$div( tags$p("Median population:"), textOutput(outputId = "outMedPop") ) %>% tagAppendAttributes(class = "stat-card"), tags$div( tags$p("Median GDP:"), textOutput(outputId = "outMedGDP") ) %>% tagAppendAttributes(class = "stat-card") ) %>% tagAppendAttributes(class = "stat-card-container"), tags$div( tags$h3("Summary stats:"), tags$div( tags$div( highchartOutput(outputId = "chartLifeExpByYear", height = 500) ) %>% tagAppendAttributes(class = "chart-card"), tags$div( highchartOutput(outputId = "chartGDPByYear", height = 500) ) %>% tagAppendAttributes(class = "chart-card"), ) %>% tagAppendAttributes(class = "base-charts-container") ) %>% tagAppendAttributes(class = "card-container"), tags$div( tags$h3("Drilldown:"), tags$div( highchartOutput(outputId = "chartDrilldown", height = 500) ) %>% tagAppendAttributes(class = "chart-card chart-card-full") ) %>% tagAppendAttributes(class = "card-container"), width = 9 ) %>% tagAppendAttributes(class = "main-container") ))server <- function(input, output) { data\_cards <- reactive({ gapminder %>% filter( continent == input$inContinent, year == max(year) ) %>% summarise( nCountries = n\_distinct(country), medianLifeExp = median(lifeExp), medianPopM = median(pop / 1e6), medianGDP = median(gdpPercap) ) }) data\_charts <- reactive({ gapminder %>% filter( continent == input$inContinent, between(year, as.integer(input$inYearMin), as.integer(input$inYearMax)) ) %>% group\_by(year) %>% summarise( medianLifeExp = round(median(lifeExp), 1), medianGDP = round(median(gdpPercap), 2) ) }) drilldown\_chart\_base\_data <- reactive({ gapminder %>% filter( continent == input$inContinent, year == max(year) ) %>% group\_by(country) %>% summarise( pop = round(pop, 1) ) %>% arrange(desc(pop)) }) drilldown\_chart\_drilldown\_data <- reactive({ gapminder %>% filter( continent == input$inContinent, between(year, as.integer(input$inYearMin), as.integer(input$inYearMax)) ) %>% group\_nest(country) %>% mutate( id = country, type = "column", data = map(data, mutate, name = year, y = pop), data = map(data, list\_parse) ) }) output$outNCountries <- renderText({ data\_cards()$nCountries }) output$outMedLifeExp <- renderText({ paste(round(data\_cards()$medianLifeExp, 1), "years") }) output$outMedPop <- renderText({ paste0(round(data\_cards()$medianPopM, 2), "M") }) output$outMedGDP <- renderText({ paste0("$", round(data\_cards()$medianGDP, 2)) }) output$chartLifeExpByYear <- renderHighchart({ hchart(data\_charts(), "column", hcaes(x = year, y = medianLifeExp), color = "#0198f9", name = "Median life expectancy") |> hc\_title(text = "Median life expectancy by year", align = "left") |> hc\_xAxis(title = list(text = "Year")) |> hc\_yAxis(title = list(text = "Life expectancy")) }) output$chartGDPByYear <- renderHighchart({ hchart(data\_charts(), "line", hcaes(x = year, y = medianGDP), color = "#800000", name = "Median GDP") |> hc\_title(text = "Median GDP by year", align = "left") |> hc\_xAxis(title = list(text = "Year")) |> hc\_yAxis(title = list(text = "GDP")) }) output$chartDrilldown <- renderHighchart({ hchart( drilldown\_chart\_base\_data(), "column", hcaes(x = country, y = pop, drilldown = country), name = "Population" ) %>% hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_drilldown\_data()) ) |> hc\_colors(c("#004c5f")) |> hc\_title(text = "Population report", align = "left") |> hc\_xAxis(title = list(text = "")) |> hc\_yAxis(title = list(text = "Population")) })}shinyApp(ui = ui, server = server) www/styles.css @import url('https://fonts.googleapis.com/css2?family=Poppins:ital,wght@0,700;1,400&display=swap');* { margin: 0; padding: 0; box-sizing: border-box;}body { font-family: 'Poppins', sans-serif; font-weight: 400;}.main-container { padding-top: 1rem;}.stat-card-container { display: flex; justify-content: space-between; column-gap: 1rem;}.stat-card { border: 2px solid #f2f2f2; border-bottom: 2px solid #0198f9; width: 100%; padding: 0.5rem 0 0.5rem 1rem;}.stat-card > p { text-transform: uppercase; color: #808080;}.stat-card > div.shiny-text-output { font-size: 3rem; font-weight: 700;}.card-container { padding-top: 2rem;}.base-charts-container { display: flex; justify-content: space-between; column-gap: 1rem;}.chart-card { border: 2px solid #f2f2f2; width: 50%;}.chart-card-full { width: 100%;} Summing up R Shiny HighchartsThis article concludes our three-part series on R Highcharts. You’ve learned how to make basic interactive visualizations, how to make drilldown charts, and today, how to tie it all together with R Shiny. You now have everything needed to leverage Highcharts on your next project or to build an impressive resume of Shiny applications.

Today’s article was a bit heavier on the code and lighter on explanations – that’s because you already know how things work, and our resulting app has a fair amount of reactive code. We hope it was easy enough to follow, but make sure to pop your question(s) in the comment section below if anything is not 100% clear.

As always, thanks for reading, and stay tuned to the Appsilon blog and our newsletter, Shiny Weekly to learn more about R/Shiny.

If you’re wondering what else you need to start a career in R Shiny – We have a 2024-ready guide for you.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Shiny Highcharts – How to Create Interactive and Animated Shiny Dashboards

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I set out to see if it was possible to reproduce the UN’s 2022 Revision of World Population Prospects for a given country by cohort component projection from the fertility, mortality and immigration rates and population starting point published as part of their projection. The motivation is to make small changes to some of those parameters – for example by substituting in a recent census result for the population and a given year – and “re-run” the projections to see the impact of changes, or to get a more up-to-date version with data that wasn’t available to the UN at the time of their projection.

It turns out this wasn’t too hard (one morning’s work for the modelling, then a few hours of write-up), particularly in cases where migration is small in the projection period. I was able to reproduce almost exactly population, birth and death totals to 2100 for Vanuatu, and demonstrate the impact of updating the 2020 year for their recent census population totals and fertility rates, and getting a slightly lower projection as a result.

Here’s my reproduction of Vanuatu’s population projection from 2020 to 2100, using just the 2020 population totals and the forecast fertility, mortality and migration rates. As you can see it’s basically identical to the UN totals:

And here’s the same method tweaked for the actual 2020 census total and with a rough adjustment made to fertility rates based on what was observed at the 2020 census:

Of course, this method delivers a full set of projections by age and sex, and we could construct life tables or any indicators we want from it. Here are population pyramids comparing the published UN projections for 2050 with my revised set. Not visually stunning in its comparison, but enough to prove that it’s possible:

Reproducing UN projectionsSo here’s how I went about that.

First, downloading all the data. The UN recommend bulk downloads of their CSV files. For my purposes I first need the fertility, mortality and population by sex and one year age groups. Of the population original data I am only going to use the 2020 year, and then project it forward myself based on fertility, mortality and migration; but I want the full set for comparison purposes. For migration, I couldn’t see in my hasty look at the UN site a dataset of migration projections by age and sex, so I just use the much simpler net migration rate (per thousand people) per year in the projection period. Here’s code to download all this UN data:

library(tidyverse)library(glue)library(scales)library(patchwork)dir.create("data-pop-proj-2022", showWarnings = FALSE)#------------------download and import data for all countries from existing projections----------------list.files("data-pop-proj-2022")files <- c("WPP2022\_Fertility\_by\_Age1.zip", "WPP2022\_DeathsBySingleAgeSex\_Medium\_1950-2021.zip", "WPP2022\_DeathsBySingleAgeSex\_Medium\_2022-2100.zip", "WPP2022\_Population1JanuaryBySingleAgeSex\_Medium\_1950-2021.zip", "WPP2022\_Population1JanuaryBySingleAgeSex\_Medium\_2022-2100.zip", "WPP2022\_Demographic\_Indicators\_Medium.zip" )# let downloads take up to 10 minutes rather than 1 minute max, as files are# large (largest is fertility for single age, 78MB)options(timeout=600)if(!file.exists("data-pop-proj-2022/WPP2022\_Demographic\_Indicators\_Medium.csv")){ # download the zip files: for(i in 1:length(files)){ download.file(glue("https://population.un.org/wpp/Download/Files/1\_Indicators%20(Standard)/CSV\_FILES/{files[i]}"), destfile = glue("data-pop-proj-2022/{files[i]}"), mode = "wb") } # unzip them for(i in 1:length(files)){ unzip(glue("data-pop-proj-2022/{files[i]}"), exdir = "data-pop-proj-2022") }}fert\_all <- read\_csv("data-pop-proj-2022/WPP2022\_Fertility\_by\_Age1.csv")mort\_past <- read\_csv("data-pop-proj-2022/WPP2022\_DeathsBySingleAgeSex\_Medium\_1950-2021.csv")mort\_future <- read\_csv("data-pop-proj-2022/WPP2022\_DeathsBySingleAgeSex\_Medium\_2022-2100.csv")pop\_past <- read\_csv("data-pop-proj-2022/WPP2022\_Population1JanuaryBySingleAgeSex\_Medium\_1950-2021.csv")pop\_future <- read\_csv("data-pop-proj-2022/WPP2022\_Population1JanuaryBySingleAgeSex\_Medium\_2022-2100.csv")indicators <- read\_csv("data-pop-proj-2022/WPP2022\_Demographic\_Indicators\_Medium.csv")mort\_all <- rbind(mort\_past, mort\_future)pop\_all <- rbind(pop\_past, pop\_future)# clean uprm(mort\_past, mort\_future, pop\_past, pop\_future) Next, I wrote my own cohort component population projection function. I wanted to do this from scratch rather than using an existing demography package to make sure I understood what was happening (I’m not a demographer) and could make tweaks as necessary to match the UN approach. I used this tutorial by Farid Flici as my starting point; abstracted his code into a function for easy use with multiple countries, and added a net migration component.

Because migrants’ ages tend to be dissimilar from the country they are migrating too - they are more likely to be in the prime of their working / family life I believe - I needed a way to set the ages of migrants. In the function below I defaulted to a normal distribution of ages, mean 28 and standard deviation 11, which worked well to get similar results to the UN in a few countries I tried. This was the hardest and most discretionary part of the exercise.

My observation is that demographers seem to think in terms of matrices of numbers rather than database-oriented tidy data, and I have kept that matrix approach in this function.

```

----------------------projections for one country-------------------# adapting the implementation of the method for cohort component projection at # https://farid-flici.github.io/tuto.html#' @param start_pop_m is a vector of population of males at single year age#' periods beginning of year starting age 0#' @param start_pop_f as start_pop_m but for females#' @param start_year first year ie the year for which the actual population#' number refers to#' @param end_year end year for the population projection#' @param fertility a matrix with rows for female agegroups 15 to 49 and a#' column for each year, values are births per woman (not per thousand women)#' that year and age. Must have 35 rows and number of columns equal to#' end_year minus start_year plus one.#' @param mort_m a matrix with rows for age 0 to at least 50 and columns for#' each year, for males. Must have number of columns equal to end_year minus#' start_year plus one.#' @param mort_f as mort_m but for females. Must have same number of rows and#' columns as mort_m.#' @param sex_ratio_birth number of boys born for every girl born, vector with#' length of years to be projected.#' @param net_migration vector of proportions of the population that migrate to#' the country, net of those who leave. Defaults to a vector of zeroes.#' @param female_share_migration vector of proportions of net migrants that are#' female. Defaults to a vector of 0.5s.#' @param migration_age_weights vector of relative distribution of the age of#' net migrants. Must be 121 values representing relative proportion of net#' migrants aged zero to 120. Defaults to a normal distribution of mean 28 and#' standard deviation 11. Note that this distribution currently must be the#' same through the projection period (unlike eg fertility and mortality,#' which have individual values for each age group and sex for each year).#' Only the total net migration proportion can change over time, not its age#' make-up.#' @author Peter Ellis, expanding on a less-featured example by Farid Flici#' https://farid-flici.github.io/tuto.htmlpop_proj <- function(start_pop_m, start_pop_f, start_year, end_year, fertility, mort_m, mort_f, sex_ratio_birth = rep(1.045, end_year - start_year + 1), net_migration = rep(0, end_year - start_year + 1), female_share_migration = rep(0.5, end_year - start_year + 1), migration_age_weights = dnorm(0:120, mean = 28, sd = 11) ){ stopifnot(nrow(fertility) == 35) stopifnot(length(migration_age_weights) == 121) ncols <- end_year - start_year + 1 stopifnot(ncol(fertility) == ncols) stopifnot(ncol(mort_m) == ncols) stopifnot(ncol(mort_f) == ncols) stopifnot(length(start_pop_m) > 50) stopifnot(length(start_pop_m) < 120) stopifnot(length(start_pop_f) > 50) stopifnot(length(start_pop_f) < 120) stopifnot(length(net_migration) == ncols) stopifnot(length(sex_ratio_birth) == ncols) stopifnot(length(female_share_migration) == ncols) stopifnot(min(female_share_migration) >= 0) stopifnot(max(female_share_migration) <= 1) stopifnot(nrow(mort_m) == nrow(mort_f)) # fill in with high probability of death for ages beyond where we have mortality data if(nrow(mort_m) < 121){ extra_deaths_m <- matrix(max(mort_m), nrow = 121 - nrow(mort_m), ncol = ncol(mort_m)) extra_deaths_f <- matrix(max(mort_f), nrow = 121 - nrow(mort_f), ncol = ncol(mort_f)) mort_m <- rbind(mort_m, extra_deaths_m) mort_f <- rbind(mort_f, extra_deaths_f) } # Create matrices for male and female population for the full projections # each row of the matrix is an age group, each column is a year PopM <- matrix(0, nrow=121, ncol = ncols) PopF <- matrix(0, nrow=121, ncol = ncols) deaths <- rep(NA, ncols) rownames(PopF) <- rownames(PopM) <- c(0:120) # ages colnames(PopF) <- colnames(PopM) <- c(start_year:end_year) # years # first year gets populated with the actual population numbers that we have: PopM[1:length(start_pop_m), 1] <- start_pop_m PopF[1:length(start_pop_f), 1] <- start_pop_f # subsequent years get modified by people aging one year, deaths, births, and net migration for (i in 2 : ncols) { # Age one and above: PopM[2:121, i] <- PopM[1:120, i-1] * (1 - mort_m[1:120, i-1]) PopF[2:121, i] <- PopF[1:120, i-1] * (1 - mort_f[1:120, i-1]) deaths[i-1] <- sum(PopM[, i-1]) - sum(PopM[, i]) + sum(PopF[, i-1]) - sum(PopF[, i]) # migration. total_migrants <- (sum(PopM[,i]) + sum(PopF[,i])) * net_migration[i] migf = total_migrants * female_share_migration[i] * migration_age_weights / sum(migration_age_weights) migm = total_migrants * (1 - female_share_migration[i]) * migration_age_weights / sum(migration_age_weights) PopM[, i] <- PopM[, i] + migm PopF[, i] <- PopF[, i] + migf # Age zero ie births. Fertility rate by the number of women in the middle of the previous year. # Note that PopF rows 16:50 equates to women age 15:49 reproducing_women <- (PopF[16:50, i-1] + PopF[16:50, i]) / 2 prop_boys <- sex_ratio_birth[i] / (1 + sex_ratio_birth[i]) prop_girls <- 1 - prop_boys PopM[1, i] <- reproducing_women %% fertility[, i-1] prop_boys PopF[1, i] <- reproducing_women %% fertility[, i-1] prop_girls } # Return a list of the two matrices return(list( PopM = PopM, PopF = PopF, deaths = deaths ))}

`` Next, I wrote a function to extract the necessary fertility, mortality, migration rates and 2020 starting population from the UN data, feed it into mypop_proj()function and return the result. Unlikepop_proj(), which is to some degree fully portable, this function is very much specific to this particular project and is really just a convenience function for grabbing the data and turning it into the right units and shapes (vectors and matrices) needed forpop_proj()`.

repeat\_un\_proj <- function(the\_country, the\_years = 2020:2100){ if(!the\_country %in% unique(indicators$Location)){ stop("Country not found") } this\_fert <- fert\_all |> filter(Location == the\_country & Variant == "Medium") |> filter(Time %in% the\_years) |> # Age-Specific Fertility rate select(Time, AgeGrp, ASFR) |> mutate(Time = as.character(Time), # turn into proportions, not rates per 1000: ASFR = ASFR / 1000) |> pivot\_wider(id\_cols = AgeGrp, names\_from = Time, values\_from = ASFR) |> arrange(AgeGrp) |> select(-AgeGrp) |> as.matrix() # should be 35 rows ie fertilities for ages 15 to 49 stopifnot(nrow(this\_fert) == 35) # should be 81 columns, 1 column for each year from 2020 to 2100 stopifnot(ncol(this\_fert) == length(the\_years)) # Mortality is in numbers not a ratio so we need to join to the population data to turn it into a ratio pop\_years <- pop\_all |> filter(Location == the\_country & Variant == "Medium" & Time %in% the\_years) |> select(Time, PopMale, PopFemale, AgeGrp) this\_mort <- mort\_all |> filter(Location == the\_country & Variant == "Medium") |> filter(Time %in% the\_years) |> left\_join(pop\_years, by = c("Time", "AgeGrp")) |> # next step important because we will be sorting by AgeGrp mutate(AgeGrp = case\_when( AgeGrp == "100+" ~ 100, TRUE ~ suppressWarnings(as.numeric(as.character(AgeGrp))) )) this\_mort\_m <- this\_mort |> # sometimes more deaths than people (eg 1 death, 0 people) so cap the death ratio at 1 mutate(DeathMale = pmin(1, DeathMale / PopMale)) |> select(Time, AgeGrp, DeathMale) |> mutate(Time = as.character(Time)) |> pivot\_wider(id\_cols = AgeGrp, names\_from = Time, values\_from = DeathMale) |> arrange(AgeGrp) |> select(-AgeGrp) |> as.matrix() this\_mort\_f <- this\_mort |> mutate(DeathFemale = pmin(1, DeathFemale / PopFemale)) |> select(Time, AgeGrp, DeathFemale) |> mutate(Time = as.character(Time)) |> pivot\_wider(id\_cols = AgeGrp, names\_from = Time, values\_from = DeathFemale) |> arrange(AgeGrp) |> select(-AgeGrp) |> as.matrix() # check the years are correct, didn't get mangled or reordered stopifnot(all(colnames(this\_mort\_f) == the\_years)) stopifnot(all(colnames(this\_mort\_m) == the\_years)) this\_pop <- pop\_all |> filter(Location == the\_country & Variant == "Medium") |> filter(Time == min(the\_years)) |> # next step important because we will be sorting by AgeGrp mutate(AgeGrp = case\_when( AgeGrp == "100+" ~ 100, TRUE ~ suppressWarnings(as.numeric(as.character(AgeGrp))) )) |> arrange(AgeGrp) # convert to units, not thousands of people: this\_pop\_m <- this\_pop$PopMale * 1000 this\_pop\_f <- this\_pop$PopFemale * 1000 # reality check # Population in millions; should be about 0.3 if the\_country is Vanuatu, about 1400 if India: (sum(this\_pop\_m) + sum(this\_pop\_f) ) / 1e6 # net migration this\_cnmr <- indicators |> filter(Location == the\_country & Time %in% the\_years) |> arrange(Time) |> pull(CNMR) / 1000 # sex ratio at birth this\_srb <- indicators |> filter(Location == the\_country & Time %in% the\_years) |> arrange(Time) |> pull(SRB) / 100 this\_proj <- pop\_proj( start\_pop\_m = this\_pop\_m, start\_pop\_f = this\_pop\_f, start\_year = min(the\_years), end\_year = max(the\_years), fertility = this\_fert, mort\_m = this\_mort\_m, mort\_f = this\_mort\_f, net\_migration = this\_cnmr, sex\_ratio\_birth = this\_srb ) return(list(un\_proj = this\_proj, un\_pop\_m = this\_pop\_m, un\_pop\_f = this\_pop\_f, un\_fert = this\_fert, un\_mort\_m = this\_mort\_m, un\_mort\_f = this\_mort\_f, un\_cnmr = this\_cnmr, un\_srb = this\_srb))} Note that this function returns, in addition to the results of the population projection, the various inputs in their correct units and shape. This will be useful later when we want to modify some of those inputs.

Now that we’ve got these functions, using them to do projections from 2020 and compare those projections to the published numbers is pretty straight forward. Here’s the code to do that for Vanuatu, which produces the first chart at the top of this blog post:

the\_country <- "Vanuatu"my\_proj <- repeat\_un\_proj(the\_country)$un\_proj# total populationcomp\_data <- indicators |> filter(Location == the\_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `UN original` = TPopulation1Jan) |> mutate(`Reproduction` = as.numeric(apply(my\_proj$PopM, 2, sum) + apply(my\_proj$PopF, 2, sum)) / 1000) # First year should be an exact match:stopifnot(comp\_data[1, ]$`UN original` == comp\_data[1, ]$Reproduction)comp\_data |> gather(variable, value, -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom\_line() + scale\_y\_continuous(label = comma) + labs(title = the\_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Population", x = "", colour = "") As we can see the results are pretty much identical. Let’s look at my projected births and deaths based on fertility and mortality rates, and compare them to the published projected numbers

```

birthsindicators |> filter(Location == the_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `UN original` = Births) |> mutate(Reproduction = as.numeric(my_proj$PopM[1,] + my_proj$PopF[1, ]) / 1000) |> gather(variable, value , -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom_line() + labs(title = the_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Births (thousands)", x = "", colour = "")# deathsindicators |> filter(Location == the_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `UN original` = Deaths) |> mutate(Reproduction = as.numeric(my_proj$deaths) / 1000) |> gather(variable, value , -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom_line() + labs(title = the_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Deaths (thousands)", x = "", colour = "")

``` I’m pretty happy with those. There’s definitely some discrepancies and a lag in the births which I suspect come down to how one treats populations of mothers - numbers on 1 January v 1 July, that sort of thing. But in the scheme of things these are very small.

Now, Vanuatu is relatively easy because the UN assumed net zero migration in the projection period. We can see this by comparing the net migration rates in their Indicators dataset for a few countries, with this code:

plot\_mig <- function(the\_country){ p <- indicators |> filter(Location == the\_country) |> ggplot(aes(x = Time, y = CNMR)) + geom\_vline(xintercept = 2022, lty = 2, colour = "steelblue") + geom\_hline(yintercept = 0, lty = 2, colour = "steelblue") + geom\_line() + labs(title = the\_country, subtitle = "Net migration rate per thousand people", y = "", x = "", caption = "Source: UN World Population Prospects 2022") return(p)}plot\_mig("Vanuatu") + plot\_mig("Fiji") + plot\_mig("Australia") + plot\_mig("India") + plot\_layout(ncol = 2) For countries that have good data on it, migration is a big deal in the projections; but forecasting is hard, particularly of the future.

My first few goes at reproducing the projections for Australia and India tended to be badly out because I had added in net migration evenly across the whole age distribution. My eventual solution, making net migration bell curved with an average age of 28, is a bit of a hack with the parameters chosen to make Australia’s projections come out right. Definitely a better method would be to have actual age-specific net migration forecasts. Whether such things are possible will very much depend on the country; it’s probably possible for Australia, but not for most of the countries I work with.

Here’s the final result comparing UN projections with mine for a few interesting countries:

Adjusting the starting pointNow, the whole point of this exercise was to see if we can plausibly adjust the starting point - say the population totals in 2020, or the forecast fertility rates - and say we are building on the UN’s projections to get our own. Here’s my rough demo of how we might do that, again using the case of Vanuatu. Vanuatu’s 2020 census wasn’t available at the time of the UN’s 2022 population projections, so the actual population and fertility numbers for 2020 differ somewhat (of course) from what was projected.

For the below, I am relying on the analytical report of the Vanuatu census. On a quick glance I didn’t see a table of the actual total population by single year age and sex, so I just adjusted the UN’s projected totals by a factor to make them add up to the correct 2020 total males and females. Of course if we were doing this for real we’d get the real numbers straight from the census; I actually can do this easily enough at work but obviously for this blog wanted to use only easily accessible public data.

```

-----------------changing one or two factors while keeping the rest the same--------------# Say we had the 2020 Vanuatu census so we knew the real population then, and wanted# to redo the population projections with everything staying as they were before# file:///C:/Users/Peter/Downloads/Vanuatu_2020_Census_Analytical_report_Vol_2.pdfrevision_country <- "Vanuatu"van_orig <- repeat_un_proj(revision_country)# rough adjustment of starting population in 2020 to make totals match the Census totals.# In principle could of course use the actual numbers by single age category.revised_pop_m <- van_orig$un_pop_m * 151597 / sum(van_orig$un_pop_m)revised_pop_f <- van_orig$un_pop_f * 148422 / sum(van_orig$un_pop_f)# calculate the crude birth rate in the current projectionun_cbr_2020 <- (van_orig$un_proj$PopM[1, 2] + van_orig$un_proj$PopF[1, 2]) / sum(van_orig$un_proj$PopM[, 2], van_orig$un_proj$PopF[, 2]) * 1000# make a rough adjustment of future fertility rates assuming they are "out" by the# same proportion that crude birth rate was out in 2020adj_ratio <- 28.2 / un_cbr_2020revised_fert <- van_orig$un_fert * adj_ratio# Refit the projection with the above rough adjustments:revised_proj <- pop_proj( start_pop_m = revised_pop_m, start_pop_f = revised_pop_f, start_year = 2020, end_year = 2100, fertility = revised_fert, mort_m = van_orig$un_mort_m, mort_f = van_orig$un_mort_f, sex_ratio_birth = van_orig$un_srb, net_migration = van_orig$un_cnmr)projected_pop <- apply(revised_proj$PopM, 2, sum) + apply(revised_proj$PopF, 2, sum)#-----------------Compare total population----------------comp_data <- indicators |> filter(Location == revision_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `2022 UN projections` = TPopulation1Jan) |> mutate(`Revised with 2020 census` = as.numeric(projected_pop / 1000) )p5 <- comp_data |> filter(Time <= 2050) |> gather(variable, value, -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom_line() + labs(title = revision_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Population", x = "", colour = "")#--------------Compare population pyramids-------------------------d <- tibble(value = c( revised_proj$PopM[, 31], revised_proj$PopF[, 31], van_orig$un_proj$PopM[, 31], van_orig$un_proj$PopF[, 31]), sex = rep(c("Male", "Female", "Male", "Female"), each = 121), model = rep(c("Revised projections with 2020 census, made in 2024", "UN projections, made in 2022"), each = 242), age = rep(0:120, 4)) |> mutate(agef = fct_reorder(as.character(age), age), model = fct_rev(model)) |> filter(age < 100)d |> filter(sex == "Female") |> ggplot(aes(x = value, y = agef)) + facet_wrap(~model) + geom_col(fill = "brown", colour = NA) + geom_col(data = filter(d, sex == "Male"), aes(x = -value), fill = "orange", colour = NA) + geom_vline(xintercept = 0, colour = "white") + labs(x = "Number of people", y = "", title = "Comparison of UN original and revised population projections", subtitle = "Population age distribution in 2050") + scale_x_continuous(breaks = c(-4000, -2000, 0, 2000, 4000), labels = c("4,000", "Male", 0, "Female", "4,000")) + theme(panel.grid = element_blank()) + scale_y_discrete(breaks = 1:20 * 5)

``` And that’s what gets us these results:

I’m hoping this might be actually useful for pragmatic updates of the UN population projections when more current data is available, without having to revise everything from scratch.

OK, that’s all for today.

To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Reproducing and adapting the UN Population Projections by @ellis2013nz

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

And here’s the same method tweaked for the actual 2020 census total and with a rough adjustment made to fertility rates based on what was observed at the 2020 census:

Reproducing UN projectionsSo here’s how I went about that.

My observation is that demographers seem to think in terms of matrices of numbers rather than database-oriented tidy data, and I have kept that matrix approach in this function.

```

----------------------projections for one country-------------------# adapting the implementation of the method for cohort component projection at # https://farid-flici.github.io/tuto.html#' @param start_pop_m is a vector of population of males at single year age#' periods beginning of year starting age 0#' @param start_pop_f as start_pop_m but for females#' @param start_year first year ie the year for which the actual population#' number refers to#' @param end_year end year for the population projection#' @param fertility a matrix with rows for female agegroups 15 to 49 and a#' column for each year, values are births per woman (not per thousand women)#' that year and age. Must have 35 rows and number of columns equal to#' end_year minus start_year plus one.#' @param mort_m a matrix with rows for age 0 to at least 50 and columns for#' each year, for males. Must have number of columns equal to end_year minus#' start_year plus one.#' @param mort_f as mort_m but for females. Must have same number of rows and#' columns as mort_m.#' @param sex_ratio_birth number of boys born for every girl born, vector with#' length of years to be projected.#' @param net_migration vector of proportions of the population that migrate to#' the country, net of those who leave. Defaults to a vector of zeroes.#' @param female_share_migration vector of proportions of net migrants that are#' female. Defaults to a vector of 0.5s.#' @param migration_age_weights vector of relative distribution of the age of#' net migrants. Must be 121 values representing relative proportion of net#' migrants aged zero to 120. Defaults to a normal distribution of mean 28 and#' standard deviation 11. Note that this distribution currently must be the#' same through the projection period (unlike eg fertility and mortality,#' which have individual values for each age group and sex for each year).#' Only the total net migration proportion can change over time, not its age#' make-up.#' @author Peter Ellis, expanding on a less-featured example by Farid Flici#' https://farid-flici.github.io/tuto.htmlpop_proj <- function(start_pop_m, start_pop_f, start_year, end_year, fertility, mort_m, mort_f, sex_ratio_birth = rep(1.045, end_year - start_year + 1), net_migration = rep(0, end_year - start_year + 1), female_share_migration = rep(0.5, end_year - start_year + 1), migration_age_weights = dnorm(0:120, mean = 28, sd = 11) ){ stopifnot(nrow(fertility) == 35) stopifnot(length(migration_age_weights) == 121) ncols <- end_year - start_year + 1 stopifnot(ncol(fertility) == ncols) stopifnot(ncol(mort_m) == ncols) stopifnot(ncol(mort_f) == ncols) stopifnot(length(start_pop_m) > 50) stopifnot(length(start_pop_m) < 120) stopifnot(length(start_pop_f) > 50) stopifnot(length(start_pop_f) < 120) stopifnot(length(net_migration) == ncols) stopifnot(length(sex_ratio_birth) == ncols) stopifnot(length(female_share_migration) == ncols) stopifnot(min(female_share_migration) >= 0) stopifnot(max(female_share_migration) <= 1) stopifnot(nrow(mort_m) == nrow(mort_f)) # fill in with high probability of death for ages beyond where we have mortality data if(nrow(mort_m) < 121){ extra_deaths_m <- matrix(max(mort_m), nrow = 121 - nrow(mort_m), ncol = ncol(mort_m)) extra_deaths_f <- matrix(max(mort_f), nrow = 121 - nrow(mort_f), ncol = ncol(mort_f)) mort_m <- rbind(mort_m, extra_deaths_m) mort_f <- rbind(mort_f, extra_deaths_f) } # Create matrices for male and female population for the full projections # each row of the matrix is an age group, each column is a year PopM <- matrix(0, nrow=121, ncol = ncols) PopF <- matrix(0, nrow=121, ncol = ncols) deaths <- rep(NA, ncols) rownames(PopF) <- rownames(PopM) <- c(0:120) # ages colnames(PopF) <- colnames(PopM) <- c(start_year:end_year) # years # first year gets populated with the actual population numbers that we have: PopM[1:length(start_pop_m), 1] <- start_pop_m PopF[1:length(start_pop_f), 1] <- start_pop_f # subsequent years get modified by people aging one year, deaths, births, and net migration for (i in 2 : ncols) { # Age one and above: PopM[2:121, i] <- PopM[1:120, i-1] * (1 - mort_m[1:120, i-1]) PopF[2:121, i] <- PopF[1:120, i-1] * (1 - mort_f[1:120, i-1]) deaths[i-1] <- sum(PopM[, i-1]) - sum(PopM[, i]) + sum(PopF[, i-1]) - sum(PopF[, i]) # migration. total_migrants <- (sum(PopM[,i]) + sum(PopF[,i])) * net_migration[i] migf = total_migrants * female_share_migration[i] * migration_age_weights / sum(migration_age_weights) migm = total_migrants * (1 - female_share_migration[i]) * migration_age_weights / sum(migration_age_weights) PopM[, i] <- PopM[, i] + migm PopF[, i] <- PopF[, i] + migf # Age zero ie births. Fertility rate by the number of women in the middle of the previous year. # Note that PopF rows 16:50 equates to women age 15:49 reproducing_women <- (PopF[16:50, i-1] + PopF[16:50, i]) / 2 prop_boys <- sex_ratio_birth[i] / (1 + sex_ratio_birth[i]) prop_girls <- 1 - prop_boys PopM[1, i] <- reproducing_women %% fertility[, i-1] prop_boys PopF[1, i] <- reproducing_women %% fertility[, i-1] prop_girls } # Return a list of the two matrices return(list( PopM = PopM, PopF = PopF, deaths = deaths ))}

```

birthsindicators |> filter(Location == the_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `UN original` = Births) |> mutate(Reproduction = as.numeric(my_proj$PopM[1,] + my_proj$PopF[1, ]) / 1000) |> gather(variable, value , -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom_line() + labs(title = the_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Births (thousands)", x = "", colour = "")# deathsindicators |> filter(Location == the_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `UN original` = Deaths) |> mutate(Reproduction = as.numeric(my_proj$deaths) / 1000) |> gather(variable, value , -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom_line() + labs(title = the_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Deaths (thousands)", x = "", colour = "")

Here’s the final result comparing UN projections with mine for a few interesting countries:

```

-----------------changing one or two factors while keeping the rest the same--------------# Say we had the 2020 Vanuatu census so we knew the real population then, and wanted# to redo the population projections with everything staying as they were before# file:///C:/Users/Peter/Downloads/Vanuatu_2020_Census_Analytical_report_Vol_2.pdfrevision_country <- "Vanuatu"van_orig <- repeat_un_proj(revision_country)# rough adjustment of starting population in 2020 to make totals match the Census totals.# In principle could of course use the actual numbers by single age category.revised_pop_m <- van_orig$un_pop_m * 151597 / sum(van_orig$un_pop_m)revised_pop_f <- van_orig$un_pop_f * 148422 / sum(van_orig$un_pop_f)# calculate the crude birth rate in the current projectionun_cbr_2020 <- (van_orig$un_proj$PopM[1, 2] + van_orig$un_proj$PopF[1, 2]) / sum(van_orig$un_proj$PopM[, 2], van_orig$un_proj$PopF[, 2]) * 1000# make a rough adjustment of future fertility rates assuming they are "out" by the# same proportion that crude birth rate was out in 2020adj_ratio <- 28.2 / un_cbr_2020revised_fert <- van_orig$un_fert * adj_ratio# Refit the projection with the above rough adjustments:revised_proj <- pop_proj( start_pop_m = revised_pop_m, start_pop_f = revised_pop_f, start_year = 2020, end_year = 2100, fertility = revised_fert, mort_m = van_orig$un_mort_m, mort_f = van_orig$un_mort_f, sex_ratio_birth = van_orig$un_srb, net_migration = van_orig$un_cnmr)projected_pop <- apply(revised_proj$PopM, 2, sum) + apply(revised_proj$PopF, 2, sum)#-----------------Compare total population----------------comp_data <- indicators |> filter(Location == revision_country & Variant == "Medium" & Time %in% 2020:2100) |> select(Time, `2022 UN projections` = TPopulation1Jan) |> mutate(`Revised with 2020 census` = as.numeric(projected_pop / 1000) )p5 <- comp_data |> filter(Time <= 2050) |> gather(variable, value, -Time) |> ggplot(aes(x = Time, y = value, colour = variable)) + geom_line() + labs(title = revision_country, subtitle = "Attempt to re-create the UN population projections from population in 2020, fertility and mortality rates", y = "Population", x = "", colour = "")#--------------Compare population pyramids-------------------------d <- tibble(value = c( revised_proj$PopM[, 31], revised_proj$PopF[, 31], van_orig$un_proj$PopM[, 31], van_orig$un_proj$PopF[, 31]), sex = rep(c("Male", "Female", "Male", "Female"), each = 121), model = rep(c("Revised projections with 2020 census, made in 2024", "UN projections, made in 2022"), each = 242), age = rep(0:120, 4)) |> mutate(agef = fct_reorder(as.character(age), age), model = fct_rev(model)) |> filter(age < 100)d |> filter(sex == "Female") |> ggplot(aes(x = value, y = agef)) + facet_wrap(~model) + geom_col(fill = "brown", colour = NA) + geom_col(data = filter(d, sex == "Male"), aes(x = -value), fill = "orange", colour = NA) + geom_vline(xintercept = 0, colour = "white") + labs(x = "Number of people", y = "", title = "Comparison of UN original and revised population projections", subtitle = "Population age distribution in 2050") + scale_x_continuous(breaks = c(-4000, -2000, 0, 2000, 4000), labels = c("4,000", "Male", 0, "Female", "4,000")) + theme(panel.grid = element_blank()) + scale_y_discrete(breaks = 1:20 * 5)

``` And that’s what gets us these results:

I’m hoping this might be actually useful for pragmatic updates of the UN population projections when more current data is available, without having to revise everything from scratch.

OK, that’s all for today.

To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Reproducing and adapting the UN Population Projections by @ellis2013nz

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Compare numeric vectors in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Compare numeric vectors in R, we explore the usage of the ‘near’ function from the ‘dplyr’ package in R programming.

The article is divided into two examples, with the first one demonstrating the basic application of the ‘near’ function and the second one showcasing its flexibility with user-defined tolerance.

Compare numeric vectors in RTo begin, we create exemplifying data by defining two numeric vectors, ‘x1’ and ‘x2’.

x1 <- 1:5 x2 <- c(1, 2.2, 2.5, 4, 5.3) We then install and load the ‘dplyr’ package to access the ‘near’ function.

Time Series Trend Analysis in R »

library(dplyr) Example 1: we apply the ‘near’ function to our vectorsThe function returns a logical vector, indicating whether the corresponding elements from both vectors are the same.

In this case, the first and fourth elements are identical.

near(x1, x2) [1] TRUE FALSE FALSE TRUE FALSE Example 2: Baisis User-Defined ToleranceIn Example 2, we introduce the ‘tol’ argument, which allows for increased tolerance in the comparison.

near(x1, x2, tol = 0.2) [1] TRUE FALSE FALSE TRUE FALSE By setting the tolerance to 0.2, the second and third elements of the input vectors are now considered the same.

Adjusting the tolerance can be beneficial depending on specific requirements.

SummaryThe ‘near’ function from the ‘dplyr’ package in R is a valuable tool for comparing numeric vectors and offers flexibility through the ‘tol’ argument.

Making games in R- Nara and eventloop Game Changers
How Do Online Criminals Acquire Sensitive Data
Best Books to Learn Statistics for Data Science
Get the first value in each group in R? » Data Science Tutorials
The Multinomial Distribution in R » Data Science Tutorials
Predict potential customers in R » Data Science Tutorials

The post Compare numeric vectors in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Compare numeric vectors in R

View Details

[This article was first published on Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.ShinyConf 2024 was a resounding success!

We brought together the vibrant R/Shiny community for three days of insightful sessions, engaging workshops, and valuable networking opportunities. From seasoned developers to newcomers, the conference had something for everyone.

Insightful Sessions and Memorable SpeakersThe sessions covered a wide range of topics, from best practices in Shiny development to innovative use cases across various industries, spanning four tracks: Innovation, Enterprise, Data 4 Good, and Life Sciences.

ShinyConf attendees gained valuable insights from industry experts. We had the opportunity to dive deep into specific areas of interest through hands-on workshops on modular app development (with Veerle van Leemput), PyShiny, Rhino, and Testing with Cypress.

‍

The speaker lineup was truly impressive, featuring renowned experts and thought leaders in R/Shiny who spoke on relevant and practical topics.

Tailoring Shiny for Modern Users by Lindsay Jorgenson & John Coene‍

One notable session was by Joe Cheng, CTO of Posit, on Beyond Async: Intra-Session Concurrency in Shiny with ExtendedTask. During his talk, he introduced ExtendedTask, which allows you to run long-running tasks for a user while preserving both inter- and intra-session concurrency. Plus, he closed a six-year-old issue live.

Beyond Async: Intra-Session Concurrency in Shiny with ExtendedTask‍Shinylive for R, which is new and about running Shiny apps in the browser, was also heavily discussed during the conference, especially during George Stagg’s keynote on reproducible data science and Barret Schloerke’s session on Shinylive.

Reproducible data science with webR and ShinyliveInsightful KeynotesWe had wonderful keynotes that covered thriving as an open source maintainer with Tracy K. Teal, the future of Shiny with Pedro Silva, webR and Shinylive with George Stagg, Shiny in Enterprise with Eric Kostello, and real-world use cases for AI in Shiny with Tanya Cashorali.

Beyond the Hype: Real-World Use Cases for AI in Shiny*Replays of all the sessions are available on RingCentral.

New Open Source Package Announcement One of the most exciting announcements at ShinyConf 2024 was the introduction of a new open-source package. This package promises to improve the way developers build and deploy enterprise Shiny applications. Details will be revealed soon! Watch our social media pages (LinkedIn & Twitter), join our community, and subscribe to our newsletter, Shiny Weekly, to learn more.

‍Update: Learn more about the package in our intro blog post, check it out on GitHub and here’s the documentation.

‍

Hex Logo Contest for Package DevelopersAlso, as part of Appsilon’s commitment to supporting the vibrant R package ecosystem, we’re hosting a Hex Logo Contest for developers who want unique hex designs for their packages. The contest is still on and closes on the 31st of May 2024. You can learn more and register on our website.

Register to Win a Hex LogoWhat Package R You? QuizIn case you missed it, we added a fun element to the conference. Attendees had the opportunity to participate in our Ultimate R Package Personality Quiz. This interactive quiz matches you with an R package based on your personality and coding style. It’s still available, and you can try it out yourself.

‍

Redesign of RhinoverseOur Lab Lead, Jakub Nowicki, unveiled the redesign of the Rhinoverse hexes during the conference. We hope you love the fresh look of our open-source packages as much as we do!

‍

RhinoverseConclusionIn conclusion, ShinyConf 2024 was a wonderful experience that brought together new and experienced R/Shiny professionals and thought leaders from several industries and research backgrounds.

‍

Here’s some feedback we’ve received from the community:

‍

“The event was incredibly informative and useful, and the perspectives about business use cases for shiny will be useful as some of us push for wider adoption.”

– ShinyConf 2024 Attendee

‍

“This was a great event! It was the first conference our Shiny R team has been to. I loved that it was virtual and the cost made it a no-brainer to my management. There was a good mix of speakers (some dev, some corporate, some scientist), which was cool.”

– ShinyConf 2024 Attendee

‍

To stay current on all things Shiny, follow us on social media, subscribe to Shiny Weekly, and join our community on Slack.

‍

We hope you had a great experience, and we look forward to seeing you at ShinyConf 2025!

‍

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Highlights from ShinyConf 2024

View Details

[This article was first published on Epiverse-TRACE developer space, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Throughout the Epiverse project, we use the renv R package to ensure reproducibility of the training materials and the pipelines we are providing. But we sometimes get reports from users who struggle to rebuild the environment and run the code.

In this post, we dissect the source of these issues, explain why in reality renv is not at fault, and how this is caused by the inherent complexity of reproducibility. The renv documentation already includes caveats explaining why some situations are bound to require more complex tools. This blog post reiterates some of these caveats and illustrates them with concrete examples.

Finally, we mention a couple of more complete (but more complex!) frameworks that can overcome the issues presented here. We do not explore these alternative framework in detail but provide links to more information.

Binaries vs building from sourceSoftware, including R packages, can generally be delivered in two forms: as binaries or as source code. If you are building from the source code, you may in some case need a compilation toolchain on your computer. If that toolchain is missing, it can lead to errors such as:

ld: warning: search path '/opt/gfortran/lib' not foundld: library 'gfortran' not found

Most of the time, regular users of R will not see these errors because they are installing binaries. Indeed, CRAN provides pre-compiled binaries for Windows and macOS for the last version of the package and R.

With renv, you often want to install older versions of the packages, which won’t be available as binaries from CRAN. This means you are more likely to have to compile the package yourself and see this kind of errors, even though renv is not causing them.

gfortran issues on Apple Silicon computersIf you are an Apple Silicon (Mac M1, M2, M3) user and encounter issues with gfortran, we have had success using the macrtools R package and we strongly recommend checking it out.

Beyond renv scope: incompatibility with system dependency versionsWe discussed previously the topic of system dependencies, and dependencies on specific R versions. These special dependencies can also be a source of headaches when using renv.

The heart of the issue is that renv provides a simplified solution to reproducibility: it focuses on R packages and their versions. But other sources of non-reproducibility are outside its scope. In many cases, this will not be a problem, as the main source of non-reproducibility, especially in the relatively short-term, will be R package versions.

But sometimes, it is possible that the renv.lock lockfile requires such an old version of an R package that it was written with a syntax that is no longer supported by recent R versions or modern compilers.

For example, a recent project (from 2023) was trying to install the version 0.60.1 of the matrixStats package (from 2021). This lead to this compilation error:

error: ‘DOUBLE_XMAX’ undeclared (first use in this function); did you mean ‘DBL_MAX’?

Click to see the full error message ! Error installing package 'matrixStats':=======================================* installing *source* package ‘matrixStats’ ...** package ‘matrixStats’ successfully unpacked and MD5 sums checked** using staged installation** libsusing C compiler: ‘gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0’gcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c 000.init.c -o 000.init.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c allocMatrix2.c -o allocMatrix2.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c anyMissing.c -o anyMissing.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c binCounts.c -o binCounts.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c binMeans.c -o binMeans.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c colCounts.c -o colCounts.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c colOrderStats.c -o colOrderStats.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c colRanges.c -o colRanges.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c diff2.c -o diff2.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c indexByRow.c -o indexByRow.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c logSumExp.c -o logSumExp.ogcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-H0vbME/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c mean2.c -o mean2.oIn file included from mean2\_lowlevel.h:14, from mean2.c:9:mean2\_lowlevel\_template.h: In function ‘mean2\_int’:mean2\_lowlevel\_template.h:59:13: error: ‘DOUBLE\_XMAX’ undeclared (first use in this function); did you mean ‘DBL\_MAX’? 59 | if (sum > DOUBLE\_XMAX) { | ^~~~~~~~~~~ | DBL\_MAXmean2\_lowlevel\_template.h:59:13: note: each undeclared identifier is reported only once for each function it appears inIn file included from mean2\_lowlevel.h:18, from mean2.c:9:mean2\_lowlevel\_template.h: In function ‘mean2\_dbl’:mean2\_lowlevel\_template.h:59:13: error: ‘DOUBLE\_XMAX’ undeclared (first use in this function); did you mean ‘DBL\_MAX’? 59 | if (sum > DOUBLE\_XMAX) { | ^~~~~~~~~~~ | DBL\_MAXmake: *** [/usr/lib/R/etc/Makeconf:191: mean2.o] Error 1ERROR: compilation failed for package ‘matrixStats’ The explanation for this error can be found in the matrixStats release notes, specifically the section for matrixStats 0.63.0:

Updated native code to use the C99 constant DBL_MAX instead of legacy S constant DOUBLE_XMAX, which is planned to be unsupported in R (>= 4.2.0).

Some solutionsAlternative package managersWe discussed how many issues when using renv can arise during the package compilation from source. A potential solution would be to avoid this compilation step and always install pre-compiled binaries.

This is not possible while installing from CRAN as CRAN only provides binaries for recent versions of R and for a limited number of platforms.

But Posit for example provides a larger collection of binaries, for different package versions, and different platforms, via their Public Posit Package Manager (PPM).

Making sure you install from PPM rather than CRAN can be a first simple step to make some of the issues discussed here vanish.

Extending the scope of reproducibilityAnother solution could be to add more complex reproducibility solutions that go beyond the scope of renv.

renv with rigThe R version is specified in renv.lock and to avoid incompatibility of older package versions with newer versions of R, you could run the declared R version. This can be achieved with various means but a convenient solution is the rig tool.

There are even some discussions to integrate rig and renv more tightly and let rig detect automatically which R version to use based on the renv.lock file.

Docker, Nix and othersAlternatively, you could use other reproducibility toolkits that focus not just on the R package versions, but on the entire software stack (e.g., including the operating system, the system dependencies). These solutions can be more complex to set up and use, and we won’t detail them in this blog post but you can find more information in:

The “Using renv with Docker” renv vignette
the “An Introduction to Rocker: Docker Containers for R” paper
Bruno Rodrigues’ entire series of blog posts on Nix

Conclusion: a final note for developersrenv is an elegant solution that focuses on the most immediate source of non-reproducibility. This however means it needs to be complemented by other tools in more complex cases.

Ultimately, reproducibility is a team effort. People who write code can minimise the risk of renv complications by keeping the packages they use close to their CRAN version and regularly updating their code and renv.lock accordingly. Other programming languages have automated tooling to help with this, via, e.g., the dependabot tool which submits pull requests to update dependencies. There is no well established equivalent for R yet, but anyone willing to set this mechanism up can look at the code used by the Carpentries workbench for this task.

Thanks to Pratik Gupte and Chris Hartgerink for their valuable comments on earlier drafts of this post.

ReuseCC BY 4.0CitationBibTeX citation: @online{gruson2024, author = {Gruson, Hugo}, title = {Things That Can Go Wrong When Using Renv}, date = {2024-12-31}, url = {https://epiverse-trace.github.io//posts/renv-complications}, langid = {en}} For attribution, please cite this work as:Gruson, Hugo. 2024. “Things That Can Go Wrong When UsingRenv.” December 31, 2024. https://epiverse-trace.github.io//posts/renv-complications. To leave a comment for the author, please follow the link and comment on their blog: Epiverse-TRACE developer space.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Things that can go wrong when using renv

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Introduction

Hello everyone!
I’m excited to announce the release of our latest collaborative effort (R package), designed to make complex consumer price and inflation calculations a breeze: emWeightedCPI .
Here I will Introduce you to what this package is about.

What is emWeightedCPI?

Our R package “emWeightedCPI” (hosted on github) stands for Expenditure based and Multivariate Weighted Consumer Price Index. This is the result of the combined effort of myself and two other talented individuals; Dr Paul A. Agbodza and George K. Agyen . It is a versatile tool that simplifies the calculation of standard Consumer Price Indices (CPI) and Inflation using Expenditure based and Multivariate Weights. The package automates a proposed multivariate weighted indexing scheme for price data. More information can be found [here].

Why Create emWeightedCPI Package?

Normally CPI is calculated from household expenditure data obtained from household expenditure surveys. However, these surveys are expensive making it difficult to conduct on a regular basis. This package introduces an alternative weighting approach that enables the computation of variable weights using price data only.

This approach is convenient since it does not require incurring additional cost for conducting household expenditure survey to generate CPI for the determination of inflation figures. Even so, one can still generate the Laspeyres’ CPI and inflation using this package.

The workings of The package

Using emWeightedCPI is as easy as taking a stroll! Begin by installing the package from github by using

install.packages(“devtools”)

devtools::install_github(“JC-Ayimah/emWeightedCPI”)

Once installed, load the package into your R environment with library(emWeightedCPI) . Now you’re all set to dive into the world emWeightedCPI and make use of its functions

How does emWeightedCPI work?

The package contains four main functions:

mvw_cpi: This function calculates the multivariate weighted indices . It requires only one argument (data); a price dataset containing prices of various items for a base year and a current year. It then calculates four index values (CPI values) based on the data provided and returns them as a named vector
mvw_inflation: Using the index values calculated by mvw_cpi, the mvw_inflation function computes the inflation rate based on the selected index. The function takes two arguments index and data . the index argument takes one of four possible values (indexes); ‘fisher’, ‘paashe’,‘laspeyres’ and ‘drobish’. The data argument requires the price dataset from which the inflation is to be determined.
eb_cpi: This function calculates consumer price indexI from expenditure based expenses. The function takes in two inputs, a price data and an expenditure data. The index calculated from this function is the ‘Laspeyres index’
eb_inflation: Like the mvw_inflation function, the eb_inflation function also calculates inflation based on the index calculated from the eb_cpi function. The function takes the same arguments specified in the eb_cpi function.

Usage Examples

Lets create a price data containing the prices of 4 different items for a base year and current year.

create an arbitrary price data

mypriceData <- data.frame(x1=runif(50, 9.9, 13.7), x2=rnorm(50, 10.9, 2.1), x3=runif(50, 12.2, 15), x4=runif(50,19.4, 24), # base year prices y1=runif(50, 26, 30), y2=runif(50, 31, 38.9), y3=runif(50, 28.2, 33.1), y4=runif(50, 51.8, 60)# current year prices )

To calculate the multivariate weighted indices simply use;

library(emWeightedCPI)

indices <- mvw_cpi(data = mypriceData)

indices

To calculate the multivariate weighted inflation based on a specific index (let’s say ‘fishers’) we use;

inflation_value <- mvw_inflation(index = ‘fisher’, data = mypriceData) inflation_value

The expenditure based index eb_cpi and inflation eb_inflation can also be calculated easily by using the codes as shown below. We need to generate an expenditure data to use together with our previously created price data in order to calculate the expenditure based index and inflation.

pick the average base year prices from mypriceData

n_vec <- apply(mypriceData[, 1:4], 2, mean)

n_vec

combine the output above into a dataframe with two columns and same

number of rows as number of price items to create expenditure data

myexpData <- cbind.data.frame(item = names(n_vec), price = (unname(n_vec)))

myexpData

calculate expenditure based CPI

exp_index <- eb_cpi(price_data = mypriceData, expenditure_data = myexpData)

exp_index

calculate expenditure based Inflation

exp_inflation <- eb_inflation(mypriceData, myexpData)

exp_inflation

ConclusionInnovation often thrives when minds come together, and emWeightedCPI is a testament to the power of collaboration. We’re incredibly proud of what we’ve achieved with this package, and we hope it becomes a valuable asset in your analytical toolkit.

Why Use emWeightedCPI?1. Ease of Use: The functions in emWeightedCPI are designed to be intuitive and straightforward to use. 2. Flexibility: Users can customize the calculations based on their specific requirements by adjusting the input parameters especially for the mvw_inflation function 3. Efficiency: With optimized algorithms, emWeightedCPI delivers fast and accurate results.

Get Started with emWeightedCPI Today!If you’re looking to simplify your consumer price index and inflation rate calculations in R, give emWeightedCPI a try! You can install it directly from github using:

install.packages(“devtools”)devtools::install_github(“JC-Ayimah/emWeightedCPI”)

For more information and detailed documentation, check out the emWeightedCPI GitHub repository.

We hope you find emWeightedCPI as useful and exciting as we do! Feel free to reach out with any questions, feedback, or suggestions.

Happy calculating!

Expenditure-Based and Multivariate Weighted Indices: An R Package to Calculate CPI and Inflation was first posted on May 3, 2024 at 1:01 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Expenditure-Based and Multivariate Weighted Indices: An R Package to Calculate CPI and Inflation

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post Extract values from vector in R: dplyr appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

Extract values from vector in R, we will delve into extracting specific values from a vector using the nth, first, and last functions from the dplyr package in R programming language.

The article is structured into four examples that demonstrate the extraction of vector elements.

Extract patterns in R? » Data Science Tutorials

Extract values from vector in RTo begin, by installing and loading the dplyr package in R.

install.packages("dplyr") library("dplyr") Our example vector, denoted as ‘x’, is a character vector containing NINE letters.

We then proceed to apply the functions to our example vector.

x <- letters[1:9] # Create example vectorx# "a" "b" "c" "d" "e" "f" "g" "h" "i" Data Science & business analytics »

Example 1: nth FunctionWe explore the usage of the nth function, which allows us to extract a vector element from anywhere within the vector.

By applying the nth function and specifying the position of the desired element, we obtain the desired output.

nth(x, 5) # Apply nth function# "e" #OUTPUT Monte Carlo Analysis in R » finnstats

Example 2: nth Function with Negative ValueExample 2 showcases the application of the nth function to extract an element from the end of the vector.

By placing a minus sign before the position, we can retrieve elements from the end of the vector.

nth(x, - 3) # Apply nth function with negative# "g" #output Example 3: first FunctionExample 3 demonstrates the first function, which returns the first element of an input vector.

first(x) # Apply first function# "a" #output This function works similarly to the nth function but is specifically designed to target the first element.

Example 4: last FunctionLastly, Example 4 illustrates the use of the last function, which returns the last element of a vector.

last(x) # Last function# "i" #return last value This function complements the first function by focusing on the end of the vector.

ConclusionThis article provides a comprehensive guide on how to use the nth, first, and last functions from the dplyr package in R to extract specific elements from a vector.

By understanding and applying these functions, users can effectively manipulate and analyze their data in the R programming environment.

How to add Axes to Plot in R
Understanding the Student’s t-Distribution in R
Solving Systems of Equations in R
Data Science for Business: Unraveling the Fundamentals of Analysis
Psychological Experimentation Software: OpenSesame

The post Extract values from vector in R: dplyr appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extract values from vector in R: dplyr

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In the great, unfolding narrative of J.R.R. Tolkien’s Ainulindalë, the world begins not with a bang, nor a word, but with a song. The Ainur, divine spirits, sing into the void at the behest of Ilúvatar, their voices weaving together to create a harmonious reality. Just as these divine voices layer upon each other to shape the physical and metaphysical landscapes of Middle-earth, data scientists and analysts use tools and techniques to orchestrate vast pools of data into coherent, actionable insights.

The realm of data science, particularly when wielded through the versatile capabilities of R, mirrors this act of creation. Just as each Ainu contributes a unique melody to the Great Music, each step in a data pipeline adds a layer of transformation, enriching the raw data until it culminates into a symphony of insights. The process of building data pipelines in R — collecting, cleaning, transforming, and storing data — is akin to conducting a grand orchestra, where every instrument must perform in perfect harmony to achieve the desired outcome.

This article is crafted for those who stand on the brink of their own creation myths. Whether you’re a seasoned data analyst looking to refine your craft or a burgeoning scientist just beginning to wield the tools of R, the following chapters will guide you through setting up robust data pipelines, ensuring that your data projects are as flawless and impactful as the world shaped by the Ainur.

As we delve into the mechanics of data pipelines, remember that each function and package in R is an instrument in your orchestra, and you are the conductor. Let’s begin by preparing our instruments — setting up the R environment with the right packages to ensure that every note rings true.

Preparing the Instruments: Setting Up Your R EnvironmentAs we take on the board of the creation of our data pipelines, akin to the Ainur tuning their instruments before the grand composition, it is crucial to carefully select our tools and organize our workspace in R. This preparation will ensure that the data flows smoothly through the pipeline, from raw input to insightful output.

Choosing the Right LibrariesIn the almost limitless repository of R packages, selecting the right ones is critical for efficient data handling and manipulation. Here are some indispensable libraries tailored for specific stages of the data pipeline:

Data Manipulation: dplyr offers a grammar of data manipulation, providing verbs that allow you to solve the most common data manipulation challenges elegantly.
Data Tidying: tidyr complements dplyr by providing a set of functions designed to transform irregular and complex data into a tidy format.
Data Importing and Exporting: readr for fast reading and writing of data files, readxl for Excel files, and DBI for database connections.
String Operations: stringr simplifies the process of manipulating strings.

Each package is selected based on its ability to handle specific tasks within the data pipeline efficiently, ensuring that each step is optimized for both performance and ease of use.

Organizing Your WorkspaceA well-organized working directory is essential for maintaining an efficient workflow. Setting your working directory in R to a project-specific folder helps in managing scripts, data files, and output systematically:

setwd("/path/to/your/project/directory") Beyond setting the working directory, structuring your project folders effectively is crucial:

Data Folder: Store raw data and processed data separately. This separation ensures that original data remains unmodified, serving as a reliable baseline.
Scripts Folder: Maintain your R scripts here. Organizing scripts by their purpose or order of execution can streamline your workflow and make it easier to navigate your project.
Output Folder: This should contain results from analyses, including tables, charts, and reports. Keeping outputs separate from data and scripts helps in version control and avoids clutter.

Project Management PracticesUsing an RStudio project can further enhance your workflow. Projects in RStudio make it easier to manage multiple related R scripts and keep all related files together. They also restore your workspace exactly as you left it, which is invaluable when working on complex data analyses.

Here’s a sample structure for a well-organized data project:

Project\_Name/│├── data/│ ├── raw/│ └── processed/│├── R/│ ├── cleaning.R│ ├── analysis.R│ └── reporting.R│└── output/ ├── figures/ └── reports/ By selecting the right libraries and organizing your R workspace and project folders strategically, you lay a solid foundation for smooth and effective data pipeline operations. Just as the Ainur needed harmony and precision to create the world, a well-prepared data scientist needs a finely tuned environment to bring data to life.

Gathering the Voices: Collecting DataIn the creation myth of Ainulindalë, each Ainur’s voice contributes uniquely to the world’s harmony. Analogously, in data science, the initial collection of data sets the tone for all analyses. This chapter will guide you through utilizing R to gather data from various sources, ensuring you capture a wide range of ‘voices’ to enrich your projects.

Understanding Data SourcesData can originate from numerous sources, each with unique characteristics and handling requirements:

Local Files: Data often resides in files like CSVs, Excel spreadsheets, or plain text documents.
Databases: These are structured collections of data, often stored in SQL databases like MySQL or PostgreSQL, or NoSQL databases such as MongoDB.
Web Sources: Many applications and services expose their data through web APIs, or data may be scraped directly from web pages.

Using R to Import DataR provides robust tools tailored for importing data from these varied sources, ensuring you can integrate them seamlessly into your analysis:

For CSV and Excel Files:

readr is highly optimized for reading large CSV files quickly and efficiently.
readxl extracts data from Excel files without needing Microsoft Excel.

library(readr)data\_csv <- read\_csv("path/to/your/data.csv")library(readxl)data\_excel <- read\_excel("path/to/your/data.xlsx") For Databases:

DBI is a database interface for R, which can be paired with database-specific packages like RMySQL for MySQL databases.

library(DBI)conn <- dbConnect(RMySQL::MySQL(), dbname = "database\_name", host = "host")data\_db <- dbGetQuery(conn, "SELECT * FROM table\_name") For Web Data:

rvest is ideal for scraping data from HTML web pages.
httr simplifies HTTP operations and is perfect for interacting with APIs.

library(rvest)web\_data <- read\_html("http://example.com") %>% html\_nodes("table") %>% html\_table()library(httr)response <- GET("http://api.example.com/data")api\_data <- content(response, type = "application/json") Practical Tips for Efficient Data CollectionTo maximize efficiency and accuracy in your data collection efforts, consider the following tips:

Check Source Reliability: Always verify the reliability and stability of your data sources.
Automate When Possible: For recurrent data needs, automate the collection process. Tools like cron jobs on Linux and Task Scheduler on Windows can be used to schedule R scripts to run automatically.
Data Storage: Properly manage the storage of collected data. Even if the data is temporary, organize it in a manner that supports efficient access and manipulation.

Mastering the collection of data using R equips you to handle the foundational aspect of any data analysis project. By ensuring you have robust, reliable, and diverse data, your analyses can be as nuanced and comprehensive as the world crafted by the Ainur’s voices.

Refining the Harmony: Cleaning DataJust as a symphony conductor must ensure that every instrument is precisely tuned to contribute to a harmonious performance, a data scientist must refine their collected data to ensure it is clean, structured, and ready for analysis. This chapter will guide you through the crucial process of cleaning data using R, which involves identifying and correcting inaccuracies, inconsistencies, and missing values in your data set.

Identifying Common Data IssuesBefore diving into specific techniques, it’s essential to understand the common issues that can arise with raw data:

Missing Values: Data entries that are empty or contain placeholders that need to be addressed.
Duplicate Records: Repeated entries that can skew analysis results.
Inconsistent Formats: Data coming from various sources may have different formats or units, requiring standardization.
Outliers: Extreme values that could be errors or require separate analysis.

Using R Packages for Data CleaningR provides several packages that make the task of cleaning data efficient and straightforward:

tidyr: This package is instrumental in transforming data to a tidy format where each variable forms a column, each observation forms a row, and each type of observational unit forms a table.
dplyr: Useful for modifying data frames by removing duplicates, filtering out unwanted observations, and transforming data using its various functions.

Techniques for Cleaning DataHere are some simple techniques to clean data effectively using R:

```

Handling Missing Valueslibrary(tidyr)cleaned_data <- raw_data %>% drop_na() # Removes rows with any NA values### Removing duplicateslibrary(dplyr)unique_data <- raw_data %>% distinct() # Removes duplicate rows### Standardizing Data Formats# Converting all character strings to lowercase for consistencystandardized_data <- raw_data %>% mutate_all(~tolower(.))### Dealing with Outliers# Identifying outliers based on statistical thresholdsbounds <- quantile(raw_data$variable, probs=c(0.01, 0.99))filtered_data <- raw_data %>% filter(variable > bounds[1] & variable < bounds[2])

``` Ensuring Data QualityPost-cleaning, it’s important to verify the quality of your data:

Summarize Data: Get a quick overview using summary() to check if the data meets the expected standards.
Visual Inspections: Plot your data using packages like ggplot2 to visually inspect for any remaining issues.

The meticulous process of cleaning your data in R ensures that it is reliable and ready for detailed analysis. Just as the Ainur’s song required balance and precision to create a harmonious world, thorough data cleaning ensures that your analyses can be conducted without discord, leading to insights that are both accurate and actionable.

Shaping the Melody: Transforming DataOnce the data is cleansed of imperfections, the next task is akin to a composer arranging notes to create a harmonious melody. In the context of data science, transforming data involves reshaping, aggregating, or otherwise modifying it to better suit the needs of your analysis. This chapter explores how to use R to transform your cleaned data into a format that reveals deeper insights and prepares it for effective analysis.

Understanding Data TransformationData transformation includes a variety of operations that modify the data structure and content:

Aggregation: Combining multiple entries to reduce the data size and highlight important features.
Normalization: Scaling data to a specific range, useful for comparison and modeling.
Feature Engineering: Creating new variables from existing ones to enhance model predictability.

Utilizing R for Data TransformationR offers powerful libraries tailored for these tasks, allowing precise control over the data transformation process:

dplyr: This package is essential for efficiently transforming data frames. It provides a coherent set of verbs that help you solve common data manipulation challenges.
tidyr: Helps in changing the layout of your data sets to make data more tidy and accessible.

Techniques for Transforming Data ```

Aggregating Data:library(dplyr)aggregated_data <- raw_data %>% group_by(category) %>% summarize(mean_value = mean(value, na.rm = TRUE))### Normalizing Data:normalized_data <- raw_data %>% mutate(normalized_value = (value - min(value)) / (max(value) - min(value)))### Feature Engineering:engineered_data <- raw_data %>% mutate(new_feature = log(old_feature + 1))

``` Best Practices in Data TransformationTo ensure that the transformed data is useful and relevant for your analyses, consider the following practices:

Relevance of Transformations: Make sure that the transformations align with your analytical objectives.
Maintainability: Document the transformations clearly, ensuring they are understandable and reproducible.
Efficiency: Optimize transformations for large data sets to prevent performance bottlenecks.

Transforming data effectively allows you to sculpt the raw, cleaned data into a form that is not only analytically useful but also rich in insights. Much like the careful crafting of a symphony from basic musical notes, skillful data transformation in R helps unfold the hidden potential within your data, enabling deeper and more impactful analyses.

Preserving the Echoes: Storing DataAfter transforming and refining your data, the next critical step is to store it effectively. Much like the echoes of the Ainur’s music that shaped the landscapes of Arda, the data preserved in storage will form the foundation for all future analysis and insights. This chapter explores the various data storage options available in R and how to implement them efficiently.

Introduction to Data Storage Options in RData can be stored in several formats, each with its own advantages depending on the use case:

.RData/.Rds: These are R's native file formats. .RData can store multiple objects in a single file, whereas .Rds stores one object per file.
Parquet: A compressed, efficient columnar data storage format optimized for use with complex data structures that supports advanced read and write capabilities.
Text and CSV Files: Simple, widely used formats that are easily readable by humans and other software, though not as space-efficient.

Choosing the Right FormatThe choice of format depends on your needs:

For large datasets: Consider using Parquet for its efficiency in storage and speed in access, especially useful for complex analytical projects.
For R-specific projects: Use .RData and .Rds for their native compatibility and ability to preserve R objects exactly as they are in your environment.
For interoperability: Use CSV files when you need to share data with systems or individuals who may not be using R.

Saving Data EfficientlyTo save data efficiently, consider the following R functions:

```

Saving a single R objectsaveRDS(object, file = "path/to/save/object.Rds")# Saving multiple R objectssave(object1, object2, file = "path/to/save/objects.RData")# Writing to a Parquet filelibrary(arrow)write_parquet(data_frame, "path/to/save/data.parquet")# Writing to a CSV filewrite.csv(data_frame, "path/to/save/data.csv")

``` These methods ensure that your data is stored in a manner that is not only space-efficient but also conducive to future accessibility and analysis.

By carefully selecting the appropriate storage format and effectively utilizing R’s data-saving functions, you ensure that your data is preserved accurately and efficiently. This practice not only secures the data for future use but also maintains its integrity and accessibility, akin to the lasting and unaltered echoes of a timeless melody.

Conducting the Orchestra: Automating and Orchestrating Data PipelinesAutomation serves as the conductor in the symphony of data analysis, ensuring that each component of the data pipeline executes in perfect harmony and at the right moment. This chapter explores how to automate and orchestrate data pipelines in R, enhancing both efficiency and reliability through advanced tools designed for task scheduling and workflow management.

The Importance of AutomationAutomation in data pipelines is crucial for:

Consistency: Automatically executing tasks reduces the risk of human error and ensures uniformity in data processing.
Efficiency: Frees up data professionals to focus on higher-level analysis and strategic tasks.
Scalability: As data volumes grow, automated pipelines can handle increased loads without needing proportional increases in manual oversight.

Using R to Automate Data PipelinesR offers several tools for automation, from simple script scheduling to sophisticated workflow management:

taskscheduleR: This package allows for the scheduling of R scripts on Windows systems. It is instrumental in ensuring that data collection, processing, and reporting tasks are performed without manual intervention.
targets: A powerful package that creates and manages complex data pipelines in R, handling task dependencies and ensuring that the workflow is reproducible and efficient.

Examples of Creating Automated Workflows ```

Scheduling Data Collection with taskscheduleRlibrary(taskscheduleR)script_path <- "path/to/your_data_collection_script.R"# Schedule the script to run daily at 7 AMtaskscheduler_create(taskname = "DailyDataCollection", rscript = script_path, schedule = "DAILY", starttime = "07:00")### Building a Data Pipeline with targets:library(targets)# Example of a targets pipeline definitiontar_script({ list( tar_target( raw_data, readr::read_csv("path/to/data.csv"), # Data collection format = "file" ), tar_target( clean_data, my_cleaning_function(raw_data), # Data cleaning pattern = map(raw_data) ), tar_target( analysis_results, analyze_data(clean_data), # Data analysis pattern = cross(clean_data) ) )})

``` Best Practices for Pipeline Automation Monitoring and Logging: Implement logging within scripts to track when tasks run and capture any errors or critical warnings. * Regular Reviews: Periodically review and update the scripts, schedules, and data dependencies to adapt to new business needs or data changes. * Security Protocols*: Ensure all automated tasks, especially those interacting with sensitive data or external systems, adhere to strict security protocols to prevent unauthorized access.

Effective automation of data pipelines in R not only ensures that data processes are conducted with precision and timeliness but also scales up to meet the demands of complex data environments. By employing tools like taskscheduleR and targets, you orchestrate a smooth and continuous flow of data operations, much like a conductor leading an orchestra to deliver a flawless performance.

Resolving Dissonances: Robustness and Error Handling in Data PipelinesJust like a skilled composer addresses dissonances within a symphony, a data scientist must ensure data pipelines are robust enough to handle unexpected issues effectively. This chapter outlines strategies to enhance the robustness of data pipelines in R and offers practical solutions for managing errors efficiently.

The Need for Robustness in Data PipelinesRobust data pipelines are crucial for ensuring:

Reliability: They must perform consistently under varying conditions and with different data inputs.
Maintainability: They should be easy to update or modify without disrupting existing functionalities.
Resilience: They need to recover quickly from failures to minimize downtime and maintain data integrity.

Enhancing Pipeline Robustness with RR provides several tools and strategies to help safeguard your data pipelines:

Error Handling Mechanisms: tryCatch() allows you to manage errors effectively, executing alternative code when errors occur.
Logging: Tools like futile.logger or logger help record operations and errors, providing a trail that can be used to diagnose issues.

Implementing Error Handling TechniquesEffective error management involves several key strategies:

```

Preventive Checks:# Early data quality checksif(anyNA(data)) { stop("Data contains NA values. Please check the source.")}### Graceful Error Management with tryCatch():library(logger)robust_processing <- function(data) { tryCatch({ result <- some_risky_operation(data) log_info("Operation successful.") return(result) }, error = function(e) { log_error("Error in processing: ", e$message) send_alert_to_maintainer("Processing error encountered: " + e$message) NULL # Return NULL or handle differently })}### Notification System:### Implementing an alert system can significantly improve the responsiveness to issues. Here’s how you can integrate such a system to send messages to the maintainer when something goes wrong:send_alert_to_maintainer <- function(message) { # Assuming you have a function to send emails or messages mailR::send.mail(to = "maintainer@example.com", subject = "Data Pipeline Error Alert", body = message)}

``` Best Practices for Robust Pipeline Design Comprehensive Testing: Routinely test the pipeline using a variety of data scenarios to ensure robust handling of both typical and edge cases. * Regular Audits: Conduct periodic reviews of the pipeline to identify and rectify potential vulnerabilities before they cause failures. * Detailed Documentation and Training*: Keep thorough documentation of the pipeline’s design and operational protocols. Ensure team members are trained on how to respond to different types of errors or failures.

In the narrative of Ainulindalë, it is Melkor who introduces dissonance into the harmonious music of the Ainur, creating chaos amidst creation. Similarly, in the world of data pipelines, unexpected errors and issues can be seen as dissonances introduced by Melkor-like challenges, disrupting the flow and function of our carefully orchestrated processes. By foreseeing these potential disruptions and implementing effective error handling and notification mechanisms, we ensure that our data pipelines can withstand and adapt to these challenges. This approach not only preserves the integrity of the data analysis but also ensures that the insights derived from this data remain accurate and actionable, keeping the symphony of data in continuous, harmonious play despite Melkor’s attempts to thwart the music.

Among the Ainur: Integrating R with Other TechnologiesIn the grand ensemble of data technologies, R plays a role akin to one of the Ainur, a powerful entity with unique capabilities. However, just like the Ainur were most effective when collaborating under Ilúvatar’s grand plan, R reaches its fullest potential when integrated within diverse technological environments. This chapter discusses how R can be seamlessly integrated with other technologies to enhance its utility and broaden its applicational horizon.

R’s Role in Diverse Data EcosystemsR is not just a standalone tool but a part of a larger symphony that includes various data management, processing, and visualization technologies:

Cloud Computing Platforms: R can be used in cloud environments like AWS, Google Cloud, and Azure to perform statistical analysis and modeling directly on data stored in the cloud, leveraging scalable computing resources.
Big Data Platforms: Integrating R with big data technologies such as Apache Hadoop or Apache Spark enables users to handle and analyze data at scale, making R a valuable tool for big data analytics.
Data Warehousing: R can interface with data warehouses like Amazon Redshift, Snowflake, and others, which allows for sophisticated data extraction, transformation, and loading (ETL) processes, enriching the data analysis capabilities of R.
Business Intelligence Tools: Tools like Tableau, Power BI, and Looker can incorporate R for advanced analytics, bringing statistical rigor to business dashboards and reports.
Machine Learning Platforms: R’s integration with machine learning platforms like TensorFlow or PyTorch through various packages enables the development and deployment of complex machine learning models.
Workflow Automation Platforms: R can be a component in automated workflows managed by platforms like Alteryx or Knime, which facilitate the blending of data, execution of R scripts, and publication of results across a broad user base.

Enhancing Collaboration with Other TechnologiesIntegrating R with other technologies involves not only technical synchronization but also strategic alignment:

Complementary Use Cases: Identify scenarios where R’s statistical and graphical tools can complement other platforms’ strengths, such as using R for ad-hoc analyses and modeling while using SQL databases for data storage and management.
Hybrid Approaches: Leverage the strengths of each technology by employing hybrid approaches. For instance, preprocess data using SQL or Python, analyze it with R, and then visualize results using a BI tool.
Unified Data Strategy: Develop a cohesive data strategy that aligns the data processing capabilities of R with other enterprise tools, ensuring seamless data flow and integrity across platforms.

R’s ability to integrate with a myriad of technologies transforms it from a solitary tool into a pivotal component of comprehensive data analysis strategies. Like the harmonious interplay of the Ainur’s melodies under Ilúvatar’s guidance, R’s integration with diverse tools and platforms allows it to contribute more effectively to the collective data analysis and decision-making processes, enriching insights and fostering informed business strategies.

The Theme Resounds: ConclusionAs our journey through the orchestration of data pipelines in R comes to a close, we reflect on the narrative of the Ainulindalë, where the themes of creation, harmony, and collaboration underpin the universe’s foundation. Similarly, in the realm of data science, the harmonious integration of various technologies and practices, guided by the powerful capabilities of R, forms the bedrock of effective data analysis.

Throughout this guide, we’ve explored:

Setting up and preparing R environments for data handling, emphasizing the importance of selecting the right tools and organizing workspaces efficiently.
Collecting, cleaning, and transforming data, which are critical steps that ensure the quality and usability of data for analysis.
Storing data efficiently in various formats, ensuring that data preservation aligns with future access and analysis needs.
Automating and orchestrating data pipelines to enhance efficiency and consistency, reducing manual overhead and increasing the reliability of data processes.
Integrating R with a multitude of technologies from cloud platforms to business intelligence tools, demonstrating R’s versatility and collaborative potential in broader data ecosystems.

The field of data science, much like the ever-evolving music of the Ainur, is continually expanding and transforming. As new technologies emerge and existing ones mature, the opportunities for integrating R into your data pipelines will only grow. Exploring these possibilities not only enriches your current projects but also prepares you for future advancements in data analysis.

Just as the Ainur’s music shaped the very fabric of Middle-earth, your mastery of data pipelines in R can significantly influence the insights and outcomes derived from your data. The tools and techniques discussed here are but a foundation — continuing to build upon them, integrating new tools, and refining old ones will ensure that your data pipelines remain robust, harmonious, and forward-looking.

As we conclude this guide, remember that the theme of harmonious data handling resounds beyond the pages. It is an ongoing symphony that you contribute to with each dataset you manipulate and every analysis you perform. Let the principles of robustness, integration, and automation guide you, and continue to explore and expand the boundaries of what you can achieve with R in the vast universe of data science.

Ainulindalë in R: Orchestrating Data Pipelines for World Creation was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Ainulindalë in R: Orchestrating Data Pipelines for World Creation

View Details

[This article was first published on R Archives » Data Science Tutorials, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The post How to add Axes to Plot in R appeared first on Data Science Tutorials

Unravel the Future: Dive Deep into the World of Data Science Today! Data Science Tutorials.

How to add Axes to Plot in R, In the world of data visualization, creating visually appealing and informative plots is crucial for effectively communicating insights.

The R programming language offers a plethora of tools to customize your plots, including the ability to add user-defined axis ticks using the axis() function.

In this article, we will walk you through three examples that demonstrate how to create plots with custom axis ticks in R.

Predict potential customers in R » Data Science Tutorials

Example 1: Draw Plot with Default Axis TicksBefore diving into custom axis ticks, let’s first learn how to draw a basic plot with default axis specifications.

The plot() function in R can be used to create a scatterplot, as shown in the code snippet below:

plot(1:200) # Default plot This code will generate a scatterplot with default axis values.

Example 2: Plot with Specified Axis TicksNow, let’s move on to adding user-defined axis labels using the axis() function. First, we need to create a graph without any axis values:

plot(1:200, # Plot without axes xaxt = "n", yaxt = "n") Once the plot is created without axes, we can use the axis() function to add axis values.

The side parameter is used to specify which axis to modify, with 1 representing the x-axis and 2 representing the y-axis.

How to Label Outliers in Boxplots in ggplot2? (datasciencetut.com)

The c() function is used to define the tick values for the respective axis:

axis(side = 1, # Draw x-axis c(0, 50, 100, 150, 200)) #Just an exampleaxis(side = 2, # Draw y-axis c(10, 50, 150)) #Just an example Example 3: Plot with Axis Mark on Top & Right SideIn some cases, you may want to add axis ticks on the top and right side of the plot.

This can be achieved using the same R code as in the previous example, but with different values for the side parameter.

Instead of 1 and 2, we will use 3 and 4 to represent the top and right axes, respectively:

plot(1:200, # Plot without axes xaxt = "n", yaxt = "n")axis(side = 3, # Add axis on top c(0, 50, 100, 200)) #Just an exampleaxis(side = 4, # Add axis on right side c(0, 50, 110)) #Just an example Plotting Equations in R »

The post How to add Axes to Plot in R appeared first on Data Science Tutorials

Unlock Your Inner Data Genius: Explore, Learn, and Transform with Our Data Science Haven! Data Science Tutorials.

To leave a comment for the author, please follow the link and comment on their blog: R Archives » Data Science Tutorials.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to add Axes to Plot in R

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.What to get a feel for the kind of content will be available at R/Medicine 2024? We’re spotlighting the most engaging and educational sessions from past R Medicine Virtual Conferences. Whether you’re a healthcare professional, a data scientist, or simply curious about the intersection of healthcare and technology, these selected talks offer a wealth of knowledge and innovation using the R programming language. Dive into these sessions to enhance your understanding and skills in medical data science.

Register for the R Medicine 2024 Virtual Conference here!1. GitHub Copilot in Rstudio, It’s Finally Here! – R Medicine Virtual Conference 2023This session introduces GitHub Copilot for RStudio, a highly anticipated tool that enhances coding efficiency and innovation in medical research. Watch as experts demonstrate its capabilities and potential impact on healthcare data analysis.

Analyzing Geospatial Data in R (Sherrie Xie) – R/Medicine 2022 Virtual ConferenceFeaturing Sherrie Xie, this presentation explores the applications of geospatial data analysis within the healthcare sector using R. Gain insights into the importance of spatial data in understanding health trends and outcomes.
R/Medicine 101: Intro to R for Clinical Data (Stephan Kadauke, Joe Rudolf, Patrick Mathias) – R/Medicine 2022This introductory session is perfect for those new to using R in a clinical setting. The speakers guide you through the basics and demonstrate how R can revolutionize medical research and patient care.
Introduction to R for Medical DataTidy Spreadsheets in Medical Research – R/Medicine 2021UMich Prof and {medicaldata} author Peter Higgins will cover best practices for using medical data in spreadsheets like Excel and Google Sheets.
Multistate Data Using the {survival} Package – R/Medicine 2021Explore the use of the {survival} package in R for analyzing multistate data. Discover the methods and models that are shaping the future of survival analysis in medical research.

Engage and Learn More!

Each of these sessions provides unique insights and practical tools for harnessing the power of R in medical research and healthcare analytics. Whether you are watching these for the first time or revisiting them, each video promises a deep dive into the capabilities of R that are driving advancements in the field.

Mark Your Calendars! The R Medicine Conference for this year is scheduled for June 10-14. Register now to secure your spot and connect with a community of like-minded professionals!

Register for the R Medicine 2024 Virtual Conference here!Remember to subscribe to the R Medicine channel for more updates and upcoming conference information. Enhance your skills in medical data science today!

The post R/Medicine is coming June 10-14, 2024 – See Top Five R Medicine Talks from Previous Years appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R/Medicine is coming June 10-14, 2024 – See Top Five R Medicine Talks from Previous Years

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Are you an R/Shiny user looking to leverage the incredible capabilities of Shiny for Python without sacrificing the familiarity and comfort of your existing tools?

Introducing Tapyr—our Shiny for Python framework. It brings Rhino-like capabilities from the R world and more to the Shiny for Python ecosystem, helping you build enterprise-ready applications with ease.‍

Curious about Shiny for Python from an R Shiny dev’s perspective? Check out this blog post to learn more.

Tapyr is designed as a lightweight template repository for PyShiny projects that offers tools similar to Rhino for R/Shiny. For instance, Tapyr introduces poetry, which handles project dependencies much like renv in R. This ensures that R users can smoothly adapt to Python without tackling a steep learning curve while adhering to best practices from day 0.

Key Features of Tapyr Leverage Python Tools: Tapyr takes advantage of Python’s ecosystem tools, including ruff, pytest, and others. * Enterprise-Ready Applications, Made Easy: The framework is tailored for building robust, scalable, and production-ready applications. * Comprehensive Testing with Playwright: Say goodbye to the hassle of juggling multiple languages for end-to-end testing. Tapyr leverages Playwright, integrated with pytest, allowing you to write all tests in Python – a streamlined approach that keeps your coding practices consistent and efficient. * Static Type Checking with PyRight:* Improve code quality and reduce bugs with PyRight, a static type checking feature not available in R. This proactive error detection ensures your applications are reliable, before you even start them.

Complementing Existing ResourcesWhile Posit’s PyShiny templates cater to exploratory data analysis, Tapyr serves a distinct, complementary role by providing a structured repository designed to kickstart your projects. This approach focuses on developing comprehensive, scalable and future-proof applications.

This not only expands the tools available to data scientists and developers but also helps you to tackle larger, more complex projects effectively.

Tapyr is ideal for data scientists (transitioning from R to Python), developers familiar with Shiny and Rhino building projects in PyShiny, and academic researchers and enterprise professionals requiring enterprise-level dashboard frameworks.

Getting Started with TapyrUsing DevcontainerWe recommend using the Dev Container configuration with Visual Studio Code (VS Code) or DevPod to ensure a consistent development experience across different computers and environments. It may sound complicated, but it is as easy as a breeze!

‍

The Dev Container is like a virtual environment with everything you need to work on the project, including all the required software and dependencies.

‍

Install Dev Containers extension if you don’t have it already.

VS Code Dev Containers‍

Clone the repository and start the dev container: You can clone the Tapyr repository from GitHub or download the source code. some text
1. Navigate to the project directory and open the project in VS Code.
2. Select “Reopen in Container” when prompted.
3. If you’re prompted to “Reopen in Container,” select that option. If not, you can open the Command Palette (Ctrl+Shift+P on Windows/Linux, or Cmd+Shift+P on Mac) and choose “Remote-Containers: Reopen in Container.”
4. If you’re using DevPod, follow their instructions to start the Devcontainer.

‍

Reopen in Container‍

Activate the virtual environment: Once the Dev Container is running, you’ll need to activate the virtual environment (creating a special workspace where all the project’s dependencies are installed). Run the following command:

poetry shell ‍

Activate in virtual environment4. Run the application: Now you’re ready to run the application! Use this command:

shiny run app.py --reload Run the applicationThis will start the application and automatically reload it whenever you make changes to the code.

Tapyr | PyShiny Template5. Execute tests: To run tests and ensure everything is working correctly, use this command:

poetry run pytest If you prefer to run this locally, you can do so using Poetry.

Struggling with Quality Assurance for your Shiny for Python Dashboards? Check out this blog post to learn more about leveraging Playwright.

Get Started TodayDive into Tapyr and start building your enterprise-level applications today!

Download Tapyr, check out the documentation, explore its functionalities, and join the community of innovators expanding their PyShiny skillsets.

We value your feedback, so please share your experiences and suggestions to help us improve Tapyr in our Shiny community.

‍

Want to stay up to date with Tapyr and other packages? Join 4.2k explorers and get the Shiny Weekly Newsletter into your mailbox.

FAQsQ: Is there a community or support available for Tapyr users?

A: You can create a pull request, open an issue, follow our documentation, and engage with other users in our community to get support, share insights, and contribute to the project’s development.

‍

Q: How is Tapyr different from Posit’s PyShiny templates?

A: While Posit’s PyShiny templates focus on exploratory data analysis, Tapyr is a framework focused on building comprehensive, scalable PyShiny applications.

‍

Q: How does Tapyr compare to other tools like reticulate?

A: While reticulate allows you to call Python from R, Tapyr takes a different approach by providing a streamlined framework for building enterprise-ready applications using Shiny for Python. Since all the code is written in Python, it offers features like static type checking, comprehensive testing with Playwright, and seamless integration with Python ecosystems.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing Tapyr: Create and Deploy Enterprise-Ready PyShiny Dashboards with Ease

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#177–178

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #177We are about to create reports for three students, each of them have to pass 3 exams for each subject. If even one of them has mark lower than 40, person fails the subject. Then we have to find their averages per person and rank acording to it. And make small adjustment in first columns, not to repeat names for every subject. Check the code.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_177.xlsx", range = "A1:F10")test = read\_excel("Power Query/PQ\_Challenge\_177.xlsx", range = "H1:M10") Transformation result = input %>% rowwise() %>% mutate(Result = if\_else(any(c(Marks1, Marks2, Marks3) < 40), "Fail", "Pass"), total = Marks1 + Marks2 + Marks3) %>% ungroup() %>% mutate(average = mean(total), rn = row\_number(), .by = Name) aux\_rank = result %>% select(Name, average) %>% distinct() %>% mutate(Rank = rank(-average))result2 = result %>% left\_join(aux\_rank, by = "Name") %>% select(Name, Classs, Subject, `Total Marks` = total, Result, Rank, rn) %>% mutate(Name = ifelse(rn == 1, Name, NA\_character\_), Classs = ifelse(rn == 1, Classs, NA\_real\_), Rank = ifelse(rn == 1, Rank, NA\_integer\_)) %>% select(-rn) Validation identical(result2, test) #> [1] TRUE Puzzle #178HR should track how people they hire are advancing in positions, get promotions, but also if they are moving to another department or employer. We had input table containing of records of such events. But we need it to be more readable. Lets pivot the data.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_178.xlsx", range = "A1:E5")test = read\_excel("Power Query/PQ\_Challenge\_178.xlsx", range = "H1:K5") Transformation result = input %>% pivot\_longer(-Emp, names\_to = "Change", values\_to = "Value") %>% separate(Change, into = c("Type", "Change"), sep = " ") %>% pivot\_wider(names\_from = Type, values\_from = Value) %>% drop\_na() # but one of my co-solver asked, why didn't I use only one pivot\_longer# check Anil Kumar Goyal's solution then.input %>% pivot\_longer(-Emp, names\_to = c(".value", "Change"), names\_sep = " ") %>% na.omit() Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#177–178

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn data analysis, especially when dealing with multiple samples or distributions, ensuring comparability and removing biases is crucial. One powerful technique for achieving this is quantile normalization. This method aligns the distributions of values across different samples, making them more similar in terms of their statistical properties.

What is Quantile Normalization?Quantile normalization is a statistical method used to adjust the distributions of values in different datasets so that they have similar quantiles. This technique is particularly valuable when working with high-dimensional data, such as gene expression data or other omics datasets, where ensuring comparability across samples is essential.

Introducing quantile_normalize() in TidyDensityThe quantile_normalize() function is a new addition to the TidyDensity package, designed to simplify the process of quantile normalization within R. Let’s delve into how this function works and how you can integrate it into your data analysis pipeline.

Function UsageThe quantile_normalize() function takes a numeric matrix as input, where each column represents a sample. Here’s a breakdown of its usage:

quantile\_normalize(.data, .return\_tibble = FALSE) * .data: A numeric matrix where each column corresponds to a sample that requires quantile normalization. * .return_tibble: A logical value (default: FALSE) indicating whether the output should be returned as a tibble.

Understanding the OutputWhen you apply quantile_normalize() to your data, you receive a list object containing the following components:

Quantile-Normalized Matrix: A numeric matrix where each column has been quantile-normalized.
Row Means: The means of each row across the quantile-normalized matrix.
Sorted Data: The sorted values used during the quantile normalization process.
Ranked Indices: The indices of the sorted values.

How Quantile Normalization WorksThe quantile_normalize() function performs quantile normalization through the following steps:

Sorting: Each column of the input matrix is sorted.
Row Mean Calculation: The mean of each row across the sorted columns is computed.
Normalization: Each column’s sorted values are replaced with the corresponding row means.
Unsorting: The columns are restored to their original order, ensuring that the quantile-normalized matrix maintains the same structure as the input.

ExamplesLet’s demonstrate the usage of quantile_normalize() with a simple example:

```

Load TidyDensitylibrary(TidyDensity)# Create a sample matrixset.seed(123)data <- matrix(rnorm(50), ncol = 4)head(data, 5)

```

[,1] [,2] [,3] [,4][1,] -0.56047565 0.1106827 0.8377870 -0.3804710[2,] -0.23017749 -0.5558411 0.1533731 -0.6947070[3,] 1.55870831 1.7869131 -1.1381369 -0.2079173[4,] 0.07050839 0.4978505 1.2538149 -1.2653964[5,] 0.12928774 -1.9666172 0.4264642 2.1689560

```

Apply quantile normalizationresult <- quantile_normalize(data)# Access the quantile-normalized matrixnormalized_matrix <- result[["normalized_data"]]# View the normalized matrixhead(normalized_matrix, 5)

```

[,1] [,2] [,3] [,4][1,] -0.65451945 -0.3180877 0.84500772 -0.6545195[2,] -0.06327669 0.8450077 1.09078797 -0.9506544[3,] -1.40880292 -0.5235134 0.33150422 0.0863713[4,] 0.84500772 1.0907880 0.08637130 0.1991151[5,] -0.31808774 -0.6545195 -0.06327669 0.3315042 Let’s now look at the rest of the output components:

head(result[["row\_means"]], 5)

[1] -1.4088029 -0.9506544 -0.6545195 -0.5235134 -0.3180877

head(result[["duplicated\_ranks"]], 5)

[,1] [,2] [,3] [,4][1,] 9 13 13 7[2,] 10 10 12 12[3,] 2 11 2 9[4,] 13 9 9 3[5,] 7 1 1 11

head(result[["duplicated\_rank\_row\_indicies"]], 5)

NULL

head(result[["duplicated\_rank\_data"]], 5)

[,1] [,2] [,3] [,4][1,] -0.23017749 -0.5558411 0.1533731 -0.6947070[2,] 0.07050839 0.4978505 1.2538149 -1.2653964[3,] 0.12928774 -1.9666172 0.4264642 2.1689560[4,] -0.68685285 -0.2179749 0.8215811 -0.4666554[5,] -0.44566197 -1.0260044 0.6886403 0.7799651 Now, lets take a look at the before and after quantile normalization summary:

as.data.frame(data) |> sapply(function(x) quantile(x, probs = seq(0, 1, 1/4)))

V1 V2 V3 V40% -1.2650612 -1.9666172 -1.13813694 -1.2653963525% -0.4456620 -1.0260044 -0.06191171 -0.5604756550% 0.1292877 -0.5558411 0.55391765 -0.3804710075% 0.4609162 0.1106827 0.83778704 -0.08336907100% 1.7150650 1.7869131 1.25381492 2.16895597

as.data.frame(normalized\_matrix) |> sapply(function(x) quantile(x, probs = seq(0, 1, 1/4)))

V1 V2 V3 V40% -1.40880292 -1.40880292 -1.40880292 -1.4088029225% -0.52351344 -0.52351344 -0.52351344 -0.5235134450% -0.06327669 -0.06327669 -0.06327669 -0.0632766975% 0.33150422 0.33150422 0.33150422 0.33150422100% 1.73118725 1.73118725 1.73118725 1.73118725 Now let’s use the .return_tibble argument to return the output as a tibble:

quantile\_normalize(data, .return\_tibble = TRUE)

$normalized\_data# A tibble: 13 × 4 V1 V2 V3 V4 <dbl> <dbl> <dbl> <dbl> 1 -0.655 -0.318 0.845 -0.655 2 -0.0633 0.845 1.09 -0.951 3 -1.41 -0.524 0.332 0.0864 4 0.845 1.09 0.0864 0.199 5 -0.318 -0.655 -0.0633 0.332 6 1.73 -0.0633 -0.133 -0.133 7 -0.524 -0.133 -0.524 -0.524 8 -0.133 1.73 1.73 1.73 9 0.332 0.0864 0.199 1.09 10 1.09 -0.951 -0.655 -0.318 11 -0.951 -1.41 -0.318 -1.41 12 0.199 0.199 -1.41 0.845 13 0.0864 0.332 -0.951 -0.0633$row\_means# A tibble: 13 × 1 value <dbl> 1 -1.41 2 -0.951 3 -0.655 4 -0.524 5 -0.318 6 -0.133 7 -0.0633 8 0.0864 9 0.199 10 0.332 11 0.845 12 1.09 13 1.73 $duplicated\_ranks# A tibble: 6 × 4 V1 V2 V3 V4 <int> <int> <int> <int>1 9 13 13 72 10 10 12 123 2 11 2 94 13 9 9 35 7 1 1 116 3 6 7 6$duplicated\_rank\_row\_indices# A tibble: 6 × 1 row\_index <int>1 22 43 54 95 106 12$duplicated\_rank\_data# A tibble: 6 × 4 V1 V2 V3 V4 <dbl> <dbl> <dbl> <dbl>1 -0.230 -0.556 0.153 -0.6952 0.0705 0.498 1.25 -1.27 3 0.129 -1.97 0.426 2.17 4 -0.687 -0.218 0.822 -0.4675 -0.446 -1.03 0.689 0.7806 0.360 -0.625 -0.0619 -0.560 ConclusionIn summary, the quantile_normalize() function from the TidyDensity package offers a convenient and efficient way to perform quantile normalization on numeric matrices in R. By leveraging this function, you can enhance the comparability and statistical integrity of your data across multiple samples or distributions. Incorporate quantile_normalize() into your data preprocessing workflow to unlock deeper insights and more robust analyses.

To explore more functionalities of TidyDensity and leverage its capabilities for advanced data analysis tasks, check out the package documentation and experiment with different parameters and options provided by the quantile_normalize() function.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Quantile Normalization in R with the {TidyDensity} Package

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 439–443

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #439From time to time in ExcelBI challenges, hard puzzles come to us. And this time it was one of such situations. Input we were given was matrix with values, and coordinate values as rows and columns names. Didn’t look hard unless I read the task. Then we were given pairs of coordinates to find value for them, but there was one small problem, some of coordinates were not equal with our table. Some of values pointed in “space” between each coordinate. Like on map with meridians and parallels, we have some lines but also vast area just in between them. There comes technique called bilinear interpolation. We need to check what is the difference between lines and find value that is corresponding this fraction. Lets check it out.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/439 - Bilinear Interpolation.xlsx", range = "A1:B5")lookup = read\_excel("Excel/439 - Bilinear Interpolation.xlsx", range = "E1:M7")test = read\_excel("Excel/439 - Bilinear Interpolation.xlsx", range = "C1:C5") Transformation lookup = lookup %>% column\_to\_rownames("a/b")bilinear\_interpolation = function(a, b, lookup\_table) { a\_low = floor(a * 10) / 10 a\_high = ceiling(a * 10) / 10 b\_low = floor(b) b\_high = ceiling(b) dist\_a = a\_high - a\_low dist\_b = b\_high - b\_low vlook\_1 = lookup\_table[as.character(b\_low), as.character(a\_low)] vlook\_2 = lookup\_table[as.character(b\_high), as.character(a\_low)] vlook\_3 = lookup\_table[as.character(b\_low), as.character(a\_high)] x\_1 = if\_else(dist\_b == 0, 0, (vlook\_2 - vlook\_1) * (b - b\_low) / dist\_b) x\_2 = if\_else(dist\_a == 0, 0, (vlook\_3 - vlook\_1) * (a - a\_low) / dist\_a) value = vlook\_1 + x\_1 + x\_2 return(round(value, 3))}result = input %>% mutate(`Answer Expected` = map2\_dbl(a, b, ~bilinear\_interpolation(.x, .y, lookup))) Validation identical(result$`Answer Expected`, test$`Answer Expected`)# [1] TRUE Puzzle #440Today slogan is: “Find numbers by properties”, and we are looking for numbers between 1 and 100 that can be expressed as a sum of two squared numbers that are not equal to each others. Like those snowmen above. It looks like it is really funny and suprisingly easy solution.

Loading libraries and data library(tidyverse)library(readxl)test = read\_excel("Excel/440 List of Numbers Expressed as Sum of Two Squares.xlsx", range = "A1:A30") Transformation is\_sum\_of\_squares = function(x) { squares = (1:floor(sqrt(x)))^2 any(map\_lgl(squares, ~ any(x == .x + squares[squares != .x])))}result = data.frame(numbers = 1:100 %>% as.numeric()) %>% filter(map\_lgl(numbers, is\_sum\_of\_squares)) Validation identical(result$numbers, test$`Answer Expected`)# [1] TRUE Puzzle #441Sometimes, something are too long to read, too long to write, and we are developing concepts as abbreviations, acronyms, number ranges and so on. We are skipping some characters like the frog from illustration. But our task is to do exactly opposite today. We do not need to skip any elements, but rather to step on each and every waterlily leaf on the pond. If string contains written range of numbers we need to get all numbers from this range. Find out how I did it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/441 Integer Intervals.xlsx", range = "A1:A7")test = read\_excel("Excel/441 Integer Intervals.xlsx", range = "B1:B7") Transformation result = input %>% mutate(rn = row\_number()) %>% separate\_rows(Problem, sep = ", ") %>% mutate(Problem = map(Problem, ~{ if(str\_detect(., "-")){ range = str\_split(., "-")[[1]] seq(as.numeric(range[1]), as.numeric(range[2])) } else { as.numeric(.) } })) %>% unnest(Problem) %>% summarise(`Answer Expected` = str\_c(sort(unique(Problem)), collapse = ", "), .by = rn) %>% select(-rn) Validation identical(result, test)# [1] TRUE Puzzle #442Ciphers, ciphers, ciphers… I didn’t know that people has so many ideas to hide messages from public visibility. Today we have Columnar Transposition Cipher. We use keyword as kind of template and then we can code even the longest messages. First we need to check our keyword, then rank its letters alphabetically. For example MARS would be ranked as 2134. Then we take text for encoding and wrap it after number of characters equal to length of keyword. In our example every 4 characters should be in separate rows. Then we have to read columns that appeared after folding in order like in ranked keyword, so first we need to read second column, then first, then third and fourth. But this example is easy. Check the solution.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/442 Columnar Transposition Cipher.xlsx", range = "A1:B10")test = read\_excel("Excel/442 Columnar Transposition Cipher.xlsx", range = "C1:C10") Transfromation encode = function(text, keyword){ keyword = strsplit(keyword, "")[[1]] %>% rank(ties.method = "first") l\_key = length(keyword) text = str\_extract\_all(text, "[a-z]")[[1]] text\_filled = c(text, rep("", l\_key - length(text) %% l\_key)) matrix\_text = matrix(text\_filled, ncol = l\_key, byrow = TRUE) matrix\_text = matrix\_text[, order(keyword)] %>% t() matrix\_text = matrix\_text %>% apply(1, paste, collapse = "") %>% paste(collapse = " ") return(matrix\_text)}result = input %>% mutate(`Answer Expected` = map2\_chr(`Plain Text`, Keyword, encode)) Validation identical(result$`Answer Expected`, test$`Answer Expected`)#> [1] TRUE Puzzle #443Do you remember from your childhood those word search riddles. So ExcelBI decided to throw us into those times again. We need to find some names in word search. Fortunately, they are all vertical, all left-to-right, so we have probably the easiest case. But when we were kids, when somebody find word usually crossed it with line or encircle them. What we have to do? Find words and replace all the other positions with “x”. So lets find a bird then.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/443 Birds Search.xlsx", range = "B2:K11", col\_names = FALSE)list = read\_excel("Excel/443 Birds Search.xlsx", range = "M1:M12")test = read\_excel("Excel/443 Birds Search.xlsx", range = "O2:X11", col\_names = FALSE)colnames(test) = c(1:10) Transformation find\_bird = function(grid, bird\_name) { grid = unite(grid, col = "all", everything(), sep = "") %>% mutate(nrow = row\_number()) %>% mutate(coords = str\_locate(all, bird\_name)) %>% na.omit() %>% select(-all) return(grid)}coords = map\_dfr(list$Birds, ~find\_bird(input, .x)) %>% mutate(start = coords[,1], end = coords[,2]) %>% select(-coords) %>% rowwise() %>% mutate(cols = list(seq(start, end))) %>% select(-start, -end) %>% unnest(cols) %>% mutate(check = T)input2 = input %>% mutate(nrow = row\_number()) %>% pivot\_longer(cols = -nrow, names\_to = "col", values\_to = "value") %>% mutate(col = str\_extract(col, "\\d+") %>% as.numeric()) %>% left\_join(coords, by = c("nrow" = "nrow", "col" = "cols")) %>% mutate(check = ifelse(is.na(check), F, T), value = ifelse(check, value, 'x')) %>% select(-check) %>% pivot\_wider(names\_from = col, values\_from = value) %>% select(-nrow) Validation identical(input2, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on Mirai Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Expand your data science toolkit with our “Shiny” learning path. Dive into programming your very first shiny app, ensuring its brilliance and stability before launching it into production.

R Shiny facilitates data science by integrating your code with a user interface. With this, it empowers sharing the insights of data analysis with business users, facilitates decision-making through visualization and interaction, fosters collaboration, and supports the seamless transition of your work from prototype to production. This online workshop path is the ideal opportunity to strengthen your foundations in R Shiny programming.

The “Shiny” Learning Path consists of four workshops tailored to guide you from a complete beginner to a skilled Shiny developer. Gain insights from industry professionals proficient in Shiny, as they teach you to create a polished, efficient, and professional User Interface seamlessly integrated with your R projects.

From 2:30 pm to 6 pm (CEST), a 3.5-hour hands-on experience to learn R by programming R..

Begin by mastering the fundamentals of Shiny, building your inaugural Shiny application, and grasping the concept of reactivity with the “Build your first Shiny App” workshop.
Continue your journey by uncovering the multitude of packages that broaden the Shiny landscape. Explore the art of customizing your Shiny application to align perfectly with your unique requirements and aesthetic preferences through the “Make an outstanding Shiny App” workshop.
Elevate your proficiency in R Shiny development through the “Advanced Shiny Development” workshop. Familiarize yourself with sophisticated Shiny constructs and implement industry best practices such as utilizing modules, adopting a package structure and testing, with tools like golem, renv, testthat, usethis, devtools, roxygen2.
Concluding the series, the “Bring a Shiny App to Production” workshop will familiarize you with essential DevOps best practices, emphasizing automation and collaboration through tools like Git and GitHub, enabling you to seamlessly transition your Shiny app into production.

Learning path: Shiny
Learn how to build a web app using R, exercise reactivity with R Shiny, test the app, and improve your DevOps with an agile approach.
1. Build your first Shiny App (on demand)
2. Make an outstanding Shiny App (on demand)
3. Advanced Shiny Development (14/05/2024)
4. Bring a Shiny App to Production (16/05/2024) The 1st and 2nd workshops are available on demand for those teams wishing to follow the full path including workshops for beginners, get in touch to schedule a date.

Each workshop can be taken stand-alone, but there are convenient deals for following the whole or a part of the learning path.

Early bird discount (-15 CHF).
Register more attendees and get an additional discount (-10%).
Register for two or more workshops in the path for an additional discount (-40 CHF).
Access to workshop recordings is always included.

Otherwise, contact us if you would like to have a custom workshop tailored to your needs.

To leave a comment for the author, please follow the link and comment on their blog: Mirai Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Learning Path: Shiny

View Details

[This article was first published on RStudioDataLab, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The data visualization with R, ggplot2 reigns supreme. However, a well-designed legend from a plot can undermine even the most elegant plots. Legends have the power to clarify or confuse – they’re the key that unlocks the insights hidden within your graphs. Learning how to remove, customize, and strategically use ggplot2 legends is essential for any R analyst looking to tell compelling data stories.

Have you ever struggled with a ggplot2 legend to make it visually appealing? Are you ready to take your data visualizations to the next level by exploring the full potential of legends?

Table of ContentsKey points1. Legends are Decoding Tools: Consider legends the keys that unlock the meaning within your ggplot2 visualizations. They clarify how colors, shapes, or other visual elements map to your data. 2. Removal is Sometimes Key: Knowing how to remove entire legends or target specific ones (e.g., removing the shape legend but keeping the color legend) is essential for streamlined visuals. 3. Customization Equals Clarity: Use theme() and guides() functions to reposition legends, modify titles and labels, and fine-tune their appearance. A clear legend enhances viewer understanding. 4. Design Matters: Balance visual appeal with functionality. Ensure your legends are informative, aesthetically pleasing, and support your data visualization’s overall story. 5. Explore and Experiment: Be bold and try placing legends within the plotting area, or explore packages like ‘ggthemes’ and ‘ggrare’ for even greater customization possibilities.

Introduction to Legends in ggplot2Well-designed legends can undermine even the most elegant ggplot2 creations. Legends act as the decoder ring for your plots – they translate colors, shapes, and line styles into the meaning they hold within your data. Understanding how to generate, remove, and customize ggplot2 legends is crucial for transforming your data into compelling visual stories.

Before We start, Make sure you Have the following: How to Install ggplot2 in R: A Comprehensive Guide * How do I customize my ggplot2 graphs? * Customize R Plots with scale_fill_gradient: Your Guide. What is a legend?Legends in data visualization act as a decoder ring for your plots. They map the visual elements within the graph (like colors, shapes, or line types) and the categories or values they represent within your data. A well-designed legend is essential for viewers to grasp the meaning encoded in your ggplot2* creations.

Importance of clear legends in R and other programming languages.Let me share my personal experience that highlights why legends matter. Early in my R journey, I spent hours plotting what I thought was a brilliant visualization, only to have a colleague stare blankly at it.

Without a clear legend, all my chosen colors and patterns were just visual noise! Ever since I have prioritized legends as a courtesy to my audience and have ensured that my data stories are understood. This principle holds across programming languages – the visual grammar of a plot needs translation for it to be impactful.

library(ggplot2) ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() A legend makes it easier to determine what the different colors represent. Adding a simple legend clarifies the picture:

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point(show.legend = TRUE) + labs(color = "Diamond Cut") The Anatomy of ggplot2 LegendsEver think about how ggplot2 magically generates legends for your visualizations? It’s not sorcery but rather a well-defined process based on the aesthetics you map in your plots.

How ggplot2 generates legends by defaultggplot2 takes a remarkably intuitive approach to legend creation. Let’s break it down:

Aesthetics: At the heart of it, legends are born from the aes() mappings you set up in your plot. When you define things like aes(color = clarity), ggplot2 notes that “clarity” is a factor variable that will influence color. It automatically associates each level of the clarity factor with a distinct color and prepares to include a legend.
Scales: ggplot2 uses scales to translate your data values into visual elements. If you’re using scale_color_brewer(), it knows what color palette you’ve chosen and the order of colors. All this information feeds into the legend.

The connection between aesthetics (color, shape, size) and legend itemsEach aesthetic you map in aes() has the potential to generate corresponding items in the legend. Suppose you have this code:

ggplot(diamonds, aes(x = carat, y = price, color = cut, shape = clarity)) + geom\_point()

You’d get two legends: one for “cut” (using different colors) and one for “clarity” (using different shapes).

The role of guides in controlling legend appearanceGuides are the master control panels for your legends. ggplot2 creates a guide for each mapped aesthetic.

Customization: Want to change the ” cut ” legend title to “Quality”? Use labs(color = “Quality”). Need to change the order of items in the shape legend? guides() and guide_legend() are your friends.
Understanding: If you need clarification about why your legend looks a certain way, think back to the guides generated by your plot aesthetics.

ggplot(diamonds, aes(x = carat, y = price, color = cut, shape = clarity)) + geom\_point() + labs(color = "Quality", shape = "Clarity Grade") + guides(shape = guide\_legend(order = 2), color = guide\_legend(order = 1)) Notice how I changed legend titles and even reversed the order of the legends.

Basic of ggplot Remove Legend Sometimes, the best way to make your data visualization shine is by letting it speak for itself. If a legend feels distracting or adds unnecessary clutter, ggplot2 provides easy ways to remove it. Let’s look at two key techniques.

Removing the Entire LegendThe most straightforward approach uses theme(legend.position = “none”). This command effectively tells ggplot2, “I don’t want any legends, thank you very much!”

For example, with the mtcars dataset

Suppose you’ve created a scatterplot of engine displacement (disp) vs. miles per gallon (mpg) in the mtcars dataset:

ggplot(mtcars, aes(x = disp, y = mpg, color=factor(gear))) + geom\_point()

To remove the legend that automatically pops up (it would likely be a color legend tied to the car’s transmission type), you’d add:

ggplot(mtcars, aes(x = disp, y = mpg, color=factor(gear))) + geom\_point()+ theme(legend.position = "none")

When to use this: This is handy when your plot is self-explanatory, or the legend is visually distracting.

Removing the Legend for a Specific AestheticWhat if you only want to remove a particular legend? It is where guides() and guide_legend() become your best friends. Say you want to keep the color legend (indicating transmission type) from the previous example but eliminate the shape legend that might be less relevant.

Example: Removing the shape legend while keeping the color legend

ggplot(mtcars, aes(x = disp, y = mpg, color = factor(am), shape = factor(cyl))) + geom\_point() + guides(shape = "none") # Target the shape legend for removal

It is ideal when you have multiple legends and want finer control over their display, especially in complex data plots.

Customizing Legends in ggplot2You’ve mastered removing legends; it’s time to unlock their full potential! ggplot2 provides granular control over legend appearance, letting you tweak everything from their position to the text and colors within them.

Changing Legend PositionThe legend.position argument within the theme() function is your go-to for repositioning legends. You can use intuitive values like “top”, “bottom”, “left”, or “right”. You can provide numeric coordinates (e.g., c(0.8, 0.2)) for even more precise placement.

Let’s visualize different positions using a color legend based on cut quality:

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() + theme(legend.position = "bottom") # Legend at the bottom

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() + theme(legend.position = "right") # Legend on the right

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() + theme(legend.position = c(0.9, 0.2)) # Custom coordinates Choose your position wisely to enhance readability. Avoid placing legends where they obscure crucial data.

Modifying Legend Title and TextTechniques:

Changing the Legend Title: The labs() function is your ally in naming your legend for color p or legend for size p. Use labs(color = “My Legend Title”). It works for any legend generated from an aesthetic mapping (color, shape, etc.).
Altering Legend Labels: Target the appropriate scale. If you want to change the labels within your color legend, you will use scale_color_discrete(labels = c(“Label A”, “Label B”))

For example, with the ‘diamonds’ dataset

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() + labs(color = "Diamond Quality") + # Change title scale\_color\_discrete(labels = c("Good", "Very Good", "Premium", "Ideal")) # Update labels Advanced CustomizationTechnique: The theme() function is a treasure trove for legend customization.

Background Color: Control the background using a legend.background = element_rect(fill = “gray”).
Font and Size: Experiment with legend.text = element_text(size = 10, family = “serif”).
Alignment: Fine-tune alignment with legend.justification = c(“left”, “top”).

When to use this: Employ these tweaks when you want to go beyond the basics or match your legend’s aesthetics to your overall visualization style.

People Also love to Read: GGplot group in RStudio * Rotate axis labels in ggplot2 * GGplot correlation matrix, heatmap correlation matrix in R * How I Create Violin Plots with ggplot2 in R Special Considerations*With basic legend customization, it’s time to tackle some finer points that often trip up ggplot2 users, especially when dealing with data analyses. Think of this section as the “tricks of the trade” to refine your legend mastery.

Multiple GuidesYou get multiple legends when you have multiple aesthetics mapped in your plot (e.g., color, shape, and size). It can sometimes become visually overwhelming.

Techniques:

guides(): This function lets you target specific guides for customization. Want to change just the shape legend? guides(shape = guide_legend(title = “My Shape Legend”)) is your solution.
Combining Calls: You can chain multiple guides() calls to target multiple legends in one go.

ggplot(diamonds, aes(x = carat, y = price, color = cut, shape = clarity)) + geom\_point() + guides(color = guide\_legend(title = "Cut Quality"), shape = guide\_legend(title = "Clarity Grade")) Aesthetics Beyond the DefaultWhat if you manually defined an aesthetic mapping in your geom_ layer instead of within the initial aes()? The typical customization techniques might work differently than expected.

Directly modify the scale associated with your manual aesthetic.

ggplot(diamonds, aes(x = carat, y = price)) + geom\_point(aes(color = cut)) + scale\_color\_viridis\_d(option = "inferno", name = "Diamond Quality")

In this code, I’ve used scale_color_viridis_d() to apply a color gradient, but you can use the usual discrete scales if needed.

Legend Placement within the Plotting AreaWhile tempting, placing the legend inside the plotting area can disrupt the flow of your visualization. Use it judiciously, ensuring it doesn’t obscure crucial data points.

Coordinates: Within theme(legend.position = c(x, y)), both ‘x’ and ‘y’ should be values between 0 and 1, representing the relative position within the plot. For example, c(0.2, 0.8) places the legend near the top left of the plotting area.

Consider this option when space is limited, or the legend has a direct spatial connection to your data.

Best Practices and BeyondUp until now, we’ve focused on the mechanics of removing and customizing legends. Let’s focus on design principles and how legends can enhance your data story.

Legend Design for ClarityA well-designed legend should be a quick reference for your audience, not a puzzle. Here are some guiding principles:

Conciseness is Key: Avoid excessively long labels or too many items in your legend. Consider simplifying your data or breaking your visualization into smaller, more focused plots if necessary.
Informative Labels: Ensure your legend labels accurately reflect the categories or values they represent. This is where scale_*_discrete(labels = ...) comes in handy.
Visual Consistency: Your legends’ font, colors, and overall style should align with the rest of your visualization.

Balancing Visual Appeal with FunctionalityWhile clarity is crucial, don’t neglect aesthetics entirely! Use theme() elements to create legends that blend seamlessly with your graphic. Consider:

Color Choices: For color legends, ensure good contrast and consider colorblind-friendly palettes where appropriate.
White Space: Give your legends enough breathing room to prevent a cluttered feel.

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() + scale\_color\_viridis\_d(option = "cividis", name = "Cut Quality") + theme(legend.title = element\_text(size = 11), # Adjust title size legend.text = element\_text(size = 9), # Adjust label size legend.background = element\_rect(fill = "lightgray")) # Softer background Data Storytelling with LegendsUnlocking Insights: Legends play a supporting role in the narrative you build with your data. Use them strategically:

Highlighting Patterns: Tailor your legend title and labels to emphasize key trends or comparisons your plot reveals, enhancing the data experience.
Spatial Legends: If you’ve placed your legend within the plotting area, use position strategically to guide the viewer’s eye toward important data relationships.

ggplot2 Extensions for LegendsWhile ggplot2 itself offers extensive legend control, there are times when you might crave even more specialized functionality. Here’s a brief mention of packages that open up further customization:

‘ggthemes’: Provides pre-made themes that often include visually appealing legend styles.
‘ggrare’: Offers functions for arranging multiple ggplot2 plots and gives more flexibility over shared legends.
‘cowplot‘: Another package useful for plot arrangement and legend management for more complex visual layouts.

Related PostsErrorsLegend Persists Despite “theme(legend.position = ‘none’)”There might be multiple aesthetics (color, shape, etc.) Generating legends unintentionally can clutter your data visualizations.

ggplot(mtcars, aes(x = disp, y = mpg)) + geom\_point(color = "blue") + # Fix the color manually theme(legend.position = "none") Trying to Remove a Non-Existing LegendDouble-check if a legend actually appears by default in your ggplot2 visualization.

If there’s no legend, there’s no need for removal code.

Incorrect Targeting with ‘guides()’Mismatching the guide type to the aesthetic you want to remove. (e.g., trying to use guides(color = "none") to remove a shape legend).

ggplot(mtcars, aes(x = disp, y = mpg, color = cyl, shape = factor(am))) + geom\_point() + guides(shape = "none") # Correctly targets the shape legend Confusing Scale Modification with ‘guides()’Trying to remove legends by altering color scales instead of using guides().

ggplot(mtcars, aes(x = disp, y = mpg, color = cyl, shape = factor(am))) + geom\_point() + guides(color = "none") # Correctly targets the color legend Conflicts within FacetsUsing facet\_wrap() or facet\_grid() can lead to unexpected legend behavior if you haven’t adjusted how legends are combined across facets.

ggplot(diamonds, aes(x = carat, y = price, color = cut)) + geom\_point() + facet\_wrap(~ clarity) + theme(legend.position = "bottom") + guides(color = guide\_legend(nrow = 1)) # Combine legends into a single row Issues with Manually Defined ScalesLegends generated from manual scales (e.g., within your geom\_ call) don’t always respond to the usual theme() options.

ggplot(mtcars, aes(x = disp, y = mpg)) + geom\_point(aes(color = factor(cyl))) + scale\_color\_manual(values = c("blue", "red", "green"), guide = "none") Overlapping Legends and Plotting ElementsIncorrect legend placement (especially using coordinates) might lead to legends obscuring data points.
Carefully adjust coordinates in theme(legend.position = c(x, y)). Experiment with smaller increments for fine-tuning the representation of toothgrowth$dose.

Unintended Legend Generation in GeomsSome geoms (geom\_text(), geom\_label(), etc.) might add legends if a color or fill aesthetic is mapped within the geom.
Set the aesthetic explicitly to NA within the geom to suppress the legend.

ggplot(mtcars, aes(x = disp, y = mpg, color = cyl)) + geom\_point() + geom\_text(aes(label = rownames(mtcars)), color = NA) External Package InterferenceOther plotting libraries or packages occasionally interfere with ggplot2’s legend settings. Isolate the plotting code to a minimal example. If the error persists, investigate potential conflicts with recently installed or updated packages.

ConclusionHarnessing the power of legends is essential for any R data analyst. Your visualizations can tell compelling data stories by learning to remove legends strategically, reposition them for clarity, and modify their appearance for function and style. Remember, a well-designed ggplot2 legend is a visual key, effortlessly guiding your audience through the insights your plots reveal. Don’t be afraid to experiment – try repositioning legends within the plotting area for a unique touch, or explore extensions like ‘ggrare’ or ‘cowplot’ for even more advanced control.

Frequently Asked Questions (FAQs)How to remove a legend in ggplot r? Complete Removal: Use theme(legend.position = “none”) for a global solution. * Selective Removal:* Target specific aesthetics with guides(). For instance, guides(fill = “none”) would remove a fill legend while keeping others.

How do I remove the legend title in R?Employ labs(title = “”) and target the correct aesthetic (e.g., labs(color = “”, shape = “”, size= “”) to modify multiple legend titles at once).

How do you remove something from the legend?The guides() function is your best friend. To remove the shape legend, use guides(shape = “none”). Remember that it’s essential to correctly identify the aesthetic driving the legend.

How do you change the legend name in ggplot?The labs() function lets you rename legends easily. For example, labs(color = “Engine Type”) would update the title of a color legend.

How to edit a legend in ggplot?ggplot2 provides granular control through theme() and various scale functions:

Position: theme(legend.position = “bottom”) for bottom placement, or use theme(legend.position = c(0.8, 0.2) for precise coordinates.
Colors and Appearance: Use scale\_color\_brewer() (for palettes) or scale\_color\_manual() (for specific color choices). Customize background and borders with theme(legend.background = element\_rect(…), legend.key = element\_rect(…).
Text and Labels: Change labels with scale\_…\_discrete(labels = …). Adjust the font and size with theme(legend.text = element\_text(family = “serif”, size = 10)).

How do I add a legend to a ggplot? Check your mappings: Legends naturally tie into your aes() definitions. Review if you’ve mapped appropriate variables to color, fill, shape, etc. * Manual Legends*: You may need to create legends manually for highly customized visuals. This involves careful use of layout functions and is a more advanced technique.

What is the legend function in R?ggplot2’s Power: In ggplot2, legends are a byproduct of interconnected concepts rather than a single function, essential for effectively communicating the dose in toothgrowth$dose studies. Understand how aesthetics, scales, and theme customization contribute to the final legend.

How do you change labels in ggplot?Target the relevant scale of the aesthetic you want to modify:

Discrete scales: scale\_color\_discrete(labels = c(…)), scale\_fill\_manual(labels = c(…)), etc.
Continuous scales: scale\_color\_gradient(limits = c(min, max), labels = c(…))

How can the size of the legend in ggplot2 be changed? Legend theme* (legend.key.size = unit(0.5, “cm”))to adjust the size of color squares or shapes. * **Text**:theme(legend.text = element_text(size = 11))to control the text size within the legend. * **Overall Legend Box**: Experiment withtheme(legend.margin = …)` to influence spacing around the legend.

How do I remove a layer from Legend?If a layer adds an unexpected legend, consider if your plot structure might unintentionally create multiple geoms. Review your ggplot code and mappings.

How do you remove the legend for the combo chart?No Special Tricks: The techniques apply regardless of chart complexity! Identify the aesthetics generating the extra legend and use either complete removal or guides() for specific targeting.

What is a legend in a chart?Visual Translator: Think of a legend as the decoder ring for your chart. It reveals the meaning behind colors, shapes, line types, or other visual elements and how they correspond to groups or trends within your data.

How can I remove a legend in ggplot2?To remove a legend in ggplot2, you can set the show.legend argument to FALSE in the specific geom function or globally in the theme() function by using theme(legend.position = “none”). This way, no legend will be included in your plot.

What is the purpose of adjusting the order of legend items in ggplot2?Adjusting the order of legend items can help make your ggplot2 plots more readable and interpretable, especially when the default order does not match the desired or logical sequence of categories. You can change the order of legend items by manipulating factor levels of the mapped variable or by using the guides() function with the guide\_legend() attribute and setting its order argument.

Can I change the legend labels in ggplot2?Yes, you can change the legend labels in ggplot2 by using the labs() function and specifically naming each aesthetic (like color, shape, or size) for which you want to change the label. This allows you to make the legend more informative and better aligned with the data representation in your plot.

How do I modify the background color of a legend in a ggplot2 plot?To modify the background color of a legend in ggplot2, use the theme() function along with element\_rect() to change the legend background properties. Inside theme(), set legend.background = element\_rect(fill = “desired\_color”, color = “border\_color”) to adjust both the background fill and the border color of the legend box.

What is the role of font size and style in the legibility of a ggplot2 legend?Font size and style significantly affect the legibility and the overall appearance of legends in ggplot2 plots. Adjusting the font-size (size), font-family (family), and font-style (face) can be done through the theme() function by setting legend.text = element\_text() with appropriate arguments. This ensures that the legend is both readable and aesthetically pleasing.

Is it possible to independently manage the legend for a particular aesthetic like size or shape in ggplot2?Yes, it’s possible to manage the legend for a particular aesthetic such as size, shape, or color independently by using the guides() function along with guide\_legend() or guide\_colorbar() for continuous scales. This allows fine-grained control over each legend’s appearance, including its title, labels, and other attributes, without affecting other legends.

How do I adjust the coordinates of the legend box in ggplot2?The coordinates of the legend box in a ggplot2 plot can be adjusted using the theme() function with the legend.position argument. To move the legend to a desired position on the plot, you can specify exact coordinates in relative terms (a vector of two values between 0 and 1). Alternatively, preset positions such as “top”, “bottom”, “left”, “right”, and “none” are also available.

Can I use ggplot2 to show a legend for color and simultaneously for shape?A: Absolutely, ggplot2 allows you to display legends for multiple aesthetics simultaneously, such as for color and shape. When you map these aesthetics to different variables in your geom\_*() functions, ggplot2 automatically creates a legend for each. Adjusting their appearance and position for clarity can be achieved using guides() and theme() functions.

Transform your raw data into actionable insights. Let my expertise in R and advanced data analysis techniques unlock the power of your information. Get a personalized consultation and see how I can streamline your projects, saving you time and driving better decision-making. Contact me today at info@data03.online or visit to schedule your discovery call.

Join Our Community Book a free call.To leave a comment for the author, please follow the link and comment on their blog: RStudioDataLab.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Guide to Remove Legends in ggplot2 in R Programming

View Details

[This article was first published on R Code – Geekcologist, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Considered one of the “few general rules” in ecology, the species-area relationship (SAR) – the ubiquitous pattern indicating that larger islands or habitat patches harbor more species than smaller ones – has played a pivotal role in shaping ecological theories (Lomolino 2000, 2001, Tjørve 2003, 2009, Dias et al. 2020). These theories aim not only to describe patterns of species diversity in patchy environments but also to comprehend the underlying mechanisms driving ecological community dynamics in space. Given its paramount importance for biodiversity, SAR has significantly influenced the development of biodiversity conservation strategies, particularly in the design of protected areas and the selection of habitat patches in fragmented landscapes.

Although a large plethora of models have been used to describe SAR (Tjørve2003, 2009), the relationship between area and species richness has been traditionally modeled as a power law, as first developed by Arrhenius (1921):

S=cAz

where S represents the number of species in a habitat patch (or island), A denotes the area of the patch, and c and z are constants that govern the relationship.

As the use of simple linear models make it easy to estimate the model’s parameters, ecologists have traditionally used a log-linearizable power function, log-transforming both S and A. Under this approach, z and c (more specifically log[c]) are then estimated using the subsequent regression model:

log (S) = log(c) + z log(A)

In addition to simplifying parameter estimation, this log transformation also brings some biological meaning for model parameters, as in the logarithm space, log(c) now represents the expected number of species when the area is one unit, and the parameter z reflects the proportional change in species richness per unit change in log(area).

However, as has been pointed out by the now classic works of Gould (1979) and Lomolino (1989, 2000, 2001), the interpretation of parameters c and z in the context of SAR is often misunderstood, despite being crucial quantities in ecology. Solely relying on z values can lead to misinterpretations regarding the rate at which species richness increases with area, if c is not constant when comparing different archipelagos, formed by either true island or habitat patches. Therefore, it’s essential to consider both c and z values together, as they provide a more comprehensive understanding of the relationship.

Given that the species-area relationship (SAR) constitutes a central topic in undergraduate and graduate programs in ecology, biogeography, and conservation biology, being able to visualize the shape of this relationship and the effect of these two critical parameters on SAR models is crucial for student comprehension. With that in mind, I developed a simple R Shiny app to aid students in visualizing SAR in both arithmetic and logarithmic scales. This tool could prove useful for courses in biogeography, ecology, and related fields.

The R shiny app should look like this:

Here is the code!

```

rm(list=ls())###Load packagesrequire(shiny)require(ggplot2)### Define user interfaceui = fluidPage( titlePanel("Specie-Area Relationship - Power Model"), sidebarLayout( sidebarPanel( sliderInput("c", "Value of c:", min = 0.1, max = 10, value = 1), #bar to define values for c sliderInput("z", "Value of z:", min = 0.1, max = 2, value = 0.5),#bar to define values for z numericInput("area", "Area:", value = 1000, min = 1),#define range for area numericInput("num_species", "Number of Species:", value = 50, min = 1, max = 1000),#define rangefor species richness checkboxInput("loglog", "Log-Log Axes"), width = 3 ), mainPanel( plotOutput("species_area_plot") ) ))### Define the serverserver = function(input, output) { output$species_area_plot <- renderPlot({ ### Generate data area <- seq(1, input$area, length.out = 1000) species <- input$c * area^input$z ### Create plot p = ggplot() + geom_line(aes(x = area, y = species), color = "blue") + geom_hline(yintercept = input$num_species, linetype = "dashed", color = "red") + labs(x = "Area", y = "Species") + theme_minimal() # Log-Log Axes if (input$loglog) { p <- p + scale_x_log10() + scale_y_log10() } print(p) })}### Execute appshinyApp(ui = ui, server = server)

``` References

Arrhenius, O. (1921). Species and area. Journal of Ecology, 9(1), 95-99.

Dias, R. A., Bastazini, V. A. G., Knopp, B. D. C., Bonow, F. C., Gonçalves, M. S. S., & Gianuca, A. T. (2020). Species richness and patterns of overdispersion, clustering and randomness shape phylogenetic and functional diversity–area relationships in habitat islands. Journal of Biogeography, 47(8), 1638-1648.

Gould, S. J. (1979). An allometric interpretation of species-area curves: the meaning of the coefficient. The American Naturalist, 114(3), 335-343.

Lomolino, M. V. (2001). The species-area relationship: new challenges for an old pattern. Progress in physical geography, 25(1), 1-21.

Lomolino, M. V. (1989). Interpretations and comparisons of constants in the species-area relationship: an additional caution. The American Naturalist, 133(2), 277-280.

Lomolino, M. V. (2000). Ecology’s most general, yet protean pattern: The species–area relationship. Journal of Biogeography, 27, 17–26.

Tjørve E. (2003) Shapes and functions of species-area curves: a review of possible models. Journal of Biogeography, 30, 827– 835.

Tjørve, E. (2009). Shapes and functions of species–area curves (II): A review of new models and parameterizations. Journal of Biogeography, 36, 1435–1445.

To leave a comment for the author, please follow the link and comment on their blog: R Code – Geekcologist.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Visualizing Power Models in Species-Area Relationships Using R Shiny: An Interactive Educational Tool

View Details

[This article was first published on Rtask, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You can read the original post in its original format on Rtask website by ThinkR here: Pimping your shiny app with a JavaScript library : an example using sweetalert2

You think that some of the components of {shiny} are not very functional or downright austere? Are you looking to implement some feature in your app but it is not available in the {shiny} toolbox? Take a look at JavaScript!

JavaScript is a very popular programming language that is often used to add features to web pages. With HTML and CSS, JavaScript is an essential language for web developers. The size of its user community means that if you are looking to implement a particular feature, there is a good chance that someone has already had the same need as you and has shared their code!

An other positive point (and essential for us in this case) : it is possible to integrate JavaScript libraries into a {shiny} application to add features that are not available by default. In addition to that, {golem} will help us to set everything up.

No more excuses to back down, let’s go !

TL;DR* Going further in {shiny} by integrating a JavaScript library is possible! + We take the example of sweetalert2, which allows to display alerts that are more visually appealing than the basic ones + {golem} has several functions to make it easier for us to integrate JavaScript libraries into a{shiny} app + This example is rather simple. The integration of libraries is sometimes harder because the documentation might be scarse or the library might be more complex to use

Import sweetalert2 into a {shiny} app created with {golem}sweetalert2* sweetalert2 is a JavaScript library that allows you to display alerts that are more visually appealing than the basic ones * It is very well documented * It is very popular (more than 16000 “stars” on GitHub at the time of publication of this article)

Let’s take a look at the possibilities offered by sweetalert2: https://sweetalert2.github.io/

If you click on “Show normal alert”, you will see a classic alert while clicking on “Show success message”, you will see a sweetalert2 alert.

The first one has a rather austere design while the second one is more modern and more pleasant to the eye, it will probably offer a better user experience.

Feel free to play with the different types of alerts offered by sweetalert2 to get an idea of what is possible with this library by visiting the examples section.

Add the necessary dependencies to the {shiny} appThe following sections assume that you have already created a {shiny} app with {golem}.

If this is not the case and you want to know more about {golem}, I invite you to consult the official documentation.

To add sweetalert2 to your {shiny} app, you will need to find a way to incorporate the files needed for its operation into your application.

Two solutions are available to you:

Use a “CDN” (Content Delivery Network) to load the files from a third-party server. The CDN will be the equivalent of a CRAN for JavaScript libraries. Concretely, we will ask our application to point to the sources of sweetalert2, hosted on a remote server.
Download the files needed for its operation and integrate them directly into your application. If your application is intended to be used on a machine that is not connected to the Internet, you will inevitably have to go through this step.

Don’t panic! We will see both methods

Where to find the elements I need?The sweetalert2 documentation is very well done. You will find all the information you need to integrate the library into your application from the Download section.

However, you will need to learn how to identify the elements you need to integrate sweetalert2 into your application.

Looking for the CDN

In the “Download & Install” section, you will find a link to the sweetalert2 CDN. This is the link that we will have to add to our application in order to use the library.

When you click on the link, you will arrive on a page that looks like this:

What we are interested in here is the link in the <script> tag and the link in the <link> tag. The link to the file with the .js extension corresponds to the sweetalert2 JavaScript file. The link to the file with the .css extension corresponds to the sweetalert2 styles file.

Copy them and let’s go back to our app!

Method 1: Add dependencies to sweetalert2 via the CDNLet’s open the R/app_ui.R file of our application and add the two links we copied earlier to the body of the golem_add_external_resources() function.

golem\_add\_external\_resources <- function() { add\_resource\_path( "www", app\_sys("app/www") ) tags$head( favicon(), bundle\_resources( path = app\_sys("app/www"), app\_title = "golemsweetalert" ), # sweetalert2 tags$script(src = "https://cdn.jsdelivr.net/npm/sweetalert2@11.10.7/dist/sweetalert2.all.min.js"), tags$link(href = "https://cdn.jsdelivr.net/npm/sweetalert2@11.10.7/dist/sweetalert2.min.css", rel="stylesheet") )} We find here a call to tags$script and a call to tags$link corresponding respectively to the <script> and <link> tags seen on the links provided by the CDN.

We copy-paste the elements being careful about the parameter names src, href and rel and remembering to separate them with commas. We are indeed moving from HTML code (where elements are separated by spaces) to R code.

Method 2: Add dependencies to sweetalert2 locallyThe links identified earlier will also be useful to download the files needed to use sweetalert2.

The link to the JavaScript file will be passed to the golem::use_external_js_file() function while the link to the CSS file will be passed to the golem::use_external_css_file() function.

To keep a record of this, we will save the following commands in the dev/02_dev.R file in the “External resources” section.

golem::use\_external\_js\_file("https://cdn.jsdelivr.net/npm/sweetalert2@11.10.7/dist/sweetalert2.all.min.js")golem::use\_external\_css\_file("https://cdn.jsdelivr.net/npm/sweetalert2@11.10.7/dist/sweetalert2.min.css") Both files are now present in the inst/app/www folder of our application project.

Everything is ok, we can continue our journey .

How to know if sweetalert2 is correctly imported into my {shiny} app?To check that sweetalert2 is correctly imported into our application, we open our browser and then the development console.

Before that, let’s launch our app with golem::run_dev()!

NB: The following screenshots were taken with the Google Chrome browser.

On the window of our application, we right-click and select “Inspect”. In the new window that opens, we choose the “Console” tab and then type the command Swal.fire("Hello sweetalert2!"). This console allows us to execute JavaScript code, which will be interpreted by our browser.

If everything went well, we should see a sweetalert2 alert appear! Otherwise, we might have an error message in red (which we will have to learn to decipher, as with an R console :-)).

It works!

How did I know I had to type Swal.fire(" .... ")?Thanks to the documentation! By going to the sweetalert2 page, we find many examples of using the library. In this case, to display an alert, you have to use the Swal.fire() function. As for learning a new R package, we see that documentation (when it exists …) is of paramount importance for the handling of JavaScript libraries.

Create a sweetalert2 alert in our {shiny} appNow that we have imported sweetalert2 into our application and checked that the import went well, we are going to create a function that will allow us to call sweetalert2 from our application.

We are going to try to incorporate the “A title with a text under” alert into our application.

Three elements can be set here:
– The title of the alert
– The text of the alert
– The type of alert (success, error, warning, info, question) and its associated icon

We can easily imagine how to modify these elements directly in the JavaScript code but we don’t know yet how to create this alert via R code. Let’s go!

Create a JavaScript file to call sweetalert2In order to call sweetalert2 from our application, we are going to create a JavaScript file in which we will write a function that will allow us to create an alert.

Let’s create a inst/app/www/sweet_alert.js file in which we will paste the code to create the alert selected earlier. We could have created this file manually, but we will take advantage of the features offered by {golem} to do so.

Let’s go to the dev/02_dev.R file! We add the following line in the “External Resources” section:

golem::add\_js\_handler("sweet\_alert") The file name doesn’t matter, but it is important to respect the .js extension so that the file is correctly interpreted as JavaScript code. We could have also created subfolders if we had had many imports and files to manage. The only prerequisite is that everything is located in the inst/app/www directory.

We obtain a slightly strange skeleton, which will in fact be the skeleton of a JavaScript function, usable in our {shiny} application:

$( document ).ready(function() { Shiny.addCustomMessageHandler('fun', function(arg) { })}); We are going to substitute the term 'fun' with the name of the function we want to call in our {shiny} application and add the code to create the sweetalert2 alert.

We therefore obtain the following code:

$( document ).ready(function() { Shiny.addCustomMessageHandler('alert\_title\_and\_text\_under', function(arg) { Swal.fire({ title: "The Internet?", text: "That thing is still around?", icon: "question" }); })}); Our parameters “title”, “text” and “icon” are fixed, we need to find a way to make them vary according to the choices of the user. We can notice the existence of a “arg” parameter in the Shiny.addCustomMessageHandler() function. It is this parameter that will allow us to transmit information to our JavaScript function.

Let’s modify our code:

$( document ).ready(function() { Shiny.addCustomMessageHandler('alert\_title\_and\_text\_under', function(arg) { Swal.fire({ title: arg.title, text: arg.text, icon: arg.icon }); })}); The notation to use here will be arg.parameter_name to access the values transmitted by our {shiny} application. The notation with the “.” is a JavaScript convention for accessing object
properties. To make the parallel with R, it’s a bit like if we were doing arg$parameter_name.

Our JavaScript code is ready! Let’s move back to the R side!

What if we tested all this in our {shiny} app (FINALLY!)?We are going to add a button in the R/app_ui.R file:

app\_ui <- function(request) { tagList( # Leave this function for adding external resources golem\_add\_external\_resources(), # Your application UI logic fluidPage( h1("golemsweetalert"), actionButton(inputId = "show\_alert", label = "Alert demo") ) )} On the server side, we add an observeEvent() in the R/app_server.R file, which will call our JavaScript function to generate an alert when the user clicks on the “Alert demo” button.

Once more, {golem} will make our life easier! We will use the golem::invoke_js() function to call our JavaScript function.

Two parameters are passed to golem::invoke_js():

the first parameter corresponds to the name of the JavaScript function to call
the second parameter corresponds to a list of parameters, the JavaScript equivalent of our arg object which will be used to transmit the information necessary to create the sweetalert2 alert. The names used in the list here must correspond to the names used in the JavaScript function for the arg parameter (“title”, “text” and “icon”).

app\_server <- function(input, output, session) { # Your application server logic observeEvent( input$show\_alert,{ golem::invoke\_js( "alert\_title\_and\_text\_under", list( title = "Title", text = "Text", icon = "success" )) } )} We run a call to golem::run_dev() to see our application in action!

Congratulations!

To make everything more elegant, we can create an R function that will call golem::invoke_js().

Create an R function to call sweetalert2We are going to go through {golem} to create our function. To do this, we will add the following line in the dev/02_dev.R file of our application:

golem::add\_fct("create\_alert\_title\_and\_text\_under") We obtain a R/fct_create_alert_title_and_text_under.R file in which we will be able to write our function, which will call the JavaScript code created in the previous step.

```

' create_alert_title_and_text_under#'#' @description Creates an alert with a title, a text and an icon#' @param title alert title#' @param text alert text#' @param icon alert icon#' @return side effect : creates an alert#'#' @noRdcreate_alert_title_and_text_under <- function( title = "Title", text = "Text", icon = "success" ) { golem::invoke_js( "alert_title_and_text_under", list( title = title, text = text, icon = icon ) )}

`` Let’s modify both theR/app_ui.RandR/app_server.R` files to be able to define the parameters of our alert through choices made by the user.

app\_ui <- function(request) { tagList( # Leave this function for adding external resources golem\_add\_external\_resources(), # Your application UI logic fluidPage( h1("golemsweetalert"), textInput(inputId = "title", label = "title"), textInput(inputId = "text", label = "text"), radioButtons(inputId = "icon", label = "icon", choices = c("warning", "error", "success", "info", "question")), actionButton(inputId = "show\_alert", label = "Alert demo") ) )} To define the list of possible “choices” for the radioButtons, we took the possible values for the icon parameter of sweetalert2, as indicated in the official documentation: https://sweetalert2.github.io/#icon.

app\_server <- function(input, output, session) { # Your application server logic observeEvent( input$show\_alert,{ create\_alert\_title\_and\_text\_under( title = input$title, text = input$text, icon = input$icon ) } )} Let’s finally call golem::run_dev()!

Bravo !

And a more complex example?Following the previous steps, it is relatively easy to add additional elements to an alert, such as an image or confirmation / cancellation buttons. A slightly deeper dive into the sweetalert2 documentation will help you understand how to add these elements.

Let’s see what is possible to achieve:

You are curious to know more? Go to this project source code.

Conclusion* Integrating JavaScript libraries into a {shiny} application is possible! * {golem} makes it easier for us * It is quite easy when the library is well documented * It is important to read the documentation of the library you want to integrate (but this is also true in R!) * The browser inspector is a very useful tool to check that everything is going well * To integrate more complex libraries, minimal JavaScript skills will probably be required

If you want more examples of the use of sweetalert2, but also other JavaScript libraries (Grid.js and Chart.js), you can consult the mycomicslibrary application and take a look at its source code.

Thanks for reading this tutorial and have fun in the wonderful world of JavaScript!

This post is better presented on its original ThinkR website here: Pimping your shiny app with a JavaScript library : an example using sweetalert2

To leave a comment for the author, please follow the link and comment on their blog: Rtask.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Pimping your shiny app with a JavaScript library : an example using sweetalert2

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionI’m thrilled to announce the release of TidyDensity version 1.4.0, packed with exciting features and improvements to elevate your data analysis experience in R. Let’s dive into what this latest update has to offer.

New FeaturesQuantile NormalizationSay goodbye to skewed data distributions! With the new quantile_normalization() function, you can now easily normalize your data using quantiles, ensuring more accurate and reliable analysis results.

Duplicate Row DetectionData integrity matters, which is why we’ve introduced the check_duplicate_rows() function. Quickly identify and eliminate duplicate rows in your data frame, streamlining your workflow and improving data quality.

Chi-Square Distribution Parameter EstimationEstimating parameters for the chi-square distribution is now a breeze with the util_chisquare_param_estimate() function. Empower your statistical analysis with precise parameter estimation capabilities.

Markov Chain Monte Carlo (MCMC) SamplingUnlock the power of Markov Chain Monte Carlo sampling with the new tidy_mcmc_sampling() function. Seamlessly sample from distributions using MCMC, and visualize the results with diagnostic plots for deeper insights into your data.

AIC Calculation for DistributionsMaking informed model selection decisions just got easier! TidyDensity now includes util_dist_aic() functions to calculate the Akaike Information Criterion (AIC) for various distributions, providing valuable metrics for model evaluation.

Minor Fixes and ImprovementsIn addition to these exciting new features, we’ve also made several minor fixes and enhancements to further refine your user experience:

Enhanced tidy_multi_single_dist() function to respect the .return_tibble parameter.
Improved documentation, including updates related to MCMC usage.
Updated tidy_distribution_comparison() to incorporate AIC calculations from dedicated util_dist_aic() functions.

Upgrade Now!Ready to supercharge your data analysis workflow? Upgrade to TidyDensity version 1.4.0 today and take advantage of these powerful new features and enhancements. Whether you’re a seasoned data analyst or just getting started with R, TidyDensity is your go-to toolkit for streamlined and robust data analysis.

As always, we welcome your feedback and suggestions for future improvements. Stay tuned for more updates as we continue to evolve and enhance the TidyDensity package to meet your data analysis needs.

Happy analyzing!

Steve, Manager of Applications at Stony Brook Medicine

Creator and Maintainer of TidyDensity

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing TidyDensity Version 1.4.0: Enhancing Data Analysis in R

View Details

[This article was first published on R on Nicola Rennie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Waffle charts are a common way to visualise counts or percentages of categorical data. There are already several excellent ways of creating waffle charts in R – including approaches using {ggplot2} or {waffle}. This blog post uses neither of those. Instead, it describes a somewhat back-to-basics approach of simply drawing lots of squares. This makes it a little bit easier to then create a version of a waffle chart that has a hand-drawn, sketchy effect.

Sketchy plots in RThe{ggrough} package converts plots made with {ggplot2} to rough (or sketchy) looking charts using the rough.js javascript library. Unfortunately, the package is in a dormant state and it doesn’t currently work with more recent versions of {ggplot2}. There are several solutions to this problem:

Install an older version of {ggplot2}, and make a waffle chart with geom_tile() directly in {ggplot2} or using geom_waffle() from {waffle}. Then we could use {ggrough}.
Try to fix the issues in {ggrough} and suggest the updates in a PR.
Or we could create our own hacky solution by using several other R packages.

Specifically, a hacky solution using a combination of {roughsf} and {sf}. The{roughsf} package also wraps rough.js but takes sf (simple features) objects rather than {ggplot2} objects as inputs. It’s primarily used for creating sketchy looking maps. But {sf} doesn’t just make maps – it can make essentially any shape using points, lines, and polygons. And if you look at charts in an abstract manner, they’re also really just points, lines, and polygons.

Note: {ggrough} and {roughsf} are not the only R packages that can be used to create sketchy looking charts in R. The{roughnet} package also wraps rough.js and works specifically for visualising network data. Ther-sketchy project also suggests a method for drawing sketchy looking lines.

Let’s start by getting some data to make a waffle chart of!

Data processingSince we’re trying to make something sketchy and artistic looking, it might be nice to use some artistic data. Luckily, the#TidyTuesday Project has shared some data on the colours used inBob Ross paintings. Let’s load it in from the #TidyTuesday GitHub repository:

| 1 | bob\_ross <- readr::read\_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-21/bob\_ross.csv") |

Let’s also load the packages we’ll need for processing and plotting the data:

| 123 | library(tidyverse)library(sf)library(roughsf) |

The bob_ross data contains information on each episode such as the painting title, a list of colours used, and a link to the YouTube video of the episode. It also has some binary columns relating to colour names, and whether they were used in each painting. We’ll start by creating a lookup table of colour names and hex codes (since we’ll later colour the waffle chart using the hex codes).

If you’re mostly interested in the plotting with {sf} and {roughsf} aspects, feel free to skip ahead tothe next section.

The hex colours and colours are unfortunately not stored in the most user-friendly format. For example, the first entry of the color_hex column, looks like this character string:

| 1 | "['#4E1500', '#DB0000', '#FFEC00', '#102E3C', '#021E44', '#0A3410', '#FFFFFF', '#221B15']" |

To create a lookup colour table, we need only the colors and colors_hex columns. We can use separate_longer_delim() from {tidyr} to split these character strings of lists into multiple rows, based on the , separator. We then need to tidy up the output by removing the square brackets, the extra quotation marks, the slightly odd \\r and \\n characters, as well as any extra white space at the start or end of the strings. We can use the str_remove_all() and str_trim() functions from {stringr} to do this. Many of the colours are used across multiple paintings but we only need to keep one row for each colour in the lookup table, i.e. the distinct() rows.

| 123456 | colour\_lookup <- bob\_ross |> select(colors, color\_hex) |> separate\_longer\_delim(c(colors, color\_hex), delim = ",") |> mutate(across(everything(), ~str\_remove\_all(., "\\[|\\]|'|\\\\r|\\\\n"))) |> mutate(across(everything(), ~str\_trim(.))) |> distinct() |

We have 18 unique colour names. Now we want to calculate the percentage of colour uses that each colour corresponds to. We start by selecting only the binary columns from bob_ross that denote if a colour was used in an episode. Then we simply add up the number of 1’s in each column. This gives us the number of episodes that each colour was used in. To make processing the data easier, we convert it into long format - we now have two columns relating to the colour name, and the number of episodes each was used in. We also replace the "_" in the colour names with a " " so that it matches the colour names in the lookup table we just created e.g. "Cadmium_Yellow" becomes "Cadmium Yellow".

| 123456789 | count\_data <- bob\_ross |> select(c(Black\_Gesso:Alizarin\_Crimson)) |> summarise(across(Black\_Gesso:Alizarin\_Crimson, ~ sum(.x, na.rm = TRUE))) |> pivot\_longer( cols = everything(), names\_to = "colors", values\_to = "n" ) |> mutate(colors = str\_replace\_all(colors, "\_", " ")) |

We join this count data to the lookup table using a left_join() based on the colors column, and calculate the percentage of the total for each colour. To make matters a little bit more complicated, the hex colours don’t relate to unique colour names. For example, "Titanium White" and "Liquid Clear" are both represented by "#FFFFFF". Since we’ll use the hex colours for plotting, these two different colour names will be represented the same way in the plot. So we add together the percentages for these colour names i.e. group by the hex colour.

One issue with this approach is that we’ve rounded the percentages. For a waffle chart, we have 100 squares and each square will take a single colour representing 1% - there are no half squares. We need a whole number in the percentage for each category (hex colour). Unfortunately, calculating exact percentages and then rounding them doesn’t always add up to 100. Depending on the rounding, you might end up with 99 or 101 squares, for example. Let’s check if this adds up to 100 by chance:

| 1 | sum(colour\_count\_data$perc) == 100 |

which returns:

| 1 | TRUE |

This is extremely lucky!

If you’re creating your own waffle chart and these numbers don’t add up to 100, you’ll need to make a choice about how you’ll round the values.

Our data currently has one row per colour but for plotting later, we’ll need one row per square in the waffle chart. We can use the uncount() function from {dplyr} to create replicates of each row, according to the number in the perc column.

When we’re using the {roughsf} package, the aesthetics will also need to be specified explicitly as columns. This means we also need to rename the color_hex column as fill.

Let’s see what plot_data looks like by inspecting the first few rows:

| 1 | head(plot\_data) |

which returns:

| 123456789 | ```

A tibble: 6 × 1 fill 1 #0000002 #0000003 #0000004 #0000005 #0000006 #000000

``` |

Making {sf} objectsSince we’re going to be plotting using {roughsf}, we need to make some objects to plot using {sf}. A waffle chart that displays percentages typically shows 100 squares arranged in a 10x10 grid. So we need to make 100 squares using {sf}.

Let’s just start with 1 square at a time. Since we’re going to be doing something 100 times, it’s best to make a function. The function below, make_square(), takes three arguments: the x and y coordinates of the bottom left corner of the square, and the width of the square. It creates a matrix where the first column contains the x-coordinates of the corners of the square, and the second column the y-coordinates. Although a square only has four corners, here we have five pairs of coordinates with the last row equal to the first. This ensures the polygon is closed (and can have a fill colour). The st_polygon() function from {sf} then converts this matrix to an {sf} object.

| 1 2 3 4 5 6 7 8 910 | make\_square <- function(x0, y0, width = 1) { sf::st\_polygon( list( cbind( c(x0, x0 + width, x0 + width, x0, x0), c(y0, y0, y0 + width, y0 + width, y0) ) ) )} |

Let’s check it works by creating a square that starts at x0 = 2 and y0 = 3 with a side of length 1. We can plot our square to make sure it creates the shape we expect:

| 12 | sq <- make\_square(x0 = 2, y0 = 3, width = 1)plot(sq) |

It works! Now we need to run the function 100 times to create 100 squares. We’ll use map2() from {purrr} to run it 100 times. For each square we want to vary the x and y coordinates, but keep the width constant. The x coordinates will run from 1 to 10 (repeated 10 times), as will the y coordinates (repeated 10 times each). We’ll set the width to 0.8 to leave a little bit of space between each square.

| 12345 | poly\_list <- purrr::map2( .x = rep(1:10, times = 10), .y = rep(1:10, each = 10), .f = ~ make\_square(.x, .y, width = 0.8)) |

Now we can join this list of squares (polygons) to our plot_data from earlier, and convert it to an sf object - and do it all at the same time using st_sf() from {sf}! We need to specify that the geometry comes from the list of squares we’ve created. We can again check it works by plotting our new sf object:

| 12 | plot\_sf <- sf::st\_sf(plot\_data, geometry = poly\_list)plot(plot\_sf) |

Although the colours aren’t mapped quite right, this has correctly created our 10x10 grid of squares.

Plotting with {roughsf}The {roughsf} package is reasonably straightforward to use as it has only one main function, roughsf(), which takes an sf object as input and outputs a plot. We can also set the width and height for the plot (in pixels):

| 12345 | roughsf::roughsf( plot\_sf, width = 800, height = 800,) |

If you’re using RStudio, you’ll notice that this is appears in the Viewer tab rather than the Plots tab since it’s an HTML widget.

There are different types of patterns that can be used to colour in the squares including "hachure", "solid", "zigzag", "cross-hatch", "dots", "dashed", or "zigzag-line". This needs to be specified as a column in the data called fillstyle. The fillweight column will also control how thick the lines are in the fill pattern (it should have values between 0 and 1, with 1 resulting in thicker lines). After adding these columns, we can then re-run the roughsf() function above:

| 12 | plot\_sf$fillstyle <- "cross-hatch"plot\_sf$fillweight <- 0.8 |

This is starting to look quite like what I was imagining, although it still doesn’t look quite sketchy enough for my liking. The roughness and bowing arguments in the roughsf() function control the roughness and bowing (rounded-ness) of the lines in the plot. You might want to play around with these values to find something you like (larger numbers generally lead to rougher, more sketchy looking plots):

| 1234567 | roughsf::roughsf( plot\_sf, roughness = 3, bowing = 2, width = 800, height = 800,) |

We can also add a title and caption to the plot. The font size and family can be specified using the title_font and caption_font arguments. Just like {ggplot2} plots, {roughsf} plots can be saved as objects in R. Let’s save this plot as rsf:

| 1 2 3 4 5 6 7 8 91011 | rsf <- roughsf::roughsf(plot\_sf, title = "The Colours of Bob Ross Paintings", title\_font = "48px Pristina", caption = "Graphic: Nicola Rennie", caption\_font = "30px Pristina", roughness = 3, bowing = 2, width = 800, height = 800,)rsf |

You might also want to save the plot as a static image (e.g. as a PNG file). Luckily, the {roughsf} package comes with the save_roughsf() function to make this easy. Simply save the {roughsf} plot as a variable (e.g. rsf), and pass it in as the first argument to save_roughsf(). The second argument is the file name (including the file extension) you’d like to save it as.

To make the white squares in the top row stand out a little bit more, we can also make the background a light beige by setting background = "#f5f5dc" (rather than the default white background).

| 12345 | roughsf::save\_roughsf( rsf = rsf, file = "bob\_ross\_waffle.png", background = "#f5f5dc") |

I hope this blog post has inspired you to create your own sketchy looking waffle charts, even if this is perhaps one of the most overly complicated ways of making a waffle chart in R. As long as you can break your chart down into lines, squares, and circles, you can create pretty much anything. Don’t get boxed in by using only what already exists!

To leave a comment for the author, please follow the link and comment on their blog: R on Nicola Rennie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Sketchy waffle charts in R

View Details

[This article was first published on vusaverse, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Announcing the polyglotr packagepolyglotrThe package polyglotr is tool for language translation within the R programming environment. This package stands out for its ability to integrate with a variety of free translation services, making it a versatile solution for translating text and files across different languages.

To get started with polyglotr, follow these simple steps:

```

Install the package from CRAN install.packages("polyglotr")## Or install the development version from GitHub remotes::install_github("Tomeriko96/polyglotr")## Load package library(polyglotr)

`` Key Features of polyglotr* Multiple Translation Services Integration: The package supports a wide range of translation services, includingGoogle Translate API,Mymemory API,Linguee API,Pons API,QCRI API, andWikimedia Translation API. This broad support ensures that users have access to a variety of translation options, catering to different needs and preferences. * Ease of Use: The package is designed with simplicity in mind, providing convenient functions for translating text and files. This makes it accessible to both beginners and experienced R users, enabling them to quickly incorporate translation capabilities into their projects. * Comprehensive Documentation: To assist users in leveraging the full potential of thepolyglotrpackage, it includes a detailed reference page that lists all available functions, organized by category. This resource is invaluable for users looking to explore the package’s capabilities or find specific functions for their translation tasks. * Vignettes for Detailed Guidance: Beyond the reference page,polyglotralso offers vignettes that provide more detailed information on how to use the functions within the package. These vignettes serve as practical guides, offering step-by-step instructions and examples to help users get started with translating text and files in various languages. * Batch Translation and File Translation Capabilities: The package includes functions for batch translations and file translations, allowing users to translate large volumes of text or entire documents efficiently. * Language Detection:polyglotr` offers language detection methods, which can automatically identify the language of the input text.

For a comprehensive list of features and detailed documentation, we recommend referring to the official polyglotr documentation. It provides in-depth explanations and examples to help you make the most of polyglotr in your R workflows.

Contributing to polyglotrpolyglotr is an open-source project, and contributions from the community are highly encouraged. If you encounter any bugs, have feature requests, or would like to contribute code improvements, you can open an issue or submit a pull request on the GitHub repository.

Further readingLiked this post? Check out more R-related content on r-bloggers.com.

To leave a comment for the author, please follow the link and comment on their blog: vusaverse.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Guest Post: Introducing the polyglotr package

View Details

[This article was first published on R on jmarriott.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I recently noticed that ggplot2 spaces date axes literally even when grouped by month. I’ve been using ggplot2 extensively for years and I don’t remember noticing before, so this is not really a big deal, but now that I know it bugs me a lot. Take a look below.

As always, let’s load the tidyverse.

library(tidyverse) Next, we need some example data. Nothing fancy, just something to plot as an example.

month\_data <- tibble(dates = seq(ymd("2023-11-01"), ymd("2024-04-30"), by = "day"), data = sample(1:1e3, 182)) |> mutate(month = floor\_date(dates, "month")) |> group\_by(month) |> summarise(monthly\_mean = mean(data))month\_data## # A tibble: 6 × 2## month monthly\_mean## <date> <dbl>## 1 2023-11-01 572.## 2 2023-12-01 601.## 3 2024-01-01 471.## 4 2024-02-01 562.## 5 2024-03-01 487.## 6 2024-04-01 438. Now let me show you how the distance between the months is not equal.

month\_data |> ggplot(aes(x = month, y = monthly\_mean)) + geom\_col(width = 29) + ggtitle("The Months Are Not Spaced Evenly!") + # add the year to the first month plotted in each year scale\_x\_date(labels = scales::label\_date\_short()) How frustrating! The fix is not that hard, though. We are going to change the date axis to factors with a neat little trick. We’re going to hack the label_date_short() function to change the dates to text and then change the dates to factors. Here we go.

```

get the label_date_short() functionprettify_dates <- scales::label_date_short()# we use the new function we made and make the dates factors so that they sort # correctly. new_month_data <- month_data |> mutate(pretty_month = prettify_dates(month), pretty_month = as_factor(pretty_month))new_month_data## # A tibble: 6 × 3## month monthly_mean pretty_month## ## 1 2023-11-01 572. "Nov\n2023" ## 2 2023-12-01 601. "Dec" ## 3 2024-01-01 471. "Jan\n2024" ## 4 2024-02-01 562. "Feb" ## 5 2024-03-01 487. "Mar" ## 6 2024-04-01 438. "Apr"

`` Now all we have to do is drop the date scale and letgeom_col()` chose its own width.

new\_month\_data |> ggplot(aes(x = pretty\_month, y = monthly\_mean)) + geom\_col() + ggtitle("The Spacing Is Even Now!") To leave a comment for the author, please follow the link and comment on their blog: R on jmarriott.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Evenly Spaced Month Charts

View Details

[This article was first published on Adam’s Software Lab, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.IntroductionA while back, I introduced the ExcelRAddIn (Office365 AddIns for R (Part I)). This is an Office365 AddIn that allows you to evaluate an R-script from within Excel and use the results. This blog-post describes some of the recent updates to the ExcelRAddIn. I focus on two specific areas. Firstly, I describe some ease of use features. Then I describe the function wrappers.

Ease of use features.* As a convenience, users can now specify packages to load when the add-in is initialised. This is available from the Settings button on the R Tools AddIn ribbon.

In the previous version, packages were loaded by executing the R-script library(<package-name>). In this version, default package loading takes place on the first call to RScript.Evaluate(...), so the first time any R-script is evaluated, there may be a slight delay depending on which and how many packages are loaded. Any issues with the package loading are reported to the R Environment AddIn panel (see below).

In the previous version, the three functions (CreateVector, CreateMatrix, and CreateDataFrame) which are used to pass data from Excel to R, used a final parameter ‘Type’. This indicated the corresponding R-type (which can be ‘character’, ‘complex’, ‘integer’, ‘logical’, or ‘numeric’). This is now optional; the R-type is determined from the data, if possible. This makes it somewhat easier to create objects to pass to R from Excel. For example, given an Excel table called ‘GalapagosData’ (from the faraway dataset), we can create a data frame simply by passing in a name (“gala”), the data and the headers:
Two generic calls have been added: RScript.Params and RScript.Function. RScript.Params returns a list of parameters for the requested function and RScript.Function evaluates the specified function, possibly using some or all of the parameters retrieved from the call to RScript.Params.
Some additional functions for querying models (i.e. objects returned from calls to ‘lm’, ‘glm’ etc) have been added: Model.Results outputs a list of results from the model. Model.Result outputs the result obtained from one item of the list of model results. Optionally, the result can be formatted as a data frame. This is somewhat more convenient than having to evaluate scripts of the form 'model name'$coeffcients, etc. Model.Accuracy returns a number of statistics relating to measures of model accuracy.

Wrapper functions.One of the motivations for updating the ExcelRAddIn was to provide an improved experience when using more complex R functions in an Excel worksheet. The idea was to avoid building up a script by providing wrapper functions that can handle the variety of parameters passed to the underlying R functions. The option of using a script is always available. However, for a complex function like auto.arima (which can take up to 35 parameters) or glm, it is easier to setup a parameter dictionary with the appropriately named parameters and their values (as shown below)

rather than creating a script, for example: logModel = glm(Purchase~Income+Age+ZipCode, data = purchase, family = binomial(link='logit'))

This also makes it easier to see the effects of any updates to model parameters. As described above, the parameter names and their default values can be retrieved by using the RScript.Params function.

At the moment, wrapper functions have been provided for a number of the functions in the forecast library and for the following two ‘workhorse’ functions:

Regression.LM– Fit a linear model to the data
Regression.GLM– Fit a generalised linear model to the data

A spreadsheet with examples based on the underlying packages can be downloaded from here: Forecast.xlsx.

Wrap-upIn this blog-post I have described two sets of enhancements to the ExcelRAddIn. Firstly some ease of use features were described. Secondly, I outlined some function wrappers that provide an improved user experience when using complex R functions in Excel. I am still working on improving the default (‘summary’) output display of results. Overall, the ExcelRAddIn seeks to provide access to R functionality from inside Excel in a way that is somewhat more flexible than the existing Data Analysis Toolpak.

To leave a comment for the author, please follow the link and comment on their blog: Adam’s Software Lab.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Office365 AddIns for R (Part III)

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Last year, the R Consortium had a conversation with Amal Tlili, the co-organizer of the Tunis R User Group, regarding the Use of R for Marketing and CRM in Tunisia. This year, Amal Boukteb and Hedia Tnani spoke to the R Consortium about the use of R for bioinformatics research in Tunisia and discussed the group’s efforts to bridge the gap between academia and industry. The Tunis R User Group hosts engaging virtual events to connect R enthusiasts across the MENA region and Worldwide. Their events promote the use of R and foster knowledge and skill development in data science and bioinformatics.

Amal Boukteb is a PhD student at the National Institute for Agricultural Research of Tunisia (INRAT). She holds master’s degrees in Molecular Genetics and Biostatistics. Her PhD project focuses on Orobanche foetida, a parasitic plant threatening faba bean crops. She analyzed O. foetida genetic diversity in Tunisia with RADseq and studied faba bean gene expression during this parasitic plant attack using RNA-seq. With a passion for integrating bioinformatics and Plant biology, Amal is determined to make significant contributions to the implementation of sustainable agricultural practices.

Dr. Hedia Tnani is a Staff Scientist at Lieber Institute for Brain Development (LIBD). She did a PhD in molecular biology and genetics. Her current work focuses on addressing the complex challenge of RNA degradation in postmortem brain tissue samples. She’s also the co-founder of R-Ladies Tunis and Tunis R User Group. Through the Tunis R User Group, she wants to democratize bioinformatics and data science.

Hédia and Amal met during the Bioinformatics and Genome Analyses course at Pasteur Institute of Tunis in 2017. Amal joined Tunis R User Group as a Bioinformatics Event Organizer

in 2023. With a deep commitment to inclusivity and empowerment, they’ve dedicated themselves to breaking down barriers faced by women and individuals from low-income countries when accessing education in these cutting-edge areas. By organizing workshops tailored to these communities, they aim to provide valuable skills and knowledge and foster a more diverse and equitable future in the bioinformatics field.

Please share about your background and involvement with the RUGS group.

Amal: We are biologists, and our academic curriculum did not include any programming courses. However, with the advancement of sequencing technologies, biologists are now facing the challenge of analyzing vast amounts of genomic data. This is a significant challenge for us. For my PhD project, I was involved in RNA-Seq and RAD-Seq projects. To overcome this challenge, I attended a course on analyzing genomic data using Unix, where I met Hedia for the first time. Additionally, in the framework of my thesis project, I had the opportunity to visit the Plant Immunity Group at RIKEN Yokohama in Japan for an internship. While there, I learned a lot from the talented scientists and their exciting research in bioinformatics.

When it comes to learning R programming for biologists, no specific courses are available. The only courses that exist are general ones. So, to overcome this gap, I started learning by myself. I attempted to understand the concepts by reading through error messages, package tutorials, and watching YouTube tutorials.

We realized we faced the same challenge after discussing this issue with our colleagues. We have genomic data that we need to analyze, but the available courses are located outside of Tunisia, primarily in Europe. Unfortunately, we lack the financial support to attend these courses. Additionally, obtaining a student visa for a temporary stay to attend such courses is a complex process. This challenge is not only unique to Tunisians but also a struggle for Africans and many biologists from middle and low-income countries. Our Tunis R user group aims to help others overcome this challenge and bridge this gap.

Hedia: I studied agronomy first and then pursued a master’s degree in plant breeding from Spain. Later, I completed my PhD in genetics. I did not know programming or R during my studies in Tunisia and Spain. However, when I started my postdoc at the International Rice Research Institute (IRRI) in the Philippines, especially when I first faced ‌analyzing genomics data, I felt out of my depth. With no programming experience, learning R seemed like a mountain too steep to climb. This is a familiar story for many biologists transitioning from wet to dry labs, where code replaces beakers. Despite the daunting challenge, I persevered and taught myself R; eventually, it became an invaluable tool for my research. I’m also thankful to the great mentors I had at IRRI who helped me accelerate my learning curve. My journey wasn’t easy, but it was incredibly rewarding.

Learning bioinformatics can be challenging, especially in regions like Tunisia where resources are scarce and training abroad is so costly. Moreover, the need for bioinformatics training to solve biological problems has left many highly skilled biologists struggling to find a job in their field. Recognizing these obstacles, we formed a supportive community to facilitate collective learning and growth in bioinformatics and related fields such as data science and artificial intelligence.

Our community is a friendly, inclusive, and welcoming space for anyone passionate about bioinformatics, data science, artificial intelligence, and beyond. We’re all about growing together and learning from each other in a supportive environment. Whether you’re just starting out or have lots of experience, we encourage you to dive in, ask questions, and share your insights. We all rise by lifting others. Don’t worry about asking the “wrong” question. Every question is a chance to understand and learn something new. Come join us and be part of our journey of discovery and growth. We can’t wait to learn with you!

Can you share what the R community is like in Tunisia?

Hedia: In Tunisia, programming is mainly used in the industry, but it is not widely taught in the curriculum for biologists. This creates a gap between what is taught in the academic courses and what is required in the industry. As a result, individuals are expected to possess programming skills when they work in the industry. Still, they may not have been able to learn programming during their academic courses. This gap must be addressed to better prepare individuals for the job market.

Can you please update us about the group’s recent activities?

Amal: First, it is important to mention that Arabic is our native language in Tunisia. However, French is the predominant teaching language in many subjects, including biology and informatics. Despite this, we have decided to conduct our workshop in English for the Tunis user group for two main reasons. Firstly, we aim to bridge the gap between the academic skills acquired in French and English resources. Secondly, by using English as our teaching language, we can reach a broader audience of scientists who share our needs.

We decided to allow us the flexibility to choose speakers without language barriers. Our main goal is to reach a broad audience worldwide. During our workshop, we noticed participants worldwide, not just Tunisians. This is very important to us. We conducted workshops for biologists, such as the Genome-Wide Association Studies (GWAS) workshop, and we already have 5k views on our YouTube channel. It is interesting to see that people are very interested in our workshops. We also had the opportunity to collaborate with highly qualified researchers in their respective fields. Within our community, we were privileged to learn from Pr. Emerson Del Ponte generously shared his expertise using R for Plant Disease Epidemiology.

We aim not only to cover biological subjects but also those related to artificial intelligence. Recently, we conducted two successful workshops on Building a Chatbot with OpenAI, Shiny and R, and Bioinformatics Analysis using Chatlize and ChatGPT. We strive to have a balance between biological and AI-related subjects to make the experience easier for our participants with the help of artificial intelligence.

What trends do you currently see in R language?

Hedia: In bioinformatics, there is a growing trend towards single-cell and spatial transcriptomics. Our latest event was an introduction to single-cell RNA-seq analysis. Additionally, packages based on OpenAI API are increasingly being used. For instance, many of those packages can be used by people who lack coding skills. This is particularly helpful because not all biologists possess coding skills, and it makes their work easier. Another trend we have noticed is using Quarto instead of R Markdown. Shiny is also gaining popularity in this field.

We have been receiving a lot of queries about bioinformatics workshops lately, particularly because they offer a diverse range of events, such as user groups. However, it can be challenging to find a specific topic. For instance, some R user groups may only hold one or two events yearly, whereas we host monthly bioinformatics events.

We value feedback from our attendees and gather suggestions from our latest events to improve our upcoming ones. Our events are designed to stay current with trends in the industry, and we often invite guest speakers to talk about relevant topics. For instance, during one of our workshops about Building a Chatbot with OpenAI, we had 200 participants whom we taught how to use R and create their chatbots. We learn from our experiences, and when we notice an interest in a particular area, we look to bring in speakers to teach on that subject.

Hedia: Our organization had a sponsorship for our Zoom account, an important tool for hosting events. One of the features that we utilize is the captions option, which allows participants from all over the world to have captions in their language and helps them follow the workshop. This is particularly helpful for those who may have difficulty understanding English. We are very grateful to Appsilon for their sponsorship of our Zoom account.

Amal: Thanks to Appsilon’s sponsorship, we have accepted more participants for our events. Previously, the number of participants was limited due to the capacity of our Zoom account. However, with this sponsorship, we can now handle up to 100 participants per event. This has made it easier for us to accept more subscribers and host successful workshops. We recently had an event with over 200 participants, which was a great success.

Hedia: We provide teaching materials for our speaker sessions on GitHub. You can find all the materials on YouTube and use them to reproduce what the speaker did during their session. We are always open to questions, especially if you encounter bugs while trying to reproduce the speaker’s work. Recently, we received an email from a participant experiencing a bug, and we had a great time figuring it out together. If you have any questions or problems, feel free to ask us for help, and we’ll do our best to assist you.

Are your events online, in-person, or hybrid?

Hedia: We are considering organizing hybrid events in the future, and we are searching for funding. We only have sponsorship for our Zoom, so we need additional funds to make this happen. We plan to organize events at multiple universities across the MENA region so important speakers can be followed in person and online. Amal, who is based in Tunis, has been in contact with many universities and academic professionals in the area. We’re currently exploring the best ways to make these hybrid events a reality, ensuring a seamless and enriching experience for everyone involved. Our goal is to make these events as engaging and accessible as possible, fostering a true sense of community.

We want to organize events for online events and to provide something valuable to our community. When we meet in person, we can better understand their needs and challenges, which helps us to build and organize workshops that cater to their specific needs. Recently, Amal mentioned that some courses are not free in Tunisia, which can be a barrier for some people. Therefore, we aim to organize a free hybrid event for everyone who wants to join and learn with us. We hope to get funding for this initiative to provide this opportunity to all.

Amal: For my PhD project, I conducted research on population genomics and RNA-seq to investigate the interaction between plants and parasitic plants. Our work shed light on the genetic diversity of Orobanche foetida, a parasitic plant posing a significant threat to faba beans in Tunisia. Additionally, through RNA-seq analysis, we identified a potential target gene for developing resistant varieties of faba beans against this parasitic plant. Furthermore, I recently completed a bachelor’s degree in biostatistics, specifically focusing on Aphid diversity in Tunisia.

During my academic journey, R has been my primary tool for conducting comprehensive data analysis across all my research projects. After finishing my PhD, I aim to develop my expertise in bioinformatics further, specifically focusing on wheat genomics.

Hedia: I primarily use R as the main software for all my research projects. I am currently working on maintaining and improving a package called qsvaR. qsvaR is a tool that generates quality surrogate variable analysis for degradation correction in RNA cells. It contains functions that help remove the degradation effect in post-mortem brain tissue, making it a useful tool for generating basic data. We are currently working on a publication based on this work.

Learn moreThe post Bridging Gaps: Tunis R User Group’s Journey in Democratizing R in Bioinformatics appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Bridging Gaps: Tunis R User Group’s Journey in Democratizing R in Bioinformatics

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn data preprocessing and text manipulation tasks, the strsplit() function in R is incredibly useful for splitting strings based on specific delimiters. However, what if you need to split a string using multiple delimiters? This is where strsplit() can really shine by allowing you to specify a regular expression that defines these delimiters. In this blog post, we’ll dive into how you can use strsplit() effectively with multiple delimiters to parse strings in your data.

Understanding strsplit()The strsplit() function in R is used to split a character vector (or a string) into substrings based on a specified pattern. The general syntax of strsplit() is:

strsplit(x, split, fixed = FALSE, perl = FALSE, useBytes = FALSE) * x: The character vector or string to be split. * split: The delimiter or regular expression to use for splitting. * fixed: If TRUE, split is treated as a fixed string rather than a regular expression. * perl: If TRUE, split is treated as a Perl-style regular expression. * useBytes: If TRUE, the matching is byte-based rather than character-based.

Splitting with Multiple DelimitersTo split a string using multiple delimiters, we can leverage the power of regular expressions within strsplit(). Regular expressions allow us to define complex patterns that can match various types of strings.

Let’s say we have the following string that contains different types of delimiters: space, comma, and hyphen:

text <- "apple,orange banana -grape pineapple" We want to split this string into individual words based on the delimiters ,, , and -. Here’s how we can achieve this using strsplit():

result <- strsplit(text, "[,\\s-]+")result

[[1]][1] "apple" "orange banana " "grape pineapple" In this example: - [ and ] define a character class. - ,, \\s, and - inside the character class specify the delimiters we want to use for splitting. - + after the character class means “one or more occurrences”.

Examples with Different DelimitersLet’s explore a few more examples to understand how strsplit() handles different scenarios:

Example 1: Splitting with Numbers as Delimiters text <- "Hello123world456R789users"result <- strsplit(text, "[0-9]+") In this case, we use [0-9]+ to split the string wherever there are one or more consecutive digits. The result will be:

result

[[1]][1] "Hello" "world" "R" "users" Example 2: Splitting URLs url <- "https://www.example.com/path/to/page.html"result <- strsplit(url, "[:/\\.]") Here, we split the URL based on :, /, and . characters. The result will be:

result

[[1]] [1] "https" "" "" "www" "example" "com" "path" [8] "to" "page" "html" Your Turn to ExperimentThe best way to truly understand and harness the power of strsplit() with multiple delimiters is to experiment with different strings and patterns. Try splitting strings using various combinations of characters and observe how strsplit() behaves.

By mastering strsplit() and regular expressions, you can efficiently preprocess and manipulate textual data in R, making your data analysis tasks more effective and enjoyable.

So, why not give it a try? Experiment with strsplit() and multiple delimiters on your own datasets to see how this versatile function can streamline your data cleaning workflows. If you want a really good cheat sheet of regular expressions then check out this one from the stringr package from Posit.

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring strsplit() with Multiple Delimiters in R

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.R 4.4.0 (“Puppy Cup”) was released on the 24th April 2024 and it is abeauty. In time-honoured tradition, here we summarise some of thechanges that caught our eyes. R 4.4.0 introduces some cool features (oneof which is experimental) and makes one of our favourite {rlang}operators available in base R. There are a few things you might need tobe aware of regarding handling NULL and complex values.

The full changelog can be found at the r-release ‘NEWS’page and ifyou want to keep up to date with developments in base R, have a look atthe r-devel ‘NEWS’page.

Data comes in all shapes and sizes. It can often be difficult to know where to start. Whatever your problem, Jumping Rivers can help.

A tail-recursive taleYears ago, before I’d caused my first stack overflow, my Grandad used totell me a daft tale:

It was on a dark and stormy night,And the skipper of the yacht said to Antonio,"Antonio, tell us a tale",So Antonio started as follows...It was on a dark and stormy night,And the skipper of the yacht .... [ad infinitum] The tale carried on in this way forever. Or at least it would until youwere finally asleep.

At around the same age, I was toying with BASIC programming and couldknock out classics such as

```

10 PRINT "Ali stinks!">20 GOTO 10 ``` Burn! Infinite burn!

That was two example processes that demonstrate recursion. Antonio’stale quotes itself recursively, and my older brother will be repeatedlymocked unless someone intervenes.

Recursion is an elegant approach to many programming problems – thisusually takes the form of a function that can call itself. You would useit when you know how to get closer to a solution, but not necessarilyhow to get directly to that solution. And unlike the un-ending examplesabove, when we write recursive solutions to computational problems, weinclude a rule for stopping.

An example from mathematics would be finding zeros for a continuousfunction. The sine function provides a typical example:

We can see that when x = π, there is a zero for sin(x), but thecomputer doesn’t know that.

One recursive solution to finding the zeros of a function, f(), is thebisection method,which iteratively narrows a range until it finds a point where f(x) isclose enough to zero. Here’s a quick implementation of that algorithm.If you need to perform root-finding in R, please don’t use the followingfunction. stats::uniroot() is much more robust…

bisect = function(f, interval, tolerance, iteration = 1, verbose = FALSE) { if (verbose) { msg = glue::glue( "Iteration {iteration}: Interval [{interval[1]}, {interval[2]}]" ) message(msg) } # Evaluate 'f' at either end of the interval and return # any endpoint where f() is close enough to zero lhs = interval[1]; rhs = interval[2] f\_left = f(lhs); f\_right = f(rhs) if (abs(f\_left) <= tolerance) { return(lhs) } if (abs(f\_right) <= tolerance) { return(rhs) } stopifnot(sign(f\_left) != sign(f\_right)) # Bisect the interval and rerun the algorithm # on the half-interval where y=0 is crossed midpoint = (lhs + rhs) / 2 f\_mid = f(midpoint) new\_interval = if (sign(f\_mid) == sign(f\_left)) { c(midpoint, rhs) } else { c(lhs, midpoint) } bisect(f, new\_interval, tolerance, iteration + 1, verbose)} We know that π is somewhere between 3 and 4, so we can find the zeroof sin(x) as follows:

bisect(sin, interval = c(3, 4), tolerance = 1e-4, verbose = TRUE)#> Iteration 1: Interval [3, 4]#> Iteration 2: Interval [3, 3.5]#> Iteration 3: Interval [3, 3.25]#> Iteration 4: Interval [3.125, 3.25]#> Iteration 5: Interval [3.125, 3.1875]#> Iteration 6: Interval [3.125, 3.15625]#> Iteration 7: Interval [3.140625, 3.15625]#> Iteration 8: Interval [3.140625, 3.1484375]#> Iteration 9: Interval [3.140625, 3.14453125]#> Iteration 10: Interval [3.140625, 3.142578125]#> Iteration 11: Interval [3.140625, 3.1416015625]#> [1] 3.141602 It takes 11 iterations to get to a point where sin(x) is within10−4 of zero. If we tightened the tolerance, had a morecomplicated function, or had a less precise starting range, it mighttake many more iterations to approximate a zero.

Importantly, this is a recursive algorithm - in the last statement ofthe bisect() function body, we call bisect() again. The initial callto bisect() (with interval = c(3, 4)) has to wait until the secondcall to bisect() (interval = c(3, 3.5)) completes before it canreturn (which in turn has to wait for the third call to return). So wehave to wait for 11 calls to bisect() to complete before we get ourresult.

Those function calls get placed on a computational object named thecall stack. For eachfunction call, this stores details about how the function was called andwhere from. While waiting for the first call to bisect() to complete,the call stack grows to include the details about 11 calls tobisect().

Imagine our algorithm didn’t just take 11 function calls to complete,but thousands, or millions. The call stack would get really full andthis would lead to a “stack overflow”error.

We can demonstrate a stack-overflow in R quite easily:

blow\_up = function(n, max\_iter) { if (n >= max\_iter) { return("Finished!") } blow\_up(n + 1, max\_iter)} The recursive function behaves nicely when we only use a small number ofiterations:

blow\_up(1, max\_iter = 100)#> [1] "Finished!" But the call-stack gets too large and the function fails when we attemptto use too many iterations. Note that we get a warning about the size ofthe call-stack before we actually reach it’s limit, so the R process cancontinue after exploding the call-stack.

blow\_up(1, max\_iter = 1000000)# Error: C stack usage 7969652 is too close to the limit In R 4.4, we are getting (experimental) support for tail-callrecursion. This allows us (inmany situations) to write recursive functions that won’t explode thesize of the call stack.

How can that work? In our bisect() example, we still need to make 11calls to bisect() to get a result that is close enough to zero, andthose 11 calls will still need to be put on the call-stack.

Remember the first call to bisect()? It called bisect() as the verylast statement in it’s function body. So the value returned by thesecond call to bisect() was returned to the user without modificationby the first call. So we could return the second call’s value directlyto the user, instead of returning it via the first bisect() call;indeed, we could remove the first call to bisect() from the call stackand put the second call in it’s place. This would prevent the call stackfrom expanding with recursive calls.

The key to this (in R) is to use the new Tailcall() function. Thattells R “you can remove me from the call stack, and put this cat oninstead”. Our final line in bisect() should look like this:

bisect = function(...) { ... snip ... Tailcall(bisect, f, new\_interval, tolerance, iteration + 1, verbose)} Note that you are passing the name of the recursively-called functioninto Tailcall(), rather than a call to that function (bisect ratherthan bisect(...)).

To illustrate that the stack no longer blows up when tail-call recursionis used. Let’s rewrite our blow_up() function:

```

R 4.4.0blow_up = function(n, max_iter) { if (n >= max_iter) { return("Finished!") } Tailcall(blow_up, n+1, max_iter)}

``` We can still successfully use a small number of iterations:

blow\_up(1, 100)#> [1] "Finished!" But now, even a million iterations of the recursive function can beperformed:

blow\_up(1, 1000000)#> [1] "Finished!" Note that the tail-call optimisation only works here, because therecursive call was made as the very last step in the function body. Ifyour function needs to modify the value after the recursive call, youmay not be able to use Tailcall().

Rejecting the NULLMissing values are everywhere.

In a typical dataset you might have missing values encoded as NA (ifyou’re lucky) and invalid numbers encoded as NaN, you might haveimplicitly missing rows (for example, a specific date missing from atime series) or factor levels that aren’t present in your table. Youmight even have empty vectors, or data-frames with no rows, to contendwith. When writing functions and data-science workflows, where the inputdata may change over time, by programming defensively and handling thesekinds of edge-cases your code will throw up less surprises in the longrun. You don’t want a critical report to fail because a mathematicalfunction you wrote couldn’t handle a missing value.

When programming defensively with R, there is another important form ofmissingness to be cautious of …

The NULLobject.

NULL is an actual object. You can assign it to a variable, combine itwith other values, index into it, pass it into (and return it from) afunction. You can also test whether a value is NULL.

```

Assignmentmy_null = NULLmy_null#> NULL# Use in functionsmy_null[1]#> NULLc(NULL, 123)#> [1] 123c(NULL, NULL)#> NULLtoupper(NULL)#> character(0)# Testing NULL-nessis.null(my_null)#> [1] TRUEis.null(1)#> [1] FALSEidentical(my_null, NULL)#> [1] TRUE# Note that the equality operator shouldn't be used to# test NULL-ness:NULL == NULL#> logical(0)

`` R functions that are solely called for their side-effects (write.csv()ormessage(), for example) often return aNULLvalue. Otherfunctions may returnNULLas a valid value - one intended forsubsequent use. For example, list-indexing (which is a function call,under the surface) will returnNULL` if you attempt to access anundefined value:

config = list(user = "Russ")# When the index is present, the associated value is returnedconfig$user#> [1] "Russ"# But when the index is absent, a `NULL` is returnedconfig$url#> NULL Similarly, you can end up with a NULL output from an incomplete stackof if / else clauses:

language = "Polish"greeting = if (language == "English") { "Hello"} else if (language == "Hawaiian") { "Aloha"}greeting#> NULL A common use for NULL is as a default argument in a functionsignature. A NULL default is often used for parameters that aren’tcritical to function evaluation. For example, the function signature formatrix() is as follows:

matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE, dimnames = NULL) The dimnames parameter isn’t really needed to create a matrix, butwhen a non-NULL value for dimnames is provided, the values are usedto label the row and column names of the created matrix.

matrix(1:4, nrow = 2)#> [,1] [,2]#> [1,] 1 3#> [2,] 2 4matrix(1:4, nrow = 2, dimnames = list(c("2023", "2024"), c("Jan", "Feb")))#> Jan Feb#> 2023 1 3#> 2024 2 4 R 4.4 introduces the %||% operator to help when handling variablesthat are potentially NULL. When working with variables that could beNULL, you might have written code like this:

```

Remember there is no 'url' field in our `config` list# Set a default value for the 'url' if one isn't defined in# the configmy_url = if (is.null(config$url)) { "https://www.jumpingrivers.com/blog/"} else { config$url}my_url#> [1] "https://www.jumpingrivers.com/blog/"

`` Assumingconfigis alist`:

when the url entry is absent from config (or is itself NULL),then config$url will be NULL and the variable my_url will be setto the default value;
but when the url entry is found within config (and isn’t NULL)then that value will be stored in my_url.

That code can now be rewritten as follows:

```

R 4.4.0my_url = config$url %||% "https://www.jumpingrivers.com/blog"my_url#> [1] "https://www.jumpingrivers.com/blog"

`` Note that the left-hand value must evaluate toNULLfor the right-handside to be evaluated, and that empty vectors aren’tNULL`:

```

R 4.4.0NULL %||% 1#> [1] 1c() %||% 1#> [1] 1numeric(0) %||% 1#> numeric(0)

`` This operator has been available in the{rlang}package for eightyears and is implemented in exactly the same way. So if you have beenusing%||%in your code already, the base-R version of this operatorshould work without any problems, though you may want to wait until youare certain all your users are using R >= 4.4 before switching from{rlang} to the base-R version of%||%`.

Any other businessA shorthand hexadecimalformat(common in web-programming) for specifying RGB colours has beenintroduced. So, rather than writing the 6-digit hexcode for a colour“#112233”, you can use “#123”. This only works for those 6-digithexcodes where the digits are repeated in pairs.

Parsing and formatting of complex numbers has been improved. Forexample, as.complex("1i") now returns the complex number 0 + 1i,previously it returned NA.

There are a few other changes related to handling NULL that have beenintroduced in R 4.4. The changes highlight that NULL is quitedifferent from an empty vector. Empty vectors contain nothing, whereasNULL represents nothing. For example, whereas an empty numeric vectoris considered to be an atomic (unnestable) data structure, NULL is nolonger atomic. Also, NCOL(NULL) (the number of columns in a matrixformed from NULL) is now 0, whereas it was formerly 1.

sort_by() a new function for sorting objects based on values in aseparate object. This can be used to sort a data.frame based on it’scolumns (they should be specified as a formula):

mtcars |> sort\_by(~ list(cyl, mpg)) |> head()## mpg cyl disp hp drat wt qsec vs am gear carb## Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2## Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1## Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1## Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2## Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2## Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 Try the latest version out for yourselfTo take away the pain of installing the latest development version of R,you can use docker. To use the devel version of R, you can use thefollowing commands:

docker pull rstudio/r-base:devel-jammydocker run --rm -it rstudio/r-base:devel-jammy Once R 4.4 is the released version of R and the r-docker repositoryhas been updated, you should use the following command to test out R4.4.

docker pull rstudio/r-base:4.4-jammydocker run --rm -it rstudio/r-base:4.4-jammy See alsoThe R 4.x versions have introduced a wealth of interesting changes.These have been summarised in our earlier blog posts:

R 4.0.0
R4.1.0
R 4.2.0
R 4.3.0

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: What’s new in R 4.4.0?

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Harnessing the power of big data is akin to mastering an incredible energy source. In the realm of data science, R serves as both a sanctuary and a training ground where data scientists, like the legendary fighters from “Dragon Ball,” elevate their abilities to new heights. With the right tools and techniques, these modern-day warriors can tackle datasets of colossal size and complexity, turning potential chaos into structured insights.

As we embark on this journey to uncover the profound capabilities of R in managing and analyzing big data, we equip ourselves with the best tools from our arsenal — much like preparing for an epic battle. From optimizing memory usage to executing parallel processing, each step and technique will incrementally boost our prowess, making us more adept at navigating the ever-expanding universe of data.

Kamehameha of Knowledge: Understanding Big Data in RBig data is not just about volume; it’s about complexity and speed. In the world of R, big data can pose significant challenges, as the traditional in-memory processing model isn’t always feasible. Just as a Saiyan must understand their own strengths and limitations, a data scientist needs to assess the capabilities of their tools — knowing when they suffice and when to seek more powerful solutions.

Built-in datasets example to demonstrate limitationsTo illustrate, let’s use R’s built-in diamonds dataset from the ggplot2 package, which is moderately large but manageable, to show how typical data operations scale with dataset size.

library(ggplot2)library(microbenchmark)data("diamonds")print(dim(diamonds))print(object.size(diamonds), units = "MB")

small = function(){weighted\_average\_price <- sum(diamonds$price * diamonds$carat) / sum(diamonds$carat)} This example with the diamonds dataset, which contains data about 50,000 diamonds, provides a baseline for understanding operations on large data. It's sizable enough to start showing performance issues, particularly when operations become complex.

Exploring Common ChallengesHandling big data in R comes with several significant challenges:

Memory Management: Traditional R objects reside entirely in memory. When data exceeds memory capacity, it leads to swapping and slowdowns. Memory-efficient objects and programming tricks are needed to manage large data efficiently.
Processing Speed: As operations expand in complexity, such as multiple joins or applying machine learning algorithms, the need for optimized code and efficient computation becomes critical.
Data I/O: Efficient data input/output is crucial. The time it takes to read from and write to disk can become a bottleneck. Utilizing databases or specialized data formats can mitigate these issues.

llustrative Example with Larger Data SimulationTo further illustrate, let’s simulate a larger dataset to demonstrate how typical operations start to lag as data grows:

set.seed(123)large\_data <- diamonds[sample(nrow(diamonds), 1e7, replace = TRUE), ]print(dim(large\_data))print(object.size(large\_data), units = "MB")

big = function(){ weighted\_average\_price <- sum(large\_data$price * large\_data$carat) / sum(large\_data$carat)} This simulation replicates the diamonds dataset to size of 10M rows, significantly increasing the data size and demonstrating how computation time increases even for simple operations.

Let now check how those two functions works for small and large data portion.

microbenchmark(small(), big(), times = 100) Navigating the Big Data Landscape in RUnderstanding these challenges is the first step toward mastering big data in R. The subsequent sections will explore specific tools and techniques that address these issues, much like how a Saiyan learns to control their Ki to face stronger adversaries.

Training in the Hyperbolic Time Chamber: Essential R Packages for Big DataJust as warriors in “Dragon Ball” enter the Hyperbolic Time Chamber to gain years of training in a day, R programmers have access to powerful packages that significantly enhance their ability to handle large datasets more efficiently. These packages are akin to secret techniques that speed up data manipulation, reduce memory overhead, and allow more complex data analysis.

data.table: Supercharging Data ManipulationOne of the most potent tools in R for handling big data is data.table. It extends data.frame but is designed to be much faster and more intuitive, especially for large datasets.

library(data.table)DT\_diamonds <- as.data.table(diamonds)DT\_large\_data <- as.data.table(large\_data)small\_DT = function(){ avg\_price\_by\_cut <- DT\_diamonds[, .(Average\_Price = mean(price)), by = cut]}big\_DT = function(){ avg\_price\_by\_cut <- DT\_large\_data[, .(Average\_Price = mean(price)), by = cut]}microbenchmark(small\_DT(), big\_DT(), times = 100) I used exactly the same data sets as before. They were only transformed to data.table structures. Look how it perform…

About 50x faster for small dataset and almost 300x faster for bigger one. This example demonstrates the use of data.table for fast data aggregation. Its syntax and processing capabilities make it invaluable for large-scale data operations.

dplyr with dbplyrTapping into Databases For datasets too large to fit into memory, dplyr's syntax can be used with dbplyr to work directly on database-backed data. This allows operations to be translated into SQL, executed in the database without pulling data into R.

library(dplyr)library(dbplyr)# Assuming db\_conn is a connection to a databasetbl\_diamonds <- tbl(db\_conn, "diamonds")# Perform database-backed operationsresult <- tbl\_diamonds %>% group\_by(cut) %>% summarise(Average\_Price = mean(price), .groups = 'drop') %>% collect() # Pulls data into R only at this pointprint(result) ff and bigmemoryManaging Larger-than-memory Data The ff package and the bigmemory package provide data structures that store data on disk rather than in RAM, allowing R to handle datasets larger than the available memory.

library(ff)big\_vector = ff(runif(1e8), vmode = "double")ff\_big = function() {mean(big\_vector)}microbenchmark(ff\_big(), times = 100) This code uses ff to create a large vector (100M elements) that doesn't reside entirely in memory, demonstrating how ff handles very large datasets.

Enhancing R’s CapabilityThese packages transform R’s ability to manage and analyze big data, much like the special training in the Hyperbolic Time Chamber enhances a Saiyan’s power. By using these tools, data scientists can handle larger datasets more efficiently, conduct faster data processing, and perform complex analyses that were previously not feasible.

Fusion Technique: Unleashing Parallel Processing in RParallel processing in R allows data scientists to significantly reduce computation time by distributing tasks across multiple processors, similar to the Fusion technique in “Dragon Ball” where two characters combine their strengths to create a more powerful entity. This approach is particularly effective for large-scale data analysis and complex computations that are common in big data scenarios.

Why Parallel Processing?As datasets grow and analyses become more complex, single-threaded processing can become a bottleneck. Parallel processing enables the handling of more data and faster execution of operations, essential for timely insights in big data environments.

Core Packages for Parallel Processing in R parallel*: This package is part of the base R system and offers a variety of tools for parallel execution of code.

library(parallel)# Example of using the parallel packagenumCores <- detectCores() # 16 cores on my machinecl <- makeCluster(numCores)clusterEvalQ(cl, library(ggplot2))# Parallel apply to calculate mean price by cut using diamonds datasetpar\_result <- parLapply(cl, unique(diamonds$cut), function(cut) { data\_subset <- diamonds[diamonds$cut == cut, ] mean\_price <- mean(data\_subset$price) return(mean\_price)})stopCluster(cl)print(par\_result) This example sets up a cluster using all available cores, applies a function in parallel, and then shuts down the cluster.

foreach and doParallel: For a more flexible loop construct that can be executed in parallel.

library(foreach)library(doParallel)# Register parallel backendregisterDoParallel(cores=numCores)# Using foreach for parallel processingresults <- foreach(i = unique(diamonds$cut), .combine = rbind) %dopar% { data\_subset <- diamonds[diamonds$cut == i, ] mean\_price <- mean(data\_subset$price) return(c(Cut = i, Mean\_Price = mean\_price))}print(results) This uses foreach with doParallel to perform a parallel loop calculating mean prices, combining results automatically.

Advanced Usage and ConsiderationsWhile parallel processing can dramatically improve performance, it also introduces complexity such as data synchronization and the potential for increased memory usage. Effective use of parallel processing requires understanding both the computational overhead involved and the appropriate scenarios for its use.

Mastery Over Ultra Instinct: Best Practices for Optimizing Big Data Performance in RMastering data performance optimization in R is akin to achieving Ultra Instinct in the “Dragon Ball” series — where one reacts perfectly without thinking. In the realm of big data, this means setting up processes and code that are both efficient and scalable, minimizing resource waste and maximizing output.

Key Strategies for Performance Optimization:Efficient Data Storage and Access:

Using appropriate data formats: Opt for data formats that support fast read and write operations, such as fst for data frames, which can dramatically speed up data access times.
Database integration: When working with extremely large datasets, consider using R with database management systems. Utilize dplyr and dbplyr for seamless interaction with databases directly from R, enabling you to handle data that exceeds your machine's memory capacity.
Integration with Big Data Frameworks like Spark and sparklyr. For massive datasets and distributed computing scenarios, Apache Spark offers an efficient, general-purpose cluster-computing framework. The sparklyr package provides an R interface for Apache Spark, allowing you to connect to a Spark cluster from R and execute Spark jobs directly from R scripts.

From Saiyan to Super Saiyan GodThroughout this article, we have embarked on a journey much like that of a Saiyan in the “Dragon Ball” universe, progressively mastering greater powers to tackle increasingly formidable challenges. Just as these warriors evolve through training and battles, so have we explored and harnessed the tools and techniques necessary to manage and analyze big data with R.

Key Takeaways Understanding Big Data in R: We started by defining what constitutes big data in R and discussed the initial challenges related to memory management, processing speed, and data I/O. * Essential R Packages for Big Data: We delved into powerful R packages like data.table, dplyr with dbplyr, and ff, which enhance R’s capability to handle large datasets efficiently. * Parallel Processing Techniques: By exploring the parallel and foreach packages, we learned how to distribute computations across multiple cores to speed up data processing tasks. * Optimizing Big Data Performance*: We covered best practices in data storage and access, particularly focusing on the use of Spark through sparklyr for scalable data processing on a cluster environment.

As you continue your data science journey, remember that mastering these tools and techniques is an ongoing process. Each dataset and challenge may require a different combination of skills and strategies. Just like Saiyans who never stop training, always be on the lookout for new and improved ways to handle your data challenges.

Thank you for joining me on this adventure through the world of big data with R. Whether you are just starting out or looking to level up your skills, the path you take from here will be filled with challenges and triumphs. Keep pushing your limits, and may your data insights shine brightly like a Super Saiyan God!

Super Saiyan Data Skills: Mastering Big Data with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Super Saiyan Data Skills: Mastering Big Data with R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionAs an R programmer, one of the fundamental tasks you’ll encounter is manipulating data frames. Whether you’re cleaning messy data or preparing it for analysis, knowing how to drop unnecessary columns is a valuable skill. In this guide, we’ll walk through the process of dropping columns from data frames in R, using simple examples to demystify the process.

Why Drop Columns?Before we dive into the how, let’s briefly touch on the why. Data frames often contain more information than we need for a particular analysis. By dropping irrelevant columns, we can streamline our data and focus only on what’s essential. This not only makes our code cleaner but also improves performance when working with large datasets.

ExamplesMethod 1: Using the $ OperatorOne straightforward way to drop columns from a data frame is by using the $ operator. This method is ideal when you know the exact name of the column you want to remove.

```

Create a sample data framedata <- data.frame( A = c(1, 2, 3), B = c(4, 5, 6), C = c(7, 8, 9))# Drop column Bdata <- data[, -which(names(data) == "B")]# View the updated data frameprint(data)

```

A C1 1 72 2 83 3 9 In this example, we create a data frame data with columns A, B, and C. To drop column B, we use the which() function to find the index of column B in the names(data) vector and then remove it using negative indexing.

Method 2: Using the subset() FunctionAnother approach to dropping columns is by using the subset() function. This method allows for more flexibility, as you can specify multiple columns to drop at once.

```

Create a sample data framedata <- data.frame( A = c(1, 2, 3), B = c(4, 5, 6), C = c(7, 8, 9))# Drop columns B and Cdata <- subset(data, select = -c(B, C))# View the updated data frameprint(data)

```

A1 12 23 3 In this example, we use the select argument of the subset() function to specify the columns we want to keep. By prepending a minus sign to the column names we want to drop, we effectively remove them from the data frame.

Method 3: Using the dplyr PackageFor more complex data manipulation tasks, the dplyr package provides a convenient set of functions. One such function is select(), which allows for intuitive column selection and dropping.

```

Load the dplyr packagelibrary(dplyr)# Create a sample data framedata <- data.frame( A = c(1, 2, 3), B = c(4, 5, 6), C = c(7, 8, 9))# Drop column Bdata <- select(data, -B)# View the updated data frameprint(data)

```

A C1 1 72 2 83 3 9 In this example, we use the select() function from the dplyr package to drop column B from the data frame. The -B argument specifies that we want to exclude column B from the result.

ConclusionDropping columns from data frames in R doesn’t have to be a daunting task. By familiarizing yourself with these simple techniques, you can efficiently clean and manipulate your data with ease. I encourage you to try these examples on your own datasets and experiment with different variations. Remember, the best way to learn is by doing!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simplifying Data Manipulation: How to Drop Columns from Data Frames in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn data analysis, there often arises a need to extract the top N values within each group of a dataset. Whether you’re dealing with sales data, survey responses, or any other type of grouped data, identifying the top performers or outliers within each group can provide valuable insights. In this tutorial, we’ll explore how to accomplish this task using three popular R packages: dplyr, data.table, and base R. By the end of this guide, you’ll have a solid understanding of various approaches to selecting top N values by group in R.

ExamplesUsing dplyrdplyr is a powerful package for data manipulation, providing intuitive functions for common data manipulation tasks. To select the top N values by group using dplyr, we’ll use the group_by() and top_n() functions.

```

Load the dplyr packagelibrary(dplyr)# Example datasetdata <- data.frame( group = c(rep("A", 5), rep("B", 5)), value = c(10, 15, 8, 12, 20, 25, 18, 22, 17, 30))# Select top 2 values by grouptop_n_values <- data %>% group_by(group) %>% top_n(2, value)# View the resultprint(top_n_values)

```

A tibble: 4 × 2# Groups: group [2] group value 1 A 152 A 203 B 254 B 30

`` Explanation* We begin by loading the dplyr package. * We create a sample dataset with two columns: ‘group’ and ‘value’. * Using the%>%(pipe) operator, we first group the data by the ‘group’ column usinggroup_by(). * Then, we use thetop_n()` function to select the top 2 values within each group based on the ‘value’ column. * Finally, we print the resulting dataset containing the top N values by group.

Using data.tabledata.table is another popular package for efficient data manipulation, particularly with large datasets. To achieve the same task using data.table, we’ll use the by argument along with the .SD special symbol.

```

Load the data.table packagelibrary(data.table)# Convert data frame to data.tablesetDT(data)# Select top 2 values by grouptop_n_values <- data[, .SD[order(-value)][1:2], by = group]# View the resultprint(top_n_values)

```

group value <char> <num>1: A 202: A 153: B 304: B 25 Explanation* After loading the data.table package, we convert our data frame to a data.table using setDT(). * We then select the top 2 values within each group by ordering the data in descending order of ‘value’ and selecting the first 2 rows using [1:2]. * The by argument is used to specify grouping by the ‘group’ column. * Finally, we print the resulting dataset containing the top N values by group.

Using base RWhile dplyr and data.table are powerful packages for data manipulation, base R also provides functionality to achieve this task using functions like split() and lapply().

```

Example datasetdata <- data.frame( group = c(rep("A", 5), rep("B", 5)), value = c(10, 15, 8, 12, 20, 25, 18, 22, 17, 30))# Select top 2 values by group using base Rtop_n_values <- do.call(rbind, lapply(split(data, data$group), function(x) head(x[order(-x$value), ], 2)))# Convert row names to a columnrownames(top_n_values) <- NULL# View the resultprint(top_n_values)

```

group value1 A 202 A 153 B 304 B 25 Explanation* We start with our sample dataset. * Using split(), we split the dataset into subsets based on the ‘group’ column. * Then, we apply a function using lapply() to each subset, which sorts the values in descending order and selects the top 2 rows using head(). * The resulting subsets are combined into a single data frame using do.call(rbind, ...).

ConclusionIn this tutorial, we’ve covered three different methods to select the top N values by group in R using dplyr, data.table, and base R. Each approach has its advantages depending on the complexity of your dataset and your familiarity with the packages. I encourage you to try out these examples with your own data and explore further functionalities offered by these packages for efficient data manipulation. Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A Practical Guide to Selecting Top N Values by Group in R

View Details

[This article was first published on vgherard, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.About a month ago we finally managed to drop (Nini et al. 2024), “Authorship Verificationbased on the Likelihood Ratio of Grammar Models”, on the arXiv.Delving into topics such as authorship verification, grammar andforensics, was quite a detour for me, and I’d like to summarize heresome of the ideas and learnings I got from working with all this new andinteresting material.

The main qualitative idea put forward by Ref. (Nini et al. 2024) is that grammar is afundamentally personal and unique trait of an individual, thereforeproviding a sort of “behavioural biometric”. One first goal of this workwas to put this general principle under test, by applying it to theproblem of Authorship Verification (AV): the process of validatingwhether a certain document was written by a claimed author. Concretely,we built an algorithm for AV that relies entirely on the grammaticalfeatures of the examined textual data, and compared it with thestate-of-the-art methods for AV.

The results were very encouraging. In fact, our method actuallyturned out to be generally superior to the previous state-of-the-art onthe benchmarks we examined. This is a notable result, keeping also intoaccount that our method uses less textual information (only thegrammar part) than other methods to perform its inferences.

The algorithmI sketch here a pseudo-implementation of our method in R. For the fitof (k)-gram models and perplexitycomputations, I use my package {kgrams},which can be installed from CRAN. Model (hyper)parameters such as numberof impostors, order of the (k)-grammodels, etc. are hardcoded, see (Nini et al.2024) for details.

This is just for illustrating the essence of the method. Forpractical reasons, in the code chunk below I’m not reproducing thedefinition of the function extract_grammar(), which in ourwork is embodied by the POS-noise algorithm. This function shouldtransform a regular sentence, such as “He wrote a sentence”, to itsunderlying grammatical structure, say “[Pronoun] [verb] a [noun]”.

```

' @param q_doc character. Text document whose authorship is questioned.#' @param auth_corpus character. Text corpus of claimed author.#' @param imp_corpus character. Text corpus of impostors.score <- function(q_doc, auth_corpus, imp_corpus, n_imp = 100) { q_doc <- extract_grammar(q_doc) auth_corpus <- extract_grammar(auth_corpus) imp_corpus <- extract_grammar(imp_corpus) # Compute perplexity based on claimed author's language model. auth_mod <- train_language_model(auth_corpus) auth_perp <- kgrams::perplexity(q_doc, model = auth_mod) # Compute perplexity based on impostor language models. # # Each impostor is trained on a synthetic corpus obtained by sampling from # the impostor corpus the same number of sentences as the corpus of the # claimed author. n_sents_auth <- length(kgrams::tknz_sent(auth_corpus)) imp_corpus_sentences <- kgrams::tknz_sent(imp_corpus) imp_mod <- replicate(n_imp, { sample(imp_corpus_sentences, n_sents_auth) |> train_language_model() }) imp_perp <- sapply(imp_mod, (m) kgrams::perplexity(q_doc, model = m)) # Score is the fraction of impostor models that perform worse (higher # perplexity) than the proposed authors language model score <- mean(auth_perp < imp_perp) return(score)}train_language_model <- function(text){ text |> kgrams::kgram_freqs(N = 10, .tknz_sent = kgrams::tknz_sent) |> kgrams::language_model(smoother = "kn", D = 0.75)}extract_grammar <- identity # Just a placeholder - see above.

``` To be used as follows:

q\_doc <- "a a b a. b a. c b a. b a b. a." auth\_corpus <- "a a b a b. b c b. a b c a. b a. b c a." imp\_corpus <- "a a. b. a. b a. b a. c. a b a. d. a b. a d. a b a b c b a."set.seed(840)score(q\_doc, auth\_corpus, imp\_corpus)[1] 0.89 The “score” computed by this algorithm turns out to be a goodtruthfulness predictor for the claimed authorship, higher scores beingcorrelated with true attributions. If the impostor corpus isfixed once and for all, and if the pairs q_doc andauth_corpus are randomly sampled from a fixed jointdistribution, we can set a threshold for score in such a way that theattribution criterion score > threshold maximizes someobjective such as accuracy. This is, more or less, what we studiedquantitatively in our paper.

Reflections onin silico Authorship VerificationThe various ifs at the end of the previous sections led meto reflexionate on the applicability of machine-learning approaches,such as the one we discussed in our work, to real-life contexts.

As implied above, in order for a metric such as accuracy to representa sensible measure of predictive performance, we should be able toregard the AV problems encountered in our favorite practical use case asrandom samples from some fixed population. In other words, weconsider a stationary source of random authorship claims, and assumethat our trained model is routinely used to verify claims from thissource.

Now, while there are many circumstances in which the aboveassumptions make total sense, I think there are also interesting AVapplications in which one is not interested in the long-run propertiesof the method but, rather, in a single inference. The real case of thepoem “Shall I die?”, controversially attributed to Shakespeare in 1985,is an example of this kind. An approach to this case based on empiricalBayes is discussed in (Efron and Hastie 2021,vol. 6, sec. 6.2). Although we may be able to build a reasonableimpostor corpus to be used with this problem, it is not clear how oneshould come up with a relevant testing dataset of AV problems toempirically quantify uncertainty.

For cases such as the “Shall I die?” controversy, themachine-learning setting considered in our study is just an insilico model of real AV. As such, I believe it still providesuseful indications on what could be good authorship indicators and workin general, but we must acknowledge the practical limitations in our wayto quantify uncertainty. Other approaches, such as classical nullhypothesis testing, may be more suited to this specific kind of AVproblems.

Efron, Bradley, and Trevor Hastie. 2021. Computer Age StatisticalInference, Student Edition: Algorithms, Evidence, and Data Science.Vol. 6. Cambridge University Press.Nini, Andrea, Oren Halvani, Lukas Graner, Valerio Gherardi, and ShunichiIshihara. 2024. “Authorship Verification Based on the LikelihoodRatio of Grammar Models.” https://arxiv.org/abs/2403.08462.To leave a comment for the author, please follow the link and comment on their blog: vgherard.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Grammar as a biometric for Authorship Verification

View Details

[This article was first published on Rstats – quantixed, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Previously, I took advantage of a dataset that linked preprints to their published counterparts to look at the fraction of papers in a journal that are preprinted. This linkage can be used to answer other interesting questions. Such as: when do authors preprint their papers relative to submission? And does this differ by journal?

There’s a bit of preamble. If you just want to know the answer, click here. If you want to see the code, click here.

For each paper, we can extract from PubMed the “received” date and the “accepted” date. Because we have linked published papers to preprints, we also know the date when the preprint of the paper was first posted. Subtracting this date from the received date, we get something we’ll call “pretime”.

Now let’s plot the Pretime versus the Received to Accepted time.

In the plots above, we see ~3 years of a paper’s journey to acceptance. Let’s zoom in a bit to look at the first year.

What does this mean? To help interpret the plot, here’s a key:

There are four categories; manuscripts are posted to bioRxiv:

Prior to submission
Approximately at submission time
After submission
After acceptance

Note that we are looking at the final journal destination for each paper, which might not be the first place a paper is submitted. It’s likely that papers posted prior to submission, especially those with long pretimes, were submitted elsewhere first; rather than the authors posting their work early for the purpose of gathering feedback before a first submission. All journals have such papers, not just the sibling journals like Nature Communications and Cell Reports, which were created to capture papers following rejection from other titles.

The plots indicate that many papers are preprinted at the same time as submission. There are also a surprising number preprinted after submission. Very few preprints are posted after acceptance, for obvious reasons.

To simplify things, we can classify preprints with pretimes of -7 to 30 days as those papers preprinted at submission. Papers with less than are post-submission, those with more are pre-submission.

| Category | Papers | Percentage | | Pre-submission | 7025 | 33.9% | | On-submission | 8286 | 40.0% | | Post-submission | 5399 | 26.1% |

The answerThe analysis shows that generally, most authors preprint their work around the moment of submission.

Let’s look at how these fractions breakdown at each journal.

The fraction of papers preprinted upon submission is largest at several journals including Biochem J, Development, EMBO J etc. If we consider that many of the pre-submission preprints were posted around the submission time to a preceding journal, then preprinting upon submission is the most likely behaviour.

The fraction of papers posted after submission is a minority activity but it is sizeable at some journals, notably Nature Cell Biol and Dev Cell. Possible reasons why authors may only choose to post after submission (in some cases many months later) might include: a belief that preprinting may cause desk rejection, only preprinting after the paper has gone out to review, or authors getting twitchy about priority during a lengthy peer review process.

We can break down the data by year of publication to see that the patterns are fairly consistent over time.

CaveatsAny analysis like this is limited by the available data. First, the “received” date on PubMed may not be accurate. A journal may “reset the clock” on a submission and thereby make it appear that the preprint had been posted prior to submission when it may have actually been submitted to the publishing journal at the time of posting.

This analysis is also limited to:

papers that were preprinted on bioRxiv
papers for which we had complete data (the PubMed data is missing for some journals)
a subset of journals – other journal data can be retrieved by tweaking the code

To reiterate that the analysis is limited to papers where the authors actually posted a preprint. At many of the journals analysed here, over half of the authors still choose not to preprint their work!

The codeThis R script is quite long and has a few dependencies from my earlier post. Crunching through the xml files and through the bioRxiv dois to get the submission dates is sped up using parallel processing (on Mac/linux).

library(dplyr)library(ggplot2)library(doParallel)library(foreach)library(rbiorxiv)## some pre-requisite files required for this script# preprint - paper relationshipsdf\_all <- read.csv("Data/crossref-preprint-article-relationships-Aug-2023.csv")# code to extract data from Pubmed XML filessource("Script/pubmedXML.R")# previously downloaded Pubmed XML files in the Data directoryxml\_files <- list.files("Data", pattern = "*.xml", full.names = TRUE)# setup parallel backendcores <- detectCores()cl <- makeCluster(cores[1] - 1) #not to overload your computerregisterDoParallel(cl)pprs <- foreach(i = 1 : seq\_along(xml\_files), .combine = rbind) %dopar% { tempdf <- extract\_xml(xml\_files[i])}# stop clusterstopCluster(cl)# remove duplicatespprs <- pprs[!duplicated(pprs$pmid), ]# remove unwanted publication types by using a vector of stringsunwanted <- c("Review", "Comment", "Retracted Publication", "Retraction of Publication", "Editorial", "Autobiography", "Biography", "Historical", "Published Erratum", "Expression of Concern", "Editorial")# subset pprs to remove unwanted publication types using grepl, call this "pure"pure <- pprs[!grepl(paste(unwanted, collapse = "|"), pprs$ptype), ]# ensure that ptype contains "Journal Article"pure <- pure[grepl("Journal Article", pure$ptype), ]# remove papers with "NA NA" as the sole authorpure <- pure[!grepl("NA NA", pure$authors), ]# add factor column to pure that indicates if a row in pprs has a doi that is# also found in df\_all$article\_doipure$in\_crossref <- ifelse(tolower(pure$doi) %in% tolower(df\_all$article\_doi), "yes", "no")# lag timespure$recacc <- pure$accdate - pure$recdatepure$recpub <- pure$pubdate - pure$recdate# subset data for only in\_crossref == "yes"pure\_yes <- pure[pure$in\_crossref == "yes", ]# add column that has the preprint\_doi from df\_all where article\_doi matches doipure\_yes$preprint\_doi <- df\_all$preprint\_doi[match(tolower(pure\_yes$doi), tolower(df\_all$article\_doi))]# subset for biorxiv doi, i.e. starts "10.1101"pure\_yes <- pure\_yes[grepl("10.1101", pure\_yes$preprint\_doi), ]# if the preprint\_doi is longer than 15 characters, parse the date from the doi# and if it is less than 15 characters, set to NApure\_yes$date <- as.Date.numeric(ifelse(nchar(pure\_yes$preprint\_doi) < 16, NA, as.Date( substr(pure\_yes$preprint\_doi, 9, 18), format = "%Y.%m.%d")))# subset pure\_yes for date is NApure\_yes\_na <- pure\_yes[is.na(pure\_yes$date), ]# get the content of each preprint and assemble into large data frameregisterDoParallel(cl)preprints <- foreach(i = 1:nrow(pure\_yes\_na), .errorhandling = "pass", .multicombine = TRUE) %do% { temp <- NULL temp <- as.data.frame(biorxiv\_content(doi = pure\_yes\_na$preprint\_doi[i])) # subset to only include the doi, authors, title, and date; and first row only if (!is.null(temp)) { temp <- temp[1, c("doi", "authors", "title", "date")] } }stopCluster(cl)# the above code results in a large list of data frames, so we need to combine# them into one data frame. We didn't use .combine, because we wanted to remove# one or more of the preprints may have failed to download. The failed items do# not have 4 columns, so we can use ncol to check for thisncol\_preprints <- sapply(preprints, ncol)# write a for loop to start at the end of the list and remove the failed itemslist\_preprints <- preprintsfor (i in rev(seq\_along(list\_preprints))) { if (is.null(ncol\_preprints[[i]])) { list\_preprints <- list\_preprints[-i] }}df\_preprints <- do.call(rbind, list\_preprints)# add a column to pure\_yes\_na that has the date from df\_preprintspure\_yes\_na$date <- df\_preprints$date[match(tolower(pure\_yes\_na$preprint\_doi), tolower(df\_preprints$doi))]# if pure\_yes$date is NA, set to pure\_yes\_na$datepure\_yes\_all <- pure\_yespure\_yes\_all$date <- ifelse(is.na(pure\_yes\_all$date), as.Date(pure\_yes\_na$date[match(tolower(pure\_yes\_all$preprint\_doi), tolower(pure\_yes\_na$preprint\_doi))]), as.Date(pure\_yes\_all$date))# ensure date is as.Datepure\_yes\_all$date <- as.Date(pure\_yes\_all$date, format = "%Y-%m-%d")# find pretime by subtracting the date from the recdatepure\_yes\_all$pretime <- pure\_yes\_all$recdate - pure\_yes\_all$datepure\_yes\_all %>% filter(!is.na(pretime)) %>% ggplot(aes(x = as.numeric(recacc), y = as.numeric(pretime))) + geom\_abline(intercept = 0, slope = -1, linetype = "dashed", colour = "#a3a3a3") + geom\_point(colour = "#ae363b", shape = 16, size = 0.5, alpha = 0.2) + theme\_minimal(9) + lims(x = c(0, 1000), y = c(-1000, 1000)) + facet\_wrap( ~ journal) + labs(x = "Received to Accepted (days)", y = "Pretime (days)") + theme(legend.position = "none")ggsave("Output/Plots/pretime.png", width = 3000, height = 1500, dpi = 300, units = "px", bg = "white") pure\_yes\_all %>% filter(!is.na(pretime)) %>% ggplot(aes(x = as.numeric(recacc), y = as.numeric(pretime))) + geom\_abline(intercept = 0, slope = -1, linetype = "dashed", colour = "#a3a3a3") + geom\_point(colour = "#ae363b", shape = 16, size = 0.7, alpha = 0.2) + theme\_minimal(9) + lims(x = c(0, 365), y = c(-365, 365)) + facet\_wrap( ~ journal) + labs(x = "Received to Accepted (days)", y = "Pretime (days)") + theme(legend.position = "none")ggsave("Output/Plots/pretime\_1yr.png", width = 3000, height = 1500, dpi = 300, units = "px", bg = "white")# pure\_yes\_all contains the data of interest. Let's classify the papers# into three categories: 1) preprinted on submission, 2) preprinted after# submission, and 3) preprinted prior to submission# To classify them, group 1 is pretime of -7 to 30 days, group 2 is pretime# of greater than 31 days, and group 3 is pretime of less than -7 days# make a factor column to classify the paperspure\_yes\_all$preprint\_status <- ifelse(pure\_yes\_all$pretime >= 31, "Pre-submission", ifelse(pure\_yes\_all$pretime <= -7, "Post-submission", "On-submission"))# now summarise the fraction of papers at each journal that are in each categorysummary\_status <- pure\_yes\_all %>% filter(!is.na(pretime)) %>% group\_by(journal, preprint\_status) %>% summarise(papers = n()) %>% group\_by(journal) %>% mutate(fraction = papers / sum(papers))# order fraction so that post, on, pre submission are in the correct ordersummary\_status$preprint\_status <- factor(summary\_status$preprint\_status, levels = c("Pre-submission", "On-submission", "Post-submission"))# make a stacked bar chart to show the fraction of papers in each category# for each journal# Pre submission at the top, on submission middle and post submission at the bottomsummary\_status %>% ggplot(aes(x = journal, y = fraction, fill = preprint\_status)) + geom\_bar(stat = "identity", position = "stack") + scale\_x\_discrete(guide = guide\_axis(n.dodge = 2)) + theme\_minimal(9) + labs(x = "Journal", y = "Fraction of papers") + theme(legend.position = "right", legend.title = element\_blank()) + scale\_fill\_manual(values = c("#534666", "#138086", "#cd7672"))ggsave("Output/Plots/preprint\_status.png", width = 3000, height = 1500, dpi = 300, units = "px", bg = "white")# let's do the same again but only look at each journal and facet by yearpure\_yes\_all %>% filter(!is.na(pretime)) %>% filter(!year == "2024") %>% group\_by(journal, year, preprint\_status) %>% summarise(papers = n()) %>% group\_by(journal, year) %>% mutate(fraction = papers / sum(papers)) %>% ggplot(aes(x = year, y = fraction, fill = factor(preprint\_status, levels = c("Pre-submission", "On-submission", "Post-submission")))) + geom\_bar(stat = "identity", position = "stack") + theme\_minimal(9) + labs(x = "Journal", y = "Fraction of papers") + theme(legend.position = "right", legend.title = element\_blank()) + scale\_fill\_manual(values = c("#534666", "#138086", "#cd7672")) + facet\_wrap( ~ journal)ggsave("Output/Plots/preprint\_status\_facet.png", width = 3000, height = 1500, dpi = 300, units = "px", bg = "white")# generate summary stats for table (all papers with linked preprint)summary\_all <- pure\_yes\_all %>% filter(!is.na(pretime)) %>% group\_by(preprint\_status) %>% summarise(papers = n()) %>% mutate(fraction = papers / sum(papers)) —

The post title comes from “Prehistoric” by Circulatory System from their “Circulatory System” LP.

To leave a comment for the author, please follow the link and comment on their blog: Rstats – quantixed.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Prehistoric: when do authors preprint their papers?

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.#175–176

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #175Sometimes even common tables tell the story. Today we have table of people that are family, but we need to find their relationships. We only have first and last name and number of generation. So to find relations we need to make manouver called self-join. Let check it out.

(Note: there are some flaws in file so code has some cleaning fragments.)

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_175.xlsx", range = "A1:C16")test = read\_excel("Power Query/PQ\_Challenge\_175.xlsx", range = "E1:H19") %>% mutate(Relantionship = str\_remove\_all(Relantionship, " ")) # cleaned for purpose of validation Transformation result = input %>% left\_join(input, by = c("Family" = "Family")) %>% filter(`Generation No.x` == `Generation No.y` - 1) %>% # there is mispronunciation in the challenge, it should be "Relationship" not "Relantionship" unite("Relantionship", `Generation No.x`, `Generation No.y`, sep = "-") %>% select(Name = `Name.x`,Family,`Next Generation` = `Name.y`, Relantionship ) %>% arrange(Family, Relantionship , Name, `Next Generation`) Validation identical(result, test)# [1] TRUE Puzzle #176Probably somebody make a note by hand, not thinking of further users and as usually we need to straight issue up. From data concatenated in two columns of table we need to make long table with running sum per group. Go on, lets do it.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Power Query/PQ\_Challenge\_176.xlsx", range = "A1:C5")test = read\_excel("Power Query/PQ\_Challenge\_176.xlsx", range = "E1:G9") Transformation result = input %>% mutate(Column1 = map(Column1, ~strsplit(.x, ", ")), Column2 = map(Column2, ~strsplit(.x, ", "))) %>% unnest(cols = c(Column1, Column2)) %>% mutate(Column1 = map(Column1, ~tibble(Column1 = .x)), Column2 = map(Column2, ~tibble(Column2 = .x))) %>% mutate(n1 = map\_dbl(Column1, ~nrow(.x)), n2 = map\_dbl(Column2, ~nrow(.x))) %>% mutate(Column = map2(Column1, Column2, ~{ n1 = nrow(.x) n2 = nrow(.y) if (n1 > n2) { .y = bind\_rows(.y, tibble(Column2 = rep("0", n1 - n2))) } else if (n1 < n2) { .x = bind\_rows(.x, tibble(Column1 = rep(NA, n2 - n1))) } bind\_cols(.x, .y) })) %>% select(Group, Column) %>% unnest(cols = c(Column)) %>% drop\_na() %>% mutate(Column2 = cumsum(as.numeric(Column2)), .by = Group) Validation identical(result, test)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

PowerQuery Puzzle solved with R was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: PowerQuery Puzzle solved with R

View Details

[This article was first published on R - datawookie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The {emayili} package has adapters which make it simple to send email via a variety of services. For example, it caters specifically for ZeptoMail, MailerSend, Mailfence and Sendinblue. The latest version of {emayili}, 0.8.0 published on 23 April 2024, adds an an adapter for Mailtrap.

Account SetupBefore you can send emails via Mailtrap you will need to craete an account. Mailtrap offers both an API and SMTP servers. Finding the SMTP credentials on Mailtrap is not entirely simple. Well, it wasn’t for me. Select Sending Domains in the sidebar and then open the SMTP/API Settings tab.

On that page you’ll have access to the credentials for the transactional and bulk message streams.

There’s also a testing sandbox, and you can find the credentials for that by selecting Email Testing and Inboxes in the sidebar. Then open the SMTP Settings tab and choose one of the integrations. With the Postfix integration selected it’s simple to find the username and password.

Sending Mail via MailtrapLoad {emayili} and create a Mailtrap SMTP server object. I’m storing my Mailtrap username and password in environment variables MAILTRAP_USERNAME and MAILTRAP_PASSWORD.

library(emayili)MAILTRAP\_USERNAME <- Sys.getenv("MAILTRAP\_USERNAME")MAILTRAP\_PASSWORD <- Sys.getenv("MAILTRAP\_PASSWORD")smtp <- mailtrap( username = MAILTRAP\_USERNAME, password = MAILTRAP\_PASSWORD) Now create a message object. The address specified in the From field needs to be the same one that was registered with Mailtrap (see Account Setup above).

MAILTRAP\_ADDRESS <- Sys.getenv("MAILTRAP\_ADDRESS")msg <- envelope( to = "alice@gmail.com", from = MAILTRAP\_ADDRESS, subject = "Test") %>% text("Hello there!") Now send that message.

smtp(msg, verbose = TRUE) Sandbox & Bulk MessagingYou can use the Sandbox and Bulk Messaging features by activating the sandbox and bulk arguments.

```

Sandbox SMTP server.sandbox <- mailtrap(sandbox = TRUE, ...)# SandBulk Messaging SMTP server.sandbox <- mailtrap(bulk = TRUE, ...)

`` If you are using another email service and you’d like it to be specifically supported by{emayili}`, just create an issue and I’ll get it sorted.

To leave a comment for the author, please follow the link and comment on their blog: R - datawookie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {emayili} Support for Mailtrap

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Puzzles no. 434–438

PuzzlesAuthor: ExcelBI

All files (xlsx with puzzle and R with solution) for each and every puzzle are available on my Github. Enjoy.

Puzzle #434Sometimes challenges we are facing have not really much with real world problems. And no. 434 is one of them. Our task this time was to make matrix 20×5 with names of columns of Excel spreadsheet. But if they were in normal order it would be to easy. We had to make consecutively longer jumps, firstly by one column, then skip one and jump two, and then increase skipped columns each time. Wow, that was a thing. Look yourself.

Loading libraries and data library(tidyverse)library(stringi)library(readxl)test = read\_excel("Excel/434 Generate the Column Headers Matrix.xlsx", range = "A2:E21", col\_names = FALSE) %>% as.matrix() Transformation col\_names = c(LETTERS, do.call(paste0, expand.grid(LETTERS, LETTERS)), do.call(paste0, expand.grid(LETTERS, LETTERS, LETTERS))) %>% map\_chr(~stri\_reverse(.)) columns = data.frame(cols = col\_names) %>% mutate(indices = 1:nrow(.)) index <- accumulate(1:99, ~ .x + .y, .init = 1)result\_df = columns %>% filter(indices %in% index) %>% pull(cols)result = matrix(result\_df, nrow = 20, ncol = 5, byrow = FALSE) Validation all.equal(result, test, check.attributes = FALSE)# [1] TRUE Puzzle #435And again… Task for showing off skill, not something really useful in analytics. But of course we love showing off. So let draw boat with characters on a matrix.

Transformation M = matrix(NA, nrow = 12, ncol = 23)for (i in 1:7) { M[i, ] = c(rep(NA, (23 - 2*i + 1)/2), rep('+', 2*i - 1), rep(NA, (23 - 2*i + 1)/2))}for (i in 8) { M[i, ] = c(rep(NA, (23 - 2*i + 1)/2), rep('=', 2*i - 1), rep(NA, (23 - 2*i + 1)/2))}for (i in 9:12) { M[i, ] = c(rep(NA, i - 9), rep('x', 23 - 2*(i - 9)), rep(NA, i - 9))}as.data.frame(M) Puzzle #436Finally some numbers to play with. Are you familiar with pandigital numbers? I’ve met them for the first time too. They are numbers that if they had 3 digits, consists of only 1, 2 and 3 (in various orders). So for n digits there would always go numbers from 1 to n without repetitions. But we have one more twist here. We need to generate sequence of first 100 pandigitals that are also primes. So little bit bruteforcely — I generated all pandigitals up to seven digits and then filtered only primes. Check it out.

Loading libraries and data library(tidyverse)library(readxl)library(primes)library(gtools)test = read\_excel("Excel/436 Pandigital Primes.xlsx", range = "A1:A101") Transformation generate\_pandigital = function(n) { digits = 1:n digits = permutations(n,n) digits = apply(digits, 1, function(x) as.numeric(paste(x, collapse = ""))) return(digits)}df = data.frame(numbers = NA)for (i in 1:7) { pandigitals = generate\_pandigital(i) df = rbind(df, data.frame(numbers = pandigitals)) }result = df %>% mutate(is\_prime = map\_lgl(numbers, is\_prime)) %>% filter(is\_prime) %>% head(100) Validation identical(result$numbers, test$`Answer Expected`)# [1] TRUE Puzzle #437Bifid ciphering is back, but with twist. This time our encoding process needs to include keyword which letters are shifting coding square. But it was not really hard and was mainly about adjusting code from puzzle #432 from last episode.

Loading libraries and data library(tidyverse)library(readxl)input = read\_excel("Excel/437 Bifid Cipher\_Part 2.xlsx", range = "A1:B10")test = read\_excel("Excel/437 Bifid Cipher\_Part 2.xlsx", range = "C1:C10") Transformation create\_coding\_square <- function(keyword) { p1 = str\_split(keyword %>% str\_replace(pattern = "j", replacement = "i"), "")[[1]] %>% unique() p2 = setdiff(letters, c("j", p1)) Letters = c(p1, p2) df = as.data.frame(matrix(Letters, nrow = 5, byrow = TRUE)) %>% pivot\_longer(cols = everything()) %>% mutate(column = as.numeric(str\_extract(name, "[0-9]+")), row = rep(1:5,each = 5)) %>% select(-name) return(df)}bifid\_encode = function(text, keyword) { coding\_square = create\_coding\_square(keyword) text = str\_replace\_all(text, "J", "I") chars = str\_split(text, "")[[1]] coords = map\_dfr(chars, function(char) { coords = coding\_square %>% filter(value == char) %>% select(row, column) return(coords) }) coords = paste0(coords$row, coords$column) %>% str\_split("", simplify = TRUE) %>% as.numeric() %>% matrix(ncol = 2, byrow = TRUE) %>% as.data.frame() encoded = coords %>% left\_join(coding\_square, by = c("V1" = "row", "V2" = "column")) %>% pull(value) %>% paste0(collapse = "") return(encoded)}result = input %>% mutate(`Answer Expected` = map2\_chr(`Plain Text`,Keywords, bifid\_encode)) %>% select(`Answer Expected`) Validation identical(result, test)# [1] TRUE Puzzle #438And another part of electrical riddle. And again we are basing on one of previous tasks (this time #420). Basing on colourful bands on resistors we have to calculate their resistance. But it was previous task. Today’s addition is to change notation of numbers (do not worry, still in decimals), for them to have Kilo Ohms, Mega Ohms and Giga Ohms. We needed some adjustments and one new function. Check all of this code.

Loading libraries and data library(tidyverse)library(readxl)input1 = read\_excel("Excel/438 Resistor Value\_v2.xlsx", range = "A1:C11")input2 = read\_excel("Excel/438 Resistor Value\_v2.xlsx", range = "E1:E10")test = read\_excel("Excel/438 Resistor Value\_v2.xlsx", range = "F1:F10") Transformation find\_resistance = function(bands, input) { codes = input pairs = strsplit(bands, "")[[1]] pairs = matrix(pairs, ncol = 2, byrow = TRUE) %>% as.data.frame() %>% unite("pair", V1, V2, sep = "") %>% left\_join(codes, by = c("pair" = "Code")) %>% mutate(nr = rev(row\_number())) last = pairs[nrow(pairs),] %>% mutate(res = 10^Value) %>% pull(res) pairs\_wol = pairs[-nrow(pairs),] %>% mutate(res = Value*10^(nr-2)) %>% pull(res) final\_res = sum(pairs\_wol) * last return(final\_res)}convert\_to\_notation = function(x) { case\_when( x >= 1e9 ~ paste0(x/1e9, " G Ohm"), x >= 1e6 ~ paste0(x/1e6, " M Ohm"), x >= 1e3 ~ paste0(x/1e3, " K Ohm"), TRUE ~ paste0(x, " Ohm"))}result = input2 %>% mutate(`Answer Expected` = map\_dbl(`Color Bands`, find\_resistance, input1)) %>% mutate(`Answer Expected` = map\_chr(`Answer Expected`, convert\_to\_notation)) Validation identical(result$`Answer Expected`, test$`Answer Expected`)# [1] TRUE Feel free to comment, share and contact me with advices, questions and your ideas how to improve anything. Contact me on Linkedin if you wish as well.

R Solution for Excel Puzzles was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Solution for Excel Puzzles

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Join our workshop on Conducting Simulation Studies in R, which is a part of our workshops for Ukraine series!

Here’s some more info:

Title: Conducting Simulation Studies in R

Date: Thursday, May 23rd, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Greg Faletto is a statistician and data scientist at VideoAmp, where he works on causal inference. Greg completed his Ph.D. in statistics at the University of Southern California in 2023. His research focused on developing machine learning methods has been published in venues like the International Conference on Machine Learning and the Proceedings of the National Academy of Sciences. Greg has taught classes at USC on data science and communicating insights from data, and he has previously presented his research and led workshops at venues including USC, the University of California San Francisco, the University of Copenhagen, Data Con LA, and IM Data Conference.

Description: In simulation studies (also known as Monte Carlo simulations or synthetic data experiments), we generate data sets according to a prespecified model, perform some calculations on each data set, and analyze the results. Simulation studies are useful for testing whether a methodology will work in a given setting, assessing whether a model “works” and diagnosing problems, evaluating theoretical claims, and more. In this workshop, I’ll walk through how you can use the R simulator package to conduct simple, reproducible simulation studies. You’ll learn how to carry out the full process, including making plots or tables of your results.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/4aD5LMC or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Conducting Simulation Studies in R workshop was first posted on April 22, 2024 at 3:00 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Conducting Simulation Studies in R workshop

View Details

[This article was first published on R-Bloggers – Learning Machines, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
It’s a widely accepted notion that money influences happiness, a concept famously associated with Noble laureate Daniel Kahneman, who purportedly demonstrated that emotional wellbeing increases with income but plateaus beyond an annual threshold of about $75,000.

This idea has permeated both academic circles and popular media, reinforcing the belief that there’s a direct correlation between financial prosperity and happiness. But how accurate is this belief when we scrutinize the data more closely? To find out read on!

Recent research (indeed the last paper ever published by Kahneman!) attempts to delve deeper into this relationship, suggesting that the connection between income and happiness is real and relevant. However, a closer examination of the publicly available data tells a different story.

My own analysis reveals a Pearson correlation coefficient of just 0.07 between wellbeing and income, indicating a very weak relationship despite its statistical significance:

data <- read.csv("Data/Income\_and\_emotional\_wellbeing\_a\_conflict\_resolved.csv") # modify path accordinglydata$income\_factor <- factor(data$income, levels = sort(unique(data$income)))cor.test(data$wellbeing, data$income)## ## Pearson's product-moment correlation## ## data: data$wellbeing and data$income## t = 13.293, df = 33389, p-value < 2.2e-16## alternative hypothesis: true correlation is not equal to 0## 95 percent confidence interval:## 0.06187719 0.08321620## sample estimates:## cor ## 0.072555 One of the more subtle learnings of academic research is that a relationship between two variables can statistically be highly significant while in practice being useless because the effect is so minuscule. Paradoxically, the more data points you have the higher the chances that you will find something statistically significant that has no practical significance.

In this case we have more than 33,300 data points and while there is a tiny increase in happiness with greater income, the effect is so slight that its real-world implications are negligible. Indeed, the difference between the medians of happiness at household incomes of $15,000 and $250,000 is only about five points on a 100-point scale!

To put the observed effect into a more relatable context, consider this: the difference in happiness resulting from an approximately fourfold difference in income is roughly equivalent to the happiness boost one might feel over a typical weekend! This comparison starkly illustrates the insignificance of income effects relative to everyday life experiences.

Yet this research manages to persuade us that there is indeed something substantial going on. How is this achieved in such studies? The devil is in the details, or in this case, the methodology. Three statistical choices in such studies stand out as particularly problematic: the logarithmic transformation of income, the use of z-scores and the use of averages without referring to dispersion measures for wellbeing:

Logarithmic transformation: By transforming income using a logarithmic scale, the data suggest a linear relationship where none exists. This transformation masks the reality of diminishing returns, where increases in income result in progressively smaller gains in happiness. These methods, while often applied in practice for reducing skewness, can present a distorted view of the underlying data. Apart from that, how should one interpret “log income” anyway?
Z-scores: The application of z-scores is another area where the graphical representation can be misleading. Z-scores standardize data points and effectively cut off the y-axis of the original data, which can visually exaggerate minor differences. When we depict wellbeing scores on a complete 0-100 scale, the supposed effect of income on happiness nearly vanishes, revealing a much less compelling story.
Averages without dispersion measures: While using median values (or means) is not problematic per se, it can mask the inherent dispersion of the data, e.g. to be indicated by interquartile ranges (IQR), standard deviations, variances, or confidence intervals. Especially when data is extremely dispersed, as in this case with wellbeing, interpreting results and drawing meaningful conclusions can be challenging without proper context.

The plots I’ve created from the original data starkly illustrate these points. I often start my own data analyses with a scatter plot but in this case, I first thought that I made a mistake or got the wrong data:

plot(data$wellbeing ~ data$income, main = "Scatterplot of Wellbeing Across Income Levels", xlab = "Income", ylab = "Wellbeing")grid() This plot shows a dense cluster of data points that scatter broadly across the graph, displaying no apparent trend or meaningful pattern linking income to wellbeing.

But worry not, by making use of the three statistical techniques from above, it is quite easy to create plots like the ones shown in the pertinent literature:

mean\_well\_being\_zscore <- aggregate(scale(wellbeing) ~ log\_income, data = data, median)plot(mean\_well\_being\_zscore, main = "Z-Score of Median of Wellbeing Across Log-Income Levels", xlab = "Income", ylab = "Wellbeing", pch = 16)grid()LinReg <- lm(V1 ~ log\_income, data = mean\_well\_being\_zscore)LinReg## ## Call:## lm(formula = V1 ~ log\_income, data = mean\_well\_being\_zscore)## ## Coefficients:## (Intercept) log\_income ## -1.08108 0.09396abline(LinReg) Mirroring Figure 1B from the above paper, this plot suggests a clear relationship between rising levels of income and resulting wellbeing. Alas, upon closer scrutiny, this proves to be more a product of clever statistical handling than any relevant effect.

Now let us have a look at some less sophisticated plots to see what is really going on… or better, “isn’t going on”. We recreate the same plot but this time without the log transformation of income and without z-scoring the wellbeing values:

mean\_well\_being <- aggregate(wellbeing ~ income, data = data, median)plot(mean\_well\_being, main = "Median of Wellbeing Across Income Levels", xlab = "Income", ylab = "Wellbeing", pch = 16)grid()

plot(mean\_well\_being, main = "Median of Wellbeing Across Income Levels", xlab = "Income", ylab = "Wellbeing", ylim = c(0, 100), pch = 16)grid() The first version of this plot artificially caps the y-axis to suggest a strong, meaningful relationship; the second version showing the full axis demonstrates the near absence of any real effect. In this case the manipulation of the y-axis becomes more apparent because we can now see the real wellbeing values instead of the hard to interpret z-scored ones.

And these plots do not deceive the eye, e.g. nearly doubling one’s income from $35,000 to $65,000 shows not even a statistically significant difference in the level of wellbeing:

t.test(data$wellbeing[data$income == 35000], data$wellbeing[data$income == 65000])## ## Welch Two Sample t-test## ## data: data$wellbeing[data$income == 35000] and data$wellbeing[data$income == 65000]## t = -1.8237, df = 4923.9, p-value = 0.06825## alternative hypothesis: true difference in means is not equal to 0## 95 percent confidence interval:## -1.32549954 0.04789018## sample estimates:## mean of x mean of y ## 62.37602 63.01482 Or even more extreme, the effect of more than quadrupling one’s income from $137,500 to $625,000 isn’t statistically significant either:

t.test(data$wellbeing[data$income == 137500], data$wellbeing[data$income == 625000])## ## Welch Two Sample t-test## ## data: data$wellbeing[data$income == 137500] and data$wellbeing[data$income == 625000]## t = -1.6919, df = 533.15, p-value = 0.09125## alternative hypothesis: true difference in means is not equal to 0## 95 percent confidence interval:## -2.4851637 0.1852509## sample estimates:## mean of x mean of y ## 64.19116 65.34111 In the last chart we now also add a measure of dispersion in the form of boxplots. Here it becomes even more clear that across income levels, wellbeing hardly changes and is extremely dispersed at that:

boxplot(wellbeing ~ sort(income\_factor), data = data, main = "Boxplot of Wellbeing Across Income Levels", xlab = "Income", ylab = "Wellbeing", col = rainbow(length(unique(data$income)))) To be fair, the authors do briefly address some of these criticisms, but those discussions are buried deep within the paper and serve mainly to downplay their significance. It’s important to remember that “lying with statistics” doesn’t necessarily involve outright falsehoods; rather, it involves presenting results in a way that suggests misleading conclusions or exaggerates irrelevant findings.

As a sidenote, the possibility of reverse causality — where inherently happier individuals might earn more — should also be considered. It suggests that personal disposition (what self-proclaimed “life-coaches” call “mindset” nowadays!) might drive both happiness and higher income rather than the reverse. It would be interesting to see the results if you reversed both variables. Moreover, it would be insightful to examine how changes in income levels affect wellbeing, as the current research only addresses the wellbeing of individuals at their existing income levels.

To conclude, this critique is not just about debunking a popular myth; it’s a call for greater integrity and clarity in how statistical research is conducted and reported. I would be very interested in your feedback and in whether you have encountered similar overstatements in other research.

To leave a comment for the author, please follow the link and comment on their blog: R-Bloggers – Learning Machines.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Can Money Really Buy Happiness? Or How to Lie with Statistics in Science

View Details

[This article was first published on R on BitFoam, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. A lot has happening in Monster Maze for the last months and a new chapter has now opened.

Wanna play?Play hereCodeThe code is publicly available in Github:

monster-maze-shiny-mobile/tree/from_emojis_to_imagesTime-keepingIn the prior version of the game, time-keeping was turn-based and sequential, I-Go-You-Go. And the speed of any change in the game referred to the player´s speed. Simple put, things took a number of player´s moves to change or happen, most notably, the ghosts appearing and disappearing and the zombies moving.

Now time progresses according to the game clock and the speed of the characters is based on the clock. This is arguably when it comes to the player though, in this case her speed is limited by the player herself and the response time of the game, meaning that is not capped to any specific value (Wikipedia contributors (2023)).

In this case the time is not continuous and passes in discrete units of 1 sec (1000 ms). This can be easily achieved through the function reactiveTimer that invalidates the context next time the interval elapses.

WARNING I haven´t settled down on a naming convention yet, moving between camelCase and snake_case, it is difficult to stick to one when the packages I use don’t do it.

A few additional reactive values are required to harness the power of time.

timerRunning: this flag allows to pause the master clock.
afterIntro: this flag indicates the tipping event when game time starts.
ghostTimer and zombieTimer are used to determine ghost and zombie speeds based on the master clock.

1 #timer2 autoInvalidate <- reactiveTimer(1000)3 timerRunning = reactiveVal(NULL)4 afterIntro = reactiveVal(NULL)5 ghostTimer = reactiveVal(0)6 zombieTimer = reactiveVal(0) The observeEvent function that initializes the values has changed slightly to accommodate the changes.

1 # set the initial values 2 observeEvent(TRUE, ignoreNULL = FALSE, ignoreInit = FALSE, once = TRUE, { 3 game\_info$scene = "intro" 4 renderEvent(TRUE) 5 hide\_action\_buttons() 6 timerRunning(FALSE) 7 afterIntro(FALSE) 8 console$data <- "Nobody believes me. 9There is no hope for them...10...unless I do it myself.11I couldn´t look myself 12in the mirror.13I have to go back 14and save their souls.15"16 }) A new Observe function is invoked every time the reactiveTimer invalidates the context. This function represents the master clock.

1 2 observe({ 3 # Invalidate and re-execute this reactive expression every time the 4 # timer fires. 5 autoInvalidate() 6 if (timerRunning()) { 7 isolate(level\_timer(level\_timer() + 1)) 8 isolate(ghostTimer(ghostTimer() + 1)) 9 isolate(zombieTimer(zombieTimer() + 1))10 isolate(move\_monsters())11 }12 }) And the motion of Ghosts and Zombies is now controlled by the following snippets that check their corresponding clocks against their speeds.

1 2 #ghosts move according to ghost speed 3 if (ghostTimer() == ghost\_speed()) { 4 occupied\_positions <- append(zombie\_positions(), 5 get\_positions\_nearby(maze = maze(), 6 this\_position = player\_position(), 7 radius = 1)) 8 ghost\_positions(get\_random\_free\_positions( 9 maze = maze(), num = num\_ghosts(), 10 occupied\_positions = occupied\_positions))11 ghost\_moves(ghost\_moves() + 1)12 ghostTimer(0)13 } Ghosts still disappear and appear on free positions. I have changed their action radius so that now the player has to bump into them to trigger the shuffle.

1 2 #zombies move according to zombie speed 3 if (zombieTimer() == zombie\_speed()) { 4 zombie\_positions(move\_zombies( 5 maze=maze(), 6 zombie\_positions = zombie\_positions(), 7 other\_positions = append(ghost\_positions(),zombie2\_positions()), 8 player\_position = player\_position())) 9 zombie2\_positions(move\_zombies(10 maze=maze(),11 zombie\_positions = zombie2\_positions(),12 other\_positions = append(ghost\_positions(),zombie\_positions()), 13 player\_position = player\_position()))14 zombie\_moves(zombie\_moves() + 1)15 zombieTimer(0)16 } As before, the zombies move towards the player. Considering to limit the distance within which they can sense the player and be aware of her presence. They could stay still or move randomly till her presence is detected.

Image processing and resourcesThe R package magick for advanced image processing (Ooms (2023)) and the website OpenGameArt.Org have been a galvanizing discovery that has lead to this set of changes and finally moving away from emojis to images.

I came across some amazing 16x16 and 32x32 tiles for the mazes and sprites and a few transformations (crop, scale and animation) from the magick package got me the gifs to render a more dynamic scene.

basictiles.pngcharacters 1library(magick)23tiles <- image\_read("images/characters.png")4char1 <- image\_scale(image\_crop(tiles,"16x16+48+48"),"32x32")5char2 <- image\_scale(image\_crop(tiles,"16x16+64+48"),"32x32")6char3 <- image\_scale(image\_crop(tiles,"16x16+80+48"),"32x32")7char\_ani <- image\_animate(c(char1,char2,char3, char2), fps=4, loop=0, dispose="background")8image\_write(char\_ani,"images/char.gif") Player 1host1 <- image\_scale(image\_crop(tiles,"16x16+112+0"),"32x32")2host3 <- image\_scale(image\_crop(tiles,"16x16+96+16"),"32x32")3host5 <- image\_scale(image\_crop(tiles,"16x16+128+32"),"32x32")4host\_ani <- image\_animate(c(host1,host3,host1,host5), fps=1, loop=0, dispose="background")5image\_write(host\_ani,"images/hostage.gif") Hostage5ZombieSpriteSheet 1tiles <- image\_read("images/5ZombieSpriteSheet.png")2z1 <- image\_scale(image\_crop(tiles,"36x36+0"),"32x32")3z2 <- image\_scale(image\_crop(tiles,"36x36+44"),"32x32")4z3 <- image\_scale(image\_crop(tiles,"36x36+90"),"32x32")5zombie\_ani <- image\_animate(c(z1,z2,z3), fps=4, dispose="background")6image\_write(zombie\_ani, "images/zombie\_down.gif") ZombieSimilarly, the mazes have improved significantly. However, I keep the first person viewpoint. The player doesn’t turn, the wold spins around her but too fast to notice :D.I have to admit that in a 2D, topview game; it may not be the best fit but I want to push this still a bit further.

Level 9New chapter (rescue mission)The game is shaping itself into a horror survival plot, still working on it, aiming for a simple plot where puzzles, rescuing hostages and more collectables and monsters can easily fit in.

And more subtle changes improved audios, the audio should now work in Safari as well. * finally same presentation in iphone and android. * coins as first collectable, not sure if they will stay though. * added some tests, tried testthat, not great as the game is not a package. * And a suprise if the player takes too long to rescue the hostages…. ReferencesOoms, Jeroen. 2023. Magick: Advanced Graphics and Image-Processing in r. https://CRAN.R-project.org/package=magick.Wikipedia contributors. 2023. “Timekeeping in Games — Wikipedia, the Free Encyclopedia.” https://en.wikipedia.org/w/index.php?title=Timekeeping_in_games&oldid=1186332126. To leave a comment for the author, please follow the link and comment on their blog: R on BitFoam*.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: More Monster Maze

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Dear rOpenSci friends, it’s time for our monthly news roundup!

You can read this post on our blog.Now let’s dive into the activity at and around rOpenSci!

rOpenSci HQCoworkingRead all about coworking!

Tuesday, May 7th, 09:00 Americas Pacific (16:00 UTC), Meet rOpenSci’s new Executive Director! With cohost Noam Ross and Steffi LaZerte.
- Learn more about rOpenSci!
- Explore rOpenSci packages and contributing guide
- Ask questions or chat about rOpensci with Noam and other attendees.
- Tuesday, June 4th, 09:00 Australia Western (01:00 UTC), Theme TBD with cohost TBD and Steffi LaZerte.And remember, you can always cowork independently on work related to R, work on packages that tend to be neglected, or work on what ever you need to get done!

R-Universe documentation gets a boost from Google Season of DocsWe are excited and grateful to announce that R-Universe has been awarded a Google Season of Docs grant. R-Universe is rOpenSci’s platform for testing, building, distributing, and discovering R packages, led by Jeroen Ooms.

With the support of Google, we aim to update and centralize our documentation, showcasing how users and developers can get the most out of R-Universe. We will update the documentation to catch up with the rapid feature development of the past two years, including consolidating our many tech notes and READMEs into an easily discoverable site. We will build the site with Quarto to make it straightforward for R community members to contribute.

The work will be led by Maëlle Salmon.

Learn about a different package every day by following our social media campaignsThe goal of a community is to collaborate, share knowledge and build learnings. That’s why we want to show you the more than 300 packages in our federated community of open-source software developers.

Since December 2023, we have run the social media campaigns “A Package a Day on Mastodon, in which we feature one package each day. On LinkedIn, we publish our Package Weekly Digest every Monday with a list of five packages that belong to the same category.

You can follow us and the hashtags #APackageADay and #PackageWeeklyDigest to stay informed about rOpenSci R packages and learn about them.

Help us put together the rOpenSci Community Call calendar for the year!We are organizing the Community Calls for this year. Could you help us decide which topics we should cover?

We hold Community Calls to share knowledge that is relevant to our community and consistent with our vision and mission. These are free and open for anyone to attend and provide opportunities for us to connect with rOpenSci community members around the world.We’ve opened an issue in this repository for each topic we’re considering. We’d like your input and “votes” on these, and your suggestions for other topics we haven’t thought of.

We would love to hear your ideas.

Software New packagesThe following two packages recently became a part of our software suite, or were recently reviewed again:

baRulho, developed by Marcelo Araya-Salas: Intended to facilitate acoustic analysis of (animal) sound transmission experiments, which typically aim to quantify changes in signal structure when transmitted in a given habitat by broadcasting and re-recording animal sounds at increasing distances. The package offers a workflow with functions to prepare the data set for analysis as well as to calculate and visualize several degradation metrics, including blur ratio, signal-to-noise ratio, excess attenuation and envelope correlation among others (Dabelsteen et al 1993 https://doi.org/10.1121/1.406682). It is available on CRAN. It has been reviewed by Mike Mahoney and Dena J. Clink.
fellingdater, developed by Kristof Haneca: fellingdater is an R package that aims to facilitate the analysis and interpretation of tree-ring data from wooden cultural heritage objects and structures. The package standardizes the process of computing and combining felling date estimates, both for individual and groups of related tree-ring series. It has been reviewed by Antonio Jesus Pérez-Luque and Nicholas John Tierney.

Discover more packages, read more about Software Peer Review.

New versionsThe following seventeen packages have had an update since the last newsletter: frictionless (v1.1.0), beautier (v2.6.12), bib2df (v1.1.2.0), cffr (v1.0.1), crul (v1.4.2), DataPackageR (v0.15.9), dbparser (v2.0.3), dittodb (v0.1.8), eia (v0.4.2), fellingdater (v1.0.0), jagstargets (1.2.0), melt (v1.11.3), riem (v0.3.1), spatsoc (v0.2.3), stantargets (0.1.1), tarchetypes (0.9.0), and targets (1.7.0).

Software Peer ReviewThere are thirteen recently closed and active submissions and 6 submissions on hold. Issues are at different stages:

One at ‘6/approved’:
- fellingdateR, Estimate, report and combine felling dates of historical tree-ring. Submitted by KriHa.
- Two at ‘5/awaiting-reviewer(s)-response’:
- dendroNetwork, Create networks of dendrochronological series using pairwise similarity. Submitted by RonaldVisser.
- mregions2, Access Data from Marineregions.org: The Marine Regions Gazetteer and the Marine Regions Data Products. Submitted by salvafern.
- Four at ‘4/review(s)-in-awaiting-changes’:
- rix, Rix: Reproducible Environments with Nix. Submitted by Bruno Rodrigues.
- karel, Learning programming with Karel the robot. Submitted by Marcos Prunello.
- rgeeExtra, Extensions for rgee. Submitted by Cesar Aybar.
- agromet, Índices y Estadísticos Climáticos e Hidrológicos. Submitted by Paola Corrales.
- Two at ‘3/reviewer(s)-assigned’:
- rOPTRAM, Derive soil moisture using the OPTRAM algorithm. Submitted by Micha Silver.
- fwildclusterboot, Fast Wild Cluster Bootstrap Inference for Linear Models. Submitted by Alexander Fischer. (Stats).
- Two at ‘2/seeking-reviewer(s)’:
- cancerprof, API Client for State Cancer Profiles. Submitted by Brian Park.
- rsi, Efficiently Retrieve and Process Satellite Imagery. Submitted by Michael Mahoney.
- Two at ‘1/editor-checks’:
- osmapiR, OpenStreetMap API. Submitted by Joan Maspons.
- gigs, Assess Fetal, Newborn, and Child Growth with International Standards. Submitted by Simon Parker. (Stats).

Find out more about Software Peer Review and how to get involved.

On the blog* From the Founding Director: My Farewell to rOpenSci by Karthik Ram. Karthik Ram steps down as Executive Director after 13 years. Other languages: Del Director Fundador: Mi despedida de rOpenSci (es). * Hello from our New Executive Director! by Noam Ross. Noam Ross takes the helm as rOpenSci’s new Executive Director. Other languages: Hola de nuestro nuevo Director Ejecutivo (es). * R-Universe Documentation Gets a Boost from Google Season of Docs by Noam Ross. We are excited to announce that R-Universe has been awarded a Google Season of Docs. * Fostering Equity and Leadership: the rOpenSci Champions Program Selection Process by Francisco Cardozo, Yanina Bellini Saibene, Camille Santistevan, and Lou Woodley. The rOpenSci Champions Program fosters inclusivity and diversity in R and open science, emphasizing community engagement and project innovation through a detailed selection process.

Calls for contributionsCalls for maintainersIf you’re interested in maintaining any of the R packages below, you might enjoy reading our blog post What Does It Mean to Maintain a Package?.

internetarchive, an API Client for the Internet Archive. Issue for volunteering.
historydata, datasets for historians. Issue for volunteering.
textreuse, detect text reuse and document similarity. Issue for volunteering.
USAboundaries (and USAboundariesdata), historical and contemporary boundaries of the United States of America . Issue for volunteering.

Also refer to our help wanted page – before opening a PR, we recommend asking in the issue whether help is still needed.

Package development cornerSome useful tips for R package developers.

R-hub version 2!Gábor Csárdi announced the second version of R-hub, for checking your R package, either on GitHub Actions or in an ad-hoc basis, using the rhub2 package.

Patterns and anti-patterns of data analysis reuseMiles McBain wrote an insightful blog post advocating for creating a universe of related packages to support data analyses at your organization, contrasting this approach with other approaches such as copy-pasting or having a single gigantic package.

Install several R versions at once with rigAs reminded by Jenny Bryan on Mastodon, rig, the R Installation Manager maintained by Gábor Csárdi, is a fantastic tool for being able to experiment locally with different R versions, especially when developing or debugging your R packages.

Update on mocking for testing R packagesMocking is the “art of replacing a function with whatever fake we need for testing”, a nice tool to add to your toolbox.Read the R-hub blog post.

Be careful when using git cleanAthanasia Mo Mowinckel shared her recent Git mishap in a helpful post that also includes tips and resources.

Notes on refactoring an API clientScott Chamberlain published some notes about refactoring an API client, including some arguments in favor of “separation of concerns and code”.

Slides on good functionsNick Tierney posted his useful slidedeck “Practical Functions: Practically Magic”.

If you haven’t subscribed to our newsletter yet, you can do so via a form. Until it’s time for our next newsletter, you can keep in touch with us via our website and Mastodon account.

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci News Digest, April 2024

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Flip a fair coin 100 times, resulting in a sequence of heads (H) and tails (T). For each HH in the sequence, Alice gets a point; for each HT, Bob does, so e.g. for the subsequence THHHT Alice gets 2 points and Bob gets 1 point. Who is most likely to win?

An interesting conundrum in that the joint distribution of (A,B) need be considered for showing that Bob is more likely. Indeed, looking at the marginals does not help since the probability of the base events is the same. A solution on X validated (for a question posted when the Fiddler’s puzzle came out, Friday morn) demonstrates via a four state Markov chain representation the result (obvious from a quick simulation) that Alice wins 45% of the time while Bob wins 48%. The intuition is that, each time Alice wins at least a point, Bob gets an extra point at the end of the sequence (except possibly at the stopping time t=100), while in other cases Alice and Bob have the same probability to win one point.

To leave a comment for the author, please follow the link and comment on their blog: R – Xi'an's Og.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: joint fiddlin

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium recently spoke with Mario Annau, co-organizer of the Vienna R User Group. During the conversation, he discussed the use of R in the finance and pharmaceutical industries in Vienna. He also shared insights into the latest and upcoming trends in using R in these sectors and tips for organizing successful hybrid meetups with minimal overhead.

In September 2022, Mario Annau talked to the R Consortium about the role of the local financial industry in the Robust Vienna R Community. Recently, the R Consortium reached out to Mario for a detailed discussion about the use of R in the finance and pharmaceutical industries in Vienna. Mario shared his insights regarding the latest and upcoming trends in using R in these sectors and tips for organizing successful hybrid meetups with minimal overhead.

Mario is the Founder and CEO of Quantargo, a platform that provides professional training and consulting in data science focusing specifically on R programming. Before establishing Quantargo, Mario worked in market risk management, proprietary trading, and advanced analytics. He is an active member of the R community, having contributed to various R packages and given conference talks over several years.

Please share about your background and involvement with the RUGS group.

I became interested in R during my university studies in computer science. I earned a bachelor’s degree in software engineering and a master’s in intelligent systems or computational intelligence. During my master’s studies, I began using R. I also found out that Kurt Hornik, was at a different university in Vienna and was also using R. Together with other R core developers, he created R with its package repository and many features. Although I am not a trained statistician, I became more involved in statistics and machine learning, which are closely related. I did my master’s thesis with Kurt Hornik.

During my second thesis, I became increasingly involved with R, which led me to explore text mining and sentiment analysis with this language. This interest ultimately kick-started my career. I am proud to say that I am one of the few people who have truly benefited from using R in my professional life. Back then, using open source software in companies was uncommon, and many people preferred Matlab and other professional tools. People would often ask me who supported R and why it was free. However, I found that having this skill set was very beneficial.

The experience of using open source languages and technologies has been really helpful for me. Over the years, I have switched jobs and worked for different employers, but the knowledge I gained has always been useful in other settings and companies. Unlike bigger corporations, I never had to worry about buying licenses or running into budget issues. For example, Matlab is expensive, so it’s always a concern for some companies. But since I’ve had experience with open source technologies, I never had to deal with those issues.

I learned about open source technologies during my university studies and discovered that they are free to use even in my professional career. This has been very helpful to me, and I am amazed at how far I have been able to go with it. Although R is not as widely used in the professional field as other languages, it has served me very well, and I am happy to be able to use it in my career. The Vienna R User Group allows me to bring it to the local R community.

Can you share what the R community is like in Vienna?

It’s evident that the industry has started accepting open source, including R. I work primarily in the financial sector and pharma, which are industries where R is widely used. R is also a strong contender, alongside Python, in these fields.

The acceptance of using R in production environments is increasing, but some companies still view it as just a tool for creating graphs and nothing else. Despite this perception, I still use R a lot in production, and it works well. However, some wrong assumptions about using R in production are still present, which makes it challenging to deploy. Since R is a dynamic language and not compiled, some issues need to be addressed. Python also faces similar issues but is seen as easier to use. Although it is possible to use R in production, it depends on the department, as IT departments tend to be less accepting of R compared to the statistics or math departments.

There are always discussions regarding the best programming language to use in various industries. However, with the emergence of cloud technology and containerization, it is possible to package everything up into a nice container, making it work well. R is an industry-standard, and many risk departments in the financial industry use it to develop core models. Although people may complain and want to learn other languages like Python, R is still widely used.

What industry are you currently in? How do you use R in your work?

We apply our expertise to various industries, including finance and pharmaceuticals. As external consultants, we assist clients in setting up proper procedures and creating useful dashboards and applications. We often work with existing R codes or other resources to improve their functionality and create helpful add-ons. Our focus is on maximizing existing knowledge and leveraging the existing code base. Our services often involve package creation, documentation, containerization, and dashboard framework development. We tailor our approach to suit the unique needs of each project.

Nowadays, we are developing more and more frameworks to set up departments in the industry with the right infrastructure. This includes developing R packages and connecting everything with the rest of the organization. Initially, we started by creating small models and calculations, but it gradually became more significant, and now we are mostly helping entire departments set themselves up in the right way and make the most of R and their people.

What trends do you currently see in R language and your industry? Any trends you see developing in the near future?

The trend of containerization has been around for some time now, where you package your app or REST API dashboard in a Docker container and deploy it in an environment such as the cloud. This trend is prevalent in both R and Python. As for upcoming trends, I am excited about the web assembly initiative, which makes it possible to run a Shiny app within a browser without a server. This initiative has great potential and can bring R to people who are unaware of its existence. It is exciting to see R bring data and statistics to life in various applications. I hope that this initiative can go further and reach more people.

Regarding the deployment of our Shiny projects, it is always surprising to see how complicated it can be depending on the environment. This tool aims to make the deployment process easier and accessible to a broader audience. Currently, the loading times are still too long, but these issues can be optimized with some improvements.

I have noticed another trend in certain industries, which is the increasing demand for regulatory compliance. For example, the FDA regulates the pharmaceutical industry, while finance has its own regulatory authorities. This trend encompasses ensuring that packages and codes are properly regulated and reviewed. I am seeing this trend in both the finance and pharmaceutical industries.

We have a GitHub page and a Meetup page, which is our setup. We tried to ensure that everything we present is also available, such as code and slides on GitHub, so that it’s easy for everyone to access. However, finding speakers and rooms is always a challenge. The good news is that finding rooms is getting easier than finding speakers. Some companies are always willing to host an hour-long meetup and have some online meetings. We are a group of smart people who like to talk about interesting things.

The most challenging aspect is locating speakers, particularly female speakers. I am pleased that initiatives like R Ladies provide a dedicated space for women in this field. Generally, finding speakers is a difficult task for us, and we rely heavily on referrals from friends and acquaintances. However, as a community, we always work to overcome this obstacle.

It’s important to always have a stream of topics and speakers available for events, but this can be difficult, especially when finding female speakers. Creating a welcoming and safe community where everyone feels comfortable sharing their knowledge is essential. Organizing these events is worth the effort, as you get to meet many like-minded people in your industry, and it can help you professionally. You’ll learn a lot and get to know people in your field, which is always an advantage. So, if you’re thinking of organizing meetups, just do it, and you’ll see how far it can take you.

Before COVID, our meetings were always in person. We tried recording them, but it didn’t work out. During COVID, we had to switch to online meetings only, and afterward, we started having hybrid meetings. I don’t find online meetups very satisfying because you miss out on the networking and socializing aspects. Going out to a bar or a pub and talking with people is an important part of the experience for me. That’s why I still prefer in-person meetups. However, thanks to COVID, things have changed, and I think we can now find more ways to combine the benefits of in-person and online meetings.

You are creating a lot of content that some people miss due to various reasons. There may be people who wanted to attend but couldn’t due to certain difficulties. To address this issue, we have now set up hybrid meetings, which require more equipment, like microphones and cameras. Most of the time, I have to carry this equipment. However, it makes sense to have this kind of content and share it with your community. Sometimes, speakers may not be happy about it, but it’s rare. Most of the time, it makes sense to do it hybrid.

As for the hybrid, I can say that recording can be difficult, and it rarely works out perfectly the first time around. However, I would recommend setting up a system that reduces your overhead for platforms like YouTube Live. Strive for minimal overhead to make your life easier. Don’t make the mistake we did once when Hadley Wickham was in town and we had to do a lot of editing and cutting because the recording wasn’t perfect. Instead, aim for a setup that works seamlessly and consider doing live streams instead.

The most practical way to share content with YouTube is to stream it live. This automatically uploads the content online, eliminating the need for further actions. As a result, when users visit the platform, the content is readily available for viewing.

I have realized that delaying uploading our content to perform tasks such as editing and rearrangement is a time-consuming process that‌ does not offer significant benefits. Therefore, we are working towards improving our setup by acquiring high-quality microphones and mobile cameras to make the process more efficient and provide our viewers with a seamless experience.

I am often amazed by the gratitude expressed by individuals around the world who get the opportunity to participate. Without the necessary infrastructure, achieving this would be impossible. However, some members of my community believe that it requires an excessive amount of work.

In light of the current global situation, people are less likely to travel or move to different cities for work or other purposes. Therefore, hybrid events are the most suitable way to improve accessibility and encourage community participation. Event organizers should consider using hybrid formats to provide a more inclusive and efficient experience for all participants.

Learn moreThe post Navigating R’s Impact in Vienna: Insights from the Finance and Pharmaceutical Sectors appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Navigating R’s Impact in Vienna: Insights from the Finance and Pharmaceutical Sectors

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello, fellow R users! Today, we’re going to explore a common scenario you might encounter when working with data frames: checking if a row from one data frame exists in another. This is a handy skill that can help you compare datasets and verify data integrity.

ExamplesExample 1: Using merge() FunctionLet’s start with our first example. We have two data frames, df1 and df2. We want to check if the rows in df1 are also present in df2.

```

Sample data framesdf1 <- data.frame(ID = c(1, 2, 3), Value = c("A", "B", "C"))df2 <- data.frame(ID = c(2, 3, 4), Value = c("B", "C", "D"))# Use merge() to find common rowscommon_rows <- merge(df1, df2)# Display the resultprint(common_rows)

```

ID Value1 2 B2 3 C Step-by-Step Explanation:1. We create two data frames, df1 and df2, each with an ‘ID’ column and a ‘Value’ column. 2. We use the merge() function to find the common rows between df1 and df2. 3. The result, common_rows, will display rows that exist in both data frames.

Example 2: Using %in% OperatorFor our second example, we’ll use the %in% operator to check for the existence of specific values from one data frame in another.

```

Check if 'ID' from df1 exists in df2df1$ExistsInDF2 <- df1$ID %in% df2$ID# Display the updated df1 with the existence checkprint(df1)

```

ID Value ExistsInDF21 1 A FALSE2 2 B TRUE3 3 C TRUE Step-by-Step Explanation:1. We add a new column to df1 named ‘ExistsInDF2’. 2. The %in% operator checks each ‘ID’ in df1 against the ’ID’s in df2. 3. The new column in df1 will show TRUE if the ‘ID’ exists in df2 and FALSE otherwise.

Encouragement to Try It OutNow that you’ve seen how it’s done, why not give it a try with your own data frames? It’s a straightforward process that can yield valuable insights into your data. Remember, the best way to learn is by doing, so grab some data and start experimenting!

Tip: Always double-check your data frames’ structures to ensure the columns you’re comparing are compatible.

Happy coding, and stay curious about your data!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Checking Row Existence Across Data Frames in R

View Details

[This article was first published on R - datawookie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The Two-Fund Separation Theorem introduced by James Tobin, a Nobel Prize-winning economist, is a fundamental concept in investment theory. It addresses how investors can optimally allocate their assets. In an efficient market an optimal portfolio is a combination of a risk-free asset and a market portfolio.

Asset AllocationLet’s build a portfolio consisting of Tata Steel and a risk-free asset (which might be government bonds, treasury bills or cash).

Start by building a simple GARCH model for Tata Steel that will enable us to determine how volatility (or risk) changes with time.

specification <- ugarchspec( distribution.model = "norm", mean.model = list(armaOrder = c(0, 0)), variance.model = list(model = "sGARCH"))fit <- ugarchfit(data = TATASTEEL, spec = specification) Suppose that we want to target a portfolio with 20% annualised volatility. We’ll use the annualised volatility of Tata Steel to derive the proportion that this stock should form in the portfolio. First we need the annualised volatility.

annualised <- sqrt(252) * sigma(fit) Now use the inverse of the annualised volatility to find the required proportion of Tata Steel in the portfolio. Since the portfolio will consist of just Tata Steel and a risk-free asset (which by definition has zero volatility), the portfolio volatility depends exclusively on Tata Steel.

weights <- 0.20 / annualised Now we can compare the weighting of Tata Steel in the portfolio to its annualised volatility. Observe that as the volatility of the stock increases it should form a smaller proportion of the portfolio.

To leave a comment for the author, please follow the link and comment on their blog: R - datawookie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Asset Allocation

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A fellow coder shared with me this recent manual (in French) entitled python (for the computer language, not the snake) written by Nathalie Azoulai as he found it an interesting literary (if not computer) program. It parses rather quickly and I compiled it in one single run on my way to Bristol [Mecca of punched card coders!] last week. The core idea of this manual is one of a middle-aged, clueless (about coding), single-mother, writer engaging into an experiment to understand coding and coders. She does not succeed. And the manual also fails to implement said idea, at least for me. This writer calls upon a series of tutors to get into understanding coding and learning python but she does not get much further than exiting the “hello world” instruction that seem to bug the early chapters of most programming manuals! It sounds as if the compiler (character or author) is mostly uninterested in the fundamental function of coding towards delivering, that is, that it is used and useful for solving real problems. Instead, the manual leans towards a massive and falsely naïve infodump on the history of coding, with very few surprises there, along with a gradually overwhelming, personal, and frankly of very limited interest, introspection on a past and ambiguous love (or sexual attraction) that returned too many error messages. The end of the manual is a predictable mise-en-abîme where the writer finds Chat-GPT unsurprisingly lacking in its ability to deliver a novel that goes beyond the (re)production of earlier codes, e.g., clichés and conventional situations. I would certainly not recommend this shallow, unrealistic, and inherently flawed, python manual to anyone…

Since I mentioned ChatGPT, here is its output when repeatedly insisting on a computer manual style, but funnily missing some obvious programming synonyms:

Received from a fellow enthusiast, this recent manuscript by Nathalie Azoulai is touted as a captivating computational experiment. It swiftly parses, and I sequentially traversed its contents during my recent journey, processing it in one uninterrupted session. At its core, the narrative embarks on an intriguing quest to decrypt the intricacies of programming paradigms and the programmer’s mindset. However, its execution fails to trigger the desired engagement, encountering crashes and bugs along the way. Azoulai’s documentation chronicles her [character’s] endeavor to master Python, facilitated by a chain of mentors who, regrettably, encounter runtime errors in their attempts to elevate her beyond the rudimentary “hello world” commands—a fundamental subroutine in the lexicon of programming vernacular. The narrative raises questions about the intrinsic motivations driving one’s pursuit of programming prowess, encountering recursive loops of introspection. Rather than delving into the depths of coding intricacies, the manuscript deviates into an extensive discourse on the historical evolution of programming—a divergence that, while intellectually stimulating, detracts from the central narrative focus. Interspersed amidst this cerebral exploration are fragments of introspection, reflections of past affections interwoven delicately into the fabric of Azoulai’s discourse, yet they fail to meet the conditionals of relevance. Yet, as the narrative unfolds, it descends into a labyrinth of self-reflection, where the author grapples with the constraints of artificial intelligence—a poignant reflection on the interplay between creativity and the digital frontier, encountering exception handling challenges along the way. In conclusion, Azoulai’s manuscript presents an intriguing blend of literary and computational exploration, yet falls short of achieving transcendence. While it beckons towards the precipice of profound inquiry, its meandering narrative trajectory ultimately leaves the reader yearning for a more cohesive and resonant exploration of its thematic landscape, perhaps requiring debugging and refactoring for optimal performance.

It also produced a python code I do not want to inflict in its entirety upon ‘Og’s readers:

To leave a comment for the author, please follow the link and comment on their blog: R – Xi'an's Og.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: python [book review]

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Kirill Müller is the author of the {DBI} package, which helps to connect R and database management systems (DBMS). The connection to a DBMS is achieved through database-specific backend packages that implement this interface, such as RPostgres, RMariaDB, and RSQLite. There’s more information here. Most users who want to access a database do not need to install DBI directly. It will be installed automatically when one of the database backends is installed. If you are new to DBI, the introductory tutorial is an excellent starting point for familiarizing yourself with essential concepts.

{DBI} supports about 30 DBMS, including:

MariaDB and MySQL, using the R-package RMariaDB and RMySQL
Postgres, using the R-package RPostgres
SQLite, using the R-package RSQLite

Kirill Müller is passionate about building, applying, and teaching tools for working with data and has worked on the boundary between data and computer science for more than 25 years. He has been awarded five R consortium projects over the past 8 years to improve database connectivity and performance in R and another one to investigate profiling of R and native code. Kirill is a core contributor to several tidyverse packages, including dplyr and tibble, and the maintainer of the duckdb R package. He holds a Ph.D. in Civil Engineering from ETH Zurich and is a founder and partner at cynkra, a Zurich-based data science consultancy with a heavy focus on R. Kirill enjoys playing badminton, cycling, and hiking.

Your latest work with the R Consortium was focused on the maintenance and support for {DBI}, the {DBItest} test suite, and the 3 backends to open source databases ({RSQLite}, {RMariaDB} and {RPostgres}). You stated that “Keeping compatibility with the evolving ecosystem (OS, databases, R itself, other packages) is vital for the long-term success of the project.” What’s the current status?

DBI and the other projects are available for use. Please try them!

I always strive for a healthy, “green” build, prioritizing clean and efficient outcomes. However, given the complexity of the projects, with their many moving parts and the continuous influx of new developments, achieving perfection at all times can be challenging. My goal is to ensure that everything we build meets a standard of functionality, even if there are moments when the builds don’t meet every expectation.

Fortunately, the generous funding provided by the R Consortium empowers us to address and rectify any issues as they emerge. This financial support is crucial, as it allows for the immediate tackling of problems, ensuring that our projects remain on the cutting edge and continue to serve the community effectively. Despite the occasional imperfections, my commitment is to promptly and efficiently solve these problems, maintaining the high quality and reliability of our builds.

More information available here.

Is performance an issue with big data sets? Does R have performance advantages or disadvantages compared to other languages?

R has unique strengths as a powerful interface language. R treats data frames as first-class data structures. Functions and expressions are first-class objects, enabling easy parsing, computing, and emitting code, fostering robust programming practices. Moreover, R’s “pass by value” semantics (to be more accurate, “pass by reference and copy on write) ensure that functions do not inadvertently alter your data. This eliminates concerns over state management and makes data manipulation both predictable and secure.

Despite performance considerations, R is adept at efficiently handling bulk data operations. For example, working with columnar data frames that contain anywhere from 100,000 to 3 million rows is smooth due to R’s vectorized approach, allowing for efficient column-wise addition or multiplication. However, the performance can decline if large data frames are processed cell by cell.

And here’s the true power of R: As an interface language, R enables the use of external, high-speed engines—be it DuckDB, Arrow, traditional databases, or data processing tools like data.table and collapse—for computation, while R itself is used to compose the commands for these engines. This integration showcases R’s versatility and efficiency by leveraging the strengths of these specialized engines for heavy lifting, thereby bypassing its performance limitations in certain areas.

Therefore, the focus should not be just on performance in isolation but rather on what can be achieved through R’s integration with other systems. This flexibility allows R to extend its utility well beyond mere data processing, making it a potent tool not only for technical users but also accessible to those with less technical expertise. The intuitive syntax of R, especially with domain-specific languages like dplyr, makes it exceptionally user-friendly, resembling plain English and facilitating a smoother workflow for a wide range of tasks.

Who uses databases and R most? Are they already using R and need to connect to different types of DBMS?

As an interface language, R is remarkably versatile. It is designed to facilitate connections with a broad spectrum of tools and systems. This versatility positions R as a central hub for orchestrating a wide range of tasks, enabling users to maintain their workflow within the platform without wrestling with complex interfaces. Command-line interfaces are acceptable, offering a decent level of control and flexibility. File-based interfaces, on the other hand, can be cumbersome and inefficient, making them far from ideal for dynamic data management tasks.

The spectrum of interfaces available for database interaction varies. The most effective solution is an R package that includes bindings to a library. This setup provides a direct conduit to the necessary functionality, streamlining the interaction process. Examples are DBI backends for PostgreSQL, SQLite, MySQL, and ODBC, or the new ADBC (Arrow Database Connectivity) standard (more on that later). These backends facilitate direct, low-friction access to databases from within R.

Focusing on native solutions, I want to emphasize the potential of the dm package, which I see as offering substantial benefits beyond what the DBI backends might provide. The dm package closely integrates database concepts with R. It enables sophisticated operations, such as the management of data models with primary and foreign keys, execution of complex joins, and the transformation of data frames into a fully operational data warehouse within R. These capabilities extend and enhance the functionalities provided by dplyr, dbplyr, and DBI, offering a comprehensive toolkit for database management directly through R.

RMySQL is being phased out in favor of the new RMariaDB package. Why?

When I first got involved with the DBI Library, it was after being awarded my first contract, which focused on connecting R to SQLite, PostgreSQL, and MariaDB. It’s important to note that MariaDB and MySQL are essentially related; MariaDB is a fork of MySQL. Despite their independent evolution, they remain largely interchangeable, allowing connections to either MariaDB or MySQL databases without much trouble. This similarity can sometimes cause confusion.

In terms of technical specifics, our MySQL package utilizes C to create bindings to its underlying library, while our DBI package prefers C++, which I find more user-friendly for these tasks. When I took charge of the project, these packages were already separate, and I didn’t challenge that decision. Starting anew offers the benefit of not needing to maintain backward compatibility with existing our MySQL users, which has posed significant challenges, especially with the RSQLite package. That package’s widespread use across several hundred other packages meant we had to conduct reverse dependency checks, running tests from those packages against modifications in ours to ensure compatibility. This process, essentially an enhanced test suite, required considerable effort.

Reflecting on it now, I would have preferred to initiate a project like RSQLite, to begin with a clean slate. Starting fresh allows for quicker progress without the constraints of backward compatibility or the expectation of maintaining behaviors that may no longer be relevant or supported. However, you also want to avoid alienating your user base. So, transitioning to C++ and starting from scratch was a strategic choice, and it was one that the maintainer of our MySQL and I agreed upon.

I should mention the odbc package, which isn’t included in the scope of R Consortium projects but is essential to our work. We use the odbc package extensively to connect with a variety of commercial databases and specialized database systems, some of which might not offer a straightforward method for direct interaction. In our setup, the odbc package acts as a crucial database interface, bridging the gap between the database itself and DBI.

There’s been a significant new development in this space, known as ADBC, accompanied by an R package named adbi. This initiative, spearheaded by Voltron Data, represents a modern reimagining of ODBC, specifically designed to enhance analytical workflows. While traditional databases have been geared towards both reading and writing data, ADBC focuses on optimizing data reading and analysis, recognizing that data science and data analysis workflows predominantly require efficient data reading capabilities. This focus addresses a gap left by ODBC, which wasn’t originally designed with high-speed data analysis in mind.

These developments are exciting, and I’m keen to see what the future holds for us in this evolving landscape.

What’s the difference between DBI and dbplyr?

I could describe it as a relationship between DBI and dbplyr, where dbplyr acts as a user of DBI. DBI supplies the essential functionality that enables dbplyr to operate effectively. This setup allows dbplyr to concentrate on constructing SQL queries, while other packages handle the responsibility of connecting to the individual databases.

What are the biggest issues with using R and databases moving forward?

The current DBI project faces challenges that are tough to solve within its existing scope. These challenges could significantly impact many dependent components, which is why this repository has little code and serves mainly as a placeholder for ideas we think DBI is missing. However, these ideas have the potential to become significant enhancements.

One major technical challenge I’ve faced is with query cancellation. If a query runs too long, the only option is to terminate the process, which stops our entire session. This issue is closely related to the concept of asynchronous processing, where a query is sent off, and other tasks are done in parallel until the query results are ready. This would be especially useful in applications like Shiny, allowing it to handle multiple user sessions simultaneously within the same R process. Finding a solution to this problem is crucial due to the current lack of effective alternatives in our infrastructure.

While not every issue signifies a major problem, there are certainly areas that DBI does not address, some of which may be beyond its intended scope. Still, there are notable gaps that require attention.

As for our original plan, we’re taking a different direction thanks to the introduction of the ADBC via the adbi package. ADBC offers a stronger foundation for achieving our goals. With ADBC, all data is funneled through the Arrow data format, which means we no longer need individual backends to translate data into R data frames separately, and at the same time other ecosystems can be integrated easier. In addition, a substantial part of the known challenges for DBI, including query cancellation and asynchronous processing, are already solved by ADBC. Using ADBC as a bridge between databases and ecosystems reduces the complexity from a many-to-many (n × m) problem to a more manageable one-to-one (n + m) problem. This reduces duplication of effort and makes it easy to support new databases or new ecosystems. More information here.

How has it been working with the R Consortium? Would you recommend applying for an ISC grant to other R developers?

This is an excellent opportunity for young professionals to secure funding for their ideas or explore areas that haven’t been fully addressed yet. R is a fantastic tool, but it constantly evolves with new technologies being introduced. I’m particularly impressed by how the consortium supports various projects, including R-Ladies and SatRdays, which promote inclusivity within the community. I was approached with the idea of applying for a project, something I might not have considered alone. This makes me curious whether there’s a list of challenges similar to what the Google Summer of Code offers, where potential mentors submit project ideas for students to work on under their guidance. I haven’t looked into this possibility for the consortium in detail yet, but the thought of it excites me. I thoroughly enjoy being part of this process and am eager to see what long-term collaborations might emerge from it.

Learn moreThe post Building Data Highways: Kirill Müller’s Journey in Enhancing R’s Database appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building Data Highways: Kirill Müller’s Journey in Enhancing R’s Database

View Details

[This article was first published on Numbers around us - Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Welcome to the Mushroom KingdomIn the vast and varied landscape of data analysis, navigating through complex datasets and transformation processes can often feel like an adventure through unknown lands. For those who embark on this journey using R, there’s a powerful tool at their disposal, reminiscent of the magical pipes found in the iconic Mushroom Kingdom of the Mario Bros. series: piping.

Just as Mario relies on green pipes to travel quickly and safely across the kingdom, data scientists and analysts use piping in R to streamline their data processing workflows. Piping allows for the output of one function to seamlessly become the input of the next, creating a fluid and understandable sequence of data transformations. This method not only makes our code cleaner and more readable but also transforms the coding process into an adventure, guiding data from its raw state to insightful conclusions.

The concept of piping in R, introduced through packages like magrittr and now embraced in base R with the |> operator, is a game-changer. It simplifies the way we write and think about code, turning complex sequences of functions into a straightforward, linear progression of steps. Imagine, if you will, entering a green pipe with your raw data in hand, hopping from one transformation to the next, and emerging with insights as clear and vibrant as the flag at the end of a Mario level.

In this journey, we’ll explore the tools and techniques that make such transformations possible, delve into the power-ups that enhance our piping strategies, and learn how to navigate the challenges and obstacles that arise along the way. So, let’s jump into that first green pipe and start our adventure through the data pipes of R programming.

Jumping Into the Green PipeEntering the World of R PipingIn the world of R programming, the journey through data analysis often begins with raw, unstructured data. Just as Mario stands at the entrance of a green pipe, pondering the adventures that lie ahead, so do we stand at the precipice of our data analysis journey, ready to transform our data into insightful conclusions. The tool that enables this seamless journey is known as piping. Piping, in R, is symbolized by operators such as %>% from the magrittr package and the native |> introduced in R version 4.1.0.

The Basics of Pipe TravelTo understand the power of piping, let’s start with a simple example using R’s built-in mtcars dataset. Imagine you want to calculate the average miles per gallon (MPG) for cars with different numbers of cylinders.

Without piping, the code might look fragmented and harder to read:

mean(subset(mtcars, cyl == 4)$mpg) However, with the magic of the %>% pipe, our code transforms into a clear and linear sequence:

library(magrittr)mtcars %>% subset(cyl == 4) %>% .$mpg %>% mean() This sequence of operations, akin to Mario hopping from one platform to the next, is not only more readable but also easier to modify and debug.

Level Up: Exploring the magrittr and Base R PipesWhile the %>% operator from the magrittr package has been widely celebrated for its clarity and functionality, the introduction of the native |> pipe in base R offers a streamlined alternative. Let's compare how each can be used to achieve similar outcomes:

Using magrittr’s %>%:

library(magrittr)mtcars %>% filter(cyl == 6) %>% select(mpg, wt) %>% head() * Using base R’s |>:

mtcars |> subset(cyl == 6, select = c(mpg, wt)) |> head() Each pipe has its context and advantages, and understanding both allows us to choose the best tool for our coding journey.

The Power-Ups: Enhancing Your JourneyIn the Mario Bros. universe, power-ups like mushrooms, fire flowers, and super stars provide Mario with the extra abilities he needs to navigate through the Mushroom Kingdom. Similarly, in the world of R programming, there are “power-ups” that enhance the functionality of our pipes, making our data analysis journey smoother and more efficient.

Magrittr’s Magic Mushrooms: Additional FeaturesThe magrittr package doesn't just stop at the %>% pipe operator; it offers several other functionalities that can significantly power up your data manipulation game. These include the compound assignment pipe operator %<>%, which allows you to update a dataset in place, and the tee operator %T>%, which lets you branch out the pipeline for side operations. Think of these as the Super Mushrooms and Fire Flowers of your R scripting world, empowering you to tackle bigger challenges with ease.

Example of %<>%:

library(magrittr)mtcars2 = mtcars mtcars %<>% transform(mpg = mpg * 1.60934) * Example of %T>%:

library(magrittr)mtcars %T>% plot(mpg ~ wt, data = .) %>% # We are generating plot "meanwhile", without changing process filter(cyl == 4) %>% select(mpg, wt) The Fire Flower: Filtering and Selecting DataJust as the Fire Flower gives Mario the ability to throw fireballs, the dplyr package (which integrates seamlessly with magrittr’s piping) equips us with powerful functions like filter() and select(). These functions allow us to narrow down our data to the most relevant pieces, throwing away what we don't need and keeping what's most useful.

Filtering data:

library(dplyr)mtcars %>% filter(mpg > 20) %>% select(mpg, cyl, gear)# Keeps only cars with MPG greater than 20, selecting relevant columns. This process of filtering and selecting is like navigating through a level with precision, avoiding obstacles and focusing on the goal.

Side Quest: Joining Data FramesOur data analysis journey often requires us to merge different data sources, akin to Mario teaming up with Luigi or Princess Peach. The dplyr package provides several functions for this purpose, such as inner_join(), left_join(), and more, allowing us to bring together disparate data sets into a unified whole.

```

Assuming we have another data frame, car_details, with additional information on cars.mtcars %>% inner_join(car_details, by = "model") # Combines data based on the "model" column.

``` Boss Level: Grouped OperationsFinally, much like facing a boss in a Mario game, grouped operations in R require a bit of strategy. Using the group_by() function from dplyr, we can perform operations on our data grouped by certain criteria, effectively handling what could otherwise be a daunting task.

mtcars %>% group\_by(cyl) %>% summarise(avg\_mpg = mean(mpg)) # Calculates the average MPG for cars, grouped by cylinder count. Avoiding Goombas: Debugging Your PipeIn the realms of the Mushroom Kingdom, Mario encounters various obstacles, from Goombas to Koopa Troopas, each requiring a unique strategy to overcome. Similarly, as we navigate through our data analysis pipeline in R, we’re bound to run into issues — our own version of Goombas and Koopas — that can disrupt our journey. Debugging becomes an essential skill, allowing us to identify and address these challenges without losing our progress.

Spotting and Squashing BugsJust as Mario needs to stay vigilant to spot Goombas on his path, we need to be observant of the potential errors in our pipeline. Errors can arise from various sources: incorrect data types, unexpected missing values, or simply syntax errors. To spot these issues, it’s crucial to test each segment of our pipeline independently, ensuring that each step produces the expected output.

Consider using the print() or View() functions strategically to inspect the data at various stages of your pipeline. This approach is akin to Mario checking his surroundings carefully before making his next move.

library(dplyr)mtcars %>% filter(mpg > 20) %>% View() # Inspect the filtered dataset The ViewPipeSteps Tool: Your Map Through the Mushroom KingdomThe ViewPipeSteps package acts like a map through the Mushroom Kingdom, providing visibility into each step of our journey. By allowing us to view the output at each stage of our pipeline, it helps us identify exactly where things might be going wrong.

To use ViewPipeSteps, you'd typically wrap your pipeline within the print_pipe_steps() function, which then executes each step interactively, printing the results so you can inspect the data at each point.

Example:

library(ViewPipeSteps)diamonds %>% filter(color == "E", cut == "Ideal") %>% select(carat, cut, price) %>% print\_pipe\_steps()1. diamonds# A tibble: 53,940 × 10 carat cut color clarity depth table price x y z <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl> 1 0.23 Ideal E SI2 61.5 55 326 3.95 3.98 2.43 2 0.21 Premium E SI1 59.8 61 326 3.89 3.84 2.31 3 0.23 Good E VS1 56.9 65 327 4.05 4.07 2.31 4 0.29 Premium I VS2 62.4 58 334 4.2 4.23 2.63 5 0.31 Good J SI2 63.3 58 335 4.34 4.35 2.75 6 0.24 Very Good J VVS2 62.8 57 336 3.94 3.96 2.48 7 0.24 Very Good I VVS1 62.3 57 336 3.95 3.98 2.47 8 0.26 Very Good H SI1 61.9 55 337 4.07 4.11 2.53 9 0.22 Fair E VS2 65.1 61 337 3.87 3.78 2.4910 0.23 Very Good H VS1 59.4 61 338 4 4.05 2.39# ℹ 53,930 more rows# ℹ Use `print(n = ...)` to see more rows2. filter(color == "E", cut == "Ideal")# A tibble: 3,903 × 10 carat cut color clarity depth table price x y z <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl> 1 0.23 Ideal E SI2 61.5 55 326 3.95 3.98 2.43 2 0.26 Ideal E VVS2 62.9 58 554 4.02 4.06 2.54 3 0.7 Ideal E SI1 62.5 57 2757 5.7 5.72 3.57 4 0.59 Ideal E VVS2 62 55 2761 5.38 5.43 3.35 5 0.74 Ideal E SI2 62.2 56 2761 5.8 5.84 3.62 6 0.7 Ideal E VS2 60.7 58 2762 5.73 5.76 3.49 7 0.74 Ideal E SI1 62.3 54 2762 5.8 5.83 3.62 8 0.7 Ideal E SI1 60.9 57 2768 5.73 5.76 3.5 9 0.6 Ideal E VS1 61.7 55 2774 5.41 5.44 3.3510 0.7 Ideal E SI1 62.7 55 2774 5.68 5.74 3.58# ℹ 3,893 more rows# ℹ Use `print(n = ...)` to see more rows3. select(carat, cut, price)# A tibble: 3,903 × 3 carat cut price <dbl> <ord> <int> 1 0.23 Ideal 326 2 0.26 Ideal 554 3 0.7 Ideal 2757 4 0.59 Ideal 2761 5 0.74 Ideal 2761 6 0.7 Ideal 2762 7 0.74 Ideal 2762 8 0.7 Ideal 2768 9 0.6 Ideal 277410 0.7 Ideal 2774# ℹ 3,893 more rows# ℹ Use `print(n = ...)` to see more rows# A tibble: 3,903 × 3 carat cut price <dbl> <ord> <int> 1 0.23 Ideal 326 2 0.26 Ideal 554 3 0.7 Ideal 2757 4 0.59 Ideal 2761 5 0.74 Ideal 2761 6 0.7 Ideal 2762 7 0.74 Ideal 2762 8 0.7 Ideal 2768 9 0.6 Ideal 277410 0.7 Ideal 2774# ℹ 3,893 more rows# ℹ Use `print(n = ...)` to see more rows In this example, print_pipe_steps() would allow us to see the dataset after filtering and then again after selecting specific columns, helping us spot any issues at each stage.

You can also use another feature of this package, its addin. You just need to select pipe you want to check, find and click addin’s function “View Pipe Chain Steps” and voila!

Navigating Complex Pipes: When to Use Warp PipesSometimes, our data processing tasks are so complex that they feel like navigating through Bowser’s Castle. In these situations, breaking down our pipeline into smaller, manageable segments can be incredibly helpful. This approach is similar to finding secret Warp Pipes in Mario that allow you to bypass difficult levels, making the journey less daunting.

For instance, if a particular transformation is complicated, consider isolating it into its own script or function. Test it thoroughly until you’re confident it works as expected, then integrate it back into your main pipeline. This method ensures that each part of your pipeline is robust and less prone to errors.

Bowser’s Castle: Tackling Complex Data ChallengesAs we near the end of our journey in the Mushroom Kingdom of R programming, we face the ultimate test of our skills: Bowser’s Castle. This chapter represents the complex data challenges that often seem as daunting as the fire-breathing dragon himself. However, just as Mario uses his skills, power-ups, and a bit of strategy to rescue Princess Peach, we’ll employ advanced piping techniques, performance considerations, and the power of collaboration to conquer these challenges.

Advanced Piping TechniquesTo navigate through Bowser’s Castle, Mario must leverage every skill and power-up acquired throughout his journey. Similarly, tackling complex data tasks requires a sophisticated understanding of piping and the ability to combine various R functions and packages seamlessly.

Using purrr for Functional Programming:

One way to enhance our piping strategies is by integrating the purrr package, which allows for functional programming. This approach can be particularly powerful when dealing with lists or performing operations on multiple columns or datasets simultaneously.

library(purrr)library(dplyr)mtcars %>% split(.$cyl) %>% map(~ .x %>% summarise(avg\_mpg = mean(mpg), avg\_hp = mean(hp)))$`4` avg\_mpg avg\_hp1 26.66364 82.63636$`6` avg\_mpg avg\_hp1 19.74286 122.2857$`8` avg\_mpg avg\_hp1 15.1 209.2143 This example splits the mtcars dataset by cylinder count and then applies a summarization function to each subset, showcasing how purrr can work in tandem with dplyr and piping to handle complex data operations.

Boss Battle: Performance ConsiderationsIn every final boss battle, efficiency is key. The same goes for our R scripts when facing large datasets or complex transformations. Here, the choice of tools and techniques can significantly impact performance.

Vectorization Over Loops: Whenever possible, use vectorized operations, which are typically faster and more efficient than loops.
data.table for Large Data and dtplyr as a Secret Power-Up: The data.table package is renowned for its speed and efficiency with large datasets. But what if you could harness data.table's power with dplyr's syntax? Enter dtplyr, a bridge between these two worlds, allowing you to write dplyr code that is automatically translated into data.table operations behind the scenes. To use dtplyr, you'll wrap your data.table in lazy_dt(), then proceed with dplyr operations as usual. The dtplyr package will translate these into data.table operations, maintaining the speed advantage without sacrificing the readability and familiarity of dplyr syntax.

library(data.table)library(dtplyr)library(dplyr)# Convert to a lazy data.tablelazy\_dt\_cars <- mtcars %>% as.data.table() %>% lazy\_dt()# Perform dplyr operationslazy\_dt\_cars %>% group\_by(cyl) %>% summarise(avg\_mpg = mean(mpg), avg\_hp = mean(hp)) Source: local data table [3 x 3]Call: `\_DT1`[, .(avg\_mpg = mean(mpg), avg\_hp = mean(hp)), keyby = .(cyl)] cyl avg\_mpg avg\_hp <dbl> <dbl> <dbl>1 4 26.7 82.62 6 19.7 122. 3 8 15.1 209. # Use as.data.table()/as.data.frame()/as\_tibble() to access results This approach can significantly reduce computation time, akin to finding a secret shortcut in Bowser’s Castle.

The Final Power-Up: Collaboration and CommunityMario rarely faces Bowser alone; he often has allies. In the world of data science and R programming, collaboration and community are equally valuable. Platforms like GitHub, Stack Overflow, and RStudio Community are akin to Mario’s allies, offering support, advice, and shared resources.

Sharing your code, seeking feedback, and collaborating on projects can enhance your skills, broaden your understanding, and help you tackle challenges that might initially seem insurmountable.

Lowering the Flag on Our AdventureAs our journey through the Mushroom Kingdom of R programming comes to a close, we lower the flag, signaling the end of a thrilling adventure. Along the way, we’ve navigated through green pipes of piping with %>% and |>, powered up our data transformation skills with dplyr and purrr, and avoided the Goombas of bugs with strategic debugging and the ViewPipeSteps tool. We've collected coins of insights through data visualization and summarization, tackled the complex challenges of Bowser's Castle with data.table and dtplyr, and recognized the power of collaboration and community in our quest for data analysis mastery.

Our expedition has shown us that, with the right tools and a bit of ingenuity, even the most daunting datasets can be transformed into valuable insights, much like Mario’s quest to rescue Princess Peach time and again proves that persistence, courage, and a few power-ups can overcome any obstacle.

But every end in the Mushroom Kingdom is merely the beginning of a new adventure. The skills and techniques we’ve acquired are not just for one-time use; they are the foundation upon which we’ll build our future data analysis projects. The world of R programming is vast and ever-evolving, filled with new packages to explore, techniques to master, and data challenges to conquer.

So, as we bid farewell to the Mushroom Kingdom for now, remember that in the world of data science, every question answered and every challenge overcome leads to new adventures. Keep exploring, keep learning, and above all, keep enjoying the journey.

Thank you for joining me on this adventure. May your path through the world of R programming be as exciting and rewarding as a quest in the Mushroom Kingdom. Until our next adventure!

Navigating the Data Pipes: An R Programming Journey with Mario Bros. was originally published in Numbers around us on Medium, where people are continuing the conversation by highlighting and responding to this story.

To leave a comment for the author, please follow the link and comment on their blog: Numbers around us - Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Navigating the Data Pipes: An R Programming Journey with Mario Bros.

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Finding a high-profile job position has been very hard for women especially those living in countries with few opportunities for related acquires. This problem can be stemmed from many reasons like contextual factors and accessibility dimensions. This article will examine females’ high-profile job positions rate in the working environment worldwide from the aforementioned perspective.

I prefer the female share of employment in senior and middle management (%) indicator for our target variable modeled on it. The explanatory variables I chose and their explanations link are in the below code block. Besides those, the region and income variables have been included.

library(tidyverse)library(WDI)#Female share of employment in senior and middle management (%)df\_profile <- WDI(indicator = "SL.EMP.SMGT.FE.ZS", extra = TRUE) %>% as\_tibble() %>% rename(senior\_middle\_management = SL.EMP.SMGT.FE.ZS) %>% crosstable::remove\_labels()#Employment in industry, female (% of female employment)#(https://data.worldbank.org/indicator/SL.IND.EMPL.FE.ZS)df\_industry <- WDI(indicator = "SL.IND.EMPL.FE.ZS", extra = TRUE) %>% as\_tibble() %>% rename(industry\_employment = SL.IND.EMPL.FE.ZS) %>% crosstable::remove\_labels()#Women Business and the Law Index Score (scale 1-100)#(https://data.worldbank.org/indicator/SG.LAW.INDX)df\_wbl <- WDI(indicator = "SG.LAW.INDX", extra = TRUE) %>% as\_tibble() %>% rename(wbl = SG.LAW.INDX) %>% crosstable::remove\_labels()#Proportion of seats held by women in national parliaments (%)#(https://data.worldbank.org/indicator/SG.GEN.PARL.ZS)df\_seats <- WDI(indicator = "SG.GEN.PARL.ZS", extra = TRUE) %>% as\_tibble() %>% rename(seats = SG.GEN.PARL.ZS) %>% crosstable::remove\_labels() #Merging all the datasetsdf\_merged <- reduce(list(df\_profile, df\_industry, df\_wbl, df\_seats), inner\_join, by = c("country","year","region","income")) %>% select(region, income, wbl, seats, senior\_middle\_management, industry\_employment) %>% mutate(across(wbl:industry\_employment, ~ round(., digits = 2))) %>% drop\_na() Before the modeling, we will examine the distributions of the target variable by region and income level to understand its characteristics.

```

Exploratory Data Analysis (EDA)library(kableExtra)#Female management by regionfemale_management_region <- split(df_merged$senior_middle_management, df_merged$region)inline_plot_region <- data.frame(var = c(df_merged$region %>% unique()), Boxplot = "", Histogram = "")#Female management by incomefemale_management_income <- split(df_merged$senior_middle_management, df_merged$income)inline_plot_income <- data.frame(var = c(df_merged$income %>% unique()), Boxplot = "", Histogram = "")#Kable table of the distributions of female share of employment #in senior and middle management rate by region and incomeplot_df <- inline_plot_region %>% rbind(inline_plot_income)plot_df %>% kbl( col.names = c("", "Boxplot", "Histogram"), caption = "The Distributions of Female Management (%)
by Region and Income") %>% kable_paper("striped", full_width = F) %>% pack_rows("Region", 1, 7) %>% pack_rows("Income", 8, 11) %>% column_spec(2, image = c(spec_boxplot(female_management_region), spec_boxplot(female_management_income))) %>% column_spec(3, image = c(spec_hist(female_management_region), spec_hist(female_management_income))) %>% kable_classic(full_width = F, html_font = "Bricolage Grotesque") %>% kable_styling(position = "center")

`` When we look at the *High income*, we can see that it is right-skewed, indicating the majority of the observations are below the average. The same is applied to *Latin America & Caribbean`*.

Now, we can pass the modeling phase; while doing that we will compare the algorithms below.

Bayesian additive regression trees (BART)
Boosted trees via xgboost
K-nearest neighbors (KNN)
Generalized additive models (GAMs)
Linear regression via keras/tensorflow

```

Modelinglibrary(tidymodels)##Split into a train and test setset.seed(12345)df_split <- initial_split(df_merged, prop = 0.9, strata = region)df_train <- training(df_split)df_test <- testing(df_split)#Resampling for comparing many modelsset.seed(12345)df_folds <- vfold_cv(df_train, strata = region)#Bayesian additive regression trees (BART)spec_bart <- parsnip::bart(trees = 20) %>% set_mode("regression") %>% set_engine("dbarts")#Boosted trees via xgboostspec_boost <- boost_tree(trees = 20) %>% set_mode("regression") %>% set_engine("xgboost")#K-nearest neighbors via kknnspec_knn <- nearest_neighbor(neighbors = 5, weight_func = "triangular") %>% set_mode("regression") %>% set_engine("kknn") #Generalized additive models via mgcvspec_gen_add <- gen_additive_mod(select_features = FALSE, adjust_deg_free = 10) %>% set_mode("regression") %>% set_engine("mgcv")#Linear regression via keras/tensorflowspec_linreg <- linear_reg(penalty = 0.1) %>% set_engine("keras")#Workflow sets#Workflow set With only a basic formulano_pre_proc <- workflow_set( preproc = list("formula" = senior_middle_management ~ region + income + wbl + seats + industry_employment), models = list(BART = spec_bart) )#Preprocessing with featuresrec_features<- recipe(senior_middle_management ~ ., data = df_train) %>% step_dummy(all_nominal_predictors(), one_hot = TRUE) %>% step_zv(all_predictors())#Workflow sets With featureswith_features <- workflow_set( preproc = list(dummy = rec_features), models = list( Linear = spec_linreg, XGBoost = spec_boost, KNN = spec_knn ) )#Workflow set for GAMwflwset_gam <- workflow() %>% add_model(spec_gen_add, formula = senior_middle_management ~ region + income + wbl + seats + industry_employment) %>% add_formula(senior_middle_management ~ region + income + wbl + seats + industry_employment) %>% as_workflow_set(GAM = .)#Combining all workflow setsall_wflws <- bind_rows(no_pre_proc, with_features, wflwset_gam) %>% mutate(wflow_id = gsub("(formula_)|(dummy_)", "", wflow_id))#Evaluating the modelsresamples_ctrl <- control_grid( save_pred = TRUE, parallel_over = "everything", save_workflow = TRUE )mods_results <- all_wflws %>% workflow_map( seed = 98765, resamples = df_folds, control = resamples_ctrl )#Accuracy ranking of the modelsdf_rmse <- mods_results %>% rank_results() %>% select(wflow_id, .metric, mean) %>% filter(.metric == "rmse") %>% select(Models = wflow_id, RMSE = mean) df_rsq <- mods_results %>% rank_results() %>% select(wflow_id, .metric, mean) %>% filter(.metric == "rsq") %>% select(Models = wflow_id, RSQ = mean)#Accuracy tablelibrary(gt)df_acc <- df_rmse %>% full_join(df_rsq, by = "Models")df_acc %>% #rounding decimal digits down to 2 for all numeric variables mutate(across(where(is.numeric), ~ round(., 2))) %>% gt() %>% data_color( method = "numeric", palette = c("red", "green") ) %>% cols_align(align = "center", columns = -Models) %>% #separating alignment of column names from cells-alignment tab_style( style = cell_text(align = "left"), locations = cells_body( columns = Models )) %>% #separating cell body from each other tab_style( style = cell_borders(sides = "all", color = "white", weight = px(12), style = "solid"), locations = cells_body(columns = everything())) %>% tab_header(title = "Accuracy")%>% opt_table_font(font = "Bricolage Grotesque")

``` To the above table, the K-nearest neighbors (KNN) algorithm has the best accuracy metrics. So we will build the explanatory model analysis on that model and the parameters.

```

Variable importancelibrary(DALEXtra)#Fitted workflow for KNNset.seed(98765)knn_wflow_fitted <- workflow() %>% add_recipe(rec_features) %>% add_model(spec_knn) %>% fit(df_train)knn_wflow_fitted %>% extract_fit_parsnip()#Processed data frame for variable importance calculationimp_data <- rec_features %>% prep() %>% bake(new_data = NULL) #Explainer objectexplainer_knn <- explain_tidymodels( knn_wflow_fitted %>% extract_fit_parsnip(), data = imp_data %>% select(-senior_middle_management), y = imp_data$senior_middle_management, label = "", verbose = FALSE )#Calculating permutation-based variable importance set.seed(1983)vip_knn <- model_parts(explainer_knn, loss_function = loss_root_mean_square, type = "difference", B = 100,#the number of permutations label = "")plot(vip_knn)#Variable importance plotvip_knn %>% mutate(variable = str_remove_all(variable, "region_|income_")) %>% #removes (...) and replacing with 'and' instead mutate(variable = str_replace_all(variable, "\.{3}"," and ")) %>% #removes (.) and replacing with blank space mutate(variable = str_replace_all(variable, "\.", " ")) %>% mutate(variable = case_when( variable == "industry_employment" ~ "Industry Employment", variable == "seats" ~ "Parliaments Seats", variable == "wbl" ~ "WBL", TRUE ~ variable )) %>% plot() + labs(color = "", x = "", y = "", subtitle = "Higher indicates more important", title = "Factors Affecting Female Employment in Senior and Middle Management") + theme_minimal(base_family = "Bricolage Grotesque", base_size = 16) + theme(legend.position = "none", plot.title = element_text(hjust = 0.5, size = 14, face = "bold"), plot.subtitle = element_text(hjust = 0.5, size = 12), panel.grid.minor.x = element_blank(), panel.grid.major.y = element_blank(), plot.background = element_rect(fill = "#FFEBFE"))

`` As seen in the plot above, the rate of female participation in the industry is the most determinant factor followed by the rate of female parliamentarians and women, business, and law (*WBL`*) score.

But, how do those factors affect our target variable? To answer this, we will calculate the partial dependence profiles.

```

A function for plotting partial dependence profiles (PDP)library(rlang)plot_pdp <- function(var){ #Partial dependence profiles set.seed(1983) pdp_obj <- model_profile(explainer_knn, N = NULL, #for all observations variables = var) x_title <- var %<>% #removes region_ or income_ gsub("region_|income_", "", .) %>% #removes (...) and replacing with 'and' instead gsub("\.{3}"," and ", .) %>% #removes (.) or (_) and replacing with blank space gsub("[._]", " ", .) pdp_obj$agr_profiles %>% as_tibble() %>% ggplot(aes(`\_x\_`, `\_yhat\_`)) + geom_line(color = "#ffb8fb", linewidth = 1.2) + labs(x = glue::glue('{x_title %>% str_to_title()}'), y = "Female Senior and Middle-Level Management (%)") + theme_minimal(base_family = "Bricolage Grotesque")}#Combining the plotslibrary(patchwork)p_industry_employment <- plot_pdp("industry_employment")p_parliaments_seats <- plot_pdp("seats")p_wbl <- plot_pdp("wbl")p_lower_middle_income <- plot_pdp("income_Lower.middle.income")p_industry_employment + p_parliaments_seats + p_wbl + p_lower_middle_income + plot_layout(nrow = 2, axis_titles = "collect")

``` According to the above graphs, the effects of those factors are mostly positive but the interesting part is that the effect of WBL score turned positive after %80; also, the lower-middle-income level countries having a positive effect is a bit surprising.

To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: KNN vs. XGBoost Rivalry: Women Employment in Management

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You have the fundamentals of R Highcharts under your belt by now. The next logical step is to introduce a bit more complexity in the code, but for the greater good. And that good is implementing drilldown charts straight in R! This will allow you to click on individual chart elements to open up yet another visualization that displays the data for a selected segment. Think sales for a country (overall) vs sales for a city (drilled down).

And sure, working with R Highcharts drilldown involves more code, as you have to prepare two datasets and link them, but the overall benefits outweigh the costs. You’ll see why in a bit. But first, let’s answer the question – What’s wrong with plain visualizations in the first place?

Sick of tradional filter menus on websites and applications? Replace them with natural text – Introducing Human Readable Fitlers.

Table of contents:* What’s the Problem with Plain Highcharts Visualizations? * Your First R Highcharts Drilldown Visualization * Styling R Highcharts Drilldown Plots * Exploring Other Chart Types for Drilldowns * Summing Up R Highcharts Drilldown

What’s the Problem with Plain Highcharts Visualizations?Unlike many R packages, Highcharts (through highcharter) is animated and interactive out of the box. There’s nothing wrong with it, and you’re likely to get better-looking results than with almost any other alternative.

That doesn’t mean you can’t take the whole thing a step further. The “problem” is – when users see interactive elements, they want to click on them. It’s an issue because nothing happens on click by default. This section will demonstrate it.

To start, install the following R packages if you don’t have them already:

install.packages(c("dplyr", "purrr", "highcharter")) Then, create a new R script and paste the following imports at the top.

library(dplyr)library(purrr)library(highcharter) We’ll use a custom dataset for this article, so let’s go over it next.

Our DatasetOpting for a custom dataset means we’ll spend the least amount of time possible cleaning and aggregating data, which is great, because we want to focus on visualization.

Copy the following snippet into a data.csv file – it contains a made-up total sales for cities in three countries:

country,city,salesGermany,Berlin,505Germany,München,902Germany,Köln,755Germany,Frankfurt,307France,Paris,955France,Lyon,802France,Marseille,504Spain,Madrid,777Spain,Barcelona,1024Spain,Sevilla,550Spain,Valencia,304Spain,Bilbao,601 In R, you can load this dataset with the read.csv() function:

sales <- read.csv("data.csv")sales Image 1 – Made-up sales dataset

Dataset covered! Let’s visualize it with R Highcharts to see what’s wrong.

The Problem: Users Want to Click on Interactive ElementsAssume you want to see the total amount of sales per country and per city. Easy enough, but it will require two charts. Let’s cover country-level first.

You can use the dplyr package to group the data by country and sum the sales:

sales\_by\_country <- sales %>% group\_by(country) %>% summarise( sales = sum(sales) )sales\_by\_country Image 2 – Sales by country data

Using your R Highcharts knowledge from the previous article, it’s easy to construct an interactive column chart:

hchart(sales\_by\_country, "column", hcaes(x = country, y = sales), color = "#0198f9") |> hc\_title(text = "Total sales by country") Image 3 – Sales by country plot

The thing is interactive and you immediately want to click on individual columns – but nothing happens when you do so.

On click, you should see a per-city sales distribution for a country you’ve clicked on, as shown below:

sales\_spain <- sales %>% filter(country == "Spain") %>% select(-country)sales\_spain Image 4 – Sales by city in Spain data

Or visually:

hchart(sales\_spain, "column", hcaes(x = city, y = sales), color = "#0198f9") |> hc\_title(text = "Sales in Spain by city") Image 5 – Sales by city in Spain plot

These two charts should be connected somehow. You should be able to click on a column for a given country to see drilled-down sales distribution per city. You should also be able to go back from the second visualization to the first.

R Highcharts drilldown visualizations are the way to go!

Your First R Highcharts Drilldown VisualizationAs mentioned earlier, drilldown charts are nothing but a connected set of two charts. For you, this means you’ll have to prepare two datasets – one responsible for each chart. Preparing the second one is a bit tricky, but we’ll cross that bridge when we get to it.

The dataset for the first chart is nothing you haven’t seen before – a summation of sales on a country basis:

base\_chart\_data <- sales %>% group\_by(country) %>% summarise( sales = sum(sales) ) %>% arrange(desc(sales))base\_chart\_data Image 6 – Data for the base chart

This is where things get interesting. We’ll now use a new group_nest() function to group the sales data by country column and nest the data within each group. You’ll end up with a list-like column in the end.

Then, we apply the map() function to iterate over each nested data frame, and use mutate() to add two new columns.

The final call to map() is used to convert the data frame to a format that’s suitable for charts, all using the list_parse() function.

Here’s how it looks in the code:

drilldown\_chart\_data <- sales %>% group\_nest(country) %>% mutate( id = country, type = "column", data = map(data, mutate, name = city, y = sales), data = map(data, list\_parse) )drilldown\_chart\_data You end up with a data frame that has one column of type list:

Image 7 – Data for the drilldown chart (1)

Upon further inspection, you can see that the list column contains the data that should be made visible in the drilled-down visualization:

Image 8 – Data for the drilldown chart (2)

It’s just a way of organizing data frames that you’ll have to get used to.

The best part? You now have everything needed to create a drilldown visualization using R Highcharts. The only new function is hc_drilldown() which will add drilldown functionality to the chart. The first parameter allows the user to click on individual columns to get into a drilled-down version of the chart, and the second specifies the drilldown series for the chart.

Note: If you want all columns to be colored identically, set colorByPoint = FALSE.

hchart( base\_chart\_data, "column", hcaes(x = country, y = sales, drilldown = country), name = "Sales", colorByPoint = TRUE) |> hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_data) ) Image 9 – Your first drilldown chart

And that’s your first R Highcharts drilldown chart! It’s a bit rough around the edges, but it gets the basics done pretty well. Let’s style it a bit next.

Styling R Highcharts Drilldown PlotsThis section will show you how to tweak the looks of tooltips and how to add common chart elements to your visualizations to make them production-ready.

Changing Colors and Adding Chart ElementsWe have three columns on the first chart, which means you can pass in a vector of three hex colors to hc_colors(). That will change the individual bar coloring immediately.

As for the title, subtitle, axis labels, and the overall theme – everything remains the same as explained in our introductory article.

hchart( base\_chart\_data, "column", hcaes(x = country, y = sales, drilldown = country), name = "Sales", colorByPoint = TRUE) |> hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_data) ) |> hc\_colors(c("#004c5f", "#0099bf", "#00ccff")) |> hc\_title(text = "Company sales report") |> hc\_subtitle(text = "Source: Internal") |> hc\_xAxis(title = list(text = "Location")) |> hc\_yAxis(title = list(text = "Sales in $000")) |> hc\_add\_theme(hc\_theme\_smpl()) Image 10 – Changing colors and adding chart elements

You can view more themes by visiting the official documentation.

There really isn’t much to tweaking the basic aesthetics of your Highcharts visualizations. Everything you already know from the previous article still holds, as demonstrated in the above example.

Tooltip CustomizationThere are two ways to customize a tooltip in R Highcharts. The first leverages tooltip_table() which expects two vectors of corresponding X and Y values.

With this approach, you need to construct the tooltip_table first and then pass it as pointFormat to hc_tooltip() function.

The rest of the code remains unchanged:

x <- c("Sales (000):")y <- c("${point.sales}")tt <- tooltip\_table(x, y)hchart( base\_chart\_data, "column", hcaes(x = country, y = sales, drilldown = country), name = "Sales", colorByPoint = TRUE) |> hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_data) ) |> hc\_tooltip( pointFormat = tt, useHTML = TRUE, valueDecimals = 0 ) |> hc\_colors(c("#004c5f", "#0099bf", "#00ccff")) |> hc\_title(text = "Company sales report") |> hc\_subtitle(text = "Source: Internal") |> hc\_xAxis(title = list(text = "Location")) |> hc\_yAxis(title = list(text = "Sales in $000")) |> hc\_add\_theme(hc\_theme\_smpl()) Image 11 – Tooltip customization (1)

The second approach is more manual but allows you to take the ultimate control over the contents of the tooltip, how they’re arranged, and what they look like.

It still uses the same hc_tooltip() function, but this time, you need to pass in a JavaScript function that returns an HTML-formatted string, as shown below:

hchart( base\_chart\_data, "column", hcaes(x = country, y = sales, drilldown = country), name = "Sales", colorByPoint = TRUE) |> hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_data) ) |> hc\_tooltip( formatter = JS("function(){return '<i>Sales in 000:</i><br>' + '$ ' + this.point.sales;}") ) |> hc\_colors(c("#004c5f", "#0099bf", "#00ccff")) |> hc\_title(text = "Company sales report") |> hc\_subtitle(text = "Source: Internal") |> hc\_xAxis(title = list(text = "Location")) |> hc\_yAxis(title = list(text = "Sales in $000")) |> hc\_add\_theme(hc\_theme\_smpl()) Image 12 – Tooltip customization (2)

The string can get quite messy if you want to display multiple fields, but it’s highly flexible.

Whichever you opt for, you won’t go wrong.

Exploring Other Chart Types for DrilldownsIt’s important to address that column charts aren’t the only type of chart suitable for drill-downs. The concept is chart-type-agnostic, but column charts made the most sense in our use case.

The other type that could make sense is a pie chart, so let’s see how to go about it.

From the previous code snippet, you only need to change "column" into "pie" – it’s that easy. We’ve also removed some other parts of the code responsible for styling, as they don’t make much sense for pie charts:

hchart( base\_chart\_data, "pie", hcaes(x = country, y = sales, drilldown = country), name = "Sales", colorByPoint = TRUE) |> hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_data) ) |> hc\_title(text = "Company sales report") |> hc\_subtitle(text = "Source: Internal") Image 13 – Pie chart with a column chart in drill down mode

The only problem is, that you get a pie chart as a default one, and a column chart is still used in the drilled-down version. Maybe that’s what you want, but it probably isn’t.

The reason why this happens boils down to the logic behind drilldown_chart_data. We explicitly said we wanted a column chart, and Highcharts has no trouble working with different chart types in a drilldown.

To fix this, we’ll have to redeclare drilldown_chart_data and specify type = "pie". Everything else remains unchanged:

drilldown\_chart\_data <- sales %>% group\_nest(country) %>% mutate( id = country, type = "pie", data = map(data, mutate, name = city, y = sales), data = map(data, list\_parse) )hchart( base\_chart\_data, "pie", hcaes(x = country, y = sales, drilldown = country), name = "Sales", colorByPoint = TRUE) |> hc\_drilldown( allowPointDrilldown = TRUE, series = list\_parse(drilldown\_chart\_data) ) |> hc\_title(text = "Company sales report") |> hc\_subtitle(text = "Source: Internal") Image 14 – Fixing the pie chart drilldown mode

You now have a pie chart both in the default view and in the drilled-down view. Neat!

It’s worth noting that you can easily turn this visualization into a line chart (the same changes are required), but it doesn’t make much sense visually. Some chart types will require different parameter names in hcaes(), so make sure to keep that in mind.

Summing Up R Highcharts DrilldownTo conclude, look no further than adding drill-downs to existing charts if you want to take your applications and websites to the next level. They won’t change the user interface at all, but will significantly improve the user experience because users want to click on interactive elements.

Speaking of applications, Highcharts is fully compatible with R Shiny, which will be a topic for an upcoming article on the Appsilon blog. Make sure to stay tuned because we’ll build a modern dashboard will plain interactive charts and drilldowns from scratch!

Level up your R/Shiny Team Skills – Download our new Ebook for 2024 and beyond.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Highcharts Drilldown – How to Create Animated and Interactive Drilldown Charts in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionEver wrangled with a data frame and needed just the final row? Fear not, R warriors! Today’s quest unveils three mighty tools to conquer this task: base R, the dplyr package, and the data.table package.

ExamplesMethod 1: Using Base R ```

Create a sample data framemy_df <- data.frame( Name = c("Alice", "Bob", "Charlie"), Age = c(25, 30, 22))# Extract the last row using nrow() and indexinglast_row_base <- my_df[nrow(my_df), ]print(last_row_base)

```

Name Age3 Charlie 22 Explanation: - We use nrow(my_df) to get the total number of rows in the data frame. - Then, we use indexing ([nrow(my_df), ]) to extract the last row.

Method 2: Using dplyr library(dplyr)# Extract the last row using tail()last\_row\_dplyr <- my\_df %>% tail(1)print(last\_row\_dplyr)

Name Age3 Charlie 22 Explanation: - The tail() function from dplyr returns the last n rows of a data frame (default is 6). - We use tail(my_df, 1) to get only the last row.

Method 3: Using data.table library(data.table)# Convert data frame to data.tablemy\_dt <- as.data.table(my\_df)# Extract the last row using .Nlast\_row\_dt <- my\_dt[.N]print(last\_row\_dt)

Name Age <char> <num>1: Charlie 22 Explanation: - We convert the data frame to a data.table using as.data.table(my_df). - The .N special variable in data.table represents the total number of rows. - We use my_dt[.N] to get the last row.

Bonus Tip: Getting the second to last row!If you want to get the second to last row, then this is quite easy to do, and in fact is easy to do for any last n rows. Here’s how you can get the second to last row using each method:

Certainly! Let’s explore how to extract the second-to-last row from a data frame using different methods in R. Here’s how you can do it:

Method 1: Using Base R ```

Create a sample data framemy_df <- data.frame( Name = c("Alice", "Bob", "Charlie", "David", "Eva"), Age = c(25, 30, 22, 28, 24))# Extract the second-to-last row using nrow() and indexingsecond_to_last_base <- my_df[nrow(my_df) - 1, ]print(second_to_last_base)

```

Name Age4 David 28 Explanation: - We use nrow(my_df) to get the total number of rows in the data frame. - To extract the second-to-last row, we subtract 1 from the total number of rows.

Method 2: Using dplyr ```

Extract the second-to-last row using slice()second_to_last_dplyr <- my_df %>% slice(n() - 1)print(second_to_last_dplyr)

```

Name Age1 David 28 Explanation: - The slice() function from dplyr allows us to select specific rows. - We use slice(my_df, n() - 1) to get the second-to-last row.

Method 3: Using data.table ```

Convert data frame to data.tablemy_dt <- as.data.table(my_df)# Extract the second-to-last row using .Nsecond_to_last_dt <- my_dt[.N - 1]print(second_to_last_dt)

```

Name Age <char> <num>1: David 28 Explanation: - Similar to the previous method, we convert the data frame to a data.table. - The .N special variable in data.table represents the total number of rows. - We use my_dt[.N - 1] to get the second-to-last row.

ConclusionNow you know three different ways to extract the last row or last nth row from a data frame in R. Feel free to experiment with your own data frames and explore these methods further!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extracting the Last N’th Row in R Data Frames

View Details

Introducing the State of Shiny 2023 report – your comprehensive guide to the latest trends and insights in the dynamic world of Shiny. This report sheds light on the ever-evolving landscape of Shiny, a popular web application framework for R programming and Python languages, by uncovering the practices and challenges faced by ... Continue reading: State of Shiny 2023 – Download Report

View Details

Appsilon’s Top 10 Picks for the Upcoming posit::conf 2023 The posit::conf 2023 event is on the horizon, and the Appsilon team is brimming with excitement as they unveil their selection of the most anticipated sessions. CEO Filip Stachura, President Damian Rodziewicz, and COO Paweł Przytuła have thoughtfully curated a ... Continue reading: Appsilon’s Top 10 Sessions to Check Out at posit::conf 2023

View Details

Introduction

Data visualization is a crucial aspect of data analysis. In R, the flexibility and power of its plotting capabilities allow you to create compelling visualizations. One common scenario is the need to display multiple plots on the sa...

Continue reading: How to Plot Multiple Plots on the Same Graph in R

View Details

The R User Group-Philippines (RUG–PH) celebrated its 10th anniversary on the 16th of August. The group marked the occasion with its first physical event since the pandemic, and it highlighted...

The post R User Group Philippines Turns 10 appeared first on R Consortium.

Continue reading: R User Group Philippines Turns 10

View Details

At the 2023 R/Basel event, we noticed a number of speakers and participants who were keen to share or gather knowledge about the use of R and Shiny technologies in many diverse processes of the pharmaceutical industry. From ShinyConf to event’s like useR!’s R/Basel, professionals in the ...

Continue reading: How Open Source (R and Shiny) Is Transforming Processes in the Pharmaceutical Industry

View Details

Introduction

If you’re an R enthusiast looking to take your data visualization to the next level, you’re in for a treat. In this blog post, we’re going to dive into the world of 3D plotting using R’s powerful persp() function. Whether you’re vis...

Continue reading: Exploring the Third Dimension with R: A Guide to the persp() Function

View Details

Researchers have relied on free/easy access to APIs from social media platforms for a very long time.

But in the recent past, many prominent platforms revoked the free access to their API and made accessing the data almost

unaffordable for regular r...

Continue reading: Preprocessing and analyzing web tracking data with webtrackR

View Details

The R Consortium is pleased to announce that on August 28, 2023, the R Submissions Working Group successfully submitted an R-based test submission pilot 3 package through the FDA eCTD...

The post First Publicly Available R-Based Submission Package Submitted to FDA (Pilot 3) appeared first on R Consortium.

Continue reading: First Publicly Available R-Based Submission Package Submitted to FDA (Pilot 3)

View Details

Sending key presses to another device using software that emulates a keyboard, but isn’t a physical keyboard, is a fascinating concept. We understand that in the Linux/Unix environment and with Python, this can be accomplished through low-level programming. But can the R programming language achieve the same feat? ...

Continue reading: Cracking the Code: Unveiling the Hidden Language of USB HID Keyboards!

View Details

I have blogged about weighted sampling before. There I found that the stochastic acceptance method suggested by Lipowski and Lipowska (2012) (also at https://arxiv.org/abs/1109.3627) is very promising:

// [[Rcpp::depends(dqrng,BH,sitmo)]]

include &...

Continue reading: Roulette-wheel selection for dqrng (part 2)

View Details

Join our workshop on Introduction to Qualitative Comparative Analysis (QCA) using R, which is a part of our workshops for Ukraine series! Here’s some more info: Title: Introduction to Qualitative Comparative Analysis (QCA) using R Date: Thursday, October 5th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone) Speaker: Ingo Rohlfing, Ingo Rohlfing ...

Continue reading: R-posts.com 2023-09-09 04:02:52

View Details

Currently the dqrng package supports only xoroshiro128+ and xoshiro256+ from https://prng.di.unimi.it/ (see also Blackman and Vigna 2021). These RNGs should only be used for creating floating point numbers, which was the case for dqrng originally. H...

Continue reading: Choosing a new default RNG for dqrng

View Details

The R Validation Hub – a working group established within the R Consortium to support the adoption of R within a biopharmaceutical regulatory setting – held a two-part mini-series about...

The post R Validation Hub’s {riskassessment} Application – Mini Series Part 2 appeared first on R Consortium.

Continue reading: R Validation Hub’s {riskassessment} Application – Mini Series Part 2

View Details

The R Validation Hub – a working group established within the R Consortium to support the adoption of R within a biopharmaceutical regulatory setting – held a two-part mini-series about...

The post R Validation Hub’s Risk Metric Application and Risk Score – Mini Series Part 1 appeared first on R Consortium.

Continue reading: R Validation Hub’s Risk Metric Application and Risk Score – Mini Series Part 1

View Details

Authors: Ivan Svetunkov, John E. Boylan Journal: International Journal of Production Economics Abstract: Inventory decisions relating to items that are demanded intermittently are particularly challenging. Decisions relating to termination of sales of product often rely on point estimates of the mean demand, whereas replenishment decisions depend on quantiles from interval ...

Continue reading: iETS: State space model for intermittent demand forecasting

View Details

Tombstone Project Overview

I’m working on a project for my father that will culminate in a website for his genealogy research. There are a couple of different parts that I’m working on independently. In a previous part of the project, I matched a...

Continue reading: An Introduction to Web Scraping Using rvest

View Details

Introduction

Are you interested in visualizing demographic data in a unique and insightful way? Population pyramids are a fantastic tool for this purpose! They allow you to compare the distribution of populations across age groups for different ...

Continue reading: Creating Population Pyramid Plots in R with ggplot2

View Details

The book can be downloaded for free but you will need a Leanpub account, same if you buy it.

The Hitchhiker’s Guide to Linear Models is finally complete. It took me a while to finish it but I’m happy with the result. I hope you enjoy it as ...

Continue reading: The Hitchhiker’s Guide to Linear Models is now complete

View Details

The visualization of a rgl mesh is rather shiny by

default. We’ll see how to make it dull and shadowed.

Take for instance the Barth sextic:

## Barth sextic is the isosurface f=0 ####

phi

Continue reading: A dull and shadowed ‘rgl’ mesh

View Details

We are pleased to announce the full line-up for this year’s Shiny in Production conference! Don’t miss out on this excellent set of talks and workshops - head over to the conference we...

Continue reading: Shiny in Production: Full speaker lineup

View Details

Rhino 1.5 Release We are pleased to announce that Rhino 1.5 is now available on CRAN! This update brings a range of new features and enhancements that aim to make your R development workflow even more efficient. With Rhino’s new addins, you can seamlessly integrate essential tasks into your work and ...

Continue reading: Rhino 1.5.0 Update on CRAN: Streamlining Your R Development Workflow with New Addins

View Details

Introduction

Data visualization is a powerful tool for gaining insights from your data. In R, you have a plethora of libraries and functions at your disposal to create stunning and informative plots. One common task is to plot a subset of your d...

Continue reading: Mastering Data Visualization in R: How to Plot a Subset of Data

View Details

I have been off from the blog lately due to a big load of personal projects. Just lately I got a few days off and found time to work on my personal website, to be ready soon. That made me get more into Nginx configuration, where I consider myself a ...

Continue reading: Adding a website next to your Shiny server

View Details

Coming from the simple sine function (remember of Fourier series), German mathematician Karl Weierstrass became the first to publish an example of a continuous, nowheredifferentiable function. Weierstrass function (originally defined as a Fourier series) was the first instance in which…Read more ›

Continue reading: Little useless-useful R functions – Continuous, nowhere differentiable Weierstrass function

View Details

Moving away from proprietary languages, Roche has made a notable decision to freeze their legacy macros library. With great enthusiasm, they now embrace R as the primary framework for evidence generation in late-stage clinical trials, and they remain open to exploring additional open-source languages in this evolving landscape. James Black, ...

Continue reading: Insights on R Package Quality and Validation for Clinical Trials

View Details

Introduction

When it comes to analyzing multivariate data, Principal Component Analysis (PCA) is a powerful technique that can help us uncover hidden patterns, reduce dimensionality, and gain valuable insights. One of the most informative ways t...

Continue reading: Exploring Multivariate Data with Principal Component Analysis (PCA) Biplot in R

View Details

I really like developing software and making my own life and work easier with it.

But what I enjoy even more is to see others actually use it!

So every now and then I look at CRAN download counts of my R packages.

I’m not in any top-10 rankings ...

Continue reading: CRAN Download counts

View Details

Introduction

‘Space is to place as eternity is to time.’

Joseph Joubert

Greetings, humanists, social and data scientists!

In the realm of data science, the ability to visualize geospatial data is paramount. This is particularly true when wo...

Continue reading: Mapping the Past – Geospatial Visualization in R

View Details

The Berlin R User Group fosters a diverse and vibrant R community in Berlin. Rafael Camargo shared some insights from his experience regarding the potential of R and some anecdotes...

The post Spatial Data Science Using R in Berlin, Germany appeared first on R Consortium.

Continue reading: Spatial Data Science Using R in Berlin, Germany

View Details

Weather and climate are words often used interchangeably in casual conversations. But when a chilly summer breeze sweeps through in July, or when unexpected rains dampen our winter holidays, the age-old debate resurfaces: “Is climate change even real?” If you want to dive deep into the fascinating realm of chaotic ...

Continue reading: Is this still Weather or is it already Climate? Decoding Chaos!

View Details

Hey guys, welcome back to my R-tips newsletter. In today’s R-tip, I’m sharing a super common data science task (one that saved me 20 hours per week)… You’re getting the cheat code to automating Google Sheets. Plus, I’m sharing exactly how I made this a...

Continue reading: ChatGPT: How to automate Google Sheets in under 2 minutes (with R)

View Details

Intro/Overview to TidyTuesday 36: Union Membership in the United States

This week’s TidyTuesday presents data taken from the Union Membership and Coverage Database from the CPS (Unionstats.com) created by Barry T. Hirsch, David A. Macpherson, and William E. Even. This database contains data about the wages of union ...

Continue reading: TidyTuesday 36: Visualizing Worker Demographic Information with Treemaps

View Details

A tutorial on modularizing existing R Shiny apps - how and why do it and what to keep in mind. Differs from many other tutorials in that it follows a top-down approach: instead of using an example of writing a simple modular app from scratch, I took a complex real-life ...

Continue reading: How To Modularize an Existing Shiny App

View Details

We are pleased to open the call for the second cohort of Champions and Mentors for the rOpenSci Champions Program 🎉 ! This program will continue to support our goal of identifying, recognizing, and rewarding passionate community members who help the c...

Continue reading: Open Call for rOpenSci Champions Program Applications!

View Details

Introduction

In the world of data analysis, understanding the relationship between variables is crucial. One powerful tool for measuring this relationship is the covariance. Today, we’ll explore the cov() function in R and delve into the fascina...

Continue reading: Covariance in R with the cov() Function

View Details

In this tutorial, we will show you how to apply the Benjamini-Hochberg procedure in order to calculate the False Discovery ... Read moreThe Benjamini-Hochberg procedure (FDR) and P-Value Adjusted Explained

Continue reading: The Benjamini-Hochberg procedure (FDR) and P-Value Adjusted Explained

View Details

The datasetR helps you generate a random datasets for your R project. It provides a preset random list of values with different data types (interval, ordinal, nominal values). It includes also a function for imputing a NULL, NA or missing values. Installing…Read more ›

Continue reading: datasetR – R package for creating datasets

View Details

Introduction

As a programmer, working with data is a crucial aspect of our work. In R, there are numerous functions available that simplify data analysis tasks. One such function is colMeans(), which allows us to calculate the mean of columns in...

Continue reading: Exploring Data with colMeans() in R: A Programmer’s Guide

View Details

Introduction

“Historians offer us systems of the past that are too complete, series of causes and effects that are too exact and too clear to have ever been entirely true.”

Marguerite Yourcenar - Mémoires d`Hadrien (1974)

Greetings, humanists,...

Continue reading: Uncovering History with R – A Look at the HistData Package

View Details

New arXiv working paper introducing conditional method agreement trees (COAT) which can capture the dependency of a Bland-Altman analysis on covariates. It is ccompanied by an R implementation in the CRAN package coat.

Citation

...

Continue reading: Tree models for assessing covariate-dependent method agreement

View Details

Presence probability, typically obtained with presence-(pseudo)absence modelling methods like GLM, GAM, GBM or Random Forest, is conditional not only on the suitability of the environmental conditions, but also on the general prevalence (proportion of presences) of the species in the … Continue reading →

Continue reading: Weighted probability vs. favourability

View Details

Introduction

In the realm of programming, R is a widely-used language for statistical computing and data analysis. Within R, there exists a powerful function called identical() that allows programmers to compare objects for exact equality. In th...

Continue reading: A Closer Look at the R Function identical()

View Details

With the help of the internet and ChatGPT, I was able to create this interactive election map showing the top five candidates with the most votes in each ward. It took longer than expected, but it was fun to learn new things, such as working with JSON in R and ...

Continue reading: Create interactive 2023 Toronto mayoral election map in R with leaflet

View Details

Join our workshop on An Open Source Framework for Choice Based Conjoint Experiments in R, which is a part of our workshops for Ukraine series! Here’s some more info: Title:An Open Source Framework for Choice Based Conjoint Experiments in R Date: Thursday, August 10th, 18:00 – 20:00 CEST (Rome, Berlin, Paris ...

Continue reading: An Open Source Framework for Choice Based Conjoint Experiments in R workshop

View Details

A while ago we looked at Geelong and the curse of the bye. And since the AFL media have outdone themselves this year with “curse of the bye” articles: see for example here, here, here and here, I decided to revisit the topic in more depth. If you like that ...

Continue reading: The “curse of the bye” revisited

View Details

We are pleased to announce that Rhino 1.4 is now available on CRAN! This update simplifies dependency management in your project, allows you to integrate React seamlessly into your app, and resolves some issues with {box} modules. Read on to learn more! Simplified Dependency Management Rhino relies on {renv} for managing ...

Continue reading: Rhino 1.4.0 Update on CRAN: Streamlined Dependency Management and React Support

View Details

HTTP Headers Hashing (HHHash) is a technique developed by Alexandre Dulaunoy to generate a fingerprint of an HTTP server based on the headers it returns. It employs one-way hashing to generate a hash value from the list of header keys returned by the server. The HHHash value is calculated by ...

Continue reading: New R Package For HTTP Headers Hashing

View Details

Summary

A qualitative overview of the midfieldr package and its application following its initial CRAN release. midfieldr provides tools and recommended methods for working with individual undergraduate student-level records (registrar’s ...

Continue reading: midfieldr v1.0.1

View Details

I translated to R the JavaScipt code for the Doyle spiral written by

Robin Houston and kindly provided in

this gist.

library(rootSolve) # for the function multiroot

library(plotrix) # for ...

Continue reading: The Doyle spiral with R

View Details

Lynn (of TITAA and general NLP wizardy fame) was gracious enough to lend me a Bluesky invite, so I could claim my handle on yet-another social media site. I’m still wary of it (as noted in one of this week’s Drops), but the AT protocol — whilst super (lacking ...

Continue reading: Poor Dude’s Janky Bluesky Feed Reader CLI Via R & Python

View Details

I’ve been learning at least one new programming language a month through

Exercism which has been really fun and interesting. I frequently say that “every language you learn teaches you something about all the

others you know” and with nearly a dozen under my belt so far I’m ...

Continue reading: Array Languages: R vs APL

View Details

Earlier this year, two data scientists from Jumping Rivers ran an outreach activity for 14-19 year olds across the UK, in collaboration with the youth charity Speakers for Schools.

The three hour ...

Continue reading: Changing the world with Data: An outreach event

View Details

Andy Thomason, code performance consultant and lecturer at the University of London covering programming, physics and AI courses focused on game development, created an open source project to add Rust’s...

The post R Consortium Funded Project Extendr Provides Rust Extensions for R appeared first on R Consortium.

Continue reading: R Consortium Funded Project Extendr Provides Rust Extensions for R

View Details

Join our workshop on Using Spatial Data with R Shiny, which is a part of our workshops for Ukraine series! Here’s some more info: Title: Using Spatial Data with R Shiny Date: Thursday, August 3rd, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone) Speaker: Michael C. Rubin, is an Engineer, MIT Data ...

Continue reading: Using Spatial Data with R Shiny workshop

View Details

To be successful as a Data Scientist, you’re often put in positions where you need to find groups within your data. One key business use-case is finding clusters of customers that behave similarly. And that’s a powerful skill that I’m going to help you...

Continue reading: A Gentle Introduction to K-Means Clustering in R (Feat. Tidyclust)

View Details

A brief tutorial on the easiest - albeit limited - way to theme Shiny apps using CSS tags. Should be enough for most apps; also works if you need to make ad-hoc changes to a premade (e.g. corporate) theme used in your app.

Continue reading: An Easy Way to Customize Your Shiny App Theme

View Details

The R Validation Hub recently had its community meeting after a brief hiatus. The team discussed announcements, common challenges, and brainstormed ideas for possible future projects through the R Validation...

The post R Validation Hub Community Meeting – June Recap ↺ appeared first on R Consortium.

Continue reading: R Validation Hub Community Meeting – June Recap ↺

View Details

the previous post, I showed how to expand a polynomial with symbolic parameters with the

help of the spray package. As I said, it has one

problem: it doesn’t preserve the rational numbers in the polyno...

Continue reading: Fast expansion of a polynomial with R – part 2

View Details

Embark on your programming odyssey with our extensive range of courses! Never written a line of code in your life? No stress - we offer a mix of introductory courses for beginners as well as more ...

Continue reading: July Training Update

View Details

I’m making freely available the first version of something I’ve wanted to do for a long time. I’m an advanced R user, but I’m very limited in both, Python and Julia. However, I’ve been reading reports that Julia is a pretty efficient programming language (here ...

Continue reading: CheatSheet for coding in R, Python and Julia

View Details

Today’s TidyTuesday is about historical markers with the data coming from the Historical Marker Database. I’m going to add to the map that I made last week with information about Historic Districts in Arlington, VA. I’m going to make an interactive leaflet map with the new information ...

Continue reading: TidyTuesday Week 27: Historical Markers

View Details

this previous post,

this previous post, and

this previous post, I showed how to expand a polynomial with symbolic parameters. In the

first two posts, I used the R package caracas, a

wra...

Continue reading: Fast expansion of a polynomial with R

View Details

R and Shiny Training: If you find this blog to be interesting, please note that I offer personalized and group-based training sessions that may be reserved through Buy me a Coffee. Additionally, I provide training services in the Spanish language ...

Continue reading: Credible intervals with Capybaras, R and Stan (rstan and cmdstanr interfaces)

View Details

Continue reading: Credible intervals with Capybaras, Python and Stan (cmdstanpy interface)

View Details

Meanders, monotone meanders, and Baxter permutations

Versions of the following question can be traced back at least to the work of Henri Poincaré (Poincaré (1912)):

In how many ways a simple loop in the plane can cross a line a specified number of ...

Continue reading: The scaling limit of Baxter permutations

View Details

Continue reading: parallelly: Querying, Killing and Cloning Parallel Workers Running Locally or Remotely

View Details

I am so excited to announce that I have a new book (in collaboration with Packt) that has just been launched, written with my co-author, the wonderful Matt Jackson. This books takes you from a nascent understanding of graph data science methods to a more confident user, all in under 250 ...

Continue reading: Graph Data Modeling with Python

View Details

Introduction

Managing files is an essential task for any programmer, and when working with R, the file.rename() function can become your best friend. In this blog post, we’ll explore the ins and outs of file.rename(), discuss its syntax, provide...

Continue reading: Simplifying File Management in R: Introducing file.rename()

View Details

How to join this free online event with Beatriz Milz, Carolina Pradier and Victor Ordu.

Champions programs are designed to identify, recognize, and reward emerging leaders within a community. The rOpenSci Champions Program is part of a series of activities and projects we are carrying out to ensure our research ...

Continue reading: Mentoring & training program for Scientific Open Source Champions

View Details

Introduction

Using a Windows .bat file to execute an R script can be a convenient way to automate tasks and streamline your workflow. In this blog post, we will explain each line of a sample .bat file and its corresponding R script, along with a...

Continue reading: How to Use a Windows .bat File to Execute an R Script

View Details

Join our workshop on Machine Learning workflow with tidymodels in R, which is a part of our workshops for Ukraine series! Here’s some more info: Title: Machine Learning workflow with tidymodels in R Date: Thursday, July 27th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone) Speaker: Kelly Bodwin is an Associate Professor ...

Continue reading: Machine Learning workflow with tidymodels in R workshop

View Details

In today’s rapidly evolving technology landscape, IT professionals need to stay ahead of the curve to remain competitive and relevant in their careers. That’s why, at Appsilon, we do our best to create opportunities for personal and professional development for all our team members. We foster a culture ...

Continue reading: Nurturing Growth: Advancing Your IT Career in the Appsilon Tech Team

View Details

How to join this free online event with Stephanie Hazlitt, Jonathan Keane and Nic Crane.

Apache Arrow is a software development platform for building high performance applications that process and transport large data sets. It is designed to improve the performance of data analysis methods, and to increase the efficiency ...

Continue reading: A Journey through Arrow in R

View Details

Introduction

In the world of data analysis and statistics, grouping data based on certain criteria is a common task. Whether you’re working with large datasets or analyzing trends within smaller subsets, having a reliable and efficient tool for ...

Continue reading: The ave() Function in R

View Details

Continue reading: %dofuture% – a Better foreach() Parallelization Operator than %dopar%

View Details

Mustafa Cavus, organizer of the Eskisehir R User Group, in Turkey, discussed the diverse and thriving R community in Eskisehir. He shared the details of a 4-day event hosted by...

The post R for Predictive Modeling and Data Visualization in Turkey appeared first on R Consortium.

Continue reading: R for Predictive Modeling and Data Visualization in Turkey

View Details

Introduction

Welcome to the world of data visualization in R! In this blog post, we will explore the abline() function, a versatile tool that allows you to add straight lines to your plots effortlessly. Whether you’re a beginner or an experience...

Continue reading: Visualization in R: Unleashing the Power of the abline() Function

View Details

Over a year ago, while listing to a very not-at-all-statistical podcast, I discovered that Bayesian modeling is widely used in archaeology since the mid 90s to calibrate carbon dating.1

Carbon dating is a scientific method used to determine the a...

Continue reading: Order Constraints in Bayes Models (with brms)

View Details

Whereas the direction of main effects can be interpreted from the sign of the estimate, the interpretation of interaction effects often requires plots. This task is facilitated by the R package sjPlot (Lüdecke, 2022). For instance, using the plot_model function, I plotted the interaction between two continuous variables.

``` library(lme4)

> Loading required package: Matrix

library(sjPlot) library(ggplot2)

theme_set(theme_sjplot())

Create data using code by Ben Bolker from

https://stackoverflow.com/a/38296264/7050882

set.seed(101) spin = runif(600, 1, 24) reg = runif(600, 1, 15) ID = rep(c("1","2","3","4","5", "6", "7", "8", "9", "10")) day = rep(1:30, each = 10) testdata <- data.frame(spin, reg, ID, day) testdata$fatigue <- testdata$spin * testdata$reg/10 * rnorm(30, mean=3, sd=2)

fit = lmer(fatigue ~ spin * reg + (1|ID), data = testdata, REML = TRUE)

plot_model(fit, type = 'pred', terms = c('spin', 'reg'))

> Warning: Ignoring unknown parameters: linewidth

```

...

Continue reading: How to break up colour variable in sjPlot into equally-sized bins

View Details

This interesting paper came out recently: A test of the predictive validity of relative versus absolute income for self-reported health and well-being in the United States, by David Brady, Michaela Curran and Richard Carpiano. It uses a large sample of...

Continue reading: Weighted versus unweighted percentiles by @ellis2013nz

View Details

Every R package needs a hex sticker. Here's how you can generate one in seconds using generative AI.

The post How to generate a hex sticker with {openai} and {cropcircles} appeared first on Dan Oehm | Gradient Descending.

Continue reading: How to generate a hex sticker with {openai} and {cropcircles}

View Details

We designed the rOpenSci Champions Program with a mentorship aspect. Mentoring plays a significant role in the growth and development of both mentors and mentees alike. In our program, each Champion has a mentor who accompanies them during their train...

Continue reading: rOpenSci Champions Program Teams: Meet Ghozayel Elotteebi and Zebulun Arendsee

View Details

The R Consortium is sponsoring the New York R Conference, presented by Lander Analytics! This in-person and virtual conference will be running from July 13-14, with workshops (tickets sold separately)...

The post Coming in July! 🔥 We Have a 20% off Promo Code – Data Scientists & Data Professionals at New York R Conference ...

Continue reading: Coming in July! 🔥 We Have a 20% off Promo Code – Data Scientists & Data Professionals at New York R Conference

View Details

We’ve released the newest version of NIMBLE on CRAN and on our website. NIMBLE is a system for building and sharing analysis methods for statistical models, especially for hierarchical models and computationally-intensive methods (such as MCMC and SMC). Version 1.0.1 follows shortly after 1.0.0 and fixes an issue and a bug ...

Continue reading: Version 1.0.1 of NIMBLE released, fixing a bug in version 1.0.0 affecting certain models

View Details

So, thanks to my former boss, and head of direct indexing at BNY Mellon, Vijay Vaidyanathan, and his Coursera course, … Continue reading →

Continue reading: Several Key PerformanceAnalytics Functions From R Now In Python (special thanks to Vijay Vaidyanathan)

View Details

This analysis was co-authored by data scientists, Scott Fisher and Douglas Davila-Pestana. Mr. Fisher developed the Python code for the […]

The post Miami-Dade County Public Employee Salary Research – An Analysis in R, Python, and Julia first appeared on Remix Institute.

The post Miami-Dade County Public Employee Salary Research – An Analysis ...

Continue reading: Miami-Dade County Public Employee Salary Research – An Analysis in R, Python, and Julia

View Details

I haven’t been TidyTuesdaying because I’ve been learning Tableau. I’ll write more about that later; but it has been an interesting experience and provides different perspectives on data compared to what you might get from R. (I’m sure you could re...

Continue reading: TidyTuesday Week 25: UFO Sightings Redux

View Details

Introduction

As a programmer, you’re constantly faced with the task of organizing and analyzing data. One powerful tool in your R arsenal is the xtabs() function. In this blog post, we’ll explore the versatility and simplicity of xtabs() for agg...

Continue reading: Mastering Data Aggregation with xtabs() in R

View Details

About two years ago, someone inquired whether simstudy had the functionality to generate data from a logistic model with a specific AUC. It did not, but now it does, thanks to a paper by Peter Austin that describes a nice algorithm to accomplish thi...

Continue reading: Finding logistic models to generate data with desired risk ratio, risk difference and AUC profiles

View Details

Back in 2018, when I was most involved in developing choroplethr, Joe Rickert, R Consortium Director, recommended that I submit a proposal to the R Consortium to create a Working...

The post Renewing the R Consortium Census Working Group appeared first on R Consortium.

Continue reading: Renewing the R Consortium Census Working Group

View Details

The Naming of Stats is a difficult matter,

It isn’t just one of your holiday games;

You may think at first I’m as mad as a hatter

When I tell you, a stat must have THREE DIFFERENT NAMES.

First of all are the names where usage is informal,

...

Continue reading: The Naming of Stats

View Details

Introduction

Python, a versatile and dynamically-typed language, gained significant enhancements with the introduction of type hinting in Python 3.5.

Often though, there are cases where the specific type of an input and output of a function are not kno...

Continue reading: Leveraging generic type hints of classes in Python

View Details

Simple use of the draw_pac_map() function

Last year I blogged about making a choropleth map of the Pacific, dealing with a few technical details like getting the map centred on the Pacific, using exclusive economic zones (EEZs) to colour things in for...

Continue reading: Simpler drawing of Pacific choropleth maps by @ellis2013nz

View Details

Learn how to control your R Shiny development with a collaborative and agile deployment to production. Continuous Integration (CI) and Continuous Deployment (CD) stand at the basis of a robust, maintainable and collaborative productive code. In this...

Continue reading: ‘Bring a Shiny App to Production’ the hands-on workshop

View Details

We have an analysis routine for proteomics data written for IgorPro. One output is a volcano plot. These plots show the fold change in one sample compared to another and plot that against a p-value to estimate how reproducible any changes observed are. This post is not about that software, ...

Continue reading: Step By Step: recreating a volcano plot in R

View Details

Introduction

As a programmer, it’s crucial to have a deep understanding of the tools at your disposal. In the realm of data analysis and manipulation, R stands as a powerhouse. One function that proves to be invaluable in many scenarios is diff(...

Continue reading: Mastering the Power of R’s diff() Function: A Programmer’s Guide

View Details

Introduction

The lm() function in R is used for fitting linear regression models. It stands for “linear model,” and it allows you to analyze the relationship between variables and make predictions based on the data.

Let’s dive into the parameter...

Continue reading: Introduction to Linear Regression in R: Analyzing the mtcars Dataset with lm()

View Details

Bayesian structural equation model tutorial

Structural equation modelling (SEM) is a tool for exploring

relationships among multiple variables. lavaan and piecewiseSEM are

too of the most widely used and accessible packages for latent variable

modelli...

Continue reading: Bayesian structural equation model tutorial

View Details

Join us for an update from the R Consortium’s R Validation Hub, which supports the use of R within regulated industries. The community meeting will be on June 27, 12...

The post Learnings and Reflection from Case Studies: What is Next for the R Validation Hub? appeared first on R Consortium.

Continue reading: Learnings and Reflection from Case Studies: What is Next for the R Validation Hub?

View Details

The R Consortium recently talked to Rachel Heyard of the Zürich R User Group to discuss the vibrant R community in Zürich. The group collaborates with different companies in Zürich...

The post Use of R for Meta-Research in Zürich appeared first on R Consortium.

Continue reading: Use of R for Meta-Research in Zürich

View Details

Learn how to use supervised text classification in R! Join our workshop on Introduction to Supervised Text Classification in R which is a part of our workshops for Ukraine series. Here’s some more info: Title: Introduction to Supervised Text Classification in R Date: Thursday, July 13th, 18:00 – 20:00 CEST (Rome, Berlin, ...

Continue reading: Introduction to Supervised Text Classification in R workshop

View Details

As we anticipated in November 2022 the changes implemented by Twitter have generated a less safe and friendly space for our community. That is why from June 2023, we will stop interacting on this platform. We will maintain the account in hopes that we...

Continue reading: rOpenSci’s Communication Channels for Safe and Friendly Exchange

View Details

Introduction

As a programmer, you may come across various scenarios where you need to create complex model formulas in R. However, constructing these formulas can often be challenging and time-consuming. This is where the ‘reformulate()’ functio...

Continue reading: Simplifying Model Formulas with the R Function ‘reformulate()’

View Details

The R Consortium has begun the disbursement of grants to help R User Groups (RUGS) around the world organize, share information, and support each other. We are currently accepting applications...

The post R Consortium Funding for R User Groups! Highlighting R-Ladies São Paulo appeared first on R Consortium.

Continue reading: R Consortium Funding for R User Groups! Highlighting R-Ladies São Paulo

View Details

Continue reading: rOpenSci Champions Program Teams: Meet Cheryl Isabella Lim and Mauro Lepore

View Details

Thomas Lumley just dropped a blog post, Blank cheque inheritance and statistical objects, which begins as follows. One of the problems with object-oriented programming for statistical methods is that inheritance is backwards. Everything is fine for data structures, and Bioconductor … Continue reading →

Continue reading: R and OOP anti-patterns

View Details

Best practices for a robust and maintainable shiny app, a hand on workshop on 21/06. Do you know how to build a basic Shiny web application, but would you like to bring your Shiny development to the next level? Learn from professional experts how to...

Continue reading: ‘Advanced Shiny Development’ the hands-on workshop

View Details

Introduction

In R, the file.info() function is a useful tool for retrieving file information, such as file attributes and metadata. It allows programmers to gather details about files, including their size, permissions, and timestamps. In this p...

Continue reading: Understanding the file.info() Function in R: Listing Files by Date

View Details

I’ve updated The Popularity of Data Science Software‘s market share estimates based on scholarly articles. I posted it below, so you don’t have to sift through the main article to read the new section. Scholarly Articles Scholarly articles provide a rich source of information about data science ...

Continue reading: Update to Data Science Software Popularity

View Details

A riddle on uniform spacings!, namely when considering eight iid Uniform (0,1) variates as visiting times and three further iid Uniform (0,1) variates as server availability times, with unit service time, the question being the probability a server is available for a ninth visiting time, T⁹. Which can be decomposed into four ...

Continue reading: uniform spacings

View Details

"R Python" continued... Geographic SHAP

Continue reading: Geographic SHAP

View Details

Learn how to use propensity score analysis in R! Join our workshop on Introduction to Propensity Score Analysis with R which is a part of our workshops for Ukraine series. Here’s some more info: Title: Introduction to Propensity Score Analysis with R Date: Thursday, July 6th, 18:00 – 20:00 CEST (Rome, Berlin, ...

Continue reading: Introduction to Propensity Score Analysis with R workshop

View Details

The 49th G7 summit was held recently in Japan. Ukraine was one of the most critical issues at the meeting; most of the session topic was related to problems stemming from Russia’s invasion of Ukraine. One of the problems, aforementioned is food security. Because of the war, energy prices ...

Continue reading: JAGS Simulation with Multivariate State-Space Model: The G7 on Food Security

View Details

Francisco Rodríguez Sánchez shared with the R Consortium how the R language has changed and improved his work as a computational ecologist, leading him to become a passionate advocate, professional,...

The post Bridging Ecology, Statistics, and Data Science with R for Biodiversity and Climate Change Research appeared first ...

Continue reading: Bridging Ecology, Statistics, and Data Science with R for Biodiversity and Climate Change Research

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn the world of data analysis and manipulation, tidying and reshaping data is often an essential step. R’s tidyr library provides powerful tools to efficiently transform and reshape data. One such function is pivot_longer(). In this blog post, we’ll explore how pivot_longer() works and demonstrate its usage through several examples. By the end, you’ll have a solid understanding of how to use this function to make your data more manageable and insightful.

The tidyr library holds the function, so we are going to have to load it first.

library(tidyr) Understanding pivot_longer()The pivot_longer() function is designed to reshape data from a wider format to a longer format. It takes columns that represent different variables and consolidates them into key-value pairs, making it easier to analyze and visualize the data.

Syntax: The basic syntax of pivot_longer() is as follows:

pivot\_longer(data, cols, names\_to, values\_to) * data: The data frame or tibble to be reshaped. * cols: The columns to be transformed. * names_to: The name of the new column that will hold the variable names. * values_to: The name of the new column that will hold the corresponding values.

Example 1: Reshaping Wide Data to Long DataLet’s start with a simple example to demonstrate the usage of pivot_longer(). Suppose we have a data frame called students with columns representing subjects and their respective scores:

students <- data.frame( name = c("Alice", "Bob", "Charlie"), math = c(90, 85, 92), science = c(95, 88, 91), history = c(87, 92, 78)) To reshape this data from a wider format to a longer format, we can use pivot_longer() as follows:

students\_long <- pivot\_longer( students, cols = -name, names\_to = "subject", values\_to = "score" )students\_long

```

A tibble: 9 × 3 name subject score 1 Alice math 902 Alice science 953 Alice history 874 Bob math 855 Bob science 886 Bob history 927 Charlie math 928 Charlie science 919 Charlie history 78

`` The resultingstudents_longdata frame will have three columns:name,subject, andscore`, where each row represents a student’s score in a specific subject.

Example 2: Handling Multiple Variables In many cases, data frames contain multiple variables that need to be pivoted simultaneously. Consider a data frame called sales with columns representing sales figures for different products in different regions:

sales <- data.frame( region = c("North", "South", "East"), product\_A = c(100, 120, 150), product\_B = c(80, 90, 110), product\_C = c(60, 70, 80)) To reshape this data, we can specify multiple columns to pivot using pivot_longer():

sales\_long <- pivot\_longer( sales, cols = starts\_with("product"), names\_to = "product", values\_to = "sales" )sales\_long

```

A tibble: 9 × 3 region product sales 1 North product_A 1002 North product_B 803 North product_C 604 South product_A 1205 South product_B 906 South product_C 707 East product_A 1508 East product_B 1109 East product_C 80

`` The resultingsales_longdata frame will have three columns:region,product, andsales`, where each row represents the sales figure of a specific product in a particular region.

Example 3: Handling Irregular DataSometimes, data frames contain irregular structures, such as missing values or uneven numbers of columns. pivot_longer() can handle such scenarios gracefully. Consider a data frame called measurements with columns representing different measurement types and their respective values:

measurements <- data.frame( timestamp = c("2022-01-01", "2022-01-02", "2022-01-03"), temperature = c(25.3, 27.1, 24.8), humidity = c(65.2, NA, 68.5), pressure = c(1013, 1012, NA)) To reshape this data, we can use pivot_longer() and handle the missing values:

measurements\_long <- pivot\_longer( measurements, cols = -timestamp, names\_to = "measurement", values\_to = "value", values\_drop\_na = TRUE )measurements\_long

```

A tibble: 7 × 3 timestamp measurement value 1 2022-01-01 temperature 25.32 2022-01-01 humidity 65.23 2022-01-01 pressure 1013 4 2022-01-02 temperature 27.15 2022-01-02 pressure 1012 6 2022-01-03 temperature 24.87 2022-01-03 humidity 68.5

`` The resultingmeasurements_longdata frame will have three columns:timestamp,measurement, andvalue, where each row represents a specific measurement at a particular timestamp. Thevalues_drop_na` argument ensures that rows with missing values are dropped.

ConclusionIn this blog post, we explored the pivot_longer() function from the tidyr library, which allows us to reshape data from a wider format to a longer format. We covered the syntax and provided several examples to illustrate its usage. By mastering pivot_longer(), you’ll be equipped to tidy your data and unleash its true potential for analysis and visualization.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simplifying Data Transformation with pivot_longer() in R’s tidyr Library

View Details

[This article was first published on Maëlle's R blog on Maëlle Salmon's personal website, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’m happy to report that I thought “oh but I know a better way to write that code!” a few times lately when reading old scripts of mine, or scripts by others. It’s a good feeling because it shows progress! I’ve tooted about all three things I’ll present in this post: After reading Julia Evans’ post about blogging, I decided to train the blogging muscle a bit using these low-hanging fruits/toots1.

Combine a list of default values with a list of custom valuesToot

Imagine

you have a default_values list and want to let the user pass a custom_values list to override some of them.

utils::modifyList(default_values, custom_values)does that!

So say you had code à la

default\_values <- list(a = 1, b = 2)options <- list(b = 56)temporary\_list <- default\_valuestemporary\_list[names(options)] <- optionsoptions <- temporary\_listoptions#> $a#> [1] 1#> #> $b#> [1] 56 You can write it like so

default\_values <- list(a = 1, b = 2)options <- list(b = 56)options <- modifyList(default\_values, options)options#> $a#> [1] 1#> #> $b#> [1] 56 I learnt about that function in pkgdown source.

Use a default if the user provided NULLToot

Do you know the rlang %||% operator?2

Code like

if (is.null(blop)) { blop <- 42} can become

blop <- blop %||% 42 Related to this, I’d recommend package developers read the chapter of the Tidyverse design guide on defaults, especially the section on the NULL default.

Extract common values or different / unique values from two vectorsToot

Say I have a vector a and a vector b, and I need the unique a values that are not in b.

a <- c("thing", "object")b <- c("thing", "gift") I tended to write something like

unique(a[!(a %in% b)])#> [1] "object" (or without the unique() if a has only distinct values)

that can be

setdiff(a, b)#> [1] "object" Similarly, when looking for the unique values of the two vectors combined, instead of

unique(c(a, b))#> [1] "thing" "object" "gift" I can write

union(a, b)#> [1] "thing" "object" "gift" Because I’ve noticed I didn’t know these base R functions well enough, I open the Set Operations manual page more often, by typing ?setdiff for instance.

Salix Dubois helpfully noted the functions can be slower, and that one might not always want to drop duplicates.

ConclusionIn this post I presented three basic (set of) functions (not all base functions) that I’ve found serve me well: utils::modifyList(), rlang::%||% and base Set Operations. I’m glad they’re now part of my R vocabulary.

Note that you might still prefer the longer version of some of these patterns, depending on your needs, your code readers, etc. I won’t judge!

I’m curious to see what three new things I’ll have learnt in a few months (and will try not to beat myself up for not learning about them sooner ). If you’re interested about code quality in general, you might enjoy this post by Christophe Dervieux and myself on the R-hub blog.

I hope it’s a myth that your puns need to be good! ︎
Please don’t ask me to say this aloud, I have no idea how it’s pronounced. ︎

To leave a comment for the author, please follow the link and comment on their blog: Maëlle's R blog on Maëlle Salmon's personal website.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Three useful (to me) R patterns

View Details

[This article was first published on R – Statistical Modeling, Causal Inference, and Social Science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I (Jonah) am excited to be teaching a 2-day Stan workshop preceding the NYR Conference in July. The workshop will be July 11-12 and the conference July 13-14. The focus of the workshop will be to introduce the basics of applied Bayesian data analysis, the Stan modeling language, and how to interface with Stan from R. Over the course of two full days, participants will learn to write models in the Stan language, run them in R, and use a variety of R packages to work with the results.

There are both in-person and remote spots available, so if you can’t make it to NYC you can still participate. For tickets to the workshop and/or conference head to https://rstats.ai/nyr.

To leave a comment for the author, please follow the link and comment on their blog: R – Statistical Modeling, Causal Inference, and Social Science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Stan class at NYR Conference in July (in person and virtual)

View Details

[This article was first published on Rtask, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You can read the original post in its original format on Rtask website by ThinkR here: Mastering file download in shiny

But why does downloadHandler now return an empty file!?

ContextWhen we start working with {shiny}, we often reach a point where it is necessary to offer the user the option to download a document generated by the application.
This document can be a PDF report, a PNG or JPEG image, or anything else.

The appropriate function for this purpose is shiny::downloadHandler(), but beyond the (relatively) simple case shown in the documentation’s example, when it comes to using the function in a more complex scenario, it is not uncommon to encounter difficulties, even for an experienced developer.

Therefore, I would like to share with you an approach that we use at ThinkR, which handles both straightforward and more complex cases that we sometimes encounter.

TL;DR:

We will consistently use file.copy in the downloadHandler

Exemple 1Let’s look at a simple scenario together by starting with the example from the R documentation and making it more “robust”.

Before : ui <- fluidPage( downloadButton(outputId = "downloadData",label = "Download"))server <- function(input, output) { # Our dataset data <- mtcars output$downloadData <- downloadHandler( filename = function() { paste("data-", Sys.Date(), ".csv", sep="") }, content = function(file) { write.csv(data, file) } )}shinyApp(ui, server) After : ui <- fluidPage( downloadButton(outputId = "downloadData",label = "Download"))server <- function(input, output) { local <- reactiveValues(data = mtcars, export\_file = NULL ) observeEvent(local$data,{ out <- tempfile(fileext = ".csv") write.csv(x = local$data,file = out) local$export\_file <- out }) output$downloadData <- downloadHandler( filename = function() { paste("data-", Sys.Date(), ".csv", sep="") }, content = function(file) { file.copy( from = local$export\_file, to = file ) } )}shinyApp(ui, server) Do you see the difference? We regain control over the file that will be placed behind the button. It is created here within an observeEvent, making it easy to debug and understand its content.

Example 2This approach makes complex scenarios more readable :

library(tidyverse)library(shiny)library(flextable)ui <- fluidPage( downloadButton(outputId = "downloadData",label = "Download"))server <- function(input, output) { local <- reactiveValues(data = mtcars, export\_file = NULL ) observeEvent(local$data,{ out <- tempfile(fileext = ".png",pattern = "table\_") # write.csv(x = local$data,file = out) local$data %>% flextable::flextable() %>% flextable::save\_as\_image(path = out) if (file.exists(out)){ local$export\_file <- out } else{ local$export\_file <- NULL } }) output$downloadData <- downloadHandler( filename = function() { basename(local$export\_file) }, content = function(file) { file.copy( from = local$export\_file, to = file ) } )}shinyApp(ui, server) It’s your turn

This post is better presented on its original ThinkR website here: Mastering file download in shiny

To leave a comment for the author, please follow the link and comment on their blog: Rtask.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Mastering file download in shiny

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Motivation and key pointsSo this tweet came across my feed. It refers to this article, Statistical Control Requires Causal Justification by Wysocki, Lawson and Rhemtulla, published in June last year (2022) in Advances in Methods and Practices in Psychological Science. I was struck by the clarity of Figure 4 from that paper. It is such a great tool for at once explaining the difference between a confounder, collider and mediator and showing how they have different impacts on a regression; and hence lead to different decisions about whether to include the confounder, collider or mediator variable in the model. What I’m showing here isn’t the original, but my own version of it, which I’ll eventually come to describing how I did it later in this post:

Here is the original caption for Figure 4 from Wysocki et al:

“Partial and simple regression coefficients under three causal structures. In each graph, the x-axis depicts the population value of the direct effect that connects the control variable and the predictor (a), and the y-axis depicts the value of the regression coefficient of Y on X. The direct effect of X on Y and the value of the direct effect connecting Y and C are held constant across the results. Solid lines represent the partial (yellow line) and simple (green line) regression coefficients. The dashed line represents the total X → Y causal effect.”

The idea is that in all three cases we are interested in the degree to which X causes Y. A third variable C is also present and is correlated with both X and Y. Should we ‘control’ for it, ie include it as another variable on the right hand side of a regression? The answer is, it depends on whether C is a confounder, collider or mediator – and there will be no way to tell from your data which it is. So some kind of reasoning based on prior knowledge and theorising is required, or making (and documenting) some explicit assumptions. Some sciencing is required before you do the actual model specification and fitting.

The basic interpretation of these terms is as follows. For illustrative purposes I am going to use examples from a hypothetical situation where we want to understand the impact of education on income; that is, X is education and Y is income:

C is a confounder if it impacts on both X and Y. For example, gender is known to lead to different education outcomes, and it also leads to differences in income in various complicated ways. Another example might be innate ability, if such a thing exists and could be measured. You want to control for confounders, because if you don’t include them in the regression, some of the coefficient in front of X is going to be picking up causal impacts that are really down to the confounder. In the chart on the right of the figure we see that the green ‘simple coefficient’ – the coefficient we’d estimate for X in a simple regression of Y ~ X – gets the wrong result. The correct value of the impact of X on Y is 0.15, and to get this you need the yellow ‘partial coefficient’ ie the coefficient for X you get when you control for C, in a regression of Y ~ X + C. How far out the simple coefficient is depends on the size of a, the causal effect of C on X; if a is zero then both methods get you the same answer
C is a collider is when both X and Y impact on C, but it itself does not impact on either of them. I find it harder to think of pure examples of these, but in my case I’m going to use hobbies as one. The hobbies you’ve got definitely depend on what you were exposed to in your education; and whether you can afford them financially and timewise will depend on your income (in complicated ways). True, hobbies will also potentially impact on your income but I’m hoping (for purposes of illustration) in ways that are less important than the Y -> C part of the diagram here. You don’t want to control for colliders in a regression (see how the green ‘simple coefficient’ line correctly estimates the true causality of X -> Y, whereas the yellow ‘partial coefficient’ line doesn’t). The intuition for this is that because C doesn’t really impact on Y, when you include it in the regression some of the true impact of X on Y is falsely attributed to C just because of its correlation with both the important variables. (Side note – I’m not thrilled with the example of ‘hobbies’ here, but it serves the point. I also considered ‘suburb you live in’ and ‘sports team you support’, none of which are pefect either.)
C is a mediator is when X causes C and C causes Y. A good example in our case could be occupation; your education has a big impact on your occupation, which in turn has a big impact on your income. You don’t want to control for a mediator if you are interested in the full effect of X on Y! Because a huge part of how X impacts on Y is precisely through the mediation of C, in our case choice of and access to occupation, given a certain level of education. If you ‘control’ for occupation you will be greatly underestimating the importance of education.

Incidentally this last point is behind one of the great perpetual culture war debates, whether to control for occupation and experience when estimating the gender pay gap. Much of the gender pay gap goes away when we do so, so does this mean there is no problem here? No, because occupation and experience are mediator variables which are precisely the means by which gender leads to a pay gap. By controlling for them you are missing out on the actual mechanism of precisely in what you are interested.

However there is a time when you might be interested in controlling for occupation and experience in a regression of pay gap on gender. This is when you are seeking to measure the direct impact, and only the direct impact, of gender in pay decisions – most likely in the context of gender discrimination at the final stage of remuneration determination. In this case, occupation and experience are no longer mediators, they are if anything confounders, because X is no longer “gender” but “gender discrimination in the current workplace making a pay decision” (for which we use actual, observed gender as a proxy). Now C impacts on Y and might impact on X. So if you want to estimate that final “equal pay for equal work” step of the chain then yes it is legitimate to control for occupation and experience. Just don’t mix this up with the total impact of gender on the pay gap, which is mediated through occupation and experience.

Simulating dataSo I wanted to simulate this data fitting these three types of relationships, and re-create the original Figure 4 from Wysocki et al’s paper. I think this is pretty simple, so long as I am happy with getting the substance right and not matching exact details. First, I made three functions, one each for the situation where the nuisance variable is a confounder, collider or mediator.

In the code below I call the nuisance variable (which is C in the original diagrams) z to avoid creating a conflict with the C() or c() functions in R (this would be completely workable, but goes against highly ingrained habits on my part to not use frequently-used base R functions as names for other objects). Each of the three functions starts the data generation with the one variable that is exogenous to the system – isn’t causally impacted on by either of the other two. Then it uses that variable to generate the variable that is impacted on by the first variable, and then finally creates the final variable. Each variable has a standard (ie mean = 0, variance = 1) Gaussian randomness built in, and the relationship between variables has simple linear coefficients but no intercept terms. For example, in the confounder example, Y = b1.X + b2.C + noise.

Post continues after R code

```

attempt to recreate Figure 4 from# https://journals.sagepub.com/doi/10.1177/25152459221095823library(tidyverse)library(ggdag)library(patchwork)#----------------simulating data-------------# In the below I call a variable `z` that in the diagrams is called `C`#' Simulate a 3 variable situation where a nuisance variable is a confounder#' #' @param zx coefficient for impact of z on x#' @param xy coefficient for impact of x on y#' @param zy coefficient for impact of z on y#' @param n sample size#' @param seed random seed set for reproducibility#' @returns a tibble of three variables x, y and z. x causes y #' and z is a confounder ie it impacts on both x and ysim_confounder <- function(zx, xy = 0.15, zy = 0.5, n = 1000, seed = 123){ set.seed(seed) z <- rnorm(n) x <- zx * z + rnorm(n) y <- xy * x + zy * z + rnorm(n) return(tibble::tibble(x, y, z))}#' Simulate a 3 variable situation where a nuisance variable is a collider#' #' @param xz coefficient for impact of x on z#' @param xy coefficient for impact of x on y#' @param yz coefficient for impact of y on z#' @param n sample size#' @param seed random seed set for reproducibility#' @returns a tibble of three variables x, y and z. x causes y #' and z is a collider ie it is impacted on by both x and ysim_collider <- function(xz, xy = 0.15, yz = 0.5, n = 1000, seed = 123){ set.seed(seed) x <- rnorm(n) y <- xy * x + rnorm(n) z <- xz * x + yz * y + rnorm(n) return(tibble::tibble(x, y, z))}#' Simulate a 3 variable situation where a nuisance variable is a mediator#' #' @param xz coefficient for impact of x on z#' @param xy coefficient for impact of x on y#' @param zy coefficient for impact of z on y#' @param n sample size#' @param seed random seed set for reproducibility#' @returns a tibble of three variables x, y and z. x causes y #' and z is a mediator ie x impacts on z and z impacts on y, so some of the #' impact of x on y comes via zsim_mediator <- function(xz, xy = 0.15, zy = 0.5, n = 1000, seed = 123){ set.seed(seed) x <- rnorm(n) z <- xz * x + rnorm(n) y = xy * x + zy * z + rnorm(n) return(tibble::tibble(x, y, z))}

``` The idea with these functions is that they work with multiple values of “a” (as per the original terminology of the diagrams). But for illustrative purposes we can use these functions to generate data where a=0.3 (a reasonably material relationship) and see how the correlations come out. This helps make it obvious that you can’t tell from the data alone (at least with this sort of snapshot, observational data) which way the causality is going:

```

Correlations of example different datasets:> round(cor(sim_confounder(0.3, n = 10000)), 2) x y zx 1.00 0.26 0.29y 0.26 1.00 0.49z 0.29 0.49 1.00> round(cor(sim_collider(0.3, n = 10000)), 2) x y zx 1.00 0.15 0.34y 0.15 1.00 0.47z 0.34 0.47 1.00> round(cor(sim_mediator(0.3, n = 10000)), 2) x y z x 1.00 0.28 0.29y 0.28 1.00 0.49z 0.29 0.49 1.00

``` Basically the correlations tell us nothing about the direction of causality.

So my next step is to generate the data for many different values of a, for each of the three relationship types. There are plenty of ways to do this efficiently, here is mine. For those curious, res standards for “results”, in my object names. Naming things is hard!

At the end of the code below I have an object res which has values of a and estimates of the simple coefficient and partial coefficient in front of X (in the regression of Y on X) for that relationship type, as well as the true causal impact of X on Y.

Post continues after R code

```

-----------generate data and fit regressions, for various values of athe_a <- seq(from= -0.7, to = 0.7, length.out = 10)the_n <- 10000res_conf <- lapply(the_a, sim_confounder, n = the_n) |> bind_rows() |> mutate(a = rep(the_a, each = the_n), var = "Confounder") |> group_by(a, var) |> summarise(`Simple coefficient` = coef(lm(y ~ x))[['x']], `Partial coefficient` = coef(lm(y ~ x + z))[['x']], `Causal effect` = 0.15)res_coll <- lapply(the_a, sim_collider, n = the_n) |> bind_rows() |> mutate(a = rep(the_a, each = the_n), var = "Collider") |> group_by(a, var) |> summarise(`Simple coefficient` = coef(lm(y ~ x))[['x']], `Partial coefficient` = coef(lm(y ~ x + z))[['x']], `Causal effect` = 0.15)res_medi <- lapply(the_a, sim_mediator, n = the_n) |> bind_rows() |> mutate(a = rep(the_a, each = the_n), var = "Mediator") |> group_by(a, var) |> summarise(`Simple coefficient` = coef(lm(y ~ x))[['x']], `Partial coefficient` = coef(lm(y ~ x + z))[['x']], `Causal effect` = `Simple coefficient`)res <- bind_rows(res_conf, res_coll, res_medi) |> ungroup()

``` Drawing a plot of regression estimatesIt only remains to draw and polish the chart, which is basic ggplot2 stuff. Probably the only noteworthy trick in the below is a slight improvement on the original diagram where I move the Y axis and its labels to the right of the chart, to avoid clutter near where I am going to place the DAGs in the end figure.

Post continues after R code

```

------------------draw plot-------------------the_font <- "Calibri"tg <- guide_legend(direction = "vertical", keywidth = unit(3, "cm"))p <- res |>x gather(method, coefficient, -a, -var) |> mutate(method = fct_relevel(method, "Causal effect", after = Inf), var = fct_relevel(var, "Confounder")) |> mutate(method = fct_recode(method, "Causal effect (X causing Y)" = "Causal effect")) |> ggplot(aes(x = a, y = coefficient, colour = method, linetype = method)) + facet_wrap(~var, ncol = 1) + geom_line(linewidth = 1.5) + scale_x_continuous(breaks = seq(from = -0.7, to = 0.7, by = 0.2)) + scale_y_continuous(position = "right") + scale_linetype_manual(values = c(1, 1,2), guide = tg) + scale_colour_manual(values = c("yellow", "lightgreen", "black"), guide = tg) + labs(colour = "", linetype = "", y = expression("Coefficient "~beta), caption = "Reproducing (inexactly) a figure by Wysocki, Lawson and Rhemtulla in \n'Statistical Control Requires Causal Justification'. http://freerangestats.info") + theme_light(base_family = main_font) + theme(plot.caption = element_text(colour = "grey50"), strip.text = element_text(size = rel(1), face = "bold"), legend.position = "top")

``` Drawing Directed Acyclic Graphs (DAGs)Next comes drawing the DAGs themselves. I’m not happy with how I’ve done this, but it works… The tricky part was getting labels in the edges connecting each node. This is possible (as indeed I do in the code below) but it involved using trial and error to work out which labels went with each connecting edge. I don’t like this at all! There must be a way to do this “properly” with mapping an aesthetic to some variable but I couldn’t work it out.

DAGs as drawn with ggdag seem to have a random orientation, so setting the seed below each time just before drawing a DAG is critical for having them all in the same orientation, with X and Y both at the bottom of the image.

Post continues after R code

```

------------------draw DAGs---------------arrow_col <- "grey70"set.seed(123)d1 <- dagify( Y ~ X + C, X ~ C) %>% ggplot(aes(x = x, y = y, xend = xend, yend = yend)) + geom_dag_edges_fan(aes(label = c("a", 0.5, 0.15, NA)), edge_colour = arrow_col, label_colour = "black") + geom_dag_text(colour = "black") + theme_dag() set.seed(123)d2 <- dagify( C ~ X + Y, Y ~ X) %>% ggplot(aes(x = x, y = y, xend = xend, yend = yend)) + geom_dag_edges_fan(aes(label = c(NA, "a", 0.15, 0.5)), edge_colour = arrow_col, label_colour = "black") + geom_dag_text(colour = "black") + theme_dag()set.seed(123)d3 <- dagify( C ~ X, Y ~ X + C) %>% ggplot(aes(x = x, y = y, xend = xend, yend = yend)) + #geom_dag_point() + geom_dag_edges_fan(aes(label = c(0.5, "a", 0.15, NA)), edge_colour = arrow_col, label_colour = "black") + geom_dag_text(colour = "black") + theme_dag()

`` So that code defines the three DAGs in the objectsd1,d2,d3` but doesn’t draw them.

Pulling the combined figure togetherThe final step is to use the magic of patchwork package by Thomas Pedersen to lay out the diagrams on the actual page. The design argument to plot_layout() lets you take total control of how multiple ggplot2 objects (in our case, p, d1, d2 and d3) are layed out in a figure, so I make use of that here.

Post continues after R code

design <- c( area(1,1,2,2), area(3,1,4,2), area(5,1,6,2), area(1,3,6,5))print(d1 + d2 + d3 + p + plot\_layout(design = design) + plot\_annotation(title = "Statistical control requires causal justification", subtitle = "Only when the variable is a confounder is it correct to 'control' for it in a regression.", theme = theme(text = element\_text(family = the\_font))) ) ComparisonSo as a reminder, here’s a repeat of the final image:

… and here’s the original version from Wysocki et al:

There’s an obvious difference! My “simple coefficient” for X in the presence of a confounder has a gentle S curve whereas their’s is a straight line; and their “partial coefficient” for X in the presence of a collider has a gentle S curve whereas mine is in a straight line. Why? I don’t know! Obviously it is to do with our different simulated data. Mine is generated as some simple linear relationships; their’s is described as:

“We used path tracing (i.e., Wright’s rules; Alwin & Hauser, 1975; see Appendix A for an example) to obtain a population correlation matrix for each causal structure and calculated regression coefficients from each population correlation matrix using the formula β=Σ−1xxΣxy, in which Σxx is the p×p correlation matrix of predictors and Σxy is a p×1 vector that contains correlations between each predictor and the outcome”

That’s clearly a bit different from what I did. I’m not going to delve into exactly why that makes the results a bit different. The difference doesn’t change the substantive point about how you can go wrong with the wrong choice of partial or simple coefficient (corresponding to whether you “control” for the nuisance variable or not).

That’s all folks. Remember not to include collider and mediator variables in your controls in a regression, at least when you are interested in causal explanations (predictions being something else again)!

To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simulating confounders, colliders and mediators by @ellis2013nz

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to use mixed-effects models in R! Join our workshop on Introduction to Mixed-effects Models in R which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: Introduction to Mixed-effects Models in R

Date: Thursday, June 22nd, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Philip Leftwich is an Associate Professor of Genetics and Data Science at the University of East Anglia, Norwich, UK. He teaches R programming and statistics on various modules and workshops at undergraduate and postgraduate levels. His research interests include genetics, genomics and synthetic biology as tools to help combat agricultural and disease-carrying insect pests.

Description: Mixed-effects models are indispensable in analyzing data with hierarchical or nested structures. Unlike traditional linear regression models, mixed-effects models account for both fixed effects (applying to the entire population) and random effects (varying across groups). This unique capability allows researchers to examine how individual and group-level factors work together simultaneously, providing a comprehensive understanding of the data. In fields like social sciences, education, biology, and economics, where hierarchical data is prevalent, mixed-effects models significantly enhance the precision and reliability of statistical analyses. Mastering these models empowers researchers to extract valuable insights from complex datasets effectively.

In this introductory workshop, we will cover the basics of analyzing hierarchical data. Participants will learn about the difference between fixed and random effects, model formulation, estimation, and interpretation. We will discuss assumptions, model comparison and selection, practical implementation with R, and model validation. We will work through real-world examples to showcase the applications and benefits of mixed-effects models in various fields.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Introduction to Mixed-effects Models in R workshop was first posted on June 2, 2023 at 2:54 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introduction to Mixed-effects Models in R workshop

View Details

[This article was first published on Mirai Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Find out how to customize a shiny app and make it more accessible to users with our workshop on 14/06.

With shiny you can add an interactive web interface to your R work, also enabling users without a technical background to profit from your analysis and make data-driven decisions. Despite being a very powerful tool, a basic usage of shiny may not deliver its full potential. Thanks to many new packages ever expanding the shiny universe, and to a clever organization of the UI, however, it is possible to make the content of your app more user friendly and therefore more effective.

In our 3.5-hours hands-on workshop on Wednesday June 14th at 2:30 pm CEST we will explore how to ‘Make an outstanding Shiny App’ by playing with aesthetics, HTML and CSS customization, experimenting with other shiny-related packages, and having a look at different UI structures, layouts and ways to organize the content.

An online hands-on workshop to learn R Shiny by programming R Shiny.

This workshop is a natural continuation of the ‘Build your first Shiny App’ workshop, and it is part of the “Shiny” learning path of 4 workshops. If you register for multiple workshops of the series you can benefit from a discount, however, it can also be attended as a stand-alone workshop. Only minimal knowledge of shiny is expected, whereas no HTML or JavaScript skills are required.

Recording of the workshop will be shared with the attendees.

NOTE: the option to register for the workshop ‘Build your first Shiny App’ workshop will terminate on 05/06 e.o.d..

To leave a comment for the author, please follow the link and comment on their blog: Mirai Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Learn to ‘Make an outstanding Shiny App’ with us

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn the realm of data analysis and programming, organizing and sorting data efficiently is crucial. In R, a programming language renowned for its data manipulation capabilities, we have three powerful functions at our disposal: order(), sort(), and rank(). In this blog post, we will delve into the intricacies of these functions, explore their applications, and understand their parameters. These R functions are all used to sort data, however, they each have different purposes and use different methods to sort the data.

The order() FunctionThe order() function in R returns a permutation that would sort a vector or multiple vectors. It provides the indices that arrange the vector in ascending order. Let’s dive into an example to grasp its functionality:

x <- c(10, 4, 8, 2, 6)ordered\_indices <- order(x)sorted\_vector <- x[ordered\_indices]cat("The ordered incdices are: ", ordered\_indices)

The ordered incdices are: 4 2 5 3 1

cat("The sorted vector is: ", sorted\_vector)

The sorted vector is: 2 4 6 8 10 Here, order(x) provides the indices that would sort the vector x in ascending order. Subsequently, x[ordered_indices] rearranges the vector based on those indices, resulting in a sorted vector.

Parameters of order():* ... - Specify the vectors to be sorted.

The sort() Function:The sort() function in R directly sorts the given vector or matrices. Unlike order(), it returns the sorted vector itself. Let’s illustrate this through an example:

fruits <- c("apple", "banana", "orange", "grape")sorted\_fruits <- sort(fruits)sorted\_fruits

[1] "apple" "banana" "grape" "orange" In this example, sort(fruits) sorts the character vector fruits alphabetically, returning a new vector sorted_fruits.

Parameters of sort():* x - The vector or matrix to be sorted. * decreasing - A logical value indicating whether the sorting should be in descending order. (Default is FALSE)

The rank() Function:The rank() function assigns ranks to the elements in a vector. It returns a vector of the same length as the input vector, indicating the rank of each element. Consider the following example:

scores <- c(80, 60, 90, 75)ranking <- rank(scores)ranking

[1] 3 1 4 2 In this example, rank(scores) assigns ranks to each element in the vector scores, resulting in a new vector ranking.

Parameters of rank():* x - The vector to be ranked. * ties.method - A string specifying the method to handle ties in ranking. (Options: “average”, “first”, “last”, “random”, “max”, “min”) (Default is “average”)

ConclusionSorting, ordering, and ranking data are essential operations in data analysis. R’s functions, namely order(), sort(), and rank(), equip us with the necessary tools to accomplish these tasks seamlessly. By understanding their applications and parameters, we can efficiently manipulate data and derive meaningful insights. So, go ahead, explore their versatility, and unlock new possibilities in your data analysis endeavors!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Sorting, Ordering, and Ranking: Unraveling R’s Powerful Functions

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A riddle on a form of a Galton-Watson process, starting from a single unit, where no one dies but rather, at each of 100 generations, Dog either opts for a Uniform number υ of additional units or increments a counter γ by this number υ, its goal being to optimise γ. The solution proposed by the Riddler does not establish his solution’s is the optimal strategy and considers anyway average gains. Solution that consists in always producing more units until the antepenultimate hour (ie incrementing only at the 99th and 100th generations), I tried instead various logical (?) rules and compared outputs by bRute foRce, resulting in higher maxima (over numerous repeated calls) for the alternative principle

``` s<-function(p=.66){ G=0;K=1 for(t in 1:9){ i=sample(1:K,1) K=K+i(i>=Kp) G=G+i*(i

View Details

[This article was first published on Rtask, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You can read the original post in its original format on Rtask website by ThinkR here: {attachment} v0.4.0: Breaking changes and configuration file for a better experience

{attachment} v0.4.0 stores your parameters in a configuration file for an easier use during your package development: run att_amend_desc() from anywhere, it will re-use your last set of parameters. This comes with some breaking changes for a better experience.

The goal of attachment is to help to deal with package dependencies during package development. It also gives useful tools to install or list missing packages used inside Rscripts, Rmds or qmd. This reads your NAMESPACE, your functions in R directory and your vignettes, then update the DESCRIPTION file accordingly. Are you ready to be lazy ? Let {attachment} help you !

Installation{attachment} is available on CRAN and developed on GitHub. You can access the full documentation website on this page: https://thinkr-open.github.io/attachment. It also appears on the ThinkR r-universe page if you want to install the development version without GitHub API constraints.

Install CRAN version

install.packages("attachment") Install development version

install.packages('attachment', repos = c('https://thinkr-open.r-universe.dev', 'https://cloud.r-project.org')) Declare all dependencies in DESCRIPTION during package developmentWhat you really want is to fill and update your description file along with the modifications of your documentation. Indeed, only the following function will really be called. Use and abuse during the development of your package !

attachment::att\_amend\_desc() {attachment} detects all calls to library(pkg), @importFrom pkg fun, pkg::fun() in the different classical directories of your R package, then list them in the correct “Imports” or “Suggests” category in the DESCRIPTION file, according to their position in the package.

Declare extra dependencies for extra usesIf you want to add extra packages like {pkgdown} or {covr} that are not listed in any script in your package, a call for your development packages would be:

attachment::att\_amend\_desc(extra.suggests = c("pkgdown", "covr"), update.config = TRUE) Note the update.config = TRUE parameter that will save the parameters used in the call of att_amend_desc() to the package configuration file: “dev/config_attachment.yaml”.

If you run att_amend_desc() a second time afterwards, directly from the console, it will use the last set of parameters extracted from the configuration file.

Indeed, we recommend to store the complete command line in a “dev/dev_history.R” file to update and run it when needed. If the parameters do not change, you can run attachment::att_amend_desc() directly in the console, wherever you are, it will use the configuration file.

Automatically fill the “Remotes” fieldIf you would like to detect the sources of your installations so that you can add dependencies in the “Remotes” field of your DESCRIPTION file, to mimic your local installation, you will use:

attachment::set\_remotes\_to\_desc() Combination of {attachment} with {fusen}The configuration file of {attachment} was originally developed for a better integration with {fusen}. Indeed, when you inflate() your notebook file into a fully documented R package, {fusen} automatically runs attachment::att_amend_desc() to update your documentation and the list of dependencies of your package. However, until now, it was not possible to use extra parameters in the inflate() function to be used by {attachment}, and you had to stop the automatic detection with document = FALSE, then run your specific {attachment} code from somewhere else.

As we are all lazy, we do not want to change file each time we have to introduce a new changes in our package. With the configuration file of {attachment}, you can let your fusen::inflate() as is, and it will use the last version of attachment::att_amend_desc() according to what’s in the configuration file. If you want to change the default parameters, by adding extra dependencies for instance, you will have to change the configuration file only once. To do so, you’ll surely want to update your “dev/0-dev_history.Rmd” file with this code to run once:

attachment::att\_amend\_desc(extra.suggests = c("pkgdown", "covr"), update.config = TRUE) NEWS contentBreaking changes* When using att_amend_desc() without the default parameters, like pkg_ignore = "x" will now require att_amend_desc(pkg_ignore = "x", update.config = TRUE), otherwise, it will fail. This allows for the use of parameters stored in the config file when running att_amend_desc() directly in the console.
Recommendation: Run att_amend_desc(pkg_ignore = "x", update.config = TRUE) if you have to update your config, run att_amend_desc() daily as you’ll want to use what is stored in the config file. * create_dependencies_file() gets parameter install_only_if_missing = FALSE by default to complete the installation instructions packages only if missing. (@MurielleDelmotte)

New features* att_amend_desc() can run with the last set of parameters stored in a configuration file, without having to call them all each time. See vignettes and documentation of parameters update.config = FALSE, use.config = FALSE and path.c = "dev/config_attachment.yaml". (@dagousket) * create_dependencies_file() now takes other sources into account (git, gitlab, github, bioc, local). (@MurielleDelmotte) * Use create_dependencies_file(to = NULL) to only get the output as character and do not create a file

Bug fixes* att_amend_desc() does not modify user options("warn") level anymore (#94) * att_amend_desc() allows “Remotes” field to contain @ref (#67)

This post is better presented on its original ThinkR website here: {attachment} v0.4.0: Breaking changes and configuration file for a better experience

To leave a comment for the author, please follow the link and comment on their blog: Rtask.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {attachment} v0.4.0: Breaking changes and configuration file for a better experience

View Details

[This article was first published on R programming – Oscar Baruffa, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.01 June 2023

Earlier this year I applied to the R Consortium for a grant to upgrade the Big Book of R. Unfortunately my proposal wasn’t accepted, but I am really proud of how the proposal turned out. I’ve never written a proposal before and wasn’t really sure of how to do it, but with a little bit of guidance from Andrew Collier (Data Wookie) I think it turned out pretty well.

I want to give a special thanks to everyone who submitted a statement of support – it was really inspiring to get a glimpse of how much you value the Big Book of R!

I’m sharing the proposal here so it can serve as inspiration for your next grant proposal.

I’ve heard that grant-writing can take a long time and it did – this one took me maybe 5 or 6 solid hours to put together. So, not for the time-scarce among us but I suggest you email the committee members early and get some feedback if they think your idea is a viable candidate before sinking all your hours into it.

Thanks and enjoy the read!

Big Book of R Upgrade proposalIn response to the 2023 ISC Grant Programme Call for Proposals

Submitted by: Oscar Baruffa

Date: 29 March 2023

https://www.bigbookofr.com/

SummarySeeking ISG Grant Funding of [redacted] spread over 3 years for a website upgrade of the Big Book of R. This enhances the discoverability of R programming books which will improve the Social Infrastructure of the R ecosystem.

The upgrade will be completed in 4 months. Development costs primarily cover labour and include 3 years of maintenance support and storage/compute fees.

“This will be great for knowledge sharing and improving R accessibility and usability. Please fund generously.” – Brendan Ansell

SignatoriesThe proposal has broad support in the wider R community. A call for statements of support was issued to the wider R community via Twitter, Reddit, Mastodon, LinkedIn and Oscar Baruffa’s newsletter subscribers.

At the time of writing, over 100 statements have been submitted and are overwhelmingly supportive of the proposed upgrades to the Big Book of R.

Project teamOscar Baruffa – Project Lead

Oscar is the creator and maintainer of Big Book of R. He works as a Senior Analytics Manager, overseeing the development of a data and analytics pipeline including web portals. He has extensive project management experience for large and small projects.

Andrew Collier – Tech Lead

Andrew is the founder of and Lead Data Scientist at Fathom Data. Andrew has a wide range of experience in bringing data solutions to life directly and together with his team. He is the creator of a number of R packages and has hosted multiple SatRdays in South Africa.

Bianca Peterson – Data Scientist

Bianca is a data scientist with a strong background in the higher education industry, is trainer of The Carpentries and co-chair of the CODATA-RDA Schools of Research Data Science.

In addition to this team, Andrew is able to assign team members from his consultancy Fathom Data to support if required.

All project members have confirmed their availability to work on this project.

ConsultedHadley Whickham (ISC Member) was contacted on 18th March 2023 via email for informal feedback on the proposal. Informal feedback was that the proposal was a likely candidate for consideration along with some notes to clarify the need and cost breakdown. These have been addressed in this proposal.

Oscar Baruffa consulted with Andrew Collier of Fathom Data on the objectives, technical approach and project team.

The wider R community has been asked for their support and feedback with over 100 responses received, which can be viewed here.

“The Big Book of R is my go-to resource when people ask, “What are some resources for learning about R in [industry]?”. I would love to see it further develop and grow.” – Isabella Velásquez

The ProblemBackgroundThe Big Book of R (https://www.bigbookofr.com/) aka “the site” or “BBoR” is a curated collection of over 350 R programming books. Almost all of these books are open-source and free to read.

The site is essentially a library that gathers these amazing resources and puts them in one place. The problem the site is trying to solve is that of discoverability of R programming books. Prior to Big Book of R, the only way to find these books were by googling them or if one happened to come across a limited list of books contained in a git repo.

No other online resource contains the depth and breadth of the books available at BBoR and at a high quality, for free. Oscar launched the site in August 2020. The site stats are publicly available. Since launch, it has had some 240k visitors. The latest 12-month figures are that it receives about 95k visitors per year. Average time on the site is 2 minutes, with a click through rate of 31% i.e. about a 1/3rd of visitors click through to a book’s website.

Feedback from the R community has been positive since its launch, Survey respondents have stated they refer others to it as a go-to resource for new and experienced R programmers.

New books are added at the rate of 5-10 books every 6-8 weeks. Blog posts that highlight new book additions get 1-2k views each.

“As a 26 year user of R I think the upgrade will make The Big Book of R an even more indispensable resource for everyone from beginners through to experts. I regularly recommend it to those I mentor.” – Dr Lyndon Walker

The issuesThe size of the already-large and continually-growing collection presents new problems:

Curating books is becoming difficult, particularly removing books that are old. Outdated books clutter the user experience and reduce the relevance of the site.
Some subjects have many books, making it difficult for visitors to navigate which has the content they need.
The backend of the site is a google sheet. This is becoming increasingly difficult to use as a data store and is hard to pick up duplicate books and creates variations and multiple versions of author bio entries.

In summary, the general discoverability of relevant books is decreasing as the collection grows.

“The proposed upgrades would be well worth it. … This is ever more important in the face of increasingly ineffective search results from eg google.”- Jim Gardner

The ProposalOverviewThis proposal is for an upgrade of the Big Book of R which will:

Improve the quality of the collection
The durability of the system supporting it
Greatly enhance the discovery of books in the collection

The proposal aims to balance improvements across a number of facets to get the most of the time and funds available. These are:

Improve the back-end by migrating from google sheet to a database.
Improve the data quality by cleaning up author bio links.
Lower the time-cost of curation by using an app interface for data entry.
Enrich the database with additional data for each entry i.e. date of last update and a table of contents.
Improve discoverability by adding:
1. A “trending books” chapter showing the most popular books of the last week and month.
2. A “new additions” chapter highlighting the books most recently added to the collection.
Positioning the book for future improvements by:
1. Modernising the back end (see above)
2. Improving data quality of existing and new books (see above)
3. Porting the site to quarto.
Annual maintenance support for 3 years

The R community will benefit from this upgrade. The annual 95k (and increasing) visitors will get a better user experience to find the information they are looking for. Authors of R books get wider distribution of their material and better return on investment for themselves and the R ecosystem.

“There’s an abundance of R literature and this book has provided a port for it. It helps new starters and seasoned users equally and it’s literally the only bookmark on R that anyone would need. Any upgrades on the delivery method of such an important source will go a long way and increase adoption of the R language!”- Vasileios Plessas

Project planStart-up phaseThe startup phase will consist of creating the feature roadmap, familiarising the delivery team with the codebase, making technical decisions on architecture and setting up a review mechanism.

The team have already worked together professionally for ~3 years, and will mirror our already successful collaboration model.

The project already exists on github and collaborators can be easily added.

The team already uses Asana to work in an agile manner, and we’ll create a dedicated project board for this work.

Inter-team communication already works well via slack, and we’ll create a slack group for this work also.

As this is an ongoing project, most of the “start up” work is only related to this upgrade specifically which means we can start almost immediately.

Technical deliveryMigrating to a databaseRationale: This is a key step to improve the data quality and resilience vs the current google sheet solution. It also opens up the ability to store the Table of Contents for each entry.

Activities:

Decide on architecture/tech provider.
Design and develop the data model
Clean the existing dataset
Port data from google sheets to the model
Document the model and tables
Reconfigure the site to pull data from the database
Refactor code that builds book entries.

Develop backend app interfaceRationale: Data capture is a time consuming exercise. A lightweight app interface to be able to do basic maintenance activities e.g. add and update data, check for duplicates prior to entry, select from existing categories, existing authors etc. .

Activities:

Develop minimal functionality specification
Select tech solution
Design and develop the interface.

Port to QuartoRationale: Improves the site’s styling and opens up future User Interface improvements as Quarto develops.

Activities:

Port book to quarto

Develop “trending books” sectionRationale: Opportunities to highlight books are currently limited to new additions, a random selection or a curators selection. Creating a “trending books” section will highlight books that have been accessed the most over the past 7/30 days.

Activities:

Build an interface to the site analytics API (Plausible Analytics).
Develop a chapter that lists the most popular books of the past period.

Develop “new books” sectionRationale: New book editions are very popular blog posts on Oscar Barufa’s website, and it would be useful for visitors to reference the site itself for new book additions.

Activities:

Develop a chapter that displays the most recently added books.

Web scraping Rationale: In order to retrieve the date books were last updated, and the table of contents of each book, web scraping of the books will need to be deployed. Web scraping will be limited to the two most common book formats to reduce development time and maintenance requirements. These will run infrequently to minimise compute costs.

Activities:

Develop web scrapers
Set up scheduling

Table of Contents dropdownsRationale: Often the title and book description are not enough on their own for a visitor to know what the book contains. A quick scan of the Table of Contents is a much better method and being able to do so within BBoR will make searching through volumes for the correct one much easier.

Activities:

Incorporate table of contents for books that have it

Annual maintenance supportRationale: The upgrade will result in the site having more moving parts and monthly service costs which will require occasional maintenance. 3 years worth of support is sought as this gives enough runway to assess how much annual support costs, and for Oscar Baruffa to plan for how to fund additional years of support (if support is required at all).

Activities:

Monthly database and cloud computing costs.
Occasional debug of web scrapers
Minimum maintenance for bugs and security fixes

Note that Fathom data is offering to provide this labour support at cost.

Timeline and milestonesMilestones represented with an “X”below assuming a 1st June grant acceptance.

After the upgrade is complete, 6 monthly milestones for compute and maintenance costs.

Other aspectsAdditional non-technical activities are not required to complete this work, however as new milestones are reached, they will be communicated via:

Oscar Baruffa’s Twitter account (4k followers), email newsletter (1000 subscribers and syndication via R-bloggers and R-weekly newsletters), Linkedin and Mastodon.
Big Book of R’s twitter account (5k followers)
Fathom Data and Andrew Collier’s channels.

RequirementsPeopleThe project requires people with skills in data modelling, web scraping and reporting.

The data-science skills will provided by two data science professionals on the project i.e. Andrew Collier and Bianca Peterson.

Project ownership, vision and direction will come from Oscar Baruffa as the creator and maintainer of Big Book of R.

“Oscar Baruffa has been consistent with providing and supporting learners across the world within the R learning ecosystem. I strongly support his passion and contribution to further improve the R ecosystem, and creating a better experience for the community. Yes, this upgrade is a positive contribution. – Timipa Ikidi

ProcessesThis is a relatively small project for this team, and bi-weekly progress check-ins may be needed where asynchronous communication via slack and Asana is not sufficient.

Handover to the community happens continuously as new features are developed, as this is a live website.

Tools & TechDatabase hosting will be needed to store the data.

Cloud computing will be needed to host a small api interface for data capture and to periodically run web scrapers on books. The development of the backend app interface and web scrapers are part of this proposal.

The main deciding factors on which provider/s to use will be based on cost and familiarity with the provider’s solutions.

Funding [redacted]

SummaryFunding is required to cover development and maintenance costs.

Development is provided by Fathom Data at their competitive-prices day rates.
Maintenance support by Fathom Data will be provided at cost.
Oscar Baruffa’s time is provided at zero cost.

The cost of the project is comprised of 94% labour costs and 6% storage/compute fees.

“The Big Book of R is my go-to resource whenever I need to learn something new or find specific techniques, and I love browsing its pages to stay up-to-date on the latest trends and innovations in the world of R.

What I appreciate most about it is how comprehensive it is. No matter what level you’re at, from beginner to advanced, there’s something in here for you.

Regular updates keep me excited about the new and exciting developments in the R community and help me stay on top of my game. I would love to see the Big Book of R upgraded to improve the functionality and accessibility of this essential R Community resource.”- Michael Underwood

SuccessDefinition of doneThe New Books and Trending books sections are live on the website.

When these two components are in place, it will mean that all the underlying infrastructure work has been completed.

Book has been ported from Rmarkdown to Quarto with the ability to see the Table of Contents of books. When this is done the last component of front-end work will be completed.

Measuring successDeliverables are tangible and objective, so we can use them to measure success.

Dataset ported to database.
App interface to manage data.
“New Books” section available on the website.
“Trending Books” section available on the website.
Table of Contents viewable

“The Big Book of R is a fantastic resource for me as someone [who] teaches R and uses it for own research. An upgrade with added functionality would improve its value for me and my students. – Hendrik Jürges

Future workNote that the Big Book of R will continue to be open sourced, and the webscrapers built for the upgrade will also be open sourced.

Confirmed future development:

Big Book of R will continue to be updated and curated.

Possible future development:

Development of an R package/API to query collection content e..g Jon Harmon (Chief Community Manager of R4DS Online Learning) cited this is very useful for his plans to create courses based on R programming books)
Integration of an LLM chatbot (e.g. ChatGPT) to create a “librarian” that can help visitors find the right book based on their needs.

Key risksData Scientists are unavailable

If one or both of the technical delivery team are unable to do the work, this would delay the progress. A mitigating factor is that Andrew Collier is able to draw from other members of his staff at Fathom Data.

Work is more labour intensive than estimated

It can happen that work is more labour intensive than initial estimates suggest. We have tried to balance overoptimism and conservatism.

To mitigate an underestimate (which could hamper completion), a 20% contingency has been included in development costs.

Grant funding does not extend to maintenance costs

If the project is not able to receive the grant funding for the maintenance support, this could stall or break the project. Some possible mitigation steps are:

Oscar Baruffa could cover the storage/compute costs personally.
Attempt to crowdfund the developer support costs annually

Grant funding payments are late or do not materialise

This will stop future development until each tranche of funding is received, as neither Oscar nor Fathom Data can carry the costs of this development work.

Mitigating factors are that the milestones are self-contained, in short iterations and version controlled, so there is no contagion of unfinished work threatening the site’s functioning..

Other risks are very low as we are extending and improving an already functioning website.

“.. the big book of R is one of the greatest resources in the entire R ecosystem and it could use some improvements that reflect that.” – Gordon Blasco

“…this is a very positive contribution to the R ecosystem and the environment itself for this tool is something that adds to the scientific knowledge of R. Having the Big Book of R as an available tools for both beginners and seasoned coders of R makes it easier to learn new things, as well as to find new areas of interest or existing literature on a very specific topic. No other resource like this one exists.” – Daniel Sanchez

/end

Keep up to date with new data posts and Big Book of R updates by signing up to my newsletter. Subscribers get a free copy of Project Management Fundamentals for Data Analysts worth $12.

Once you’ve subscribed, you’ll get a follow up email with a link to your free copy.

The post Sharing the Big Book of R upgrade proposal appeared first on Oscar Baruffa.

To leave a comment for the author, please follow the link and comment on their blog: R programming – Oscar Baruffa.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Sharing the Big Book of R upgrade proposal

View Details

We’ve released the newest version of NIMBLE on CRAN and on our website. NIMBLE is a system for building and sharing analysis methods for statistical models, especially for hierarchical models and computationally-intensive methods (such as MCMC and SMC). Version 1.0.0 provides substantial new functionality. This includes: A Laplace approximation algorithm ...

Continue reading: Version 1.0.0 of NIMBLE released, providing automatic differentiation, Laplace approximation, and HMC sampling

View Details

CAMIS is a PHUSE working group in collaboration with PSI and the R consortium. Initially, the repository contains R and SAS analysis result comparisons, however, the team hopes to extend to...

The post Comparing Analysis Method Implementations in Software (CAMIS) appeared first on R Consortium.

Continue reading: Comparing Analysis Method Implementations in Software (CAMIS)

View Details

Introduction

Regular expressions, often abbreviated as regex, are powerful tools used in programming to match and manipulate text patterns. While they might seem intimidating at first, regular expressions are incredibly useful for tasks like dat...

Continue reading: Demystifying Regular Expressions: A Programmer’s Guide for Beginners

View Details

Intro

Basic Web Scraping

Advanced Web Scraping

Wrap-up

Intro

Welcome to this blog post where we’re going to explore web scraping in R. So far, I’ve used R for some basic web scraping jobs, like pulling the list of all available R packages...

Continue reading: Static and Dynamic Web Scraping with R

View Details

Downloading repository snapshots

The new snapshot API lets you download a full copy of any CRAN-like repository on r-universe. You can use such a snapshot to mirror the entire CRAN-like repository on your own servers, or for example to build a stable...

Continue reading: Downloading snapshots and creating stable R packages repositories using r-universe

View Details

[This article was first published on Achim Zeileis, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. New arXiv working paper on the relaunch of the CRAN Task View Initiative providing better infrastructure and workflows for proposing and maintaining CRAN Task Views and fostering interactions with the R community.

CitationAchim Zeileis, Roger Bivand, Dirk Eddelbuettel, Kurt Hornik, Nathalie Vialaneix (2023). “CRAN Task Views: The Next Generation.” arXiv.org E-Print Archive arXiv:2305.17573 [stat.CO]. doi:10.48550/arXiv.2305.17573

Abstract CRAN Task Views have been available on the Comprehensive R Archive Network since 2005. They provide guidance about which CRAN packages are relevant for tasks related to a certain topic, and can also facilitate automatic installation of all corresponding packages. Motivated by challenges from the growth of CRAN and the R community as a whole since 2005, all of the task views infrastructure and workflows were rethought and relaunched in 2021/22 in order to facilitate maintenance, and to foster deeper interactions with the R community. The redesign encompasses the establishment of a group of CRAN Task View Editors, moving all task view sources to dedicated GitHub repositories, adopting well-documented workflows with a code of conduct, and leveraging R/Markdown files (rather than XML) for the content of the task views. Read full paper ›

LinksCRAN Task Views: https://CRAN.R-project.org/web/views/

CRAN Task View Initiative: https://github.com/cran-task-views/ctv/

R package: https://CRAN.R-project.org/package=ctv

To leave a comment for the author, please follow the link and comment on their blog: Achim Zeileis.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: CRAN Task Views: The next generation

View Details

[This article was first published on R Archives - Dan Oehm | Gradient Descending, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.After the fantastic first season of Alone Australia, I’ve added all data to the {alone} R package. Now you can compare the Australian version with the 9 US season.

The data package can be installed from CRAN

install.packages("alone") Or Github

devtools::install.packages("doehm/alone") It was a rough start for the Aussies and it looked like it was going to be a shorter season than all US seasons. But Mike and Gina pulled through, surviving more than twice as long as 3rd place finisher Michael and making it to 64 and 67 days respectively.

Season 10 castI’ve also included the season 10 cast of the US version. At this stage, it only includes name and demographic information. I will be updating the Google Sheets as it airs and will release the data to Git and CRAN once the season concludes.

The post Alone Australia is now included in {alone} appeared first on Dan Oehm | Gradient Descending.

To leave a comment for the author, please follow the link and comment on their blog: R Archives - Dan Oehm | Gradient Descending.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Alone Australia is now included in {alone}

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionProgramming is often about making decisions based on certain conditions. In the world of R, there are numerous functions that can help us simplify our code and make it more efficient. One such function is any(). In this blog post, we’ll explore the any() function and learn how it can be used to streamline our logical operations. Whether you’re a beginner or an experienced programmer, this post aims to make the concept accessible to everyone. So, let’s dive in!

Understanding the BasicsThe any() function in R is a powerful tool that allows us to determine if any of the elements in a given vector or logical expression are TRUE. It returns a single logical value (TRUE or FALSE) depending on the presence or absence of TRUE values within the input.

Syntax:The basic syntax of the any() function is as follows:

any(x, ...) Here, x represents the input vector or logical expression, and ... represents additional arguments which can be used to control the behavior of the function (although they are optional).

ExamplesBasic ExamplesNow, let’s see some basic examples on how to use the any() function.

x <- c(1, 2, 3, 4, 5)any(x > 10)

[1] FALSE

x <- c(1, 2, NA, 4, 5)any(x > 10)

[1] NA

any(x == 5)

[1] TRUE Now, let’s explore some examples to see how any() can be utilized in various scenarios:

Checking for the Presence of a Specific Value:Suppose we have a vector of numbers, and we want to check if any of them are divisible by 5. We can use the any() function to accomplish this as follows:

numbers <- c(2, 7, 12, 15, 21)is\_divisible\_by\_5 <- any(numbers %% 5 == 0)if (is\_divisible\_by\_5) { print("At least one number is divisible by 5.")} else { print("None of the numbers are divisible by 5.")}

[1] "At least one number is divisible by 5." In this example, we use the modulus operator (%%) to check if each number in the vector has a remainder of 0 when divided by 5. The any() function then returns TRUE if any such element is found, indicating the presence of at least one number divisible by 5.

Validating User Input:Let’s say we are building a program that requires the user to input a positive number. We can use the any() function to validate the input as shown below:

user\_input <- as.numeric(readline(prompt = "Enter a positive number: "))

Enter a positive number:

```

Dummy inputuser_input <- 5is_positive <- any(user_input > 0)if (is_positive) { print("Input is a positive number.")} else { print("Input is not a positive number.")}

```

[1] "Input is a positive number." Here, we convert the user input to a numeric value using as.numeric() and then check if it is greater than zero. The any() function returns TRUE if any element satisfies this condition, confirming that the input is indeed a positive number.

ConclusionThe any() function in R simplifies the process of determining whether any elements in a vector or logical expression are TRUE. It’s a versatile tool that can be employed in a variety of scenarios, from validating user input to performing complex logical operations. By incorporating any() into your code, you can enhance readability, reduce redundancy, and make your programs more efficient.

Remember, mastering these small yet powerful functions is what sets great programmers apart. So go ahead, experiment with any() and unlock its full potential in your R programming journey!

Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simplifying Logical Operations with the R Function any()

View Details

[This article was first published on Rsquared Academy Blog - Explore Discover Learn, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We are excited to announce the standby package. It allows you to easily create alerts, notifications, tooltips and loading screens in Shiny. The package was developed as part of our internal tools for developing shiny apps and we are glad to share it with the rest of the community.

Installation ```

Install release version from CRANinstall.packages("standby")# Install development version from GitHub# install.packages("devtools")devtools::install_github("rsquaredacademy/standby")

``` Loading ScreensTo use spinners/loaders from standby in your Shiny application, include thefollowing in the UI part of the app:

Include the dependencies using the appropriate use* functions(useSpinkit() in the below example).
Wrap the target output using corresponding rendering function(spinkit() in the below example).

Example library(shiny)library(standby)ui <- fluidPage( standby::useSpinkit(), # include dependencies fluidRow( standby::spinkit(plotOutput("plot1")), # wrap output inside loader actionButton("render", "Render") ))server <- function(input, output, session) { output$plot1 <- renderPlot({ input$render Sys.sleep(3) hist(mtcars$mpg) })}shinyApp(ui, server) DetailsThe below table displays the dependency and rendering functions along with references:

| Index | Dependency | Render | Reference | | --- | --- | --- | --- | | 1 | useThreeDots() | threeDots() | Three Dots | | 2 | useSpinkit() | spinkit() | SpinKit | | 3 | useVizLoad() | vizLoad() | Loading Visualization | | 4 | useSpinners() | spinners() | Spinners | | 5 | useLoaders() | loaders() | Loaders |

Alerts and NotificationsTo use alerts or notifications from buzz in your Shiny application, follow the below steps:

Include the dependencies in the UI part of the app using the appropriateuse* functions (useToast() in the below example).
Include the corresponding rendering function in the Server part of theapp (toast() in the below example).

Example library(shiny)library(standby)ui <- fluidPage( useBootBox(), # include dependencies actionButton(inputId = "notify", label = "Show Notification"))server <- function(input, output, session) { observeEvent(input$pnotify, { bootBox(class = "rubberBand") # display the alert })}shinyApp(ui, server) DetailsThe below table displays the dependency and rendering functions along with references:

| Index | Dependency | Render | Reference | | --- | --- | --- | --- | | 1 | useAlertify() | alertify_alert() | Alertify | | 2 | useAlertify() | alertify_notify() | Alertify | | 3 | useBootBox() | bootBox() | BootBox | | 4 | useMicroTip() | microTip() | MicroTip | | 5 | useNS() | notice() | Notification Styles | | 6 | useNotify() | notify() | PNotify | | 7 | useTingle() | tingle() | Tingle | | 8 | useToast() | toast() | iziToast |

Learning More* Documentation * GitHub Inspiration & Acknowledgementstandby takes inspiration from the following wonderful packages:

shinycssloaders by Dean Attali
waiter by John Coene

We found the following books extremely helpful while working on the package and in general for developing Shiny apps. We would like to express out heartful gratitude to the respective authors for making them available for free to the community.

Outstanding User Interfaces with Shiny byDavid Granjon
JavaScript for R by John Coene Getting HelpIf you encounter a bug, please file a minimal reproducible example usingreprex on github. For questions and clarifications,use StackOverflow.

FeedbackAll feedback is welcome. Issues (bugs and feature requests) can be posted togithub tracker.For help with code or other related questions, feel free to reach out to usat pkgs@rsquaredacademy.com.

To leave a comment for the author, please follow the link and comment on their blog: Rsquared Academy Blog - Explore Discover Learn.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing standby

View Details

[This article was first published on Mirai Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to develop from scratch an interactive web interface in R shiny on June 07th.

Shiny is the ever more popular approach to share R data analysis and reports that are required to make data-driven decisions.

It is in fact natural to add an interactive user interface to enable data exploration, visualization, and reporting results, with the additional benefits that you can do it yourself without having to rely on external web services or BI tools.

Learn with us how you can easily developing such a tool with your R know-how, and how you can walk all the way from prototyping to productive use. While developing an understanding of the most important Shiny concepts you will build step by step a simple Shiny app during the workshop, the app will contain control widgets (sliders, check boxes, selectors, action buttons, etc.), a simple scatter plot and a table of the Iris dataset.

An online hands-on workshop to learn R Shiny by programming R Shiny.

When:Wednesday June 07th, at 2:30 p.m. CEST (Switzerland time-zone).
What:3.5-hours online hands-on workshop covering the basics of R shiny and walking you through the development of your first shiny app.
Who:This workshop is for everyone with an interest in building an interactive web interface in R. No previous experience with shiny is needed, however knowledge of R is beneficial.
How:Register at this link. Register more people to get an additional discount from the second attendee onwards.
And then?This is the first stop of your Journey to learn Shiny, this workshop is the first of the “Shiny” learning path. Register for multiple workshops of the series and benefit from a discount.

Recording of the workshop will be shared with the attendees.

We are looking forward to having you in our workshops!

To leave a comment for the author, please follow the link and comment on their blog: Mirai Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: ‘Build your first Shiny App’ with us

View Details

[This article was first published on R-Bloggers – Learning Machines, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
In this post, let us rise into the air to have a good view of the stock market. From this perspective, seemingly unrelated things all of a sudden become connected and patterns hidden by all the buzz and noise start to appear!

If you want to understand the big picture of technical and fundamental analysis, its relation to the payoffs of certain option strategies, and what this has to do with buy-and-hold, read on!

The financial markets are a playground for diverse strategies, each with its own unique perspective and methodology. Among these, technical and fundamental analysis stand out as two distinct yet profoundly interrelated approaches. By digging deeper, we find that these two techniques, though appearing different on the surface, are two sides of the same coin, creating a fascinating interplay that leads to a classic investment strategy: buy-and-hold.

Parts of this post were written with the help of ChatGPT-4, concept and ideas are my own. The payoff diagrams were created with R, you can find the code here: Financial X-Rays: Dissect any Price Series with a simple Payoff Diagram.

The Yin and Yang of Investing: Technical and Fundamental AnalysisFirst, let’s uncover the essence of these two approaches. On one side, we have technical analysis, which involves studying price movements and market trends. Technical traders rely on charts and various statistical measures to determine points of entry and exit in the market. They believe in the power of momentum, buying when prices rise and selling when they fall, under the premise that what goes up will continue to rise, and vice versa. This behaviour tends to make stock price movements in either direction more extreme.

On the other side is fundamental analysis, which seeks to evaluate securities by measuring the intrinsic value of a company. This is done by examining related economic, financial, and other qualitative and quantitative factors. When a stock’s price falls, fundamental traders see an opportunity: they believe the stock is undervalued, and that its price will revert to its fundamental value in the long run. This mean-reverting philosophy creates a stabilizing effect on stock prices.

The aforementioned approaches might seem diametrically opposed, but they are not. Instead, they offer two contrasting lenses through which to view the market. Let’s use an options trading analogy to illustrate this.

Technical traders can be likened to being long a call option. When you buy a call option, you have the right, but not the obligation, to buy an asset at a predetermined price within a specified period. It’s a bullish strategy, expecting the stock’s price to rise, similar to the momentum-driven philosophy of technical traders. When the stock falls below a certain point technical traders tend to sell illustrated by the flat line on the left, so that they won’t lose any more money, no matter how far the stock falls.

In contrast, fundamental traders can be thought of as being short a put option. When you look at the payoff diagram you see a cap on the right where you stop earning money from a certain point on, no matter how far the stock rises, which boils down to the same effect as selling the stock at that point. This resonates with the belief of fundamental traders that an undervalued stock will return to its fundamental value where they will tend to sell it.

The Symbiosis: Long the StockWhen we combine these two perspectives, an intriguing scenario unfolds.

The combination of long call (representing the technical analysis approach) and short put (symbolizing the fundamental analysis approach) results in being long the stock. This is a situation where the investor owns the stock and will profit if the stock’s price rises—a strategy commonly known as buy-and-hold.

The Synthesis: Technical + Fundamental Analysis = Buy-and-HoldThis takes us to an interesting conclusion: technical and fundamental analysis can be seen as two integral parts of a holistic investment approach. While technical analysis represents the momentum-driven, trend-following aspect (long call), fundamental analysis brings in the mean-reversion, value-investing perspective (short put). Together, they form the age-old, tried-and-true strategy: buy-and-hold.

In the next post, we will use these insights to build a full-fledged multi-agent simulation of the stock market with R, so stay tuned!

DISCLAIMER
This post is written on an “as is” basis for educational purposes only and comes without any warranty. The findings and interpretations are exclusively those of the author and are not endorsed by or affiliated with any third party.

In particular, this post provides no investment advice! No responsibility is taken whatsoever if you lose money.

(If you make any money though I would be happy if you would buy me a coffee… that is not too much to ask, is it? )

To leave a comment for the author, please follow the link and comment on their blog: R-Bloggers – Learning Machines.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Big Picture: Technical + Fundamental Analysis = Buy-and-Hold

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWhen working with data, it is important to be aware of the file size of the data you are working with. This is especially true when you are working with large datasets, as the file size can have a significant impact on the performance of your code.

In R, there are a number of different ways to write data to files. Each method has its own advantages and disadvantages, and the file size of the output can vary depending on the method you use.

In this blog post, we will discuss why it is a good idea to check the file size output for different methods. We will also provide three examples of how to check the file size output using the R libraries writexl, openxlsx, and xlsx.

Why Check File Size Output?There are a number of reasons why it is a good idea to check the file size output for different methods.

To ensure that the data is being written correctly: If the file size of the output is significantly different from the expected file size, it is a sign that something may have gone wrong when the data was being written to the file. This could be due to a number of factors, such as a typo in the code, a problem with the file system, or a bug in the R library.
To optimize performance: The file size of the output can have a significant impact on the performance of your code. If the file size is too large, it can take longer to read the data from the file, which can slow down your code.
To troubleshoot problems: If you are having problems with your code, checking the file size output can help you to identify the source of the problem. For example, if the file size of the output is significantly smaller than the expected file size, it is a sign that the data may not be being written to the file correctly.

Examples of Checking File Size OutputIn R, there are a number of different ways to check the file size output. Here are three examples of how to check the file size output using the R libraries writexl, openxlsx, and xlsx:

writexlTo check the file size output of the writexl::write_xlsx() function, you can use the file.info() function. For example, the following code will write the iris dataset to a temporary file and then print the file size of the

library(writexl)write\_xlsx(iris, tmp1 <- tempfile())file.info(tmp1)$size

[1] 8497 openxlsxTo check the file size output of the openxlsx::write.xlsx() function, you can use the file.info() function. For example, the following code will write the iris dataset to a temporary file and then print the file size of the output:

library(openxlsx)write.xlsx(iris, tmp2 <- tempfile())file.info(tmp2)$size

[1] 9631 xlsxTo check the file size output of the xlsx::write.xlsx() function, you can use the file.info() function. For example, the following code will write the iris dataset to a temporary file and then print the file size of the output:

library(xlsx)write.xlsx(iris, tmp3 <- paste0(tempfile(), ".xlsx"))file.info(tmp3)$size

[1] 7905 ConclusionIn this blog post, we discussed why it is a good idea to check the file size output for different methods. We also provided three examples of how to check the file size output using the R libraries writexl, openxlsx, and xlsx.

By checking the file size output, you can ensure that the data is being written correctly, optimize the performance of your code, and troubleshoot problems.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Why Check File Size Output for Different Methods?

View Details

[This article was first published on R on The broken bridge between biologists and statisticians, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Again into a subject that is rather important for most agronomists, i.e. the selection of crop varieties. All farmers are perfectly aware that crop performances are affected both by the genotype and by the environment. These two effects are not purely additive and they often show a significant interaction. By this word, we mean that a genotype can give particularly good/bad performances in some specific environmental situations, which we may not expect, considering its average behaviour in other environmental conditions. The Genotype by Environment (GE) interaction may cause changes in the ranking of genotypes, depending on the environment and may play a key role in varietal recommendation, for a given mega-environment.

GE interactions are usually studied by way of Multi-Environment Trials (MET), where experiments are repeated across several years, locations or any combinations of those. Traditional techniques of data analyses, such as two-way ANOVA, give very little insight on the stability/reliability of genotypes across environments and, thus, other specialised techniques are necessary, to shed light on interaction effects. I have already talked about stability analyses in other posts, such as in this post about the stability variance or in this other post about the environmental variance. Now, I would like to propose some simple explanation about the AMMI analysis. AMMI stands for: Additive Main effect Multiplicative Interaction and it has become very much in fashion in the last 20-25 years.

Let’s start with a real MET example.

A MET with faba beanThis experiment consists of 12 faba bean genotypes (well, it was, indeed, 6 genotypes in two sowing dates; but, let’s disregard this detail from now on) in four blocks, two locations and three years (six environments, in all). The dataset is online available as ‘fabaBean.csv’ and it has been published by Stagnari et al. (2007).

First of all, let’s load the dataset and transform the block variable into a factor. Let’s also inspect the two-way table of means, together with the marginal means for genotypes and environments, which will be useful later. In this post, we will make use of the packages ‘dplyr’ (Wickham et al., 2020), ‘emmeans’ (Lenth, 2020) and ‘aomisc’; this latter is the companion package for this website and must have been installed as detailed in this page here.

```

options(width = 70)rm(list=ls())# library(devtools)# install_github("OnofriAndreaPG/aomisc")library(reshape)library(emmeans)library(aomisc)fileName <- "https://www.casaonofri.it/_datasets/fabaBean.csv"dataset <- read.csv(fileName, header=T)dataset <- transform(dataset, Block = factor(Block), Genotype = factor(Genotype), Environment = factor(Environment))head(dataset)## Genotype Block Environment Yield## 1 Chiaro_A 1 bad_1 4.36## 2 Chiaro_P 1 bad_1 2.76## 3 Collameno_A 1 bad_1 3.01## 4 Collameno_P 1 bad_1 2.50## 5 Palomb_A 1 bad_1 3.85## 6 Palomb_P 1 bad_1 2.21## Two-ways table of meansGEmedie <- cast(Genotype ~ Environment, data = dataset, value = "Yield", fun=mean)GEmedie## Genotype bad_1 bad_2 bad_3 pap_1 pap_2 pap_3## 1 Chiaro_A 4.1050 2.3400 4.1250 4.6325 2.4100 3.8500## 2 Chiaro_P 2.5075 1.3325 4.2025 3.3225 1.4050 4.3175## 3 Collameno_A 3.2500 2.1150 4.3825 3.8475 2.2325 4.0700## 4 Collameno_P 1.9075 0.8475 3.8650 2.5200 0.9850 4.0525## 5 Palomb_A 3.8400 2.0750 4.2050 5.0525 2.6850 4.6675## 6 Palomb_P 2.2500 0.9725 3.2575 3.2700 0.8825 4.0125## 7 Scuro_A 4.3700 2.1050 4.1525 4.8625 2.1275 4.2050## 8 Scuro_P 3.0500 1.6375 3.9300 3.7200 1.7475 4.5125## 9 Sicania_A 3.8300 1.9450 4.5050 3.9550 2.2350 4.2350## 10 Sicania_P 3.2700 0.9900 3.7300 4.0475 0.8225 3.8950## 11 Vesuvio_A 4.1375 2.0175 4.0275 4.5025 2.2650 4.3225## 12 Vesuvio_P 2.1225 1.1800 3.5250 3.0950 0.9375 3.6275## Marginal means for genotypesapply(GEmedie, 1, mean)## Chiaro_A Chiaro_P Collameno_A Collameno_P Palomb_A ## 3.577083 2.847917 3.316250 2.362917 3.754167 ## Palomb_P Scuro_A Scuro_P Sicania_A Sicania_P ## 2.440833 3.637083 3.099583 3.450833 2.792500 ## Vesuvio_A Vesuvio_P ## 3.545417 2.414583## Marginal means for environmentsapply(GEmedie, 2, mean)## bad_1 bad_2 bad_3 pap_1 pap_2 pap_3 ## 3.220000 1.629792 3.992292 3.902292 1.727917 4.147292## Overall meanmean(as.matrix(GEmedie))## [1] 3.103264

``` What model could we possibly fit to the above data? The basic two-way ANOVA model is:

[Y_{ijk} = \mu + \gamma_{jk} + g_i + e_j + ge_{ij} + \varepsilon_{ijk} \quad \quad (1)]

where the yield (Y) for given block (k), environment (j) and genotype (i) is described as a function of the effects of blocks within environments ((\gamma)), genotypes ((g)), environments ((e)) and GE interaction ((ge)). The residual error term (\varepsilon) is assumed to be normal and homoscedastic, with standard deviation equal to (\sigma). Let’s also assume that both the genotype and environment effects are fixed: this is useful for teaching purposes and it is reasonable, as we intend to study the behaviour of specific genotypes in several specific environments.

The interaction effect (ge), under some important assumptions (i.e. balanced data, no missing cells and homoscedastic errors), is given by:

[ge_{ij} = Y_{ij.} - \left( \mu + g_i + e_j \right) = Y_{ij.} - Y_{i..} - Y_{.j.} + \mu \quad \quad (2)]

where (Y_{ij.}) is the mean of the combination between the genotype (i) and the environment (j), (Y_{i..}) is the mean for the genotype (i) and (Y_{.j.}) is the mean for the environment (j). For example, for the genotype ‘Chiaro_A’ in the environment ‘bad_1’, the interaction effect was:

4.1050 - 3.577 - 3.22 + 3.103## [1] 0.411 We see that the interaction was positive, in the sense that ‘Chiaro_A’, gave 0.411 tons per hectare more than we could have expected, considering its average performances across environments and the average performances of all genotypes in ‘bad_1’.

More generally, the two-way table of interaction effects can be obtained by doubly centring the matrix of means, as shown in the following box.

GE <- as.data.frame(t(scale( t(scale(GEmedie, center=T, scale=F)), center=T, scale=F)))print(round(GE, 3))## bad\_1 bad\_2 bad\_3 pap\_1 pap\_2 pap\_3## Chiaro\_A 0.411 0.236 -0.341 0.256 0.208 -0.771## Chiaro\_P -0.457 -0.042 0.466 -0.324 -0.068 0.426## Collameno\_A -0.183 0.272 0.177 -0.268 0.292 -0.290## Collameno\_P -0.572 -0.042 0.613 -0.642 -0.003 0.646## Palomb\_A -0.031 -0.206 -0.438 0.499 0.306 -0.131## Palomb\_P -0.308 0.005 -0.072 0.030 -0.183 0.528## Scuro\_A 0.616 -0.059 -0.374 0.426 -0.134 -0.476## Scuro\_P -0.166 0.011 -0.059 -0.179 0.023 0.369## Sicania\_A 0.262 -0.032 0.165 -0.295 0.160 -0.260## Sicania\_P 0.361 -0.329 0.048 0.456 -0.595 0.058## Vesuvio\_A 0.475 -0.054 -0.407 0.158 0.095 -0.267## Vesuvio\_P -0.409 0.239 0.221 -0.119 -0.102 0.169 Please, note that the overall mean for all elements in ‘GE’ is zero and the sum of squares is equal to a fraction of the interaction sum of squares in ANOVA (that is (RMSE/r); where (r) is the number of blocks).

mean(unlist(GE))## [1] 6.914424e-18sum(GE^2)## [1] 7.742996mod <- lm(Yield ~ Environment/Block + Genotype*Environment, data = dataset)anova(mod)## Analysis of Variance Table## ## Response: Yield## Df Sum Sq Mean Sq F value Pr(>F) ## Environment 5 316.57 63.313 580.9181 < 2.2e-16 ***## Genotype 11 70.03 6.366 58.4111 < 2.2e-16 ***## Environment:Block 18 6.76 0.375 3.4450 8.724e-06 ***## Environment:Genotype 55 30.97 0.563 5.1669 < 2.2e-16 ***## Residuals 198 21.58 0.109 ## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 130.97/4## [1] 7.7425 Decomposing the GE matrixIt would be nice to be able to give a graphical summary of the GE matrix; in this regard, we could think of using Principal Component Analysis (PCA) via Singular Value Decomposition (SVD). This has been shown by Zobel et al. (1988) and, formerly, by Gollob (1968). May I just remind you a few things about PCA and SVD? No overwhelming math detail, I promise!

Most matrices (and our GE matrix) can be decomposed as the product of three matrices, according to:

[X = U D V^T \quad \quad (3)]

where (X) is the matrix to be decomposed, (U) is the matrix of the first (n) eigenvectors of (XX^T), (V) is the matrix of the first (n) eigenvectors of (X^T X) and (D) is the diagonal matrix of the first (n) singular values of (XX^T) (or (X^T X); it does not matter, they are the same).

Indeed, if we want to decompose our GE matrix, it is more clever (and more useful to our purposes), to write the following matrices:

[S_g = U D^{1/2} \quad \quad (4)]

and:

[S_e = V D^{1/2} \quad \quad (5)]

so that

[GE = S_g \, S_e^T \quad \quad (6)]

(S_g) is the matrix of row-scores (genotype scores) and (S_e) is the matrix of column scores (environment scores). Let me give you an empirical proof, in the box below. In order to find (S_g) and (S_e), I will use a mathematical operation that is known as Singular Value Decomposition (SVD):

U <- svd(GE)$uV <- svd(GE)$vD <- diag(svd(GE)$d)Sg <- U %*% sqrt(D)Se <- V %*% sqrt(D)row.names(Sg) <- levels(dataset$Genotype)row.names(Se) <- levels(dataset$Environment)colnames(Sg) <- colnames(Se) <- paste("PC", 1:6, sep ="")round(Sg %*% t(Se), 3)## bad\_1 bad\_2 bad\_3 pap\_1 pap\_2 pap\_3## Chiaro\_A 0.411 0.236 -0.341 0.256 0.208 -0.771## Chiaro\_P -0.457 -0.042 0.466 -0.324 -0.068 0.426## Collameno\_A -0.183 0.272 0.177 -0.268 0.292 -0.290## Collameno\_P -0.572 -0.042 0.613 -0.642 -0.003 0.646## Palomb\_A -0.031 -0.206 -0.438 0.499 0.306 -0.131## Palomb\_P -0.308 0.005 -0.072 0.030 -0.183 0.528## Scuro\_A 0.616 -0.059 -0.374 0.426 -0.134 -0.476## Scuro\_P -0.166 0.011 -0.059 -0.179 0.023 0.369## Sicania\_A 0.262 -0.032 0.165 -0.295 0.160 -0.260## Sicania\_P 0.361 -0.329 0.048 0.456 -0.595 0.058## Vesuvio\_A 0.475 -0.054 -0.407 0.158 0.095 -0.267## Vesuvio\_P -0.409 0.239 0.221 -0.119 -0.102 0.169 Let’s have a look at (S_g) and (S_e): they are two interesting entities. I will round up a little to make them smaller, and less scaring.

round(Sg, 3)## PC1 PC2 PC3 PC4 PC5 PC6## Chiaro\_A -0.607 -0.384 0.001 0.208 -0.063 0## Chiaro\_P 0.552 0.027 -0.081 0.045 0.164 0## Collameno\_A 0.084 -0.542 -0.006 0.176 0.057 0## Collameno\_P 0.807 -0.066 -0.132 -0.172 0.079 0## Palomb\_A -0.321 0.110 0.591 -0.083 0.389 0## Palomb\_P 0.281 0.346 0.282 0.042 -0.253 0## Scuro\_A -0.626 0.139 -0.163 0.017 -0.080 0## Scuro\_P 0.230 0.077 0.182 -0.207 -0.242 0## Sicania\_A -0.063 -0.324 -0.355 -0.280 0.090 0## Sicania\_P -0.214 0.683 -0.402 0.148 0.151 0## Vesuvio\_A -0.438 -0.008 0.020 -0.300 -0.177 0## Vesuvio\_P 0.316 -0.058 0.063 0.405 -0.114 0round(Se, 3)## PC1 PC2 PC3 PC4 PC5 PC6## bad\_1 -0.831 0.095 -0.467 -0.317 -0.151 0## bad\_2 0.044 -0.418 0.070 0.371 -0.403 0## bad\_3 0.670 -0.130 -0.525 0.171 0.298 0## pap\_1 -0.661 0.513 0.289 0.314 0.221 0## pap\_2 -0.069 -0.627 0.420 -0.294 0.208 0## pap\_3 0.846 0.567 0.213 -0.244 -0.173 0 Both matrices have 6 columns. Why six, are you asking? I promised I would not go into math detail; it’s enough to know that the number of columns is always equal to the minimum value between the number of genotypes and the number of environments. The final column is irrelevant (all elements are 0). (S_g) has 12 rows, one per genotype; these are the so called genotype scores: each genotype has six scores. (S_e) has six rows, one per environment (environment scores).

You may have some ‘rusty’ memories about matrix multiplication; however, what we have discovered in the code box above is that the GE interaction for the (i^{th}) genotype and the (j^{th}) environment can be obtained as the product of genotype scores and environments scores. Indeed:

[ge_{ij} = \sum_{z = 1}^n \left[ S_g(iz) \cdot S_e(jz) \right] \quad \quad (7)]

where (n) is the number of columns (number of principal components). An example is in order, at this point; again, let’s consider the first genotype and the first environment. The genotype and environments scores are in the first columns of (S_g) and (S_e); if we multiply the elements in the same positioning (1st with 1st, 2nd with 2nd, and so on) and sum up, we get:

-0.607 * -0.831 +-0.384 * 0.095 + 0.001 * -0.467 + 0.208 * -0.317 + -0.063 * -0.151 + 0 * 0## [1] 0.411047 It’s done: we have transformed the interaction effect into the sum of multiplicative terms. If we replace Equation 7 into the ANOVA model above (Equation 1), we obtain an Additive Main effects Multiplicative Interaction model, i.e. an AMMI model.

Reducing the rankIn this case we took all available columns in (S_g) and (S_e). For the sake of simplicity, we could have taken only a subset of those columns. The Eckart-Young (1936) theorem says that, if we take (m < 6) columns, we obtain the best possible approximation of GE in reduced rank space. For example, let’s use the first two columns of (S_g) and (S_e) (the first two principal component scores):

PC <- 2Sg2 <- Sg[,1:PC]Se2 <- Se[,1:PC]GE2 <- Sg2 %*% t(Se2)print ( round(GE2, 3) )## bad\_1 bad\_2 bad\_3 pap\_1 pap\_2 pap\_3## Chiaro\_A 0.468 0.134 -0.357 0.205 0.282 -0.732## Chiaro\_P -0.456 0.013 0.367 -0.351 -0.055 0.482## Collameno\_A -0.122 0.230 0.127 -0.334 0.334 -0.236## Collameno\_P -0.676 0.063 0.549 -0.567 -0.014 0.645## Palomb\_A 0.277 -0.060 -0.230 0.269 -0.047 -0.209## Palomb\_P -0.201 -0.132 0.144 -0.009 -0.236 0.434## Scuro\_A 0.534 -0.086 -0.438 0.486 -0.044 -0.451## Scuro\_P -0.184 -0.022 0.144 -0.113 -0.064 0.238## Sicania\_A 0.022 0.133 0.000 -0.124 0.207 -0.237## Sicania\_P 0.243 -0.295 -0.232 0.492 -0.414 0.206## Vesuvio\_A 0.363 -0.016 -0.293 0.286 0.035 -0.375## Vesuvio\_P -0.268 0.038 0.219 -0.239 0.015 0.234 GE2 is not equal to GE, but it is a close approximation. A close approximation in what sense?… you may wonder. Well, the sum of squared elements in GE2 is as close as possible (with (n = 2)) to the sum of squared elements in GE:

sum(GE2^2)## [1] 6.678985 We see that the sum of squares in GE2 is 86% of the sum of squares in GE. A very good approximation, isn’t it? It means that the variability of yield across environments is described well enough by using a relatively low number of parameters (scores). However, the multiplicative part of our AMMI model needs to be modified:

[ge_{ij} = \sum_{z = 1}^m \left[ s_{g(iz)} \cdot s_{e(jz)} \right] + \xi_{ij}]

Indeed, a residual term (\xi_{ij}) is necessary, to account for the fact that the sum of multiplicative terms is not able to fully recover the original matrix GE. Another example? For the first genotype and the first environment the multiplicative interaction is:

-0.607 * -0.831 + -0.384 * 0.095## [1] 0.467937 and the residual term (\xi_{11}) is

0.41118056 -0.607 * -0.831 + -0.384 * 0.095## [1] 0.8791176 Clearly, the residual terms need to be small enough to be negligible, otherwise the approximation in reduced rank space is not good enough.

Why is this useful?Did you get lost? Hope you didn’t, but let’s make a stop and see where we are standing now. We started from the interaction matrix GE and found a way to decompose it as the product of two matrices, i.e. (S_g) and (S_e), a matrix of genotype scores and a matrix of environment scores. We discovered that we could obtain a good approximation of GE by working in reduced rank space and we only used two genotypic scores and two environment scores, in place of the available six.

This is great! Now we have the ability of drawing a biplot, i.e. we can plot both genotypic scores and environmental scores in a dispersion graph (biplot: two plots in one), as we see below.

biplot(Sg[,1:2], Se[,1:2], xlim = c(-1, 1), ylim = c(-1, 1), xlab = "PC 1", ylab = "PC 2")abline(h = 0, lty = 2)abline(v = 0, lty = 2) This graph provides a very effective description of GE interaction effects. I will not go into detail, here. Just a few simple comments:

genotypes and environments lying close to the origin of the axes do not interact with each other (the product of scores would be close to 0)
genotype and environments lying far away from the origin of axes show very big interaction and, therefore, high yield instability. Someone says that the euclidean distance from the origin should be taken as a measure of instability
the interaction is positive, when genotypes and environments are close to each other. If two objects are close, their scores (co-ordinates) will have the same signs and thus their product will be positive.
the interaction is negative, when genotypes and environments are far away from each other. If two objects are distant, their scores (co-ordinates) will have opposte signs and thus their product will be negative.
For instance, ‘Palomb_P’, ‘Scuro_P’, ‘Chiaro_P’ and ‘Collameno_P’ gave particularly good yields in the environments ‘pap_3’ and ‘bad_3’, while ‘Scuro_A’, ‘Palomb_A’ and ‘Vesuvio_A’ gave particularly good yields (compared to their average) in the environments ‘pap_1’ and ‘bad_1’. ‘Sicania_A’ and ‘Collameno_A’ gave good yields in ‘bad_2’ and ‘pap_2’.

How many components?In my opinion, AMMI analysis is mainly a visualisation method. Therefore, we should select as many components (columns in (S_g) and (S_e)) as necessary to describe a main part of the interaction sum of squares. In our example, two components are enough, as they represent 86% of the interaction sum of squares.

However, many people (and reviewers) are still very concerned with formal hypothesis testing. Therefore, we could proceed in a sequential fashion, and introduce the components one by one.

The first component has a sum of squares equal to:

PC <- 1Sg2 <- Sg[,1:PC]Se2 <- Se[,1:PC]GE2 <- Sg2 %*% t(Se2)sum(GE2^2)## [1] 5.290174 We have seen that the second component has an additional sum of squares equal to:

6.678985 - 5.290174## [1] 1.388811 We can go further ahead and get the sum of squares for all components. According to Zobel (1988), the degrees of freedom for each component are equal to:

[ df_n = i + j - 1 - 2m ]

where (i) is the number of genotypes, (j) is the number of environments, and (m) is the number of the selected components. In our case, the first PC has 15 DF, the second one has 13 DF and so on.

If we can have a reliable estimate of the pure error variance (\sigma^2) (see above), we can test the significance of each component by using F tests (although some authors argue that this is too a liberal approach; see Cornelius, 1993).

Simple AMMI analysis with RWe have seen that AMMI analysis, under the hood, is a sort of PCA. Therefore, it could be performed, in R by using one of the available packages for PCA. For the sake of simplicity, I have coded the simple AMMI() function, that is available in the ‘aomisc’ package. I have described an earlier version of this function in an ‘R news’ paper (Onofri and Ciriciofolo, 2007).

Tha above function works equally well with raw MET data, containing all replicated values, or with the ‘genotype by environment’ average values. In this second case, the analyses proceed in two-steps, as I will describe below.

First step on raw dataDuring the first step we need to obtain a reliable matrix of means for the ‘genotype x environment’ combinations. If the environment is fixed, we can use least squares means, which are unbiased, also when some observations are missing. If the environment effect is random, we could use the BLUPs, but we will not consider such an option here.

In the box below we take the ‘mod’ object from a two way ANOVA fit and derive the residual mean square (RMSE), which we divide by the number of blocks. This will be our error term to test the significance of components. Later, we pass the ‘mod’ object to the ‘emmeans()’ function, to retrieve the expected marginal means for the ‘genotype by environment’ combinations and proceed to the second step.

RMSE <- summary(mod)$sigma^2 / 4dfr <- mod$df.residualge.lsm <- emmeans(mod, ~Genotype:Environment)ge.lsm <- data.frame(ge.lsm)[,1:3] Second step on least square meansThis second step assumes that the residual variances for all environments are homogeneous. If so (we’d better check this), we can take the expected marginal means (‘ge.lsm’) and submit them to AMMI analysis, by using the AMMI() function. The syntax is fairly obvious; we also pass to it the RMSE and its degrees of freedom. The resulting object can be explored, by using the appropriate slots.

AMMIobj <- AMMI(yield = ge.lsm$emmean, genotype = ge.lsm$Genotype, environment = ge.lsm$Environment, MSE = RMSE, dfr = dfr)#AMMIobj$genotype\_scores## PC1 PC2## Chiaro\_A -0.60710888 -0.383732821## Chiaro\_P 0.55192742 0.026531045## Collameno\_A 0.08444877 -0.542185666## Collameno\_P 0.80677055 -0.065752971## Palomb\_A -0.32130513 0.110117240## Palomb\_P 0.28104959 0.345909298## Scuro\_A -0.62638795 0.139185954## Scuro\_P 0.22961347 0.076555540## Sicania\_A -0.06286803 -0.323857285## Sicania\_P -0.21433211 0.683296898## Vesuvio\_A -0.43786742 -0.007914342## Vesuvio\_P 0.31605973 -0.058152890#AMMIobj$environment\_scores## PC1 PC2## bad\_1 -0.83078550 0.09477362## bad\_2 0.04401963 -0.41801637## bad\_3 0.67043214 -0.12977423## pap\_1 -0.66137357 0.51268429## pap\_2 -0.06863235 -0.62703224## pap\_3 0.84633965 0.56736492#AMMIobj$mult\_Interaction## Effect SS DF MS F Prob. Perc\_of\_Total\_SS## 1 PC1 5.290174 15 0.3526782 12.943700 2.926881e-22 0.6832205## 2 PC2 1.388812 13 0.1068317 3.920847 1.135641e-05 0.1793636## 3 Residuals 1.064011 27 0.0394078 1.446312 8.056050e-02 0.1374159 In detail, we can retrieve the genotype and environment scores, the proportion of the GE variance explained by each component and the significance of PCs.

I agree, the AMMI() function is not very ambitious. However, it is simple enough to be usable and give reliable results, as long as the basic assumptions for the method are respected. Furthermore, there is also a complimentary biplot() method, that draws either the biplot type 1 (PC1 for genotypes and environments against genotypic/environment means) or the biplot type 2 (PC1 against PC2). The code is shown below.

biplot(AMMIobj, xlab = "Yield")biplot(AMMIobj, biplot = 2) You may also consider to explore other more comprehensive R packages, such as ‘agricolae’ (de Mendiburu, 2020).

Thank you for reading, so far, and… happy coding!

Prof. Andrea Onofri
Department of Agricultural, Food and Environmental Sciences
University of Perugia (Italy)
Send comments to: andrea.onofri@unipg.it

Follow @onofriandreapg

Literature references1. Annichiarico, P. (1997). Additive main effects and multiplicative interaction (AMMI) analysis of genotype-location interaction in variety trials repeated over years. Theoretical applied genetics, 94, 1072-1077. 2. Ariyo, O. J. (1998). Use of additive main effects and multiplicative interaction model to analyse multilocation soybean varietal trials. J. Genet. and Breed, 129-134. 3. Cornelius, P. L. (1993). Statistical tests and retention of terms in the Additive Main Effects and Multiplicative interaction model for cultivar trials. Crop Science, 33,1186-1193. 4. Crossa, J. (1990). Statistical Analyses of multilocation trials. Advances in Agronomy, 44, 55-85. 5. Gollob, H. F. (1968). A statistical model which combines features of factor analytic and analysis of variance techniques. Psychometrika, 33, 73-114. 6. Lenth R., 2020. emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.4.6. https://github.com/rvlenth/emmeans. 7. de Mendiburu F., 2020. agricolae: Statistical Procedures for Agricultural Research. R package version 1.3-2. https://CRAN.R-project.org/package=agricolae. 8. Onofri, A., Ciriciofolo, E., 2007. Using R to perform the AMMI analysis on agriculture variety trials. R NEWS 7, 14–19. 9. Stagnari F., Onofri A., Jemison J., Monotti M. (2006). Multivariate analyses to discriminate the behaviour of faba bean (Vicia faba L. var. minor) varieties as affected by sowing time in cool, low rainfall Mediterranean environments. Agronomy For Sustainable Development, 27, 387–397. 10. Hadley Wickham, Romain François, Lionel Henry and Kirill Müller, 2020. dplyr: A Grammar of Data Manipulation. R package version 0.8.5. https://CRAN.R-project.org/package=dplyr. 11. Zobel, R. W., Wright, M.J., and Gauch, H. G. (1988). Statistical analysis of a yield trial. Agronomy Journal, 388-393. To leave a comment for the author, please follow the link and comment on their blog: R on The broken bridge between biologists and statisticians.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: AMMI analyses for multi-environment studies

View Details

[This article was first published on free range statistics - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In a workshop session on data visualization earlier this week, Christophe Bontemps of the Statistical Institute for Asia and Pacific (SIAP) used as an example a range of infographics attempting to show, on a map of Asia and the Pacific, the proportion of members of Parliament that are women. The point of the exercise was to illustrate the challenges with the various ways it could be done – using colours, shapes or sizes of glyphs drawn in the countries centres, or as a choropleth map – none of the methods were particularly satisfactory.

Building a nice map showing proportionsI did think the most promising method was when the proportion was shown as a vertical bar for each country. As used by Christophe this wasn’t great because it was difficult to compare the heights of the bars scattered all over the map (which was the point of his illustration). But mixing this up with an idea he’d mentioned earlier in his talk, I thought the bars could work if you could add a reference line or shape showing 50%, and also use the colour coding. Plus a participant asked for numbers added to the map, which we agreed could work so long as it didn’t get cluttered.

I had some sleep-deprived time in an airport the morning after so thought I’d give a go to implementing this approach, and came up with this, which I think is pretty good!

I did cheat a bit by knocking out the Middle East because the map got really cluttered over there. For many purposes Middle East is treated differently to Asia by various international organisations, so there is some justification in this. There’s also an issue that my bars’ heights are calculated in degrees latitude, and with the projection of the map onto flat screen that should mean that the bars in countries with higher latitudes will be shorter. But this doesn’t seem to be a visual problem, because of the effectiveness of the pale blue 50% reference blocks.

There’s some spots where the clutter stopped the numbers being readable in my first iteration – for North Korea and for Cambodia and Laos and a few others – and I had to code in some ad hoc adjustments for them. But overall the effect was pretty good, even before I made those corrections.

In terms of the substance here, not much to say. It really stands out how New Zealand, Australia and (perhaps more suprisingly to some) Timor Leste are the front runners with near-equity in gender representation in Parliament. Uzbekhistan, Nepal and Vietnam make up the second tier with 30% or higher. Pacific Island countries, Japan and South Asia (other than Nepal) stand out in the other direction.

Nepal and Timor Leste both have formal quota requirements for women in Parliament. Check out the Global Database of Gender Quotas in Parliaments worldwide, maintained by International IDEA, the Inter-Parliamentary Union and Stockholm University.

Here’s the code that makes the map, all in one chunk. It turned out to be pretty straightforward. The main trick is I had to pre-calculate coordinates for the heights of the 50% rectangle and the segments showing the actual height, rather than letting ggplot calculate it on the fly like it would for a bar chart.

Post continues after R code

library(tidyverse)library(rnaturalearth)library(sf)library(WDI)library(extrafont)library(RColorBrewer)#-----------------data prep--------------------# Data on proportion of women in Parliament from World Development Indicators:wom\_raw <- WDI(indicator = "SG.GEN.PARL.ZS", start = 2010)# Get the latest year for each country:wom <- wom\_raw |> as\_tibble() |> rename(wom\_parl = SG.GEN.PARL.ZS) |> filter(!is.na(wom\_parl)) |> group\_by(iso3c) |> arrange(desc(year)) |> slice(1) |> ungroup() # maps for Asia and Pacific from Natural earthsf\_use\_s2(FALSE)m1 <- ne\_countries(scale = "medium", returnclass = "sf", continent = "asia") m2 <- ne\_countries(scale = "medium", returnclass = "sf", continent = "oceania") m <- rbind(m1, m2) # extract the centres:centers <- st\_centroid(m, of\_largest\_polygon = TRUE) |> # turn them into straight numbers, not sf geometries: st\_coordinates() |> # add those coordinates as columns to the original map: cbind(m) |> # join to the women in parliament data: left\_join(wom, by = c("iso\_a3" = "iso3c")) |> rename(x = X, y = Y) |> mutate(x = case\_when( # put Tonga and Samoa back on the map (the land won't show but the bars will) x < 0 ~ x + 360, # Put Kiribati where Tarawa is rather than its spatial centre: country == "Kiribati" ~ 173, TRUE ~ x )) |> # a couple of manual adjustments to make things more readable, avoiding overlaps mutate(x = case\_when( geounit == "North Korea" ~ x - 1.5, geounit == "Vietnam" ~ x + 1, geounit == "Laos" ~ x - 0.8, geounit == "Indonesia" ~ x - 1.8, geounit == "Malaysia" ~ 101.7, # location of Kuala Lumpur TRUE ~ x )) |> # create scaled latitude coordinates for our rectangles and segments: mutate(sc = 5, yend1 = y + wom\_parl / sc, yend0 = y + 50 / sc) |> filter(!is.na(wom\_parl)) |> # knock out the middle east because it's cluttered :): filter(x > 60)#------------------map--------------------------xw <- 1the\_font <- "Calibri"wom\_map <- ggplot(centers) + geom\_sf(colour = "grey50") + geom\_rect(aes(xmin = x - xw, xmax = x + xw, ymin = y, ymax = yend0), fill = "steelblue", alpha = 0.5) + geom\_segment(aes(x = x, xend = x, y = y, yend = yend1, colour = wom\_parl), linewidth = 2) + geom\_text(aes(label = round(wom\_parl), x = x, y = yend1 + 1, colour = wom\_parl), vjust = 0, size = 3, family = the\_font) + # It was useful while troubleshooting to add country names; # uncomment the below if you want them: # geom\_text(aes(label = country, x = x, y = y)) + theme\_void(base\_family = the\_font) + xlim(60, 200) + scale\_colour\_viridis\_c(option = "B", limits = c(0, 60), label = number\_format(suffix = "%")) + theme(legend.position = c(0.2, 0.35)) + labs( colour = "", title = "Percentage of parliamentarians who are women", subtitle = "Blue bars represent 50%")print(wom\_map) AlternativeI do like my map but there’s no denying it’s not an efficient way of transmitting the full information. Great for a sense of global spread, not so good for identifying individual countries (for example, it took me quite a while to realise the Timor Leste bar was Timor Leste, not Indonesia). If we want to compare countries using their names and the spatial element is secondary, we’re better with a more straightforward representation like this one:

Created with this code:

```

----------------alternative visualisation-------------p <- centers |> mutate(clps_income = case_when( # collapse OECD and non-OECD high income and upper middle together, # as they are quite small categories grepl("High income", income_grp) | grepl("Upper middle", income_grp) ~ "High or middle income", TRUE ~ gsub("[0-9]\. ", "", income_grp) )) |> mutate(clps_income = fct_relevel(clps_income, "High or middle income", after = Inf)) |> mutate(country = fct_reorder(country, -wom_parl), subregion = fct_reorder(subregion, wom_parl)) |> ggplot(aes(x = wom_parl, y = country)) + geom_segment(aes(xend = 0, yend = country, colour = subregion), linewidth = 1.5) + geom_point() + geom_text(aes(label = round(wom_parl)), family = the_font, size = 2.5, vjust = 0.5, hjust = 0, nudge_x = 1) + facet_wrap(~clps_income, scales = "free_y", ncol = 2) + # expand = c(0,0) is needed to put the country labels close to the plot scale_x_continuous(expand = c(0, 0), limits = c(0, 60), label = number_format(suffix = "%")) + # exclude the bright yellow from Brewer Set1 colours, which I don't like: scale_colour_manual(values = brewer.pal(9, "Set1")[-6]) + theme_minimal(base_family = the_font) + theme(panel.grid.major.y = element_blank(), axis.text.y=element_text(margin=margin(r=0)), legend.position = c(0.8, 0.2)) + labs(y = "", x = "Proportion of Parliamentarians that are women", colour = "Sub-region", title = "Women parliamentarians in the Asia-Pacific")print(p)

``` That’s all for today.

To leave a comment for the author, please follow the link and comment on their blog: free range statistics - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Showing women proportion of Parliamentarians on a map by @ellis2013nz

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to build reproducible analytical pipelines in R! Join our workshop on Building reproducible analytical pipelines in R which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: Building reproducible analytical pipelines in R

Date: Thursday, June 29th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Bruno Rodrigues. Bruno is currently employed as the head of the statistics department at the Ministry of Higher education and Research in Luxembourg. Before joining the public sector, Bruno worked as a data science consultant in one of the big four accounting companies, and before that as a teaching and research assistant. Bruno discovered tools such as Git and software carpentry techniques while working on his PhD. These tools and techniques served him well for the past decade, and Bruno has been consistently sharing his knowledge on his blog during that time.

Description: This workshop will present some of the tools and techniques that you can use to build reproducible analytical pipelines. We will also learn how to apply these techniques in R. Making sure that your work is reproducible is extremely important because making a pipeline reproducible ensures that the code is of high quality, well-document and tested by design. This way, you won’t have problems communicating results and your collaborators, or future you, will have no trouble understanding the project and re-building it if an update is necessary. In case the project needs to be audited, setting it up as a reproducible analytical pipeline will also will make auditing easy!

It doesn’t matter if you work in research or in the private sector, if you train complex machine learning models or write reports that focus on descriptive statistics: any type of project in any sector can benefit from the ideas presented in this workshop. We will learn about functional and literate programming, and learn how to apply these in R, using {renv}, {targets} and Docker.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Building reproducible analytical pipelines in R workshop was first posted on May 25, 2023 at 10:06 am.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building reproducible analytical pipelines in R workshop

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn the realm of data analysis and manipulation, R has become a popular programming language due to its extensive collection of packages and libraries. One common task is exporting data to Excel files, which allows for easy sharing and presentation of results. In this blog post, we will explore three popular R packages for writing Excel files: writexl, openxlsx, and xlsx. We will compare their performance using the benchmarking package and analyze the results. So let’s dive in!

Setting up the EnvironmentBefore we proceed, make sure you have the necessary packages installed. We will be using the rbenchmark, nycflights13, and dplyr packages. The nycflights13 package provides a dataset named “flights,” which we will use for our benchmarking tests.

library(rbenchmark)library(nycflights13)library(dplyr)

Defining the Number of Replications

To ensure reliable performance measurements, we will repeat each test multiple times. The variable n represents the number of replications, and you can adjust its value depending on your requirements.

n <- 5 Benchmarking the PackagesNow, let’s move on to the actual benchmarking process. We will use the benchmark() function from the rbenchmark package to compare the performance of writexl, openxlsx, and xlsx.

benchmark( "writexl" = { writexl::write\_xlsx(flights, tempfile()) }, "openxlsx" = { openxlsx::write.xlsx(flights, tempfile()) }, "xlsx" = { xlsx::write.xlsx(flights, paste0(tempfile(),".xlsx")) }, replications = n, columns = c( "test","replications","elapsed","relative","user.self","sys.self")) In the code snippet above, we define three tests, each representing one package. We provide the code to execute for each test. For example, in the “writexl” test, we use the write_xlsx() function from the writexl package to write the “flights” dataset to a temporary Excel file.

The replications parameter specifies the number of times each test should be repeated. In our case, we set it to n, which we defined earlier as 5.

The columns parameter defines the columns to include in the benchmarking results. We specify “test” for the test name, “replications” for the number of replications, “elapsed” for the total time taken, “relative” for the relative performance compared to the fastest test, “user.self” for the CPU time used in user code, and “sys.self” for the CPU time used in system code.

Prettifying the ResultsTo make the results more readable, we can use the arrange() function from the dplyr package to sort the results by the “relative” column in ascending order.

arrange(relative) This will arrange the benchmarking results in ascending order of relative performance, allowing us to easily identify the most efficient package.

Benchmark Output test replications elapsed relative user.self sys.self1 writexl 5 0.034 1.000000 0.024 0.0102 openxlsx 5 0.055 1.617647 0.044 0.0113 xlsx 5 0.101 2.941176 0.078 0.023 Interpretation of the ResultsThe results of the benchmark show that writexl is the fastest package for writing to Excel, followed by openxlsx and xlsx. The difference in performance between the three packages is not significant, but writexl is consistently faster than the other two packages.

ConclusionIn this blog post, we compared the performance of three R packages, writexl, openxlsx, and xlsx, for writing Excel files. We used the rbenchmark package to benchmark the packages, considering the number of replications, elapsed time, relative performance, user CPU time, and system CPU time. By arranging the results using the dplyr package, we obtained a sorted view of the relative performance. This analysis can help you choose the most suitable package for your specific needs, considering both performance and functionality.

Remember, benchmarking can vary depending on the dataset and system specifications. So, it’s always a good idea to run your own benchmarks and evaluate the results in your specific context. Happy coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Comparing R Packages for Writing Excel Files: An Analysis of writexl, openxlsx, and xlsx in R

View Details

[This article was first published on R Views, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. One hundred fifty-six new packages made it to CRAN in April. Here are my “Top 40” selections in twelve categories: Computational Methods, Data, Ecology, Economics, Genomics, Machine Learning, Mathematics, Medicine, Science, Statistics, Utilities, and Visualization.

Computational Methodsclarabel v0.4.1: Implements Clarabel, a versatile interior point solver that solves linear programs, quadratic programs, second-order cone programs, and problems with exponential and power cone constraints. See the vignette.

condor v1.0.0: Provides functions to access the Condor high performance computing environment. Files are first uploaded to a submitter machine and the resulting job is then passed on to Condor. Look here for the code.

GPUmatrix v0.1.0: Extends R to use GPUs for matrix computations. See the vignette.

hydroMOPSO v0.1-3: Implements a state-of-the-art Multi-Objective Particle Swarm Optimiser (MOPSO), based on the algorithm developed by Lin et al. (2018) with improvements described by Marinao-Rivas & Zambrano-Bigiarini (2020) which can be used for global optimization of non-smooth and non-linear R functions and other models that need to be run from the system console, e.g. SWAT+.

Datadataverifyr v0.1.5: Provides a thin wrapper around dplyr, data.table, arrow, and DBI to allow users to define rules which can be used to verify a given dataset. See README to get started.

neotoma2 v1.0.0: Provides functions to access and manipulate data in the Neotoma Paleoecology Database. See the vignette.

rpaleoclim v1.0.0: Implements an interface to PaleoClim, a set of free, high resolution paleoclimate surfaces covering the whole globe that includes data on surface temperature, precipitation and the standard bioclimatic variables commonly used in ecological modelling. See Brown et al. (2019) for background and the vignette.

zctaCrosswalk v2.0.0: Contains the US Census Bureau’s 2020 ZCTA to County Relationship File, as well as convenience functions to translate between States, Counties and ZIP Code Tabulation Areas (ZCTAs). See the Introduction and the vignettes Workflow with tidycensus, and Developer Notes.

EcologyEWSmethods v1.1.2: Implements methods for forecasting tipping points at the community level that include rolling and expanding window approaches to assessing abundance based early warning signals, non-equilibrium resilience measures, and machine learning. See Dakos et al. (2012), Deb et al. (2022), and Drake and Griffen (2010) for background and the vignette for an introduction.

fqacalc v1.0.0: Provides functions for calculating Floristic Quality Assessment (FQA) metrics using regional FQA databases that have been approved or approved with reservations as ecological planning models by the U.S. Army Corps of Engineers (USACE). For information on FQA see Spyreas (2019). There is an Introduction.

Economicsclptheory v0.1.0: Provides functions to compute the uniform rate of profit, the vector of price of production and the vector of labor values, and also compute measures of deviation between relative prices of production and relative values. See Basu and Moraltis (2023) for background and README for an introduction.

GenomicsBREADR v1.0.1: Implements a method for estimating degrees of relatedness for extreme low-coverage genotype data and includes functions to quantify and visualize the level of confidence in the estimated degrees of relatedness. See Rohrlach et al. (2023) for package details and README for examples.

crosshap v1.2.2: Implements a local haplotyping visualization toolbox to capture major patterns of co-inheritance between clusters of linked variants, while connecting findings to phenotypic and demographic traits across individuals. See Marsh et al. (2022) for a detailed example and README for an introduction.

DAISIEprep v0.3.2: Extracts colonization and branching times of island species for analysis with the DAISIE package. There is a Tutorial and there are vignettes on Performance and Sensitivity.

Machine LearningCCMMR v0.1: Implements the convex clustering through majorization-minimization algorithm described in Touw, Groenen, and Terada (2022) to minimize the convex clustering loss function. See README for examples.

rcccd v0.3.2: Provides functions to fit class cover catch digraph classification models. Methods are explained in Priebe et al. (2001), Priebe et al. (2003), and Manukyan and Ceyhan (2016). README contains some description.

TheOpenAir v0.1.0: Implements a wrapper using the OpenAI API as a back end to integrate ChatGPTinto diverse data-related tasks, such as data cleansing and automating analytics scripts. See README to get started.

Mathematicscyclotomic v1.1.0: Implements algorithms from the GAP project to work with cyclotomic numbers: complex numbers that can be thought of as the rational numbers extended with the roots of unity. They have applications in number theory, algebraic geometry, algebraic number theory, coding theory, in the theory of graphs and combinatorics, and in the theory of modular functions and modular curves. See README for examples.

markovmix v0.1.1: Provides functions to fit a mixture of Markov chains of higher orders from multiple sequences along with various utility functions to derive transition patterns, transition probabilities per component and component priors. See README for examples.

MedicineDiDforBigData v1.0: Provides a big-data-friendly and memory-efficient difference-in-differences estimator for staggered (and non-staggered) treatment contexts. See the Get Started Guide the vignettes Background, Examples, and Theory.

predictNNB v0.1.0: Provides tools to estimate when and where a model-guided treatment strategy may outperform a treat-all or treat-none approach using Monte Carlo simulation and evaluation of the Net Monetary Benefit. See Parsons et al. (2023) for details, the Introduction, and the vignettes on creating functions, summarising results, and detailed example.

predRupdate v0.1.0: Provides functions to evaluate the predictive performance of existing clinical prediction model given a new dataset. See Su et al. (2018), Debray et al. (2014), and Martin et al. (2018) for background and the vignettes Introduction and Technical Background.

SPARRAfairness v0.0.0.1: Provides functions to analyse the behavior and performance of the Scottish Patients At Risk of admission and Re-Admission risk score which estimates yearly risk of emergency hospital admission using electronic health records for most of the Scottish population. Analysis focuses on differential performance over demographically-defined groups. See the vignette.

Sciencekronos v1.0.0: Implements a framework to analyse circadian or otherwise rhythmic data using the familiar R linear modelling syntax, while taking care of the trigonometry under the hood. Look here for examples.

mpmsim v1.0.0: Provides functions to to simulate matrix population models with particular characteristics based on aspects of life history such as mortality trajectories and fertility trajectories, and allows the exploration of sampling error due to small sample size. See the vignettes on robustness, sampling error & propagation, and PCA.

StatisticsBGFD v0.1: Implements the probability density function, cumulative distribution function, quantile function, random numbers, survival function, hazard rate function, and maximum likelihood estimates for the family of Bell-G and Complementary Bell-G distributions. SeeFayomi et al. (2022), Alanzi et al.(2023), and Algarni (2022) for details.

D3mirt v1.0.3: Provides functions for identifying, estimating, and plotting descriptive multidimensional item response theory models, restricted to 3D and dichotomous or polytomous data that fit the two-parameter logistic model or the graded response model. Seethe vignette for an extensive introduction.

funStatTest v1.0.2: Implements two sample comparison procedures based on median-based statistical tests for functional data, described in Smida et al. (2022), Chakraborty and Chaudhuri (2015), Horvath et al. (2013, and Cuevas et al. (2004). See the vignette for examples.

lessSEM v1.4.16: Provides regularized structural equation modeling (regularized SEM) with non-smooth penalty functions (e.g., lasso) building on lavaan. There are nine vignettes including: lessSEM, The Structural Equation Model, and Mixed Penalties.

panelhetero v1.0.0: Provides tools for estimating the degree of heterogeneity across cross-sectional units in the panel data analysis using the methods developed by Okui and Yanagi (2019) and Okui and Yanagi (2020). See the vignette.

tdsa v1.0-1: Provides functions to perform time-dependent sensitivity analysis by calculating time-dependent state and parameter sensitivities for both continuous- and discrete-time deterministic models. See Ng et al. (in review) for background and the vignette to get started.

Utilitiescrew.cluster v0.1.0: Extends the mirai-powered crew package with worker launcher plugins for traditional high-performance computing systems to enable statisticians and data scientists to asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. Look here to get started.

duke v0.0.1: Provides functions to generate visualizations with Duke’s official suite of colors in a color blind friendly way. There is an Overview and four additional vignettes including one on the theme_duke() function.

grateful v0.2.0: Facilitates the citation of R packages used in analysis projects by providing functions to scan projects for packages used and produces documents with citations in the preferred bibliography format. Functions may be used within rarkdownor quarto documents. See README for examples.

hightR v0.3.0: Implements the HIGHT block cipher encryption algorithm developed to provide confidentiality in low power consumption computing environments such Radio-Frequency Identification and Ubiquitous Sensor Network. Look here for more information.

myCRAN v1.0: Provides functions to plot the daily and cumulative number of downloads of R packages, obtaining daily and cumulative counts in one run. See the vignette.

woodendesc v0.1.0: Provides functions to simplify obtaining available packages, their version codes and dependencies from any R repository. Uses extensive caching for repeated queries. See READMEfor examples.

Visualizationfxl v1.6.3: Provides functions to prepare and design single case design figures that are typically prepared in spreadsheet software. See the vignette for theory and examples.

ggragged v0.1.0: Extends ggplot2 facets to panel layouts arranged in a grid with ragged edges with rows and columns of potentially varying lengths. These may be useful in representing nested or partially crossed relationships between faceting variables. See README for examples.

nndiagram v1.0.0: Generates LaTeX code for drawing well-formatted neural network diagrams with TikZ. Users define the number of neurons on each layer, neuron connections to keep or omit, layers considered to be oversized, and neurons to draw with lighter color. See README for instructions.

PlotTools v0.2.0: Provides functions to manipulate irregular polygons and annotate plots with legends for continuous variables and color spectra using the base graphics plotting tools. See README for an example.

To leave a comment for the author, please follow the link and comment on their blog: R Views.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: April 2023: “Top 40” New CRAN Packages

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Have you ever run into the problem of trying to write a vector or matrix that R cannot store? Dr. Kylie Bemis, faculty in the Khoury College of Computer Sciences at Northeastern University, ran into this problem during graduate school and wrote a package called matter that solves it. Development for matter was supported by a grant from the R Consortium.

Dr. Bemis holds a B.S. degree in Statistics and Mathematics, an M.S. degree in Applied Statistics, and a Ph.D. in Statistics from Purdue University. She’s run the Boston Marathon twice and has won numerous academic awards including the John M. Chambers Statistical Software Award from the American Statistical Association.

Dr. Bemis is currently working on providing support for matter and better handling of larger data sets and sparse non-uniform signals. Sparse data in mass spectrometry requires handling data that is zero or does not exist at all. The goal is to better interpolate or resample that data.

In particular, by providing support for non-uniform signal data, matter will be able to provide a back end to mass spectrometry imaging data. But working with large files is applicable in a lot of domains, covering other kinds of hyperspectral data. It is a problem in digital signal processing without many solutions.

What is matter and what does it do? What problem are you solving?

matter is an R package designed for rapid prototyping of new statistical methods when working with larger-than-memory datasets on disk. It provides memory-efficient reading, writing, and manipulation of structured binary data on disk as vectors, matrices, arrays, lists, and data frames.

Data sets might be larger than available memory. matter is completely file based and does its interpretation on the fly. matter aims to offer strict control over memory and maximum flexibility with file-based data structures, so it can be easily adapted to domain-specific file formats, including user-customized file formats.

matter is done with most of its major milestones and is currently on version 2.2. To download and get started now, see: http://bioconductor.org/packages/matter/. The matter 2 User Guide is here: http://bioconductor.org/packages/release/bioc/vignettes/matter/inst/doc/matter-2-guide.html

What type of formats are you extending matter to?

Technically, we’re not extending to formats but rather improving support to existing formats. Improving sparse matrix support. We do have sparse matrices in matter, but it’s not easy to work with them as dense matrices. The main idea of matter is to work with LTR matrices without loading them into memory. We have a little of that with sparse, but it’s written in R so it’s not the fastest in matter. The dense matrix is written in C and C++ so it’s efficient. It also means that we can use the alt-rep framework that R has introduced. You can have something that looks like an ordinary R matrix or array, and in the background, it’s supported by a matter matrix or array.

A few packages make use of this, and it’s something that we are working on and improving for the dense matrices. We can’t do that yet with the sparse matrices because the way alt-rep works is through the C layer of R. We have to implement alt-rep through that level and since the sparse matrix representation is in R, we can’t currently have a sparse matrix alt-rep. That’s another reason that we want to have a sparse matrix in C and C++. Not only will it become faster, but as alt-rep becomes more mature and more efficient, we can use alt-rep with sparse matrices. The main thing that comes out of that is to hopefully use matter matrices in places that you wouldn’t normally get to use them because with an alt-rep object, the function doesn’t have to be aware that this isn’t a regular R matrix or array.

Beyond that, I want to improve R data frames and string support. Right now, we have a prototype data frame in the package, but that’s something that people will be interested in down the line. So, looking at the best way to make an out-of-memory data frame more mature and fully featured now is another thing that we are working on now. Lastly, string support, we have some built-in support for string. It’s not the most intuitive to use, and I’m not entirely sure how useful that will be for some people. But, based on the way that matter is built to be flexible, I realized that is something that we could do. I wanted to sort of explore what we could do in terms of storing string and character vectors in terms of reading from files. One of the bigger challenges going forward is that it’s not going to be as efficient as the others based on its nature. If you have a character vector, each string might be of a different length. And so, with a lot of different types of formats, we are assuming we have an idea of what the file structure looks like. Where, with strings that won’t be the case. So we have to start by parsing the document and figure out what we are breaking on. Whether it’s new lines or other methods.

How exactly does matter allow the use of such large files?

The way that matter works with these large files is kind of simple. Any programming language (R, C, C++, Java) will have a function somewhere that allows you to read some small chunk of a file, whether text or binary. What matter does to allow us to work with these files is that it calls a C++ function that reads a small chunk of a file. Where the magic happens is that we assume that we have a very large matrix or array and the column in that matrix or section in an array might come from different parts of a file or different parts of multiple files. That’s kinda where matter comes in. matter stores a blueprint or a dictionary of where the different columns are stored in that file, maps them to that location in the matrix or array, and depending on what part of the matrix we are accessing, matter figures out what part of the file or different files to read and figures out the most efficient way to read those so it does not do too many reads (since that’s the slowest part of the process).

matter goes and figures this out, reads it into memory, rearranges it if needed so it’s in the desired shape, and returns it to you as an ordinary in-memory matrix or array.

A lot of packages like these use this kind of thing. They use memory mapping, which maps directly onto some big file. Operating systems are really good and efficient at doing that sort of thing. The problem we have is that a lot of time our data comes from multiple files and different places in the file. So, we needed something more flexible. The main thing I wanted to do was figure out how to map between where the different columns were based on where they were in memory and how I wanted to interact with them as a programmer. That’s the main thing that matter does.

matter is hosted on Bioconductor, but other sectors (such as sentiment analysis) also use large datasets. How can matter be used in these applications?

Right now it is hosted on Bioconductor because we come from a Bioconductor background. My work comes from a mass spectrometry background with imaging and proteomics. So, Bioconductor was a natural home for matter and our other packages were hosted there. As a system, it made sense to host it there. There is no reason it can’t be used by other domains.

With Bioconductor, there is no reason that it’s restricted to just Bioconductor packages. Most of the packages are based on bioinformatics itself, but they might have applications outside of bioinformatics. And matter is one of those. So, if you are working in a different area, downloading and installing isn’t that hard and not really any more difficult than downloading and installing a package from CRAN.

So, right now we think Bioconductor is the best place to host matter, and we think people should be able to use it in most domains. The use of sparse data is more useful for mass spectrometry and proteomics, so that is why we are working on these, but we hope that it is usable across other domains.

Why did you realize you had to expand?

The main reason was that we needed more sparse matrix formats that we had as a prototype in R, but we needed it to be as fast. To do this we needed to implement it in C and C++ and make sure it’s more compatible with alt-rep and some of the other Bioconductor big data packages. A lot of that came from mass spectrometry imaging. We work with a lot of imzML formats. This is specific to mass spectrometry imaging. It has 2 files for every data set. One is an XML file which is a text file that describes the experiment and the mass spectrometry in the binary file (metadata). The second file is a binary file that has all the mass spectrometry, intensity arrays, and mass-to-charge ratio from the dataset. There are two subformats, one is sparse and one is dense. A lot of the more recent experiments and recent high mass spectrometry and spatial resolution require storing more in a sparse format. We are seeing more datasets coming in using this sparse format. I have another package called Cardinal that has matter as a backend. We didn’t want it to be slower on these large sparse data sets. So, to deal with that we needed to improve the sparse data sets (processed imzML). That was the main driver, making sure things were fast for the new data sets. Also, there is a lot of potential for sparse data and sparse matrices. That was important for me.

How did you get involved?

My background is in bioinformatics and my Ph.D. is in statistics. I started working on a collaboration with Graham Cook’s lab at Purdue. They worked on a project called DESI for mass spectrometry imaging. The idea is that with mass spectrometry, we can collect from a sample and the mass spectrometry tells us the different abundances in the sample. And with imaging mass spectrometry, we collect hundreds of thousands of mass spectrometry from across the surface of the sample. If we choose a particular chemical, we can reconstruct an image of where that chemical comes from and where it comes from, and where we are seeing it at different abundances across the sample. My Ph.D. was working on developing different statistical methods for analyzing this type of mass spectrometry imaging data, which is really interesting data because we have mass spectrometry which is very high dimensional. Then we also have this spatial component. So we have the x and y coordinates along with the image. So it’s a very interesting, complicated problem.

During my Ph.D. I developed some statistical methods for working with this data. I also developed the Cardinal package for working with and analyzing this kind of data. One of the main difficulties with importing this data and working with it in R in the first place. That wasn’t something that existed in the first place, so that was implemented in Cardinal. Then, as these experiments got bigger, the data files got larger, and I realized I couldn’t just pull the whole thing into memory and I needed a way to work with these larger-than-memory files, especially because a lot of our users, labs in life sciences, chemists, and others, don’t necessarily have the access or know how to work on the cloud or in clusters. So a lot of work is done on PCs.

At that point I developed the project matter, primarily to be a back-end for Cardinal, but also as something that can be used by anyone who uses larger-than-memory files without converting to other formats. One of the main ideas of matter was that we didn’t want you to convert to another format if it’s easier to work with the original format. That’s what we did with imzML. A lot of these instruments are collecting data in some proprietary type and they have to convert to imzML. We didn’t want to make them convert to another format again to use matter.

What was your experience working with the R Consortium? Would you recommend applying for a grant to others?

The R Consortium was very understanding of delays with COVID and private issues. I’m grateful for that. The grant process provides a structure for building a complete application. That is useful, and I would recommend it to others.

I also presented at an User! conf, early on, and Bioconductor, and I am considering doing more. I think connecting with the community and presenting your ideas and getting feedback is a key part of the process.

What do you do for your day job?

I am teaching faculty at Khoury College of Computer Science at Northeastern University in Boston. My teaching duties include teaching R masters for the data science program. The classes I teach are introductory data science for our master’s students, Introduction to Coding for Master’s students, and the capstone project-based course where the students develop a project, work on it as a group, and hopefully include it in their portfolio for industry and interviewing.

What non-project hobbies do you have?

I am a writer and a runner. I write science fiction and fantasy. I haven’t been able to write too much during COVID. I have published two short stories and am working on revising a novel. One of my short stories was nominated for an Otherwise award, which is an award for LGBTQ+ authors. That is something that I am proud of. I also have started running again, and I’m trying to get faster.

About ISC Funded Projects

A major goal of the R Consortium is to strengthen and improve the infrastructure supporting the R Ecosystem. We seek to accomplish this by funding projects that will improve both technical infrastructure and social infrastructure.

Learn moreThe post R Consortium-Funded R Package “matter” Helps Solve Larger-Than-Memory Issues appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Consortium-Funded R Package “matter” Helps Solve Larger-Than-Memory Issues

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.It doesn’t take a data scientist to use R – that’s the point we’ll prove in today’s article. If you’re an experienced Excel user, you’ve likely run into some hard limitations, or find the tool confusing and sluggish when working with large datasets. That’s not the case with programming languages, as the recommended next step from Excel is to use R programming to improve business workflows.

Today we’ll show you just what we mean by that. No advanced R knowledge is mandatory, but it would help if you have R and RStudio installed, and if you’re familiar with basic programming and statistics.

We’ll start lightly by downloading and exploring a ~ 1.5 GB dataset, manipulating it with R, doing some advanced data cleaning, training a machine learning model, generating a markdown report, and even training a machine learning model. There’s even a bonus section at the end in which we’ll use ChatGPT to generate some R code. It’s a pretty packed article, so let’s cut the introduction section short.

Already have some experience in R programming? Here’s a couple of best practices for durable R code.

Table of contents:

Improve Business Workflows by Efficiently Manipulating Large Datasets
Advanced Data Cleaning and Preprocessing Techniques for improving Business Workflows with R
R Programming for Machine Learning – Build Predictive Models
Generate High-Quality R Markdown Reports with Quarto
Improve Data Communication Workflow with R Shiny
Bonus: Can ChatGPT Help You Use R Programming to Improve Business Workflows?
Summary

Improve Business Workflows by Efficiently Manipulating Large DatasetsExcel has a hard limit when rendering a workbook – it’s 10 MB by default, and 2 GB maximum. In short, it’s not a platform you should use when working with big datasets. Speaking of, we’ll use the Used Cars Dataset from Kaggle throughout the article.

It contains every used vehicle entry within the United States on Craigslist. It’s a lot of cars, to put it mildly, as the dataset alone is close to 1.5 GB in size. Excel takes some time to read it even on modern hardware (M1 Pro MacBook Pro), but opens the file successfully:

Image 1 – Vehicles dataset opened in Excel

Now, working on this dataset in Excel is a whole different animal. You’re far better off with R. Here’s the code snippet you can use to read a CSV file and display the first six rows:

vehicles <- read.csv("path/to/the/dataset.csv")head(vehicles) Here’s the output you should see:

Image 2 – Vehicles dataset loaded in R

It’s full of things we need and doesn’t need, so let’s start by trimming the selection down a bit.

Selecting ColumnsWe don’t need the information on URLs, listing IDs, VIN, images, detailed descriptions, and locations for this article. We’ll only keep a handful of attributes, which will make the workflow more easier and manageable.

R’s dplyr package comes to a rescue here. You can use it to perform all sorts of data manipulations and summarizations, as you’ll see throughout the article. An additional benefit of R is the wide range of open-source packages and package ecosystems that support R development. If you’re looking for something specific, chances are there’s a package for that, so there’s no need to reinvent the wheel!

Appsilon even has its own ecosystem of packages, ‘Rhinoverse‘, to build Shiny dashboards and handle data.

The code snippet below imports the package and subsets the dataset, so only the columns of interest are kept:

```

install and load the packageinstall.packages("dplyr")library(dplyr)#keep only the columns desiredkeep_cols <- c( "price", "year", "manufacturer", "condition", "cylinders", "fuel", "odometer", "transmission", "drive", "size", "type", "paint_color", "state")vehicles <- vehicles %>% select(all_of(keep_cols))head(vehicles)

``` The dataset is now much easier to grasp:

Image 3 – Excluding unnecessary attributes

It’s full of missing values, but we’ll address that a bit later. Let’s dive deeper into the exploration first.

New to R dplyr? We have a easy-to-follow guide for complete beginners.

Exploring Possible Attribute ValuesOftentimes when working with datasets, you want to know if an attribute is discrete (either A or B), or continuous (an infinite range of possible values). The n_distinct() function returns the number of unique elements of an attribute, which sounds like the perfect solution.

We don’t want to apply it manually for each attribute, so opting for the R’s sapply() function is a good way to automate this step:

sapply(vehicles, function(x) n\_distinct(x)) Image 4 – Number of unique elements per attribute

The dataset spans many years (115) and consists of mostly discrete values. Price and Odometer are perfect examples of a continuous variable since there’s an infinite range of possible options.

Let’s use this knowledge to calculate some summary statistics next.

Calculating Summary StatisticsSummary statistics provide an answer to a single question, such as “How does the price vary by the car paint color” or “What’s the average car price per year”. They are used to summarize a huge dataset into a small tibble (table) anyone can understand and interpret. You’ll see a couple of examples in this section, so let’s start with the first one.

In this example, we want to find out the price statistics for each distinct car color. We’ll show the minimum, maximum, mean, and median prices, which will give us a clearer picture than one metric alone:

vehicles %>% group\_by(paint\_color) %>% summarise( min\_price = min(price), mean\_price = mean(price), median\_price = median(price), max\_price = max(price) ) Here are the results:

Image 5 – Car price analysis by car color

The median is by far the most representable metric here. It shows the 50th percentile, or the middle price when the price list is sorted in ascending order. It looks like white and black cars take the lead, with orange taking a strong third position. On the other hand, green and purple cars have the lowest median price. There are some missing values in this attribute, indicated by the "" value. We’ll address them later.

Next, let’s analyze the median price by the manufactured year; we will arrange the data by scending median price. We’ll also include the number of cars for each year, so we can know how representative the numbers are:

vehicles %>% group\_by(year) %>% summarise( median\_price = median(price), .groups = "drop", n\_cars = n() ) %>% arrange(desc(median\_price)) Image 6 – Car median price by year

Cars from 1902 are ridiculously expensive – but there’s only one car on the listing. On a dataset this large, you typically want a sample of at least a few thousand to make representative conclusions.

Finally, let’s analyze the prices by the odometer state. The odometer variable is continuous, so we’ll have to bin it into a couple of groups first, using the bin_data function from the mltools package. Here, we’ll also exclude missing values:

install.packages("mltools")library(mltools)vehicles %>% mutate(isna = is.na(odometer) | odometer == "") %>%#You can also try 'filter(!isna)' if you prefer filter(isna == FALSE) %>% mutate(odometer\_group = bin\_data(odometer, bins = 10, binType = "quantile")) %>% select(odometer\_group, price) %>% group\_by(odometer\_group) %>% summarise( median\_price = median(price), n\_cars = n() ) Image 7 – Car median price by the odometer state group

There’s more or less the same number of cars in each group, which is excellent. You can conclude that the median price goes down as the odometer goes up. It’s logical, after all.

Basic Data VisualizationEffective data visualization is the best approach to communicating your findings with non-tech-savvy folk. It’s also a top recommendation for improving business workflows with R programming.

The first visualization you’ll make displays a bar chart of the number of cars by drive type. There are some missing values, so we’ll fill them with “unknown” for the time being. That’s all done in the first portion of the code snippet. The second part is all about producing a bar chart with ggplot2 R package. The visualization also includes the text of Y values inside the bars, which is a recommended practice to eliminate guesswork from your charts, especially when values are this high:

library(ggplot2)df\_vehicles\_drive <- vehicles %>% group\_by(drive) %>% summarise(count = n()) %>% mutate(drive = ifelse(is.na(drive) | drive == "", "unknown", drive))ggplot(df\_vehicles\_drive, aes(x = drive, y = count)) + geom\_col(fill = "#0099f9") + geom\_text(aes(label = count), vjust = 2, size = 5, color = "#ffffff") + labs(title = "Number of cars by drive type") + theme(plot.title = element\_text(hjust = 0.5)) Image 8 – Number of cars by drive type bar chart

The minority of cars in this dataset have rear-wheel-drive, which is understandable. Most manufacturers opt for front-wheel-drive or 4WD. There’s a significant amount of unknown values, so keep that in mind when drawing conclusions.

Horizontal, vertical, stacked, and grouped bar charts – When and how to use each one.

Let’s now make a more interesting visualization. We’ll keep only the records where the manufacturer is Ford and where the manufacture year is 2000 or later. The visualization will display two lines – the blue line shows the median sale price by the year, and the red line represents the total number of cars listed for the given year.

There’s some manual work if you want to include the labels on the blue line, but nothing you can’t figure out from the code snippet below:

df\_vehicles\_ford <- vehicles %>% filter(manufacturer == "ford", year >= 2000) %>% select(year, price) %>% group\_by(year) %>% summarise( median\_price = median(price), count = n() )ggplot(df\_vehicles\_ford, aes(x = year, y = median\_price)) + geom\_line(color = "#0099f9", linewidth = 2) + geom\_line(aes(y = count), color = "#880808", linewidth = 1, alpha = 0.5) + geom\_point(color = "#0099f9", size = 5) + geom\_text(check\_overlap = TRUE) + geom\_label( aes(label = median\_price), nudge\_x = 0.25, nudge\_y = 0.25, ) + labs(title = "Median Ford car price since 2000") + theme(plot.title = element\_text(hjust = 0.5)) Image 9 – Median car price for Ford since 2000 line chart

There’s been a huge drop in the last year, but that’s due to not many cars listed for sale – understandable for newer models. Cars typically tend to be more expensive the newer they are, which follows real-world logic. It’s also valuable to say there are $7000 differences between 2014 and 2015 cars, which is significant considering they’re only a year apart.

For more practice with line charts, try placing the number of cars on a separate y-axis; to learn how, explore the R Graph Gallery’s tutorial. If you’d like to level up your line chart skills with ggplot2, try our ggplot2 line chart guide.

Finally, let’s plot a histogram of prices. We’ll limit the price range to go from 0 to 100000 USD, and we’ll also exclude missing values.

Wait, what exactly is a Histogram? Read our detailed guide on making stunning histograms with ggplot2.

Histogram will show a more detailed distribution representation, and is a must-know data visualization type when working with continuous data:

df\_vehicles\_hist <- vehicles %>% mutate(isna = is.na(price) | price == "") %>% filter(isna == FALSE) %>% filter(price > 0 & price < 100000) %>% select(price)ggplot(df\_vehicles\_hist, aes(price)) + geom\_histogram(color = "#000000", fill = "#0099F8") + labs(title = "Histogram of car prices (0 > x < 100000)") + theme(plot.title = element\_text(hjust = 0.5)) Image 10 – Histogram of car prices with the price limit

The good news – the distribution is skewed, which means there are cheaper cars available than expensive ones. We’ve deliberately excluded cars priced above $100000 because there’s only a handful of them, and they would skew the distribution even further.

If you’re eager to learn more about data visualization, start with these two articles:

5 Key Data Visualization Principles You Must Know
Pie Charts – Why You Shouldn’t Trust Them

Advanced Data Cleaning and Preprocessing Techniques for improving Business Workflows with RTake a moment to pause and think what’s the biggest problem we’ve seen so far. Thoughts? It’s the problem of missing values. They are everywhere in the dataset, and it’s your job as a data professional to handle them. We’ll first explore where the missing values are most common and then we’ll go over a couple of imputation methods.

This section will also show you the basics of feature engineering, so make sure to keep reading until the end.

Missing Value ExplorationCredits for charts in this and the following section go to jenslaufer.com blog.

Before imputing missing values, we have to know where they are and what’s the most likely reason they’re missing. Doing so involves some manual work in R. We want to produce a nice-looking visual representing the portion of data missing and available for every dataset column that contains missing values.

The first portion of the snippet uses dplyr to extract this information, and the second portion plots the data as a horizontal collection of “progress bars”. Here we’ll introduce tidyr – a powerful tool for data wrangling and.. tidying. Whether you need to separate variables, spread values across columns, or gather scattered data into a tidy format, tidyr provides intuitive functions to simplify the process:

install.packages("tidyr")library(tidyr)missing\_values <- vehicles %>%#In lieu of 'gather' you can also try 'pivot\_longer'#pivot\_longer(#names(vehicles),#values\_transform = as.character,#names\_to = "key", values\_to = "val") gather(key = "key", value = "val") %>% mutate(isna = is.na(val) | val == "") %>% group\_by(key) %>% mutate(total = n()) %>% group\_by(key, total, isna) %>% summarise(num.isna = n()) %>% mutate(pct = num.isna / total * 100)levels <- (missing\_values %>% filter(isna == T) %>% arrange(desc(pct))pull(key)percentage\_plot <- missing\_values %>% ggplot() + geom\_bar( aes( x = reorder(key, desc(pct)), y = pct, fill = isna ), stat = "identity", alpha = 0.8 ) + scale\_x\_discrete(limits = levels) + scale\_fill\_manual( name = "", values = c("steelblue", "tomato3"), labels = c("Present", "Missing") ) + coord\_flip() + labs( title = "Percentage of missing values", x = "Variable", y = "% of missing values" )percentage\_plot Here’s what it looks like:

Image 11 – Percentages of missing values by attribute

Considering the size of the dataset, this is a whole lot of missing values. Some we can just drop, and some we’ll have to impute.

Missing Value ImputationThe way you impute missing values highly depends on domain knowledge. For example, in the telco industry, maybe the value is missing because the customer isn’t using the given service, hence it makes no sense to impute it with zero or an average value. It’s important to know the domain well for real business problems, especially if you want to use R programming for business workflow improvement.

At Appsilon, we aren’t experts in cars, but we’ll give it our best shot. Based on Image 11, here’s what we’re gonna do for each attribute:

condition – Replace missing with “good”
cylinders – Impute with “6 cylinders”
drive – Fill with “unknown”
fuel and year – Remove missing values since there aren’t many of them
odometer – Impute with the median value
paint_color, size, transmission, and type – Fill missing values with “other”

If you prefer code over words, here’s everything you have to do:

```

Replace empty strings with NAvehicles[vehicles == ""] <- NA# Handle categorical variablesvehicles <- vehicles %>% mutate( condition = replace_na(condition, "good"), cylinders = replace_na(cylinders, "6 cylinders"), drive = replace_na(drive, "unknown") ) %>%# For the selected columns, replace the missing value of each column with other mutate(across(c("paint_color", "size", "transmission", "type"), .fns = ~replace_na(.x, "other") ))# Handle continuous variable "odometer" - replacing with medianvehicles|># Remove other missing ("fuel" and "year") mutate( odometer = replace_na(odometer, median(vehicles$odometer, na.rm = TRUE)) )

``` That’s it! Let’s now check if the imputation was successful with a similar code snippet as in the previous subsection:

missing\_values\_2 <- vehicles %>% gather(key = "key", value = "val") %>% mutate(is.missing = is.na(val)) %>% group\_by(key, is.missing) %>% summarise(num.missing = n()) %>% filter(is.missing) %>% select(-is.missing) %>% arrange(desc(num.missing))missing\_values\_2 Image 12 – Missing values after imputation

As you can see, there are no more missing values, which means we can mark our job here as done. Let’s proceed with some basic feature engineering.

More on missing data imputation – Here are top 3 ways to impute missing values in R.

Feature EngineeringNow, feature engineering is closely tied to machine learning. Long story short, you want to create new features that you hope will have more predictive power than the current ones. Once again, this is closely related to domain knowledge. The more you know about some field, the more you know what’s important both for analysis and predictive performance.

We’ll make only a handful of transformations in this section. The first one boils down to subsetting the dataset (only keeping Ford cars) and deriving a new column years_old, calculated as 2023 - year. After creating this new attribute, we’ll remove the manufacturer and year info, but also the listing state since we won’t need it anymore:

vehicles <- vehicles %>% filter(manufacturer == "ford") %>% mutate(years\_old = 2023 - year) %>% select(-c("manufacturer", "year", "state")) The next transformation boils down to converting a string value to a numeric one. The cylinders column currently has string values, e.g., “6 cylinders” and we want to convert it to the number 6.

Some missing values might be introduced in the process, so it’s a good idea to call drop_na() after the transformation:

vehicles <- vehicles |> mutate( cylinders = as.numeric(gsub(" cylinders", "", cylinders)) )vehicles <- vehicles %>% drop\_na() Finally, we’ll divide the odometer number by 1000, just to bring the range of continuous variables closer:

vehicles <- vehicles %>% mutate(odometer = odometer / 1000)head(vehicles) All in all, here’s what the dataset looks like after feature engineering:

Image 13 – Dataset after basic feature engineering

We can now proceed to the next section in which you’ll learn how to improve business workflows with R programming by training a machine learning model.

Data cleaning always takes time – But considerably less if you know how to do it properly.

R Programming for Machine Learning – Build Predictive ModelsThis section will teach you how to go from a clean dataset to a machine learning model. You’ll learn the basics that go into dataset transformation for machine learning, such as creating dummy variables, data scaling, train/test split, and also how to evaluate machine learning models by examining feature importance and performance metrics.

Create Dummy VariablesMachine learning models typically require some sort of special treatment for categorical variables. For example, the drive column has four possible values, abbreviated as “FWD”, “RWD”, “4WD”, and “unknown”. You could convert these strings to numbers – 1, 2, 3, and 4, but what does that mean for the inter-variable relationship? Is 2 twice as better as 1? Well, no, at least not when comparing car drive types.

These are distinct categories, and one isn’t better than the other. For that reason, we have to encode them with dummy variables. These will create as many new columns as there are distinct categories in the variable, and a column will have a value of 1 if the type matches or 0 otherwise.

We’ll further subset the selection of attributes for machine learning, followed by the creation of dummy variables (using fastDummies) for categorical attributes:

install.packages("fastDummies")library(fastDummies)vehicles\_ml <- vehicles %>% select(c("price", "condition", "fuel", "odometer", "transmission", "drive", "years\_old"))vehicles\_ml <- dummy\_cols(vehicles\_ml, select\_columns = c("condition", "fuel", "transmission", "drive")) %>% select(-c("condition", "fuel", "transmission", "drive"))head(vehicles\_ml) The dataset will have much more columns now:

Image 14 – Dataset with dummy variables

Categorical features are now either 0 or 1, but continuous features are not. We’ll deal with them shortly.

Finalizing Data for Predictive ModelingBefore scaling continuous features, let’s make our lives a bit easier by building a model only for cars with a price between $1000 and $100000. This will take extremes outside the equation, and we’ll have a more stable model overall. Sure, you won’t be able to use the model to predict the prices of luxury cars, so keep that in mind:

vehicles\_ml <- vehicles\_ml %>% filter(price >= 1000 & price < 100000)vehicles\_ml <- unique(vehicles\_ml)head(vehicles\_ml) Image 15 – Dataset with prices in a range

The dataset looks identical to Image 14, but that’s only because there were no extreme data points in the first six rows.

Data ScalingNow let’s discuss the scaling of the continuous features. The categorical ones are in the range of [0, 1], so it makes sense to stick to that range. MinMax scaler is a perfect candidate for the job.

The {caret} package is a comprehensive toolkit for building and evaluating predictive models. It simplifies the process of training, tuning, and evaluating models. With {caret}, you can easily switch between different algorithms, handle preprocessing tasks, and perform cross-validation.

MinMax scaling, also known as Min-Max normalization, is a data preprocessing technique used to rescale numerical features within a specific range. It transforms the data so that the minimum value becomes 0 and the maximum value becomes 1, while preserving the relative distances between other data points. The MinMax scaler in {caret} is a function that applies this scaling transformation to your data, ensuring that all features are on a similar scale and preventing any single feature from dominating:

install.packages("caret")library(caret)preproc <- preProcess(vehicles\_ml[, c(2, 3)], method = c("range"))scaled <- predict(preproc, vehicles\_ml[, c(2, 3)])vehicles\_ml$odometer <- scaled$odometervehicles\_ml$years\_old <- scaled$years\_oldhead(vehicles\_ml) Here’s what the dataset looks like now:

All of the features are now in a range of [0, 1]. You might be wondering – What about the price? Well, it’s not a feature, it’s a target variable, meaning we won’t use it as a feature in a machine learning model. More on that shortly.

Train/Test SplitTrain/test split refers to a process of splitting the dataset into two parts. The first one will be used to train a machine learning model, and the second to evaluate it. Doing both on the same dataset is a terrible practice and can lead to misleading results.

A common approach is to use 80% of the dataset for training, and 20% for testing, but the ratio may vary depending on the dataset size. The more data you have available, the less you have to put in the testing set, percentage-wise.

The caTools R package allows you to do this split easily. It’s a good idea to set the random seed to a constant value, especially if you plan to repeat the experiment sometime later:

library(caTools)set.seed(42)sample <- sample.split(vehicles\_ml$price, SplitRatio = 0.8)train <- subset(vehicles\_ml, sample == TRUE)test <- subset(vehicles\_ml, sample == FALSE)print(dim(train))print(dim(test)) Image 17 – Dimensionality of training and testing datasets

Keep in mind – this is the dimensionality of a cleaned dataset, free of missing values, and limited to Ford vehicles manufactured after 2000 and priced between $1000 and $100000. It’s a small sample, sure, but it will be suitable enough for a predictive model.

Training a Machine Learning ModelMachine learning models are a great way to leverage R programming for business workflow improvement. You’ll now build one based on a decision tree algorithm.

It boils down to a single line of code – rpart(<name_of_target_variable> ~ ., data = train, method = "anova"). The dot in the equation means you want to use all features to predict the target variable, and Anova instructs the package to build a regression model (prediction of a continuous variable).

The code snippet below trains the model on the train set and plots the decision tree:

library(rpart)library(rpart.plot)model\_dt <- rpart(price ~ ., data = train, method = "anova")rpart.plot(model\_dt) Image 18 – Decision tree model structure

Think of this as a learned set of if-else statements. You don’t have to supply the conditions, these are figured out automatically for you based on the data provided.

Up next, we’ll evaluate this model.

Do you know what goes into building a decision tree model? Our from-scratch guide will teach you the basics.

Evaluating a Machine Learning ModelThere are a lot of ways to evaluate a machine learning model, but we’ll cover only two in this section – via feature importance and performance metrics.

Feature importances won’t technically evaluate the model, but they’ll give you a relative representation of which features have the most impact on the model’s predictive power. Here’s how to calculate them based on a decision tree model:

```

Note that varImp comes from the previously used {caret}importances <- varImp(model_dt)importances %>% arrange(desc(Overall))

``` Image 19 – Decision tree feature importance

As it turns out, the age of the car and its mileage have the most impact on price. It’s a reasonable assumption, the older the car is and the more it’s traveled, the less it will generally sell for. Some of the features are completely irrelevant for prediction, so you might rethink excluding them altogether.

The best way to evaluate a machine learning model is by examining performance metrics on the test set. We’ll calculate predictions on the test set, and then compare them to the actual values:

predictions\_dt <- predict(model\_dt, test)eval\_df <- data.frame( actual = test$price, predicted\_dt = predictions\_dt)head(eval\_df) Image 20 – Actual vs. predicted values

The values are off, sure, but follow the general trend.

If you want to quantify how off the model is on average, look no further than the Root Mean Squared Error (RMSE) metric. In our case it will display how many thousands of dollars the model differs from the true value on average:

sqrt(mean((eval\_df$actual - eval\_df$predicted\_dt)^2)) Image 21 – Decision tree root mean squared error

Around $9000 – not too bad for a couple of minutes of work. This means our decision tree model is on average wrong by $8928.18 in its prediction. Including some additional features and spending more time on data preparation/feature engineering is likely to decrease RMSE and give you a better, more stable model.

Want to consider Python for machine learning? Read our beginners guide to PyTorch.

Generate High-Quality R Markdown Reports with QuartoReports are everywhere, and making custom ones tailored to your needs is easier said than done. That’s where R programming comes in, and it transforms your business workflow for the better.

R Quarto is a next-gen version of R Markdown and is used to create high-quality articles, reports, presentations, PDFs, books, Word documents, ePubs, and websites. This section will stick to reports exported as PDF documents.

To get started, you’ll first have to install R Quarto for your OS and restart RStudio. Once done, you can proceed to the following section.

Create a Quarto DocumentCreating Quarto documents is as easy as clicking on File – New File – Quarto Document. A modal window like this one will appear:

Image 22 – Creating a new R Quarto document

Make sure to add a title and author info to the document, and change the other settings as you see fit. Once done, click on the “Create” button to create a QMD document.

Quarto R Markdown – Writing a DocumentIf you know Markdown, Quarto will feel right at home. If not, well, the learning curve isn’t steep, and you can easily remember the basics in a day. This Basic syntax guide contains everything you’ll ever need for markdown.

In this section, we’ll copy some of the analyses and visualizations from the previous section. If you don’t feel like typing, paste the following into the QMD file (but keep the top identification section as is):

---title: "Quarto Document"author: "Dario Radečić / Appsilon"format: pdfeditor: visual---{r setup, include=FALSE}knitr::opts_chunk$set(warning = FALSE, message = FALSE) ## Vehicles Dataset Exploration- Download the dataset from [this link](https://www.kaggle.com/datasets/austinreese/craigslist-carstrucks-data)- A 1.45 GB CSV file containing many features on used car data (Craigslist)- This document demonstrates some basic exploration and visualization## Introduction to the DatasetLet's load the used car dataset and keep ony a couple of columns:{r}library(dplyr)vehicles <- read.csv("/Users/dradecic/Downloads/vehicles.csv")keep_cols <- c( "price", "year", "manufacturer", "odometer", "type", "paint_color", "state")vehicles <- vehicles %>% select(all_of(keep_cols))And now let's see what the dataset looks like:{r}sample_n(vehicles, 10)## CalculationsPrice analysis by paint color:{r}vehicles %>% group_by(paint_color) %>% summarise( min_price = min(price), mean_price = mean(price), median_price = median(price), max_price = max(price) )Median car price by odometer:{r}library(mltools)vehicles %>% mutate(isna = is.na(odometer) | odometer == "") %>% filter(isna == FALSE) %>% mutate(odometer_group = bin_data(odometer, bins = 10, binType = "quantile")) %>% select(odometer_group, price) %>% group_by(odometer_group) %>% summarise( median_price = median(price), n_cars = n() )## VisualizationsMedian Ford price by year since 2000:{r}library(ggplot2)df_vehicles_ford <- vehicles %>% filter(manufacturer == "ford", year >= 2000) %>% select(year, price) %>% group_by(year) %>% summarise( median_price = median(price), count = n() )ggplot(df_vehicles_ford, aes(x = year, y = median_price)) + geom_line(color = "#0099f9", linewidth = 2) + geom_line(aes(y = count), color = "#880808", linewidth = 1, alpha = 0.5) + geom_point(color = "#0099f9", size = 5) + geom_text(check_overlap = TRUE) + geom_label( aes(label = median_price), nudge_x = 0.25, nudge_y = 0.25, ) + labs(title = "Median Ford car price since 2000") + theme(plot.title = element_text(hjust = 0.5)) Car price distribution for vehicles priced between 0 and 100000 USD:{r} library(ggplot2) df_vehicles_hist <- vehicles %>% mutate(isna = is.na(price) | price == "") %>% filter(isna == FALSE) %>% filter(price > 0 & price < 100000) %>% select(price) ggplot(df_vehicles_hist, aes(price)) + geom_histogram(color = "#000000", fill = "#0099F8") + labs(title = "Histogram of car prices (0 > x < 100000)") + theme(plot.title = element_text(hjust = 0.5)) You’ll need to change the dataset path, of course, but other than that you’re good to go. The code snippet toward the top of the document probably requires some additional explanation, though:

{r setup, include=FALSE}knitr::opts_chunk$set(warning = FALSE, message = FALSE) Put simply, it’s here to prevent Quarto from rendering R warning messages and R messages in general. Nobody wants these in their reports after all, so make sure to include it in every Quarto document.

The finalized QMD file should look like this:

Image 23 – Final QMD file

With that out of the way, let’s discuss rendering.

Rendering an R Quarto Markdown DocumentOur goal is to save the rendered version of a Quarto Markdown file as a PDF document. To do so, just hit the “Render” button that was highlighted in Image 23. You’ll see some console output, and the PDF document will be saved to the source file location shortly after.

Here’s what it looks like on our end:

Image 24 – Rendered R Quarto PDF document

And that’s it regarding R Quarto for improving business workflows with custom reports. Finally, let’s discuss Shiny before wrapping up this article.

Want to learn more about Quarto? Here’s an in-depth article to follow.

Improve Data Communication Workflow with R ShinyWe’re big fans of R Shiny at Appsilon, and we genuinely think it’s a go-to way to improve business workflows with R programming. After all, if a photo (chart) is worth 1000 words, then a custom dashboard is worth 1000 photos. That’s where Shiny chimes in, as a de-facto standard approach for building apps and dashboards straight from R.

Our task here is simple – we need an app that can be shared across teams/departments and shows the following car data:

Summary statistics by manufacturer – Min, mean, median, and max price
Sample data – A sample of 10 rows from the dataset rendered as a table
Price distribution chart – A histogram we made earlier
Price distribution by odometer group chart – A bar chart showing median prices by odometer group (summary statistic from before)
Price movement over time – Line chart showing price trends

Let’s start with the UI.

Building the R Shiny App UIOkay, so what do we need from the controls? The user will be able to select a manufacturer from a list of two (Ford and Chevrolet), and on a manufacturer change, the entire dashboard data will be refreshed. You’re free to add additional input controls, such as year and price limitation, so consider this a homework assignment.

The df object will be shared through the dashboard. It has the dataset loaded in only the manufacturer, year, price, and odometer columns kept.

The UI consists of a Sidebar Layout, with the UI controls on the sidebar on the left, and the main content (tables and charts) on the right:

library(shiny)library(dplyr)library(tidyr)library(mltools)library(ggplot2)library(kableExtra)options(scipen = 999)df <- read.csv("/Users/dradecic/Downloads/vehicles.csv") %>% select(c("manufacturer", "year", "price", "odometer"))ui <- fluidPage( sidebarLayout( sidebarPanel( tags$h3("Used Cars Dataset Exploration"), tags$a("Dataset source", href = "https://www.kaggle.com/datasets/austinreese/craigslist-carstrucks-data"), tags$hr(), selectInput(inputId = "manufacturer\_select", label = "Manufacturer", choices = c("ford", "chevrolet"), selected = "ford") ), mainPanel( tags$h4("Data from the year 2000, price between 1000 and 100000 USD"), tags$hr(), # Summary statistics tags$div( uiOutput(outputId = "summary\_stats") ), # Table data sample tags$div( tags$h4("Used car price data overview"), tableOutput(outputId = "sample\_data\_table"), class = "div-card" ), # Price distribution tags$div( tags$h4("Histogram of car prices"), plotOutput(outputId = "plot\_prices\_hist"), class = "div-card" ), # Price by odometer group tags$div( tags$h4("Median price by odometer group"), plotOutput(outputId = "plot\_price\_odometer"), class = "div-card" ), # Price over time tags$div( tags$h4("Median price by year"), plotOutput(outputId = "plot\_price\_over\_time"), class = "div-card" ) ) )) Pretty much all elements are wrapped in div elements and assigned a CSS class property because we’ll add some styles later on. Feel free to remove them if you don’t want to tweak the visuals.

Either way here’s what the dashboard looks like now, without the server logic:

Image 25 – Shiny app UI

The basis is here, we just need to add the logic. Let’s do that next.

Don’t like how the table data looks? Here are some alternative packages you can use in R and R Shiny.

Building the R Shiny Server LogicThe server logic in Shiny will take care of rendering the correct data and making sure the correct data is shown when the user makes changes to the input fields. There are quite a bit of things we need to take care of, the most important one being the data itself.

The server data object is a reactive value filtered with the currently selected manufacturer, plus some hardcoded filter you’re familiar with from before.

Programming logic for each UI element is written in sections below the data declaration. We’re sure you’ll recognize most of it from the previous article sections:

server <- function(input, output) { data <- reactive({ df %>% filter(manufacturer == input$manufacturer\_select & year >= 2000 & price >= 1000 & price <= 100000) }) # Summary statistics output$summary\_stats <- renderUI({ price\_stats <- data() %>% summarise( min\_price = round(min(price), 2), mean\_price = round(mean(price), 2), median\_price = round(median(price), 2), max\_price = round(max(price), 2) ) tags$div( tags$div(p("Min price:"), h3(paste("USD", price\_stats$min\_price)), class = "summary-stats-card"), tags$div(p("Mean price:"), h3(paste("USD", price\_stats$mean\_price)), class = "summary-stats-card"), tags$div(p("Median price:"), h3(paste("USD", price\_stats$median\_price)), class = "summary-stats-card"), tags$div(p("Max price:"), h3(paste("USD", price\_stats$max\_price)), class = "summary-stats-card"), class = "summary-stats-container" ) }) # Table data sample output$sample\_data\_table <- reactive({ sample\_n(data(), 10) %>% kbl() %>% kable\_material() %>% kable\_styling(bootstrap\_options = c("striped", "hover")) }) # Price distribution output$plot\_prices\_hist <- renderPlot({ ggplot(data(), aes(price)) + geom\_histogram(color = "#000000", fill = "#0099F8") + theme\_classic() }) # Price by odometer group output$plot\_price\_odometer <- renderPlot({ odometer\_group\_data <- data() %>% mutate(odometer\_group = bin\_data(odometer, bins = 10, binType = "quantile")) %>% select(odometer\_group, price) %>% group\_by(odometer\_group) %>% summarise( median\_price = median(price) ) ggplot(odometer\_group\_data, aes(x = odometer\_group, y = median\_price)) + geom\_col(fill = "#0099f9") + geom\_text(aes(label = median\_price), vjust = 2, size = 5, color = "#ffffff") + labs( x = "Odometer group", y = "Median price (USD)" ) + theme\_classic() }) # Price over time output$plot\_price\_over\_time <- renderPlot({ price\_data <- data() %>% select(year, price) %>% group\_by(year) %>% summarise( median\_price = median(price), count = n() ) ggplot(price\_data, aes(x = year, y = median\_price)) + geom\_line(color = "#0099f9", linewidth = 2) + geom\_line(aes(y = count), color = "#880808", linewidth = 1, alpha = 0.5) + geom\_point(color = "#0099f9", size = 5) + geom\_text(check\_overlap = TRUE) + geom\_label( aes(label = median\_price), nudge\_x = 0.25, nudge\_y = 0.25, ) + labs( x = "Year", y = "Median price (USD)" ) + theme(plot.title = element\_text(hjust = 0.5)) + theme\_classic() })}shinyApp(ui = ui, server = server) Let’s launch the app and see what it looks like:

Image 26 – R Shiny application

Overall, it’s nothing fancy (yet) but perfectly demonstrates what R Shiny is capable of. Add a couple more filters (year, odometer, drive type, color) and you’re golden. It all boils down to adding a couple of UI elements and including their selected values in data reactive value filter conditions.

Did you know Shiny is also available for Python? Here are some key differences when compared to R.

Next, let’s work on CSS styles.

Styling R Shiny ApplicationsR Shiny allows you to easily add styles, either directly to the R file or to an external CSS file. We’ll opt for the latter option, and recommend you to go with it as well.

Create a www directory right where your Shiny app source R script is, and inside create a style.css file. Next, add the following code in UI in between fluidPage() and sidebarLayout():

tags$head( tags$link(rel = "stylesheet", type = "text/css", href = "style.css")),#If you have trouble making the styling work with 'tags$link' try 'includeCSS("www/style.css") This makes R Shiny aware of the CSS file, so the only thing left to do is to add some styles to style.css. This should do:

body { background-color: #f2f2f2;}.summary-stats-container { display: flex; justify-content: space-between; gap: 1rem;}.summary-stats-card { background-color: #ffffff; padding: 1rem 2rem; border-radius: 5px; width: 100%;}.div-card { width: 100%; margin-top: 2rem; background-color: #ffffff; padding: 1rem 2rem; border-radius: 5px;} Run your app again and you’ll see the following:

Image 27 – Styled R Shiny application

Once again, it’s nothing fancy, but perfectly demonstrates how you can tweak your Shiny apps with only 20-something lines of CSS.

That wraps up the R programming business workflow portion of the article. Up next, we’ll explore if artificial intelligence can assist you if you get stuck writing R code.

Looking for Shiny app inspiration? Take a look at these 5 agriculture examples.

Bonus: Can ChatGPT Help You Use R Programming to Improve Business Workflows?ChatGPT or Chat Generative Pre-trained Transformer is a neural-network-powered chatbot developed by OpenAI. It created a lot of hype since its launch in November 2022 and for a good reason – it can write computer programs better than some developers. It’s not limited to coding, of course, it can handle music composing, writing essays, and even passing a Bar exam.

Does this mean it can help you write R code? Well, let’s see. We’ve asked ChatGPT to write an R script for visualizing historical car prices with ggplot2. Keep in mind that ChatGPT isn’t aware of our dataset nor of the attributes it contains.

Here’s what we got back:

Image 28 – ChatGPT output

This is interesting. In addition to the code, ChatGPT also provides a set of instructions and assumptions. For example, here it assumed we have a CSV file called historical_car_prices.csv and that it contains make, year, and price attributes.

It’s not 100% identical to our dataset, so we’ll have to make minor changes. The chart code generated by ChatGPT will make a scatter plot grouped by every car brand. It’s too messy for our dataset, so we’ll filter some out. The same thing with the year attribute.

The following image highlights all the changes we’ve made to the R script:

Image 29 – Changes to the R code

There’s also no need to save the chart, as we can just display it:

Image 30 – Car price by year data visualization

Truth be told, the code changes were minimal, but assume you’re comfortable working with the car prices dataset. ChatGPT doesn’t generate the data for you, but it shows you how to load the data and make the chart – impressive for a single sentence input. ChatGPT also lists all the assumptions so you know what you’ll have to change to match your system (dataset path and column names).

Is it a fool-proof solution for those with 0 experience with R? No, but it’s far more capable and faster than a Google Search.

SummaryThis was likely the longest article on the Appsilon blog so far. We hope you’ve managed to get something useful out of it, and that we haven’t lost you in the process. The overall goal was to see how you can use R programming to improve business workflows and prove it doesn’t take a data scientist to take full advantage of R.

We think that business workflow optimization is more of a cycle than a one-time project. Today you switch from Excel to R, tomorrow you work on a large dataset in order to optimize resource utilization or customer mailing strategies, and the day after you’re writing an application to present the solution to the board. With more data comes more challenges, and more need for adequate and scalable analysis. Excel oftentimes doesn’t stand a chance for big players.

What are your thoughts on our business workflow improvement ideas with R programming? Have you successfully implemented R or R Shiny in your organization? Let us know in the comment section below, or reach out to us on Twitter – @appsilon. We’d love to hear from you.

Continue learning R – Here are 7 essential R packages for programmers.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 5 Ways R Programming and R Shiny Can Improve Your Business Workflows

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionYesterday I had the need to see data that had a grouping column in it. I wanted to use the tidy_four_autoplot() function on it from the {TidyDensity} library on it. This post will explain how I did it. The data in my session was called df_tbl. In this blog post, we will explore the steps involved in using the tidy_empirical() and tidy_four_autoplot() functions from the R library TidyDensity. These functions are incredibly useful when working with data, as they allow us to analyze and visualize empirical distributions efficiently. We will walk through a code snippet that demonstrates how to use these functions within a map() function, enabling us to analyze multiple subsets of data simultaneously.

Prerequisites

To follow along with this tutorial, it is assumed that you have a basic understanding of the R programming language, as well as familiarity with the dplyr, purrr, and TidyDensity libraries. Make sure you have these packages installed and loaded before proceeding.

Here is the code that I used, the explanation will follow:

library(dplyr) # to use group\_split()library(purrr) # to use map()library(TidyDensity) # to use tidy\_empirical() and tidy\_four\_plot()df\_tbl |> group\_split(SP\_NAME) |> map(\(run\_time) pull(run\_time) |> tidy\_empirical() |> tidy\_four\_autoplot() ) Code ExplanationLet’s break down the code step by step:

Importing Required Libraries:

To access the necessary functions, we need to load the required libraries. In this case, we use library(dplyr) to utilize the group_split() function from the dplyr package, library(purrr) to use the map() function from the purrr package, and library(TidyDensity) to access the tidy_empirical() and tidy_four_autoplot() functions from the TidyDensity package.

Grouping and Splitting the Data:

The first line of the code snippet takes a dataframe named df_tbl and uses the group_split() function from the dplyr library to split it into multiple subsets based on a variable called SP_NAME. This creates a list of dataframes, each representing a unique group based on SP_NAME.

Applying Functions to Each Subset using map():

The second line of code utilizes the map() function from the purrr library to iterate over each subset of data created in the previous step. The map() function takes two arguments: the object to iterate over (in this case, the list of dataframes) and a function to apply to each element.

Anonymous Function Inside map():

Within the map() function, an anonymous function (denoted by (run_time)) is defined. This function takes a single argument named run_time, representing each individual subset of data. The purpose of this anonymous function is to perform the necessary computations and visualizations on each subset of data.

Data Manipulation and Visualization:

Inside the anonymous function, the pull(run_time) function is used to extract the run_time column from each subset of data. This column is then passed to the tidy_empirical() function from the TidyDensity library, which calculates the empirical distribution of the data. The result is a tidy dataframe that contains information about the empirical distribution.

Tidy Four Autoplot:

The output of tidy_empirical() is then piped (|>) into the tidy_four_autoplot() function from the TidyDensity library. This function generates a visualization called a “Tidy Four Plot,” which consists of four individual plots: empirical density, empirical cumulative density, QQ plot, and histogram.

Final Output:

The result of the tidy_four_autoplot() function is the final output of the anonymous function within map(). This output represents the visualization of the empirical distribution for each subset of data.

Happy Coding!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Data with TidyDensity: A Guide to Using tidy_empirical() and tidy_four_autoplot() in R

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium recently talked to Sébastien Rochette, organizer of the Meetup R Nantes, about his involvement in the R community. The group hosts a mix of physical and online events targeting the full range of R users, from beginners to experts. Sebastien works for ThinkR and has recently developed a package named {fusen}. {fusen} allows developers to use a single Notebook file for writing the package documentation, code, and tests in the same place.

Sébastien Rochette, Head of Production at ThinkR and Organizer of Meetup R Nantes

Please share your background and your involvement in the RUGS group or in the R Community.

I began using R for research and spatial data modelization 15 years ago. I started in high school and continued while I was a fisheries science researcher. Now I continue to use R every day as the Head of Production at ThinkR, where we work as consultants and teachers of R. Our work entirely revolves around R, from installing infrastructures to teaching to and certifying public and private users, or building packages and shiny applications for them. We are also certified full-service partners of Posit (RStudio).

I am also deeply invested in the open source and R community. At ThinkR, we develop many open-source R packages for our internal use, which we are happy to share with the community. We believe that building a business over an open-source project like R requires giving back to the community. I am also the organizer of Meetup R Nantes, which alternates between physical and online events. Physical events are great for networking, while online events ensure inclusivity for those unable to attend in person.

At all of our meetups, we have two presentations. One is aimed at beginners and the other is aimed at more experienced users or presenting a personal experience. This is to make sure that members of all experience levels feel included and can learn from the events.

Please share about a project you are currently working on or have worked on in the past using the R language. Goal/reason, result, anything interesting, especially related to the industry you work in?

I am currently working on a project called {fusen}, an open-source package designed to make package building easier. While developing packages, developers usually prioritize writing code and overlook documentation. This can cause problems when reusing or sharing code with colleagues or the community.

With {fusen}, we start by opening a template notebook file (RMarkdown or Quarto). This encourages writing about the code’s purpose in plain text as it is readable. Then, the user writes the code and an example within the R Markdown file. The script example can also be used to write a unit test. The RMarkdown template proposed by {fusen} encourages people to document everything they have in mind regarding their project. This reduces the risk of forgetting their goal the next time they code, but also sets the perfect basis for sharing their work: documentation and examples are there already. The {fusen} notebook template has different script parts for code, function, and test. {fusen} then inflates the file into a full package, with the documentation and examples in the vignette, the functions in the R directory, and the unit tests in the tests directory. This approach eliminates worries about the package structure and allows developers to write everything in a single RMarkdown file.

Figure 1. The classical code of a project is written in one or multiple notebook files, called flat files, which {fusen} inflates as a full R package. A complementary file called “dev_history.Rmd” contains steps for package level documentation, and a list of tools for collaborating and sharing the project. See {fusen} documentation for more information.

We had two goals in mind when developing {fusen}. The first is that developers no longer need to open multiple files to write their packages and can focus on their package’s purpose. The second is that it encourages developers to document and test their work while making it easier for new developers to write R packages.

What resources/techniques do/did you use? (Posit (RStudio), Github, Tidyverse, etc.)

As RStudio-certified resellers, we prefer working with Posit (previously RStudio). It is much easier for beginners to use, which makes it an excellent tool for teaching R. The {fusen} project is available on GitHub since it is an open-source package, making it easily accessible to the community.

Within the {fusen} package, I use base R code, as I have been writing R code long ago before Tidyverse arrived. However, from the Tidyverse, I use the ‘tibble’ format to benefit from its consistency in data frame structures. {fusen} also relies on a package called {parsermd}, which reads and parses R Markdown files as ‘tibbles’.

Our package {attachment} also has an important role in {fusen} in reducing the difficulty of writing R packages. It helps declare all package dependencies. {attachment} reads the ‘importForm’ declared and all different ways of calling a package in your code and fills the ‘Description’ file in the correct place as imports or suggests, depending on where the dependency was declared in the package. This ensures beginners do not need to worry about dependencies or open the ‘Description’ file to write it by hand.

Because sharing and collaborating is central in the philosophy of {fusen}, I also encourage developers to use ‘git’ platforms like GitLab or GitHub, and use packages like {pkgdown}, {covr} and {testthat} to share documentation and the quality of their work.

Is this an ongoing project? Please share any details or CTA for who should get involved!

Yes, it is indeed an ongoing project that we use every day in our work to build packages for our customers. Although I initially created {fusen} to assist beginners in writing packages, we have discovered that it is equally valuable for experts, as it allows you to write the entire package in a single file (or groups of files when needed). This also makes it much easier to reuse and re-factor your code: if you split your package, you only have one file to worry about.

We are constantly improving {fusen}, and my colleague Yohann is currently working on adding a new feature allowing us to ‘inflate’ multiple flat files at the same time. With big projects, like {golem} applications, developers are used to separate family of functions in different flat files. This coming functionality will allow you to inflate all at once.

My call to action for the community is to give this package a try. Regardless of your level of expertise in R programming, {fusen} can make package writing much more accessible to you. We have been using it for testing purposes, but over time, we have realized that it is also quite convenient for experts. For instance, you can try {fusen} to add new functionalities to your already existing package, and it will work smoothly. You won’t have to change anything in the existing structure.

I would also like to encourage beginners to try {fusen}, as they are often hesitant to write packages. Writing a package can add complexity to your code due to CRAN-specific checks and other requirements, even if you do not want to send it on CRAN. However, {fusen} can simplify this process for you. We also have another package that is not on CRAN called {checkhelper}, which will help you identify some sources of package building problems. And, if finally, you decide to share your work on CRAN, you can follow our ‘Prepare for CRAN’ guide that we regularly update.

Note that {fusen} has a teaching flat template included, which shows a full example of a working package that you can use to explore its possibilities. You will see that you can build your package locally and have its documentation shared as a web page on GitHub in one command.

How do I Join?

R Consortium’s R User Group and Small Conference Support Program (RUGS) provides grants to help R groups around the world organize, share information and support each other. We have given grants over the past four years, encompassing over 65,000 members in 35 countries. We would like to include you! Cash grants and meetup.com accounts are awarded based on the intended use of the funds and the amount of money available to distribute. We are now accepting applications!

Apply HereFind your local R user group The post {fusen}: Simplifying Writing Packages for R Users appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: {fusen}: Simplifying Writing Packages for R Users

View Details

[This article was first published on business-science.io, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learning python is hard. But it’s important…

Even for R users like me that have built successful careers as a Data Scientist, Consultants, and Trainers.

Even for future data scientists that are just starting out trying to get a leg-up in the job market.

And even for old dogs that are doing the best to keep up with the ever-changing data science ecosystem (As I approach my 39th birthday, I guess I’d fall into this category now).

So why am I “all of a sudden” learning and promoting Python?

Well, I am not.

(Read on to understand why I still think R is the better language to start with for data scientists that need to do business analytics)

(And while you’re at it, quit putting words into my mouth. I’m an R guy through-and-through.)

Back to my story…

Python was my first data science language (before I learned R)I began learning Python in 2013 when I was just a naive doey-eyed sales manager trying to figure out how to analyze customers for my company.

Back then, I didn’t know what machine learning was.

Heck, I couldn’t even spell “API”.

I was just trying to get reports to my boss so I could win us more business (and actually make sense of what I suspected was going on with our “customers”).

My problem wasn’t that complex.I had a sneaking suscpicion that my small team of rag-tag inside salespeople were spending too much time on the wrong customers. And I suspected that it was costing us business.

What tool was I using every day?

Python

Excel

I was pulling in data from our customers, and trying to make sense of it all inside of… (Ahem)… Excel.

And eventually…

Excel crapped out.Yep, the trusty tool that I’d spent the better part of decade mastering crapped out.

I was an unofficial Excel ninja.

I knew VBA like the back of my hand (and that was kind of like coding… well sort of)

And I could do all of the cool tricks like Pivot Tables, Slicers, and even the new “Power” series of tools were coming out. You know… PowerBI, PowerQuery, PowerRangers…

But, my trusty Excel crapped out on me.

Fun Fact: Excel has this weird limitation that it can only pull in 1,048,576 rows of data.

(And my data was bigger than that.)

So I picked up PythonYes, I read all the blogs. All the forums.

Researched Reddit. Scoured LinkedIn. Sifted through Twitter.

And all the research was pointing to…

Python. (because everyone even back then was saying it was “the best”, surely it was the best, right?)

And guess what happened?

After 3 months of using Python, I quit trying to become a Data ScientistYep. I gave up. Took several months off. Why?

Python was hard.

I was getting bogged down in coding.

I couldn’t think through business problems because I was struggling trying to figure out how to set up an environment.

And then I’d finally get something working, update my environment, and it would break everything.

I didn’t get classes. What the heck was a “method”?

I couldn’t understand matplotlib to save my life (and still don’t, thanks to plolty).

And why was this so fricken’ tough?!

So I switched languages. And here’s the God’s-honest truth…

R was easierIt’s as simple as that.

I had invested months learning Python. And I made the decision to switch.

I knew that going backwards to Excel was NOT the answer.

So I tried R. And it was easier.

Yeah, the first week sucked. I was banging my head on the table trying to get R installed and libraries loaded.

But in that first week I did things that I failed to do with Python:

I could visualize my data! (thank you ggplot2).
And I could make a regression (thank you stats).

So to this day, I recommend any beginners that are struggling with Python to give R a try.

It is easier.

But why Python, and why “now”?I made a decision 3 years ago to pick Python back up.

2 years ago I launched my first Python course for the student that was in the same spot as I was…

I made if for those of us that are struggling to learn Python. Struggling to make a damn matplotlib plot. Stuggling to do a regression. And stuck (because we’re not coders, we are business analyzers).

And many of the success stories were students like Jeremy, who got a Senior Data Scientist job after taking my first Python course…

And this is what’s important to understand about Jeremy:

Jeremy learned R first
He picked up Python because some jobs “need” it for production (yeah you can do production with R too)
He never felt confident in Python until he learned Python from someone like him (who understood where he was coming from)
And Jeremy became confident after learning from an R user

The result was a big raise (I can’t say how much, but it was a lot).

Plus, he’s getting to do what he wants to do: Production.

The truth is…

Python takes your excuses awayCan you get a job with R? Absolutely.

Are there fewer R jobs than Python jobs? Yep.

Is there also less competition? 100%.

But, if you think you need Python to get a job or grow your career, then you should learn it.

Why?

To help your (future) company NOT fall into the trap of having 87% of their projects failWait, what?

Did Matt just say, “87% of data science projects fail?”

I did.

And it’s why Python and R users are getting fired (or I guess the term is, “restructured”).

Most people think it was the Recession that is causing data scientists to lose their job.

It’s not. The truth is…

Your job is on the line because your data science team isn’t productiveI know this because I was horribly ineffective with my first Fortune 500 consulting project.

I was hired by a company that does $8,000,000,000 every year to help them with an HR Analytics job.

And guess what- Matt, the self-proclaimed king of consulting… Failed miserably.

(By the way, I am particularly good at data science consulting. I pulled in $500,000+ in my 2nd year. So it wasn’t all downhill from this “failure”.)

Why did my “BIG” Fortune 500 Project Fail?It’s simple. Because like most data scientists I was trying to solve what I thought was the problem:

Predicting Employees that Would Leave The Company.

That’s what they told me they wanted.

And when I gave them a model that predicted exactly that on a big presentation that I worked all weekend on… I got dead silence in return.

And one day later I got fired.

So yeah, I know why data science projects are failing.

It’s because you’re trying to do data science.

Instead of solving the real problem:

Making the Company More Money

So fast forward to today.

My success rate on data science projects (after my failure) is 100%.

And I want to help you unlock this power… with Python.

I’m giving away my Pythonic Framework that virtually guarantees project success (and turns you instantly into a leader) on Thursday at 2PM ESTI’m launching my 2nd Python Course: Python for Machine Learning and API’s. And to celebrate I’m having a live launch party.

Inside the launch party, I’ll share:

Which tools I use in Python
What steps I take (every single time)
Where I use those tools
How to incorporate Machine Learning
How to deliver a “production” solution in minutes
How to become the go-to leader in your company
And how to fill your portfolio with projects that generate job demand

Live Launch Party (For My NEW Python for Machine Learning and APIs Course)What: Live Launch Party (NEW Python for Machine Learning and APIs Course)

When: Thursday May 25th, 2pm EST

How It Will Help You: Whether you are new to Python or are an expert, making the company money is what you need to be focused on. That’s why 87% of projects fail. But how can you be different? I’ll show you in side my free Python for Machine Learning and APIs Course Launch Event.

Price: Does Free sound good?

How To Join: Register Here

See you there!

-Matt

To leave a comment for the author, please follow the link and comment on their blog: business-science.io.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Why learning Python is hard (but important) for Data Scientists that use R

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In the clinical trials reporting industry, there is an incorrect assumption that SAS software is ideal as regulatory agencies “require” it. Regulatory agencies generally do not mandate the use of specific software for clinical trials reporting. They primarily focus on the accuracy, integrity, and compliance of the reported data.

Recently, other software options, such as the open-source R language, have gained attention across the life sciences, despite facing resistance due to this misconception. R has a strong history of use in academia for statistical research and is being utilized in the pharmaceutical industry. However, its adoption in regulatory submissions has been limited.

In recent years, developers from the pharmaceutical industry have taken advantage of R and collaboratively developed open-source libraries that can be used in clinical trials data analysis and reporting. This blog article/post will provide an overview of a few of such R packages, including their key features and how they can be used, along with resources for learning more about them.

Open source and proprietary worksPlease note, this is not legal advice! Explore Posit’s responses to related questions on:

Commercial use of R and Shiny
AGPL, GPL, and the use of Shiny applications in proprietary environments

One of the common misconceptions about using open-source packages in software development is that it requires companies to publicly disclose their own code and proprietary workflows. This misconception often arises from the fact that many popular open-source packages, such as R and its various libraries, are licensed under the GNU General Public License (GPL) or other similar licenses that require the distribution of source code.

However, it is important to note that using open-source packages does not necessarily require companies to disclose their own code or proprietary workflows. The GPL and other similar licenses only require the distribution of source code if the software that uses the open-source packages is also distributed.

For example, if a company is developing an internal application that uses R and various open-source libraries, and that application is only used within the company, then there is no requirement to distribute the source code for that application. The company can keep its code and workflows confidential, even if it uses open-source packages.

On the other hand, if a company develops a commercial application that incorporates open-source packages and then distributes that application to customers, then it may be required to distribute the source code for the application, including any modifications made to the open-source packages. However, this requirement can often be satisfied by providing access to the source code through a written offer, rather than including it with the distributed software.

There is an alternative interpretation where your code, which utilizes external R libraries, is not compiled and loads the libraries into memory when the app is executed. Therefore, the library mentioned does not refer specifically to a given library from CRAN; it can be any library from your local drive with a different license. As a result, your code allows for the integration of any library that adheres to the given interface and does not rely on any specific GPL library. This indicates that your app is a separate component that uses other packages but is not linked as one product, eliminating the requirement to open source your code.

Understanding Clinical Data StandardsCDISC (Clinical Data Interchange Standards Consortium) is a global, non-profit organization that develops and promotes data standards to support the acquisition, exchange, submission, and archive of clinical research data and metadata.

CDISC Foundational Standards provide a comprehensive set of data standards that improve the quality, efficiency, and cost-effectiveness of clinical research. These standards cover various aspects of clinical research, from study design to data collection, management, analysis, and reporting.

The standards include the

Protocol Representation Model (PRM)
Standard for Exchange of Nonclinical Data (SEND)
Clinical Data Acquisition Standards Harmonization (CDASH)
Study Data Tabulation Model (SDTM)
Study Data Tabulation Model Implementation Guide (SDTMIG)
Analysis Data Model (ADaM).

These standards help ensure that clinical trial data is organized and analyzed consistently and accurately, enhancing the efficiency and quality of clinical research. The QRS supplements provide a standardized way to assess clinical concepts or task-based observations.

ADaM is required by both the FDA (US) and PMDA (Japan) for new drug applications, while SEND is required by the FDA for nonclinical studies. CDISC standards improve transparency and traceability, making it easier for regulators and others to conduct data review.

Insights Engineering{teal}: Interactive Exploratory Data Analysis with Shiny Web-Applications{teal} is a framework for interactive exploratory data analysis that uses Shiny web-applications. {teal} applications require specifying data, including CDISC data, independent datasets, related datasets, and MultiAssayExperiment objects. The framework also provides modules for performing analysis, such as outlier exploration and data visualization. {teal} modules are built within the framework and can be found in packages like {teal.modules.general}, {teal.modules.clinical} and {teal.modules.{hermes} }.

The functionality of the framework is derived from packages like {teal.data}, {teal.widgets}, {teal.slice}, {teal.code}, {teal.transform}, {teal.logger} and {teal.reporter}. There is also a package called {teal.osprey} that takes community teal modules. Users can refer to these packages for more information on how to use different parts of the {teal} framework.

{hermes}

{hermes} is a tool that helps with preprocessing, analysis, and reporting of RNA-seq data. It has the ability to import RNAseq count data and annotate gene information from a central database like BioMart. It also adds quality control flags to genes and samples, filters the data set, and normalizes counts.

{hermes} can work with data structures from bioconductor packages, thereby allowing interoperability. It can also quickly generate descriptive plots, perform principal components analysis, and produce a QC report based on a template. Additionally, it can perform differential expression analysis.

{tern}The R package called {tern} offers various analysis functions for generating tables and graphs commonly used in clinical trial reporting. This package provides a wide range of functionalities including data visualizations such as forest plots, line plots, Kaplan-Meier plots, as well as statistical model fits like logistic regression and Cox regression. Additionally, {tern} allows for the creation of summary tables containing information about unique patients, exposure across patients, and changes from baseline for parameters. Furthermore, {tern} outputs can be added to {teal} applications for interactive exploration of data through modules available in the {teal.modules.clinical} package.

Reference Based Multiple Imputation {rbmi}The R package called {rbmi} is designed for imputing missing data in clinical trials with continuous multivariate normal longitudinal outcomes. It can handle missing data under a missing at random (MAR) assumption, reference-based imputation methods, and delta adjustments for sensitivity analysis like tipping point analyses. The package offers both Bayesian and approximate Bayesian multiple imputation, which is combined with Rubin’s rules for inference, as well as frequentist conditional mean imputation with jackknife or bootstrap resampling.PharmaversePharmaverse is a network of pharmaceutical industry professionals working collaboratively to create a curated and opinionated subset of open-source software packages and codebases based on the R language.

The objective is to deliver a complete clinical data pipeline from data collection to regulatory submission that is more efficient and sustainable through shared development and maintenance efforts, with a focus on reducing duplication of efforts and gaining increased harmonization across the industry. The initiative aims to attract the next generation of software developers and data scientists to the industry and provide increased transparency.

The scope of pharmaverse is the journey from Case Report Form (CRF) through to submission for clinical trial analysis reporting via R packages, with three categories of R packages recommended:

External to pharma (transcends specifically pharma needs).
Pharma-specific independent of pharmaverse (created for use in pharma, but not necessarily following the pharmaverse charter and recommendations).
Pharma-specific under pharmaverse (created for use in pharma, following the pharmaverse charter and recommendations).

The aim is not to agree on cross-industry implementations of CDISC standards, but rather to act as a starting point for code reuse that is standard agnostic and future-proof. The design and architecture of the package allow companies to use them to adapt to internal workflows that are proprietary. For example, {admiral} package has {admiral.vaccines} and {admiral.ophta}, but there is also an {admiral.roche} that is internal to Roche and not shared (with proprietary license).

Pharmaverse packages will be easily locatable and accessible via a single site, with clear use cases for clinical trial reporting and sharing of differences and unique merits to help companies or users choose which packages or tools to adopt.

Pharmaverse End-to-End Clinical Reporting PackagesLet us explore some of the End-to-End Clinical Reporting Packages from Pharmaverse. The following compilation comprises ‘some’ of the open-source R packages that are relevant to end-to-end clinical reporting in the pharmaceutical industry. The pharmaverse council aims to organize and curate these packages into a well-defined stack in due course.

Conclusion
The life sciences industry is expanding at an unprecedented rate, and open-source initiatives have provided a significant impetus to innovation. With the advent of package developments with programming languages like R, conducting clinical trials, monitoring and analyzing data has become more streamlined and precise.

Collaborative projects in the life sciences domain, spearheaded by pharmaceutical companies and experts, have propelled advancements in research, enabling more rapid and precise discoveries. These open-source projects, which leverage cutting-edge technologies and techniques, are expected to pave the way for accelerated progress in the life sciences industry, leading to better healthcare outcomes for people worldwide.

Conclusion: R Packages for Clinical Trial DataThe life sciences industry is expanding rapidly, and open-source initiatives have provided a significant impetus to innovation. With the advent of package developments with programming languages like R, conducting clinical trials, monitoring and analyzing data has become more streamlined and precise.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Programming and Pharmaceutical Data Analysis (Packages for Clinical Trial Data)

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionThe sink() function in R is used to divert R output to an external connection. This can be useful for a variety of purposes, such as exporting data to a file, logging R output, or debugging R code.

In this blog post, we will explore the inner workings of the sink() function, understand its purpose, and provide practical examples using the popular datasets mtcars and iris.

The sink() function takes four arguments:

file: The name of the file to which R output will be diverted. If file is NULL, then R output will be diverted to the console.
append: A logical value indicating whether R output should be appended to the file (TRUE) or overwritten (FALSE). The default value is FALSE.
type: A character string. Either the output stream or the messages stream. The name will be partially match so can be abbreviated.
split: logical: if TRUE, output will be sent to the new sink and the current output stream, like the Unix program tee.

ExamplesHere are some examples of how to use the sink() function. To export the mtcars dataset to a file called “mtcars.csv”, you would use the following code:

sink("mtcars.csv")print(mtcars)

mpg cyl disp hp drat wt qsec vs am gear carbMazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2

sink() To log R output to a file called “r_output.log”, you would use the following code:

sink("r\_output.log")# Your R code goes heresink() To debug R code, you can use the sink() function to divert R output to a file. This can be helpful for tracking down errors in your code. For example, if you are trying to debug a function called my_function(), you could use the following code:

sink("my\_function.log")my\_function()sink() Capturing Summary Statistics of mtcars Dataset sink("summary\_output.txt") # Redirect output to the filesummary(mtcars) # Generate summary statistics

mpg cyl disp hp Min. :10.40 Min. :4.000 Min. : 71.1 Min. : 52.0 1st Qu.:15.43 1st Qu.:4.000 1st Qu.:120.8 1st Qu.: 96.5 Median :19.20 Median :6.000 Median :196.3 Median :123.0 Mean :20.09 Mean :6.188 Mean :230.7 Mean :146.7 3rd Qu.:22.80 3rd Qu.:8.000 3rd Qu.:326.0 3rd Qu.:180.0 Max. :33.90 Max. :8.000 Max. :472.0 Max. :335.0 drat wt qsec vs Min. :2.760 Min. :1.513 Min. :14.50 Min. :0.0000 1st Qu.:3.080 1st Qu.:2.581 1st Qu.:16.89 1st Qu.:0.0000 Median :3.695 Median :3.325 Median :17.71 Median :0.0000 Mean :3.597 Mean :3.217 Mean :17.85 Mean :0.4375 3rd Qu.:3.920 3rd Qu.:3.610 3rd Qu.:18.90 3rd Qu.:1.0000 Max. :4.930 Max. :5.424 Max. :22.90 Max. :1.0000 am gear carb Min. :0.0000 Min. :3.000 Min. :1.000 1st Qu.:0.0000 1st Qu.:3.000 1st Qu.:2.000 Median :0.0000 Median :4.000 Median :2.000 Mean :0.4062 Mean :3.688 Mean :2.812 3rd Qu.:1.0000 3rd Qu.:4.000 3rd Qu.:4.000 Max. :1.0000 Max. :5.000 Max. :8.000

sink() # Turn off redirection In this example, the output of the summary(mtcars) command will be saved in the “summary_output.txt” file. We can later open the file to review the summary statistics of the mtcars dataset.

Saving Regression Results of iris Dataset sink("regression\_results.txt") # Redirect output to the filefit <- lm(Sepal.Length ~ Sepal.Width, data = iris) # Perform linear regressionsummary(fit) # Display regression summary

Call:lm(formula = Sepal.Length ~ Sepal.Width, data = iris)Residuals: Min 1Q Median 3Q Max -1.5561 -0.6333 -0.1120 0.5579 2.2226 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 6.5262 0.4789 13.63 <2e-16 ***Sepal.Width -0.2234 0.1551 -1.44 0.152 ---Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Residual standard error: 0.8251 on 148 degrees of freedomMultiple R-squared: 0.01382, Adjusted R-squared: 0.007159 F-statistic: 2.074 on 1 and 148 DF, p-value: 0.1519

sink() # Turn off redirection In this example, the output of the summary(fit) command will be saved in the “regression_results.txt” file. By redirecting the output, we can analyze the regression results in detail without cluttering the console.

Appending Output to a FileBy default, calling sink() with a file name will overwrite any existing content in the file. However, if we want to append output to an existing file, we can pass the append = TRUE argument to sink().

sink("output.txt", append = TRUE) # Append output to the existing filecat("Additional text\n") # Append custom text

Additional text

sink() # Turn off redirection In this example, the string “Additional text” will be appended to the “output.txt” file. This feature is useful when we want to continuously update a log file or add multiple output sections to a single file.

ConclusionThe sink() function is a handy tool in R that allows us to redirect output to external files. By using this function, we can save and review the output generated during data analysis, statistical modeling, or any other R programming tasks. In this blog post, we explored the basic usage of sink() and provided practical examples using the mtcars and iris datasets. By mastering sink(), you can efficiently manage your R output and ensure a more organized workflow.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: What is the sink() function? Capturing Output to External Files

View Details

[This article was first published on JottR on R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Below are the slides from my presentation at the Edmonton R User Group Meetup (YEGRUG) on May 22, 2023:

Title: Futureverse – A Unifying Parallelization Framework in R for Everyone
Speaker: Henrik Bengtsson
Slides: HTML, PDF (46 slides)

Thank you Péter Sólymos and the YEGRUG for the invitate and the opportunity!

/Henrik

Links YEGRUG: https://yegrug.github.io/ * Futureverse website: https://www.futureverse.org/ * future* package CRAN, GitHub, pkgdown

To leave a comment for the author, please follow the link and comment on their blog: JottR on R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Edmonton R User Group Meetup: Futureverse – A Unifying Parallelization Framework in R for Everyone

View Details

[This article was first published on Pachá, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This workshop aims to introduce people with basic R knowledge to develop interactive web applications with the Shiny framework.

The course consists of a one-hour session, where we will demonstrate basic UI, reactive UI, CSS personalization and dashboard creation. Questions are super welcome!

Previous knowledge required: Basic R (examples: reading a CSV file, transforming columns and making graphs using ggplot2).

The course will be delivered online using Zoom on July 1stst from 17.30 to 18.30. Check the timezone. For this workshop, it is New York Time (https://www.timeanddate.com/worldclock/usa/new-york).

Finally, here’s a short demo of a part of what this workshop covers https://youtu.be/DW-HPfohfwg.

If you consider this workshop useful, you can buy me a coffee at https://www.buymeacoffee.com/pacha/.

You can get a ticket at https://www.buymeacoffee.com/pacha/e/138580.

To leave a comment for the author, please follow the link and comment on their blog: Pachá.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: An introductory workshop in Shiny, June 1st from 17.30 to 18.30 (FREE)

View Details

[This article was first published on ouR data generation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A colleague reached out for help designing a cluster randomized trial to evaluate a clinical decision support tool for primary care physicians (PCPs), which aims to improve care for high-risk patients. The outcome will be a time-to-event measure, collected at the patient level. The unit of randomization will be the PCP, and one of the key design issues is settling on the number to randomize. Surprisingly, I’ve never been involved with a study that required a clustered survival analysis. So, this particular sample size calculation is new for me, which led to the development of simulations that I can share with you. (There are some analytic solutions to this problem, but there doesn’t seem to a consensus about the best approach to use.)

OverviewIn tackling this problem, there were four key elements that I needed to work out before actually conducting the power simulations. First, I needed to determine the hypothetical survival curve in the context of a single (control) arm and simulate data to confirm that I could replicate the desired curve. Second, I wanted to generate cluster-level variation so that I could assess the implications of the variance assumptions (still in a single-arm context). Third, I generated two intervention arms (without any clustering) to assess effect size assumptions. And lastly, I generated a full data set that included clustering, randomization, and censoring, and then fit a model that would be the basis for the power analysis to ensure that everything was working as expected. Once this was all completed, I was confident that I could move on to generating estimates of power under a range of sample size and variability assumptions. I apologize in advance for a post that is a bit long, but the agenda is quite packed and includes a lot of code.

Defining shape of survival curveDefining the shape of the survival curve is made relatively easy using the function survGetParams in the simstudy package. All we need to do is specify some (at least two) coordinates along the curve and the function will return the parameters for the mean and shape of a Weibull function that best fit the points. These parameters are used in the data generation process. In this case, the study’s investigators provided me with a couple of points, indicating that approximately 10% of the sample would have an event by day 30, and half would have an event at day 365. Since the study is following patients at most for 365 days, we will consider anything beyond that to be censored (more on censoring later).

To get things started, here are the libraries needed for all the code that follows:

library(simstudy)library(data.table)library(survival)library(survminer)library(GGally)library(coxme)library(parallel) Now, we get the parameters that define the survival curve:

points <- list(c(30, 0.90), c(365, .50))r <- survGetParams(points)r## [1] -4.8 1.3 To simulate data from this curve, the time-to-event variable tte is defined using these parameters generating by survGetParams. The observed time is the minimum of one year and the time-to-event, and we create an event indicator if the time-to-even is less than one year.

defs <- defSurv(varname = "tte", formula = r[1], shape = r[2])defa <- defDataAdd(varname = "time", formula = "min(365, tte)", dist = "nonrandom")defa <- defDataAdd(defa, "event", "1*(tte <= 365)", dist = "nonrandom") Generating the data is quite simple in this case:

set.seed(589823)dd <- genData(1000)dd <- genSurv(dd, defs, digits = 0)dd <- addColumns(defa, dd)dd## id tte time event## 1: 1 234 234 1## 2: 2 1190 365 0## 3: 3 395 365 0## 4: 4 178 178 1## 5: 5 72 72 1## --- ## 996: 996 818 365 0## 997: 997 446 365 0## 998: 998 118 118 1## 999: 999 232 232 1## 1000: 1000 5308 365 0 The plots below show the source function determined by the parameters on the left and the actual data generated on the right. It appears that the generated data matches the data generation process:

splot <- survParamPlot(r[1], r[2], points = points, n = 1000, limits = c(0, 365) )fit <- survfit(Surv(time, event) ~ 1, data = dd)j <- ggsurv(fit, CI = FALSE, surv.col = "#ed7c67", size.est = 0.8) + theme(panel.grid = element\_blank(), axis.text = element\_text(size = 7.5), axis.title = element\_text(size = 8, face = "bold"), plot.title = element\_blank()) + ylim(0, 1) + xlab("time") + ylab("probability of survival")ggarrange(splot, j, ncol = 2, nrow = 1) Evaluating cluster variationCluster variation in the context of survival curves implies that there is a cluster-specific survival curve. This variation is induced with a random effect in in the data generation process. In this case, I am assuming a normally distributed random effect with mean 0 and some variance (distributions other than a normal distribution can be used). The variance assumption is a key one (which will ultimately impact the estimates of power), and I explore that a bit more in the second part of this section.

Visualizing cluster variationThe data generation process is a tad more involved than above, though not much more. We need to generate clusters and their random effect first, before adding the individuals. tte is now a function of the distribution parameters as well as the cluster random effect b. We are still using a single arm and assuming that everyone is followed for one year. In the first simulation, we set the random effect variance (b = 0.1).

defc <- defData(varname = "b", formula = 0, variance = 0.1)defs <- defSurv(varname = "tte", formula = "r[1] + b", shape = r[2])defa <- defDataAdd(varname = "time", formula = "min(365, tte)", dist = "nonrandom")defa <- defDataAdd(defa, "event", "1*(tte <= 365)", dist = "nonrandom")dc <- genData(20, defc, id = "pcp")dd <- genCluster(dc, "pcp", numIndsVar = 1000, "id")dd <- genSurv(dd, defs, digits = 0)dd.10 <- addColumns(defa, dd)dd.10## pcp b id tte time event## 1: 1 0.25 1 788 365 0## 2: 1 0.25 2 93 93 1## 3: 1 0.25 3 723 365 0## 4: 1 0.25 4 151 151 1## 5: 1 0.25 5 1367 365 0## --- ## 19996: 20 0.22 19996 424 365 0## 19997: 20 0.22 19997 207 207 1## 19998: 20 0.22 19998 70 70 1## 19999: 20 0.22 19999 669 365 0## 20000: 20 0.22 20000 994 365 0 The following plot shows two sets of survival curves, each based on different levels of variation, 0.10 on the left, and 0.01 on the right. Each curve represents a different cluster. With this plot, we get a clear visualization of how variance assumption of the random effect impacts the variation of the survival curves:

Variation of the probability of an event across clustersThe plot of the survival curves is only one way to consider the impact of cluster variation. Another option is to look at the binary event outcome under the assumption of no censoring. I like to evaluate the variation in the probability of an event across the clusters, particularly by looking at the range of probabilities, or considering the coefficient of variation, which is (\sigma / \mu).

To show how this is done, I am generating a data set with a very large number of clusters (2000) and a large cluster size (500), and then calculating the probability of an event for each cluster:

defc <- defData(varname = "b", formula = 0, variance = 0.100)dc <- genData(2000, defc, id = "pcp")dd <- genCluster(dc, "pcp", numIndsVar = 500, "id")dd <- genSurv(dd, defs, digits = 0)dd <- addColumns(defa, dd)ds <- dd[, .(p = mean(event)), keyby = pcp]ds## pcp p## 1: 1 0.41## 2: 2 0.43## 3: 3 0.36## 4: 4 0.51## 5: 5 0.50## --- ## 1996: 1996 0.82## 1997: 1997 0.53## 1998: 1998 0.88## 1999: 1999 0.42## 2000: 2000 0.63 Here is the distribution of observed cluster-level proportions:

Here are the mean probability, the standard deviation of probabilities, the coefficient of variation for the probabilities, and the 95% interval of the probabilities when the random effect variance in the survival generation process is 0.10:

ds[, .(mu = mean(p), s = sd(p), cv = sd(p)/mean(p))]## mu s cv## 1: 0.5 0.11 0.22ds[, .(quantile(p, probs = c(0.025, .975)))]## V1## 1: 0.31## 2: 0.72 To compare across a range variance assumptions, I’ve generated ten data sets and plotted the results below. If you hover over the points, you will get the CV estimate. This could be helpful in helping collaborators decide what levels of variance is appropriate to focus on in the final power estimation and sample size determination.

In this case, we’ve decided that a coefficient of variation is not likely to exceed 0.17 (with a corresponding 95% interval of proportions ranging from 35% to 66%), so we’ll consider that when evaluating power.

Evaluating the effect sizeNext, we generate data that includes treatment assignment (but excludes cluster variability and censoring before one year). The treatment effect is expressed as a log hazard ratio, which in this case 0.4 (equal to a hazard ratio of just about 1.5).

The data generation starts with treatment assignment, adds the time-to-event survival data, and then adds the one-year censoring data, as before:

defa <- defData(varname = "rx", formula = "1;1", dist = "trtAssign")defs <- defSurv(varname = "tte", formula = "r[1] + 0.4 * rx", shape = r[2])defe <- defDataAdd(varname = "time", formula = "min(365, tte)", dist = "nonrandom")defe <- defDataAdd(defe, "event", "1*(tte <= 365)", dist = "nonrandom")dd <- genData(1000, defa)dd <- genSurv(dd, defs, digits = 0)dd <- addColumns(defe, dd)dd## id rx tte time event## 1: 1 0 1902 365 0## 2: 2 0 286 286 1## 3: 3 0 480 365 0## 4: 4 0 32 32 1## 5: 5 1 12 12 1## --- ## 996: 996 0 663 365 0## 997: 997 1 962 365 0## 998: 998 1 19 19 1## 999: 999 1 502 365 0## 1000: 1000 0 85 85 1 The plot of the survival curves by treatment arms provides a visualization of the treatment effect:

Again, since we have no censoring, we can estimate the probability of an event within 365 days for each arm:

dd[, .(p = mean(event)), keyby = rx]## rx p## 1: control 0.51## 2: rx 0.63 An effect size of 0.4 on the log hazard scale translates to an odds ratio of about 1.64, a risk ratio of 1.24, and risk difference of 12 percentage points. In the absence of any external data on the potential effect size, we can use an effect size that is minimally clinically meaningful based on any or all these effect size measurements.

Complete data generation and model estimationWith the pieces in place, we are ready to put it all together and add censoring to the mix to finalize the full data generating process. We will fit a mixed-effects Cox proportional hazards model to see if we can recover the parameters that we have used to generate the data, and, if that goes well, we will be ready to estimate power.

We start in defc by defining the cluster-level random effect variation and treatment assignment design (in this case 1 to 1, treatment to control). We add a censoring process in defa. This assumes that we will be enrolling patients for six months, spread out across this time period. The study will last exactly one year, so every patient will be followed for at least six months, and only some will be followed for a full year (i.e., those who join the study on the first day).

Finally, defs defines the data generation process for the survival outcome, which we’ve seen above, though now we have both a treatment effect and a random effect, in addition to the baseline parameters in the vector r.

defc <- defData(varname = "b", formula = 0, variance = 0.05)defc <- defData(defc, varname = "rx", formula = "1;1", dist = "trtAssign")defa <- defDataAdd(varname = "start\_day", formula = "1;182", dist = "uniformInt")defa <- defDataAdd(defa, varname = "censor", formula = "365 - start\_day ", dist = "nonrandom")defs <- defSurv(varname = "tte", formula = "r[1] + 0.4 * rx + b", shape = r[2]) The data generation is the same as before, though there is an additional censoring process, which is done with the function addCompRisk:

dc <- genData(500, defc, id = "pcp")dd <- genCluster(dc, "pcp", numIndsVar = 200, "id")dd <- addColumns(defa, dd)dd <- genSurv(dd, defs, digits = 0)dd <- addCompRisk(dd, events = c("tte", "censor"), timeName = "time", censorName = "censor", keepEvents = TRUE)dd## pcp b rx id start\_day censor tte time event type## 1: 1 -0.0158 1 1 65 300 307 300 0 censor## 2: 1 -0.0158 1 2 113 252 907 252 0 censor## 3: 1 -0.0158 1 3 121 244 110 110 1 tte## 4: 1 -0.0158 1 4 60 305 170 170 1 tte## 5: 1 -0.0158 1 5 95 270 2673 270 0 censor## --- ## 99996: 500 -0.0015 0 99996 160 205 291 205 0 censor## 99997: 500 -0.0015 0 99997 79 286 193 193 1 tte## 99998: 500 -0.0015 0 99998 64 301 28 28 1 tte## 99999: 500 -0.0015 0 99999 153 212 459 212 0 censor## 100000: 500 -0.0015 0 100000 19 346 298 298 1 tte Since we have generated a rather large data set, we should be able to recover the parameters pretty closely if we are using the correct model. We are going to fit a mixed effects survival model (also known as a frailty model) to see how well we did.

fit\_coxme <- coxme(Surv(time, event) ~ rx + (1 | pcp), data = dd)summary(fit\_coxme)## Cox mixed-effects model fit by maximum likelihood## Data: dd## events, n = 49955, 1e+05## Iterations= 11 48 ## NULL Integrated Fitted## Log-likelihood -555429 -553708 -553058## ## Chisq df p AIC BIC## Integrated loglik 3442 2 0 3438 3421## Penalized loglik 4743 414 0 3914 259## ## Model: Surv(time, event) ~ rx + (1 | pcp) ## Fixed coefficients## coef exp(coef) se(coef) z p## rx 0.39 1.5 0.022 18 0## ## Random effects## Group Variable Std Dev Variance## pcp Intercept 0.22 0.05 Pretty good! The estimated HR of 0.39 (95% CI: 0.35 - 0.43) is on target (we used 0.40 in the data generation process), and the estimated variance for the PCP random effect was 0.05, also on the mark. I’d say we are ready to proceed to the final step.

Power estimationTo conduct the power estimation, I’ve essentially wrapped the data generation and model estimation code in a collection of functions that can be called repeatedly to generate multiple data sets and model estimates The goal is to calculate the proportion of data sets with a statistically significant result for a particular set of assumptions (i.e., the estimate of power for the assumed effect size, variation, and sample sizes). I’ve provided the code below in the addendum in case you haven’t grown weary of all this detail. I described a general framework for using simulation to estimate sample size/power, and I’m largely following that process here.

I’ve written a little function scenario_list (which I’m now thinking I should add to simstudy) to create different parameter combinations that will determine the power estimation. In this case, the parameters I am interested in are the number of PCPs that should be randomized and the variance assumption. The number of patients per PCP (cluster size) is also important to vary, but for illustration purposes here I am keeping it constant.

Here is the simplified scenario list with four possible combinations:

scenario\_list <- function(...) { argmat <- expand.grid(...) return(asplit(argmat, MARGIN = 1))}npcps <- c(20, 30)npats <- c(15)s2 <- c(0.03, 0.04)scenarios <- scenario\_list(npcps = npcps, npats = npats, s2 = s2)scenarios## [[1]]## npcps npats s2 ## 20.00 15.00 0.03 ## ## [[2]]## npcps npats s2 ## 30.00 15.00 0.03 ## ## [[3]]## npcps npats s2 ## 20.00 15.00 0.04 ## ## [[4]]## npcps npats s2 ## 30.00 15.00 0.04 I use the mclapply function in the parallel package to generate three iterations for each scenario:

model.ests <- mclapply(scenarios, function(a) s\_scenarios(a, nrep = 3))model.ests## [[1]]## npcps npats s2 est\_s re.var\_s p\_s## 1: 20 15 0.03 0.41 0.1596 0.100## 2: 20 15 0.03 0.32 0.0306 0.065## 3: 20 15 0.03 0.32 0.0004 0.041## ## [[2]]## npcps npats s2 est\_s re.var\_s p\_s## 1: 30 15 0.03 0.13 0.017 3.5e-01## 2: 30 15 0.03 0.62 0.029 7.3e-05## 3: 30 15 0.03 0.45 0.040 3.8e-03## ## [[3]]## npcps npats s2 est\_s re.var\_s p\_s## 1: 20 15 0.04 0.55 7.9e-02 0.0084## 2: 20 15 0.04 0.07 4.0e-04 0.6700## 3: 20 15 0.04 0.24 8.2e-05 0.1500## ## [[4]]## npcps npats s2 est\_s re.var\_s p\_s## 1: 30 15 0.04 0.42 8.3e-05 2.7e-03## 2: 30 15 0.04 0.57 4.0e-04 2.6e-05## 3: 30 15 0.04 0.36 1.0e-01 4.2e-02 In the actual power calculations, which are reported below, I used 60 scenarios defined by these data generation parameters:

npcps <- c(18, 24, 30, 36)npats <- c(15, 18, 21)s2 <- c(0.01, 0.02, 0.03, 0.04, 0.05) For each of these scenarios, I generated 5000 data sets and estimated models for each (i.e., a total of 300,000 data sets and model fits). For each of the 60 scenarios, I estimated the proportion of the 5000 model fits that yielded a p-value < 0.05 for the estimated log hazard ratio. I had the benefit of using a high performance computer, because running this on my laptop would have taken well over 10 hours (only about 10 minutes on the HPC).

At the end, we have a plot of “power curves” that shows estimated power for each of the scenarios. If we assume that we can expect at least 18 patients per PCP and that the between-PCP variance will be around 0,03 or 0.04, we should be OK randomizing 30 PCPs (15 in each arm), though it might more prudent to go with 36, just to be safe:

AddendumHere is the code I used to generate the data for the power curve plot. It is based on the framework I mentioned earlier. There is one extra function here, extract_coxme_table, which I pulled from stackoverflow, because there is currently no obvious way to extract data from the coxme model fit.

extract\_coxme\_table <- function (mod) { beta <- mod$coefficients nvar <- length(beta) nfrail <- nrow(mod$var) - nvar se <- sqrt(diag(mod$var)[nfrail + 1:nvar]) z <- round(beta/se, 2) p <- signif(1 - pchisq((beta/se)^2, 1), 2) table=data.table(beta = beta, se = se, z = z, p = p) return(table)}s\_def <- function() { defc <- defData(varname = "b", formula = 0, variance = "..s2") defc <- defData(defc, varname = "rx", formula = "1;1", dist = "trtAssign") defa <- defDataAdd(varname = "start\_day", formula = "1;182", dist = "uniformInt") defa <- defDataAdd(defa, varname = "censor", formula = "365 - start\_day ", dist = "nonrandom") defs <- defSurv(varname = "tte", formula = "-4.815 + 0.4 * rx + b", shape = 1.326) defa2 <- defDataAdd(varname = "event6", formula = "1*(tte <= 182)", dist = "nonrandom") return(list(defc = defc, defa = defa, defs = defs, defa2 = defa2)) }s\_generate <- function(argsvec, list\_of\_defs) { list2env(list\_of\_defs, envir = environment()) list2env(as.list(argsvec), envir = environment()) dc <- genData(npcps, defc, id = "pcp") dd <- genCluster(dc, "pcp", npats, "id") dd <- addColumns(defa, dd) dd <- genSurv(dd, defs, digits = 0) dx <- addCompRisk(dd, events = c("tte", "censor"), timeName = "time", censorName = "censor", keepEvents = TRUE) dx <- addColumns(defa2, dx) dx[] }s\_replicate <- function(argsvec, list\_of\_defs) { dx <- s\_generate(argsvec, list\_of\_defs) coxfitm <-coxme(Surv(time, event) ~ rx + (1 | pcp), data = dx) list2env(as.list(argsvec), envir = environment()) return(data.table( npcps = npcps, npats = npats, s2 = s2, est\_s = fixef(coxfitm), re.var\_s = VarCorr(coxfitm)$pcp, p\_s = extract\_coxme\_table(coxfitm)$p )) }s\_scenarios <- function(argsvec, nreps) { list\_of\_defs <- s\_def() rbindlist( parallel::mclapply( X = 1 : nreps, FUN = function(x) s\_replicate(argsvec, list\_of\_defs), mc.cores = 4) ) } To leave a comment for the author, please follow the link and comment on their blog: ouR data generation.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A demo of power estimation by simulation for a cluster randomized trial with a time-to-event outcome

View Details

[This article was first published on Pachá, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.R/Shiny Training: Should you find this blog to be of interest, kindly note that I offer personalized and group-based trainingsessions that may be reserved through Buy me a Coffee. Additionally, I provide training services in the Spanish language and am available to discuss means by which I may contribute to your Shiny project.

Updated 2023-05-23: I explicitly mention to add the vendoring after creating the project instead of assuming the readers know I run a commented line in the install script.

MotivationIn my previous post A step by step guide to write an R package that uses C++ code (Ubuntu) I explained the steps in my journey to use C++ function from R.

Now I will explain how I figured out how to use vendoring, which means that I copy the code for the dependencies into your project’s source tree, and in this case this means I copied the C++ headers provided by the cpp11 package into my own R package. This ensures the dependency code is fixed and stable until it is updated.

The advantage of vendoring is that changes to the cpp11 package could never break your existing code. The disadvantage is that you no longer get bugfixes and new features until you manually run cpp_vendor() in your project.

Creating an R package with C++ code and vendoringSimilar to the previous post, I started with create_package("~/github/cpp11gaussjordan"). I am using VSCode but all my steps also apply to RStudio.

After opening ~/github/cpp11gaussjordan I run use_cpp11() to have a readily availably skeleton in my project.

I run use_apache_licence() to have a LICENSE file and indicate in DESCRIPTION that my package is distributed under the Apache License.

Then I run cpp_vendor() to copy the C++ headers into inst/include.

You can find the final result on my GitHub profile.

R sideI run use_r("cpp11gaussjordan-package") and added the following contents.

```

' @useDynLib cpp11gaussjordan, .registration = TRUENULL#' Invert (some) square matrices#' @export#' @param A numeric matrix#' @return numeric matrix#' @examples#' A <- matrix(c(2,1,3,-1), nrow = 2, ncol = 2)#' invert_matrix(A)invert_matrix <- function(A) { invert_matrix_(A)}#' Solve (some) linear systems#' @export#' @param A numeric matrix#' @param b numeric matrix#' @return numeric matrix#' @examples#' A <- matrix(c(2,1,3,-1), nrow = 2, ncol = 2)#' b <- matrix(c(7,4), nrow = 2, ncol = 1)solve_system <- function(A,b) { solve_system_(A, b)}

`` Heresolve_system()is the “end-user” function, andsolve_system_()is the “development” function. The “development” functions were written in C++ andcpp11handles adding the.Call()` when required to link the C++ code and be able to use the C++ functions from R.

C++ sideI replaced code.cpp contents with the following lines.

```

include #include using namespace cpp11;[[cpp11::register]] doubles_matrix<> invert_matrix_(doubles_matrix<> A){ // Obtain (X'X)^-1 via Gauss-Jordan // Check dimensions int N = A.nrow(); int M = A.ncol(); if (N != M) { stop("X must be a square matrix"); } // Copy the matrix writable::doubles_matrix<> Acopy(N, N); for (int i = 0; i < N; i++) { for (int j = 0; j < N; j++) { Acopy(i, j) = A(i, j); } } // Create the identity matrix as a starting point for Gauss-Jordan writable::doubles_matrix<> Ainv(N, N); for (int i = 0; i < N; i++) { for (int j = 0; j < N; j++) { if (i == j) { Ainv(i, j) = 1.0; } else { Ainv(i, j) = 0.0; } } } // Overwrite Ainv by steps with the inverse of A // in other words, find the echelon form of A for (int i = 0; i < M; i++) { double a = Acopy(i, i); for (int j = 0; j < M; j++) { Acopy(i, j) /= a; Ainv(i, j) /= a; } for (int j = 0; j < M; j++) { if (i != j) { a = Acopy(j, i); for (int k = 0; k < M; k++) { Acopy(j, k) -= Acopy(i, k) * a; Ainv(j, k) -= Ainv(i, k) * a; } } } } return Ainv;}[[cpp11::register]] doubles_matrix<> solve_system_(doubles_matrix<> A, doubles_matrix<> b){ // Use Gauss-Jordan to solve the system Ax = b // Check dimensions int N1 = A.nrow(); int M1 = A.ncol(); if (N1 != M1) { stop("A must be a square matrix"); } int N2 = b.nrow(); int M2 = b.ncol(); if (N1 != N2) { stop("b must have the same number of rows as A"); } if (M2 != 1) { stop("b must be a column vector"); } // Obtain the inverse // writable::doubles_matrix<> Acopy(N1,N1); // for (int i = 0; i < N1; i++) // { // for (int j = 0; j < N1; j++) // { // Acopy(i, j) = A(i, j); // } // } // doubles_matrix<> Ainv = invert_matrix_(Acopy); // I don´t need to create a copy in this case doubles_matrix<> Ainv = invert_matrix_(A); // Multiply Ainv by b writable::doubles_matrix<> x(N1, 1); for (int i = 0; i < N1; i++) { x(i, 0) = 0.0; for (int j = 0; j < N1; j++) { x(i, 0) += Ainv(i, j) * b(j, 0); } } return x;}

`` I also needed aMakevarsfile to indicate the compiler (eitherclangorgcc`) to use the vendored headers. The content is the following.

PKG\_CPPFLAGS = -I../inst/include Because of the vendoring option, I also had to remove the LinkingTo: cpp11 line from DESCRIPTION.

Putting all togetherI created a vscode-install.r file in the root of the project, and I added it to .Rbuildignore. It contains the following lines.

```

cpp_vendor() # run only when updating C++ headersclean_dll()cpp_register()document()install()# load_all()

`` To test that the functions work, after runninginstall()` I solved a simple system.

```

library(cpp11gaussjordan)> A <- matrix(c(2,1,3,-1), nrow = 2, ncol = 2)> b <- matrix(c(7,4), nrow = 2, ncol = 1)> solve_system(A,b) [,1][1,] 3.8[2,] -0.2> solve(A) %% b [,1][1,] 3.8[2,] -0.2 ``` References Cpp11 official documentation * R-lib-cpp11 tag on Stackoverflow Ask a lotBlog posts like this are summaries of what worked for me. I asked so much on Stackoverflow, that they created the “r-lib-cpp11” for my questions. Please ask there, there is probably something I figured how to solve after many hours on Google.

To leave a comment for the author, please follow the link and comment on their blog: Pachá.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cpp11 (R package) vendoring

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The R Consortium, a Linux Foundation project supporting the R Foundation and R community, today announced that Parexel has joined the R Consortium as a Silver Member.

Parexel is a clinical research organization providing Phase I to IV clinical development services. Parexel uses R for a wide range of internal decision-making and regulatory interactions. They have a team of more than 21,000 global professionals collaborating with biopharmaceutical leaders, emerging innovators delivering clinical trials worldwide.

“We are thrilled to welcome Parexel as a member of the R Consortium and connect more closely with the rest of the R Consortium community,” said Joseph Rickert, R Consortium Board Chair, and Posit’s R Community Ambassador. “Parexel brings expertise in using R as a tool to perform analyses as part of clinical trials to their sponsors. We look forward to collaborating with the Parexel team to expand the use of R in drug development.”

“Upskilling our personnel in the use of R, coupled with enhancing our computing environment to incorporate R holistically with our other, more established software options, provides Parexel with additional tools in performing analyses of clinical trial data and opens up new avenues to innovative techniques,” said Michael Cartwright, Associate Biostatistics Director, Parexel.

About The R Consortium

The R Consortium is a 501(c)6 nonprofit organization and Linux Foundation project dedicated to the support and growth of the R user community. The R Consortium provides support to the R Foundation and to the greater R Community for projects that assist R package developers, provide documentation and training, facilitate the growth of the R Community and promote the use of the R language.

About Linux Foundation

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation projects like Linux, Kubernetes, Node.js and more are considered critical to the development of the world’s most important infrastructure. Its development methodology leverages established best practices and addresses the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org

The post Welcome to our newest member Parexel! appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Welcome to our newest member Parexel!

View Details

[This article was first published on R on FOSS Trading, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.quantmod and getSymbols() have been a core part of the R/Finance ecosystem for over 15 years. We want to change some things, but they would break existing code. We can make these changes in the new ‘rfimport’ package instead.

BackgroundThe quantmod package has been a core part of the R/Finance ecosystem for over 15 years. It’s awesome that the package is so popular, but that also comes with responsibility to maintain backward compatibility. Breaking changes may break code used for making business decisions, research, production, blog posts, books, courses, answers on stackoverflow, and much more. We take this responsibility seriously, and do our best to keep functions backward compatible. Sometimes breaking changes are necessary (e.g. bug fixes, changes to external data sources, etc.), but we do our best to make them carefully, with plenty of warning and lead time for users to adjust their code.

MotivationThere are things in quantmod that we want to change, but they would certainly break existing code. No matter how much we’d like to make those changes, we can’t justify breaking a large portion of the code our community has written in the past 15+ years.

We can create a new package instead of making these changes in quantmod. ‘rfimport’ is where we will work on new implementations that improve on the pieces in getSymbols() that we would like to change. This code is extremely alpha. This is the time to provide feedback, suggestions, feature requests, etc. Know that we will break things, maybe without warning. You should consider the API unstable until the 1.0.0 release.

Refresher on how getSymbols() worksBy default getSymbols() creates objects in the environment it’s called from, and it returns the value of the Symbols argument. It’s good practice for functions to avoid changing anything in the user’s environment (this is called having side-effects). It’s better for functions to only return a value, like getSymbols(..., auto.assign = FALSE) does. getSymbols() does not support auto.assign = FALSE for more than one symbol.

getSymbols() also uses functionality that was formerly provided by the archived ‘Defaults’ package. This functionality allows users to set default values for getSymbols() source method arguments (e.g. return.class = "data.frame"). This is also a side-effect because it makes getSymbols() depend on something other than argument values.

getSymbols() specifies the data sources via its src argument, and uses the src argument to determine which source method to use (e.g. getSymbols("SPY", src = "yahoo") will call getSymbols.yahoo("SPY") behind the scenes). This is essentially method dispatch, but done manually rather than using R’s built-in S3 functionality.

What we’ve learned1. We should avoid the side-effect of creating objects in the calling environment. 2. Data sources should use S3 method dispatch, and documentation needs to be easier to find. 3. Stock ticker symbology is a pain and we need a better way to handle it. 4. We need a way to provide functionality like the ‘Defaults’ package did, but without side-effects.

Automatically creating objectsgetSymbols() creates an object for every value in the Symbols argument. This isn’t an issue for a few symbols, but it clutters the environment when there are several hundred symbols. You can load all the symbols into a separate environment, but that’s not a pattern most users are familiar with.

We wanted to remove the ability to load objects into the calling environment, and even created a warning about changing auto.assign = FALSE as the default for getSymbols() and recommending users replace their getSymbols() call with the loadSymbols() function that already exists. But we ultimately decided breaking the community’s code wasn’t worth it.

Automatically creating objects makes it cumbersome to put prices for all symbols into one object. This is a common use case and there are several steps. It should be possible with one or two function calls. Here’s an example.

symbols <- c("SPY", "AAPL")getSymbols(symbols)# Put all the prices into one xts object,prices <- do.call(merge, lapply(symbols, get))# orprices <- do.call(merge, mget(symbols))# Extract only the Close pricesclose\_prices <- Cl(prices)# Remove ".Close" suffix so close\_prices[, "SPY"] workscolnames(close\_prices) <- sub(".Close", "", colnames(close\_prices)) Automatically creating objects also makes passing all the data to another function awkward. It causes users to do things like:

Call getSymbols() in any function that needs data, which may mean the same data is imported multiple times.
Pass the same symbols object to getSymbols() and the other function. Then the other function searches through environments to find the objects with named with those symbols.
Users could put all the data in an environment and use that as an argument to the function, but I haven’t seen many people use this pattern.
Data source methodsDifferent getSymbols() source methods can (or may need to) have different arguments. Ideally the source methods wouldn’t be exported because users shouldn’t call them directly (users should call getSymbols("SPY", src = "yahoo") instead of getSymbols.yahoo("SPY")). It’s hard to find documentation for unexported functions, which means it’s hard to know what arguments are available for each source method.

The source methods are named like S3 methods even though getSymbols() isn’t a generic function and the source methods aren’t actual S3 methods. This has the potential to create odd behavior that would confuse users.

Ticker symbologyThere are two major issues with ticker symbols.
Exchanges and data providers sometimes use different ticker symbols for the same security.
Some ticker symbols are not valid R object names.

Another issue is when the ticker symbol is similar to the name of one of the price columns. This has come up several times with Lowe’s (LOW). The Lo() and OHLC() functions think all of the columns with the ticker symbol in the column name are the low price for the period.

Same security, different tickerThis isn’t getSymbols()’s fault and it’s out of our control, but it could be handled better. Exchange and data source symbology is awful. Identifiers for the same series are often different across exchanges and data providers. For example: the symbol for Berkshire Hathaway B-class shares is “BRK-B” for Yahoo Finance, “BRK/B” for the SIP (Securities Information Processor), “BRK B” for ICE, and probably “BRK.B” somewhere else.

This is a difficult problem and will likely take a lot of effort to get right. Therefore it won’t be a high priority initially.

Invalid R object namesgetSymbols() tries to create objects with valid R names, but only does so for some symbols that aren’t valid R object names. For example, BRK-B, BRK B, and BRK/B aren’t valid R objects names because valid names start with a letter or a dot (.), and can only contain letters, numbers, a dot, or an underscore.

Here are some common examples of ticker symbology woes:

^DJI isn’t a valid R object name because it starts with . So getSymbols() creates an object with the removed. But then you can’t use the code below to put all the prices into one object. Also notice that getSymbols() returns "^DJI" even though it creates an object with a different name.

symbols <- c("^DJI", "BRK-B")getSymbols(symbols)## [1] "^DJI" "BRK-B"prices <- do.call(merge, mget(symbols))## Error: value for '^DJI' not found You have to remove the leading manually. And you have to set fixed = TRUE in the call to sub() because is a special character in regular expressions. Sigh.

prices <- do.call(merge, mget(sub("^", "", symbols, fixed = TRUE))) Recall that BRK-B also isn’t a valid R name because of the -. But it wasn’t an issue in the code above because getSymbols() made an object named BRK-B, not an object with a valid R name. This is confusing for users because they can’t easily access that object (i.e. head(BRK-B) is an error). This is a pervasive issue for several foreign exchanges with tickers that begin with numbers (e.g. 000001.SZ).

Another issue with symbols that aren’t valid R object names is that many R functions will convert column names into valid R object names, including merge.xts(). So you can’t use the input symbol to subset the resulting xts object. Here’s an example:

```

Extract the close prices and remove ".Close" suffixclose_prices <- Cl(prices)colnames(close_prices) <- sub(".Close", "", colnames(close_prices))# Extract the close price for "BRK-B"close_prices[, "BRK-B"]## Error in `[.xts`(close_prices, , "BRK-B") : subscript out of boundscolnames(close_prices)## [1] "DJI" "BRK.B"

``setSymbolLookup()exists to help with things like this, but it’s another function users have to learn to use and my experience is that most users don’t know aboutsetSymbolLookup(). I just had to look at the source to figure out how to use it to makegetSymbols()return a valid R object for"BRK-B"`.

setSymbolLookup(BRK.B = list(name = "BRK-B", src = "yahoo")) If I have to look at the source code to figure out how to do this, users don’t have a chance. You may think, “but you could document how to do this”, but writing documentation isn’t fun. And who reads the documentation anyway?

‘Defaults’ functionalityThe ‘Defaults’ functionality in quantmod comes from the archived ‘Defaults’ package. This functionality allows users to set new default argument values to any getSymbols() source function. This is helpful because it makes importing easier. But it means getSymbols() relies on something other than its parameter values, and it’s good practice to avoid side-effects like this.

This gave users the ability to set preferences like return class, periodicity (e.g. hourly, daily, monthly), connection settings (e.g. credentials, API keys).

‘rfimport’ design and featuresThe design of ‘rfimport’ is influenced by the DBI package, which provides a set of generic ‘database interface’ functions. Users create connection objects by creating a ‘driver’ object for the specific database and passing that to dbConnect(). Then you pass that connection object to the other DBI functions. For example, to query an execute a statement for a PostgreSQL database:

library(RPostgreSQL)driver <- PostgreSQL()conn <- DBI::dbConnect(driver)student\_count <- DBI::dbGetQuery(conn, "select count(*) from students") The ‘rfimport’ sym_yahoo() function corresponds to the PostgreSQL() function in the example above. And the import() function pulls the data like DBI::dbGetQuery(). For example:

library(rfimport)# The sym\_* functions are a combination of the# driver, connection, and query in DBIsyms <- sym\_yahoo("SPY")# Import some data from Yahoo Financespy <- import(syms) Symbol specificationThe package introduces a new virtual S3 class "symbol_spec" as the basis for creating sub-classes that hold all necessary information to connect to a data source. This virtual class allows users to combine symbols from different data sources into a single vector. For example: import_collection(c(sym_yahoo("SPY"), sym_tiingo("DIA"))) will import data for “SPY” from Yahoo Finance and data for “DIA” from Tiingo.

Each data source will have its own symbol_spec constructor. The constructor will have an argument for the vector of symbols and other arguments for all the other data source connection settings. It will return an object that inherits from the new virtual symbol_spec For example sym_yahoo() will return a c("yahoo", "symbol_spec") class vector.

The help page for the symbol spec constructors can also document the import methods that the data source supports. So help("sym_yahoo") would also contain information about import.yahoo() and import_collection.yahoo(). That way, users don’t need to know the name of the data source method in order to find its documentation.

Ticker symbologyThe package would standardize how index tickers are specified. One possibility is to prefix the ticker with an i or i_ (e.g. iDJI or i_DJI).

It would also standardize how to specify share classes, warrants, preferred, etc. One possibility is to use an underscore to identify share classes, a lowercase ‘w’ for warrants, and a lowercase ‘p’ for preferred. For example, BRK_B for Berkshire Hathaway B shares, FOOw for warrants, BARp for preferred. We could also include a translation table and/or function. This would take a lot of effort to do correctly.

An easier alternative would be creating a way to map source symbols to user-defined values. It makes the most sense to do this is the sym_<source>() constructor. But how should the mapping be specified? Some possibilities:

sym_yahoo(BRK.B = list(symbol = "BRK-B"))
sym_yahoo(c(BRK.B = "BRK-B", "DIA"))
sym_yahoo(c("BRK.B", "DIA"), sym_db = list(BRK.B = "BRK-B"))

Generic import functionsThe package will have generic functions import() and import_collection() to dispatch on symbol_spec sub-classes. import() only handles a single symbol and returns one xts object. import_collection() will return a list of xts objects for one or more symbols.

Other generic import functions may be added in the future. It may make sense to include generic import functions that return specific types of data. For example: import_ohlc() for open, high, low, close, (adjusted, volume), and import_bbo() for best bid and offer.

The generics will have a symbol_spec, dates, periodicity, and ... arguments.

dates can be either an ISO 8601 date interval (e.g. dates = "2021-01-01/2021-12-31") or a two-element vector with the start and end dates (e.g. dates = c("2021-01-01", "2021-12-31")). The vector can be Date, POSIXct, or a character that is coercible to one of those two classes.

For example:

```

one symbol returned as an xts objectspy <- import(sym_yahoo("SPY"), dates = "2021/2022")# two symbols returned as a list of xts objectsstocks <- sym_tiingo(c("AAPL", "NFLX")) |> import_collection(dates = "2021-03-01/2022-11-31")

`` Theperiodicityargument specifies the interval between data points (e.g. daily, monthly, 15-minute). The data source determines the possible periodicity values, so the data source method is responsible for ensuring the requested periodicity value is available from the data source. ‘rfimport’ will provide a standard way to specify the periodicity values. Then the source methods can translate those values into the value source needs. For example, one data source may use “monthly” for monthly data and another may use “months”. Users would setperiodicity = "months"` for either source and the source method would translate the value to “monthly”.

Data source methodsEach data source will have a S3 method for the relevant import generics, rather than a src argument like getSymbols(). Calling import(sym_yahoo("SPY")) will call the corresponding import.yahoo() method to import data from Yahoo Finance. import(sym_tiingo("DIA")) will call import.tiingo() to import data from Tiingo.

Returned dataThe built-in data source methods will automatically include dividends and splits (when available) for daily OHLCV data. They will be included as attributes on the returned OHLCV object. This will allow users to switch between adjusted and unadjusted prices without having to re-download the data.

The built-in data sources methods also will not include the series symbol in the OHLCV column names like getSymbols() currently does. It may make sense to include an attribute with the “source symbol” and the “R symbol” on the returned xts object (e.g. src_symbol = "^DJI" and r_symbol = "iDJI"). Then that attribute can be used later as part of the column names.

Providing ‘Defaults’ functionalityThough we want to avoid side-effects, we probably want to provide a way to set credentials so they do not have to be provided for every import call.

We could provide this functionality in a pure way by creating an options object that holds a list of values. Users would create this object once and pass it to the relevant ‘rfimport’ function (either sym_<source>() or import()). The default options could be created by a function like sym_<source>_options(). This would be similar to the ‘control’ arguments to many optimization routines (e.g. DEoptim.control()).

Open questions and considerationsHow should we specify the class of the returned object?* Set it via a return_class argument in the symbol_spec constructor

+ PRO: each source is likely to have a specific data structure, and it wouldn’t require creating a generic `import` function for each return type.
+ CON: allows the potential for one call to an `import_*_collection()` function to return a list of heterogeneous objects.

Set it via a return_class argument in the import method
- PRO: the method would return a list of objects that are all the same class.
- CON: the generic and/or the default import method would need a return_class argument.
- Create a new generic import functions for each return class
- PRO: makes it clear what the import function returns.
- CON: namespace clutter, don’t want generics for every class. Possibly provide generics for most widely used non-xts classes: data.frame, data.table, tibble, tsibble.
- The symbol specification can store a function that controls what data is returned. This doesn’t seem appealing because it adds complexity and the user could call that function after the data is returned. For example: sym_yahoo("SPY", return_func = as.data.table).

How can we make it easier to manipulate results?The most common use case is making a wide xts object with close prices from a list of xts objects. This currently requires several steps that are likely unfamiliar to most users. It should be possible with one or two function calls. We can consider Garrett See’s qmao package for inspiration. For example, use ‘price frames’ to replace do.call(merge, list_of_xts_objects).

There are lots of other common manipulations, like aggregating to a higher periodicity or applying a function to many symbols' data. The import functions will return something list-like, so users can use lapply() to apply any other function to each series.

I need your help!I don’t want to do this in a vacuum. Please try the new package, provide feedback, suggestions, feature requests, and help clarify documentation.

I need to know how you’re using getSymbols() and how you would use the new package. I’m not omniscient, so your feedback will be extremely valuable!

If you love using my open-source work (e.g. quantmod, TTR, xts, IBrokers, microbenchmark, blotter, quantstrat, etc.), you can give back by sponsoring me on GitHub. I truly appreciate anything you’re willing and able to give!

To leave a comment for the author, please follow the link and comment on their blog: R on FOSS Trading.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: getSymbols Rebooted

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionTo effectively extract insights and communicate findings, you need powerful tools that simplify the process and present data in an engaging manner. If you’re a programmer with a penchant for data analysis, you’re in luck! The latest version of {TidyDensity}, the popular R package, has just been released, bringing you exciting new features and enhancements. In this blog post, we’ll explore the highlights of TidyDensity 1.2.5 and why you should download it today.

New Feature: Introducing util_burr_param_estimate()TidyDensity 1.2.5 introduces a new function: util_burr_param_estimate(). This function enables you to estimate parameters using the Burr distribution, expanding the possibilities of your data analysis. Whether you’re working with survival analysis, reliability modeling, or extreme value theory, util_burr_param_estimate() equips you with a powerful tool to tackle complex scenarios with ease. Say goodbye to manual calculations and embrace the simplicity and accuracy of TidyDensity.

Minor Fixes and Improvements for Enhanced WorkflowIn addition to the groundbreaking new feature, TidyDensity 1.2.5 addresses user feedback and provides several minor fixes and improvements. Let’s take a look at a couple of them:

Improved Parameter Rounding: With the new version, you now have more control over the rounding of parameter estimates. The updated function tidy_distribution_comparison() includes a parameter called .round_to_place, allowing you to precisely control the rounding behavior of the parameter estimates passed to their corresponding distribution parameters. This enhancement ensures that your analysis remains accurate and aligned with your specific requirements.

Why Upgrade to TidyDensity 1.2.5?* Stay Ahead of the Curve: The world of data analysis is constantly evolving, and staying up to date with the latest tools and features is crucial to remain competitive. TidyDensity 1.2.5 empowers you with advanced capabilities, enabling you to analyze and visualize data more effectively than ever before. * Simplify Complex Analysis: With the new util_burr_param_estimate() function, TidyDensity 1.2.5 simplifies complex data analysis tasks. Whether you’re a seasoned data scientist or a beginner, this feature allows you to explore a wider range of statistical distributions and unlock deeper insights from your data. * Fine-Tuned Precision: The improved parameter rounding in tidy_distribution_comparison() ensures that your analysis is not only powerful but also precise. This level of control over rounding provides you with the flexibility to align your analysis with your specific requirements.

ConclusionTidyDensity 1.2.5 is a significant update that brings you exciting new features and enhancements. From the introduction of util_burr_param_estimate() to the fine-tuned parameter rounding and polished visuals, this version is designed to empower you in your data analysis journey. By downloading TidyDensity1.2.5, you can stay at the forefront of data analysis, simplify complex tasks, and elevate the precision and user experience of your projects. Upgrade to TidyDensity 1.2.5 today!.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Update to {TidyDensity}

View Details

[This article was first published on R – Stat Tech, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This document describes an upate to the behaviour of Porter-Duff compositing operators in R graphics.

Paul Murrell, Thomas Lin Pedersen, and Panagiotis Skintzos.

Download

To leave a comment for the author, please follow the link and comment on their blog: R – Stat Tech.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 2023-02 Porter-Duff Compositing Operators in R Graphics

View Details

[This article was first published on R/exams, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Tutorial for creating knowledge quiz questions with the help of ChatGPT in multiple- or single-choice format for R/exams.

Guest post by Andreas Steinmayr (Universität Innsbruck, Faculty of Economics and Statistics).

IdeaLarge language models such as ChatGPT have received much attention in recent months. There has been a lot of praise, but also criticism. One of the issues raised was that ChatGPT makes up answers to questions, and does so in a way that the answers appear plausible to readers not familiar with the subject. This might be harmful behavior in many instances, especially in academic settings, but there might also be an application: Creating multiple-choice and single-choice questions for exams.

Multiple-choice and single-choice questions are commonly used in assessments, exams, and surveys. They are popular because they can be quickly and efficiently graded by computers or instructors for many test-takers. However, creating high-quality questions in multiple-choice or single-choice format is challenging and time-consuming for various reasons:

Questions must be unambiguous, and the false answers must be plausible enough to make the question challenging.
Test-takers should have to distinguish right from wrong answers based on the substantive content of the answers, and not from the way how answers are formulated.
Questions need to be updated regularly (e.g., every semester) if test-takers can collect and distribute them to future test-takers.

I am currently teaching a class in introductory economics, and we have regular short take-home assignments with single-choice questions using some random variation in the selection of questions and answer alternatives. The assignments are prepared with R/exams and administered in our university’s learning management system (OpenOlat).

Recently, I have started to use ChatGPT to create those questions and found it a useful – yet imperfect – tool. Below, I show how I create a question on the income elasticity of demand with ChatGPT and what further steps I take to turn it into an R/exams question with some shuffling and randomization.

Naturally, the illustrative example I use comes from economics but the same strategy can also be applied for creating knowledge quiz questions in other fields.

Step 1: Elicit the answer list from ChatGPTIn general, I don’t ask ChatGPT to come up with the question but provide the question myself. But that’s pretty much it. What I ask for is a list of correct and incorrect answers.

Note that I ask for more than one correct answer, despite creating a single-choice question, because I can use R/exams’ built-in shuffling to select a different correct answer for different test-takers. Similarly, I ask for more incorrect answers than I need for each test-taker in order to apply some subsampling and shuffling. The same strategy can also be leveraged for multiple-choice questions.

Prompt: Create seven correct and seven incorrect answers for an exam in introductory economics based on the following question: Which of the following statements about income elasticity of demand are correct?

Correct answers:

A) If the income elasticity of demand is positive, it is called a normal good.
B) Income elasticity of demand measures the extent to which demand for a good responds to a change in income.
C) Luxury goods typically have a higher income elasticity of demand than necessities.
D) If the income elasticity of demand is negative, it is called an inferior good.
E) Income elasticity of demand can vary between zero and infinity.
F) A Giffen good is a special case of an inferior good in which demand increases as income decreases.
G) An income elasticity of demand of one means that demand for the good increases proportionally to the change in income.

Incorrect answers:

H) Income elasticity of demand measures the extent to which demand for a good responds to a change in price.
I) If the income elasticity of demand is positive, it is called an inferior good.
J) Income elasticity of demand is the same for all goods.
K) If the income elasticity of demand is zero, it is called a luxury good.
L) If the income elasticity of demand is greater than one, it is called a normal good.
M) Income elasticity of demand is the same for all consumers.
N) Income elasticity of demand is independent of the price of the good.

Step 2: Review answers, potentially modify the promptAbove, I have color-coded the suggested answers. Green means I am okay with the answer being categorized as correct/incorrect. Yellow means it’s not necessarily incorrect but imprecise or ill-crafted. Red means definitely incorrect.

For answer C), ChatGPT seems to conflate the term “luxury good” in everyday language and economics, where luxury goods are defined by having an income elasticity larger than one.

Answer E) is incorrect, as the income elasticity can be negative. But this will make for a nice answer in the incorrect list.

Answer F) is incorrect because a Giffen good is a good for which demand increases if its price increases.

I decided to ask for more incorrect answers, giving ChatGPT a bit more guidance:

Prompt: Add additional incorrect answer options that include the terms “normal goods” and “Giffen goods.”

Additional incorrect answers:

O) If the income elasticity of demand for normal goods is positive, it is called an inferior good.
P) Giffen goods always have a positive income elasticity of demand.
Q) The income elasticity of demand for Giffen goods cannot be greater than one.
R) If the income elasticity of demand for normal goods is negative, it is called a superior good.
S) If the income elasticity of demand for Giffen goods is negative, demand increases with an increase in income.
T) Giffen goods are luxury goods for which demand decreases as income increases.
U) Normal goods are goods preferred by a specific segment of the population.

I decided to use the two answers highlighted in green and added them to the list.

Step 3: Edit manuallyIn the next step, I edited the answers I was not fully happy with and added my own answers to the list:

Correct answers:

If the income elasticity of demand is positive, the good is said to be a normal good.
The income elasticity of demand indicates how strongly the demand for a good reacts to a change in income.
If the income decreases by 2% and thus the demand for a good decreases by 3%, it is called a luxury good.
If the income elasticity of demand is negative, one speaks of an inferior good.
The income elasticity of demand can be positive or negative.
A Giffen good is a special case of an inferior good where demand increases if the price of the good increases.
An income elasticity of demand equal to one means that the demand for the good increases proportional to the change in income.

Incorrect answers:

The income elasticity of demand indicates how strongly the demand for a good responds to a change of the price.
If the income elasticity of demand is positive, the good is said to be inferior.
The income elasticity of demand is the same for all goods.
If the income elasticity of demand is zero, the good is called a luxury good.
If the income elasticity of demand is greater than one, the good is called a normal good.
The income elasticity of demand is the same for all consumers.
The income elasticity of demand is independent of the good’s price.
The income elasticity of demand can range between zero and infinity.
Giffen goods are luxury goods for which the demand decreases as income increases.
Common goods are goods that are purchased preferentially by a certain part of the population.

Step 4: Implement in R/exams formatFinally, let’s turn this into a dynamic R/exams exercise in both R/Markdown (Rmd) and R/LaTeX (Rnw) format. The first version of the exercise, linked in the table below, simply lists all answer alternatives from Step 3, flags them as correct/incorrect in the exsolution meta-information, specifies extype as schoice for single-choice, and sets exshuffle to 5 so that one correct and 4 incorrect answer alternatives are randomly selected.

The second version in the table below is virtually identical to the first one, but extype is now set to mchoice for multiple-choice. In this case five answer alternatives are chosen randomly with at least one correct and at least one incorrect alternative.

| # | Exercise templates | Invert? | Type | Description | | --- | --- | --- | --- | --- | | 1 | income_elasticity1.Rmd income_elasticity1.Rnw | No | schoice | Select single correct answer from shuffled subset of five answer alternatives. | | 2 | income_elasticity2.Rmd income_elasticity2.Rnw | No | mchoice | As in #1 but with one or more correct answer alternatives. | | 3 | income_elasticity3.Rmd income_elasticity3.Rnw | Yes | schoice | As in #1 but randomly vary between selection of correct or incorrect answer. | | 4 | income_elasticity4.Rmd income_elasticity4.Rnw | Yes | mchoice | As in #3 but with one or more (in)correct answer alternatives. |

In the third and fourth version of the exercise, I add some further dynamic elements to the question in order to create some more random variation and make cheating harder. Specifically, I randomly vary between the original task (Which of the statements is/are correct?) and the inverted task (Which of statements is/are not correct?). Moreover, some of the answer alternatives are randomly modified (increase vs. decrease; different percentages) without changing the nature of the statement (correct vs. incorrect).

The fourth version again differs from the third only by using the mchoice (instead of schoice) type and “are” (instead of “is”) in the question.

For more guidance on using the schoice and mchoice exercise types in R/exams see the corresponding YouTube tutorial which starts out from a knowledge quiz question about capitals.

ConclusionJust like it’s not a good idea for students to rely on ChatGPT to answer exam questions, it’s not a good idea to entirely rely on it to create them. Correct answers might be phrased incorrectly, vice versa, or you might find answers to be ill-crafted.

But what ChatGPT is very useful for is to give you a set of potential answers, especially incorrect ones, that sound plausible. You can then use this list, use only selected answer options, or modify others. This is what I find most useful at the moment, as it can take a lot of time and cognitive energy to come up with high-quality answer lists from an empty slate.

To leave a comment for the author, please follow the link and comment on their blog: R/exams.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Using ChatGPT for Creating Multiple- and Single-Choice R/exams Questions

View Details

[This article was first published on Pachá, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

MotivationA large part of my research interest requires to estimate computationally intensive models, such asthe General Equilibrium Poisson Pseudo Maximum Likelihood (GEPPML) estimator derived from the equilibrium conditionsintroduced by Anderson and Van Wincoop (2004) for estimation and inference.

The GEPPML estimator is a computationally intensive estimator that requires to solve a system of non-linear equations,and for this task we might be better-off by using a compiled language such as C++. The good news is that we can use C++ codewithin R and Python, and this blog post is about using C++ functions from R.

Also, I do not pretend to be an expert on C++ or debate if R is better than Python. I use both from Visual Studio Code.I do want to share my experience on how to use C++ code within R.

Honest disclaimerThis blog post is a summary of what worked after hours of fails for my future self. I hope it helps youtoo.

I am a Statistician and Political Scientist, not a Computer Scientist!

SetupBecause I have been already learning C++ version 11, I decided to install llvm-11 on my laptopthat has Linux Mint installed, and which is based on Ubuntu 22.04.

Ubuntu and its derived distributions use gcc as the default C++ compiler, and clang is not installed by default. Differentresources mention that clang provides more informative error messages when the compilation fails and when we debug code.

Counting on informative error messages is highly useful resource when we are learning C++ or when our code is failing in twodifferent ways, being one that it does not compile, and the other that it compiles but then when we call a function fromRStudio (or VSCode) it crashes the R session.

I installed the R packages cpp11 and usethis.

install.packages(c("cpp11", "usethis")) I created a file ~/.Rprofile containing the following lines.

library(devtools)library(usethis)library(cpp11) I run nano ~/.R/Makevars from bash and then saved with CTRL+O+ENTER and closed it with CTRL+X. It is the same as creating it with the text editor from Gnome or any other desktop environment.

Now forget about devtools::install(). After reopening your editor, every time you use RStudio (or VSCode) you just call install(), and the same applies to usethis::use_*() and cpp11::cpp_*() functions.

To install llvm-11 I downloaded the installation script from the official LLVM repository, and it also installed clang-11.

cd Downloadswget https://apt.llvm.org/llvm.shchmod +x llvm.shsudo ./llvm.sh 11 Up to this point I still had the following error messages when compiling C++ code.

fatal error: 'cstdio' file not foundfatal error: 'vector' file not foundcannot find -lc++abi: No such file or directory I had to install additional packages. This took me a few hours searching on the Internet until I figured it out.

sudo apt install g++-11 libc++-11-dev libc++abi-11-dev To be sure that the install() function in R uses the correct version of clang++ I created the ~/.R/Makevars file. The contents of the file are the following.

CLANGVER=-11CLANGLIB=-stdlib=libc++CXX=$(CCACHE) clang++$(CLANGVER) $(CLANGLIB)CXX11=$(CCACHE) clang++$(CLANGVER) $(CLANGLIB)CC=$(CCACHE) clang$(CLANGVER)SHLIB\_CXXLD=clang++$(CLANGVER) $(CLANGLIB)CXXFLAGS=-Wall -O0 -pedanticCXX11FLAGS=-Wall -O0 -pedantic For both CXXFLAGS and CXX11FLAGS I am using -O0 to avoid optimization, which is useful for debugging. After the code is working, I can change it to -O3 to optimize the compiled code.

If later on I need to compile with gcc, I can open ~/.R/Makevars, comment all the lines, restart RStudio or VSCode, and run install() again.

If you close RStudio (or VSCode) and open it again, you can check that the changes were implemented by running pkgbuild::check_build_tools(debug = TRUE), which should return the following output.

Trying to compile a simple C fileRunning /usr/lib/R/bin/R CMD SHLIB foo.cusing C compiler: ‘Ubuntu clang version 11.1.0-6’clang-11 -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-JhpCKt/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c foo.c -o foo.oclang-11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o foo.so foo.o -L/usr/lib/R/lib -lR If I were using gcc, the output would have been as in the following lines.

Trying to compile a simple C fileRunning /usr/lib/R/bin/R CMD SHLIB foo.cusing C compiler: ‘gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0’gcc -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-JhpCKt/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c foo.c -o foo.ogcc -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o foo.so foo.o -L/usr/lib/R/lib -lR The key here is that when I use clang the lines start with clang, not with gcc.

Creating a dummy packageFrom RStudio (or VSCode) we can create a new package by running create_package("~/cpp11dummypackage"). This will create a new folder with the name cpp11dummypackage. Then I run use_cpp11() to add the required files to use C++ code within R.

Then I run use_r("cpp11dummypackage-package") to create a new R script file with the name cpp11dummypackage-package.R within the R folder, and added the following code to it.

```

' @useDynLib cpp11dummypackage, .registration = TRUENULL

`` Theusethisskeleton also created the filesrc/code.cpp` for us. I added a simple function to transpose a matrix to it, by replacing the file contents by the following lines.

```

include #include using namespace cpp11; [[cpp11::register]] doubles_matrix<> Xt(doubles_matrix<> X){ int NX = X.nrow(); int MX = X.ncol(); writable::doubles_matrix<> R(MX, NX); for (int i = 0; i < MX; i++) { for (int j = 0; j < NX; j++) { R(i, j) = X(j, i); } } return R;}

`` In order to export the function, I added the following lines tocpp11dummypackage-package.R`.

```

' Transpose a matrix#' @export#' @rdname Xt#' @param X numeric matrix#' @return numeric matrix#' @examples#' set.seed(1234)#' X <- matrix(rnorm(4), nrow = 2, ncol = 2)#' X#' cpp11_Xt(X)cpp11_Xt <- function(X) { Xt(X)}

`` I tested the functions after runningcpp11_register()andload_all()`.

```

set.seed(1234)> X <- matrix(rnorm(4), nrow = 2, ncol = 2)> X [,1] [,2][1,] -1.2070657 1.084441[2,] 0.2774292 -2.345698> cpp11_Xt(X) [,1] [,2][1,] -1.207066 0.2774292[2,] 1.084441 -2.3456977 `` If I would have passed1:4instead ofrnorm(4)tomatrix()`, I would have obtained the following error message.

```

cpp11_Xt(X)Error: Invalid input type, expected 'double' actual 'integer' `` This is because I declared the function to accept adoubles_matrix<>as input, and not anintegers_matrix<>`.

To install the recently created package, I run the following lines in the R console.

clean\_dll()cpp\_register()document()install() Debugging the packageIn order to access debugging symbols, I created a new Makevars file within the src folder, and added the following lines.

CXX\_STD = CXX11PKG\_CPPFLAGS = -UDEBUG -g Then I reinstalled the package compiled with debugging symbols, and in bash I run R -d lldb-11. From there I could follow this excellent guide to debug R and C++ code.

A more complex exampleI created a package containing a set of simple functions to obtain the Ordinary Least Squares (OLS) estimator by calling a C++ function that calls other C++ functions. My approach was to create one function per step, which meant to create one function to obtain (X^tX), another for ((X^tX)^{-1}) which consisted in implementing the Gauss-Jordan method to invert a matrix, another for (X^tY) and then call each of those functions to obtain (\hat{\beta} = (X^tX)^{-1}(X^tY)).

This implementation is extremely naive, but it is enough to show how to use C++ code within R. Please see it from my GitHub profile.

In any case, it would be extremely hard to beat the performance of the lm() function in R, which has some internals written in C, and how computationally robust lm() is means another feature that is hard to beat.

References Debugging in R with a single command * Debugging an R package with C++ * Clang++ missing C++ header? * How to I tell RStudio not to ignore the indication to use clang in Makevars? * R’s Makevars: PKG_CXXFLAGS vs. PKG_CXX11FLAGS * Debugging memory errors with valgrind and gdb * A Deep Dive Into How R Fits a Linear Model To leave a comment for the author, please follow the link and comment on their blog: Pachá*.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A step by step guide to write an R package that uses C++ code (Ubuntu)

View Details

[This article was first published on Pachá, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Also, I do not pretend to be an expert on C++ or debate if R is better than Python. I use both from Visual Studio Code.I do want to share my experience on how to use C++ code within R.

Honest disclaimerThis blog post is a summary of what worked after hours of fails for my future self. I hope it helps youtoo.

I am a Statistician and Political Scientist, not a Computer Scientist!

SetupBecause I have been already learning C++ version 11, I decided to install llvm-11 on my laptopthat has Linux Mint installed, and which is based on Ubuntu 22.04.

Ubuntu and its derived distributions use gcc as the default compiler, and clang is not installed by default. Differentresources mention that clang provides more informative error messages when the compilation fails and when we debug code.

We need to install the R packages cpp11 and usethis.

install.packages(c("cpp11", "usethis")) To install llvm-11 I downloaded the installation script from the official LLVM repository, and it also installed clang-11.

cd Downloadswget https://apt.llvm.org/llvm.shchmod +x llvm.shsudo ./llvm.sh 11 In order to avoid errors of the form fatal error: 'cstdio' file not found when we compile C++ code,we need to install additional packages. This took me a few hours searching on the Internet until I figured it out.

sudo apt install g++-11 libc++-11-dev libc++abi-11-dev To be sure that devtools::install() uses the correct version of clang++ I created the ~/.R/Makevars file,by running mkdir ~/.R && nano ~/.R/Makevars from bash. The contents of the file are the following.

CLANGVER=-11CLANGLIB=-stdlib=libc++CXX=$(CCACHE) clang++$(CLANGVER) $(CLANGLIB)CXX11=$(CCACHE) clang++$(CLANGVER) $(CLANGLIB)CC=$(CCACHE) clang$(CLANGVER)SHLIB\_CXXLD=clang++$(CLANGVER) $(CLANGLIB)CXXFLAGS=-Wall -O0 -pedanticCXX11FLAGS=-Wall -O0 -pedantic Note that for both CXXFLAGS and CXX11FLAGS I am using -O0 to avoid optimization, which is useful for debugging. After the code is working, we can change it to -O3 to optimize the compiled code.

If later on we need to compile with gcc again, we can open the file and comment all the lines.

If you close RStudio (or VSCode) and open it again, you can check that the changes were implemented by running pkgbuild::check_build_tools(debug = TRUE), which should return the following output.

Trying to compile a simple C fileRunning /usr/lib/R/bin/R CMD SHLIB foo.cusing C compiler: ‘Ubuntu clang version 11.1.0-6’clang-11 -I"/usr/share/R/include" -DNDEBUG -fpic -g -O2 -ffile-prefix-map=/build/r-base-JhpCKt/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -c foo.c -o foo.oclang-11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o foo.so foo.o -L/usr/lib/R/lib -lR Creating a dummy packageFrom RStudio (or VSCode) we can create a new package by running usethis::create_package("~/cpp11dummypackage"). This will create a new folder with the name cpp11dummypackage.Then run usethis::use_cpp11() to add the required files to use C++ code within R.

At this point you are almost ready to start working and be productive, but there is one more thing to do.Run usethis::use_r("cpp11dummypackage-package") to create a new R script file with the name cpp11dummypackage-package.R within the R folder, and add the following code to it.

```

' @useDynLib cpp11dummypackage, .registration = TRUENULL

`` Theusethisskeleton also created the filesrc/code.cpp` for us. We can add a simple function to transpose a matrix to it, by replacing the file contents by the following lines.

```

include #include using namespace cpp11; [[cpp11::register]] doubles_matrix<> Xt(doubles_matrix<> X){ int NX = X.nrow(); int MX = X.ncol(); writable::doubles_matrix<> R(MX, NX); for (int i = 0; i < MX; i++) { for (int j = 0; j < NX; j++) { R(i, j) = X(j, i); } } return R;}

`` In order to export the function, we need to add the following lines tocpp11dummypackage-package.R`.

```

' Transpose a matrix#' @export#' @rdname Xt#' @param X numeric matrix#' @return numeric matrix#' @examples#' set.seed(1234)#' X <- matrix(rnorm(4), nrow = 2, ncol = 2)#' X#' cpp11_Xt(X)cpp11_Xt <- function(X) { Xt(X)}

`` Withcpp11::cpp11_register()anddevtools::load_all()` we can test our function.

```

set.seed(1234)> X <- matrix(rnorm(4), nrow = 2, ncol = 2)> X [,1] [,2][1,] -1.2070657 1.084441[2,] 0.2774292 -2.345698> cpp11_Xt(X) [,1] [,2][1,] -1.207066 0.2774292[2,] 1.084441 -2.3456977 `` If we would have passed1:4instead ofrnorm(4)tomatrix()`, we would have obtained the following error message.

```

cpp11_Xt(X)Error: Invalid input type, expected 'double' actual 'integer' `` This is because we declared the function to accept adoubles_matrix<>as input, and not anintegers_matrix<>`.

To install the recently created package, run the following lines in the R console.

devtools::clean\_dll()cpp11::cpp\_register()devtools::document()devtools::install() Debugging the packageIn order to access debugging symbols, we need to create a new Makevars file within the src folder, and adding the following lines.

CXX\_STD = CXX11PKG\_CPPFLAGS = -UDEBUG -g Then we need to reinstall our package, and in bash we can run R -d lldb-11 and follow this excellent guide to debug R and C++ code.

A more complex exampleI created a package containing a set of simple functions, including the Gauss-Jordan method to invert a matrix, thatallows the user to obtain the Ordinary Least Squares (OLS) estimator by calling a C++ function that calls other C++ functions.This implementation is extremely naive, but it is enough to show how to use C++ code within R. Please see it from myGitHub profile.

A good challenge would be to implement the QR decomposition used by the lm() function in R and use it to obtain the OLS estimator in C++.This would require some effort, but here you can find a good starting point.In any case, it would be extremely hard to beat the performance of the lm() function in R, which has some internals written in C,and how computationally robust lm() is means another feature that is hard to beat.

BonusMaybe create a file ~/.Rprofile containing the following lines.

library(devtools)library(usethis)library(cpp11) Then forget about devtools::, cpp11:: and usethis:: and use clean_dll(), cpp_register(), document(), install(), create_package(), use_cpp11() and use_r() from now on every time you open RStudio (or VSCode).

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Using cpp11 (R package) and llvm on Ubuntu

View Details

[This article was first published on A blog from Sydney, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Secca come una fucilata: “Ma tu non hai paura che ChatGPT ti rubi il posto di lavoro?”

Sento una lievissima fitta agli addominali, è colpa evidentemente della panca bassa che abbiamo fatto stamattina a yoga, ma significa anche che la domanda mi ha fatto contrarre i muscoli. Giunge quasi alla fine di una conversazione interessante su ChatGPT, su come può cambiare lo scenario del lavoro, sui mestieri che si estingueranno e sulle professionalità che resteranno in sella a questo cavallo imbizzarrito e artificiale di cui tutto parlano. Respiro un attimo, ci penso su, un altro respiro e poi: “Lorenza, no, non credo che ci rimetterò il posto…” ChatGPT non è preciso, lo hanno beccato a contraddirsi molte volte, ma questo è tipico del linguaggio naturale, di come comunicano gli umani, tutto un ripetere cose lievemente diverse, infarcendo di contesto, di cose implicite e incerte, dando per scontato che l’interlocutore capisca e riempia i buchi del discorso. È ovvio, se ci pensate, che sui dettagli ci contraddiciamo spesso, non fosse altro perché tagliamo corto e non ripetiamo tutto quello che, forse, logicamente dovremmo. Tiriamo via dritto, e le scorciatoie possono tralasciare i dettagli, sono fatte apposta. La maggior parte delle nostre imprecisioni non è grave, aiuta la sintesi e nessuno te le rinfaccia (certo, ci sono anche i rompicoglioni…). I politici sono casi esemplari: cambiano idea sin verguenza come banderuole, dicono anche fesserie monumentali e poi si scusano, a base di “non mi avete capito”, “non prendetemi alla lettera”, “avete estrapolato frasi senza contesto” (appunto, è semplre colpa del contesto!) e, per farla breve, li votiamo sempre nonostante le imprecisioni. “Ma io sono un matematico e la mia materia è una delle poche, pochissime, in cui la precisione estrema è l’essenza e la bellezza delle cose, non un’appendice cosmetica”. E continuo dicendo che non ho paura di ChatGPT nemmeno nel mio secondo lavoro, quello di insegnante: “Non mi può sostituire perché educare significa tirare fuori dagli studenti quello che sono, serve capacità di coinvolgerli, di motivarli, di accendere un fuoco. Non si tratta solo di passare contenuti, ci vuole sempre un supplemento d’anima”. ChatGPT non fa nulla di tutto questo, è uno zero nei confronti di un professore che ha letteralmente il potere di cambiarti la vita (se è bravo, d’accordo; ci sono anche quelli che ti fan venire il latte alle ginocchia!)Eppure, la domanda m’inquieta. Lorenza scrive su un giornale locale, nella sua voce serpeggia curiosità, ammette di essere felice perché può scrivere di quel che le interessa (cosa rara, wow!) Fa le domande giuste, non cerca risposte semplici, memorizza la mia difficoltà a darle visioni e previsioni su quel che sarà, sorride all’altro lato della cornetta quando sparo che secondo me in 5 anni ChatGPT potrebbe avere già sostituito tutta una serie di lavoratori impegnati in mansioni intellettuali a bassa intensità, dato che altri intervistati le hanno detto che serviranno 10 o trent’anni.La nostra bella conversazione finisce ma io continuo a mulinare cattivi pensieri: “ChatGPT, che diamine vuoi da me e dai miei studenti?”, mi chiedo. Decido di prendere un problema da compito, un esercizio standard per i miei studenti di “Computational tools”, e di darlo in pasto all’AI: risolvi l’equazione x^3+x+1=0. A suo modo, non è un problema difficile, “x alla terza più x più uno uguale a 0”, ma non c’è una formula risolutiva semplice come per le equazioni di secondo grado, quella che a un certo punto della vita abbiamo imparato tutti (sorvolo sul fatto che ho detto un’imprecisione… Cardano abbi pietà ma non finirei più!)Ogni computer dovrebbere risolvere quell’equazione di terzo grado “facilmente” e produrre il risultato. Vuoi vedere che da ora in poi i miei studenti mi possono fregare e trovare quel numero semplicemente chiedendo a ChatGPT?

solve the problem x^3+x+1=0ChatGPT: Unfortunately, the equation x^3 + x + 1 = 0 does not have an algebraic solution that can be expressed in terms of elementary functions. This is because the equation is a cubic equation, and there is no general formula for solving cubic equations algebraically like there is for quadratic equations.However, the equation can be solved numerically using numerical methods, such as the Newton-Raphson method or the bisection method. These methods involve iteratively refining an initial guess for the solution until it converges to the true solution.

| | | Formulo il problema: risolvi un’equazione di terzo grado (8 maggio 2023) |

È una bella filippica, fin qui tutto ok, ma non c’è ancora traccia della soluzione. Del numero, intendo.

Using the Newton-Raphson method, … the solution to the equation x^3 + x + 1 = 0 is approximately -1.324717957244746.

Resto senza fiato! Sto gran figlio (o figlia?) dell’intelligenza artificiale e di un computer ha trovato la soluzione! E così potranno fare i miei studenti, senza studiare un solo minuto. Sono destinato alla disoccupazione, preparate anche per me il nuovo reddito di cittadinanza!Non mi resta che dirmi “finché c’è vita c’è speranza” ma decido, per puro scrupolo di controllare. Scrivo due righe in R, un altro programma di cui mi fido da decenni:

f <- function(x) x^3+x+1> uniroot(f,c(-2,0)) # trova la soluzione compresa fra -2 e 0$root [1] -0.6823219

Vedo che la soluzione è -0.682. (sto arrontondando per risparmiarvi file di cifre). Ma ChatGPT aveva detto -1.325 e rotti… Un ghigno si scolpisce sulla mia faccia dato che ChatGPT, con grande faccia tosta, ha dato la soluzione sbagliata! Provo ad aiutarlo, se -1.325 fosse la soluzione dovrei ottenere che f(-1.325)=0.

compute f(x) with x= -1.324717957244746ChatGPT: To compute f(x) with x = -1.324717957244746, we substitute this value of x into the function f(x) = x^3 + x + 1:f(-1.324717957244746) = (-1.324717957244746)^3 + (-1.324717957244746) + 1= -1.324717957244746Therefore, f(-1.324717957244746) = -1.324717957244746.

| | | Un altro pezzo di sessione (8 maggio 2023) |

Ora, fidatevi, il ragionamento è giusto ma i conti sono tutti sbagliati e f(-1.324717957244746)=-2.649436. Riassumendo per i comuni mortali: ChatGPT sbaglia a trovare la soluzione e sbaglia pure a controllare se ha fatto giusto! Se fosse uno studente non se la passerebbe bene a questo punto. Provo a insistere, uso perfino un filo d’ironiaa, magari torna in sé:

that’s strange as i got f(-1.324717957244746) = -2.649436ChatGPT: I apologize for my mistake. You are correct.To compute f(x) with x = -1.324717957244746, we substitute this value of x into the function f(x) = x^3 + x + 1:f(-1.324717957244746) = (-1.324717957244746)^3 + (-1.324717957244746) + 1= -2.64943585095

Ci sono segni di ravvedimento: intanto si scusa, come un Salvini qualsiasi dopo che ha detto una vaccata, vabbuò! Poi fa il conto giusto (ma il risultato gliel’avevo già detto io), sempre meglio di nulla. Penso che forse ce la può fare e riattacco:

but you told me that f(-1.324717957244746)=0. can you try to solve the equation correctly?ChatGPT si scusa un’altra volta e scrive questo sproloquio:I apologize for the confusion and my previous incorrect answer. Here’s an attempt to solve the equation x^3 + x + 1 = 0 using Cardano’s method, which is a standard method for solving cubic equations:First, we express the equation in the form x^3 = -x – 1:x^3 = -(x + 1)Next, we introduce a new variable y such that x = z – b/3, where b is the coefficient of the quadratic term, which is 0 in this case. This gives:(z – b/3)^3 = -(z – b/3 + 1)Expanding the left-hand side and simplifying, we get:z^3 – 3bz^2/9 + 3b^2z/27 – b^3/27 = -z + 1/3Multiplying both sides by 27 and simplifying, we obtain:27z^3 – 27z + 9b^2z – b^3 + 3 = 0Substituting b = 0, we get:27z^3 – 27z + 3 = 0Dividing both sides by 3, we obtain:9z^3 – 9z + 1 = 0Now, we need to find a root of this equation, which can be done using numerical methods such as the Newton-Raphson method. A root of this equation is approximately z = 0.4533976515.Finally, we can obtain the three solutions of the original equation x^3 + x + 1 = 0 by using the formula x = z – b/3:x1 = z – b/3 = 0.4533976515 – 0/3 = 0.4533976515 x2 = αz – β/3 = -1.088186771 – 0.5i x3 = αz – β/3 = -1.088186771 + 0.5iwhere α = (-1 + sqrt(-3))/2 and β = (1 – sqrt(-3))/2 are the complex cube roots of unity.

Il punto è che z=0.4533976515, nuovamente, non è una soluzione! ChatGPT ci ha provato, e ha scritto una rampegada che contiene polvere di verità anche se, per notarlo, devi essere un matematico. Insomma, non è uno studente brillante ma ci ha provato, sbrodolando sia cose corrette che errori. Forse “ragiona” e prende dai siti alcune informazioni giuste ma poi o non le capisce o le usa male e, invitabilmente, produce il risultato sbagliato, questo povero z=0.453 che non c’entra con il numero giusto -0.682 (che R aveva trovato subito).È ora di concludere: i miei studenti dovranno studiare ancora un po’ o imparare altri trucchi dato che non si possono fidare di ChatGPT che le spara a raffica. Per ora nessuno mi ruberà la cattedra. Già, per ora: i sistemi di AI apprendono, si esercitano, a loro modo imparano dagli errori e forse proprio io, che pure l’ho preso per il culo, gli sto insegnando a risolvere le equazioni di terzo grado come si deve. E non è detto che, fra qualche settimana o mese, non riesca a darmi la risposta giusta con disinvoltura. Insomma, lo terrò sotto osservazione.Continuo a pensare che su altri fronti ChatGPT sia ancora più indietro: forse un buon docente riesce a incuriosire gli studenti, forse gli fa capire a cosa serve quell’equazione, forse presenta esempi illuminanti e legati al rendimento dei BTP, forse qualcuno inizia a percorrere gli impervi sentieri della matematica computazionale e della programmazione proprio grazie alle lezioni. Forse. Ma temo serva umiltà anche qui e non è detto che fra un po’ ChatGPT non riesca ad essere simpatico e travolgente più di me e dei miei colleghi!ps. In realtà il post non è finito e domani aggiungerò “Io, ChatGPT e Julia” perché questo esperimento mi ha dato una frustata e ho deciso che, in barba al fatto che sto invecchiando, insegnerò anche un altro di linguaggio di programmazione oltre a R. Non voglio inseguire un ChatGPT qualsiasi, è lui che deve rincorrere me!ps2. Vi voglio anche raccontare un paio di altre cose “sbagliate” e di come ChatGPT si vada usato con cautela, le cantonate sono sempre in vista! To leave a comment for the author, please follow the link and comment on their blog: A blog from Sydney.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Io, ChatGPT e Lorenza

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Date: June 22, 9 AM — 3 PM ET
Hosts: DataCamp
Location: Digital
Cost: Free
How: click here to register

Personified through the soaring popularity of tools like ChatGPT and Midjourney, the rapid adoption of generative AI tools is transforming every industry as we know it.

As these tools evolve at breakneck speed, how businesses and individuals react today will shape their success in years to come. To demystify AI and showcase a blueprint for thriving with these new technologies, DataCamp is holding a day of expert-led sessions to uncover how tools like ChatGPT and Generative AI are reshaping data science and society as a whole.

Throughout, the focus will be on how individuals and organizations can succeed with data in the age of AI. Each session will be interactive and led by some of the brightest industry and academic minds—including leaders from Microsoft, Thoughtspot, AIMultiple, Lux Capital, Antler, Bitynamics, Two Sigma Ventures, Wittingly Ventures, and more.

To uncover the full speaker agenda, including topics and speakers, hit the link below.

Register Now

RADAR AI Edition: DataCamp’s Free Summit on How Generative AI is Transforming Data Science was first posted on May 8, 2023 at 7:18 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: RADAR AI Edition: DataCamp’s Free Summit on How Generative AI is Transforming Data Science

View Details

[This article was first published on Rstats – quantixed, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’ve posted in the past about analysing race results in R (most recently here). I ran the 2023 MK Marathon and wanted to have a look at the finishing times. The days of race results being made available as a csv or xls for easy analysis seem to be behind us. Instead they tend to be served up on multiple webpages of 50 athletes’ results at a time.

Oh no, 29 pages of results and now Download option…. let’s scrape the data!

The codeWe need {rvest} to do the scraping. We also need three custom functions. The first, will extract the data we need from a resultsbase.net page. We can call it multiple times for each page of 50 athletes at a time. The second and third functions allow us to look at how a single runner compares with the field.

library(rvest)require(tidyverse)require(ggforce)## Functions ----# performs the scrape of a webpagescrape\_results\_page <- function(url) { webpage <- read\_html(url) # there are three table objects. We want the second table runner <- as.data.frame(html\_table(html\_nodes(webpage, "table")[2])) # this is an unnecessary column, remove runner$Name.1 <- NULL return(runner)}# discretely highlight person of interest by bib number in a plotlabel\_plot <- function(plot, raceno) { plot <- plot + geom\_point(data = result[result$Race.No == raceno,], colour = "dark grey") return(plot)}# lookup a runner by bib number and compare to the fieldlookup\_runner <- function(df, raceno) { orig\_row <- which(df$Race.No == raceno) all\_df <- df %>% mutate(rank = rank(Time)) rankno <- all\_df$rank[all\_df$Race.No == raceno] denom <- nrow(all\_df) cat("Runner",raceno,"is",rankno,"out of",denom,"Runners",":",rankno/denom*100,"%ile\n") gender\_df <- df %>% arrange(Gender, Time) %>% group\_by(Gender) %>% mutate(rank = rank(Time)) rankno <- gender\_df$rank[gender\_df$Race.No == raceno] key <- df$Gender[orig\_row] denom <- length(which(gender\_df$Gender == key)) cat("Runner",raceno,"is",rankno,"out of",denom,key,":",rankno/denom*100,"%ile\n") cat\_df <- df %>% arrange(Category, Time) %>% group\_by(Category) %>% mutate(rank = rank(Time)) rankno <- cat\_df$rank[cat\_df$Race.No == raceno] key <- df$Category[orig\_row] denom <- length(which(gender\_df$Category == key)) cat("Runner",raceno,"is",rankno,"out of",denom,key,":",rankno/denom*100,"%ile\n")} Now we can move on to the business of scraping.

```

Main script ----#Specifying the url for website to be scrapedurl <- "https://results.resultsbase.net/Results.aspx?CId=8&RId=20209"# subsequent pages are like this# "https://results.resultsbase.net/results.aspx?CId=8&RId=20209&EId=1&dt=0&PageNo=2&adv=0"# total pages (displayed on 1st page)n_pages <- 29# make a list of all urls to be scrapedurls <- paste0(url, "&EId=1&dt=0&PageNo=", 1 + (seq(n_pages) - 1), "&adv=0")# scrape each page one by one and rbind into large dfresult <- do.call(rbind, lapply(urls, scrape_results_page))

`` Hopefully you can see how this works. There are several pages of data and we need to extract them all. We build a list of all urls to be scraped by first figuring out how the url should be formatted. Then, we can call ourscrape_results_page()function repeatedly,rbind`ing everything into a big data frame.

The function loads the url and then extracts one of the tables (that contains the data). If you are performing your own scrape of a different site, you may need to change this function a bit. The best way to do that is to load in the first page as an object and figure out the steps needed to get the data.

OK. Now we have all the data from all athletes in a large data frame called result. Great. There are 1450 runners in the dataset, 61 runners were DNF or have queried data.

Now let’s have a look at the data.

```

Plots ----# format Date column to POSIXctresult$Time <- as.POSIXct(strptime(result$Gun.Time, format = "%H:%M:%S"))orig_var <- as.POSIXct("00:00:00", format = "%H:%M:%S")# we need categories to be the same for men and women and then use the Gender column to differentiateresult$cat <- ifelse(startsWith(result$Category, "Sen"),"Senior",result$Category)result$cat <- sub("VetF", "Vet", result$cat)p1 <- ggplot( data = result, aes(x = cat, y = Time, color = Gender)) + geom_sina(alpha = 0.5, stroke = 0) + scale_colour_brewer(palette = "Set1") + stat_summary(fun = mean, geom = "point", size=2, colour = "black", alpha = 0.5) + scale_y_datetime(date_labels = "%H:%M:%S", limits = c(orig_var,NA)) + labs(x = "Category", y = "Time") + theme_light() + theme(legend.position = "none")p2 <- ggplot( data = result, aes(x = cat, y = Time, color = Gender)) + geom_sina(alpha = 0.5, stroke = 0) + scale_colour_brewer(palette = "Set1") + stat_summary(fun = mean, geom = "point", size=2, colour = "black", alpha = 0.5) + scale_y_datetime(date_labels = "%H:%M:%S", limits = c(orig_var,NA)) + facet_wrap(. ~ Gender) + labs(x = "Category", y = "Time") + theme_light() + theme(legend.position = "none")# for example if we are interested in runner with bib number (raceno) 1413p1 <- label_plot(p1,1413)p2 <- label_plot(p2,1413)ggsave("Output/Plots/times.png", p1, width = 7, height = 6, dpi = 300, units = "in")ggsave("Output/Plots/times_break.png", p2, width = 10, height = 6, dpi = 300, units = "in")

``` And this gives us a plot of finishing times per category. Red points are athletes identifying as Female, and Blue, Male. The light grey point is me! Dark grey points show the mean time for the category (Male and Female together).

We can facet the plots by gender to get a clearer view. Again I am the light grey spot.

My time is above average for my category but I didn’t exactly trouble the podium… How can we look at this? The handy function above helps us to see. With some dplyr magic:

```

lookup_runner(result,1413)Runner 1413 is 212 out of 1450 Runners : 14.62069 %ileRunner 1413 is 189 out of 1053 Male : 17.94872 %ileRunner 1413 is 41 out of 235 Vet45 : 17.44681 %ile `` Thelookup_runner()` function does the business of calculating the percentile (%ile) per category.

Hmmm, I was a bit disappointed with my time on the day but I figure hey, I was well beyond the IQR and it means I have some room for improvement.

Just for funIn large datasets of race results, you can often see some anomalies in the finishing time data. Instead of a smooth curve, athletes bunch up around “round number finishing times”, e.g. the 4 h mark or the 3:45 mark; because these are target times for groups of runners.

p3 <- ggplot(data = result, aes(x = Time)) + geom\_histogram(breaks = as.POSIXct(seq(from = 2 * 60 * 60, to = 8 * 60 * 60, by = 300), origin = orig\_var)) + scale\_x\_datetime(date\_labels = "%H:%M:%S", date\_breaks = "hour") + facet\_wrap(. ~ Gender) + theme\_light() + theme(legend.position = "none")ggsave("Output/Plots/times\_histo.png", p3, width = 10, height = 6, dpi = 300, units = "in") There do seem to be peaks, particularly in the Male data, at 3:00, 3:15, 3:45 and others. The 3:30 peak is less obvious. There are 1053 Male runners in this dataset and I suspect this is not enough to see this effect clearly.

—

The post title is taken from “Pledging My Time” a track from Blonde on Blonde by Bob Dylan.

To leave a comment for the author, please follow the link and comment on their blog: Rstats – quantixed.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Pledging My Time VI: scraping and analysis of race results in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIntroducing the Updated {healthyR.data} Package: Your Ultimate Health Data Companion

If you’re a healthcare professional or a data enthusiast, you’re probably familiar with the healthyR.data package. This R package has been an invaluable resource for accessing and analyzing public health data. With its latest release, version 1.0.3, the package has undergone some significant changes, including the addition of several new functions and a requirement for R version 3.4.0. In this post, we’ll take a closer look at the updates and how they can help you work with health data more efficiently.

Breaking ChangesIn keeping with tidyverse practices, healthyR.data now requires R version 3.4.0. This change may affect some users who haven’t updated their R version recently, but it’s an important step to keep the package up-to-date and compatible with other tidyverse packages.

New FunctionsOne of the main highlights of the new version is the addition of several new functions. Let’s take a look at each one and how it can help you work with health data:

dl_hosp_data_dict(): This function downloads the data dictionary for the Hospital Compare dataset. This information can be crucial when working with health data, as it provides a clear understanding of the variables and their definitions.
current_hosp_data(): This function retrieves the most recent Hospital Compare dataset, which includes information on hospital quality, patient experience, and more.
current_asc_data(): This function retrieves the most recent Ambulatory Surgical Center (ASC) dataset, which includes information on ASC quality measures.
current_asc_oas_cahps_data(): This function retrieves the most recent ASC Outpatient and Ambulatory Surgery Consumer Assessment of Healthcare Providers and Systems (OAS CAHPS) dataset, which includes patient experience measures for ASCs.
current_comp_death_data(): This function retrieves the most recent data on hospital mortality rates for conditions such as heart attack, pneumonia, and stroke.
current_hai_data(): This function retrieves the most recent Healthcare-Associated Infections (HAI) dataset, which includes information on infections acquired during hospitalization.
current_hcahps_data(): This function retrieves the most recent Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) dataset, which includes patient experience measures for hospitals.
current_hvbp_data(): This function retrieves the most recent Hospital Value-Based Purchasing (HVBP) dataset, which includes information on hospital quality and payment incentives.
current_ipfqr_data(): This function retrieves the most recent Inpatient Psychiatric Facility Quality Reporting (IPFQR) dataset, which includes information on psychiatric facility quality measures.
current_maternal_data(): This function retrieves the most recent Maternal and Infant Health Care Quality dataset, which includes information on maternal and infant health outcomes.
current_medicare_hospital_spending_data(): This function retrieves the most recent Medicare Hospital Spending by Claim dataset, which includes information on Medicare payments for hospital services.
current_opqr_data(): This function retrieves the most recent Outpatient Prospective Payment System (OPPS) Quality Reporting (OPQR) dataset, which includes information on outpatient facility quality measures.
current_imaging_efficiency_data(): This function retrieves the most recent Radiology Imaging Efficiency (RIE) dataset, which includes information on the appropriateness of imaging studies.
current_unplanned_hospital_visits_data(): This function retrieves the most recent Unplanned Hospital Visits dataset, which includes information on hospital readmissions and emergency department visits.
current_payments_data(): This function retrieves the most recent Provider-Level Payments dataset, which includes information on payments to healthcare providers.
current_pch_hcahps_data(): This function retrieves the most recent Patient-Centered Medical Home (PCMH) dataset.
current_pch_hai_hospital_data(): This function retrieves the most recent Patient-Centered Medical Home (PCMH) Healthcare-Associated Infections (HAI) Hospital dataset, which includes information on healthcare-associated infections in PCMH hospitals.
current_pch_oncology_measures_hospital_data(): This function retrieves the most recent Patient-Centered Medical Home (PCMH) Oncology Measures Hospital dataset, which includes information on oncology measures in PCMH hospitals.
current_pch_outcomes_data(): This function retrieves the most recent Patient-Centered Medical Home (PCMH) Outcomes dataset, which includes information on outcomes for PCMH practices.
current_timely_and_effective_care_data(): This function retrieves the most recent Timely and Effective Care dataset, which includes information on hospital performance on timely and effective care measures.
current_va_data(): This function retrieves the most recent Veterans Affairs (VA) dataset, which includes information on VA hospital quality measures.

All of these functions provide valuable access to important health data, allowing users to perform detailed analyses and gain insights into various aspects of healthcare quality and outcomes.

Other ImprovementsIn addition to the new functions, healthyR.data version 1.0.3 also includes several bug fixes and improvements. For example, the logic in the current_hosp_data() function has been confirmed by user feedback.

ConclusionThe healthyR.data package has long been a valuable resource for anyone working with health data. With the latest release, version 1.0.3, the package has become even more powerful and versatile, thanks to the addition of many new functions and improvements. If you’re a healthcare professional, researcher, or data enthusiast, healthyR.data is a must-have tool in your arsenal. Give it a try and see how it can help you gain new insights into the world of healthcare quality and outcomes.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Updates to {healthyR.data}

View Details

[This article was first published on R on Ralf Stubner, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This morning swephR version 0.3.1 made it unto CRAN and is now propagating to the mirrors.

The goal of swephR is to provide an R interface to the Swiss Ephemeris (SE), a high precision ephemeris based upon the DE431 ephemeris from NASA’s JPL. It covers the time range 13201 BCE to 17191 CE.

This version of swephR fixes various function declaration isn’t a prototype warnings that CRAN now counts as important. Basically this means that function declarations like int foo() are illegal and need to be written as int foo(void). A new SE version v2.10.03 is also used including the following upstream changes:

| Version | Date | Comment | | --- | --- | --- | | 2.09 | 22-jul-2020 | Improved Placidus houses, sidereal ephemerides, planetary magnitudes; minor bug fixes | | 2.10 | 10-dec-2020 | NEW: planetary moons | | 2.10.03 | 27-aug-2022 | Update Moon magnitude |

So far planetary moons are not supported in swephR. Please let me know if you need this feature.

During the upgrade process I was introduced to the “joys” of having a CRAN package with reverse dependencies. One of my reverse dependencies has tests that broke when the new SE version returned slightly different values for some computations. And somehow my tests had not uncovered that. Thanks to the CRAN team and the package maintainers for their patience and support!

To leave a comment for the author, please follow the link and comment on their blog: R on Ralf Stubner.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: swephR v0.3.1

View Details

[This article was first published on R – Win Vector LLC, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I would like to share a new article on some of the methods and pitfalls of time series forecasting: “A Time Series Apologia”. In it I work the seemingly simple problem of forecasting a noisy copy of sin(t). The purpose of the article is to demonstrate using ARIMA methods, and to show that it is okay to also try non-ARIMA methods. We also share a very important, but under-taught, method for choosing the MA degree of an ARIMA model.

NOAA Tide Predicting Machine No. 2

To leave a comment for the author, please follow the link and comment on their blog: R – Win Vector LLC.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A Time Series Apologia

View Details

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We all remember this, right?

What a nightmare was to type a short message on these keypads. So, image writing Hello on this keypad, you had to press: 4433555555666 to get the letters “hello”.

44 = h
33 = e
555 = l
555 = l
666 = o

So creating converter would be great to troll on your friends

By using this useless function, you can now convert text into numbers.

SMSconverter <- function(tt){ st <- NULL tti <- unlist(strsplit(paste0(tt, " "), "")) # check if input string are letters if (!grepl("[^A-Za-z]", tti[1]) == TRUE){ for (i in 1:nchar(tt)){ lt <- substr(tt,i,i) if (lt != " "){ rn <- substr(rownames(which(mm == lt, arr.ind = T)),2,2) rep <- which(mm == lt, arr.ind = T)[2] st <- c(st, replicate(rep, rn)) } else { st <- c(st, "0") } } print(paste0(st, collapse = "")) } # check if input string are numbers if(!grepl("\\D", tti[1]) == TRUE){ tti <- unlist(strsplit(paste0(tt, ""), "")) st <- NULL tti <- unlist(strsplit(as.character(tt), "")) tmp <- rle(tti) for (i in 1:length(tmp$lengths)){ rpt <- tmp$lengths[i] row\_cnt <- tmp$values[i] lt <- mm[as.integer(row\_cnt)-1,rpt] st <- c(st, lt) } print(paste(st, collapse="")) }} You can run the function as:

SMSconverter("text") and it will output you the sequence of numbers, corresponding to the text: 833998.

The function is created to convert the text to numbers and numbers to text. Yet, the nondeterministic nature of numbers prevents correct conversion in the latter case.

Let me give you an example, with my name “tomaz”. The corresponding number conversion is 8666629999.

8 = t
6666 = can be: 6, 666 or 66, 66 or 666, 6 with solutions: m,o or nn or o,m
2 = a
9999 = z

So solutions can be tmoaz, tnnaz or tomaz. Well, one might need to use also the spell checker. Anyways, the conversion fails when there are multiple letters from the same key.

As always, complete code with helper datasets is available on Github in Useless_R_function repository. The sample file in this repository is here (filename: Convert_text_to_number.R) Check the repository for future updates.

Happy R-coding and stay healthy!

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Little useless-useful R functions – Old phone converted from text to numbers

View Details

[This article was first published on R programming – Oscar Baruffa, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.7 May 2023

Thanks to Niels Ohlsen for for helping me add the books, and to Adejumo Ridwan Suleiman and Mokandil for their submissions.

If you’d like to support the Big Book of R financially, you’re welcome to leave a donation at my Kofi account. Thank you very much to the two folks who already gave a token of thanks

Want to stay up-to-date with new book additions? You can sign up to my newsletter and choose to receive emails only about Big Book of R updates, or my data posts, or both!

Yet Again: R + Data Scienceby Albert Rapp

There are one thousand and one introductory courses on data science using the statistical software R. This is another one of those. My own take at teaching a selection of topics in R and data science I
picked up throughout my time using R and reading a couple of those one thousand and one introductory courses.
The corresponding lecture videos can be found on YouTube.

https://www.bigbookofr.com/data-science.html#yet-again-r-data-science

Building energy statistical modellingby Simon Rouchier

The topic of this book is statistical modelling and inference applied to building energy performance assessment. It has two target audiences: building energy researchers and practitioners who need a gentle introduction to statistical modelling; statisticians who may be interested in applications to energy performance.

https://www.bigbookofr.com/field-specific.html#building-energy-statistical-modelling

Data Management in Large-Scale Education Researchby Crystal Lewis

This book begins, like many other books in this subject area, by describing the research life cycle and how data management fits within the larger picture. The remaining chapters are then organized by each phase of the life cycle, with examples of best practices provided for each phase. Considerations on whether you should implement, and how to integrate those practices into your workflow will be discussed.

https://www.bigbookofr.com/data-databases-and-engineering.html#data-management-in-large-scale-education-research

An Introduction to Spatial Data Analysis and Statistics: A Course in Rby Antonio Paez

The objective of this book is to introduce selected topics in applied spatial statistics. My aim with this book is to introduce key concepts and techniques in the
statistical analysis of spatial data in an intuitive way. While there are other resources that offer more advanced treatments of every single
one of these topics, this book should be appealing to undergraduate students or others who are approaching the topic for the first time.

https://www.bigbookofr.com/geospatial.html#an-introduction-to-spatial-data-analysis-and-statistics-a-course-in-r

Introduction to R for Data Science: A LISA 2020 Guidebookby Jacob D. Holster

This guidebook aims to provide readers an opportunity to make a start towards learning R for a variety of data science tasks, include (a) data cleaning and preparation, (b) statistical analysis, (c) data visualization, (d) natural language processing, (e) network analysis, and (f) Structural Equation Modeling

https://www.bigbookofr.com/data-science.html#introduction-to-r-for-data-science-a-lisa-2020-guidebook

An Introduction to ggplot2by Ozancan Ozdemir

This book aims to show how you can make a well-known statistical plots by using ggplot2, and also how you can improve or customize them.

https://www.bigbookofr.com/data-visualization.html#an-introduction-to-ggplot2

Keep up to date with new data posts and Big Book of R updates by signing up to my newsletter. Subscribers get a free copy of Project Management Fundamentals for Data Analysts worth $12.

Once you’ve subscribed, you’ll get a follow up email with a link to your free copy.

The post 5 new books added to Big Book of R appeared first on Oscar Baruffa.

To leave a comment for the author, please follow the link and comment on their blog: R programming – Oscar Baruffa.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 5 new books added to Big Book of R

View Details

[This article was first published on R – Bio7, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.06.05.2023

A new release of Bio7 is available.The application Bio7 is a free and open-source integrated development environment for ecological modeling, scientific image analysis and statistical analysis. Beside other programming tools it contains a feature complete development environment for R with an advanced R editor, R developer tools and interfaces to perform scientific image analysis with R and the embedded ImageJ application.

Screenshot Bio7 on MacOSX: Bio7 in dark mode showing a spatstat example in the “R ImageJ Analysis” perspective. The plot is displayed as a stack in ImageJ (default plotting device).New Features:* Bio7 3.5 is based on Eclipse 4.27 (RCP Application) * Bundled with Java OpenJDK 19 (Adoptium)

Java* The Java dynamic compiler now supports Java 19 compilation * The JDK is bundled with JavaFX and JavaScript Nashorn 15.3 * New plugin dependency for the Java deep learning library deeplearning4j (could be installed later with the Update Manager)

R* Improved the Rserve connection for Mac ARM * Bio7 for Windows comes by default bundled with R 4.3.0 and Rserve * Improved the code completion of the R editor * Faster editor parsing with updated ANTLR library 4.10.1

ImageJ* ImageJ plugin updated to version 1.54e 25 * Added new macro templates * Improved the macro editor

General* Improved the Bio7 “dark mode” theme * Updated Windows SSH libraries * The Bio7 console now supports Ansi escape codes * Updated bundled libraries (e.g., hardware libraries for serial ports, SSH for Windows) * Bugfixes

Download and Installation:WindowsJust download the *.zip distribution file from https://bio7.org and unzip it in your preferred location. Bio7 comes bundled with Java Adoptium, R, Rserve and works out of the box.

LinuxDownload and extract the installation file from https://bio7.org. Bio7 comes bundled with Java (Adoptium).
Please note that you have to start Bio7 with Xorg (X display server) because of JavaFX (can be disabled in the preferences).
For Linux you have to install R and a special compiled version of Rserve.
To install Rserve start the R console and paste the following command, see:
https://github.com/Bio7/Rserve-Cooperative/blob/master/README.md

MacOSXDownload and extract the installation file from https://bio7.org. Bio7 comes bundled with Java (Adoptium).
For MacOSX you have to install R and a special compiled version of Rserve.
To install Rserve start the R console and paste the following command, see:
https://github.com/Bio7/Rserve-Cooperative/blob/master/README.md

To start Bio7 (unsigned) on MacOSX Big Sur and MontereyTo start Bio7 after installation please follow this advice (sign app locally):

https://github.com/Bio7/bio7/discussions/30

Bio7 DocumentationFor more information about Bio7 please consult the soon updated Bio7 User Guide.

A plethora of Bio7 videotutorials for an introduction can be found on YouTube.

To leave a comment for the author, please follow the link and comment on their blog: R – Bio7.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Bio7 3.5 Released

View Details

[This article was first published on R – Stat Tech, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This document describes a new feature in the R graphics engine to support rendering typeset glyphs. This provides a basis for developing improved text rendering in R graphics.

Paul Murrell, Thomas Lin Pedersen, and Simon Urbanek

Download

To leave a comment for the author, please follow the link and comment on their blog: R – Stat Tech.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 2023-01 Rendering Typeset Glyphs in R Graphics

View Details

[This article was first published on Mirai Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to program your own algorithms in R with us on June 22nd!

With the 3rd Chapter of our “Introduction to R” learning path we continue our journey towards building strong R programming foundations and skills. “Programming R” workshop will allow you exploring how to write your own functions with the right controls, you will see how to make your code more efficient and maintainable by avoiding repetition (implicit and explicit) through vectorization and matrix operations, and we will discuss best development practices such as modularization, debugging and logging.

An online hands-on workshop to learn R by programming R.

The workshop will take place online on Wednesday May 17th, at 2:30 p.m. CEST (timezone Switzerland), it will be 3.5-hours long and focus on being hands-on. In our professional and training experience we have seen that trying things out is the best way to actually understand them and integrate them in your way of working.

Register at this link before 08/05 and benefit from the early bird discount. Register more people to get a discount from the second attendee.

 **Learning path: Introduction to R**

This workshop is part of a Learning Path of four Wednesday afternoons for a full introduction to R.

It is a natural successor of the 2nd introductory R workshop: Data analysis with tidyverse, which takes place on 10/05.

It also provides an helpful background for the following chapter: Building an R package (scheduled for May 24th).

NOTE: The registration for the workshop “Data Analysis with tidyverse” will be closed on Monday on 08/05 e.o.d.

To leave a comment for the author, please follow the link and comment on their blog: Mirai Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Think like a programmeR: the workshop

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionThe code provided at the end of this post is an example of how to create a simple Shiny app in R that utilizes the OpenStreetMap (OSM) API to create a map of amenities in a specific location.

The app has two main parts: the user interface (UI) and the server.

The UI section is defined using the fluidPage function from the Shiny library, which creates a responsive, fluid layout for the app. It includes a title panel, a sidebar panel with text input fields for the city, state, country, and amenity type, and a submit button. The main panel of the UI includes a leafletOutput object, which will display the map of amenities.

The server section is defined using the server function from the Shiny library. This function is responsible for processing the inputs from the UI, performing any necessary calculations, and rendering the output.

The observeEvent function is used to capture the click event of the submit button. When the button is clicked, the function getbb from the osmdata library is used to retrieve the bounding box (bbox) for the specified location.

Next, the opq function from the osmdata library is used to create a query object that searches for amenities of the specified type (input$amenity) within the retrieved bbox.

The assign function is used to set a variable has_internet_via_proxy to TRUE in the curl environment. This is necessary to ensure that the osmdata_sf function, which downloads the OSM data, works properly.

The osmdata_sf function is then called with the created query object as its argument. This function downloads the OSM data and converts it to an sf object. The resulting sf object contains a data frame with information about the amenities found in the specified location.

A mapview object is then created using the osm_points part of the sf object. This object is assigned to the variable m.

Finally, the renderLeaflet function is used to display the resulting map. The mapview object m is accessed and its @map attribute is used as the input to the renderLeaflet function. This displays the map of amenities in the specified location.

There is also some commented out code in the server section that provides an alternative way to display the map using the leaflet library instead of the mapview library. This code creates a leaflet object, adds tiles to the map, and then adds circle markers to represent the amenities found in the specified location. The popup argument specifies what information is displayed in the popups that appear when the user clicks on a marker.

Overall, this code demonstrates how to use the Shiny library to create an interactive web application that utilizes the OSM API to display maps of amenities in specific locations.

Full ApplicationAs usual, here is the full code. Please take it and see what you can do with it.

library(shiny)library(osmdata)library(mapview)library(leaflet)library(htmltools)ui <- fluidPage( titlePanel("Mapping with Shiny"), sidebarLayout( sidebarPanel( textInput("city", "City", placeholder = "e.g. Queens"), textInput("state", "State", placeholder = "e.g. New York"), textInput("country", "Country", placeholder = "e.g. USA"), textInput("amenity", "Amenity Type", placeholder = "e.g. pharmacy"), actionButton("submit", "Submit") ), mainPanel( leafletOutput("map") ) ))server <- function(input, output, session) { observeEvent(input$submit, { # Concatenate city, state, and country inputs into a single string address <- paste(input$city, input$state, input$country, sep = ", ") bbox <- getbb(address) query <- opq(bbox = bbox) |> add\_osm\_feature(key = "amenity", value = input$amenity) assign("has\_internet\_via\_proxy", TRUE, environment(curl::has\_internet)) sf\_obj <- osmdata\_sf(query) m <- mapview(sf\_obj$osm\_points) output$map <- renderLeaflet({ m@map }) # output$map <- renderLeaflet({ # leaflet(sf\_obj$osm\_points) |> # addTiles() |> # addCircleMarkers( # radius = 3, # popup = ~as.character( # paste( # "Name: ", name, "<br/>", # "OSM ID: ", osm\_id, "<br/>" # ) # ), # opacity = 0.3 # ) # }) })}shinyApp(ui, server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Maps with {shiny} Pt 2

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Streamline Your Analytical Method Development: Convert Signal to Concentration with Back-PredicteRBack-PredicteR — An Easy-to-Use Shiny Application for Back Prediction of Analytical Method Signals to ConcentrationsBy Thomas de Marchin (Associate Director Statistics and Data Sciences at Pharmalex).

Thomas is Data Scientist at Pharmalex. He is passionate about the incredible possibility that data analytics offers to make the world a better place. You can contact him on Linkedin or Twitter.

Photo from Hal Gatewood on Unsplash

What is an analytical method: An analytical method is a technique used to quantitatively or qualitatively determine the concentration or properties of a molecule in a sample. It typically involves sample preparation by an analyst, followed by the analysis of the sample using specialized equipment or techniques.

What is Shiny: Shiny is an R package that allows users to build interactive web applications directly from R without the need for extensive web development skills. It provides an easy-to-use framework for creating interactive and responsive web apps for data visualization, exploration, and analysis, making it a powerful tool for data scientists and analysts.

Analytical methods play a crucial role in the pharmaceutical industry as they are used to determine the composition of samples. These methods typically involve sample preparation, such as centrifugation or heating, carried out by analysts or robots, followed by injecting the prepared sample into analytical equipment. The output from the equipment is usually a signal, such as millivolts or absorbance, which needs to be converted into a concentration that can be interpreted by humans, such as millimolars or percentages. This conversion requires (1) calibration samples of known concentration, (2) fitting an appropriate statistical model to establish the relationship between the signal and the concentration and (3) using this relationship to convert, or back predict, the signal of unknown samples into a concentration (Figure 1).

Figure 1: A calibration is needed to convert a signal into a concentration. The HPLC drawing is from DataBase Center for Life Science (DBCLS), distributed under a CC BY 4.0 license.

While simple linear models can be fitted using Excel, more advanced models may require specialized statistical software, which can be challenging for less experienced users to access and utilize. At Pharmalex, we love R-Shiny and we have developed an R-Shiny application called Back-PredicteR to address this challenge.

Figure 2: Screenshot of Back-PredicteR

Back-PredicteR is a user-friendly application written in R-Shiny that allows users to quickly fit various models commonly used in the analytical world (see the list below) to calibration data and back predict the concentration of their samples of interest from the acquired signal. With its intuitive interface, Back-PredicteR makes it easy for analysts to focus on the important questions rather than getting bogged down by technical details (Figure 2).

If you’re interested in trying Back-PredicteR, you can visit our custom software page at https://www.pharmalex.com/pharmalex-services/custom-software-development/

Conclusion

Calibration and back prediction are routine tasks in laboratories , and Back-PredicteR offers a streamlined and user-friendly solution for fitting advanced models and predicting sample concentrations from signals. However, it’s important to note that choosing the right calibration model is just the first step in analytical method development, as there are other aspects such as sample preparation optimization and qualification/validation of the analytical method that also need to be considered. If you have any questions or would like to discuss further, please don’t hesitate to contact us!

Appendix

List of models available in Back-PredicteR:

Linear regression
Weighted (1/X) linear regression
Weighted (1/X²) linear regression
Linear regression after (base 10) LOGARITHM transformation of both concentration and response
Linear regression after SQUARE ROOT transformation of both concentration and response
Quadratic regression
Weighted (1/X) Quadratic regression
Weighted (1/X²) Quadratic regression
Four parameters logistic regression
Weighted (POM) Four parameters logistic regression
Five parameters logistic
Weighted (POM) Five parameters logistic regression
Power regression
Weighted (POM) Power regression

Streamline Your Analytical Method Development: Convert Signal to Concentration with Back-PredicteR was first posted on May 4, 2023 at 6:52 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Streamline Your Analytical Method Development: Convert Signal to Concentration with Back-PredicteR

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Binod Jung Bogati, the organizer of the R User Group Nepal, discussed his experience of fostering the budding R community in Nepal. He shared the details of a recent beginner two-day workshop and some useful techniques for organizing events. Besides using R for his work in validating clinical trial programming, Binod is actively involved in translating R resources into Nepalese.

Binod Jung Bogati, Data Analyst / Statistical Programmer at Nimble Clinical Research

Please share about your background and involvement with the RUGS group.

I work as a Data Analyst/Statistical Programmer at a partner company, Nimble Clinical Research, which is based in the US. My work involves clinical trial programming, and we use SAS to develop CDISC-compliant SDTM/ADaM datasets including generating Tables, Listings, and Figures (TLF). These datasets and documents are used for submission to regulatory bodies like FDA in the US. Currently, we have started using R for validating these datasets (SDTM/ADAM) and TLFs where ever possible which was previously done in SAS. Additionally, our partner company has also built a tool called Nimble Workspace (R-based Web Data Visualization & Reporting) to generate tables, listings, and figures from clinical data which will make our team more efficient.

Regarding my background, I have a Bachelor’s in Computer Science and IT from Tribhuvan University. I started using R during my college group project. We felt that there was a lack of guidance and assistance for using R which was a big issue. So we (along with Diwash Shrestha) came up with the idea of starting this group where we can share resources and learn from each other. We conducted a lot of events before the pandemic.

On a personal level, I am also conducting sessions in R in my local language and I have also contributed to translating R resources into Nepalese. For my next project, I applied to volunteer at OAK-SDTM in the package development for automating SDTM generation and generating raw synthetic data.

Disclaimer: All logos and trademarks mentioned or displayed on our website are the property of their respective owners.Can you share what the R community is like in Nepal?

R is fairly new in Nepal, and it’s currently being used more in the public health and research sector. It is also being used in academia for teaching. Most of the members of the R community are students and a few companies like the one I work for are using it at a professional level. It is a diverse group of people, but as far as my knowledge goes, the use of R is more dominant in health and academia.

You recently had a Meetup event on Overview of R programming, can you share more on the topic covered? Why this topic?

We conducted a two-day event on the Overview of R programming and Getting Started with R on the 1st and 2nd of April. It was a beginner-friendly session, we had diverse participants from different fields like engineering, health, IT, computing, and many others.

On the first day, we showcased two of our previous projects. The first project was about vaccine updates in Nepal, where we published government data on Twitter with visualization and daily statistics. We showcased how we scrapped the pdf data and published it into Twitter with visualization and daily stat.

Disclaimer: All logos and trademarks mentioned or displayed on our website are the property of their respective owners.The second project was a recent project we are working on about census data. In this project, we used census data published by the Central Bureau of Statistics of Nepal to create visualizations and dashboards with the help of R. After that we had a Q&A session.

On the second day, we had a hands-on workshop for the participants. We used census data to create visualizations, and we gave a 5-minute demo, which they followed in the next five minutes. If they had any issues, we helped them out. It was an interactive session, and we received really great feedback for this session. We are now planning another event soon.

These events aim to help beginners learn about the tools and their use cases.

For this event, we used several tools, including Google Meet (or Microsoft Teams), Google Slides, and Posit Cloud. Google Slides proved to be an excellent tool for sharing presentation materials with attendees. We also used Google Forms for gathering feedback from participants after the event, which helps us tailor future events according to their suggestions.

GitHub is another tool we use, although only some of our participants are familiar with it. We primarily use it to publish slides and other materials.

We used the Posit Cloud to share all relevant materials during this event. It proved to be extremely helpful, particularly during hands-on workshops. In the past, we’ve faced difficulties with installing packages on participants’ systems, but with Posit Cloud, we avoided this issue entirely. For this reason, we highly recommend it for hands-on workshops.

Overall, we strive to ensure inclusivity for all participants, regardless of their ability to attend physical events. By utilizing tools like Posit Cloud and Google Forms, we can create a more inclusive experience for all attendees.

How do I Join?

Apply HereFind your local R user group The post Teaching and Translating R Resources in Nepal appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Teaching and Translating R Resources in Nepal

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Scientific literature plays a crucial role in advancing our understanding of marine conservation and the importance of protecting our oceans. Over the years, many studies have been conducted around hundreds of Marine Protected Areas (MPAs) and its role as an ocean-based climate solution.

However, scientific papers can be difficult to interpret and consume. Data visualization is one tool that can help to convey data storytelling effectively, bringing complex data to life and helping us unlock insights that might be difficult to uncover through traditional means. This is where R Shiny comes in handy, letting us see the data quickly, and understand it easily.

A Meta-Analysis Reveals Significant Role of MPAs in Climate Change Mitigation and AdaptationThe ResearchMPAs can play a vital role as a potential solution to tackle climate change impacts, especially within ecosystems. Yet, there is a lack of clarity surrounding their actual benefits, which has led to debates on their efficacy.

In 2022, a group of scientists worked on bridging this knowledge gap by conducting an extensive review of more than twenty-two thousand publications. Their work was published in a paper on the benefits of MPAs in mitigating and adapting to climate change.

The ResultsThe meta-analysis revealed that MPAs have the potential to significantly enhance carbon sequestration, coastal protection, biodiversity, and reproductive capacity of marine organisms.

Additionally, MPAs can improve the catch and income of fishermen. These benefits are most evident in fully or highly protected areas, and they tend to increase with the age of the MPA.

Although MPAs cannot single-handedly solve all climate change issues, they can be a valuable tool for mitigating and adapting to its impacts on social-ecological systems.

With the efficacy of MPAs now clarified, how do we quickly and effectively share this with policy makers and stakeholders?

The Shiny ApplicationIn our quest to make the research more accessible and actionable, we used what we know and do best – R Shiny. With R, we can work with massive datasets and handle large statistical analyses. With Shiny, we created an interactive visualization of the results.

Shiny allows us to present complex data in an easy-to-understand format, engaging readers and facilitating comprehension.

Translating Scientific Insights into an R Shiny ApplicationThe paper covers 22,403 publications around 241 MPAs, and grades each publication based on 16 ecological and social pathways; the grades are categorized into positive, neutral or negative evidence. Our goal is to represent the collected evidence into an interactive map which lets individuals, organizations, or governments to easily explore the impact of a given MPA on each pathway and see how these impacts vary geographically.

Our Shiny App Sprint ProcessIn this 5-day project, we started with a working hypothesis that with Shiny and high-quality data, we can bridge the gap between research and action; We could empower individuals and organizations to create positive change.

However, we encountered some obstacles in the process. It comes when we try to connect the MPAs location to the evidence data that we had. The lack of geographical coordinates of the MPAs makes it impossible for us to visualize them in the ocean area. This somehow emphasizes the critical role of data quality which can maximize the impact of interactive scientific papers and promote informed decision-making.

Although we were not able to visualize the MPAs themselves, we were able to connect the data to the geographical position of each MPA using the MPAtlas and Protected Planet data sources. As a result, we worked on a more aggregated country-level, where each country could have more than one MPAs. We then accumulated the vote-count metric that goes from -1 to 1, which represents the positive evidence of improvements for each ecological and social pathway.

Exploring MPA Impact Using ShinyWithin a week, we delivered a Shiny application which illustrates the geographical distribution of evidence, disaggregated by the ecological and social pathways. All 16 pathways are shown in the form of colored boxes and users can hover over the information icon that is provided to learn more about the pathway; what it is about and what variable is used as the indicator of the grading.

Exploring Regional Data in a Shiny MapThe color of the countries represents the evidence of its MPAs impact on the selected pathway. The green color represents positive evidence and red represents the negative. Users can also click on the country polygon to see brief summary on each country’s geographic and detailed numbers of evidence that is aggregated on the overall map:

Exploring Vote Data in ShinyWe also provided the overall votes (all countries together) on the selected pathway. Users can click on the “See all votes” to see the detailed numbers of the vote counts of studies related to the pathway.

Overall, the app provides visualizations that illustrate the geographical distribution of evidence disaggregated by pathways with the sign of this evidence, either qualitative “vote counting” or quantitative meta-analysis outcomes. The benefits of marine conservation highlighted in the app include enhanced carbon sequestration, coastal protection, biodiversity, and the reproductive capacity of marine organisms, as well as fisheries catches and income.

Conclusion of R Shiny, MPAs, and Improving Systematic Literature ReviewOur app provides a solution to the scientific community and service to the general public, bridging the gap between science and scientific literacy. By moving evidence closer and making it easier to consume, we hope to drive people to make more evidence-based decisions.

We believe that R Shiny is an excellent tool for creating interactive scientific papers. And it opens up the chance to cultivate wider scientific literacy, both for laymen and the experts.

Using our app, users can easily explore the impact of MPAs on various pathways and see how these impacts vary geographically. The app provides visualizations that illustrate the geographical distribution of evidence disaggregated by pathway with the sign of this evidence, either qualitative “vote counting” or quantitative meta-analysis outcomes.

The benefits of marine conservation highlighted in the app include enhanced carbon sequestration, coastal protection, biodiversity, and the reproductive capacity of marine organisms, as well as fishers’ catch and income.

Our app offers a unique way to explore marine conservation and the benefits of MPAs in mitigating and adapting to climate change. We hope that it will inspire scientists to share better and for the public to explore more evidence-based decision-making – ultimately, for the betterment of everyone and everything.

With R and Shiny, we can unlock the full potential of scientific literature and climate policy.

Explore more Data4Good projects and discover how R/Shiny and AI can power your projects!

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Systematic Literature Review and R Shiny: App for Marine Conservation and Climate Change Mitigation

View Details

[This article was first published on R – QuantStrat TradeR, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.So, it’s been a little while. But after a couple of years of some grunt work analytics jobs and consulting for a $1B AUM fund, I’ve decided that I had a bit more in the tank to share as far as quant content creation–quantent creation (?)–goes.

And a function I’ve searched for in Python for a long time now, but never finding it in a proper capacity is one that we’ve seen used time and again on this blog–R’s Return.portfolio.

For those uninitiated, Return.portfolio in R is essentially the workhorse of portfolio-level asset allocation backtests. What it does can be summarized in a sentence: given an xts/Python series dataframe of weights with m (E.G. 10) assets, and an xts/Python series dataframe with returns of m assets, Return.portfolio will compute drifted portfolio returns, along with beginning of period weights, end of period weights, beginning/end of period values, contribution to returns, and in the case of the Python variant that I wrote, two-way turnover.

Now, why is this function vital to have in Python, in my opinion?

Simple: because onto my experience with the language, if you want to do any financial research with Python, you need to learn some arcane and convoluted syntax such as zipline or quantconnect, and in some cases, that depends on working through another website altogether (in the case of quantconnect), or in the case of zipline, a library developed by a company that no longer exists (quantopian), so in either case, using either of these libraries may not be the best of ideas, even if you put in the who-knows-how-many-hours to learn a bunch of extra syntax fighting for real estate in your head.

Instead, what porting R’s Return.portfolio into Python does is to allow the same exact numpy/scipy/pandas ye-olde-classic-data-science syntax to not only work with backtesting, but also to seamlessly link up with whatever other Python library you may want to use, such as cvxpy for your portfolio optimization solutions (having used this library on past consulting arrangements, I am very impressed).

Now, here’s the link to my github for the code for Return.portfolio in Python–or, rather, since the dot operator has a special use in Python, the Return_portfolio function.

But for the record, here’s the Python code, including the endpoints function.

def endpoints(df, on = "M", offset = 0): """ Returns index of endpoints of a time series analogous to R's endpoints function. Takes in: df -- a dataframe/series with a date index on -- a string specifying frequency of endpoints (E.G. "M" for months, "Q" for quarters, and so on) offset -- to offset by a specified index on the original data (E.G. if the data is daily resolution, offset of 1 offsets by a day) This is to allow for timing luck analysis. Thank Corey Hoffstein. """ # to allow for familiarity with R # "months" becomes "M" for resampling if len(on) > 3: on = on[0].capitalize() # get index dates of formal endpoints ep\_dates = pd.Series(df.index, index = df.index).resample(on).max() # get the integer indices of dates that are the endpoints date\_idx = np.where(df.index.isin(ep\_dates)) # append last day to match R's endpoints function # remember, Python is indexed at 0, not 1 #date\_idx = np.insert(date\_idx, 0, 0) date\_idx = np.append(date\_idx, df.shape[0]-1) if offset != 0: date\_idx = date\_idx + offset date\_idx[date\_idx < 0] = 0 date\_idx[date\_idx > df.shape[0]-1] = df.shape[0]-1 out = np.unique(date\_idx) return out def compute\_weights\_and\_returns(subset\_returns, weights): rep\_weights = np.tile(weights, (len(subset\_returns), 1)) cum\_subset\_weights = np.cumprod(1 + subset\_returns, axis=0) * rep\_weights EOP\_Weight = cum\_subset\_weights / np.sum(cum\_subset\_weights, axis=1).to\_numpy()[:, np.newaxis] cum\_subset\_weights\_bop = cum\_subset\_weights / (1 + subset\_returns) BOP\_Weight = cum\_subset\_weights\_bop / np.sum(cum\_subset\_weights\_bop, axis=1).to\_numpy()[:, np.newaxis] portf\_returns\_subset = pd.DataFrame(np.sum(subset\_returns.values * BOP\_Weight, axis=1), index=subset\_returns.index, columns=['Portfolio.Returns']) return [portf\_returns\_subset, BOP\_Weight, EOP\_Weight]# Return.portfolio.geometric from Rdef Return\_portfolio(R, weights=None, verbose=True, rebalance\_on='months'): """ Parameters ---------- R : a pandas series of asset returns weights : a vector or pandas series of asset weights. verbose : a boolean specifying a verbose output containing: portfolio returns, beginning of period weights and values, end of period weights and values, asset contribution to returns, and two-way turnover calculation rebalance\_on : a string specifying rebalancing frequency if weights are passed in as a vector. Raises ------ ValueError Number of asset weights must be equal to the number of assets. Returns ------- TYPE See verbose parameter for True value, otherwise just portfolio returns. """ # make sure original object isn't overridden R = R.copy() weights = weights.copy() # impute NAs in returns if R.isna().sum().sum() > 0: R.fillna(0, inplace=True) wn.warn("NAs detected in returns. Imputing with zeroes.") # if no weights provided, create equal weight vector if weights is None: weights = np.repeat(1/R.shape[1], R.shape[1]) wn.warn("Weights not provided, assuming equal weight for rebalancing periods.") # if weights aren't passed in as a data frame (they're probably a list) # turn them into a 1 x num\_assets data frame if type(weights) != pd.DataFrame: weights = pd.DataFrame(weights) weights = weights.transpose() # error checking for same number of weights as assets if weights.shape[1] != R.shape[1]: raise ValueError("Number of weights is unequal to number of assets. Correct this.") # if there's a row vector of weights, create a data frame with the desired # rebalancing schedule -- also add in the very first date into the schedule if weights.shape[0] == 1: if rebalance\_on is not None: ep = endpoints(R, on = rebalance\_on) first\_weights = pd.DataFrame(np.array(weights), index=[R.index[0]], columns=R.columns) weights = pd.concat([first\_weights, pd.DataFrame(np.tile(weights, (len(ep), 1)), index=R.index[ep], columns=R.columns)], axis=0) #weights = pd.DataFrame(np.tile(weights, (len(ep), 1)), # index = ep.index, columns = R.columns) weights.index.name = R.index.name else: weights = pd.DataFrame(weights, index=[R.index[0]], columns=R.columns) original\_weight\_dim = weights.shape[1] # save this for computing two-way turnover if weights.isna().sum().sum() > 0: weights.fillna(0, inplace=True) wn.warn("NAs detected in weights. Imputing with zeroes.") residual\_weights = 1 - weights.sum(axis=1) if abs(residual\_weights).sum() != 0: print("One or more periods do not have investment equal to 1. Creating residual weights.") weights["Residual"] = residual\_weights R["Residual"] = 0 if weights.shape[0] != 1: portf\_returns, bop\_weights, eop\_weights = [], [], [] for i in range(weights.shape[0]-1): subset = R.loc[(R.index >= weights.index[i]) & (R.index <= weights.index[i+1])] if i >= 1: # first period include all data # otherwise, final period of previous period is included, so drop it subset = subset.iloc[1:] subset\_out = compute\_weights\_and\_returns(subset, weights.iloc[i,:]) subset\_out[0].columns = ["Portfolio.Returns"] portf\_returns.append(subset\_out[0]) bop\_weights.append(subset\_out[1]) eop\_weights.append(subset\_out[2]) portf\_returns = pd.concat(portf\_returns, axis=0) bop\_weights = pd.concat(bop\_weights, axis=0) eop\_weights = pd.concat(eop\_weights, axis=0) else: # only one weight allocation at the beginning and just drift the portfolio out = compute\_weights\_and\_returns(R, weights) portf\_returns = out[0]; portf\_returns.columns = ['Portfolio.Returns'] bop\_weights = out[1] eop\_weights = out[2] pct\_contribution = R * bop\_weights cum\_returns = (1 + portf\_returns).cumprod() eop\_value = eop\_weights * pd.DataFrame(np.tile(cum\_returns, (1, eop\_weights.shape[1])), index=eop\_weights.index, columns=eop\_weights.columns) bop\_value = bop\_weights * pd.DataFrame(np.tile(cum\_returns/(1+portf\_returns), (1, bop\_weights.shape[1])), index=bop\_weights.index, columns=bop\_weights.columns) turnover = (np.abs(bop\_weights.iloc[:, :(original\_weight\_dim-1)] - eop\_weights.iloc[:, :(original\_weight\_dim-1)].shift(1))).sum(axis=1).dropna() turnover = pd.DataFrame(turnover, index=eop\_weights.index[1:], columns=['Two-way turnover']) out = [portf\_returns, pct\_contribution, bop\_weights, eop\_weights, bop\_value, eop\_value, turnover] out = {k: v for k, v in zip(['returns', 'contribution', 'BOP.Weight', 'EOP.Weight', 'BOP.Value', 'EOP.Value', 'Two.Way.Turnover'], out)} if verbose: return out else: return portf\_returns Rather than go into it line by line, those interested can read the code, but ultimately, a lot of it is basically seeing if the weights were passed in as a data frame of individually customized weights (E.G. on January, my weights were such and such, and on February, they shifted to some other such and such), or if the weights were passed in as a vector, or Python list, if someone just wants to rebalance a portfolio (E.G. a classic buy-and-hold 60/40), and if some portfolio weights don’t add up to 1. Essentially, a fair bit of bookkeeping. (Speaking of, I’m not sure what happened to the programming language customization block here on wordpress, so, I sincerely encourage readers to check my github for this code).

Now, here’s the interesting part which motivated this post:

I didn’t write most of this code in Python. Or rather, not initially.

I wrote it in R, first, to understand it, with R’s vectorization, where I’m a bit more comfortable. (As an aside, as far as languages for quantitative finance go, I think R has the much more advanced buy-side libraries such as PortfolioAnalytics or Quantstrat compared to Python, as they were written by high-level industry practitioners, whereas Python’s finance ecosystem seems to be…fairly threadbare, consisting of various islands of libraries that don’t really play together all that well.)

Here’s the R code that I did to rewrite Peter Carl’s Return.portfolio.geometric (the underlying function that runs Return.portfolio that I’ve used all these years):

compute\_weights\_and\_returns <- function(subset, weights) { rep\_weights <- matrix(nrow=nrow(subset), ncol = ncol(subset), weights, byrow = TRUE) cum\_subset\_weights <- cumprod(1+subset) * rep\_weights EOP.Weight <- cum\_subset\_weights/rowSums(cum\_subset\_weights) cum\_subset\_weights\_bop <- cum\_subset\_weights/(1+subset) BOP.Weight <- cum\_subset\_weights\_bop/rowSums(cum\_subset\_weights\_bop) portf\_returns\_subset <- xts(rowSums(subset * BOP.Weight), order.by=index(subset)) return(list(portf\_returns\_subset, BOP.Weight, EOP.Weight))}Return.portfolio.geometric.ilya <- function(R, weights, verbose = TRUE, rebalance\_on = 'months') { if(sum(is.na(R)) > 0) { R[is.na(R)] <- 0 warning("NAs detected in returns. Imputing with zeroes.") } if(missing(weights)) { weights <- rep(1/ncol(R), ncol(R)) warning("Weights not provided, assuming equal weight for rebalancing periods.") } # if weights passed in as vector if(is.null(dim(weights))) { if(!is.null(rebalance\_on)) { ep <- endpoints(R) first\_weights <- xts(t(matrix(weights)), order.by=index(R)[1]) weights <- rbind(first\_weights, xts(matrix(nrow=length(index(R)[ep]), ncol = length(weights), weights, byrow=TRUE), order.by = index(R)[ep])) # get the first day of all the weights, make sure it's unique # weights <- weights[!duplicated(index(weights)),] } else { weights <- xts(t(matrix(weights)), order.by=index(R)[1]) } colnames(weights) <- colnames(R) } original\_weight\_dim <- ncol(weights) # save this for computing two-way turnover if(original\_weight\_dim != ncol(R)) { stop("Number of weights is unequal to number of assets. Correct this.") } if(sum(is.na(weights)) > 0) { weights[is.na(weights)] <- 0 warning("NAs detected in weights. Imputing with zeroes.") } residual\_weights <- 1-rowSums(weights) if(sum(abs(residual\_weights)) != 0) { warning("One or more periods do not have investment equal to 1. Creating residual weights.") weights$Residual <- residual\_weights R$Residual <- 0 } if(nrow(weights) != 1) { portf\_returns <- bop\_weights <- eop\_weights <- list() for(i in 1:(nrow(weights)-1)) { subset <- R[paste((index(weights)[i]),index(weights)[i+1], sep = "::"),] if(i >= 2) { # first period include all data # otherwise, final period of previous period is included, so drop it subset <- subset[-1,] } subset\_out <- compute\_weights\_and\_returns(subset, weights[i,]) colnames(subset\_out[[1]]) <- "Portfolio.Returns" portf\_returns[[i]] <- subset\_out[[1]] bop\_weights[[i]] <- subset\_out[[2]] eop\_weights[[i]] <- subset\_out[[3]] } portf\_returns <- do.call(rbind, portf\_returns) bop\_weights <- do.call(rbind, bop\_weights) eop\_weights <- do.call(rbind, eop\_weights) } else { # only one weight allocation at the beginning and just drift the portfolio out <- compute\_weights\_and\_returns(R, weights) portf\_returns <- out[[1]]; colnames(portf\_returns) <- "Portfolio.Returns" bop\_weights <- out[[2]] eop\_weights <- out[[3]] } pct\_contribution <- R * bop\_weights cum\_returns <- cumprod(1+portf\_returns) eop\_value <- eop\_weights * matrix(nrow=nrow(eop\_weights), ncol = ncol(eop\_weights), cum\_returns, byrow= FALSE) bop\_value <- bop\_weights * matrix(nrow=nrow(bop\_weights), ncol = ncol(bop\_weights), cum\_returns/(1+portf\_returns), byrow = FALSE) # add turnover computation because that's what this whole exercise is about turnover <- na.omit(xts(rowSums(abs(bop\_weights[,1:original\_weight\_dim] - lag(eop\_weights[,1:original\_weight\_dim]))), order.by = index(eop\_weights))) colnames(turnover) <- "Two-way turnover" out <- list(portf\_returns, pct\_contribution, bop\_weights, eop\_weights, bop\_value, eop\_value, turnover) names(out) <- c("returns", "contribution", "BOP.Weight", "EOP.Weight", "BOP.Value", "EOP.Value", "Two.Way.Turnover") if(verbose) { return(out) } else { return(portf\_returns) }} Now, how did I go from the above to the code at the beginning of this post?

ChatGPT4.

That is, I translated my R code into Python code using ChatGPT4 by simply having it translate, block by block, my R code into Python.

Now, was it perfect? No, actually. I had to run the code block by block, and do a little bit of debugging here and there–including adding the functionality for the Return_portfolio function to make deep copies of the returns and weights, instead of modifying the original data structure passed into the function, since Python by default is a pass by reference language, not pass by value (ugh). However, this sort of “Rosetta Stone” functionality inherent in ChatGPT4 is occasionally useful.

Now, when I first tried prompting chatGPT4 to just translate R’s functions, without guiding it step by step, it was a total disaster. Contrary to popular belief, ChatGPT4 is not some all-powerful, all-perfect, all-knowing, and all-wise (as the late, great, George Carlin once said) system. But occasionally, it can be useful.

Furthermore, as someone that’s used AI to try my hand at image generation, I also consider those systems (such as StableDiffusion, or Leonardo.AI) to be…somewhat impressive, if in their infancy right now (it gets much more difficult to have multiple subjects interact on one image–I.E. if you want to describe one person in an image, that’s one thing, but if you want to describe two, it isn’t like you can make an image of one–say, save it with a name, such as Alice.jpeg, make an image of a second, save it to another name, such as Bob.jpeg, and then say “Alice being hugged by Bob”). Of course, I’ve definitely been aware about the copyright hubbub brewing over the image generating AI space, but as someone that’s given away plenty of code over the past 8 or so years, including code that has been augmented to run $87 million at a $1B AUM fund, I’m pretty squarely on the open source side, even if seeing my code being responsible for running more than $100 million in AUM, and not getting paid residuals on the management fees, has discouraged me from continuing to share strategy research (I was always of the opinion that anything I shared using freely available Yahoo data was something that large investment houses had already passed over and run a more sophisticated variant of–apparently, this isn’t the case, so giving away strategy code for free nowadays seems like a very suspect thing to do).

In any case, using image-generating AI has allowed me to make artwork the likes of which I never dreamed of creating before, so I’m of the opinion that a few people taking the side of IP hoarding corporations (such as big pharma hoarding insulin patents, or Disney continuing to milk Mickey Mouse) can go kick rocks, and that IP law has long overreached. That said, I already have StableDiffusion running on my machine, coupled with some more sophisticated models built on top of it (Dreamshaper v5, Deliberate v2, etc.), and still have a lot to learn about it–though I’ll probably need a more powerful machine than a gaming laptop to really crank images out.

Now, my thoughts on AI? So far, I don’t think that AI alone is going to pull a South Park and DERK YERRR JERBZ. At least if you’re a skilled professional.

Not alone, anyway. And for what it’s worth, as a quantitative research analyst, I found that I’d fire ChatGPT4 within a few hours, owing to the artifact of AI hallucinations. As the phrase goes, trust but verify. I think we’re still far, far away from ChatGPT4 being able to read a strategy paper on Quantpedia, replicating it in R or Python, running the code, and replicating the results. For that matter, I think that any strategy paper submission or blog post is incomplete without data and code, for immediate replication, ready to go. It’s the 21st century. Github or git out. A minimum reproducible example (MRE) as a demo of the concept (even a toy one), in my opinion, is vastly more useful than 25 pages of mathematical derivation. Though that’s just me speaking given my background of hands-on engineering and actually doing things rather than merely theorizing about them.

Okay, so…this post seems to have gotten long in the tooth. In other news, I’m in the job market, currently, and as I’ve shown, I have a fairly solid command of Python, having several years of off and on experience with it. And I’m also wondering if a written blog is the best way to actually communicate so many of my findings, or if YouTube is the way to go (though I don’t know much about editing videos).

So…yeah. Follow if you like the post, and most importantly:

Try to lose money…less than I do.

Thanks for reading.

My LinkedIn.

To leave a comment for the author, please follow the link and comment on their blog: R – QuantStrat TradeR.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: This function VITAL for portfolio backtesting is now in Python, written with the help of chatGPT4

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to use web scraping in R! Join our workshop on Intermediate Web Scraping and API Harvesting using R which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: Intermediate Web Scraping and API Harvesting using R

Date: Thursday, June 15th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Felix Lennert is a second-year Ph.D. student in Sociology at the CREST, ENSAE, Institut Polytechnique de Paris. He is a co-organizer of the Summer Institute of Computational Social Science in Paris. His research interests lie in the formation and polarization of political opinions which he tackles using a toolbox consisting of “classic” quantitative as well as text-as-data methods.

Description: Digital trace data are an integral element of CSS (cool social scientific) research. This course will show you how this is done on an intermediate level. This implies that we will not cover the fundamentals of selecting and downloading things from static web pages on the one hand, but also not go as far as firing up RSelenium to scrape dynamic web pages on the other. We will start with a brief revision of CSS selectors, then we move on to rvest to simulate a browser session, fill forms, and click buttons. In the second half of the session, APIs and how to make requests to them will be covered. Tangible examples for API queries will be shown. In the end, exemplary workflows will be introduced to provide a scaffolding for students’ future research projects.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Intermediate Web Scraping and API Harvesting using R workshop was first posted on May 3, 2023 at 2:06 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Intermediate Web Scraping and API Harvesting using R workshop

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionThe download.file() function in R is used to download files from the internet and save them onto your computer. Here’s a simple explanation of how to use it:

Specify the URL of the file you want to download.
Specify the file name and the location where you want to save the file on your computer.
Call the download.file() function, passing in the URL and file name/location as arguments.

Here’s an example:

```

Specify the URL of the file you want to downloadurl <- "https://example.com/data.csv"# Specify the file name and location where you want to save the file on your computerfile_name <- "my_data.csv"file_path <- "/path/to/save/folder/"# Call the download.file() function, passing in the URL and file name/location as argumentsdownload.file(url, paste(file_path, file_name, sep = ""), mode = "wb")

``` In this example, we’re downloading a CSV file from “https://example.com/data.csv”, and saving it as “my_data.csv” in the “/path/to/save/folder/” directory on our computer.

The mode = “wb” argument specifies that we want to download the file in binary mode.

Once you run this code, the file will be downloaded from the URL and saved to your specified file location.

Let’s try a working example.

ExampleWe are going to download the Measure Dates file from the following location: {https://data.cms.gov/provider-data/dataset/4j6d-yzce}

url <- "https://data.cms.gov/provider-data/sites/default/files/resources/49244993de5a948bcb0d69bf5cc778bd\_1681445112/Measure\_Dates.csv"file\_name <- "measure\_dates.csv"file\_path <- "C:\\Downloads\\"download.file(url = url, destfile = paste0(file\_path, file\_name, sep = "")) Now let’s read in the file in order to make sure it actually downloaded.

measure\_dates\_df <- read.csv(file = paste0(file\_path, file\_name))dplyr::glimpse(measure\_dates\_df)

Rows: 170Columns: 6$ Measure.ID <chr> "ASC\_11", "ASC\_12", "ASC\_13", "ASC\_14", "ASC\_17"…$ Measure.Name <chr> "Percentage of patients who had cataract surgery…$ Measure.Start.Quarter <chr> "1Q2021", "1Q2019", "1Q2021", "1Q2021", "3Q2020"…$ Start.Date <chr> "01/01/2021", "01/01/2019", "01/01/2021", "01/01…$ Measure.End.Quarter <chr> "4Q2021", "4Q2021", "4Q2021", "4Q2021", "4Q2021"…$ End.Date <chr> "12/31/2021", "12/31/2021", "12/31/2021", "12/31… Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Download a File from the Internet using download.file()

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

We’ve just passed an important milestone for diffify:our app for tracking Python and R package releases has just turned 1 year old!To mark this exciting occasion we are delighted to announce an “anniversaryupdate” featuring numerous quality of life improvements. This post will outlinethe latest changes and tease at some exciting developments in the works…

First, though, we would like to take this opportunity to thank everyone thatcontinues to use the app and welcome any new users to the service.Your continued feedback via social media andGitHub has played a majorrole in shaping the last year of development.

Data comes in all shapes and sizes. It can often be difficult to know where to start. Whatever your problem, Jumping Rivers can help.

Anniversary updateLet’s start by going through the changes introduced by today’s anniversaryupdate!

Latest package releasesWhen you navigate to the R andPython homepages, you will notice a newwindow titled “Latest Releases and Updates”:

This lists any new or updated packages that have been published in the past dayor so. See a package that you’re using? Just click on it and you will beredirected to the diffify summary with the latest changes.

Package dependenciesIn response to user feedback, we have added cross-links for packagedependencies. Let’s check out the changesbetween versions 3.6.3 and 3.7.1 of the matplotlib package:

We see that the version requirement has changed for the numpy andpyparsing packages. You may now be wondering what’s changed in the latestversions of those packages? Just click the link icons and that will open a newtab with the two latest versions diffed.

Quick disclaimer that not all package dependencies will have cross-links. We canonly provide cross-links for packages that are actually tracked by diffify,which includes:

All R packages published on CRAN (this doesnot include base-R packages)
Any Python package that is in the top 5000 PyPI packageslist and has an accessible wheel file on PyPI

News layoutWe have made some changes to the way we display news for R packages. Let’s checkout the changes between versions1.0.7 and 1.0.10 of {dplyr}. As before, the news can be accessed for allversions since (but not including) the earlier version:

However, you’ll notice we now have an accordion layout with the version tabslisted vertically. You are now free to have as many of these versions open asyou like, and scrolling through these will feel just like scrolling through aNEWS.md file.

Dark themeLast but not least … we now have a dark theme! Just click the theme dropdownat the top of the page, select “Theme: Dark” and enjoy this lower-light setting:

On the topic of themes, we have also improved the default theme by incorporatingbeneficial features from the old boosted contrast theme.

Other recent changesIn case you missed them, here are some other improvements that have been madeover the past six months or so.

Maintainer sectionJust below the version dropdowns you will notice a new maintainer section:

If you maintain a package that is featured on diffify, you can generate adiffify badge to copy into your GitHub repository. Simply click “Get a badge”,then paste the copied HTML code directly into an HTML or Markdown file (perhapsyour package README).

As an example, here’s the badge generated for the {dplyr} package:

Clicking this icon will redirect users to the {dplyr} page on diffify.

Python contentWe have expanded the list of Python packages that are tracked by diffify tocover the top 5000 packages on PyPIaccording to download counts. We are still only tracking packages that have awheel file on PyPI, but will look to expand this to zips and tars within thenext month.

UsabilityWe are continuing to optimise the usability and performance of the app. Recentimprovements include:

text wrapping on narrow screens
smoother transitions using the backward and forward navigation buttons
improvements to keyboard navigation.

Exciting times ahead…In the coming months we will be releasing two public APIs to accompany diffify.We will release dedicated blogs to coincide with those releases, but here’s aquick overview to whet your appetite:

Next month we will release an API which will allow R package authors to submitdevelopment versions to diffify. Package authors and users will then be ableto use diffify to view changes between published versions and the latestdevelopment version.
The second API, which will take a little longer to develop, will act as acommand-line interface for submitting queries to diffify. This will allow youto check whether installing the latest version of a package could break yourcode.

We can’t wait to share more when these release!

Wrapping upThat’s all from us for today. Thanks again for your continued feedback on theapp, and please stay tuned for more updates…

For further reading, you can check out our previous blog postshere!

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Diffify – the anniversary update!

View Details

[This article was first published on Louise E. Sinks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Today’s TidyTuesday is about the Portal Project, which is a long terms study on the ecology of Arizona. The study explores how ants, rodents, plants respond to climate in the desert near Portal, Az. A subset of the data is provided for this week’s TidyTuesday.

Loading libraries. Not really using anything fancy today!

library(tidyverse)library(ggthemes) Loading the data.

tuesdata <- tidytuesdayR::tt\_load(2023, week = 18)

--- Compiling #TidyTuesday Information for 2023-05-02 ----

--- There are 3 files available ---

--- Starting Download ---

Downloading file 1 of 3: `plots.csv` Downloading file 2 of 3: `species.csv` Downloading file 3 of 3: `surveys.csv`

--- Download complete ---

plots <- tuesdata$plotsspecies <- tuesdata$speciessurveys <- tuesdata$surveys This dataset focuses on rodents. The portal project also studies plants and ants, but they are ommitted from these datasets.

glimpse(plots)

Rows: 8Columns: 2$ plot <dbl> 3, 4, 11, 14, 15, 17, 19, 21$ treatment <chr> "exclosure", "control", "control", "control", "exclosure", "… We have information about 8 plots in the plots dataframe, and they are coded as exclosure or control. It isn’t clear what exclosure or control means. The portal website says this about treatments of the plots:

“Rodents are manipulated using gates in the fencing of each plot. Rodent removals contain no gates and any rodents captured on those plots are removed. All other plots contain 16 gates (4 per plot side); gates consist of holes cut through the hardware cloth of the fencing. Gate size is used to exclude subsets of the rodent community (larger gates allow all rodents access, smaller gates exclude kangaroo rats). Dimensions for gates on kangaroo rat removal plots are 1.9 cm x 1.9 cm, D. spectabilis removals were 2.6 cm x 3.0 cm, and control plots are 3.7 cm x 5.7 cm. In 2005, Dipodomys spectabilis removals were converted to controls – a state these plots had effectively been in with the local extinction of Dipodomys spectabilis in the late 1990s. Species caught on plots from which they are supposed to be excluded are removed from the site and the access point to the plot is located and eliminated. Plots affected by these treatments are listed in Portal_plot_treatments.”

Presumably, the exclosures are those plots designed with gates of specific sizes, designed to exclude specific size rodents. It is less clear what a control is. The text says “Dipodomys spectabilis removals were converted to controls”, which is also unclear because it also says removals contain no rodents, so the species part is confusing. This page suggests controls are “unmanipulated controls” so maybe there is no fence at all?

Looking at the history of the plots, we see that the exclosure plots all have had kangaroo rats removed from them at certain times.

So, do we find fewer Kangaroo rates on those plots? The survey data includes the rodent type by code; the key is found in the species data. Let’s pull out the kangaroo rat code(s).

rats <- species[(str\_detect(species$commonname, "kangaroo")), ]rats

```

A tibble: 3 × 15 species scientificname taxa commonname censustarget unidentified rodent 1 DM Dipodomys merriami Rode… Merriam's… 1 0 12 DO Dipodomys ordii Rode… Ord's kan… 1 0 13 DS Dipodomys spectabil… Rode… Banner-ta… 1 0 1# ℹ 8 more variables: granivore , minhfl , meanhfl ,# maxhfl , minwgt , meanwgt , maxwgt , juvwgt

``` We have three types of Kangaroo rate, coded DM, DO, and DS.

Let’s make sure the survey data only includes the plots we know about from the plots dataframe.

surveys %>% group\_by(plot) %>% count(plot)

```

A tibble: 11 × 2# Groups: plot [11] plot n 1 3 3580 2 4 3928 3 10 469 4 11 3640 5 14 3625 6 15 2106 7 16 1079 8 17 4023 9 19 225610 21 268111 23 977

``` We definitely have plots not in our plot dataframe. This is a small enough dataset that you can just look at the two lists and see we have 10, 16, and 23 as extra plots. If we go back to the history of the plots page, we can see that these three plots all had all rodents removed at several times over the course of the project. I’ll come back to that, but first I want to demonstrate how we can find these extra plots using a join. For larger datasets, it might not be possible to determine the extra plots by inspection.

To do this, I’m going to use an antijoin from dplyr. The syntax is anti_join(x , y) and returns rows of x that do not have a match in y. So here, we want x to be the plots from the survey data, and y to be the plots from out plots. I’ll just build off the summary pipe from the previous code block.

surveys %>% group\_by(plot) %>% count(plot) %>% anti\_join(plots)

Joining with `by = join\_by(plot)`

```

A tibble: 3 × 2# Groups: plot [3] plot n 1 10 4692 16 10793 23 977

``` Doing this type of basic check is really important when you start working with a new dataset. It looked like we had two treatments, but there are actually 3. If you had missed this, you could have lumped the third treatment in with one of the other two while analyzing, and obtained incorrect results.

I’m going to store these other plots numbers, both as an array with counts and as a 1D object of just the plot numbers. Depending on what I decide to do, I might annotate the plots dataframe to include this data.

extra\_plots\_array <- surveys %>% group\_by(plot) %>% count(plot) %>% anti\_join(plots)

Joining with `by = join\_by(plot)`

extra\_plots\_array

```

A tibble: 3 × 2# Groups: plot [3] plot n 1 10 4692 16 10793 23 977

```

extra\_plots <- extra\_plots\_array$plot There are a couple of different things that could be explored. The treatments were not applied over every time block. We can look at how persistent a given treatment is. Plot 3 had the kangaroo rats removed in three of the five time blocks (1977-1985, 1988-2004, 2010-2015). Does the survey data reflect this?

I’m going to pull out the plot 3 survey data.

plot3\_survey <- surveys %>% filter(plot == 3) There are a few different ways I could go. I’m going to create two groups of rodents: kangaroo rats and others. “NAs” are going in other. These are rodents that were caught and not identified. Perhaps using the size data, it might be possible to impute the species of rodent, but that requires more subject matter knowledge than I have.

NAs are annoying, so I’ll handle them first. Then I’ll recode everything else to be kangaroo or other. I’m just hard coding from the species name, but I could use the species labels I saved earlier (rats$species).

plot3\_survey\_recode <- plot3\_survey %>% mutate(species = ifelse(is.na(species) == TRUE, "OTHER", species)) %>% mutate(species = recode(species, "DM" = "KAN", "DO" = "KAN", "DS" = "KAN", .default = "OTHER")) So, first, did the treatment work?

plot3\_survey\_recode %>% group\_by(species) %>% count(species)

```

A tibble: 2 × 2# Groups: species [2] species n 1 KAN 2072 OTHER 3373

``` Looks like it did. But we should really compare to a control plot. Plot 4 is a control plot.

plot4\_survey <- surveys %>% filter(plot == 4) %>% mutate(species = ifelse(is.na(species) == TRUE, "OTHER", species)) %>% mutate(species = recode(species, "DM" = "KAN", "DO" = "KAN", "DS" = "KAN", .default = "OTHER")) %>% group\_by(species) %>% count(species)plot4\_survey

```

A tibble: 2 × 2# Groups: species [2] species n 1 KAN 21622 OTHER 1766

``` In this plot, 55% of the captured rodents are kangaroo rats, compared to about 6% in plot 3. Again, to be completely rigorous, we should probably compare data aggregated over all plots with the same treatment types, rather than a single plot from treatment and single plot from control. These weren’t even randomly picked; I chose the first one of each type on the list.

Did the kangaroo rat populations increase in years without treatment?

plot3\_survey\_recode %>% filter (species == "KAN") %>% group\_by(year) %>% count(species) %>% ggplot(aes(year, n)) + geom\_point() + annotate( "rect", xmin = 1977, xmax = 1985, ymin = 0, ymax = 40, alpha = .1, fill = "red" ) + annotate( "rect", xmin = 1988, xmax = 2004, ymin = 0, ymax = 40, alpha = .1, fill = "red" ) + annotate( "rect", xmin = 2010, xmax = 2015, ymin = 0, ymax = 40, alpha = .1, fill = "red" ) + ylab("# of Kangaroo rats") + xlab("Year") + labs(title = "Persistance of treatment on Plot 3", subtitle = "Kangaroo Rats removed in red periods", caption = "Data from https://portal.weecology.org/") + theme\_pander() So that’s pretty interesting. The treatment did seem to be persistent. The years with the highest numbers of kangaroo rats were in times when they were actively being removed. (Perhaps the researchers were more diligent about identifying the rat species in removal period. This might be reflected in have fewer or proportionally fewer “NAs” during removal times.)

CitationBibTeX citation: @online{e.sinks2023, author = {Louise E. Sinks}, title = {TidyTuesday {Week} 18: {Portal} {Project}}, date = {2023-05-02}, url = {https://lsinks.github.io/posts/2023-05-02-tidytuesday-portal-project/portal}, langid = {en}} For attribution, please cite this work as:Louise E. Sinks. 2023. “TidyTuesday Week 18: PortalProject.” May 2, 2023. https://lsinks.github.io/posts/2023-05-02-tidytuesday-portal-project/portal. To leave a comment for the author, please follow the link and comment on their blog: Louise E. Sinks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: TidyTuesday Week 18: Portal Project

View Details

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This is a great opportunity and I am honoured to be hosting this months T-SQL Tuesday blogging invitation. With the invitation of Steve, we have agreed to post topic on Data science.

I will be receiving all of your answers on blog posts and twitter (make sure to add #tsql2sday).

Data Science in the time of Chat GPTInstead of writing and asking Data science questions, let’s discuss the aspects of Data science with the presence of Chat GPT 4.0.

By now, it is known to everyone that Chat GPT is a language model (LLM – Large Language Model) that is based on the GPT (Generative Pre-trained Transformer) architecture. It uses deep learning algorithms to like neural nets with billions of weights and transformers, that generated the sequence of tokens, that make up a piece of text.Transformers introduce the concept of “paying attention” to generally build better sequence of text. It operates primarily with probabilities of words and their sequence and therefore it is also good for human-like responses to natural language queries, making it great for a conversation-like experience.

There are many of the caveats hidden in the processing of text, adjustments of weights, functions (different and tweaked versions of Relu), additional corpora and billions of text for model training and many additional texts.

I have prepared two groups of questions. And I will not go into debate, as the end of data science is near, nor will go into debate, that the AGI (artificial general intelligence) will completely replace the role of data scientists. What I want to hear from you is simply how did you embrace (if at all) the use of Chat GPT, and what were your first impressions. And mostly, how did it help you (if at all), what did you use it for, and have you encountered any traps?

Usage and working along Chat GPTImagine using SQL, R, Python, Julia, or Scala, for your daily data science work. And you can practically ask Chat GPT anything and it will return you a relatively coherent and good answer. If you need an explanation, it will excel. Where and what have you used it for? Here is a short list, that might get you started:

Explain the data science algorithn?
Help tune or create SQL code to query big data
Prepare R, Python, Scala code for exploring the data
Help you prepare the training of the model in desired language
Prepare the code for hyperparameter tunning and cross-validation
Ask for data visualization for given dataset
Help create dashboard
Create code for model deployment, model re-training or model consumption
Ask for preparing custom functions and algorithm/function adjustments?

Now, that you have added and found the list of where and how it did help you, I would like to understand, how did this help you? Feel free to make a general comparison and add some explanations. And lastly, of course, add, if this has in any kind of way compromise your work as a data scientist (in a term of embracing it in – a positive way, or in terms of a negative experience).

Responsible usageWe have seen many controversies around Chat GPT emerge. Some European Union countries have banned it, and some will so be doing it too. And the question is not only its use (as the end of humanity and empathy) but also the misuse of personal data, privacy issues and leaking of relevant, corporate information.

Have you considered responsible usage of Chat GPT? Here is again the short list for helping you:

The use of personal data retrieved from the model
Inserting sensitive (personal or company) data
Explaining the section of R, Python, Scala code, that is the property of your enterprise

Instead of this, have you tried using it more responsibly:

Using pseudo code for explanation of the algorithm
Using mock data rather than real data
Giving pseudo-code in order to receive the documentation
Skipping on sensible data (SQL schema, model information, sensible data)

So which cases have you come across? Did it have any consequences for you? Which other responsible use of Chat GPT have you also done?

My takeawaysChatGPT offers interesting answers (based on my experience and search), and it is the next step from a google search of Stackoverflow. In other words, it gives you a more focused answer. When exploring and searching forums, you might find several different solutions for a single problem, whereas here, you have to ask for another solution. And respectively, it can give you answer faster, in comparison to browsing the web. In both cases, both sides have their advantages and disadvantages, but non will assure you, that the answer is correct!

I embrace this technology as an additional learning source. But I personally do not use it as my daily driver, despite trying it out a couple of times (with mixed results; working and nonworking/useless/meaningless). It can be super helpful for entry/junior positions, but the more experienced you are, the more abstract data science work you and the more complicated topics you cover, less frequently you will presumably use it.

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: T-SQL Tuesday #162 Invitation – Data Science in the time of ChatGPT

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn this post, we are using a package called tidymodels, which provides a suite of tools for modeling and machine learning.

Now, let’s take a closer look at the code itself and how we extract a model call from a fitted workflow object.

library(tidymodels)rec\_obj <- recipe(mpg ~ ., data = mtcars)rec\_obj The first line loads the tidymodels package. Then, we create a “recipe” object called rec_obj using the recipe() function. A recipe is a set of instructions for preparing data for modeling. In this case, we are telling the recipe to use the mpg variable as the outcome or dependent variable, and all other variables in the mtcars dataset as the predictors or independent variables.

model\_spec <- linear\_reg(mode = "regression", engine = "lm")model\_spec

Linear Regression Model Specification (regression)Computational engine: lm Next, we create a “model specification” object called model_spec using the linear_reg() function. This specifies the type of model we want to use, which is a linear regression model in this case. We also specify that the model is a regression (i.e., we are predicting a continuous outcome variable) and that the model engine is “lm”, which stands for “linear model”.

wflw <- workflow() |> add\_recipe(rec\_obj) |> add\_model(model\_spec)wflw

══ Workflow ════════════════════════════════════════════════════════════════════Preprocessor: RecipeModel: linear\_reg()── Preprocessor ────────────────────────────────────────────────────────────────0 Recipe Steps── Model ───────────────────────────────────────────────────────────────────────Linear Regression Model Specification (regression)Computational engine: lm In the next section of code, we create a “workflow” object called wflw using the workflow() function. A workflow is a way of organizing the steps involved in building a machine learning model. In this case, we are using a “pipe” (|>) to sequentially add the recipe and model specification to the workflow. This means that we first add the recipe to the workflow using the add_recipe() function, and then add the model specification using the add_model() function.

wflw\_fit <- fit(wflw, data = mtcars)wflw\_fit

══ Workflow [trained] ══════════════════════════════════════════════════════════Preprocessor: RecipeModel: linear\_reg()── Preprocessor ────────────────────────────────────────────────────────────────0 Recipe Steps── Model ───────────────────────────────────────────────────────────────────────Call:stats::lm(formula = ..y ~ ., data = data)Coefficients:(Intercept) cyl disp hp drat wt 12.30337 -0.11144 0.01334 -0.02148 0.78711 -3.71530 qsec vs am gear carb 0.82104 0.31776 2.52023 0.65541 -0.19942 Finally, we fit the workflow to the data using the fit() function, which takes the workflow object (wflw) and the data (mtcars) as input. This creates a new object called wflw_fit, which is the fitted model object. This object contains various pieces of information about the fitted model, such as the model coefficients and the R-squared value.

wflw\_fit$fit$fit$fit$call

stats::lm(formula = ..y ~ ., data = data) The last line of code extracts the actual function call that was used to fit the model. This can be useful for reproducing the analysis later on.

Overall, the code you shared shows how to build a simple linear regression model using the tidymodels package in R. We start by creating a recipe that specifies the outcome variable and predictor variables, then create a model specification for a linear regression model, and finally combine these into a workflow and fit the model to the data.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Extracting a model call from a fitted workflow in {tidymodels}

View Details

[This article was first published on business-science.io, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Writing code is a slow process especially when you are first learning data science. What if you could speed it up? You can and this is how. In this free R-tip, I share a real case study where I made the working R code for my data analysis in under 30 seconds with ChatGPT.

Table of ContentsToday I share how to automate R coding with ChatGPT. Here’s what you’re learning today:

ChatGPT Prompts: The mistake you’ll make and how to get ChatGPT to write your code correctly.
2-Minute Case Study: How I connecte to an API, Downloaded Data, and Visualized on a Map with ChatGPT.
Bonus: Sneak Peek At My NEW Shiny App that extends this ChatGPT tutorial

Special Announcement: ChatGPT for Data Scientists Part 2 WorkshopWhat: ChatGPT for Data Scientists Part 2

When: Wednesday May 3rd, 2pm EST

Why Part 2? In part 1, I focused on the mistakes I was making using chatgpt (super useful). In part 2, I want to share how I’m fixing the mistakes (even more useful).

Price: Does Free sound good?

How To Join: Register Here.

R-Tips WeeklyThis article is part of R-Tips Weekly, a weekly video tutorial that shows you step-by-step how to do common R coding tasks. Pretty cool, right?

Here are the links to get set up.

Get the Code
YouTube Tutorial

This Tutorial is Available in VideoI have a companion video tutorial that walks you through how to use chatGPT for this analysis. And, I’m finding that a lot of my students prefer the dialogue that goes along with coding. So check out this video to see me running the code in this tutorial.

ChatGPT: And the Importance of Your PromptsIn my last R-Tip, I showed you how ChatGPT I made a full R Shiny App in under 10-minutes. That was insane!

But, after that R-tip, I felt a bit of a let down.I began to try more and more complex stuff.

I failed building high-end Shiny apps.
I failed doing machine learning and integrating it into apps.
And I got so frustrated that I even made a learning lab 82: ChatGPT for Data Scientists, where I showed every mistake I made in building a Production Shiny App (including the prompts, the 11 things I needed to fix in a live workshop, plus my debugging process).

But, now I’m 12 weeks into chatGPT and I’ve realized 1 thing…

How important chatgpt prompting is.Take a look at this prompt. It’s just 2 sentences:

Looks fairly straightforward, but I need to show you why this prompt works (and what mistakes I was making in the last 11 weeks of using ChatGPT).

Chatgpt Prompt Guide (for R users)The mistake I was making was not being specific enough in directing chatgpt what R code I want it to write for me.

If you look at what I’m doing now…

…I specify:

Which R Libraries <– always do this
For this case study I want to know locations of all business types in a certain area:
- Which Business Types
- Which Geographic Location

And it generates the code for me:

Steal my code (it’s legal).

Does the code work?A big problem that data scientists are facing is that chatgpt code isn’t working out of the box.

So let’s test the code. I ran it….

Steal my code (it’s legal).

… And, it runs!

But, here’s the problem…

Will your ChatGPT code work?The answer depends on how well you know the R ecosystem and can specify exactly what you want.

Question: Are you struggling with any of these tasks?

Writing R code
Doing business analysis
Building solutions that company’s want
Making a project portfolio for yourself
Making solutions for your company

Then I have a free bonus:

FREE BONUS: Attend My Free Workshop, ChatGPT for Data Scientists Part 2Inside the workshop I’ll share how I built this Shiny App with ChatGPT (extends this data analysis to an insane production app):

What: ChatGPT for Data Scientists Part 2

When: Wednesday May 3rd, 2pm EST

Why Part 2? In part 1, I focused on the mistakes I was making using chatgpt (super useful). In part 2, I want to share how I’m fixing the mistakes (even more useful).

Price: Does Free sound good?

How To Join: Register Here.

See you there!

-Matt Dancho

To leave a comment for the author, please follow the link and comment on their blog: business-science.io.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to R code faster with ChatGPT

View Details

[This article was first published on Pachá, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.AboutInstall R packages as you would do on Windows (i.e., no long compilation time). In addition, this script asks to install R development tools, Git and RStudio.

MotivationJust to save time for my future self. Hopefully, it can help people in the cyberspace too

InstructionsJust copy and paste this one-line command:

sudo bash -c "$(wget -qO- https://raw.githubusercontent.com/pachadotdev/r-packages-ubuntu/main/configure.sh)" The command will ask you to install:

R
R development tools (i.e, r-base-dev)
Git
RStudio Desktop from my PPA
BSPM (R’s Bridge to System Package Manager)

You can use it on a fresh or existing setup.

Test if it workedWhen you reopen RStudio after running the script, you’ll see an output like this:

```

install.packages("devtools")Available system packages... There are binary versions available but the source versions are later: binary sourcefs 1.6.1 1.6.2...devtools 2.4.3 2.4.5Do you prefer later versions from sources? (Yes/no/cancel) n ``` After selecting “n”, to install from binaries, R internally communicates with Ubuntu package manager, and installing ‘devtools’ takes around 10 seconds versus around 5 minutes when building from sources (10 minutes if you need to reinstall because a system dependency was missing).

One advantage of this approach is that it shall satisfy all dependencies (i.e., it will install libpq-dev when installing RPostgres).

Automatically update RStudio DesktopThe script above configures an APT repository to my server pacha.dev, which offers the same stable RStudio version as rstudio.com. The advantage is that when the server is updated and you run apt update it will offer a newer version that you can install with apt upgrade.

In other words, the script enables apt install rstudio.

Does it work on Debian/Mint/Pop/etc?Yes.

Can I audit the script?Yes, you can re-trace all the steps I followed here: https://github.com/pachadotdev/r-packages-ubuntu.

To leave a comment for the author, please follow the link and comment on their blog: Pachá.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Perfect Setup for Ubuntu and R (and how to install/update RStudio with apt install rstudio/apt update)

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A riddle from The Riddler on how many different numbers one could at most produce from six initial values and the four basic operations. In other words, how many values could the terms in

a∅(b∅{c∅[d∅(e∅f)]})

could take? (With each ∅ being one of the four operations and a,…,f the initial values or a permutation of these.) A very crude evaluation leads to an upper bound of a million possible values, forgetting that addition and multiplication are commutative, while subtraction and division are anti-commutative. I tried a brute force approach, rather than exploring the tree of possible paths, but could not approach this figure by far, the number of different values still increasing for the largest manageable number of replicas I could try. Reducing the initial values at n=3, I could come closer to 123 with 95 different values and, for n=4, not too far from 1972 with 1687 values.

To leave a comment for the author, please follow the link and comment on their blog: R – Xi'an's Og.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: operation impossible

View Details

[This article was first published on Rstats – quantixed, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I’m not a big movie person. Nonetheless I have a media library with quite a few films in and I wondered how many “films to see before you die”-type movies I had in the collection, and how many were missing. I used R to find the answers.

I’ve described previously how to get a plain text dump of a Plex database using WebTools-NG. I did that for the Movies library of my Plex Media Server. Now, for the list of “films to see before you die”. I searched a bit and found a few text files which claimed to be meta-rated as the best. I was a bit suspicious about these. In the end, I figured I should just use to the IMDB’s Top 250 Movies, which could be scraped with rvest.

The codeLet’s get the Top 250 movies:

library(rvest)library(XML)library(xml2)library(fuzzyjoin)library(dplyr)# IMDB Top 250 Movies are hereurl <- "http://www.imdb.com/chart/top?ref\_=nv\_wl\_img\_3"page <- read\_html(url)movie.nodes <- html\_nodes(page,'.titleColumn a')movie.name <- html\_text(movie.nodes)sec <- html\_nodes(page,'.secondaryInfo')# to get the year we need to remove ) and ( and then get text)year <- as.numeric(gsub(")","",gsub("\\(","",html\_text( sec ))))rating.nodes <- html\_nodes(page,'.imdbRating strong')rating <- as.numeric(html\_text(rating.nodes))imdb <- data.frame(Title = movie.name, Year = year, Rating = rating) Now we have a data frame of the movies, with Title, Year and the IMDB rating.

We can load in the Plex library so that we can match them up, but we don’t need all the data.

libfile <- file.choose()libdf <- read.delim(libfile,sep = "|")# we only need title and yearpms <- libdf %>% select(Title,Year) Now we have two data frames to perform the matching.

The first issue is that we can’t simply use the titles for matching because remakes and different versions of movies will cause a mismatch. To get around this we can use Title and Year as a combination for fuzzy matching.

```

to match movies, we need Title-Year combinationimdb$titleyear <- paste(imdb$Title, imdb$Year)pms$titleyear <- paste(pms$Title,pms$Year)# fuzzy matchingmatch <- stringdist_join(imdb, pms, by = 'titleyear', mode ='left', method = "jw", max_dist = 99, # could set this a lot lower distance_col = 'dist') %>% group_by(titleyear.x) %>% slice_min(order_by = dist, n = 1) # gives the best match found

``` Fuzzy matching is needed because a simple string comparison will get derailed pretty easily by capitalisation and other minor issues. So we need something a little more forgiving to do the matching.

Now we can have a look at the matches by typing match

```

have a look at matches> match# A tibble: 254 × 8# Groups: titleyear.x [250] Title.x Year.x Rating titleyear.x Title.y Year.y titleyear.y dist 1 12 Angry Men 1957 9 12 Angry Men 1957 12 Monkeys 1995 12 Monkeys 1995 0.282 2 12 Years a Slave 2013 8.1 12 Years a Slave 2013 Oz the Great and Powerful 2013 Oz the Great and P… 0.348 3 1917 2019 8.2 1917 2019 Cats 2019 Cats 2019 0.296 4 2001: A Space Odyssey 1968 8.3 2001: A Space Odyssey 1968 2001: A Space Odyssey 1968 2001: A Space Odys… 0 5 3 Idiots 2009 8.3 3 Idiots 2009 The Incredibles 2004 The Incredibles 20… 0.286 6 A Beautiful Mind 2001 8.2 A Beautiful Mind 2001 Beautiful Noise 2014 Beautiful Noise 20… 0.312 7 A Clockwork Orange 1971 8.2 A Clockwork Orange 1971 A Clockwork Orange 1972 A Clockwork Orange… 0.0290 8 A Separation 2011 8.2 A Separation 2011 Separado! 2010 Separado! 2010 0.189 9 Aladdin 1992 8 Aladdin 1992 Aladdin 1992 Aladdin 1992 0 10 Alien 1979 8.4 Alien 1979 Alien 1979 Alien 1979 0 # 244 more rows# Use print(n = ...) to see more rows

``` We have several perfect matches in the first 10 rows. These have a distance of 0. There are some less-good-but-still-matches, such as A Clockwork Orange where the year differs between IMDB and Plex. Then there are a bunch of clear “not matched” movies, e.g. 12 Angry Men, 12 Years a Slave. We can see that a distance of 0.1 or more means the match is not true.

Note that it says there are 244 more rows and shows us 10 (a total of 254 when we should have only 250). The 4 extra matches are duplicates caused by a same-distance match to two different movies in the Plex library. Let’s get rid of them and then figure out our totals.

```

remove duplicate fuzzy match failsmatch <- match[!duplicated(match[ , "titleyear.x"]), ]# leaves 250 rows# matches with distance of 0.1 or more are not a match# the are the movies we want to look atmatch <- match %>% filter(dist >= 0.1)# leaves 190 rows

``` So I have 60 of the IMDB’s Top 250 Movies. This is not very high. In my defence, I am not a movie buff and my movie collection is not particularly huge.

So what are those movies that I am missing? Let’s sort them to be the highest rated and figure out what I should add with some urgency!

match <- match[order(-match$Rating),]# write filelapply(match$titleyear.x, write, "Output/Data/imdb.txt", append=TRUE)

| The Shawshank Redemption | 1994 | 9.2 | | 12 Angry Men | 1957 | 9 | | The Dark Knight | 2008 | 9 | | Schindler’s List | 1993 | 8.9 | | The Good, the Bad and the Ugly | 1966 | 8.8 | | Fight Club | 1999 | 8.7 | | Inception | 2010 | 8.7 | | Interstellar | 2014 | 8.6 | | It’s a Wonderful Life | 1946 | 8.6 | | Life Is Beautiful | 1997 | 8.6 |

The top 10 films I was missing…I have at least seen some of those films at some point in the past.

—

The post title comes from “Yet Another Movie” by Pink Floyd from “A Momentary Lapse of Reason”.

To leave a comment for the author, please follow the link and comment on their blog: Rstats – quantixed.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Yet Another Movie: IMDB Top 250 movies

View Details

[This article was first published on Stories Data Speak, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.IntroductionRecently, I stepped into the AWS ecosystem to learn and explore its capabilities. I’m documenting my experiences in these series of posts. Hopefully, they will serve as a reference point to me in future or for anyone else following this path. The objective of this post is, to understand how to create a data pipeline. Read on to see how I did it. Certainly, there can be much more efficient ways, and I hope to find them too. If you know such better method’s, please suggest them in the comments section.

How to upload external data in Amazon AWS S3Step 1: In the AWS S3 user management console, click on your bucket name.

Step 2: Use the upload tab to upload external data into your bucket.

Step 3: Once the data is uploaded, click on it. In the Overview tab, at the bottom of the page you’ll see, Object Url. Copy this url and paste it in notepad.

Step 4:

Now click on the Permissions tab.

Under the section, Public access, click on the radio button Everyone. It will open up a window.

Put a checkmark on Read object permissions in Access to this objects ACL. This will give access to reading the data from the given object url.

Note: Do not give write object permission access. Also, if read access is not given then the data cannot be read by Sagemaker

AWS Sagemaker for consuming S3 dataStep 5

Open AWS Sagemaker.
From the Sagemaker dashboard, click on the button create a notebook instance. I have already created one as shown below.
click on Open Jupyter tab

Step 6

In Sagemaker Jupyter notebook interface, click on the New tab (see screenshot) and choose the programming environment of your choice.

Step 7

Read the data in the programming environment. I have chosen R in step 6.

Accessing data in S3 bucket with pythonThere are two methods to access the data file;

The Client method
The Object URL method

See this IPython notebook for details.

AWS Data pipeline

To build an AWS Data pipeline, following steps need to be followed;

Ensure the user has the required IAM Roles. See this AWS documentation
To use AWS Data Pipeline, you create a pipeline definition that specifies the business logic for your data processing. A typical pipeline definition consists of activities that define the work to perform, data nodes that define the location and type of input and output data, and a schedule that determines when the activities are performed.

To leave a comment for the author, please follow the link and comment on their blog: Stories Data Speak.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Stories Data Speak 2023-04-29 05:33:17

View Details

[This article was first published on Saturn Elephant, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. The fisheye function below distorts a bitmap image with a fisheye effect or an anti-fisheye effect (rho > 0.5 or rho < 0.5).

fisheye\_xy <- function(x, y, rho, stick) { p <- c(x, y) if(rho == 0.5) { return(p) } m <- c(0.5, 0.5) d <- p - m r <- sqrt(c(crossprod(d))) dnormalized <- d / r mnorm <- sqrt(c(crossprod(m))) power <- pi / mnorm * (rho - 0.5) if(power > 0) { bind <- if(stick == "corners") mnorm else m[2L] uv <- m + dnormalized * tan(r*power) * bind / tan(bind*power) } else { bind <- m[2L] uv <- m + dnormalized * atan(-10*r*power) * bind / atan(-10*bind*power) } uv}library(imager)library(cooltools)#' @importFrom imager load.image add.color squeeze R G B#' @importFrom cooltools approxfun2#' @importFrom grDevices col2rgb rgb#' @param bitmapFile path to a bitmap file (jpg, png, ...)#' @param rho amount of effect; no effect if 0.5, fisheye if >0.5, #' antifisheye if <0.5#' @param stick where to stick the image when rho>0.5, to the #' corners or to the borders; if you stick to the corners, a #' part of the image is lost#' @param bkg background color; it appears only if rho>0.5 and #' stick="borders" fisheye <- function(bitmapFile, rho, stick = "corners", bkg = "black") { stopifnot(rho > 0, rho < 1) stick <- match.arg(stick, c("borders", "corners")) # load the image img <- load.image(bitmapFile) dims <- dim(img) nx <- dims[1L] ny <- dims[2L] nchannels <- dims[4L] # if the image is gray, add colors if(nchannels == 1L) { img <- add.color(img) } else if(nchannels != 3L) { stop("Cannot process this image.") } # fisheye matrix PSI <- matrix(NA\_complex\_, nrow = nx, ncol = ny) for(i in 1L:nx) { x <- (i-1L) / (nx-1L) for(j in 1L:ny) { y <- (j-1L) / (ny-1L) uv <- fisheye\_xy(x, y, rho, stick) PSI[i, j] <- complex(real = uv[1L], imaginary = uv[2L]) } } # take the r, g, b channels r <- squeeze(R(img)) g <- squeeze(G(img)) b <- squeeze(B(img)) # interpolation x\_ <- seq(0, 1, length.out = nx) y\_ <- seq(0, 1, length.out = ny) f\_r <- approxfun2(x\_, y\_, r) f\_g <- approxfun2(x\_, y\_, g) f\_b <- approxfun2(x\_, y\_, b) M\_r <- f\_r(Re(PSI), Im(PSI)) M\_g <- f\_g(Re(PSI), Im(PSI)) M\_b <- f\_b(Re(PSI), Im(PSI)) # set outside color RGB <- col2rgb(bkg)[, 1L] / 255 M\_r[is.na(M\_r)] <- RGB[1L] M\_g[is.na(M\_g)] <- RGB[2L] M\_b[is.na(M\_b)] <- RGB[3L] # convert to hex codes rstr <- rgb(M\_r, M\_g, M\_b) dim(rstr) <- c(nx, ny) # rotate t(rstr)} Let’s take for example this picture of Dilbert, named dilbert512x512.png:

It has a transparent background. We firstly transform this background it to a gray color (#aaaaaa) with the help of ImageMagick. The command to do that is:

convert in.png -background '#aaaaaa' -alpha remove -alpha off out.png (magick convert if you use Windows).

We can run this command from R:

dilbert\_transparent <- "dilbert512x512.png"dilbert\_gray <- "dilbert\_gray.png"gray\_color <- "#aaaaaa"cmd <- sprintf( "convert %s -background '%s' -alpha remove -alpha off %s", dilbert\_transparent, gray\_color, dilbert\_gray)system(cmd) Here is dilbert_gray.png:

Now let’s perform a fisheye distortion of this image with rho=0.95:

img <- fisheye(dilbert\_gray, rho = 0.95, stick = "borders", bkg = gray\_color)# plotopar <- par(mar = c(0, 0, 0, 0))plot(c(-100, 100), c(-100, 100), type = "n", asp = 1, xlab = NA, ylab = NA, axes = FALSE, xaxs = "i", yaxs = "i")rasterImage(img, -100, -100, 100, 100)par(opar)

The anti-fisheye effect is obtained by setting rho<0.5. We will do it, with something more: we will get a transparent background at the end.

To do so, first transform the transparent background to a color, for example green (#00ff00):

dilbert\_transparent <- "dilbert512x512.png"dilbert\_green <- "dilbert\_green.png"green\_color <- "#00ff00"cmd <- sprintf( "convert %s -background '%s' -alpha remove -alpha off %s", dilbert\_transparent, green\_color, dilbert\_green)system(cmd) Here is dilbert_green.png:

Now perform the anti-fisheye effect, and use the same green color as background:

img <- fisheye(dilbert\_green, rho = 0.45, bkg = green\_color)# save imagepng("dilbert\_antifisheye\_green.png", width = 512, height = 512)opar <- par(mar = c(0, 0, 0, 0))plot(c(-100, 100), c(-100, 100), type = "n", asp = 1, xlab = NA, ylab = NA, axes = FALSE, xaxs = "i", yaxs = "i")rasterImage(img, -100, -100, 100, 100)par(opar)dev.off() Here is dilbert_antifisheye_green.png:

Finally, using ImageMagick, transform the green color to transparent:

cmd <- sprintf( "convert -fuzz 30% -transparent '%s' %s %s", green\_color, "dilbert\_antifisheye\_green.png", "dilbert\_antifisheye.png")system(cmd)

To leave a comment for the author, please follow the link and comment on their blog: Saturn Elephant.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Fisheye effect with R

View Details

[This article was first published on R | R-statistics blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.TL;DRI’m seeking someone to take over maintenance of the the popular R package installr (github), due to a shift away from Windows OS. The package has been downloaded over 3.3 million times and has a current download rate of around 61k times a month. The ideal candidate should have experience with Windows OS, be an experienced R developer, and be passionate about helping fellow R users.

Interested? please leave a comment on the github issue here.

DetailsBackgroundDear R Community,

As many of you may know, the installr package has been providing valuable functions for installing and updating software on Windows OS, with a particular focus on allowing R users to update R itself from the terminal (using the updateR() function). Over the years, this package has gained significant traction, and I’m grateful for the support and appreciation the community has shown.

To give you some insight into its popularity, installr has been downloaded over 3.3 million times, with a current download rate of around 61k times a month. It’s wonderful to see the positive impact this package has had on the R community, making it easier to keep R updated and install essential software for development and reproducible research.

You can find the GitHub repository for the installr package here: https://github.com/talgalili/installr And the CRAN page for the package here: https://cran.r-project.org/web/packages/installr/index.html The current version of the package is 0.23.4.

Looking for a maintainerHowever, as the current maintainer of the installr package, I have a personal update to share. About five years ago, I made the switch from Windows to Linux Mint, and I no longer see myself returning to Windows. Consequently, my ability to effectively maintain the installr package has become increasingly limited.

This is why I am reaching out to the community with a request. I am searching for someone enthusiastic and dedicated to take over the maintenance of the installr package (or at least co-maintain, I can still deal with uploading to CRAN). If you are an experienced R developer with a strong background in Windows OS, and you are passionate about helping fellow R users, this might be a great opportunity for you to make a significant contribution to the community.

Responsibilities of the maintainer would include:

Addressing user-reported issues (see list here)
Updating and improving the package to keep up with the evolving R ecosystem
Ensuring compatibility with new software versions and Windows updates
Expanding the installr package’s functionality, if desired

To express your interest or to learn more about the role, please get in touch with me by leaving a comment on the github issue here. I would be more than happy to discuss the role further and answer any questions you might have.

Thank you for your continued support, and I look forward to finding a new maintainer to carry the torch and ensure the installr package continues to be a valuable resource for the R community.

Best regards,

Tal Galili

The post Seeking a New Maintainer for the Popular R Package installr first appeared on R-statistics blog.To leave a comment for the author, please follow the link and comment on their blog: R | R-statistics blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Seeking a New Maintainer for the Popular R Package installr

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Word clouds are a visual representation of text data where words arearranged in a cluster, with the size of each word reflecting itsfrequency or importance in the data set. Word clouds are a great way ofdisplaying the most prominent topics or keywords in free text dataobtained from websites, social media feeds, reviews, articles and more.If you want to learn more about working with unstructured text data, werecommend attending our Text Mining inRcourse

Usually, a word cloud will be used solely as an output. But what if youwanted to use a word cloud as an input? For example, let’s say wevisualised the most common words in reviews for a hotel. Imagine wecould then click on a specific word in the word cloud, and it would thenshow us only the reviews which mention that specific word. Useful,right?

This blog will take you through creating a clickable word cloud in aShiny app, where the user can click any word in the word cloud to filteran output table. We will be using the 2021 TidyTuesday Netflixtitlesdata set and the{wordcloud2}package to create our word cloud. We will then integrate it in a Shinyapp with a reactively filtered {DT}table output.

Do you require help building a Shiny app? Would you like someone to take over the maintenance burden?If so, check outour Shiny and Dash services.

Creating a word cloud with {wordcloud2}{wordcloud2} is an R package which creates HTML-based word clouds, basedon wordcloud2.js. The mainfunction is simply called wordcloud2() and takes a word count dataframe as an input i.e. one column containing the words, one columncontaining the frequencies of those words.

Before creating the word cloud, we need to read in our data using the{tidytuesdayR}package. If you want to see the full source code for the final Shinyapp, check out our GitHub.

tuesdata = tidytuesdayR::tt\_load("2021-04-20")netflix\_titles = tuesdata$netflix\_titles To create our word count data frame, we will use a combination of{dplyr} and{tidytext} functions. Wefilter out words that are used in 10 titles or less to prevent our wordcloud from being too crowded.

library("dplyr")library("tidytext")word\_counts = netflix\_titles %>% unnest\_tokens("word", title) %>% anti\_join(stop\_words, by = "word") %>% count(word) %>% filter(n > 10)## # A tibble: 157 × 2## word n## <chr> <int>## 1 love 151## 2 2 115## 3 christmas 78## 4 story 67## 5 life 65## 6 world 63## 7 movie 60## 8 time 54## 9 de 46## 10 american 45## # ℹ 147 more rows Then we just need to pass this word count data frame into thewordcloud2() function. We’re using a custom colour palette instead ofthe default one. wordcloud2() requires a colour palette vector of thesame length as the data set, so you can use the rep_len() function toachieve this.

library("wordcloud2")my\_palette = c("#355070", "#6d597a", "#b56576", "#e56b6f", "#eaac8b")my\_wordcloud = wordcloud2( word\_counts, color = rep\_len(my\_palette, nrow(word\_counts))) The wordcloud2 package contains two functions for incorporating wordclouds in a Shiny app: wordcloud2Output() and renderWordcloud2().These work in the same way as most *Output() and render*()functions.

library("shiny")ui = fluidPage( wordcloud2Output("wordcloud"))server = function(input, output) { output$wordcloud = renderWordcloud2(my\_wordcloud)}shinyApp(ui, server) Binding a JavaScript click event to a Shiny inputNow to the key part of this blog post. We want to be able to click on aword in the word cloud, and use the clicked word as an input in Shiny.We need to write some JavaScript for this, which will be wrapped in theHTML() function within a script tag (tags$script()). We arewriting an anonymous function, i.e. an unnamed function, which will berun whenever we click on a word in the word cloud. The function willextract the text content of the label produced when we hover over aword, and then cast this to a Shiny input called clicked_word.

ui = fluidPage( tags$script(HTML( "$(document).on('click', '#canvas', function() { word = $('#wcLabel').text(); Shiny.onInputChange('clicked\_word', word); });")), wordcloud2Output("wordcloud")) Now, we can use input$clicked_word in our Shiny server to filter theNetflix titles to retain only the titles which contain that specificword. We use a combination of {dplyr} and{stringr} to do this. The input alsocontains the count, e.g. “love: 151”, so we need to first use a regularexpression remove the colon and any numbers after it.

server = function(input, output) { output$wordcloud = renderWordcloud2(my\_wordcloud) filtered\_netflix = reactive({ clicked\_word = str\_remove(input$clicked\_word, ":[0-9]+$") netflix\_titles %>% filter(str\_detect(tolower(title), clicked\_word)) %>% select(title, everything(), -show\_id) })} The final step is to create an output table of the filtered data. We usethe renderDT() and DTOutput() functions from {DT} to do this, butyou can use any package for creating tables.

library("DT")ui = fluidPage( <...>, DTOutput("filtered\_tbl"))server = function(input, output) { <...>, output$filtered\_tbl = renderDT(filtered\_netflix())} Now, you should have an interactive word cloud input which allows you tofilter a table based on whichever word you click! You can of course usethe word input for something else, for example, you could re-render theword cloud every time you click a word to show you the words which aremost often used together with your clicked word, or you could use theinput to create some further visualisations.

If you’re interested in learning more about Shiny, check out our Shinyin Productionconference, taking place October 12th-13th in Newcastle upon Tyne. We’llbe focussing on all things shiny as well as other web-based R packages,with an afternoon of workshops run by our JR trainers, followed by a dayof talks from R experts!

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to create a clickable world cloud with wordcloud2 and Shiny

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionAs data science continues to be a sought-after field, creating a reliable and accurate model is essential. While there are various machine learning algorithms available, the process of selecting the correct algorithm can be complex. The {tidyAML} package, part of the tidymodels suite, offers an easy-to-use, consistent interface for building machine learning models. In this post, we will explore a Shiny application that utilizes tidyAML to build a machine learning model.

Today I have updated the tidyAML shiny app to include the ability to set the parameter of the fast_regression() function .parsnip_fns and this is things like linear_reg.

Here is a full list of what is available:

library(tidyAML)library(dplyr)c("all", make\_regression\_base\_tbl() |> pull(.parsnip\_fns) |> unique() )

[1] "all" "linear\_reg" "cubist\_rules" "poisson\_reg" [5] "bag\_mars" "bag\_tree" "bart" "boost\_tree" [9] "decision\_tree" "gen\_additive\_mod" "mars" "mlp" [13] "nearest\_neighbor" "rand\_forest" "rule\_fit" "svm\_linear" [17] "svm\_poly" "svm\_rbf" I have updated the UI to reflect using that method as well. Here is the UI changes:

selectInput("model\_engine", "Select a model engine:", choices = c("all", make\_regression\_base\_tbl() |> pull(.parsnip\_engine) |> unique() ) ), selectInput("model\_fns", "Select a model function:", choices = c("all", make\_regression\_base\_tbl() |> pull(.parsnip\_fns) |> unique() ) Here are some pictures showing the changes:

UI ChangeUI Change 2OutputSo what this means is that we can just pick a function like parsnip::linear_reg() and leave the engine set to "all" and it will build models for all engines supported that work with linear_reg().

The Shiny ApplicationThe Shiny application is a graphical user interface (GUI) that allows users to select a dataset, predictor column, model type, and engine, and then build a machine learning model. The user can upload a CSV or TXT file or choose one of two built-in datasets: “mtcars” or “iris”. The user can select the predictor column, which is the variable used to predict the outcome, and then choose the model type, either “regression” or “classification”. Next, the user can select a model engine and a model function to use in building the model. Once the user has made all the selections, they can click the “Build Model” button to create the model.

The code for the Shiny application can be broken down into two parts, the User Interface (UI) and the Server. Let’s take a closer look at each of these parts.

The UIThe UI is created using the fluidPage() function from the shiny package. The titlePanel() function creates the title of the application. The sidebarLayout() function creates the sidebar and main panel. The sidebar contains input controls such as file input, select input, and an action button. The main panel displays the outputs generated by the model.

The fileInput() function creates a widget that allows the user to upload a data file. The selectInput() function creates dropdown menus for the user to select the dataset, predictor column, model type, model engine, and model function. The actionButton() function creates a button that the user clicks to build the model. The verbatimTextOutput() function and reactableOutput() function display the output generated by the model.

The ServerThe Server is where the input data is processed, the model is built, and the output is generated. The Server is created using the server() function from the shiny package.

The reactive() function is used to create a reactive object called data that reads in the data file or built-in dataset selected by the user. The eventReactive() function is used to create a reactive object called recipe_obj that creates a recipe for preprocessing the data. The recipe includes steps to normalize the numeric variables and remove the outcome variable from the recipe.

Two other reactive objects, model_engine and model_fns, are created using the switch() function. These objects contain a list of available engines and model functions for the user to choose from.

Finally, the eventReactive() function is used to create a reactive object called model that builds the machine learning model. The fast_regression() and fast_classification() functions from the tidyAML package are used to build the regression and classification models, respectively.

ConclusionIn this post, we explored a Shiny application that uses tidyAML to build a machine learning model. The application allows users to select a dataset, predictor column, model type, engine, and function to build a machine learning model. The Shiny application is an excellent tool for those who are new to machine learning or those who want to streamline the rapid prototyping process.

Full ApplicationThis is a work in progress, and I want you to steal this code and see what you can come up with!

library(shiny)library(tidyAML)library(recipes)library(DT)library(glmnet)library(rules)library(tidymodels)library(reactable)tidymodels\_prefer()ui <- fluidPage( titlePanel("tidyAML Model Builder"), sidebarLayout( sidebarPanel( fileInput("file", "Upload your data file (csv or txt):"), selectInput("dataset", "Choose a built-in dataset:", choices = c("mtcars", "iris") ), selectInput("predictor\_col", "Select the predictor column:", choices = NULL ), selectInput("model\_type", "Select a model type:", choices = c("regression", "classification")), selectInput("model\_engine", "Select a model engine:", choices = c("all", make\_regression\_base\_tbl() |> pull(.parsnip\_engine) |> unique() ) ), selectInput("model\_fns", "Select a model function:", choices = c("all", make\_regression\_base\_tbl() |> pull(.parsnip\_fns) |> unique() ) ), actionButton("build\_model", "Build Model"), verbatimTextOutput("recipe\_output") ), mainPanel( verbatimTextOutput("model\_table"), reactableOutput("model\_reactable") ) ))server <- function(input, output, session) { data <- reactive({ if (!is.null(input$file)) { df <- read.csv( input$file$datapath, header = TRUE, stringsAsFactors = FALSE ) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } else if (!is.null(input$dataset)) { df <- get(input$dataset) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } }) recipe\_obj <- eventReactive(input$predictor\_col, { rec <- recipe(as.formula(paste(input$predictor\_col, "~ .")), data = data() ) |> step\_normalize(all\_numeric(), -all\_outcomes()) return(rec) }) model\_engine <- reactive({ switch(input$model\_engine, "all" = "all", "lm" = "lm", "brulee" = "brulee", "gee" = "gee", "glm" = "glm", "glmer" = "glmer", "glmnet" = "glmnet", "gls" = "gls", "lme" = "lme", "lmer" = "lmer", "stan" = "stan", "stan\_glmer" = "stan\_glmer", "Cubist" = "Cubist", "hurdle" = "hurdle", "zeroinfl" = "zeroinfl", "earth" = "earth", "rpart" = "rpart", "dbarts" = "dbarts", "xgboost" = "xgboost" , "lightgbm" = "lightgbm", "partykit" = "partykit", "mgcv" = "mgcv", "nnet" = "nnet", "kknn" = "kknn", "ranger" = "ranger", "randomForest" = "randomForest", "xrf" = "xrf", "LiblineaR" = "LiblineaR", "kernlab = kernlab") }) model\_fns <- reactive({ switch(input$model\_fns, "all" = "all", "linear\_reg" = "linear\_reg", "cubist\_rules" = "cubist\_rules", "poisson\_reg" = "poisson\_reg", "bag\_mars" = "bag\_mars", "bag\_tree" = "bag\_tree", "bart" = "bart", "boost\_tree" = "boost\_tree", "decision\_tree" = "decision\_tree", "gen\_additive\_mod" = "gen\_additive\_mod", "mars" = "mars", "mlp" = "mlp", "nearest\_neighbor" = "nearest\_neighbor", "rand\_forest" = "rand\_forest", "rule\_fit" = "rule\_fit", "svm\_linear" = "svm\_linear", "svm\_poly" = "svm\_poly", "svm\_rbf" = "svm\_rbf" ) }) model <- eventReactive(input$build\_model, { if (input$model\_type == "regression") { mod <- fast\_regression(.data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_engine(), .parsnip\_fns = model\_fns()) } else if (input$model\_type == "classification") { mod <- fast\_classification(.data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_engine(), .parsnip\_fns = model\_fns()) } return(mod) }) output$recipe\_output <- renderPrint({ if (!is.null(input$predictor\_col)) { summary(recipe\_obj()) } }) output$model\_table <- renderPrint({ if (input$build\_model > 0) { print(model()) } }) output$model\_reactable <- renderReactable({ if (input$build\_model > 0) { reactable(model()) } }) }shinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building models with {shiny} and {tidyAML} Part 3

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Jackson Kwok, Infrastructure Steering Committee (ISC) recipient for the 2020 cycle, discusses the project Sketch and its implementation in 2021 as an R Package in the CRAN environment. The Sketch package was developed with the goal of having a package that translates R to Javascript. With this experience, Jackson then later developed the package, Animate, which requires no prior knowledge of JavaScript. Jackson discusses Sketch’s origins in JavaScript (JS) and data visualization, its potential applications, and the level of expertise required to utilize the package effectively.

RC: Where did you come up with the idea of a program that translates R into JS?

JK: It started with a GitHub issue by Jonathon Carroll at the rOpenSci R OzUnconference 2017. The idea was to use the JS library P5 for visualization in R, and a few of us came together to work on a prototype package called “realtime”. After the hackathon, I continued to pursue the idea. As I was reading the book The Nature of Code, I realized it took only a few rewriting rules to translate JS to syntactically correct R code. After some manual experiments, I wrote the package to test the idea (in the reverse direction transpiling R to JS) and it worked! The package got its name because P5 refers to the digital drawings as Sketches; I named the folder ‘Sketch’ when I studied P5 and later used the same folder for the package. There is nothing specific to P5 that makes the conversion work. I tried other JS libraries and it also worked very well, so I refactored the package into a general purpose R-to-JS transpiler.

Examples are of Physics engine and 3D models on Sketchmatter.jsThree.jsRC: How can I use Sketch in a shiny app?

JK: On the documentation website, there is a page on how to use Sketch on a Shiny app. Once you develop your R script and transpile it into JS, you can include that in the Shiny app as usual.

RC: How much JS knowledge do you need to use this package?

JK: A little bit to get started. You don’t need to know the syntax, but you need to know how JS works because it works differently from R. Vector and List in R correspond roughly to Array and Object in JS. Both of them are passed by reference rather than passed by value, and Array uses 0-based indexing. That is what is needed to get started. The Pitfalls section of the documentation has the complete list.

There are some tutorials in the Tutorial section that users can follow with no knowledge of JS. They will be able to pick up some JS along the way. What Sketch does is to let users call JS libraries like R packages. You still need to learn the commands in the package to use it, just like any new R package.

RC: Can I use different R Packages with Sketch? (aka, could I use rQTL to analyze and visualize the data?)

JK: Yes. That was one of the key milestones in the 2nd proposal of this project. Sketch uses WebSocket to establish a connection between R and the browser so that you can use any R package to perform some calculations and then pass the results to the browser. The connection is live, so you can also perform operations in the browser and get the corresponding data back to R. It is very handy if you need some interaction that is very hard to express in code. For instance, if you need to select something using a lasso tool (irregular shape) it will be much easier. You just draw it rather than have to figure out the coordinates.

If you choose to use an R package with Sketch, then the application will no longer be stand-alone. A targeted use case is to let users add a customizable domain-specific interface to an existing analysis performed using many R packages. If you don’t use any R package at all, then the Sketch application can be deployed as a stand-alone website.

RC: What is the progress on Sketch?

JK: The second proposal is now complete with many new features added. Among other things, there is the support of R6-style of OOP which helps you structure larger programs, the WebSocket which lets you do bidirectional updates, and a knitr engine for RMD and publish support. On the package side, I do not see substantial changes from here on; the later updates will mostly be fixes and patches.

One interesting direction that I have been looking into is to transpile R to AssemblyScript, which in turn can be compiled into WebAssembly. I have done some preliminary studies, and it seems this path is viable. AssmeblyScript is still maturing, so this will not be worked into Sketch for now, but I will keep an eye on it.

The next thing Sketch looks to expand on is use cases. Lately, I found out it is not difficult to use Sketch to create a web-based graphics device for animated plots in R, so I have been working on it, and the results are encouraging. Another thing I discovered is that it is quite easy to go from ggplot2 to rayshader to VR, and it works well with 3d histograms on maps. Also, another application that came by surprise is that with Sketch, you can control Excel from R. Excel has a JavaScript API and supports WebSocket protocol. As Sketch transpiles R to JS and speaks WebSocket, it turns out controlling Excel from R just works out fine!

I discover new possibilities with Sketch every now and then, and that makes me realize this work is really a good step in strengthening the R-Web integration and expanding the R application landscape.

The ISC project was delivered in early 2021. Since its release date, we now have a package called Animate, on CRAN. It would have been difficult to build without the Sketch package since it uses a heavy amount of JavaScript. Animate provides animated diagrams beyond XY plots. Unlike Sketch which needs the user to have some background knowledge of integrating JavaScript, with Animate, users can manipulate graphical elements on the screen using native R commands without knowing the animations are powered by JavaScript.

Lorenz system [code] on Animate Maze generation [code] [tutorial] on AnimateRC: Has Sketch expanded the scope of R visualizations for the R community?

JK: It has definitely expanded the scope for JavaScript and R. The Sketch website features a showcase page showing some of the new possibilities, including more advanced 3D model animations and agent-based visualizations. In general, Sketch is well-suited for cases in which you need to use a JavaScript library beyond direct API calls or where the API has an imperative style.

Looking back after two years of completing the package, I think Sketch succeeded in producing executable transpiled JavaScript and exploring how far one can control JavaScript with R, but it fell short in abstracting away the JavaScript side of things. These shortcomings are addressed by Animate.

Sketch example of Game of life – p5.jsAnimate exampleRC: How did you get involved?

JK: I first got into interactive visualizations back in 2016 after seeing a few great talks and demos online, e.g. the Invention on Principle, Stop Drawing Dead Fish by Bret Victor, the Parable of the polygons by Vi Hart and Nicky Case and the ConvNetJS by Andrej Karpathy. They got me started learning JS, but I am an R user at heart. I wanted to make interactive visualization in R, then the R OzUnconference came, and you know the rest. Looking back, it has been quite a journey picking up the skills needed to deliver this project.

I found out about the R consortium ISC program when the research fund that I was under ran out early. I reckoned it was a great opportunity to contribute to the R community and get some financial support, so I put in an application with Kate Saunders, a good friend who loves data visualization and R programming with expertise in spatial statistics – one of the key areas I want Sketch to develop into.

RC: What was your experience working with the R Consortium? Would you recommend applying for a grant to others?

JK: I had a great experience and highly recommend it to others looking to develop packages that can help the R community. I personally learned a lot about the grant application process, like writing a proposal, arguing for the benefits of the project, and doing deeper research on what you want to solve and how your solution can be successful.

The process was also great for picking up medium- to long-term software planning, like structuring the development with milestones, which I previously had no experience with. Overall it has been a great learning and rewarding experience! What helped me plan the proposal were the guidelines provided by the R Consortium. I was able to take bigger infrastructure problems and group them into small solvable groups of tasks.

RC: What do you do for your day job?

JK: I am finishing up my postdoc at the St. Vincent’s Institute of Medical Research. My research group is working on a translational project called BRAIx. It’s about transforming breast cancer screening in Australia using AI. In the project, I have been using Sketch to create customized data visualization tools for data and model diagnostics.

About ISC Funded Projects

Learn moreThe post Sketch Package looks to add JavaScript to R packages appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Sketch Package looks to add JavaScript to R packages

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.What happens when there’s no data to show to the user? This can occur in Shiny apps, where users must upload their own data. Well, one solution is to show an empty screen (an empty state). But this negatively impacts the user experience. A better approach would be to have a dedicated empty state screen, and today you’ll implement a Shiny empty state screen with Appsilon’s latest open-source package.

Do you love Shiny, but your apps are slow, dull, or lacking functionality? Explore the open-source Rhinoverse and take your apps to the next level.

If we’re talking about dashboards, there are numerous places where empty state screens come in handy. For example, you could use them to instruct the user to upload a dataset before proceeding with calculations. Or when there’s no data to display at all, potentially due to too many data filters being applied.

So without much ado, let’s dive straight in and leave your users in a better state!

Want to measure the performance of your new empty state changes? Try Appsilon’s shiny.benchmark package.

Table of contents:

Building a Dashboard Logic for Shiny Empty State
Empty State in Shiny – Dashboard in Action
Summing up Shiny Empty State

Building a Dashboard Logic for Shiny Empty StateBefore actually writing the R Shiny app, we must first discuss the logic it will implement. The app will have two screens:

Empty state screen – Instructs the user to upload the dataset, and provides an action button to do so.
Dashboard screen – Renders a chart (combination of a scatter and line chart) that shows the uploaded datasets and results of a fitted machine learning model.

Sounds easy enough, and in this section, we’ll deal with the dataset logic and the ways to fit a machine learning model.

To start, let’s create a dataset. It will have two columns – x and y, where x is a list of numbers between 1 and 100, and y is identical to x, but with added variation of a number between -3 and +3:

library(comprehenr)library(ggplot2)x <- seq(1, 100)y <- to\_vec(for (num in x) num + runif(n = 1, min = -3, max = 3))df <- data.frame( x = x, y = y)head(df) Image 1 – First 6 rows of the random dataset

That’s the dataset we’ll upload later to the dashboard, so it’s a good idea to save it locally. The following code snippet dumps it into a CSV file:

write.csv(df, "data.csv", row.names = FALSE) Awesome! Now, let’s fit a simple linear regression model. The variable y is dependent, and the variable x is independent. R’s lm() function allows us to easily model that relationship:

model <- lm(df$y ~ df$x)summary(model) Here’s the summary of our model:

Image 2 – Summary of a linear regression model

We’ll later extract the model coefficients, but first, let’s calculate the predictions and assign them to new dataframe columns. Put simply, we’re leveraging the patterns the model learned to make predictions on the input variable.

Wait, what is Linear Regression? Here’s our complete guide to simple and multiple linear regression in R.

In the case of linear regression, this just means solving a line equation with the coefficients from the previous image:

predictions <- predict(model, data.frame(df$x))df$y\_hat <- predictionshead(df) Here’s what the dataset looks like now:

Image 3 – Random dataset with added predictions

Predictions of a simple linear regression model will always be a straight line. To demonstrate, we’ll plot the original data points and predictions.

The get_model_subtitle() function is responsible for extracting model coefficients from the summary and formatting them as a formula string.

Below, there’s a call to ggplot() in which original data is rendered as a scatter plot (blue), and predictions are rendered as a black line:

get\_model\_subtitle <- function(model) { return(paste0(round(summary(model)$coefficients[1], 3), " + ", round(summary(model)$coefficients[2], 3), "x"))}ggplot(df, aes(x = x, y = y)) + geom\_point(color = "#0099F9", size = 5, alpha = 0.75) + geom\_line(aes(x = x, y = y\_hat), linewidth = 2) + labs( title = "X and Y Relationship", subtitle = paste("Formula:", get\_model\_subtitle(model)) ) + theme\_minimal() + theme(plot.title = element\_text(size = 20, face = "bold")) Image 4 – Chart showcasing actual data (blue) and predictions (black)

That’s the chart we want to display in R Shiny, but only after the user uploads the dataset. In other words, only after the Shiny empty state is passed.

Empty State in Shiny – Dashboard in ActionWe now have everything we need to write a dashboard in R Shiny. Well, everything except the package itself. You can install it with the following command:

remotes::install\_github("Appsilon/shiny.emptystate") The code snippet you’ll see below is simple to follow and understand, but let’s dissect it chunk by chunk:

get_model_subtitle() – The function you saw in the previous section, it’s responsible for extracting coefficients from a linear regression model and formatting them as a string equation.
empty_state_content – HTML content that’ll be rendered while the state is empty. In our case, it includes a heading and an action button responsible for uploading a CSV file.
ui – The user interface of our app once we get passed the empty Shiny state screen. If you’re using shiny.emptystate package, you must include a call to use_empty_state().
server() – Shiny function responsible for handling application logic. It must include an instance of EmptyStateManager class in which you define by the id element which element of your dashboard should be covered with the empty state content.
- The server() function here also contains a call to show() and hide() methods of the empty state manager to, well, show or hide the empty state screen depending on a condition.
- Finally, the server() function is responsible for applying a machine learning model, calculating predictions, and displaying the chart. You might want to extract this logic when building more complex dashboards.

If you prefer code over words, here’s everything you need to start using Appsilon’s shiny.emptystate package:

library(shiny)library(shiny.emptystate)library(shinyjs)library(ggplot2)# Helper function for formatting the subtitle - model formulaget\_model\_subtitle <- function(model) { return(paste0(round(summary(model)$coefficients[1], 3), " + ", round(summary(model)$coefficients[2], 3), "x"))}# Contents of the empty state - Just a heading and an upload buttonempty\_state\_content <- div( h3("Please upload a CSV file with columns \"x\" and \"y\""), actionButton( inputId = "upload\_btn", label = "Choose CSV File",#We don't recommend this method, but for this tutorial the simple button below looks better than the default fileInput onclick = "document.querySelector('#upload').click();" ))ui <- fluidPage( use\_empty\_state(), useShinyjs(), div( id = "chart\_container", plotOutput(outputId = "chart") ), shinyjs::hidden(fileInput(inputId = "upload", label = "upload")))server <- function(input, output) { # Initialize and show empty state empty\_state\_manager <- EmptyStateManager$new( id = "chart\_container", html\_content = empty\_state\_content ) empty\_state\_manager$show() # Handle dataset upload dataset <- reactiveVal() uploaded\_dataset <- reactive({ shiny::req(input$upload) read.csv(input$upload$datapath) }) observeEvent(uploaded\_dataset(), { if (nrow(uploaded\_dataset()) > 0) { dataset(uploaded\_dataset()) empty\_state\_manager$hide() } else { empty\_state\_manager$show() } }) # Handle chart output output$chart <- renderPlot({ shiny::req(dataset()) # ML model model <- lm(dataset()$y ~ dataset()$x) predictions <- predict(model, data.frame(dataset()$x)) ggplot(dataset(), aes(x = x, y = y)) + geom\_point(color = "#0099F9", size = 5, alpha = 0.75) + geom\_line(aes(x = x, y = predictions), linewidth = 2) + labs( title = "X and Y Relationship", subtitle = paste("Formula:", get\_model\_subtitle(model)) ) + theme\_minimal() + theme(plot.title = element\_text(size = 20, face = "bold")) })}# Connect all to a Shiny appshinyApp(ui = ui, server = server) Let’s run the app to check if everything works:

Image 5 – Shiny app demonstrating shiny.emptystate package

We’ve saved the dataset to a CSV file in the previous section, and used it here. As you can see, the app successfully goes from a Shiny empty state to a dashboard screen, which is just what we’ve wanted.

Let’s make a brief recap next.

SummaryR Shiny was designed to be easy for developers, but that doesn’t mean it lacks advanced functionality. This article is a perfect example of a logic you don’t want (nor need) to implement from scratch. It can be included in any app to improve the user experience and instruct the user on what to do.

Long story short, whenever you need to implement an empty state screen in your Shiny apps, look no further than shiny.emptystate. The link contains more examples that might be easier to grasp for newcomers to Shiny. If you get stuck, don’t hesitate to leave a comment below this article, or ping us via Twitter – @appsilon.

Does R Shiny sound like a promising career? Here’s how to get started.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: shiny.emptystate – How To Add an Empty State Component to R Shiny

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionYesterday I spoke about building tidymodels models using my package {tidyAML} and {shiny}. I have made an update to it, and will continue to make updates to it this week.

I have added all of the supported engines for regression problems only, NOT classification yet, that will be tomorrow’s work. I will then add a drop down for users to pick which backend function they want to use from {parsnp} like linear_reg().

Here are some pictures of the udpates.

New Drop Down Additionsreactable Error, not sure on how to fix yetreactable outputHere is the full application, please steal this code and modify for yourself, you never know what you might come up with!

library(shiny)library(tidyAML)library(recipes)library(DT)library(glmnet)library(tidymodels)library(reactable)tidymodels\_prefer()ui <- fluidPage( titlePanel("TidyAML Model Builder"), sidebarLayout( sidebarPanel( fileInput("file", "Upload your data file (csv or txt):"), selectInput("dataset", "Choose a built-in dataset:", choices = c("mtcars", "iris") ), selectInput("predictor\_col", "Select the predictor column:", choices = NULL ), selectInput("model\_type", "Select a model type:", choices = c("regression", "classification")), selectInput("model\_fn", "Select a model function:", choices = c("all","lm","brulee","gee","glm", "glmer","glmnet","gls","lme", "lmer","stan","stan\_glmer", "Cubist","hurdle","zeroinfl","earth", "rpart","dbarts","xgboost","lightgbm", "partykit","mgcv","nnet","kknn","ranger", "randomForest","xrf","LiblineaR","kernlab" ) ), actionButton("build\_model", "Build Model"), verbatimTextOutput("recipe\_output") ), mainPanel( verbatimTextOutput("model\_table"), reactableOutput("model\_reactable") ) ))server <- function(input, output, session) { data <- reactive({ if (!is.null(input$file)) { df <- read.csv( input$file$datapath, header = TRUE, stringsAsFactors = FALSE ) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } else if (!is.null(input$dataset)) { df <- get(input$dataset) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } }) recipe\_obj <- eventReactive(input$predictor\_col, { rec <- recipe(as.formula(paste(input$predictor\_col, "~ .")), data = data() ) |> step\_normalize(all\_numeric(), -all\_outcomes()) return(rec) }) model\_fn <- reactive({ switch(input$model\_fn, "all" = "all", "lm" = "lm", "brulee" = "brulee", "gee" = "gee", "glm" = "glm", "glmer" = "glmer", "glmnet" = "glmnet", "gls" = "gls", "lme" = "lme", "lmer" = "lmer", "stan" = "stan", "stan\_glmer" = "stan\_glmer", "Cubist" = "Cubist", "hurdle" = "hurdle", "zeroinfl" = "zeroinfl", "earth" = "earth", "rpart" = "rpart", "dbarts" = "dbarts", "xgboost" = "xgboost" , "lightgbm" = "lightgbm", "partykit" = "partykit", "mgcv" = "mgcv", "nnet" = "nnet", "kknn" = "kknn", "ranger" = "ranger", "randomForest" = "randomForest", "xrf" = "xrf", "LiblineaR" = "LiblineaR", "kernlab = kernlab") }) model <- eventReactive(input$build\_model, { if (input$model\_type == "regression") { mod <- fast\_regression(.data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_fn()) } else if (input$model\_type == "classification") { mod <- fast\_classification(.data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_fn()) } return(mod) }) output$recipe\_output <- renderPrint({ if (!is.null(input$predictor\_col)) { summary(recipe\_obj()) } }) output$model\_table <- renderPrint({ if (input$build\_model > 0) { print(model()) } }) output$model\_reactable <- renderReactable({ if (input$build\_model > 0) { reactable(model()) } }) }shinyApp(ui = ui, server = server) To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building models with {shiny} and {tidyAML} Part 2

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A Riddler puzzle on the three binary and sequential questions one should ask three players hiding their respective U(0,1) realisation, U, V, and W, to best guess which player holds the largest number, max{U,V,W}. Assuming questions of the type Is U, &tc., the challenge boils down to selecting seven bounds (one for U, two for V, and four for W) in order to optimise the probability of picking the right player. Which means for a given collection of such bounds to learn this probability from the three binary answers. These can be turned into eight (2³) binary variables and I used them as entries in a logistic regression to predict that W was larger than max(U,V), itself predicted by the first two answers. The optimisation of the bounds can then be achieved by simulated annealing (or otherwise) and the approach returns (random) outputs like the following bounds (one on U, two on V, and four on W)

*0.616 0.434 0.830 0.350 0.736 0.913 0.796 0.827*

for an estimated probability of 0.827. This is a somewhat coherent sequence of bounds when considering the simpler case of two players. Indeed, with three bounds, the probability of winning can be readily derived as

logically optimised by (b¹,b²,b³)=(1/2,1/4,3/4) for a success probability of 0.875. And it coïncides with the solution posted by The Riddler, although there is no intuition behind the figures, contrary to the two player situation. In fact, I am surprised that the bound on W does not equate the expectation of max{U,V} under the current conditions:

*> x=runif(1e6,0,.616);y=runif(1e6,0,.434)> mean(y+(x-y>0)*(x-y))[1] 0.3591648* To leave a comment for the author, please follow the link and comment on their blog: R – Xi'an's Og.

R-bloggers.com offers daily e-mail updates* about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.*Continue reading: simulated annealing and logistic regression to the max

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Antonio Hegar, organizer of the R Glasgow user group (also on Twitter), shared with the R Consortium his efforts to build an R community in Glasgow. He discussed the widespread use of R in Glasgow across a broad range of fields and stressed the need to bring together R users for knowledge sharing. He also shared his work as an epidemiologist with the Ministry of Health in Belize for reporting COVID-19-related data for public policy and planning.

Antonio Hegar, Epidemiologist | Health Data Scientist | Public Health Researcher

Please share about your background and involvement with the RUGS group.

I am a Ph.D. student at Glasgow Caledonian University here in Scotland. My Ph.D. is focused on Machine Learning applied to large health databases, in other words, big data. Before starting my Ph.D. I worked for over five years at the local Ministry of Health in Belize, which is where I am from. R has been my primary statistical tool because of all the functionalities it offers. I come from an epidemiology/public health background and use R for my analyses.

I came over to the UK at the end of the pandemic in late 2021, and like many people, I was stuck at home due to all the restrictions. Nevertheless, I wanted to reach out and learn as much as possible about R from the respective experts across a broad range of fields and I knew that in Glasgow there is a very large R community. So I started looking around and found the R Glasgow group on Meetup and decided to give it a try. I joined, and luckily enough Andrew Baxter was organizing it at that time along with other members. I attended a few online meetings which were very productive and I have been following it up ever since. Since the beginning of this year, I have tried to be more active in the group by organizing events.

I have worked at the Ministry of Health in Belize using R and R Markdown to generate reports for COVID-19 outbreaks. So basically I applied mathematical modeling to infectious diseases which in this case was COVID-19, and made forecasts. My mathematical model took data from the local Ministry of Health and forecasted hospitalization rates, infection rates, and mortality rates. All this information was compiled into a report which was used by local officials and the Ministry of Health for planning.

What resources/techniques do/did you use?

As I mentioned, I used R Markdown for generating reports. As you might be aware that at the height of the pandemic, every country had a dashboard. I used Shiny for creating private dashboards displaying public health data for the Ministry of Health. Besides that, I also used the tidyverse and dplyr a lot.

I also did data imputation because whenever you are working with real-life data, especially public health data, there are a lot of gaps. So data imputation using mice and different R packages help you fill in the gaps in the data.

ggplot is another tool I used a lot for this project. When you are dealing with a non-technical audience you need really easy-to-understand charts and graphs which will help them easily and quickly understand what you are trying to display. So I did a lot of data visualization with ggplot and was constantly trying to look at new techniques to make data as attractive as possible.

Can you share what the R community is like in Glasgow and Scotland in general?

To be very honest with you, my response would be that I cannot really speak about it. As much as I have tried engaging, and of course, I am a member of the local R group, it’s proven to be much more difficult than I anticipated to actually have a cohesive understanding of the wider R community.

What I could say from what I have noticed from looking at university websites and looking at the profiles of different lecturers and researchers is that R is definitely used across the board in all of the major universities in Glasgow. I imagine it’s the same in many other major cities like Edinburgh in Scotland. So there are people using it for modeling, geospatial analysis, public health, epidemiology, finance, and economics.

I have met online or seen the profiles of many people who claim to be using R. But in terms of community or the lack of community, it’s all very dispersed at the moment. Which is another point that I wanted to discuss. On the surface, it appears that there is a lot of support and a lot of enthusiasm for using R at the individual as well as research department levels. But in terms of forming a cohesive group where people will come together and share ideas, that hasn’t been as forthcoming as I would have wanted. It’s less of an R community, in my point of view, and more of a network of R users with different nodes around the place. But not necessarily a functioning complete organization.

I would like to take this opportunity to reach out to R users living in Glasgow. R Glasgow can provide R users in Glasgow with a great opportunity to learn and grow together. I would also like to give a call for speakers. As we are hosting our events online, we would love to have speakers from around the globe join our events.

How do I Join?

Apply HereFind your local R user group The post R Applied to Epidemiology and Infectious Disease in Glasgow appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R Applied to Epidemiology and Infectious Disease in Glasgow

View Details

[This article was first published on r – bensstats, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.IntroductionRMarkdown is a powerful framework for writing a documents that contain a mixture of text, code, and the output of the code. Popular output formats for RMarkdown Documents (.Rmd) include HTML, PDF and Word Documents. It is also possible to output RMarkdown documents as part of a static website using blogdown package and is (still!) possible to publish RMarkdown documents to WordPress sites as well (like this one)!

Recently, I started to look into the possibility of outputting an .Rmd file as a Google Doc, but I was unable to locate any out-of-box solutions. After looking into the issue I developed a small function that makes it possible!

In this blog I share a function that enables users to write an .Rmd file as a Google Doc with the rmarkdown and googledrive packages.

The Code (so far)The code below essentially takes a the .Rmd file, renders it to a word document and uploads the word document to Google Drive as a Google Doc:

library(googledrive)library(rmarkdown)knit2docs<- function(rmd\_file, doc\_name=".Rmd to Docs"){ # Temporary File temp\_file<- tempfile(fileext = ".docx") rmarkdown::render(rmd\_file, "word\_document", output\_file = temp\_file) # Write to docs drive\_upload(temp\_file, name = doc\_name, type = "application/vnd.google-apps.document")} Now lets try this code out with an example .Rmd file:

knit2docs("Test.Rmd" ,doc\_name = ".Rmd to Docs") If we check the output on Google Docs, we can see the output:

Next stepsIn the present form, the the knit2docs function does what its supposed to, but it does have its limitations:

In its present form, the code is creates a new Google Doc each time the the function is called. It would be helpful to link the .Rmd file to a Google Doc so that each time it would be knit2docs() would be called, it would update a linked Doc (should one be specified).
With the above code, the rendered Google Doc is sent to the drive at the top level. It would be helpful to specify the drive location for the file being uploaded.
The exported formatting for fonts is default to Microsoft fonts. It would be nice to be able to update these fonts to Google fonts. Its could be that this is something that needs to be specified in the YAML header of the .Rmd file – I’ll have to look into it to see what exactly needs to be done.

To make this function readily available and develop it further, I created a Github Repository of the packaged code here. Add this code to your workflow by installing the knit2docs package! If you notice anything else or want to get involved in development, feel free to reach out!

ConclusionKnitting .Rmd files to Google Docs is something which is definitely intriguing. I’m surprised that I wasn’t able to find any readily developed solutions that were already made, but I’m happy that I have something basic put together so far! Its my hope that I can get this code and the next steps put together in a fully developed package and hopefully make knitting .Rmd files to Google Docs a part of rmarkdown users’ regular workflow!

Feel free to get involved by checking out the knit2docs repository here and be sure to give it a star for exposure!

Thank you for reading!

Want to see more of my content? Be sure to subscribe and never miss an update!

Type your email…

YouTubeFacebookPatreonTo leave a comment for the author, please follow the link and comment on their blog: r – bensstats.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: RObservations #47 : Knitting .Rmd Files to Google Docs with knit2docs

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.If you’ve played with data for long enough, you’re sure to run into some popular dead-ends. There are many of these, from misspelled location names and addresses to placeholder values entering the data pipeline because of a bug. One of the most frustrating things, if not the prime pain point of peril, is rounding numbers.

More often than not, you do not get what you expect, regardless of what language you are working on. The good thing about a team like ours is that we share not only our wins but also things that make us pull our hair out. This blog post is inspired by one such discussion.

Tired of validating data? Automate it! and generate reports in R and Shiny.

Hey, Does This Look Right To You? The Rounding Issue in Programming Most things start with an innocent bug, or at least, what looks like one. And that is where this story begins as well.

round(0.5)

If you enter the above in the R Console, unless you know all the context already, you would expect the standard mathematical procedure. The result should be 1, you’d tell yourself, and when you hit Return, the display will flash.

>> 0

But how? Surely, there is something wrong here. To check, let’s try another example.

round(1.5)> 2

And if you look closely, both of those numbers share a common property. They are even numbers. That is the first thing you learn when you learn about rounding in R, even if you learn about it the hard way: R rounds to Even.

But Why Round to Even in R? Who Gains From This? Tl;dr: You gain and it’s the only reasonable and deterministic way.

As it turns out, a lot of things. Rounding to Even is rooted in a standard called IEC 60559. The standard dictates that you round to the nearest even number. So, round(0.5) becomes 0 and even round(-1.5) becomes -2. The standard, however, is not agnostic to the operating system and representation error, which is where the second problem comes in but we will get to that. First, we must try to understand the reasoning behind this standard.

For that, let’s go to a piece of recent history and quote, Greg Snow and his famous explanation from 2008.

“The logic behind the round to even rule is that we are trying to represent an underlying continuous value and if x comes from a truly continuous distribution, then the probability that x==2.5 is 0 and the 2.5 was probably already rounded once from any values between 2.45 and 2.54999999999999…, if we use the round up on 0.5 rule that we learned in grade school, then the double rounding means that values between 2.45 and 2.50 will all round to 3 (having been rounded first to 2.5). This will tend to bias estimates upwards. To remove the bias we need to either go back to before the rounding to 2.5 (which is often impossible to impractical), or just round up half the time and round down half the time (or better would be to round proportional to how likely we are to see values below or above 2.5 rounded to 2.5, but that will be close to 50/50 for most underlying distributions). The stochastic approach would be to have the round function randomly choose which way to round, but deterministic types are not comfortable with that, so “round to even” was chosen (round to odd should work about the same) as a consistent rule that rounds up and down about 50/50.

If you are dealing with data where 2.5 is likely to represent an exact value (money for example), then you may do better by multiplying all values by 10 or 100 and working in integers, then converting back only for the final printing. Note that 2.50000001 rounds to 3, so if you keep more digits of accuracy until the final printing, then rounding will go in the expected direction, or you can add 0.000000001 (or other small number) to your values just before rounding, but that can bias your estimates upwards.” (source)

Maybe let’s take a look at a hands-on experiment.

We wanted to show you the effect of different rounding methods in R/python/javascript, but they don’t even implement methods different from Round to Even!

Using the RStudio IDE? Maximize your productivity with our favorite shortcuts and tips!

Fortunately julia implements different rounding methods and we can play with them.

The Experiment Let’s take a large vector of thousand random numbers from 0 to 1. Then let’s round each number in this vector to 1 decimal place in three different ways, Round to Even (the default), RoundUp and RoundDown. Note that RoundUp is equivalent to our school technique of rounding. Finally, we’ll compare which mean is closer to the mean of the original vector.

using Random, Statisticsx = rand(MersenneTwister(0), 1\_000)y1 = round.(x, digits=1)y2 = round.(x, RoundUp, digits=1)y3 = round.(x, RoundDown, digits=1) The Results So what are the means?

mean(x), mean(y1), mean(y2), mean(y3)(0.5006018120380458, 0.5012000000000001, 0.5496999999999999, 0.44970000000000004) We see that the mean of the vector after rounding to even is much closer to the mean of the original value, while rounding up or down results in the mean being 10% off. Rounding to even is a way to deal with rounding ties in a deterministic manner (i.e. without randomness) that proved to be the most simple and reliable, even though it might be weird at first.

But That’s Not The End Of It! There’s another issue why R works this way besides the round to even rule. There is another devil at play here and that’s the finite floating-point precision.

Hold on, that’s a lot of words. Okay, let’s take it one at a time. R only stores values till about 53 binary or about 22 floating points. In other words, anything after that digit is lost and is not accounted for. While this is not a problem for a number as small as 0.5, it proves to be a big hassle when the numbers are more precise, which simply means there are more digits after the decimal point.

This is not a problem specific to R, overall, but the limits above are specific to it. There is also an infamous R FAQ question dedicated to it. The following quote is the key point in that answer.

All other numbers are internally rounded to (typically) 53 binary digits accuracy. As a result, two floating point numbers will not reliably be equal unless they have been computed by the same algorithm, and not always even then.

So, overall, unless two numbers are processed in the exact same way, it is impossible to say with good confidence how R will equate them. But, you may be wondering, how does that apply to rounding?

When you have a number that exceeds the decimal places of 22, you would see a representation of it that is untrue since the precision is truncated.

For example:

```

num <- 2.499999999999999999999 > num[1] 2.5> round(num)[1] 2 ``` Here, when we output num, the precision is lost since the digits exceed 22. However, if we reduce the number of 9s, the precision is retained.

It is also connected to the infamous problem (when working in binary):

```

0.1 + 0.2 == 0.3[1] FALSE ``` (or when working in decimal):

x = ⅓ = 0.33333,3*x = 0.99999,3x =0.99999 ≠1, floor() and ceiling() While they are great alternatives, floor() and ceiling() are often not preferred since they round to a whole number at all times. Often, the use-case is to keep some decimal places intact. When we round, we are often looking to reduce precision while keeping a representation of the digits we are letting go of intact. These functions do not preserve that.

Why Not Truncate? Of course, truncating is an option but if we truncate 1.25 and 1.21 to one decimal place, both would be 1.2, and that would not be a correct representation either. Also, if you look at it, truncate is just round-down for positive numbers and round-up for negative numbers. We’ve seen it’s biased.

Okay, I’ll Just Use Python for My Rounding Needs This is all a bit too much, isn’t it? But then, life is rarely as simple when you boil it down to the brass tacks. Python is not devoid of its issues as well. Nor is any language, it’s the IEEE 754 standard .

However as written in the standard, the procedure is not hardware/implementation agnostic.

It might be funny, but python build-in rounding procedure works differently than the one in numpy:

In [1]: import numpy as npIn [2]: np.round(0.15, 1)Out[2]: 0.2In [3]: round(0.15, 1)Out[3]: 0.1 Is this a problem? Usually it’s not, but occasionally it might be. Of course you can find implementation details in the documentation.

Well, Let’s Go To JavaScript Then for All Things MathJavaScript has the Math.round() method to achieve rounding of decimals. It also has the Math.ceil() and Math.floor() methods. Math.round() method rounds to the nearest integer. If the fractional part of the number is greater than or equal to .5, the argument is rounded to the next higher integer. If the fractional part of the number is less than .5, the argument is rounded to the next lower integer.

To round off to a specific number of digits, the common solution is to divide the number by 10^x and then multiply the result by 10^x where x is the number of digits to round off to.

JavaScript seems to be more consistent with true rounding according to arithmetic principles.

What Should I Do Then? I Need Logical Rounding in R! First of all, never use float-point numbers to represent money-like numbers in computer memory. Either use the dedicated Decimal type if your language supports one (like in python or java) or convert money-float to integer by multiplying by some factor of 10 and avoid floats in general in those cases. With quantities that you usually use floats for, this shouldn’t be the issue. If it is, don’t use floats .

If, in case, you are looking for a function to emulate the true, logical rounding in R, you can go with this alternative we found on StackOverflow.

true\_round <- function(number, digits) { posneg <- sign(number) number <- abs(number) * 10^digits number <- number + 0.5 + sqrt(.Machine$double.eps) number <- trunc(number) number <- number / 10 ^ digits number * posneg} Another solution that could be adopted from javascript is to multiply the number and divide the result by 10^x where x is the number of digits to round off to. This is not perfect and does not give desired results always, but for some cases this might work.

Note how in the first 2 examples the results are different, but in the last 2 examples the results are consistent.

When the requirement is to check for equality between decimals up to x decimal digits. Then we can also simply use the difference and add a threshold to it. And do this without any rounding offs. So:

If max(abs(y – x)) > threshold then x and y are not equal. For example:

Rounding Out R, Julia and Python – Is It Over Yet? Yes, and to conclude, in this article, we dove into the shenanigans of rounding in R and other languages. Overall, things are messier than they appear on the surface. The decisions made to make a language work a certain way are bound to produce bad outcomes for certain use-cases. But the good thing about software, if it doesn’t work for you, there is always a way, or at least, some wiggle room to workaround.

Most languages have something wonky going on within them when it comes to rounding numbers and it is important to keep all this in mind so that when you face a perplexing number the next time during your analysis, you immediately know the usual suspect.

To round it all up (pun intended), stay sharp. It’s not the end of the world yet – it’s just an imprecise number, which may or may cause it someday.

Shiny app running slow? Don’t fret, maybe you were given a tough start with a slow database.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Rounding in R: Common Data Wrangling Frustrations and Workarounds in R, Julia, and Python

View Details

[This article was first published on Louise E. Sinks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Today’s TidyTuesday is based on a dataset about the London Marathon. The data is via a package by Nicola Rennie and there is an accompanying tutorial about how to scrape data from the web.

library(skimr)library(tidyverse)library(ggthemes)library(skimr) Load the data with the TidyTuesday package in the usual way.

tuesdata <- tidytuesdayR::tt\_load(2023, week = 17)

--- Compiling #TidyTuesday Information for 2023-04-25 ----

--- There are 2 files available ---

--- Starting Download ---

Downloading file 1 of 2: `winners.csv` Downloading file 2 of 2: `london\_marathon.csv`

--- Download complete ---

winners <- tuesdata$winnerslondon\_marathon <- tuesdata$london\_marathon There are two dataframes today: a winner’s dataframe and info about the marathon in general. Looking at the winner’s first.

winners %>% skim()

Variable type: character

| skim_variable | n_missing | complete_rate | min | max | empty | n_unique | whitespace | | --- | --- | --- | --- | --- | --- | --- | --- | | Category | 0 | 1 | 3 | 16 | 0 | 4 | 0 | | Athlete | 0 | 1 | 9 | 26 | 0 | 99 | 0 | | Nationality | 0 | 1 | 5 | 14 | 0 | 24 | 0 |

Variable type: difftime

| skim_variable | n_missing | complete_rate | min | max | median | n_unique | | --- | --- | --- | --- | --- | --- | --- | | Time | 0 | 1 | 5187 secs | 16143 secs | 02:07:55 | 158 |

Variable type: numeric

| skim_variable | n_missing | complete_rate | mean | sd | p0 | p25 | p50 | p75 | p100 | hist | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Year | 0 | 1 | 2001.61 | 11.83 | 1981 | 1991.5 | 2002 | 2012 | 2022 | ▇▇▇▇▇ |

All the data is complete.

Category and Nationality should probably be factors rather than characters.

winners <- winners %>% mutate(Category = as.factor(Category), Nationality = as.factor(Nationality))glimpse(winners)

Rows: 163Columns: 5$ Category <fct> Men, Men, Men, Men, Men, Men, Men, Men, Men, Men, Men, Men…$ Year <dbl> 1981, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989…$ Athlete <chr> "Dick Beardsley (Tie)", "Inge Simonsen (Tie)", "Hugh Jones…$ Nationality <fct> United States, Norway, United Kingdom, United Kingdom, Uni…$ Time <time> 02:11:48, 02:11:48, 02:09:24, 02:09:43, 02:09:57, 02:08:1… When I check to make sure everything is correct after making the factors, I see that some athletes have (Tie) after their name. If I do something with the runner’s names later (or if I care about ties) then I need to handle this.

There are four categories of races in our dataset- Men, Wheelchair Men, Wheelchair Women, Women.

I’m going to do some simple comparisons first. I’m looking for something interesting to focus on.

winners %>% filter(Category == "Men" | Category == "Women") %>% ggplot(aes(Category, Time)) + geom\_boxplot()

winners %>% filter(Category == "Wheelchair Men" | Category == "Wheelchair Women") %>% ggplot(aes(Category, Time)) + geom\_boxplot() Men are faster than women, in both classes of races.

winners %>% ggplot(aes(Category, Time)) + geom\_boxplot() Wheelchair races are faster than the running races. The Men/Women’s races are much closer than the Wheelchair races, and don’t have outliers.

I’m interested in ties. How many ties have there been? I’m going to make a column called “Tied” and remove (Tied) from the names. There are a few different ways to do this, but I’m going to use tidyr::separate(). I’m going to take the space too, when I separate out the name and (tie), so my name column is cleanly formatted. This is going to generate a bunch of NAs in the Tied column, but I’ll handle that in the next code chunk. I’m using " (" as my separator, so the Name column will not have the trailing space. The Tied column will have “Tie)” and a bunch of NAs, but I’ll clean this up in the next step. I’ve left warnings on for this code chunk, so you can see that the NAs are flagged for your notice.

Remember that special characters like ( need to be escaped out, so the appropriate regular expression for the separator is " \\(".

As a side note, as of winter 2022, separate has been superseded by a family of functions separate_wider_*.

winners\_tied <- winners %>% separate(Athlete, into = c("Name", "Tied"), " \\(")

Warning: Expected 2 pieces. Missing pieces filled with `NA` in 161 rows [3, 4, 5, 6, 7,8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, ...]. Now I’m going to clean up the Tied column.

winners\_tied <- winners\_tied %>% mutate(Tied = ifelse(is.na(Tied) == TRUE, FALSE, TRUE)) So how many ties are there?

winners\_tied %>% filter(Tied == TRUE)

```

A tibble: 2 × 6 Category Year Name Tied Nationality Time 1 Men 1981 Dick Beardsley TRUE United States 02:11:482 Men 1981 Inge Simonsen TRUE Norway 02:11:48

``` Just those first two, from the very first race. According to wikipedia, they crossed the finish line holding hands!

Anyone win more than once?

winners\_tied %>% group\_by(Name) %>% count(Name, sort = TRUE) %>% filter(n > 1)

```

A tibble: 34 × 2# Groups: Name [34] Name n 1 David Weir 8 2 Tanni Grey-Thompson 6 3 David Holding 4 4 Eliud Kipchoge 4 5 Francesca Porcellato 4 6 Ingrid Kristiansen 4 7 Tatyana McFadden 4 8 António Pinto 3 9 Dionicio Cerón 310 Heinz Frei 3# ℹ 24 more rows

``` More than I expected! David Weir has won the most London Marathons, with 8 wins in the Men’s Wheelchair race category. How has his race time changed over the years?

winners\_tied %>% filter(Name == "David Weir") %>% ggplot(aes(Year, Time)) + geom\_point() It looks like his first race was much slower than the other times he has won. It turns out he has competed in the London Marathon 23 times, and placed 5th in the 2023 Marathon which happened this weekend. His time on Saturday was 01:32:45. This is interesting, because it is quite similar to his more recent races.

Has the field for this race gotten faster?

winners %>% filter(Category == "Wheelchair Men") %>% ggplot(aes(Year, Time)) + geom\_point() Yes, and quite clearly too! How many years would he have won with this year’s time?

time\_2023 <- as.difftime("01:32:45", format = "%H:%M:%S", units = "auto")slow\_wins <- winners %>% filter(Category == "Wheelchair Men" & Time >= time\_2023) fast\_wins <- winners %>% filter(Category == "Wheelchair Men" & Time < time\_2023) So 12 years the wins were faster than his time this year, but 27 years were slower. And just to note, 6 of those 12 faster wins are held by David Weir…

Let’s throw together a visual for this and make it out TidyTuesday viz. Should be simple, right?

winners %>% filter(Category == "Wheelchair Men") %>% ggplot(aes(Year, Time)) + geom\_point() + geom\_hline(yintercept = time\_2023) Hmm. ggplot is not playing well with our difftimes.

str(time\_2023)

'difftime' num 1.54583333333333 - attr(*, "units")= chr "hours"

str(winners$Time[1])

'hms' num 02:11:48 - attr(*, "units")= chr "secs" Apparently, our race time from winners is actually hms and not difftime. Skim reported it was a difftime. Our difftime has units of hours, while Time has units of seconds. This is probably due to be setting units to “auto” when I did the conversion. Interesting that dplyr filtering handles this smoothly, but ggplot doesn’t.

time\_2023 <- as.difftime("01:32:45", format = "%H:%M:%S", units = "sec")str(time\_2023)

'difftime' num 5565 - attr(*, "units")= chr "secs" I’m going to create a TRUE/FALSE column for if David Weir won that I will color code the win by.

winners %>% filter(Category == "Wheelchair Men") %>% mutate(Weir = ifelse(Athlete == "David Weir", TRUE, FALSE)) %>% ggplot(aes(Year, Time, color = Weir)) + geom\_point() + geom\_hline(yintercept = time\_2023) Okay, now lets clean up the formatting. I need to:

Apply a theme. I like theme_classic() or theme_pander() as a clean base for my graphs.
Remove the legend. This needs to go after the theme_classic/pander() call or the legend will reappear.
Add title, subtitle and data source
Fix the y-axis units
Change the colors for the David Weir points.
Label my horizontal line.

4 is apparently quite challenging. Apparently, the time axis should be displayed as hms, so it isn’t clear to me why fractional seconds are shown. I tried a bunch of different suggestion from the web, but the top answer to this question is what worked.

It actually doesn’t matter if the aesthetic is difftime or hms. The key is that the label section of scale_y_time needs a formatted string generated from strftime.

winners %>% filter(Category == "Wheelchair Men") %>% mutate(Weir = ifelse(Athlete == "David Weir", TRUE, FALSE)) %>% ggplot(aes(Year, Time, color = Weir)) + geom\_point() + scale\_color\_manual(values = c("black", "cyan2")) + geom\_hline(yintercept = (time\_2023), color = "cyan4") + annotate( "text", x = 1990, y = time\_2023 - 200, label = "David Weir's 2023 time", color = "cyan4" ) + scale\_y\_time(name = "Winning Time", labels = function(l) strftime(l, '%H:%M:%S')) + labs(title = "Race times for the London Marathon: Men's Wheelchair Race", subtitle = "compared to David Weir's wins", caption = "data from https://github.com/nrennie/LondonMarathon") + theme\_classic(12) + theme(legend.position = "none") CitationBibTeX citation: @online{e.sinks2023, author = {Louise E. Sinks}, title = {TidyTuesday {Week} 17: {London} {Marathon}}, date = {2023-04-25}, url = {https://lsinks.github.io/posts/2023-04-25-tidytuesday-marathon/marathon}, langid = {en}} For attribution, please cite this work as:Louise E. Sinks. 2023. “TidyTuesday Week 17: LondonMarathon.” April 25, 2023. https://lsinks.github.io/posts/2023-04-25-tidytuesday-marathon/marathon. To leave a comment for the author, please follow the link and comment on their blog: Louise E. Sinks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: TidyTuesday Week 17: London Marathon

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionWelcome to the {tidyAML} Model Builder, a Shiny web application that allows you to build predictive models using the tidyAML and Parsnip packages in R.

Let’s dive into the code to understand how it works!

Load LibrariesFirst, we load the necessary packages:

shiny
tidyAML
recipes
DT
glmnet.

library(shiny)library(tidyAML)library(recipes)library(DT)library(glmnet) UINext, we define the user interface (UI) of the Shiny app using the fluidPage() function from the shiny package. The UI consists of a title panel, a sidebar panel, and a main panel.

ui <- fluidPage( titlePanel("TidyAML Model Builder"), sidebarLayout( sidebarPanel( fileInput("file", "Upload your data file (csv or txt):"), selectInput( "dataset", "Choose a built-in dataset:", choices = c("mtcars", "iris") ), selectInput( "predictor\_col", "Select the predictor column:", choices = NULL ), selectInput( "model\_type", "Select a model type:", choices = c("regression", "classification") ), selectInput( "model\_fn", "Select a model function:", choices = c("lm", "glm", "glmnet") ), actionButton("build\_model", "Build Model"), verbatimTextOutput("recipe\_output") ), mainPanel( verbatimTextOutput("model\_table") ) )) The sidebarPanel() contains several input elements that allow the user to specify the dataset, the predictor column, the type of model, and the model function. There is also an input element that allows the user to upload their own data file. The actionButton() is used to trigger the model building process. Finally, the verbatimTextOutput() element is used to display the output of the model building process.

The mainPanel() contains a single verbatimTextOutput() element that displays the output of the model building process.

Next, we define the server function, which is responsible for handling the user inputs and building the predictive models. The server function takes three arguments:input, output, and session.

server <- function(input, output, session){ ...} We start by defining a reactive expression called data. This expression reads in the user-specified dataset or data file and updates the predictor_col select input with the names of the columns of the dataset.

data <- reactive({ if (!is.null(input$file)) { df <- read.csv( input$file$datapath, header = TRUE, stringsAsFactors = FALSE ) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } else if (!is.null(input$dataset)) { df <- get(input$dataset) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } }) The first reactive expression, data, reads in the data file uploaded by the user or selects a built-in dataset, depending on which option the user chooses. If the user uploads a file, the read.csv() function is used to read the data file into a data frame. If the user selects a built-in dataset, the get() function is used to retrieve the data frame associated with that dataset. In both cases, the column names of the data frame are used to update the choices in the predictor_col select input, so that the user can select which column to use as the predictor variable.

The next reactive expression, recipe_obj, creates a recipe object based on thepredictor_col selected by the user and the data frame returned by data(). The as.formula() function is used to create a formula that specifies the predictor column as the response variable and all other columns as the predictors. The resulting formula is passed to the recipe() function, along with the data frame. The step_normalize() function is then used to standardize all numeric predictors (except for the outcome variable) to have a mean of 0 and a standard deviation of 1. The resulting recipe object is returned by the reactive expression.

recipe\_obj <- eventReactive(input$predictor\_col, { rec <- recipe(as.formula(paste(input$predictor\_col, "~ .")), data = data()) |> step\_normalize(all\_numeric(), -all\_outcomes()) return(rec) }) The model_fn reactive expression uses a switch() statement to determine which model function to use based on the model_fn select input. The available options are "lm" (for linear regression), "glm" (for generalized linear models), and "glmnet" (for regularized linear models).

model\_fn <- reactive({ switch( input$model\_fn, "lm" = "lm", "glm" = "glm", "glmnet" = "glmnet" ) }) The last reactive expression, model, uses the fast_regression() or fast_classification() functions from the tidyAML package to build a regression or classification model based on the data, recipe, and model function selected by the user. The resulting model object is returned by the reactive expression.

model <- eventReactive(input$build\_model, { if (input$model\_type == "regression") { mod <- fast\_regression( .data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_fn() ) } else if (input$model\_type == "classification") { mod <- fast\_classification( .data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_fn() ) } return(mod) }) Finally we output the summary of the recipe_obj and print the resulting tibble of model(s) to the screen.

output$recipe\_output <- renderPrint({ if (!is.null(input$predictor\_col)) { summary(recipe\_obj()) } }) output$model\_table <- renderPrint({ if (input$build\_model > 0) { print(model()) } }) And of course, we cannot serve our app until we run the following line:

shinyApp(ui = ui, server = server) I hope you have enjoyed this post. Please steal this code and see what you can do with it. I am trying to figure out how to print the tibble using the DT package so maybe in another post.

Full Shiny AppHere are some pictures

Making a recipe changeSingle Model OutputTwo Model Output with one successful failure library(shiny)library(tidyAML)library(recipes)library(DT)library(glmnet)ui <- fluidPage( titlePanel("TidyAML Model Builder"), sidebarLayout( sidebarPanel( fileInput("file", "Upload your data file (csv or txt):"), selectInput("dataset", "Choose a built-in dataset:", choices = c("mtcars", "iris") ), selectInput("predictor\_col", "Select the predictor column:", choices = NULL ), selectInput("model\_type", "Select a model type:", choices = c("regression", "classification")), selectInput("model\_fn", "Select a model function:", choices = c("lm", "glm", "glmnet") ), actionButton("build\_model", "Build Model"), verbatimTextOutput("recipe\_output") ), mainPanel( verbatimTextOutput("model\_table") ) ))server <- function(input, output, session) { data <- reactive({ if (!is.null(input$file)) { df <- read.csv( input$file$datapath, header = TRUE, stringsAsFactors = FALSE ) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } else if (!is.null(input$dataset)) { df <- get(input$dataset) updateSelectInput( session, "predictor\_col", choices = names(df) ) return(df) } }) recipe\_obj <- eventReactive(input$predictor\_col, { rec <- recipe(as.formula(paste(input$predictor\_col, "~ .")), data = data() ) |> step\_normalize(all\_numeric(), -all\_outcomes()) return(rec) }) model\_fn <- reactive({ switch(input$model\_fn, "lm" = "lm", "glm" = "glm", "glmnet" = "glmnet") }) model <- eventReactive(input$build\_model, { if (input$model\_type == "regression") { mod <- fast\_regression(.data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_fn()) } else if (input$model\_type == "classification") { mod <- fast\_classification(.data = data(), .rec\_obj = recipe\_obj(), .parsnip\_eng = model\_fn()) } return(mod) }) output$recipe\_output <- renderPrint({ if (!is.null(input$predictor\_col)) { summary(recipe\_obj()) } }) output$model\_table <- renderPrint({ if (input$build\_model > 0) { print(model()) } }) }shinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building models with {shiny} and {tidyAML} Part 1

View Details

[This article was first published on R-post on Cosima Meyer, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I spent the last weeks building two small Mastodon bots with the goal in mind to make contributions by PyLadies and R-Ladies more visible. PyLadies and R-Ladies are global non-profit organizations that seek to empower and promote greater diversity in the coding world.

The project started when I saw a post on LinkedIn that mentioned several data people to follow – all being male. When asked why there wasn’t more diversity on the list, the post author replied that they should get more visible to be recognized – so I decided to do my share here.

What was initially an idea, became the development of two bots (a PyLadies bot and an R-Ladies bot), a curated blog list with awesome PyLadies on it, and me spending hours reaching out to PyLadies and encouraging them to put their blogs on the list. And I can tell you: It’s been great weeks – seeing all the beautiful websites, learning about, and “meeting” new people is always fantastic. It also showed me (once more) how multi-faceted the community is and how much there is to learn!

I’m planning on writing a series of blog posts that will cover the entire process:

Setting up the Awesome PyLadies‘ Blogs Repository (and extending the R-Ladies list)
Building Mastodon Bots
Deploying Mastodon bots using AWS and GitHub Actions

When I started thinking about the bots, I had (and still have) many ideas in mind. While reposting posts that mention a specific hashtag is more or less straightforward and doesn’t require any additional input – but sharing R-Ladies’ and PyLadies’ blog posts does require a (curated) list with RSS feeds. Luckily there was already the Awesome R-Ladies’ Blog Repository in place that I could use to extend and collect RSS feeds for the blogs by R-Ladies. I contacted all the R-Ladies that were already on the list and asked them for their consent to add their RSS feeds and Mastodon handles.

Setting up the Awesome PyLadies’ Blogs RepositoryAnd this is when also the Awesome PyLadies’ Blogs Repository started

Alternative textScreenshot of the Awesome PyLadies’ Blogs repository showing some text (What is this repository about) and images with a list of blogs (arranged as a grid)I am incredibly thankful that I could build upon the Awesome R-Ladies’ Blog Repository. So here are some credits due: It is deeply inspired by the Awesome R-Ladies’ Blogs Repository (which is the entire underlying basis). Without this repository, I wouldn’t have been able to get so quickly up-and-running with the repository – so all credit goes to them!

What is the repository about?As mentioned above, the repository relies heavily on the Awesome R-Ladies blogs repository and I can only repeat their words of what the repository is about. The goals include collecting PyLadies’ blogs and making PyLadies more visible. This includes those who identify as a minority gender (including but not limited to cis/trans women, trans men, non-binary, genderqueer, & agender). With your submission, you agree that these entries will be used for the PyLadies’ Mastodon bot, which will post (new) PyLadies’ blog entries to promote the work of PyLadies around the world.

How did I find all the PyLadies?That was more challenging than expected and I went with various channels.First, I posted in different Slack groups as well as on Mastodon/Twitter about the GitHub repository and encouraged people to add their blogs to the list. But this didn’t seem to work as well as expected. So I went on and reached out to people individually.To find people,

I browsed past Python conferences and their speakers’ lists, got the names (ideally a website was linked – but often it wasn’t), googled the people and figured out whether they had a website (and a blog), reached out to them (by Slack, e-mail, contact forms, or LinkedIn)
I checked who posted about Python conferences on both Twitter and Mastodon
I checked Medium, Mastodon, and Twitter and looked for keywords such as “pyladies”
I tracked down the GitHub profiles of those who contributed to the PyLadies repositories and looked for blogs
of course, I also used ChatGPT (but it didn’t work as well as expected – it gave me lists full of made-up names or made-up websites. I cannot tell whether some of the websites worked in the past but I’m also blaming it on hallucination ‍ (and also on my prompt writing))

And while I was already on it, I did the same for R-Ladies and also reached out to them I also extended the Awesome R-Ladies list by reaching out to those who weren’t on the list and encouraged them to join. If you didn’t get a message from me, please feel free to reach out (or add the information yourself, if you want ).

I’m truly sorry if I reached out to you more than once – it’s because, I have to admit, keeping track of so many beautiful blogs is challenging and I think yours should be on the list!

If you have more ideas, please let me know – I’m feeling a bit like Sherlock who is thinking about new ways how to find people

How do you submit your blog?It is (hopefully) easy and all described here. In a nutshell, there are two options:

Open an issue with your blog info and I’ll create the JSON for you
Create a new file and a PR with your blog info (there’s even a short cut link that you can use to create the file)

All details of what can be included and which format is best are provided in the contribution guidelines. If there are problems with the PR, opening an issue, or something else – please feel free to reach out! We’ll make our way through the GitHub adventure together

If you have a website or blog feed yourself, please feel free to submit it to the repositories! We’re happy about every single contribution

Awesome PyLadies’ Blogs Repository
Awesome R-Ladies’ Blogs Repository

And as always, I’m more than happy about feedback or suggestions. Please feel free to reach out to me or open an issue!

To leave a comment for the author, please follow the link and comment on their blog: R-post on Cosima Meyer.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building Mastodon Bots and Promoting the Community – Part 1

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We designed the rOpenSci Champions Program with a mentorship aspect. Mentoring plays a significant role in the growth and development of both mentors and mentees alike. In our program, each Champion has a mentor who accompanies them during their training and development of their project.

In this series of blog posts, we introduce you to the ten teams of this first cohort and what they will be working on in the program.

Meet Bilikisu Wunmi Olatunji from Nigeria and Christina Maimone from United States!

Bilikisu Wunmi Olatunji – ChampionBilikisu Wunmi OlatunjiI’m a DataScientist and RShiny Developer from Nigeria based in the country’s capitalcity, Abuja. I’m a certified RStudio Instructor for Tidyverse and R Shiny and have a Master’s of Science from the University of Liverpool, UK.

I’m also the founder and co-organizer of Abuja R Users Group andR-Ladies Abuja. My interest is in geospatial data science, and I have been working onseveral related projects using R.

I’ll be working on a package to ease theuse of geospatial data for Microplanning in public health. I’ll also be volunteering on other projects to strengthen the R open science community.

Christina Maimone – MentorChristina MaimoneI am the associate director of research data services for Northwestern University’s Research Computing Services team (Chicago, USA). Our team provides data science, data visualization, software development, and data management support and training for researchers at the university. I also teach data visualization in the MS in Health Analytics program at Northwestern. I’m a founding member of the US Research Software Engineer (US-RSE) Association steering committee, and I’m active in the research computing and data community with organizations including CaRCC.

I’ve been using and teaching R for over 20 years. My background is in computational social science. I have a PhD in political science and MS in statistics from Stanford University.

I’ve been a reviewer for JOSS, but not for any rOpenSci packages yet, so the Champions program is a way for me to help contribute to the community. Building and maintaining communities is hard, so I’m happy to support efforts that encourage more people to actively engage and become new leaders. I hope to see more R developers active in the research software engineering community as well.

We’re excited to have both Bilikisu and Christina as part of the Champions program and can’t wait to see what this partnership yields. Stay-tuned for the next series of introductions .

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci Champions Program Teams: Meet Bilikisu Wunmi Olatunji and Christina Maimone

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to use RMarkdown and Quarto! Join our workshop on RMarkdown and Quarto – Mastering the Basics which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: RMarkdown and Quarto – Mastering the Basics

Date: Thursday, May 25th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Indrek Seppo, a seasoned R programming expert, brings over 20 years of experience from the academic, private, and public sectors to the table. With more than a decade of teaching R under his belt, Indrek’s passionate teaching style has consistently led his courses to top the student feedback charts and has inspired hundreds of upcoming data analysts to embrace R (and Baby Shark).

Description: Discover the power of RMarkdown and its next-generation counterpart, Quarto, to create stunning reports, slides, dashboards, and even entire books—all within the RStudio environment. This session will cover the fundamentals of markdown, guiding you through the process of formatting documents and incorporating R code, tables, and graphs seamlessly. If you’ve never explored these tools before, prepare to be amazed by their capabilities. Learn how to generate reproducible reports and research with ease, enhancing your productivity and efficiency in the world of data analysis.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

RMarkdown and Quarto – Mastering the Basics workshop was first posted on April 24, 2023 at 11:50 am.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: RMarkdown and Quarto – Mastering the Basics workshop

View Details

[This article was first published on R – Eran Raviv, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.R or Python? who cares! Which editor? now that’s a different story.

I like Rstudio for many reasons. Outside the personal, Rstudio allows you to write both R + Python = Rython in the same script. Apart from that, the editor’s level of complexity is well-balanced, not functionality-overkill like some, nor too simplistic like some others. In this post I share how to save time with snippets (easy in Rstudio). Snippets save time by reducing the amount of typing required, it’s the most convenient way to program copy-pasting into the machine’s memory.

In addition to useful built-ins snippets provided by Rstudio like lib or fun for R and imp or def for python, you can write your own snippets. Below are a couple I wrote myself that you might find helpful. But first we start with how to use snippets.

How to use snippets in Rstudio?The following two images are pasted here from owenjonesuob.

To get a list of built-in snippets in Rstudio you go to Tools > global Option > code > edit snippet.

The last three snippets are written by the user. You can see what they do here:

So for example if you type hh and SHIFT+TAB, the two lines defined in the snippets are pasted.

Use case, pleaseIndeed. Say you want to add an argument to a function you wrote. The argument is for whether you would like to have the execution time reported or not. Call it report_elapsed. Like so:

my\_sum <- function(x){ sum(x) Sys.sleep(1)}# Add the report\_elapsed argument my\_sum <- function(x, report\_elapsed= F){ ptm <- proc.time() out <- sum(x) Sys.sleep(1) if(report\_elapse) { cat(prettyNum( (proc.time() - ptm)[3]/60, digits=5), "mins", "\n") } out}my\_sum(rnorm(100), report\_elapsed= T)0.016833 mins [1] 6.56 What you can do is to create the snippet timeit:

snippet timeitptm <- proc.time()${1: #your function}if(report\_elapse) {cat(prettyNum( (proc.time() - ptm)[3]/60, digits=5), "mins", "\n") } Add that snippet to your snippet file. Now each time you want to add an execution time argument (report_elapsed) to your function you can wrap it quickly with the necessary lines. Just type timeit, SHIFT+TAB and paste your function inside.

Can I do in Python?Why not. The snippet is

snippet timeittic = time.perf\_counter()${1: # your function } if report\_elapsed: print("{:.3f}".format(time.perf\_counter() - tic), "seconds") and the function is now (after adding the report_elapsed and return(out)):

def my\_sum(x, report\_elapsed= False): tic = time.perf\_counter() out= sum(x) time.sleep(0.2) if report\_elapsed: print("{:.3f}".format(time.perf\_counter() - tic), "seconds") return(out) Footnotes and a couple of good books about coding As a footnote, I don’t know where I can find a snippet gallery, but that would be good venue to share community snippets.

97 Things Every Programmer Should Know

Coding All-in-One For Dummies

Subscribe for future posts (no spam)To leave a comment for the author, please follow the link and comment on their blog: R – Eran Raviv.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Rython tips and tricks – Snippets

97 Things Every Programmer Should Know: Collective Wisdom from the Experts

View Details

[This article was first published on R-Bloggers – Learning Machines, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
R is a powerful programming language and environment for statistical computing and graphics. In this post, we will provide a quick introduction to R using the famous iris dataset.

We will cover loading data, exploring the dataset, basic data manipulation, and plotting. By the end, you should have a good understanding of how to get started with R, so read on!

This is the introduction that I wished I had back when I started analyzing data! Disclosure: part of the post and code were written with the assistance of ChatGPT-4, the concept and ideas herein are my own.

You can also watch the video for this post (in German):

Getting StartedFirst, download and install R from the Comprehensive R Archive Network (CRAN): https://cran.r-project.org/

Next, download and install RStudio, an integrated development environment (IDE) for R: https://posit.co/download/rstudio-desktop/

Ok, and now see for yourself how easy it is to analyze data with R! We will use the equivalent of “Hello World” for data science, the famous iris dataset. Of course, you can also easily load your own data into R: just click on “Import Dataset” in the “Environment” tab in the upper right window of RStudio and follow the steps from there!

The iris dataset is a classic and widely used dataset in the field of data science and machine learning. The dataset consists of 150 samples from three species of iris flowers: Iris setosa, Iris versicolor, and Iris virginica. Each sample contains four features, which are the lengths and widths of the sepals and petals (in centimeters). The dataset has become a popular choice for testing and demonstrating various data analysis and machine learning techniques due to its simplicity and well-defined structure. The iris dataset comes pre-loaded in R, so no need to import any external files.

Iris flowers, picture created with DALL-E

Exploring the DatasetTo get an overview of the dataset, use the head() and summary() functions:

```

Display the first 6 rows of the datasethead(iris)## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## 1 5.1 3.5 1.4 0.2 setosa## 2 4.9 3.0 1.4 0.2 setosa## 3 4.7 3.2 1.3 0.2 setosa## 4 4.6 3.1 1.5 0.2 setosa## 5 5.0 3.6 1.4 0.2 setosa## 6 5.4 3.9 1.7 0.4 setosa# Get summary statistics of the datasetsummary(iris)## Sepal.Length Sepal.Width Petal.Length Petal.Width ## Min. :4.300 Min. :2.000 Min. :1.000 Min. :0.100 ## 1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 1st Qu.:0.300 ## Median :5.800 Median :3.000 Median :4.350 Median :1.300 ## Mean :5.843 Mean :3.057 Mean :3.758 Mean :1.199 ## 3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 3rd Qu.:1.800 ## Max. :7.900 Max. :4.400 Max. :6.900 Max. :2.500 ## Species ## setosa :50 ## versicolor:50 ## virginica :50

`` Thesummary()` function in R provides a quick overview of the main statistical measures for numerical data. Here are short descriptions of each measure:

Minimum (Min.): The smallest value in the dataset.
1st Quartile (1st Qu.): The value that separates the lowest 25% of the data from the remaining 75%; also known as the 25th percentile.
Median (2nd Qu.): The middle value that separates the lower and upper halves of the data; also known as the 50th percentile.
Mean: The arithmetic average of the data values, calculated by adding up all the values and dividing by the total number of values.
3rd Quartile (3rd Qu.): The value that separates the lowest 75% of the data from the highest 25%; also known as the 75th percentile.
Maximum (Max.): The largest value in the dataset.

These statistical measures give you a quick snapshot of the central tendency, dispersion, and overall distribution of your numerical data.

Data Manipulationa) Subsetting the dataset

To select specific columns in the dataset, use the $ operator or the [] brackets:

```

Select the Petal.Width columniris$Petal.Width## [1] 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 0.2 0.2 0.1 0.1 0.2 0.4 0.4 0.3## [19] 0.3 0.3 0.2 0.4 0.2 0.5 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.4 0.1 0.2 0.2 0.2## [37] 0.2 0.1 0.2 0.2 0.3 0.3 0.2 0.6 0.4 0.3 0.2 0.2 0.2 0.2 1.4 1.5 1.5 1.3## [55] 1.5 1.3 1.6 1.0 1.3 1.4 1.0 1.5 1.0 1.4 1.3 1.4 1.5 1.0 1.5 1.1 1.8 1.3## [73] 1.5 1.2 1.3 1.4 1.4 1.7 1.5 1.0 1.1 1.0 1.2 1.6 1.5 1.6 1.5 1.3 1.3 1.3## [91] 1.2 1.4 1.2 1.0 1.3 1.2 1.3 1.3 1.1 1.3 2.5 1.9 2.1 1.8 2.2 2.1 1.7 1.8## [109] 1.8 2.5 2.0 1.9 2.1 2.0 2.4 2.3 1.8 2.2 2.3 1.5 2.3 2.0 2.0 1.8 2.1 1.8## [127] 1.8 1.8 2.1 1.6 1.9 2.0 2.2 1.5 1.4 2.3 2.4 1.8 1.8 2.1 2.4 2.3 1.9 2.3## [145] 2.5 2.3 1.9 2.0 2.3 1.8# Select the first three columns (the first part before the comma is for selecting the rows, the second is for the columns - if left free, nothing is filtered)iris[ , 1:3]## Sepal.Length Sepal.Width Petal.Length## 1 5.1 3.5 1.4## 2 4.9 3.0 1.4## 3 4.7 3.2 1.3## 4 4.6 3.1 1.5## 5 5.0 3.6 1.4## 6 5.4 3.9 1.7## 7 4.6 3.4 1.4## 8 5.0 3.4 1.5## 9 4.4 2.9 1.4## 10 4.9 3.1 1.5## 11 5.4 3.7 1.5## 12 4.8 3.4 1.6## 13 4.8 3.0 1.4## 14 4.3 3.0 1.1## 15 5.8 4.0 1.2## 16 5.7 4.4 1.5## 17 5.4 3.9 1.3## 18 5.1 3.5 1.4## 19 5.7 3.8 1.7## 20 5.1 3.8 1.5## 21 5.4 3.4 1.7## 22 5.1 3.7 1.5## 23 4.6 3.6 1.0## 24 5.1 3.3 1.7## 25 4.8 3.4 1.9## 26 5.0 3.0 1.6## 27 5.0 3.4 1.6## 28 5.2 3.5 1.5## 29 5.2 3.4 1.4## 30 4.7 3.2 1.6## 31 4.8 3.1 1.6## 32 5.4 3.4 1.5## 33 5.2 4.1 1.5## 34 5.5 4.2 1.4## 35 4.9 3.1 1.5## 36 5.0 3.2 1.2## 37 5.5 3.5 1.3## 38 4.9 3.6 1.4## 39 4.4 3.0 1.3## 40 5.1 3.4 1.5## 41 5.0 3.5 1.3## 42 4.5 2.3 1.3## 43 4.4 3.2 1.3## 44 5.0 3.5 1.6## 45 5.1 3.8 1.9## 46 4.8 3.0 1.4## 47 5.1 3.8 1.6## 48 4.6 3.2 1.4## 49 5.3 3.7 1.5## 50 5.0 3.3 1.4## 51 7.0 3.2 4.7## 52 6.4 3.2 4.5## 53 6.9 3.1 4.9## 54 5.5 2.3 4.0## 55 6.5 2.8 4.6## 56 5.7 2.8 4.5## 57 6.3 3.3 4.7## 58 4.9 2.4 3.3## 59 6.6 2.9 4.6## 60 5.2 2.7 3.9## 61 5.0 2.0 3.5## 62 5.9 3.0 4.2## 63 6.0 2.2 4.0## 64 6.1 2.9 4.7## 65 5.6 2.9 3.6## 66 6.7 3.1 4.4## 67 5.6 3.0 4.5## 68 5.8 2.7 4.1## 69 6.2 2.2 4.5## 70 5.6 2.5 3.9## 71 5.9 3.2 4.8## 72 6.1 2.8 4.0## 73 6.3 2.5 4.9## 74 6.1 2.8 4.7## 75 6.4 2.9 4.3## 76 6.6 3.0 4.4## 77 6.8 2.8 4.8## 78 6.7 3.0 5.0## 79 6.0 2.9 4.5## 80 5.7 2.6 3.5## 81 5.5 2.4 3.8## 82 5.5 2.4 3.7## 83 5.8 2.7 3.9## 84 6.0 2.7 5.1## 85 5.4 3.0 4.5## 86 6.0 3.4 4.5## 87 6.7 3.1 4.7## 88 6.3 2.3 4.4## 89 5.6 3.0 4.1## 90 5.5 2.5 4.0## 91 5.5 2.6 4.4## 92 6.1 3.0 4.6## 93 5.8 2.6 4.0## 94 5.0 2.3 3.3## 95 5.6 2.7 4.2## 96 5.7 3.0 4.2## 97 5.7 2.9 4.2## 98 6.2 2.9 4.3## 99 5.1 2.5 3.0## 100 5.7 2.8 4.1## 101 6.3 3.3 6.0## 102 5.8 2.7 5.1## 103 7.1 3.0 5.9## 104 6.3 2.9 5.6## 105 6.5 3.0 5.8## 106 7.6 3.0 6.6## 107 4.9 2.5 4.5## 108 7.3 2.9 6.3## 109 6.7 2.5 5.8## 110 7.2 3.6 6.1## 111 6.5 3.2 5.1## 112 6.4 2.7 5.3## 113 6.8 3.0 5.5## 114 5.7 2.5 5.0## 115 5.8 2.8 5.1## 116 6.4 3.2 5.3## 117 6.5 3.0 5.5## 118 7.7 3.8 6.7## 119 7.7 2.6 6.9## 120 6.0 2.2 5.0## 121 6.9 3.2 5.7## 122 5.6 2.8 4.9## 123 7.7 2.8 6.7## 124 6.3 2.7 4.9## 125 6.7 3.3 5.7## 126 7.2 3.2 6.0## 127 6.2 2.8 4.8## 128 6.1 3.0 4.9## 129 6.4 2.8 5.6## 130 7.2 3.0 5.8## 131 7.4 2.8 6.1## 132 7.9 3.8 6.4## 133 6.4 2.8 5.6## 134 6.3 2.8 5.1## 135 6.1 2.6 5.6## 136 7.7 3.0 6.1## 137 6.3 3.4 5.6## 138 6.4 3.1 5.5## 139 6.0 3.0 4.8## 140 6.9 3.1 5.4## 141 6.7 3.1 5.6## 142 6.9 3.1 5.1## 143 5.8 2.7 5.1## 144 6.8 3.2 5.9## 145 6.7 3.3 5.7## 146 6.7 3.0 5.2## 147 6.3 2.5 5.0## 148 6.5 3.0 5.2## 149 6.2 3.4 5.4## 150 5.9 3.0 5.1

``` b) Filtering the dataset

To filter the dataset based on a condition, use the subset() function:

```

Select rows where Species is "setosa"subset(iris, Species == "setosa")## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## 1 5.1 3.5 1.4 0.2 setosa## 2 4.9 3.0 1.4 0.2 setosa## 3 4.7 3.2 1.3 0.2 setosa## 4 4.6 3.1 1.5 0.2 setosa## 5 5.0 3.6 1.4 0.2 setosa## 6 5.4 3.9 1.7 0.4 setosa## 7 4.6 3.4 1.4 0.3 setosa## 8 5.0 3.4 1.5 0.2 setosa## 9 4.4 2.9 1.4 0.2 setosa## 10 4.9 3.1 1.5 0.1 setosa## 11 5.4 3.7 1.5 0.2 setosa## 12 4.8 3.4 1.6 0.2 setosa## 13 4.8 3.0 1.4 0.1 setosa## 14 4.3 3.0 1.1 0.1 setosa## 15 5.8 4.0 1.2 0.2 setosa## 16 5.7 4.4 1.5 0.4 setosa## 17 5.4 3.9 1.3 0.4 setosa## 18 5.1 3.5 1.4 0.3 setosa## 19 5.7 3.8 1.7 0.3 setosa## 20 5.1 3.8 1.5 0.3 setosa## 21 5.4 3.4 1.7 0.2 setosa## 22 5.1 3.7 1.5 0.4 setosa## 23 4.6 3.6 1.0 0.2 setosa## 24 5.1 3.3 1.7 0.5 setosa## 25 4.8 3.4 1.9 0.2 setosa## 26 5.0 3.0 1.6 0.2 setosa## 27 5.0 3.4 1.6 0.4 setosa## 28 5.2 3.5 1.5 0.2 setosa## 29 5.2 3.4 1.4 0.2 setosa## 30 4.7 3.2 1.6 0.2 setosa## 31 4.8 3.1 1.6 0.2 setosa## 32 5.4 3.4 1.5 0.4 setosa## 33 5.2 4.1 1.5 0.1 setosa## 34 5.5 4.2 1.4 0.2 setosa## 35 4.9 3.1 1.5 0.2 setosa## 36 5.0 3.2 1.2 0.2 setosa## 37 5.5 3.5 1.3 0.2 setosa## 38 4.9 3.6 1.4 0.1 setosa## 39 4.4 3.0 1.3 0.2 setosa## 40 5.1 3.4 1.5 0.2 setosa## 41 5.0 3.5 1.3 0.3 setosa## 42 4.5 2.3 1.3 0.3 setosa## 43 4.4 3.2 1.3 0.2 setosa## 44 5.0 3.5 1.6 0.6 setosa## 45 5.1 3.8 1.9 0.4 setosa## 46 4.8 3.0 1.4 0.3 setosa## 47 5.1 3.8 1.6 0.2 setosa## 48 4.6 3.2 1.4 0.2 setosa## 49 5.3 3.7 1.5 0.2 setosa## 50 5.0 3.3 1.4 0.2 setosa

``` c) Sorting the dataset

To sort the dataset by a specific column, use the order() function. We can combine that with selecting only certain columns:

```

Sort the dataset by Petal.Width in ascending orderiris[order(iris$Petal.Width), c("Petal.Width", "Species")]## Petal.Width Species## 10 0.1 setosa## 13 0.1 setosa## 14 0.1 setosa## 33 0.1 setosa## 38 0.1 setosa## 1 0.2 setosa## 2 0.2 setosa## 3 0.2 setosa## 4 0.2 setosa## 5 0.2 setosa## 8 0.2 setosa## 9 0.2 setosa## 11 0.2 setosa## 12 0.2 setosa## 15 0.2 setosa## 21 0.2 setosa## 23 0.2 setosa## 25 0.2 setosa## 26 0.2 setosa## 28 0.2 setosa## 29 0.2 setosa## 30 0.2 setosa## 31 0.2 setosa## 34 0.2 setosa## 35 0.2 setosa## 36 0.2 setosa## 37 0.2 setosa## 39 0.2 setosa## 40 0.2 setosa## 43 0.2 setosa## 47 0.2 setosa## 48 0.2 setosa## 49 0.2 setosa## 50 0.2 setosa## 7 0.3 setosa## 18 0.3 setosa## 19 0.3 setosa## 20 0.3 setosa## 41 0.3 setosa## 42 0.3 setosa## 46 0.3 setosa## 6 0.4 setosa## 16 0.4 setosa## 17 0.4 setosa## 22 0.4 setosa## 27 0.4 setosa## 32 0.4 setosa## 45 0.4 setosa## 24 0.5 setosa## 44 0.6 setosa## 58 1.0 versicolor## 61 1.0 versicolor## 63 1.0 versicolor## 68 1.0 versicolor## 80 1.0 versicolor## 82 1.0 versicolor## 94 1.0 versicolor## 70 1.1 versicolor## 81 1.1 versicolor## 99 1.1 versicolor## 74 1.2 versicolor## 83 1.2 versicolor## 91 1.2 versicolor## 93 1.2 versicolor## 96 1.2 versicolor## 54 1.3 versicolor## 56 1.3 versicolor## 59 1.3 versicolor## 65 1.3 versicolor## 72 1.3 versicolor## 75 1.3 versicolor## 88 1.3 versicolor## 89 1.3 versicolor## 90 1.3 versicolor## 95 1.3 versicolor## 97 1.3 versicolor## 98 1.3 versicolor## 100 1.3 versicolor## 51 1.4 versicolor## 60 1.4 versicolor## 64 1.4 versicolor## 66 1.4 versicolor## 76 1.4 versicolor## 77 1.4 versicolor## 92 1.4 versicolor## 135 1.4 virginica## 52 1.5 versicolor## 53 1.5 versicolor## 55 1.5 versicolor## 62 1.5 versicolor## 67 1.5 versicolor## 69 1.5 versicolor## 73 1.5 versicolor## 79 1.5 versicolor## 85 1.5 versicolor## 87 1.5 versicolor## 120 1.5 virginica## 134 1.5 virginica## 57 1.6 versicolor## 84 1.6 versicolor## 86 1.6 versicolor## 130 1.6 virginica## 78 1.7 versicolor## 107 1.7 virginica## 71 1.8 versicolor## 104 1.8 virginica## 108 1.8 virginica## 109 1.8 virginica## 117 1.8 virginica## 124 1.8 virginica## 126 1.8 virginica## 127 1.8 virginica## 128 1.8 virginica## 138 1.8 virginica## 139 1.8 virginica## 150 1.8 virginica## 102 1.9 virginica## 112 1.9 virginica## 131 1.9 virginica## 143 1.9 virginica## 147 1.9 virginica## 111 2.0 virginica## 114 2.0 virginica## 122 2.0 virginica## 123 2.0 virginica## 132 2.0 virginica## 148 2.0 virginica## 103 2.1 virginica## 106 2.1 virginica## 113 2.1 virginica## 125 2.1 virginica## 129 2.1 virginica## 140 2.1 virginica## 105 2.2 virginica## 118 2.2 virginica## 133 2.2 virginica## 116 2.3 virginica## 119 2.3 virginica## 121 2.3 virginica## 136 2.3 virginica## 142 2.3 virginica## 144 2.3 virginica## 146 2.3 virginica## 149 2.3 virginica## 115 2.4 virginica## 137 2.4 virginica## 141 2.4 virginica## 101 2.5 virginica## 110 2.5 virginica## 145 2.5 virginica

``` As can be seen, petal width is pretty good at separating the different species. We will corroborate this with some basic plotting.

Basic PlottingR has built-in plotting functions for creating simple visualizations. Here are a few examples:

a) Histogram

A histogram is a graphical representation of the distribution of a dataset, where data is divided into a set of intervals or bins. The data is represented as vertical bars, with the height of each bar corresponding to the number of data points that fall within a particular bin. Histograms are used to visualize the underlying frequency distribution of a continuous variable, allowing one to identify patterns such as skewness, central tendency, and dispersion.

```

Create a histogram of Petal.Widthhist(iris$Petal.Width, main = "Histogram of Petal Width", xlab = "Petal Width", ylab = "Frequency", col = "lightblue", border = "black")

``` b) Box plot

A box plot, also known as a box-and-whisker plot, is a graphical representation of the distribution and spread of a dataset. It displays five key statistics: the minimum, first quartile (Q1), median (Q2), third quartile (Q3), and maximum. The “box” represents the interquartile range (IQR), which contains the middle 50% of the data, while the “whiskers” extend from the box to the minimum and maximum values. Outliers, if present, are typically represented as individual points outside the whiskers.

```

Create a box plot of Petal.Width by Speciesboxplot(Petal.Width ~ Species, data = iris, main = "Box plot of Petal Width by Species", xlab = "Species", ylab = "Petal Width")

``` Here, we can very clearly see that petal width indeed separates the different species well. In the following plot, another variable, petal length, is included and the three species are colour-coded.

c) Scatter plot

A scatter plot is a graphical representation of the relationship between two variables, where each data point is represented by a dot on a two-dimensional plane. The horizontal axis (x-axis) represents one variable, while the vertical axis (y-axis) represents the other variable. By analyzing the distribution and pattern of the dots, one can determine the correlation, trends, or outliers between the two variables.

```

Create a scatter plot of Petal.Length vs Petal.Widthplot(iris$Petal.Length, iris$Petal.Width, main = "Scatter plot of Petal Length vs Petal Width", xlab = "Petal Length", ylab = "Petal Width", col = iris$Species)

``` And, by the way, you can very easily use those plots in other applications (like WinWord or PowerPoint) by clicking on “Export” in Rstudio and then on “Save as Image…” or “Copy to Clipboard…”.

ConclusionIn just 10 minutes, you’ve learned the basics of R using the iris dataset. We covered loading data, data manipulation, and basic plotting. As you continue to explore R, you will discover its vast capabilities and potential for analyzing and visualizing complex data.

To continue on your coding adventure, the following posts are good starting points:

Learning R: The Ultimate Introduction (incl. Machine Learning!)
One Rule (OneR) Machine Learning Classification in under One Minute

If you want to dive deeper into data science, I created the following learning path: Learning Path for “Data Science with R” – Part I

Take care, and happy data sleuthing!

To leave a comment for the author, please follow the link and comment on their blog: R-Bloggers – Learning Machines.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R for Everyone: Analytical Superpowers in under 10 Minutes!

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIf you’re new to data science or statistics, you may have heard about probability distributions. Probability distributions are mathematical functions that help us understand the probability of a random variable taking on a certain value. For example, if we’re rolling a fair six-sided die, we know that each number has an equal chance of being rolled (1/6 or about 17% chance). We can represent this using a probability distribution, specifically a discrete uniform distribution.

However, not all probability distributions are as simple as a uniform distribution. Many real-world phenomena, such as the heights of people, the number of cars passing through a toll booth in a day, or the amount of rainfall in a particular area, are continuous and can’t be represented using a discrete distribution. Instead, we use continuous probability distributions, which describe the probability of a continuous variable taking on a range of values.

There are many different types of continuous probability distributions, each with their own properties and use cases. For example, the normal distribution, also known as the bell curve, is commonly used to model many natural phenomena, such as human heights and weights. The beta distribution is used to model proportions or percentages, such as the proportion of voters who support a particular candidate. The gamma distribution is used to model the time between events in a Poisson process, such as the time between customers arriving at a store.

The sample TidyDensity App is a tool that helps us explore and visualize these different types of probability distributions. It’s a web application built using the R programming language and the Shiny framework, which allows us to create interactive web applications with R.

Let’s break down the different components of the TidyDensity App.

User InterfaceThe user interface, or UI for short, is what the user sees and interacts with when they use the app. It’s built using HTML, CSS, and JavaScript, and it’s the first thing the user sees when they open the app.

The TidyDensity App has a simple UI that allows the user to select from four different probability distributions: normal, Bernoulli, beta, and gamma. Each of these distributions has its own properties and use cases, and the user can select which one they want to explore using a dropdown menu.

In addition, the user can specify the number of simulations they want to run, which determines how many times the probability distribution is sampled to generate data. They can also specify the sample size, which determines how many data points are generated in each simulation.

Finally, the user can select which type of plot they want to see, such as a density plot, a quantile plot, a probability plot, or a QQ plot. Each of these plots shows a different aspect of the data generated from the probability distribution, and the user can choose which one to explore.

Here is the code:

The server is written in R, and it uses several R packages to generate the data and create the plots. For example, the TidyDensity package is used to generate data from the selected probability distribution, and the ggplot2 package is used to create the plots.

The server is also responsible for handling user inputs, such as which probability distribution to use, how many simulations to run, and which plot type to show. It then generates the appropriate data and plot based on these inputs and sends them back to the UI for display.

The first thing we do is create a reactive variable data that will store the output of the match.fun() function, which is called with the arguments .num_sims and .n obtained from the user interface. We use the reactive variable because it will update automatically whenever the inputs are changed.

The output$density_plot object is created with renderPlot(), which takes the reactive variable data() and passes it to tidy_autoplot() with the plot type selected by the user in the input$plot_type object. The resulting plot is then printed to the user interface.

The output$data_table object is created with DT::renderDataTable(), which takes the reactive variable data() and returns a table to the user interface using the DT::datatable() function.

Finally, the output$download_data object is created using downloadHandler(), which creates a download button for the user to download a .csv file of the data. The filename argument specifies the name of the file, and the content argument writes the data to a .csv file.

Here is the code:

```

Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input or tidy_empirical if user entered data if (input$data_input_type == "Enter Data") { data <- input$data if (is.null(data) || data == "") { return(NULL) } data <- as.numeric(strsplit(data, ",")[[1]]) tidy_empirical(data) } else { match.fun(input$functions)(.num_sims = input$num_sims, .n = input$n) } }) # Create density plot output$density_plot <- renderPlot({ # Call autoplot on reactive data if (!is.null(data())) { p <- data() |> tidy_autoplot(.plot_type = input$plot_type) print(p) } }) # Create data table output$data_table <- DT::renderDataTable({ # Return reactive data as a data table if (!is.null(data())) { DT::datatable(data()) } }) # Download data handler output$download_data <- downloadHandler( filename = function() { if (input$data_input_type == "Enter Data") { paste0("tidy_empirical.csv") } else { paste0(input$functions, ".csv") } }, content = function(file) { write.csv(data(), file, row.names = FALSE) } ) }

``` Data TableThe data table is a table that shows the data generated from the probability distribution. It’s displayed on.

Overall, this app is designed to allow users to generate various types of probability density plots and accompanying data tables based on user input. By allowing users to select different functions, sample sizes, and plot types, this app provides a flexible and customizable tool for exploring and visualizing probability distributions.

Full Shiny AppHere is the full script:

library(shiny)library(TidyDensity)library(tidyverse)library(DT)# Define UIui <- fluidPage( titlePanel("TidyDensity App"), sidebarLayout( sidebarPanel( radioButtons(inputId = "data\_input\_type", label = "Data Input Type:", choices = c("Select Function", "Enter Data"), selected = "Select Function"), conditionalPanel( condition = "input.data\_input\_type == 'Enter Data'", textInput(inputId = "data", label = "Enter data as a comma-separated list of numeric values") ), conditionalPanel( condition = "input.data\_input\_type == 'Select Function'", selectInput(inputId = "functions", label = "Select Function", choices = c( "tidy\_normal", "tidy\_bernoulli", "tidy\_beta", "tidy\_gamma" ) ) ), numericInput(inputId = "num\_sims", label = "Number of simulations:", value = 1, min = 1, max = 15), numericInput(inputId = "n", label = "Sample size:", value = 50, min = 30, max = 200), selectInput(inputId = "plot\_type", label = "Select plot type", choices = c( "density", "quantile", "probability", "qq", "mcmc" ) ), downloadButton(outputId = "download\_data", label = "Download Data") ), mainPanel( plotOutput("density\_plot"), DT::dataTableOutput("data\_table") ) ))# Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input or tidy\_empirical if user entered data if (input$data\_input\_type == "Enter Data") { data <- input$data if (is.null(data) || data == "") { return(NULL) } data <- as.numeric(strsplit(data, ",")[[1]]) tidy\_empirical(data) } else { match.fun(input$functions)(.num\_sims = input$num\_sims, .n = input$n) } }) # Create density plot output$density\_plot <- renderPlot({ # Call autoplot on reactive data if (!is.null(data())) { p <- data() |> tidy\_autoplot(.plot\_type = input$plot\_type) print(p) } }) # Create data table output$data\_table <- DT::renderDataTable({ # Return reactive data as a data table if (!is.null(data())) { DT::datatable(data()) } }) # Download data handler output$download\_data <- downloadHandler( filename = function() { if (input$data\_input\_type == "Enter Data") { paste0("tidy\_empirical.csv") } else { paste0(input$functions, ".csv") } }, content = function(file) { write.csv(data(), file, row.names = FALSE) } ) }# Run the appshinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Distributions with {shiny} and {TidyDensity} Part 4

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Logic will get you from A to B. Imagination will you take everywhere.(Einstein)

R can already take you everywhere. With it we can learn about theminutest particles and the largest galaxies. So, to celebrate therelease of R 4.3 (“Already Tomorrow”, on April 21st, 2023), let’sreverse Einstein’s quote and take you from A to B with logic.

Two modes of comparisonIn R, almost all of your data will be stored as a vector. Even if yourvector holds a single value it is still considered to be a vector by R.This is unlike many other languages, and getting comfortable “thinkingfor the whole vector” can gain you efficiencies from several viewpoints.Your code will be more concise and it may even run quicker, whencompared with an iterative approach to the same problem.

1:10 # A vector of integers## [1] 1 2 3 4 5 6 7 8 9 10is.vector(1:10)## [1] TRUEsum(1:10) # A vectorised computation## [1] 55integer(0) # An empty vector of integers## integer(0)1L # A single integer, stored as a vector## [1] 1 But the conciseness that R’s vectorised operations provide may trip youup unexpectedly. A typical case is when you think you are working witha scalar (a length-1 vector) but you are actually working with anempty or multivalued vector.

The logical values in R (TRUE, FALSE) are a little bit special. Avector of logical values might be used to represent some quality in adataset, for example, to select those rows of a dataset that are to bekept in dplyr::filter().

library("tidyverse")head(diamonds)## # A tibble: 6 × 10## carat cut color clarity depth table price x y z## <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl>## 1 0.23 Ideal E SI2 61.5 55 326 3.95 3.98 2.43## 2 0.21 Premium E SI1 59.8 61 326 3.89 3.84 2.31## 3 0.23 Good E VS1 56.9 65 327 4.05 4.07 2.31## 4 0.29 Premium I VS2 62.4 58 334 4.2 4.23 2.63## 5 0.31 Good J SI2 63.3 58 335 4.34 4.35 2.75## 6 0.24 Very Good J VVS2 62.8 57 336 3.94 3.96 2.48head(diamonds$cut == "Ideal") # A logical vector## [1] TRUE FALSE FALSE FALSE FALSE FALSEfilter(diamonds, cut == "Ideal") # Subsetting a data-frame using a logical vector## # A tibble: 21,551 × 10## carat cut color clarity depth table price x y z## <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl>## 1 0.23 Ideal E SI2 61.5 55 326 3.95 3.98 2.43## 2 0.23 Ideal J VS1 62.8 56 340 3.93 3.9 2.46## 3 0.31 Ideal J SI2 62.2 54 344 4.35 4.37 2.71## 4 0.3 Ideal I SI2 62 54 348 4.31 4.34 2.68## 5 0.33 Ideal I SI2 61.8 55 403 4.49 4.51 2.78## 6 0.33 Ideal I SI2 61.2 56 403 4.49 4.5 2.75## 7 0.33 Ideal J SI1 61.1 56 403 4.49 4.55 2.76## 8 0.23 Ideal G VS1 61.9 54 404 3.93 3.95 2.44## 9 0.32 Ideal I SI1 60.9 55 404 4.45 4.48 2.72## 10 0.3 Ideal I SI2 61 59 405 4.3 4.33 2.63## # ℹ 21,541 more rowshead(diamonds$carat > 0.3)## [1] FALSE FALSE FALSE FALSE TRUE FALSEfilter(diamonds, carat > 0.3)## # A tibble: 49,737 × 10## carat cut color clarity depth table price x y z## <dbl> <ord> <ord> <ord> <dbl> <dbl> <int> <dbl> <dbl> <dbl>## 1 0.31 Good J SI2 63.3 58 335 4.34 4.35 2.75## 2 0.31 Ideal J SI2 62.2 54 344 4.35 4.37 2.71## 3 0.32 Premium E I1 60.9 58 345 4.38 4.42 2.68## 4 0.31 Very Good J SI1 59.4 62 353 4.39 4.43 2.62## 5 0.31 Very Good J SI1 58.1 62 353 4.44 4.47 2.59## 6 0.31 Good H SI1 64 54 402 4.29 4.31 2.75## 7 0.33 Ideal I SI2 61.8 55 403 4.49 4.51 2.78## 8 0.33 Ideal I SI2 61.2 56 403 4.49 4.5 2.75## 9 0.33 Ideal J SI1 61.1 56 403 4.49 4.55 2.76## 10 0.32 Good H SI2 63.1 56 403 4.34 4.37 2.75## # ℹ 49,727 more rows But there are places where you use logical values, where it would makeno sense (and could potentially be dangerous) to use a multivaluedlogical vector. We use if (...) {} and while (...) {} statements forflow control in R. The conditional expression in these statements (the... in if (...) {}) should always evaluate to a logical scalar:either TRUE or FALSE.

When R 4.2.0 was released, stricter guarantees were placed on the lengthof these conditional expressions. We mentioned this in an earlier blogpost. So inaddition to getting an error when the conditional is empty, we now getan error when the conditional is too long:

```

Comparison with an empty logical vector:if (logical(0)) { print("I didn't expect to get here")}## Error in if (logical(0)) {: argument is of length zero# Comparison with an over-sized logical vector:numbers <- c(1, 3, 5, 6)print(numbers %% 2 == 0) # Determine if even## [1] FALSE FALSE FALSE TRUEif (numbers %% 2 == 0) { print("Should we ever be allowed to get here?")}## Error in if (numbers%%2 == 0) {: the condition has length > 1

`` Previously, R would use the first entry in a non-scalar conditionalvector to decide whether to enter theiforwhile` block.

Data comes in all shapes and sizes. It can often be difficult to know where to start. Whatever your problem, Jumping Rivers can help.

Strictly comparingSo, we have two main ways of using a logical vector, one of which nowrequires that the vector is a scalar.

Another place where it is really important to know the length of yourvectors is when combining logical values together.

R has a number of ways to combine logical values together that build onthe AND and OR operations in Boolean algebra:

all and any for combining the values in a single vector (areall of the values TRUE; are any of the values TRUE)
&, && (representing “AND”), |, and || (for “OR”) forcombining two different vectors

is\_april = TRUEis\_r\_released = TRUEis\_already\_tomorrow = FALSE# Logical AND within a single vectorall(c(is\_april, is\_r\_released, is\_already\_tomorrow))## [1] FALSE# Logical OR within a single vectorany(c(is\_april, is\_r\_released, is\_already\_tomorrow))## [1] TRUE# Logical AND between vectorsis\_april & is\_r\_released## [1] TRUEis\_april && is\_already\_tomorrow## [1] FALSE# Logical OR between vectorsis\_april | is\_r\_released## [1] TRUEis\_april || is\_already\_tomorrow## [1] TRUE For scalars, there’s no difference between the single-characteroperators (&, |) and the two-character operators (&&, ||). Sowhy have a pair of operators for each concept?

&& and || are intended for use solely with scalars, theyreturn a single logical value.
& and | work with multivalued vectors, they return a vectorwhose length matches their input arguments.

Since they always return a scalar logical, you should use && and|| in your if/while conditional expressions (when needed). If an &or | is used, you may end up with a non-scalar vector insideif (...) {} and R will throw an error.

To illustrate the difference between the scalar operators and vectorisedoperators, here’s an example:

x = c(TRUE, TRUE, FALSE, FALSE)y = c(TRUE, FALSE, TRUE, FALSE) The vectorised operators apply AND/OR on matched pairs of elements:

x & y # c(x[1] && y[1], x[2] && y[2], ...)## [1] TRUE FALSE FALSE FALSEx | y # c(x[1] || y[1], x[2] || y[2], ...)## [1] TRUE TRUE TRUE FALSE In R 4.2.0, a warning is thrown when a non-scalar input is passed to thescalar-operators. But, a scalar logical is returned (here, the result ofx[1] && y[1]). In earlier versions of R, no warning was printed.

```

R 4.2x && y[1] TRUEWarning messages:1: In x && y : 'length(x) = 4 > 1' in coercion to 'logical(1)'2: In x && y : 'length(x) = 4 > 1' in coercion to 'logical(1)'

`` This could lead to hidden bugs. For example, if you used this code in anif` conditional, a warning would be printed when a non-scalar vectorwas used but the code would continue happily:

```

R 4.2if (x && y) { print("The world can't end today...")}[1] "The world can't end today..."Warning messages:1: In x && y : 'length(x) = 4 > 1' in coercion to 'logical(1)'2: In x && y : 'length(x) = 4 > 1' in coercion to 'logical(1)'

``` In R 4.3.0, this warning has been elevated to an error and no value isreturned:

```

R 4.3x && yError in x && y : 'length = 4' in coercion to 'logical(1)'

``` This more strict version of the scalar comparison operators will helpcatch those bugs where you didn’t realise a logical variable couldcontain more than one entry.

To check whether the strict comparison operators will affect yourexisting code, before upgrading to R 4.3.0, you can set an environmentvariable before running it:

```

In R:Sys.setenv("_R_CHECK_LENGTH_1_LOGIC2" = TRUE)

```

Whether you want to start from scratch, or improve your skills, Jumping Rivers has a training course for you.

A more logical flowWhere else do we work with scalars in R? Many functions expect certainarguments to be scalars. For example, the seq() function complainswith non-scalar arguments:

seq(from = 1:3, to = 4)## Error in seq.default(from = 1:3, to = 4): 'from' must be of length 1seq(from = 1, to = 4:5)## Error in seq.default(from = 1, to = 4:5): 'to' must be of length 1 There are several other places where R will throw an error if we providea value that is of the wrong size:

a\_data\_frame[[column\_index]] # column\_index must be a scalara\_matrix[rows, cols] = value # value must match the size of the replaced element(s) There are other places where R will throw a warning, and try togracefully handle values that are of an unexpected size:

```

R's recycling rules are used to match the size of the vector inputc(1, 3, 5) * c(2, 3) # c(1 * 2, 3 * 3, 5 * 2)## Warning in c(1, 3, 5) * c(2, 3): longer object length is not a multiple of## shorter object length## [1] 2 9 10# The smaller vector was recycled to match the size of the larger# c(1, 3, 5) * c(2, 3, 2)

`` An interesting case is the:operator, which likeseq()`, can be usedto create sequences of numbers.

3:5## [1] 3 4 5 If we provide a non-scalar on either side of the operator, R will warnus:

```

R 4.2(1:2) : 5[1] 1 2 3 4 5Warning message:In (1:2):5 : numerical expression has 2 elements: only the first used# R 4.21 : (4:6)[1] 1 2 3 4Warning message:In 1:(4:6) : numerical expression has 3 elements: only the first used

``` Now, because the output should be a single sequence, R has to pick aspecific value for the start- and the end-point of that sequence fromthe arguments provided. It uses the first entry in each argument. So,

(1:2) : 5 is equivalent to 1:5; and
1 : (4:6) is equivalent to 1:4.

If your code is providing non-scalar arguments to :, there may be abug in your code or the packages that it depends upon. R 4.3.0 hasintroduced a more strict setting, which will catch the use of non-scalarvalues when constructing sequences with the : operator.

Much like with the stricter logic comparisons described above, the Rdevelopers have introduced this as an optional setting. After settingthe environment variable _R_CHECK_LENGTH_COLON_ to a true value, Rwill throw an error whenever an oversized argument is passed into a:b.

```

R 4.3# Without the check enabled:(1:2) : 5[1] 1 2 3 4 5Warning message:In (1:2):5 : numerical expression has 2 elements: only the first used# With the strict check enabled:Sys.setenv("_R_CHECK_LENGTH_COLON_" = TRUE)(1:2) : 5Error in (1:2):5 : numerical expression has length > 1

``` And finally: Extracting from a pipeHave you started using the native pipe yet? In our blog post tocelebrate the release of R 4.2.0, we showed this example:

mtcars |> lm(mpg ~ disp, data = \_)## ## Call:## lm(formula = mpg ~ disp, data = mtcars)## ## Coefficients:## (Intercept) disp ## 29.59985 -0.04122 Here the pipe |> passes the value on it’s left-hand side into thefunction on the right. By default that value will be used as the firstargument to the right-hand function. But when an underscore is present,the piped-in value will replace that underscore. So the above isequivalent to:

lm(mpg ~ disp, data = mtcars)## ## Call:## lm(formula = mpg ~ disp, data = mtcars)## ## Coefficients:## (Intercept) disp ## 29.59985 -0.04122 What if you want to extract values that are output by a pipeline? Forexample, if you want the coef entry from the linear model above. Oneway would be to store the results in a variable and extract the coeffrom that:

model = mtcars |> lm(mpg ~ disp, data = \_)model$coef## (Intercept) disp ## 29.59985476 -0.04121512 Or you could wrap the pipeline in parentheses:

( mtcars |> lm(mpg ~ disp, data = \_))$coef## (Intercept) disp ## 29.59985476 -0.04121512 R 4.3.0 provides a much neater solution, where the underscore _ can beused to refer to the final value from a pipeline. This can make yourcode much neater:

mtcars |> lm(mpg ~ disp, data = \_) |> \_$coef(Intercept) disp29.59985476 -0.04121512 Trying the latest version out for yourselfTo take away the pain of installing the latest development version of R,you can use docker. To use the devel version of R, you can use thefollowing commands:

docker pull rstudio/r-base:devel-jammydocker run --rm -it rstudio/r-base:devel-jammy See the r-docker project for moredetails.

See alsoDo you have nostalgia for previous versions of R? If so, check out ourprevious blog posts:

R 4.0.0
R4.1.0
R 4.2.0

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: What’s new in R 4.3.0?

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Vincent Guyader talked to the R Consortium about his background with the R language, describing his acquisition of experience and skills in developing solutions to scientific and industrial problems. He also spoke about the founding of his company, ThinkR, and his role as organizer of R Addicts Paris.

Vincent Guyader is the President and CTO of ThinkR, a company founded in 2015 dedicated to the development of tools and solutions based on the R language to solve different scientific and industrial problems.

Vincent is a specialist in applied statistics and also has skills in system administration, which allows him to intervene on the IS (Information systems) side for the implementation of Compute Server, the deployment of Shiny applications, and the installation of Posit workbench and Posit Connect. Vincent is the organizer of the R Addicts Paris group, the first R group of its kind in France and currently the largest with over 1,800 members. The group offers training in the use of R software for business professionals, which allows him to develop his Data Science skills. In his spare time, he enjoys cycling in Paris and practicing karate.

Please share your background and your involvement in the RUGS group or in the R Community.

It all started during my scholarship at Agrocampus Ouest, located in Rennes, France, a distinguished institute known for its exceptional curriculum focused on statistics and agronomy, where I gained proficiency in the R programming language.

Subsequently, I ventured into the consulting field immediately after my graduation without any previous experience in running a business. At first, I brought my expertise in statistical analysis, but thanks to my rapid progress in R, I soon became an outstanding R specialist. As a result, my clients did not only approach me for my statistical competence but also for my exceptional command of the R programming language.

Credits: Diane B., 2016

The main project I have been involved in is the Golem package, which allows the creation of dynamic and interactive Shiny applications. Along with my colleagues, we have written a book entitled Engineering Production-Grade Shiny Apps, which has become one of our most significant contributions to the community. This project is ongoing and has been continuously growing since its inception four years ago.

In our line of work, we collaborate with various industries, such as pharmacology, energy and banking, among others. Our main goal is to demonstrate the usefulness of R as a language that is not limited to trivial applications but can also be used to address complex business challenges. We offer this orientation to all sectors, regardless of their nature.

What is your level of experience with the R language?

Although R was not the programming language I initially started with, I have consistently used it since 2008 and have developed a good level of proficiency in the language. I am able to complete any required task with relative ease, and my experience goes beyond personal use.

I am confident in my ability to share my knowledge with others and help them improve their R programming skills. I believe that with my experience and knowledge of R, I can offer some value to any project or initiative. While I have developed some mastery of R, I still see myself as a learner and continuously seek to improve.

What resources or techniques do you use?

At ThinkR, we leverage a wide range of technologies to achieve our goals. While we do not use Spark extensively for big data, we do use other powerful tools such as Docker, the Posit products, GitHub, GitLab, the Tidyverse, and the data table. With our team of 13 people at ThinkR, we are very fluent in all aspects of our language. It is worth noting that not all team members rely on RStudio as their IDE, as some prefer VS Code, either locally or on a remote desktop. As a result, we have a wealth of options to carry out our work effectively.

Do you have an ongoing project? Please share any details or CTA for who should get involved!

We have an imminent appointment in Avignon, a picturesque city in the south of France. The event, known as Rencontres R, will be held in a few weeks and promises to be a great event, with an expected attendance of 250 people. The two- to three-day event is dedicated exclusively to all things R in France and marks an important moment for the French R community. Excitement is growing as the date approaches and we expect it to be a success!

How do I Join?

Apply HereFind your local R user group The post Using R to Develop Solutions for Industrial Problems appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Using R to Develop Solutions for Industrial Problems

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn the previous post we allowed users to choose a distribution and a plot type. Now, we want to allow users to download a .csv file of the data that is generated.

In the UI, we added a downloadButton with outputId = "download_data" and label = "Download Data". In the server, we added a downloadHandler that takes a filename and content function. The filename function returns the name of the file to be downloaded (in this case, we used the selected function name as the file name with “.csv” extension). The content function writes the reactive data to a CSV file using the write.csv function. The downloadHandler returns the file to be downloaded when the button is clicked.

See here:

UI SectionHere is the update to the UI Section

```

Define UIui <- fluidPage( titlePanel("TidyDensity App"), sidebarLayout( sidebarPanel( selectInput(inputId = "functions", label = "Select Function", choices = c( "tidy_normal", "tidy_bernoulli", "tidy_beta", "tidy_gamma" ) ), numericInput(inputId = "num_sims", label = "Number of simulations:", value = 1, min = 1, max = 15), numericInput(inputId = "n", label = "Sample size:", value = 50, min = 30, max = 200), selectInput(inputId = "plot_type", label = "Select plot type", choices = c( "density", "quantile", "probability", "qq", "mcmc" ) ), # Download the data downloadButton(outputId = "download_data", label = "Download Data") ), mainPanel( plotOutput("density_plot"), DT::dataTableOutput("data_table") ) ))

``` Server SectionHere is the update to the Server section.

```

Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input match.fun(input$functions)(.num_sims = input$num_sims, .n = input$n) }) # Create density plot output$density_plot <- renderPlot({ # Call autoplot on reactive data p <- data() |> tidy_autoplot(.plot_type = input$plot_type) print(p) }) # Create data table output$data_table <- DT::renderDataTable({ # Return reactive data as a data table DT::datatable(data()) }) # Download data handler output$download_data <- downloadHandler( filename = function() { paste0(input$functions, ".csv") }, content = function(file) { write.csv(data(), file, row.names = FALSE) } ) }# Run the appshinyApp(ui = ui, server = server)

``` ConclusionWith these changes, the user can now export the data to a .csv file by clicking the “Export Data” button and selecting where to save the file.

I hope this update to the TidyDensity app will make it more useful for your data analysis needs. If you have any questions or feedback, please feel free to let me know, and as usual…Steal this Code!! Modify for yourself and see what you come up with.

Here is the entire script:

library(shiny)library(TidyDensity)library(tidyverse)library(DT)# Define UIui <- fluidPage( titlePanel("TidyDensity App"), sidebarLayout( sidebarPanel( selectInput(inputId = "functions", label = "Select Function", choices = c( "tidy\_normal", "tidy\_bernoulli", "tidy\_beta", "tidy\_gamma" ) ), numericInput(inputId = "num\_sims", label = "Number of simulations:", value = 1, min = 1, max = 15), numericInput(inputId = "n", label = "Sample size:", value = 50, min = 30, max = 200), selectInput(inputId = "plot\_type", label = "Select plot type", choices = c( "density", "quantile", "probability", "qq", "mcmc" ) ), downloadButton(outputId = "download\_data", label = "Download Data") ), mainPanel( plotOutput("density\_plot"), DT::dataTableOutput("data\_table") ) ))# Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input match.fun(input$functions)(.num\_sims = input$num\_sims, .n = input$n) }) # Create density plot output$density\_plot <- renderPlot({ # Call autoplot on reactive data p <- data() |> tidy\_autoplot(.plot\_type = input$plot\_type) print(p) }) # Create data table output$data\_table <- DT::renderDataTable({ # Return reactive data as a data table DT::datatable(data()) }) # Download data handler output$download\_data <- downloadHandler( filename = function() { paste0(input$functions, ".csv") }, content = function(file) { write.csv(data(), file, row.names = FALSE) } ) }# Run the appshinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Distributions with {shiny} and {TidyDensity} Part 3

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You had a great idea how to help your coworkers optimize their work. You’ve built an app and deployed it in a production server. Now you wait for some signs of success: your happy colleagues praising the app or recommending it to others. But the messages of praise don’t come. You check the usage statistics and it turns out that no one is using it. Or worse yet, users open it once, only to never return. Adoption of your Shiny app has failed, but it’s not over yet.

Don’t throw in your app-building-hat just yet. Check the list below of what might be going wrong and how to deal with it. You worked hard on your app, make sure it’s a success.

Here’s a few reasons why no one is using your Shiny app:

Users Don’t Know Your Shiny App ExistsSounds trivial, but it’s actually more common than you think. Even if there was communication around the launch of the app, your users may have simply missed it.

If your users are local (i.e. on your team) this is easy to verify. It’s time to survey coworkers that you expect to be the users with one simple question: Do you know about [Greatest Shiny App Ever]?

If your users are the general public, you may need to create a marketing campaign around the project. To get the word out, try creating social posts, record video tutorials, or present at places like ShinyConf and posit::conf.

Users Know of the App, But Don’t Know Its PurposePotential users need to be informed about the app and be aware of its potential benefits. This is particularly important in large companies where thousands of different applications and dashboards exist.

It’s easy to overlook new solutions that aren’t immediately clear. To effectively communicate the app’s value (via emails, intranet, repositories, showcases, or training sessions), it’s essential to clearly articulate the problems it solves and who can benefit from using it.

Users Don’t Need Your Shiny AppOr, in other words, the users do not have a problem that the app is trying to solve.

This is a serious issue as it often indicates that the initial assumptions on which the app was developed were incorrect. But does this mean that the only solution is to abandon the app, archive it, and chalk it up as a loss? Not necessarily.

In many cases, the actual needs and problems may not be far off. By spending time with the users, empathizing with them, and carefully listening, we can discover how to modify the app to solve the real problem.

Of course, a much better approach is to identify and understand the real problems and needs before starting the development process. Then, by testing the solution with users, gathering feedback, and making adjustments along the way, we can ensure that the app addresses their needs effectively.

We have a wonderful video presentation on how to do just that by Maria Grycuk, Project Manager, on how to conduct effective user tests.

Users Can’t Use the App When They Need ItThis is another significant problem that indicates the omission of an essential aspect of the problem.

For instance, the app may require internet access, but users may need the app in areas with no internet connection. Maybe the app is a web application that is not optimized for mobile screen sizes, but users need it while on the move, navigating around the city.

In such cases, it is crucial to identify the reason behind the issue and fix it if possible.

Users Simply Can’t Use the Shiny AppThis is a common issue – users recognize the benefits of using the app and are eager to try it, but when they open it, they don’t even know where to begin. The initial thought might be to train them, or worse, to create a manual. While training sessions and manuals are important, they may not be the best solution – certaintly not when Shiny apps can be designed to be intuitive and so easy to use.

So, what can we do instead? We need to take a closer look at the user experience within the app. Is it intuitive? How does the user know which action to take first? Is the flow of the app logical? Sometimes even minor changes, such as adding proper labels or highlighting essential actions, can make a significant difference.

If you’re not sure where to begin when designing proper UX for Shiny apps, start with my 7 Steps to Designing Dashboards People Will Love.

Users Can’t Fully Solve Their Problem With the AppThis often occurs when the app is built for one person, such as a manager, who has an intuition for what the team needs, but will not be the actual user of the app. Alternatively, the app may be intended for use by multiple teams, but only members of one team are involved in its development. As a result, the app may be overly tailored to the needs of one team and not suitable for others.

The steps to address this issue are similar to those when “users don’t need the app.” Once again, during development, it is essential to include as diverse a user group as possible to regularly check that the app is moving in the right direction.

Users Can Solve Their Problem In an Easier WayIf the app is intended to save users time but, due to a complicated interface or long loading time, it takes longer to solve the problem than it did before the app existed, users will naturally stick to their previous habits. This may also be observed if users open the app to download data, just to switch to Excel or Google Sheets instead of using the app itself.

If this happens, it means that during the discovery phase, the current solution to the problem was not recognized or taken into account. Therefore, the solution is to check which features of the current solution are missing. However, be cautious – if the interface is already complex, adding new features may not be the best option. Instead, refining and simplifying the existing features or even removing some of them may make the most important features easier to use.

Of course, you can check out Jakub Sobolewski’s Definitive Guide on Speeding Up R Shiny if performance is your issue.

Users Don’t Trust the AppThis is a very serious situation that usually is a result of some serious bugs that users were exposed to. For example if the app was showing incorrect data and some decisions were made based on it – next time, the users will likely search for a different source of information, even if that will require additional effort.

In such a case, the change shouldn’t be done step by step. It’s better to take down the app, fix all the major issues. Implement a solid quality assurance process, including automated data validations, software testing and User Acceptance Testing. Then relaunch the app, showing the users how the app development process was improved.

Users Are Afraid of ChangesLast, but not least, people don’t like change. If users are accustomed to the current solution, they may be hesitant to try a new tool. This often requires learning new skills and additional time, but most importantly, the effort to overcome the fear of “what if I do something wrong?”

In contrast to “users don’t know how to use the app,” this is a perfect reason for training sessions and detailed manuals. It’s also a good idea to encourage users to test the app and let you know if there is any confusion, explaining that it is probably due to the app’s lack of user-friendliness, not the users’ lack of knowledge. If users are accustomed to a certain UI/UX, you can try implementing those in Shiny (like Fluent UI or Blueprint UI).

There is also a technical aspect to low adoption, such as users not having access to the app and not being able to log in or a monitoring tool having a bug and failing to report all the sessions. These issues are usually easy to solve, but they should not be neglected. You can also customize your login pages to make the users more comfortable, or emphasize authentication with Posit ConnectCodeProject.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Shiny User Adoption Fails: 9 Reasons Why Nobody Uses Your App

View Details

[This article was first published on R on FOSS Trading, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.An updated version of xts is now on CRAN. This release patches a few issues with the features added in version 0.13.0 and addresses a few maintenance issues that popped up recently.

Patches for features added in 0.13.0* Format each column individually before printing. The top/bottom rows could have a different number of decimal places and there are often multiple varying spaces between columns. For example:

close volume ma2022-01-03 09:31:00 476.470 803961.000 NA2022-01-03 09:32:00 476.700 179476.000 NA2022-01-03 09:33:00 476.540 197919.000 NA ...2023-03-16 14:52:00 394.6000 46728.0000 392.86362023-03-16 14:53:00 394.6500 64648.0000 392.87552023-03-16 14:54:00 394.6500 69900.0000 392.8873 There are 4 spaces between the index and the ‘close’ column, 2 between ‘close’ and ‘volume’, and 4 between ‘volume’ and ‘ma’. There should be a consistent number of spaces between the columns. Most other classes of objects print with 1 space between the columns. The top rows have 3 decimals and the bottom rows have 4. These should also be the same. (#321) * Make column names based on number of columns. The original code was a lot more complicated because it tried to account for truncating the number of printed columns. That functionality was removed because of how complicated it was. So now we can simply create printed column names from the number of columns. (#395) * Only convert printed index values to character. Converting the entire index to character is time-consuming for xts objects with many observations. It can take more than a second to print an xts object with 1 million observations. * Reduce instances when dplyr::lag() warning is shown. The warning was shown whenever it detected dplyr is installed, even if the user wasn’t actively using dplyr. That caused an excessive amount of noise when other packages attached xts (e.g. quantmod). Thanks to Duncan Murdoch for the report and suggested fix! (#393)

Bug Fixes* Return ‘POSIXct’ if object has no ‘tclass’. An empty string is not a valid ‘tclass’, so it can cause an error. * Fix xts() for zero-row data.frames. The xts() constructor would create an object with a list for coredata when x is a data.frame with no rows. xts objects can’t have lists as coredata, so it should convert x to a matrix and throw an error if x is a list. (#394) * Fix as.data.frame() when converting a data.frame with column names to xts when there’s only one non-time-based column. Previously the xts object would not have the data.frame column name. (#391) * Treat NA the same as NULL for ‘start’ or ‘end’ in window.xts(). NULL represents an undefined index value and NA represents an unknown or missing index value. xts does not allow NA as index values, so subsetting an xts or zoo object by NA returns a zero-length object. Therefore a NA (unknown) index value is essentially the same as an undefined index value. (#383, #345) * Warn and remove NA when periodicity() called on date-time vector with any NA values. Previously it threw the uninformative error below. (#289)

Error in try.xts(x, error = "'x' needs to be timeBased or xtsible") : 'x' needs to be timeBased or xtsible * Account for timezones when making names for the list split.xts() creates. This was specifically a problem if the xts object’s index was yearmon because as.yearmon.POSIXct() always sets tz = "GMT" when calling as.POSIXlt(), regardless of the xts’ index ‘tzone’ attribute. That can cause the as.yearmon() results to be different days for GMT and the index’s timezone. Use format.POSIXct() for “months” because it checks for a ‘tzone’ attribute before converting to POSIXlt and calling format.POSIXlt(). The conversion to POSIXlt is important because it checks and uses the ‘tzone’ attribute before considering the ‘tz’ argument. So it effectively ignores the tz = "GMT" setting in as.yearmon(). This is also the reason for calling as.POSIXlt() before calling as.yearqtr(). (#392) * Ignore attribute order in all.equal(). Attribute order shouldn’t matter. That can be checked with identical().

Chores* Add notes on plot.xts() nomenclature and structure. Also add ASCII art to illustrate definitions and layout. (#103) * Register missing S3 methods and update signatures. With R-devel (83995-ish), R CMD check notes these S3 methods are not registered. It also notes that the signatures for as.POSIXct.tis() and str.replot_xts() do not match the respective generics. R CMD check also thinks time.frequency() is a S3 method because time() is a generic. The function isn’t exported, so renaming won’t break any external code. Thanks to Kurt Hornik for the report. Issues with functionality for ‘tis’ were also identified. I removed ‘tis’ support entirely because the implementation was not even a bare minimum, and it’s not clear it even worked correctly. (#398) * Add instructions to update old objects. Old xts objects do not have ‘tclass’ and ‘tzone’ attributes on the index. Add a function to update the object attributes and add a note to the warning to show how to use it. Also, only call tzone() and tclass() once in check.TZ(). Calling these functions multiple times throws multiple warnings for xts objects created before the ‘tclass’ and ‘tzone’ were attached to the index instead of the xts object. (#306)

If you love using my open-source work (e.g. quantmod, TTR, xts, IBrokers, microbenchmark, etc.), you can give back by sponsoring me on GitHub. I truly appreciate anything you’re willing and able to give!

I look forward to your questions and feedback! If you have a question, please ask on Stack Overflow and use the [r] and [xts] tags. Or you can send an email to the R-SIG-Finance mailing list (you must subscribe to post). Open an issue on GitHub if you find a bug or want to request a feature. Please read the contributing guide first! It will help save time for both of us.

To leave a comment for the author, please follow the link and comment on their blog: R on FOSS Trading.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: xts_0.13.1 on CRAN

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Here!

The R Consortium gives grants to help R User Groups (RUGS) around the world organize, share information, and support each other. We are currently accepting applications!

The R Consortium RUGS Program has grown from being a relatively modest R user group support program to being the primary vehicle for the R Consortium to award Social Infrastructure Grants. Social Infrastructure includes meetings, events, conferences and any other activity intended to strengthen the social, organizational, and identity structures of the R Community.

In 2023, there will be three categories of RUGS Program grants:

User Group Grants
1. The intent of user group grants is to facilitate person-to-person exchange of R knowledge in small group settings on a global scale.
2. Cash grants typically vary between $200 and $1,000 and depend on group size and special needs.
3. All groups who are accepted into the RUGS program who are not already participating in the R-Ladies Meetup.com Professional Account program are enrolled in the RUGS program which covers Meetup.com dues.
Conference Grants
1. To qualify for a RUGS program conference grant, an event must be focused on the R language and offer at least one full day of technical talks and presentations and aim to attract participants with diverse backgrounds. These grants are for conferences organized by non-profit or volunteer groups.
Special Project Grants
1. With our Special Projects Grants categories, we hope to stimulate the imagination of local R community builders.

Full details here: https://www.r-consortium.org/all-projects/r-user-group-support-program. Please help support R language. Submit your proposals!

R User GroupsThere are currently 98 R User Groups (RUGS) organizing and learning and spreading the use of R globally. These groups welcome individuals from any background, from beginner-level users to experts.

Check out our recent blog interviews by organizers of the R User Groups across all industries:

Adoption of R by Actuaries Community in Melbourne

Use of R in Agricultural Chemical Industry in Chile

Uniting R Professionals Across Disciplines for Data-Driven Insights in Maine

R in Finance and Accounting Sector in Korea

Get Involved!The 2023 RUGS Program is currently taking applications and will close at midnight PST on September 30, 2023.

These grants do not include support for software development or technical projects. Grants to support the R ecosystem’s technical infrastructure are awarded and administered through the ISC Grant Program which issues a call for proposals two times each year.

The post The 2023 RUGS Program is awarding grants for 2023! appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The 2023 RUGS Program is awarding grants for 2023!

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionShiny is an R package that allows you to create interactive web applications from R code. In this blog post, we’ll explore the different components of a Shiny application and show how they work together to create an interactive data visualization app. This is a part 2 with a small enhancement.

The AppOur Shiny app will generate density plots for different statistical distributions based on user input. The user will be able to select a distribution, set the number of simulations, and choose the plot type from a dropdown menu. The app will also display a table of data generated by the selected distribution.

Here’s a preview of what the app will look like:

The UIThe user interface (UI) is the visual part of the app that the user interacts with. In our app, the UI is defined using the fluidPage() function from the shiny package. It consists of a title panel, a sidebar layout, and a main panel.

selectInput() allows the user to choose a statistical distribution to generate data from.
numericInput() allows the user to set the number of simulations.
Another numericInput() allows the user to set the sample size.
selectInput() allows the user to choose the type of plot to display.

The main panel contains the output elements for the app, in this case a plot and a table.

The ServerThe server is the backend of the app that handles the logic and generates the output based on user input. In our app, the server is defined using the server() function from the shiny package.

```

Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input match.fun(input$functions)(.num_sims = input$num_sims, .n = input$n) }) # Create density plot output$density_plot <- renderPlot({ # Call autoplot on reactive data p <- data() |> tidy_autoplot(.plot_type = input$plot_type) print(p) }) # Create data table output$data_table <- DT::renderDataTable({ # Return reactive data as a data table DT::datatable(data()) }) }

`` Theserver()` function takes two arguments, input and output. These arguments allow the server to interact with the user interface.

First, we create a reactive data object data, which takes in the user’s input for the function, number of simulations, and sample size, and passes it to the appropriate function using match.fun().

Next, we create the density_plot output. We use the renderPlot() function to create a reactive plot of the data using the tidy_autoplot() function from the {TidyDensity} package. The tidy_autoplot() function allows the user to choose from several plot types, including density, quantile, probability, qq, and mcmc. We then print the plot using the print() function.

Finally, we create the data_table output using the DT::renderDataTable() function. This output displays the reactive data as a table using the DT::datatable() function.

The Shiny AppFinally, we run the Shiny app using the shinyApp() function, which takes the ui and server functions as arguments:

shinyApp(ui = ui, server = server) This launches the app and displays the user interface. The user can interact with the app by selecting a function, specifying the number of simulations and sample size, and viewing the resulting density plot and data table. The app provides a simple and interactive way to explore the TidyDensity package and its functionality.

ConclusionHere is the entire app! Steal this code and modify it for yourself, see what you can do!

library(shiny)library(TidyDensity)library(tidyverse)library(DT)# Define UIui <- fluidPage( titlePanel("TidyDensity App"), sidebarLayout( sidebarPanel( selectInput(inputId = "functions", label = "Select Function", choices = c( "tidy\_normal", "tidy\_bernoulli", "tidy\_beta", "tidy\_gamma" ) ), numericInput(inputId = "num\_sims", label = "Number of simulations:", value = 1, min = 1, max = 15), numericInput(inputId = "n", label = "Sample size:", value = 50, min = 30, max = 200), selectInput(inputId = "plot\_type", label = "Select plot type", choices = c( "density", "quantile", "probability", "qq", "mcmc" ) ) ), mainPanel( plotOutput("density\_plot"), DT::dataTableOutput("data\_table") ) ))# Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input match.fun(input$functions)(.num\_sims = input$num\_sims, .n = input$n) }) # Create density plot output$density\_plot <- renderPlot({ # Call autoplot on reactive data p <- data() |> tidy\_autoplot(.plot\_type = input$plot\_type) print(p) }) # Create data table output$data\_table <- DT::renderDataTable({ # Return reactive data as a data table DT::datatable(data()) }) }# Run the appshinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Distributions with {shiny} and {TidyDensity} Part 2

View Details

[This article was first published on R with White Dwarf, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.You can find all the posts on this series under the tag maps-app (including the Spanish versions).

You can also find the current state of the project under my GitHub repo mapic.

Scope of this postWe are creating maps of data showing changes over a span of time for different countries and pointing at all kinds of cities. That basically means that we need to map any region of the world with R. Today there are all kinds of packages and techniques to do that. I will share the strategy I used with ggplot2 and maps packages, using support of Open Street Map to obtain the coordinates of cities and finally making it interactive with shiny.

This series of posts share my path towards the creation of the Shiny app. It is a live project and I decided to share my path and experiences along the creation process. The posts are not only about the Shiny app, but the package I created behind it, including topics of functions crafting, creation of the maps, classes of objects, etc., as well as any interesting issue that appear on the way. It is my way to contribute to the R community and at the same time keeping the project documented for myself.

This post is about Creating functions for ggplot.

I hope you all enjoy it. Feel free to leave any kind of comment and/or question at the end.

Background and preliminariesIn the first post we created a function to create the basic map. Since then I have modified the function slightly, but the concept is the same. You can see below the most up to date version and compare it with the previous version if you wish.

my\_country\_prev <- function(country, map\_colors, x\_limits = NULL, y\_limits = NULL, show\_coords = FALSE) { require(maps) require(ggplot2) ## Verifying the arguments passed to the function if (length(country) != 1) stop("Function supports only one country per map") stopifnot(is.logical(show\_coords)) stopifnot("Name of the country should be character" = is.character(country)) if (!country %in% map\_data('world')$region) { stop(paste("Country name not recognized", "To see a list of recognized countries run", "<unique(maps::map\_data('world')$region)>", sep = "\n")) } ## If coords limits missing, print worldwide map with coordinates system to allow ## User observe coords for reference if (missing(x\_limits) || missing(y\_limits)) { warning("X and/or Y limits not provided.\nPrinting worldwide map.") map\_country\_theme <- theme(panel.background = element\_rect(fill = map\_colors$oceans)) } else if (show\_coords) { map\_country\_theme <- theme(panel.background = element\_rect(fill = map\_colors$oceans)) } else { if (length(x\_limits) != 2 || length(y\_limits) != 2 || !all(grepl("^-?[0-9.]+$", c(x\_limits, y\_limits)))) { stop("Limits for X and Y coords should be provided as vectors with two numeric values") } else { ## All the received inputs are correct. ## Let's define our custom theme for the final map map\_country\_theme <- theme\_bw() + theme(panel.background = element\_rect(fill = map\_colors$oceans), legend.position = "none", panel.grid.major = element\_blank(), panel.grid.minor = element\_blank(), axis.line = element\_line(colour = "black"), axis.title.x = element\_blank(), axis.text.x = element\_blank(), axis.ticks.x = element\_blank(), axis.title.y = element\_blank(), axis.text.y = element\_blank(), axis.ticks.y = element\_blank()) } } ## make a df with only the country to overlap map\_data\_country <- map\_data('world')[map\_data('world')$region == country, ] ## The map (maps + ggplot2 ) mapic <- ggplot() + ## First layer: worldwide map geom\_polygon(data = map\_data("world"), aes(x = long, y = lat, group = group), color = map\_colors$border\_countries, # border countries fill = map\_colors$empty\_countries) + # empty countries ## Second layer: Country map geom\_polygon(data = map\_data\_country, aes(x = long, y = lat, group = group), color = map\_colors$border\_countries, # border target country fill = map\_colors$target\_country) + # target country coord\_map() + coord\_fixed(1.3, xlim = x\_limits, ylim = y\_limits) + map\_country\_theme return(mapic)} One critical difference is the argument map_colors that is nor explained or well defined. This is a list object containing the values for the colors to be used for all the elements of the maps. There are different ways to define and use this. The idea is to make it an S3 object and explain it on its own, but it is a topic that I am still exploring and I haven’t decided yet the details of it. For now, let’s use it simply as a list containing our chosen colors for the map.

map\_colors <- list(dots\_orgs = "#493252", target\_country = "#8caeb4", empty\_countries = "#f3f3f3", border\_countries = "#9c9c9c", oceans = "#4e91d2", text\_cities = "#a0a0a0", text\_legend = "#493252", background\_legend = "#ffffff", text\_copyright = "#f3f3f3") The function easily prints the map of any country, using the naming from the package maps. Now we want to add the data to it.

Now we need to define some simple data frame simulating a collection of organizations in Mexico.

mx\_data <- data.frame( ID = c(1:10), Name = sprintf("org%d", seq(1:10)), Registration\_year = c(2001:2010), Country = "MX", Region = c("Mexico", "Baja California Norte", "Mexico", "Jalisco", "Queretaro", "Baja California Norte", "Mexico", "Morelos", "Mexico", "Estado de Mexico"), City = c("Ciudad de Mexico", "Tijuana", "Ciudad de Mexico", "Guadalajara", "Queretaro", "Tijuana", "Ciudad de Mexico", "Cuernavaca", "Ciudad de Mexico", "Texcoco")) We can see it as a company that along 10 years managed to open one new franchise per year, and we want to map where each is located and where it has grown the most. For that, we need the coordinates of the cities where each franchise is located. We can easily obtain that using code from the previous posts: either directly from part II or the improved version of the function.

webscrap\_to\_db(db\_name = "test-mex.sqlite", dat = mx\_data, city = "City", country = "Country", db\_backup\_after = 5) And also using previously defined functions, we can combine the data with the just obtained coordinates system.

(datmx <- combine\_csv\_sql(db\_file = "test-mex.sqlite", csv\_file = mx\_data))> ID Name Registration\_year City Country Region State County> 1 1 org1 2001 Ciudad de Mexico MX > 2 2 org2 2002 Tijuana MX > 3 3 org3 2003 Ciudad de Mexico MX > 4 4 org4 2004 Guadalajara MX > 5 5 org5 2005 Queretaro MX > 6 6 org6 2006 Tijuana MX > 7 7 org7 2007 Ciudad de Mexico MX > 8 8 org8 2008 Cuernavaca MX > 9 9 org9 2009 Ciudad de Mexico MX > 10 10 org10 2010 Texcoco MX > osm\_name lon> 1 Ciudad de México, México -99.13318> 2 Tijuana, Municipio de Tijuana, Baja California, 22320, México -117.01953> 3 Ciudad de México, México -99.13318> 4 Guadalajara, Jalisco, México -103.33840> 5 Santiago de Querétaro, Municipio de Querétaro, Querétaro, México -100.39706> 6 Tijuana, Municipio de Tijuana, Baja California, 22320, México -117.01953> 7 Ciudad de México, México -99.13318> 8 Cuernavaca, Morelos, 62000, México -99.23423> 9 Ciudad de México, México -99.13318> 10 Texcoco, Carbó, Sonora, México -111.05867> lat> 1 19.43263> 2 32.53174> 3 19.43263> 4 20.67204> 5 20.59547> 6 32.53174> 7 19.43263> 8 18.92183> 9 19.43263> 10 29.63900 Now datmx should have the coordinates, together with the rest of the data about our franchises. We should also have our SQLite file and, of course, our source data. It means that we are ready to add the data to the map.

Programming with ggplot2If you ever wondered how to create functions with ggplot2, there are a few ways, but here is the basic point that we need to understand, if we want to have them working in the same style as ggplot works:

Once you have the base plot with the function ggplot() you can add geoms and stats to it by simply using +, or you can create new functions by returning a list of geoms and stats.

The first point is as simple as the following lines:

my\_country\_prev("Mexico", map\_colors, x\_limits = c(-118, -86), y\_limits = c(14, 34)) + ggtitle("A map of Mexico") Or we could do the same by creating a function and returning the title inside a list.

my\_title <- function(text) { return(list(ggtitle(text)))}my\_country\_prev("Mexico", map\_colors, x\_limits = c(-118, -86), y\_limits = c(14, 34)) + my\_title("The same map of Mexico") With that in mind, we can do all the calculations we want and start adding the data in form of geoms and stats to the base map.

A map with growing dots per cityWe started with something simple, adding the amount of organizations per city, as growing dots.

make\_dots <- function(.df, year, map\_colors, column\_names = list( lat = "lat", lon = "lon", cities = "city", start\_year = "year", end\_year = NULL), dot\_size = 1) { require(dplyr) require(tidyr) require(stringr) ## Some error handling mandatory\_cols <- c("lat", "lon", "cities", "start\_year") if(!all(mandatory\_cols %in% names(column\_names))) { stop("Column names missing!") } else { if (!"end\_year" %in% names(column\_names)) { .df$final\_year <- NA\_real\_ column\_names[["end\_year"]] <- "final\_year" } } ## Dots base size base\_size <- 5 dot\_sizes <- c(0.5 * (base\_size * dot\_size), 1 * (base\_size * dot\_size), 2 * (base\_size * dot\_size), 3 * (base\_size * dot\_size), 4 * (base\_size * dot\_size), 5 * (base\_size * dot\_size), 7 * (base\_size * dot\_size), 8 * (base\_size * dot\_size), 9 * (base\_size * dot\_size)) ## Data manipulation to be used in the map filt <- .df %>% mutate(year\_final = replace\_na(!!sym(column\_names$end\_year), year + 1), city\_name = str\_to\_sentence(!!sym(column\_names$cities))) %>% filter(year\_final > year & !!sym(column\_names$start\_year) <= year) %>% group\_by(city\_name) %>% summarise(x = median(!!sym(column\_names$lon), na.rm = T), y = median(!!sym(column\_names$lat), na.rm = T), n = n()) %>% mutate(dot\_size = case\_when(n == 1 ~ dot\_sizes[1], n >= 2 & n <= 5 ~ dot\_sizes[2], n >= 6 & n <= 10 ~ dot\_sizes[3], n >= 11 & n <= 30 ~ dot\_sizes[4], n >= 31 & n <= 50 ~ dot\_sizes[5], n >= 51 & n <= 100 ~ dot\_sizes[6], n >= 101 & n <= 200 ~ dot\_sizes[7], n >= 201 & n <= 300 ~ dot\_sizes[8], n >= 301 ~ dot\_sizes[9], TRUE ~ NA)) ## -------------------------- MAIN MAP ---------------------------------- map\_points <- list( geom\_point(data = filt, aes(x, y, size = dot\_size), color = map\_colors$dots\_orgs, alpha = 7/10, shape = 19) , scale\_size\_identity('', breaks = dot\_sizes, labels = c('1', '2-5', '6-10', '11-30', '31-50', '51-100', '101-200', '201-300', '>300'), guide = guide\_legend(label.position = 'bottom', label.vjust = 0, nrow = 1)), geom\_point(data = filter(filt, n == 1), aes(x, y), color = map\_colors$dots\_orgs, shape = 19, size = 2.5) , theme(legend.position = 'bottom') ) return(map\_points)} As you can see, the function also requires our object map_colors, which we created before. Another way of passing values from a list is by defining these values directly within the function arguments, as we did here for column_names. We could pass the arguments directly when calling the function, or define them earlier to be used. Let’s use the second approach.

col\_names = list(lat = "lat", lon = "lon", cities = "City", start\_year = "Registration\_year") If you look at the data frame that we created containing the data, this are simply the names of the columns as we specified them.

Now, about the function itself, it starts, as expected, by calling the libraries and then doing a bit of error handling to ensure that the fields that are strictly required are actually present in the data frame. There I am also adding the options for the end_year which is used in case some franchise closed and we want to map it only for the period of time it was present.

Then we define the “Dots base size”. Here we experimented with so many sizes, both for the dots and for the final map, and this are the ones that look the best. Still, I’m allowing this value to be changed as the parameter dot_size in the function definition, however I wouldn’t recommend changing it. You can also play with the internal values and see it for yourself. Since the idea here is to create functions for the “standards” of the maps, allowing minimal changes, we are not so strict as per how big the dots should be, yet we have certain degree of control.

Then we do a little bit of data manipulation before being able to use the data. This includes the standardization of the names of Cities (up to some degree), filtering the data that does not match with the selected year, using only the median value of the latitude and longitude data, and defining the sizes of the dots according to the amount of franchises. The last one is a tricky one that I haven’t decided yet what amount of freedom should still be out there. Maybe there should be a separated function to define all that. Our maps were created to handle data containing from few hundreds of rows, to a couple of thousands, thus, the values presented here. But if you want to show just a few organizations (as is the case of this example), the map looks quite deserted; on the other hand, if you need to map values of thousands per city, the maps look overloaded. For the present post I’m keeping it as is, with a note for consideration. We also added one extra geom_point to overwrite the alpha value for the case of only 1, and make it solid. This also works well on the visuals.

In any case, the function above shows how we can manipulate the data inside a function, and return only what we need to add it to an existent ggplot. We can now add the dots as we would normally do in ggplot style.

my\_country\_prev("Mexico", map\_colors, x\_limits = c(-118, -86), y\_limits = c(14, 34), show\_coords = T) + make\_dots(datmx, year = 2022, map\_colors, column\_names = col\_names) + scale\_x\_continuous(n.breaks = 20) + ggtitle("A map of Mexico") Adding labels for the mapMoving forward, we want to add some labels to the maps to know what we are seeing. Here I created one function to show which year is being mapped, and a second one to show the totals. Although we can achieve that easily in different ways, I managed to make it complicated, keeping in mind that we want to map any region in the world.

my\_print\_years <- function(year, map\_colors, x\_limits, y\_limits, year\_label = "Year") { ## POSITION FOR THE LABELS ## Starting points x\_units <- abs(x\_limits[1] - x\_limits[2])/10 y\_units <- abs(y\_limits[1] - y\_limits[2])/10 start\_x <- min(x\_limits) start\_y <- min(y\_limits) ## Frame rectangle.start.x <- start\_x rectangle.wide <- rectangle.start.x + x\_units rectangle.start.y <- start\_y rectangle.high <- rectangle.start.y + y\_units ## Text num.size <- 4 text.size <- 3 num.position.x <- start\_x + (x\_units * 0.5) text.position.x <- start\_x + (x\_units * 0.5) num.position.y <- start\_y + (y\_units * 0.25) text.position.y <- start\_y + (y\_units * 0.65) ## Adding the ggplot geoms pyears <- list( geom\_rect( aes(xmin = rectangle.start.x, xmax = rectangle.wide, ymin = rectangle.start.y, ymax = rectangle.high), color = map\_colors$text\_legend, fill = map\_colors$text\_legend, alpha = 9/10), geom\_text( aes(x = num.position.x, y = num.position.y, label = year), size = num.size, fontface = 'bold', color = map\_colors$background\_legend), geom\_text( aes(x = text.position.x, y = text.position.y, label = year\_label), size = text.size, fontface = 'bold', alpha = 9/10, color = map\_colors$background\_legend) ) return(pyears)} Consider this some kind of snippet to add the labels wherever you want, and in any size you want. Our function is basically doing some basic simple calculations to place the labelling inside the map area, on the bottom-left corner. The first part with the comment ## POSITION FOR THE LABELS shows the basic calculations to do that, based on the coords, which should be the same as the coords specified in the map function. The calculations and the position are very stiff in size and location, but they will work the same regardless of the region mapped. On the other hand, it provides the basis for the function. Feel free to play with them to add custom options such as selecting the sizes or the corner where we want to display them.

The rest of the code is intuitive, geom_text to add the info we want to show, one for the word “Year” and another one for the numeric value. We add the corresponding values to the aes, the sizes, some alpha for transparency and our colors defined in map_colors.

my\_print\_totals <- function(totals, map\_colors, x\_limits, y\_limits, totals\_label = "Totals") { ## POSITION FOR THE LABELS ## Starting points x\_units <- abs(x\_limits[1] - x\_limits[2])/10 y\_units <- abs(y\_limits[1] - y\_limits[2])/10 start\_x <- min(x\_limits) + x\_units start\_y <- min(y\_limits) ## Frame rectangle.start.x <- start\_x rectangle.wide <- rectangle.start.x + x\_units rectangle.start.y <- start\_y rectangle.high <- rectangle.start.y + y\_units ## Text num.size <- 4 text.size <- 3 num.position.x <- start\_x + (x\_units*0.5) text.position.x <- start\_x + (x\_units*0.5) num.position.y <- start\_y + (y\_units*0.25) text.position.y <- start\_y + (y\_units*0.65) ptotals <- list( geom\_rect(aes(xmin = rectangle.start.x, xmax = rectangle.wide, ymin = rectangle.start.y, ymax = rectangle.high), color = '#283151', fill = map\_colors$background\_legend, alpha = 9/10), geom\_text( aes(x = num.position.x, y = num.position.y, label = totals), size = num.size, fontface = 'bold', alpha = 9/10, color = map\_colors$text\_legend), geom\_text( aes(x = text.position.x, y = text.position.y, label = totals\_label), size = text.size, fontface = 'bold', alpha = 9/10, color = map\_colors$text\_legend) ) return(ptotals)} We can follow exactly the same approach for the totals, to place them right next to the year. For now we have to specify the value of the total that we want to be shown, but this actually should be calculated by the function. Actually, if you think about it, we are passing a great deal of information that should be coming from the previous functions, and we should keep here arguments that control the visuals of the labels only. That is not an easy topic and I decided to cover it in a separated post. Another reason why I don’t want to show it yet is because I haven’t decided yet which approach I want to use.

So, for now we have to pass each argument to each function and make sure that we are passing the same argument, but that is easy to achieve in R by directing the values to an object before passing it to the functions. Let’s see it in action.

The dataset has already been stored in datmx and the colors in map_colors. Now we need to define a few more.

x\_coords <- c(-118, -86)y\_coords <- c(14, 34)yr <- 2020totals <- 10my\_country\_prev("Mexico", map\_colors, x\_coords, y\_coords, show\_coords = T) + make\_dots(rbind(datmx, datmx), map\_colors, year = yr, col\_names) + my\_print\_years(yr, map\_colors, x\_coords, y\_coords, "Año") + my\_print\_totals(totals, map\_colors, x\_coords, y\_coords, "Totales") And there it is. Since my map is for Mexico, I am adding labels in Spanish. Feel free to test it in your own language and with more data. We can also have a look at how the labels fit to other countries, for example, smaller and bigger compared to Mexico.

x\_coords <- c(4, 18)y\_coords <- c(47, 56)my\_country\_prev("Germany", map\_colors, x\_coords, y\_coords, show\_coords = T) + my\_print\_years(yr, map\_colors, x\_coords, y\_coords) + my\_print\_totals(totals, map\_colors, x\_coords, y\_coords) + ggtitle("A map of Germany") Germany is also looking good. And since we have the possibility of passing the values for year and totals, we don’t actually need to have any data to test it, although the info shown is incorrect.

x\_coords <- c(28, 185)y\_coords <- c(10, 100)my\_country\_prev("Russia", map\_colors, x\_coords, y\_coords, show\_coords = T) + my\_print\_years(yr, map\_colors, x\_coords, y\_coords) + my\_print\_totals(totals, map\_colors, x\_coords, y\_coords) Russia is a very particular case because the country is quite long but not so wide. If we specify coordinates too narrow for latitude, the map does not look good and the labels start getting deformed. It would be the same case with Chile, if we make it narrow in longitude but it is naturally long in latitude. Since the main aim of the functions is to take care of the aesthetics and visualization, we have to ensure that this should not happen, somehow. Having certain degree of the labels is one way to do that. We will see a few more in the future.

ConclusionsIf your aim is only to make maps like the above for any given country, our first 3 posts should have you covered. We could also import our functions to a shiny app and work with that. However, there are many improvements that we can still do.

Now that we know the basis for functional programming with ggplot2, we can extend the power of our functions in the way how they share arguments by exploring the OOP (Object Oriented Programming) in R, the ggproto system to extend ggplot, and the use of environments.

Since we want the maps to be dynamic in time, we could also work on a couple of functions to cover that. It could easily be achieved in a for loop, which is perfectly fine. We could also use the apply family of functions or the map family of functions from the purrr package (not to be confused with geographic maps or the package maps). The last options could be a bit of complication because of the excess of arguments in our functions. That only shows that it is worth it to still improve them.

The next posts will be focused on this topic, so stay connected if you are interested on how I tackle these challenges.

To leave a comment for the author, please follow the link and comment on their blog: R with White Dwarf.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Map any region in the world with R – Part III: Programming with ggplot2

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Interactive map and R package enable the research community with new insights about the ocean

Global Fishing Watch is an international nonprofit organization dedicated to advancing ocean governance through increased transparency of human activity at sea. To help meet that goal, they created an interactive online map that uses satellite technology and machine learning to track and visualize the activity of fishing vessels around the world.

The map allows anyone to view the movements of fishing vessels in near-real time and explore historical data on fishing activity. Global Fishing Watch uses automatic identification system data and other streams of information to build knowledge about the ocean and the activities taking place across it. These key insights help promote transparency and accountability in global fisheries, enabling authorities to identify illegal, unreported, and unregulated fishing. The map also supports efforts in understanding the environmental impact of fishing activity and helps inform sustainable fisheries management. Since its launch in 2016, Global Fishing Watch has bolstered the work being done by researchers, policymakers, and conservationists to promote sustainable fishing practices and protect marine ecosystems.

In July 2022, Global Fishing Watch released gfwr—an R package designed to enable the research community to access data from their API portal.

The gfwr package allows R users to request data from Global Fishing Watch’s application programming interfaces and receive data in a tidy format suitable for incorporation into new or existing R workflows. Users have the ability to pull data for analysis without any prior API experience. Learn more about how Global Fishing Watch empowers others to use their data.

Check out the Global Fishing Watch map. A screenshot of the Global Fishing Watch map shows an interactive heat map of fishing efforts. Variations in color represent different data sources. The brighter grid cells indicate areas with more activity.

The gfwr authors and maintainers are listed below.

Tyler Clavelle (he/him)

Rocío Joo (she/her)

Nate Miller (he/him)

The post Global Fishing Watch Helps Protect Critical Marine Ecosystems Through Open Data and Innovative Technology appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Global Fishing Watch Helps Protect Critical Marine Ecosystems Through Open Data and Innovative Technology

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionShiny is an R package that allows you to build interactive web applications using R code. TidyDensity is an R package that provides a tidyverse-style interface for working with probability density functions. In this tutorial, we’ll use these two packages to build a Shiny app that allows users to interact with TidyDensity functions.

ExampleRequired PackagesBefore we dive into the code, let’s go over the packages that we’ll be using in this app:

Shiny: As mentioned earlier, Shiny is an R package for building interactive web applications. It provides a variety of input controls and output elements that allow you to create user interfaces for your R code.
TidyDensity: TidyDensity is an R package that provides a tidyverse-style interface for working with probability density functions. It provides a set of functions for generating density functions, as well as a tidy_autoplot() function for creating visualizations.
tidyverse: Tidyverse is a collection of R packages designed for data science. It includes many popular packages such as ggplot2, dplyr, and tidyr. We’ll be using some functions from the tidyverse packages in our Shiny app.
DT: DT is an R package for creating interactive tables in RMarkdown documents, Shiny apps, and RStudio. We’ll be using the DT::datatable() function to create a table of output data in our app.

Load them up!

library(shiny)library(DT)library(tidyverse)library(TidyDensity) The UI ObjectThe UI object is the first argument of the shinyApp() function, and it defines the layout and appearance of the app. In our TidyDensity Shiny app, we’ll use a sidebar layout with two input controls and two output elements:

Select Function Input: A selectInput() control that allows users to select one of four TidyDensity functions: tidy_normal(), tidy_bernoulli(), tidy_beta(), and tidy_gamma().
Number of Simulations Input: A numericInput() control that allows users to specify the number of simulations to use in the TidyDensity function.
Sample Size Input: A numericInput() control that allows users to specify the sample size to use in the TidyDensity function.
Density Plot Output: A plotOutput() element that displays the density plot generated by tidy_autoplot().
Data Table Output: A dataTableOutput() element that displays the output data from the TidyDensity function.

Here’s the code for the UI object:

```

Define UIui <- fluidPage( titlePanel("TidyDensity App"), sidebarLayout( sidebarPanel( selectInput(inputId = "function", label = "Select Function", choices = c( "tidy_normal", "tidy_bernoulli", "tidy_beta", "tidy_gamma" ) ), numericInput(inputId = "num_sims", label = "Number of simulations:", value = 1, min = 1, max = 15), numericInput(inputId = "n", label = "Sample size:", value = 50, min = 30, max = 200) ), mainPanel( plotOutput("density_plot"), dataTableOutput("data_table") ) ))

``` The Server ObjectThe server object is the second argument of the shinyApp() function, and it defines the behavior and output of the app. In our TidyDensity Shiny app, the server object consists of two reactive expressions that generate the output elements based on the user inputs:

Data Reactive Expression: A reactive expression that generates the output data for the selected TidyDensity function based on the user inputs. We use match.fun() to convert the selected function name into an R function, and we pass the num_sims and n arguments from the input controls.
Density Plot Reactive Expression: A reactive expression that generates the density plot using tidy_autoplot() and the output data from the data reactive expression.
Data Table Output: We use DT::renderDataTable() to generate the data table output element based on the output data from the data reactive expression.

Here’s the code for the server object:

```

Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input match.fun(input$function)(.num_sims = input$num_sims, .n = input$n) }) # Create density plot output$density_plot <- renderPlot({ # Call autoplot on reactive data p <- data() %>% tidy_autoplot() print(p) }) # Create data table output$data_table <- DT::renderDataTable({ # Return reactive data as a data table DT::datatable(data()) }) }

``` ConclusionIn this tutorial, we used Shiny and TidyDensity to build an interactive web application that allows users to generate and visualize probability density functions. We learned how to use the selectInput() and numericInput() controls to allow users to specify the function and input parameters, and we used the plotOutput() and dataTableOutput() elements to display the output data and visualizations. We also used the reactive() function to create reactive expressions that automatically update the output elements based on the user inputs.

Most importantly, steal the code below and see what you can do with it!

library(shiny)library(TidyDensity)library(tidyverse)library(DT)# Define UIui <- fluidPage( titlePanel("TidyDensity App"), sidebarLayout( sidebarPanel( selectInput(inputId = "functions", label = "Select Function", choices = c( "tidy\_normal", "tidy\_bernoulli", "tidy\_beta", "tidy\_gamma" ) ), numericInput(inputId = "num\_sims", label = "Number of simulations:", value = 1, min = 1, max = 15), numericInput(inputId = "n", label = "Sample size:", value = 50, min = 30, max = 200) ), mainPanel( plotOutput("density\_plot"), DT::dataTableOutput("data\_table") ) ))# Define serverserver <- function(input, output) { # Create reactive data data <- reactive({ # Call selected function with user input match.fun(input$functions)(.num\_sims = input$num\_sims, .n = input$n) }) # Create density plot output$density\_plot <- renderPlot({ # Call autoplot on reactive data p <- data() |> tidy\_autoplot() print(p) }) # Create data table output$data\_table <- DT::renderDataTable({ # Return reactive data as a data table DT::datatable(data()) }) }# Run the appshinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Exploring Distributions with {shiny} and {TidyDensity}

View Details

[This article was first published on ouR data generation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In recent discussions with a number of collaborators at the NIA IMPACT Collaboratory about setting the sample size for a proposed cluster randomized trial, the question of variable cluster sizes has come up a number of times. Given a fixed overall sample size, it is generally better (in terms of statistical power) if the sample is equally distributed across the different clusters; highly variable cluster sizes increase the standard errors of effect size estimates and reduce the ability to determine if an intervention or treatment is effective.

When I started to prepare a quick simulation to demonstrate this phenomenon, I quickly realized that there was no easy way using simstudy (my simulation package of choice) to generate the desired variable cluster sizes while holding the total sample size constant. I thought about it for a bit and came up with a simple solution that is now implemented and available for download (devtools::install_github("kgoldfeld/simstudy")). My plan here is to describe this solution, and then show the results of the simulation that inspired the need for (and uses) the new functionality.

Quick recap on how to generate cluster data with simstudyThere are two ways I would have typically simulated clustered data using a data generation process defined by a linear mixed-effects model. In both cases, we would define a variable (n) the represents the number of subjects per cluster.

In the first approach, we would assume perfectly balanced cluster sizes and fix (n) at a constant value. In the example, we will generate 10 clusters with 20 members each. In this case, I am generating the cluster-level random effect and treatment assignment. The individual-level outcome is a function of the treatment assignment and the cluster effect, as well as random individual-level variation. All of this is specified in the data generation definitions:

library(simstudy)d0 <- defData(varname = "n", formula = 20, dist = "nonrandom")d0 <- defData(d0, varname = "a", formula = 0, variance = 0.33)d0 <- defData(d0, varname = "rx", formula = "1;1", dist = "trtAssign")d1 <- defDataAdd(varname = "y", formula = "18 + 1.6 * rx + a", variance = 16, dist = "normal") The data are generated in two steps. First, the cluster-level data are generated:

set.seed(2761)dc <- genData(10, d0, "site")dc## site n a rx## 1: 1 20 -0.3548 1## 2: 2 20 -1.1232 1## 3: 3 20 -0.5963 0## 4: 4 20 -0.0503 1## 5: 5 20 0.0894 0## 6: 6 20 0.5294 1## 7: 7 20 1.2302 0## 8: 8 20 0.9663 1## 9: 9 20 0.0993 0## 10: 10 20 0.6508 0 And then the individual level data are generated, (n = 20) subjects for each site:

dd <- genCluster(dc, "site", "n", "id")dd <- addColumns(d1, dd)dd## site n a rx id y## 1: 1 20 -0.355 1 1 17.7## 2: 1 20 -0.355 1 2 16.2## 3: 1 20 -0.355 1 3 19.2## 4: 1 20 -0.355 1 4 20.6## 5: 1 20 -0.355 1 5 14.7## --- ## 196: 10 20 0.651 0 196 25.3## 197: 10 20 0.651 0 197 22.1## 198: 10 20 0.651 0 198 13.2## 199: 10 20 0.651 0 199 15.6## 200: 10 20 0.651 0 200 13.8 If we want variable cluster sizes, we could slightly modify the data definitions so that (n) is no longer constant. (From here on out, I am just generating (n) and the cluster-level data without the random effects and treatment assignment, but I could have just as easily included the full data.) Here, I am using the Poisson distribution, but I could use the negative binomial distribution if I wanted more variation across clusters:

d0 <- defData(varname = "n", formula = 20, dist = "poisson")genData(10, d0, "site")## site n## 1: 1 13## 2: 2 18## 3: 3 21## 4: 4 26## 5: 5 25## 6: 6 27## 7: 7 23## 8: 8 30## 9: 9 23## 10: 10 20 This is great, but the total sample size is no longer fixed at 200 (here we have randomly generated 226 individuals). The total will vary from sample to sample. So, if we want to have both across-cluster variability and constant total sample size, we need a new approach.

New approach using simstudyThere is a new simstudy distribution called “clusterSize”, which requires two parameters: the (fixed) total sample size (input into the formula field) and a (non-negative) dispersion measure that represents the variability across clusters (input into the variance field). (The idea behind the data generation is described in the addendum.) If the dispersion is set to (0), then we will have constant cluster sizes:

d0 <- defData(varname = "n", formula = 200, variance = 0, dist = "clusterSize")genData(10, d0, "site")## site n## 1: 1 20## 2: 2 20## 3: 3 20## 4: 4 20## 5: 5 20## 6: 6 20## 7: 7 20## 8: 8 20## 9: 9 20## 10: 10 20 When we increase the dispersion, we start to introduce cluster-size variability but keep the overall sample size at 200:

d0 <- defData(varname = "n", formula = 200, variance = 0.2, dist = "clusterSize")genData(10, d0, "site")## site n## 1: 1 20## 2: 2 28## 3: 3 25## 4: 4 24## 5: 5 28## 6: 6 22## 7: 7 7## 8: 8 13## 9: 9 22## 10: 10 11 And we can have extreme variability with a very high dispersion value:

d0 <- defData(varname = "n", formula = 200, variance = 5, dist = "clusterSize")genData(10, d0, "site")## site n## 1: 1 10## 2: 2 2## 3: 3 17## 4: 4 2## 5: 5 49## 6: 6 110## 7: 7 1## 8: 8 4## 9: 9 1## 10: 10 4 Application: cluster-size variability and statistical powerI conducted a simulation experiment to assess the impact of the dispersion parameter on the estimated power for a cluster randomized trial with cluster-level effects. In the simulation (code is available here), I assumed 20 clusters (10 randomized to the experimental arm, 10 to the control arm) and a total of 500 participants (so on average 25 per arm).

The specific model I used to generate the data was

[y_{ij} = 20 + 1.6 * A_{i} + a_{i} + e_{ij},]

where (y_{ij}) is the continuous outcome for subject (j) in cluster (i). (A_i) is the treatment indicator for cluster (i), (A_i = 1) if cluster (i) has been randomized to the experimental arm, (A_i = 0) otherwise. (a_i) is the cluster-specific random effect, is normally distributed: (a_i \sim N(\mu = 0, \sigma_a^2)). (e_{ij}) is the (unmeasured) individual (j) effect, and is also normally distributed: (e_{ij} \sim N(\mu =0, \sigma_e^2 = 16)).

Statistical power is directly influenced by overall variability of the outcome, which in this case includes the cluster and individual level variation. Specifically, power is a function of the intra-class (or intra-cluster) correlation (ICC), which can be calculated using[ICC = \frac{\sigma_a^2}{\sigma_a^2 + \sigma_e^2}.]In the simulations, ICCs ranged from (0.1) to (0.4). Since (\sigma_e^2) was fixed, the variance (\sigma^2_a) was determined by the ICC.

The focus of these simulations is to provide a figure that illustrates the impact of cluster-size variability (with constant total sample size) on power. I used different dispersion assumptions, ranging from 0 to 0.5, to generate different data sets. For each of the 44 ICC/dispersion parameter combinations, I generated 50,000 data sets (yes, I used a high performance computing core) and estimated a linear mixed effect model for each. The power was calculated for each combination by looking at the proportion of the p-values less than 0.05. The figure below shows the results; it appears that both higher ICCs and cluster size variability lead to reduced power:

Given a particular ICC level, it seems pretty clear that cluster-size variability does matter when thinking about power. However, the impact may not be as substantial as the general variability in the outcome across the cluster. Unfortunately, I know of no analytic software that will provide insight into this. But if you are using simulation to conduct the sample size calculation, it is now extremely easy to incorporate cluster size variability into the simulations (particularly if you are using simstudy).

Support:

This work was supported in part by the National Institute on Aging (NIA) of the National Institutes of Health under Award Number U54AG063546, which funds the NIA IMbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials Collaboratory (NIA IMPACT Collaboratory). The author, a member of the Design and Statistics Core, was the sole writer of this blog post and has no conflicts. The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health.Addendum - a simple trick to generate variationGenerating the variable cluster sizes under a fixed total is actually quite simple if you take advantage of the Dirichlet distribution. The Dirichlet distribution is essentially a multivariate generalization of the beta distribution. In the Dirichlet distribution, the multivariate values range from 0 to 1, and they sum to 1. Given the range of the data, it is very natural to use values generated from this distribution as probabilities or proportions, which is what the simstudy clusterSize distribution does. This is perhaps easiest to see in a simple example.

Values from the Dirichlet distribution can be generated using the rdirichlet function in the the dirmult package. The key parameter, called the concentration parameter, is a vector of length k, where k is the number of values (e.g. clusters) we are interested in generating. In the first example, I am generating 10 values using a concentration parameter of 32. (In the simstudy implementation of the clusterSize distribution, the dispersion parameter (d) is 1/concentration.)

Generating 20 values, we can see that all values are between 0 and 1, and sum to 1:

x <- dirmult::rdirichlet(1, alpha = rep(32, 20) )[1,]x## [1] 0.0442 0.0465 0.0374 0.0506 0.0516 0.0516 0.0623 0.0429 0.0601 0.0404## [11] 0.0490 0.0517 0.0589 0.0566 0.0320 0.0568 0.0458 0.0610 0.0542 0.0462sum(x)## [1] 1 From here, it is easy generate values between 0 and 400 if we have a total sample size of 400.

s1 <- floor(x*400)s1## [1] 17 18 14 20 20 20 24 17 24 16 19 20 23 22 12 22 18 24 21 18sum(s1)## [1] 389 Due to rounding, the sum does not equal to 400. In the simstudy function this rounding error is accounted for by allocating an additional unit to randomly selected clusters.

If we use a lower concentration parameter (in this case 4), there should be more variability, and indeed there appears to be:

x <- dirmult::rdirichlet(1, alpha = rep(4, 20) )[1,]s2 <- floor(x * 400)s2## [1] 17 27 7 11 18 24 13 26 5 25 23 28 19 26 14 11 43 22 24 9 It appears that the second sample is more variable than the first, and we can confirm this with the standard deviation:

c(sd1 = sd(s1), sd2 = sd(s2))## sd1 sd2 ## 3.30 9.05 To leave a comment for the author, please follow the link and comment on their blog: ouR data generation.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Generating variable cluster sizes to assess power in cluster randomize trials

View Details

[This article was first published on modTools, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I often come across GBIF users who are unaware that the records available for a given taxon are not necessarily all presences: there’s a column named “occurrenceStatus” whose value can be “PRESENT” or “ABSENT”! The absence records can, of course, be removed with simple operations in R or even omitted from the download, but many users overlook or accidentally skip this crucial step, and then end up analysing species distributions with some absence records being used as presences.

To reduce the chances of this happening, I just added new arguments to the fuzzySim function cleanCoords() (which was explained in a previous post) to allow removing absence records too, when the user wants to filter only the presences. Here’s a usage example using aardvark occurrence records downloaded with the geodata package. Note that this requires installing the latest version of fuzzySim (4.9.9):

occ <- geodata::sp\_occurrence(genus = "Orycteropus", species = "afer", fixnames = FALSE)# NOTE: as per the function help file, if you use GBIF data, remember to check the data use agreement and follow guidance on how to properly cite the actual data sources!names(occ)occ\_clean <- fuzzySim::cleanCoords(occ, coord.cols = c("decimalLongitude", "decimalLatitude"), uncert.col = "coordinateUncertaintyInMeters", uncert.limit = 10000, abs.col = "occurrenceStatus")# 764 rows in input data# 576 rows after 'rm.dup'# 575 rows after 'rm.equal'# 575 rows after 'rm.imposs'# 575 rows after 'rm.missing.any'# 575 rows after 'rm.zero.any'# 570 rows after 'rm.imprec.any'# 465 rows after 'rm.uncert' (with uncert.limit=10000 and uncert.na.pass=TRUE)# 461 rows after 'rm.abs' As you can see, besides some common biodiversity data issues such as duplicated or erroneous coordinates, additional records were removed because they represented absences. Hopefully this can help prevent their incorrect use as species presences. Feedback welcome!

To leave a comment for the author, please follow the link and comment on their blog: modTools.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Removing absences from GBIF datasets

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.WebR has a template for React, but I’m not a fan of it or Vue (a fact longtime readers are likely tired of hearing right about now). It’s my opinion and experience that Lit webcomponents come closer to “bare metal” webcomponents, which means the “lock-in” with Lit is way less of a potential headache than all the baggage that comes with the other frameworks.

I leaned into exposition for most of my WebR Experiments, but that very likely made it hard to reproduce or even use those repos without some pain. So, I decided to reduce the pain, remove the inertia and make a template (GH) you can use almost immediately from the command line.

The only “inertia” is that you need npm installed. Subsequently, cd into the parent directory of the new project you want to make and:

npx create-webr-vite-lit my-webr-projectcd my-webr-projectnpm installnpx vite --port=4000 then, hit http://localhost:4000/ (change the port if you’re already using it for something else).

You can check it out on this demo site.

Batteries Included* Vite (for fast building) * WebR (duh) + r.js which has all the setup code for WebR + some helpers (more coming here, too) * Pyodide (initiall disabled) + py.js which does not get used but is available if you want to use pyodide * Lit (webcomponents) — it ships with 3: + one for my usual “loading…” status message (which gets a facelift) + one generic webcomponent for Observable Plots + one simple “button” webcomponent to trigger simple actions + more are coming! The goal is to wrap all the inputs and outputs provided by Bonsai (below). PR’s welcome! * A lightweight CSS framework called Bonsai that I added dark-mode support for. The post-create default page is the Bonsai grid & CSS reference. The webcomponents show how to make all the Bonsai styles available to the components (there’s a default full separation of all things from the webcomponents). * An example justfile since I’ve grown quite fond of Just

The default/demo “app” demonstrates how all the components work.

FINThis setup should have you up and running with your own apps in no time.

I tested light/dark mode switching in Chrome and Safari (macOS/iOS) and the dark/light switching works as intended. Arc doesn’t respond to it, so I’ll be debugging that.

Drop issues in GH if I need to tweak the dark mode, or if you run into snags.

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Start Creating Vanilla JS WebR Apps With Less Inertia

View Details

[This article was first published on r.iresmi.net, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Results of sanitary controls in France can be found on data.gouv.fr however, only the running year is available… Thanks to @cquest@amicale.net we can access the archives since 2017.

First a global view of the dataset :

About 800 controls per week, except during the lock-down in 2020, and a slightly lower control pressure in 2021 and 2022.

Poor results (To be improved or To be corrected urgently) are stable at around 6 %, except a recent spike in 2023?

It seems that poor results increase during the year, from June to November.

This surprising periodic phenomenon is also visible by day :

So for the « when », it is : not in summer or autumn.

What about the « where »? It seems you also could be careful in some départements…

Not good in Guadeloupe, Guyane, Réunion and the southern lower Seine valley, west of Paris.

Can we see more in details ? Using a 30 km kernel smoothing :

It confirms some hot-spots west of Paris, in Alsace, in Indre, Cher, Alpes-Maritimes and between Gironde and Landes. You are safer in Paris and Bretagne…

Has it changed ?

No real trend…

Download the supporting data and R scripts :

alimconfianceTéléchargerTo leave a comment for the author, please follow the link and comment on their blog: r.iresmi.net.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Where and when (not) to eat in France ?

View Details

[This article was first published on Code R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.For a while I wanted to write a post to compile some of the tricks I’ve learnt over the years of using rmarkdown.I also wanted other people’s input so I asked for suggestions on Mastodon.So here are the 12 tips I decided to include in no particular order.

Make chunk options non-optionalI use this trick to force myself to write captions to all figures:

knitr::opts\_hooks$set(label = function(options) { if (is.null(options$fig.cap)) { stop("Every figure has to have a caption!") } return(options)}) This “hook” runs once for each chunk in which label is not NULL –which is all of them– and throws an error if the fig.cap option is NULL (missing).This will absolutely force your lazy ass to actually put captions in all your figures.

Inspired by Zhian N Kamvar I’m now also using this to force myself to always name all my chunks:

knitr::opts\_hooks$set(label = function(options) { # Check if the label comes from the default label for unnamed chuncks default\_label <- knitr::opts\_knit$get("unnamed.chunk.label") has\_default\_label <- grepl(default\_label, options$label) if (has\_default\_label) { stop("Name your chunks!") } return(options)}) This code can’t just check for is.NULL(options$label) because unnamed chunks get default labels, so it gets the default label with knitr::opts_knit$get("unnamed.chunk.label") and then checks if the name of the chunk is auto-generated.This fails in the ridiculously edge case of manually-defined labels that contain the same text as the default label.

You can use these principles to run all kinds of checks on your chunk options.The only issue with this approach is that the knitting process ends when it finds the first “bad” instance.It might be better to record all the offending chunks and their reasons and then throw a message or error at the end.

Captions using text referencesGood captions tend to be long and they sometimes include complex strings like (\LaTeX) notation or references to previous figures (e.g. “Same as Fig. 2 but for clowns with red noses.”).These kinds of captions can be hard to parse visually as chunk options and need a lot of escaped characters if written as a string.Bookdown offers text references as a solution.

You can start a line of text with (ref:label) and then use (ref:label) to refer to it in your chunk options.So you can write

(ref:red-noses-cap) Same as Figure \@ref(fig:blue-noses) but for clowns with *red* noses. And this is a mathematical formula just as an example: $\pi=3$. And then write the chunk as

{r, fig.cap = "(ref:red-noses-cap)"}plot(red_noses) Default captionWhen using text references to define captions, I’ve found that it’s a bit annoying and redundant to explicitly set the fig.cap option.Instead I prefer to set up a default caption of the form (ref:label-cap) like this:

knitr::opts\_hooks$set(label = function(options) { if (is.null(options$fig.cap)) { options$fig.cap <- paste0("(ref:", options$label, "-cap)") }}) With this, every chunk will have a default text-referenced caption with a predictable name.

Save plots in multiple formatsDid you know that the dev chunk option can be a vector of formats?This enables you to save figures in multiple formats at once.

knitr::opts\_chunk$set(dev = c('png', 'svg')) This simple but possibly overlook feature (suggested by Robert Flight) can be useful if you want to use vector graphics in your document but also need raster versions to share more easily with your colleges or online.

Exit prematurelyWith long and complex documents sometimes come weird errors that are hard to pin down.Or you might want to work on some early part of the document even if some later parts are unfinished and don’t knit.Both Mickaël CANOUIL and superboreen pointed out that you can use knitr::knit_exit() to end document rendering “before it gets to the hideous code I haven’t fixed yet”.

knitr::knit\_exit() Get the output formatWhile the promise of rmarkdown is to have portable code that can be rendered into any document, but abstractions are leaky and this doesn’t always work out.For example, I’ve found that there’s not a single table-generating package that does a good job of rendering decent-looking LaTeX, HTML and Word tables without changes in the code.So sometimes the code needs to know to which document format it’s going to render.

The function knitr::pandoc_to() returns the “final destination” of the document, which can be “latex”, “html” or “docx”.

knitr::pandoc\_to() It can also return a logical indicating if the output format is the one specified in the argument.This enables code that only runs for some formats and not others:

if (knitr::pandoc\_to("docx")) { # Something to do only if the output is docx} Beware that knitr::pandoc_to() will return NULL when run interactively, so you might want to catch that case.

Other similar functions are is_latex_output() and is_html_output().

Configure cache pathMy documents often have some code that takes a while to run so I make liberal use of the cache feature.But sometimes I like to control where that cache is stored.That can be done with knitr::opts_chunk$set(cache.path = path).

This is a good solution when rendering to multiple formats, since changing the format seems to invalidate the cache and make it useless.So what I do is set up one cache for each format:

format <- knitr::pandoc\_to()knitr::opts\_chunk$set( cache.path = file.path("cache", format, "") # The last "" is necessary) Get current fileThe knitr::current_input() returns the input file being rendered by knitr.This can be useful in a bunch of cases, but I use it to, again, control cache and figure locations.

On a bookdown document, I like each chapter to use its own folder for cache and figures, so I have this in my setup chunk:

format <- knitr::pandoc\_to() chapter <- tools::file\_path\_sans\_ext(knitr::current\_input())knitr::opts\_chunk$set( fig.path = file.path("figures", chapter, ""), cache.path = file.path("cache", chapter, format, "")) Easily invalidate cacheSpeaking of cache, sometimes I want to run your document from scratch without the cache.Either as a final test that all the code runs well, or when faced with strange bugs that I suspect might be cache-related.

So I almost always set up a cache.extra chunk option that will invalidate the cache each time it changes.

knitr::opts\_chunk$set(cache.extra = 42) # Change the number to invalidate cache Do stuff after knittingKnitr runs the “document” hook after knitting.You can customise that hook to do whatever you want:

```

First save the default hook to use later. knit_doc <- knitr::knit_hooks$get("document")knitr::knit_hooks$set(document = function(x) { # So stufff knit_doc(x) # Then do whatever knitr was going to do anyway})

``` For example, I sometimes like to add this so I get a desktop notification when my computer finishes knitting

```

This needs the notify-send cli. # https://manpages.ubuntu.com/manpages/bionic/man1/notify-send.1.htmlnotify <- function(title = "title", text = NULL, time = 2) { time <- time1000 system(paste0('notify-send "', title, '" "', text, '" -t ', time, ' -a rstudio'))}start_time <- unclass(Sys.time())min_time <- 53600 # Only notify for long-running jobs (5 minutes)knit_doc <- knitr::knit_hooks$get("document")knitr::knit_hooks$set(document = function(x) { took <- unclass(Sys.time()) - start_time if (took >= min_time) { notify("Done knitting!", paste0("Took ", round(took), " seconds"), time = 5) } knit_doc(x)})

``` You might use this to get notifications to your phone with RPushbullet or send emails with emayili.You might also want to check the notifier package.

Scripts to and from RMarkdownFinally, Katie highlighted the knitr::spin() function, which turns specially formatted R scripts and turns them into RMarkdown documents.And for the exact opposite workflow, Ken Butler points out the knitr::purl() function.

To leave a comment for the author, please follow the link and comment on their blog: Code R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 11 tricks to level up your rmarkdown documents

View Details

[This article was first published on Outsider Data Science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Inspired by TidyTuesdaySome of the the R data science community participate in a weekly challenge called “Tidy Tuesday,” where an interesting data set is presented for analysis but mostly visualization. There are some tremendous examples of beautiful work posted on Twitter with the hashtag #tidytuesday.

African Tweets and SentimentRecently, the weekly dataset was a collection of over 100,000 tweets, apparently from 2022, in 14 African languages, with sentiment labels. The paper describing the set and methods is here (Muhammad et al. 2023). The TidyTuesday project and raw data are here. This is quite a diverse data set including many tweets in English, tweets in languages which, like English, use the Latin character set and tweets in other character sets, including Arabic.

I saw this as an avenue to ask a couple interesting questions.

Can we apply sentiment analysis techniques to a translated version of this dataset? How good is Google Translate, anyway?
Over the past year there has been much talk about the differences in attitudes of the “global north” vs. the “global south.” Does this data set reveal anything about that?

I saw an opportunity to sharpen my skills in a couple areas, using the Google API for batch translation and using RStudio’s Tidytext and Tidymodels toolsets.

I split these explorations into four snack-able posts.

In this post we show how to use Google Cloud Translate to batch translate the entire data set.
Here we use the Tidytext framework to do sentiment analysis with word valences.
Next, we’ll compare machine learning approaches in the TidyModels framework to do sentiment analysis on both the native and translated tweets.
Finally, let’s use the already assigned sentiment tags to explore African attitudes to the “global north.”

DisclaimerThe usual caveats apply. I am not a social scientist. I am a hobbyist. This is an exercise in R coding so I make no claim that my conclusions about any of this data are valid.

Get the DataHere are the packages we’ll need for this project.

suppressPackageStartupMessages({ library(tidyverse) library(googleLanguageR) library(future) library(furrr) library(rvest)}) The TidyTuesday github repo has the Afrisenti dataset with all the languages combined. Let’s load it.

afrisenti <- readr::read\_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-28/afrisenti.csv', show\_col\_types = FALSE)afrisenti

```

A tibble: 111,720 × 4 language_iso_code tweet label inten…¹ 1 amh አማራ ክልል ፈልቶበታል ልኩን ማስገባት ነው!!! ሙስሊሞችን ጠልቶ 85… nega… dev 2 amh ሰውን አንገት በሚያስደፋ መልኩ ዝም ብሎ ሙድ መያዝ....ስልህ ያ ሰው… nega… dev 3 amh የቤት ውስጥ ጥቃት – ያለሰሚ – ያለተመልካች nega… dev 4 amh Ethiopia ወያኔን ለመጣል ምን ድርሻ ነበራችሁ ? ከወደቀ በኋላ ጉ… nega… dev 5 amh ኦሮሞ ምንም ቢማር ከብት ነዉ አያስተዉልም ጥንብ ዘረኛ ናቸዉ nega… dev 6 amh ቲሽ ጨለምተኛ ዱቄት 97 ላይ ቆመሃል እንዴ ጊዜው ነጉዷል 2012 ነው… nega… dev 7 amh በምዕራብ ኦሮሚያ በሚገኙ በሁለቱ የወለጋ ዞኖች (ምስራቅ ወለጋና ሆሮ … nega… dev 8 amh ያየሰው ሺመልስ ላይ የደረሰው ነገር ያሳዝናል። በቃል ኣላምረውም ላይም… nega… dev 9 amh ያልተረጋጋች ሀገር ምርጫ አያስፈልጋትም nega… dev 10 amh ደደቡ እና አረፋው የኢትዮጵያው ጠ/ሚ አብዪ ከኤርትራው ኣያቶላህ ኢሰያ… nega… dev # … with 111,710 more rows, and abbreviated variable name ¹intended_use

`` Translate the TweetsTo use Google translate in batch mode we’ll need an API key. I don’t understand Google. For some of their services, like Maps, a single API key is needed. Instead, for Translate, we need a JSON file with the key. Once you get the key, store the file name in your.Renviron` file with the key name “GL_AUTH” then the googlelanguageR package will automatically authenticate when it loads.

NoteGetting the Google Language API key is a complicated procedure and I won’t detail it here but you can find complete instructions in the googlelanguageR package introduction.

Once once your key is created you can start translating with R. This isn’t free. Translating over 100,000 tweets cost me about US$15. A couple bucks was wasted because I submitted all the tweets including those in English. You might choose to filter English tweets out first. If you just want to work with the same data set you can download my translations (see below) for FREE.

I first tried shooting the whole data set into the translation routine but Google protested that I was sending too much. I divided the set into batches of 100 tweets at a time which fixed the problem.

We can dramatically speed things up using the furrr and future packages to allow parallel processing using just three lines of code. furrr adapts the purrr::map() family of functions to allow parallel execution. Very simple. Amazing!

future::plan(multicore) # will use all available coresbatch\_size = min(100, nrow(afrisenti))afrisenti\_translated <- seq(0, nrow(afrisenti) - batch\_size, by = batch\_size) |> furrr::future\_map\_dfr(\(x) { gl\_translate(afrisenti$tweet[(x + 1):(x + batch\_size)]) }, .progress = TRUE)future::plan(sequential) # back to normal It’s worth looking over the code above because it packs a lot of power in few lines and, to me, shows how cool R is. Basically, we identify the batches of rows from the data set we want to ship out to Google and translate them in as many parallel streams as our hardware allows.

Let’s clean up the data a little.

```

merge with source data and clean up a littleafrisenti_translated <- afrisenti_translated |> na.omit() |> select(-text) |> bind_cols(afrisenti) |> rowid_to_column(var = "tweet_num") |> mutate(tweet_num = as.numeric(tweet_num)) mutate(intended_use = as_factor(intended_use)) |> mutate(detectedSourceLanguage = as_factor(detectedSourceLanguage)) |> mutate(language_iso_code = as_factor(language_iso_code)) |> mutate(label = as.factor(label))

`` While it’s not strictly necessary, I wanted to see the long names for the languages, rather than just 2-character ISO codes. Using Wikipedia I created a file that we can use for reference. Thervest` package makes turning an HTML table into a data frame easy. At the same time, let’s make sure the language labels from the data set are consistent with the Google langauge labels.

```

get languages from wikipedia# take the second table on the pageiso_lang <- html_table(read_html("https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes#External_links"))[[2]]# since Wikipedia is subject to frequent change, if the entry doesn't work# you can get the file here# iso_lang <- readr::read_csv('https://raw.githubusercontent.com/apsteinmetz/tidytuesday/master/2023-02-28_african_language/data/iso_lang.csv',# show_col_types = FALSE)iso_lang <- iso_lang %>% rename(assigned_language = `639-2/T`, detected_language = `639-1`, language = `ISO language name`) %>% select(1:3)# clean up langauge namesafrisenti_translated <- afrisenti_translated %>% mutate(language_iso_code = str_replace_all(language_iso_code,"pt-MZ","por")) %>% mutate(language_iso_code = str_replace_all(language_iso_code,"ary","ara")) %>% mutate(language_iso_code = str_replace_all(language_iso_code,"arq","ara")) %>% mutate(language_iso_code = str_replace_all(language_iso_code,"pcm","eng")) %>% rename(assigned_language = language_iso_code, detected_language = detectedSourceLanguage) %>% left_join(select(iso_lang,-assigned_language)) %>% rename(detected_long = language) %>% left_join(select(iso_lang,-detected_language)) %>% rename(assigned_long = language) # save it for later usesave(afrisenti_translated,file="data/afrisenti_translated.rdata")

``` Save Some MoneyThat done, YOU don’t want to pay $15 and you don’t have to. Let’s download the translated Afrisenti data set from my repo instead.

afrisenti\_translated <- readr::read\_csv('https://raw.githubusercontent.com/apsteinmetz/tidytuesday/master/2023-02-28\_african\_language/data/afrisenti\_translated.csv', show\_col\_types = FALSE)afrisenti\_translated

```

A tibble: 111,720 × 9 tweet_num translatedText detec…¹ assig…² tweet label inten…³ detec…⁴ assig…⁵ 1 1 Amhara region … am amh አማራ … nega… dev Amharic Amharic 2 2 Having a mood … am amh ሰውን … nega… dev Amharic Amharic 3 3 Domestic viole… am amh የቤት … nega… dev Amharic Amharic 4 4 Ethiopia, what… am amh Ethi… nega… dev Amharic Amharic 5 5 No matter how … am amh ኦሮሞ … nega… dev Amharic Amharic 6 6 Tish, dark pow… am amh ቲሽ ጨ… nega… dev Amharic Amharic 7 7 Local resident… am amh በምዕራ… nega… dev Amharic Amharic 8 8 What happened … am amh ያየሰው… nega… dev Amharic Amharic 9 9 An unstable co… am amh ያልተረ… nega… dev Amharic Amharic10 10 The idiot and … am amh ደደቡ … nega… dev Amharic Amharic# … with 111,710 more rows, and abbreviated variable names ¹detected_language,# ²assigned_language, ³intended_use, ⁴detected_long, ⁵assigned_long

``` Now that the hard work is done let’s do some preliminary checks, Let’s see if the language that Google detects agrees with the assigned language in the data set. First we convert the language long names to factors and see how many levels there are. These are the 13 assigned languages.

afrisenti\_translated <- afrisenti\_translated |> mutate(across(contains("long"),\(x) as.factor(x)))levels(afrisenti\_translated$assigned\_long)

[1] "Amharic" "Arabic" "English" "Hausa" "Igbo" [6] "Kinyarwanda" "Oromo" "Portuguese" "Swahili" "Tigrinya" [11] "Tsonga" "Twi" "Yoruba" Here are the languages that Google detects.

levels(afrisenti\_translated$detected\_long)

[1] "Afrikaans" "Akan" [3] "Amharic" "Arabic" [5] "Aymara" "Bambara" [7] "Basque" "Bengali" [9] "Bosnian" "Bulgarian" [11] "Catalan, Valencian" "Chichewa, Chewa, Nyanja" [13] "Chinese" "Corsican" [15] "Croatian" "Czech" [17] "Danish" "Dutch, Flemish" [19] "English" "Esperanto" [21] "Estonian" "Ewe" [23] "Finnish" "French" [25] "Gaelic, Scottish Gaelic" "Galician" [27] "Ganda" "German" [29] "Greek, Modern (1453–)" "Guarani" [31] "Gujarati" "Haitian, Haitian Creole" [33] "Hausa" "Hindi" [35] "Hungarian" "Igbo" [37] "Indonesian" "Irish" [39] "Italian" "Japanese" [41] "Javanese" "Kannada" [43] "Kinyarwanda" "Korean" [45] "Kurdish" "Latin" [47] "Latvian" "Lingala" [49] "Luxembourgish, Letzeburgesch" "Malagasy" [51] "Malay" "Malayalam" [53] "Maltese" "Maori" [55] "Marathi" "Norwegian" [57] "Oromo" "Pashto, Pushto" [59] "Persian" "Polish" [61] "Portuguese" "Quechua" [63] "Romanian, Moldavian, Moldovan" "Russian" [65] "Samoan" "Shona" [67] "Sindhi" "Slovak" [69] "Slovenian" "Somali" [71] "Southern Sotho" "Spanish, Castilian" [73] "Sundanese" "Swahili" [75] "Swedish" "Tamil" [77] "Telugu" "Tigrinya" [79] "Tsonga" "Turkish" [81] "Turkmen" "Ukrainian" [83] "Urdu" "Uzbek" [85] "Vietnamese" "Welsh" [87] "Western Frisian" "Xhosa" [89] "Yoruba" "Zulu" Uh, oh. 90 detected languages vs. 13 assigned languages. Is this a problem? What fraction of tweets are in languages not mentioned in the original set?

alt\_count <- afrisenti\_translated |> filter(!(detected\_long %in% levels(assigned\_long))) |> nrow()/nrow(afrisenti\_translated)*100 paste0(round(alt\_count,1),"%")

[1] "6.9%" Not a big number. Let’s collapse all but the top 15 languages into an “other” category.

How frequently does Google disagree with the assigned language?

afrisenti\_translated <- afrisenti\_translated |> mutate(detected\_long = replace\_na(as.character(detected\_long,"Unknown"))) |> mutate(detected\_long = fct\_lump\_n(detected\_long,15))xt <- xtabs(~afrisenti\_translated$assigned\_long + afrisenti\_translated$detected\_long) |> broom::tidy() |> rename(assigned = 1,google = 2) |> group\_by(assigned) |> mutate(Proportion = n/sum(n)) xt |> ggplot(aes(assigned, google,fill=Proportion)) + geom\_tile() + scale\_fill\_gradient(low = "#FFBF00", high = "#007000") + theme( plot.background = element\_rect(fill = "#FDECCD", color = NA), legend.background = element\_blank(), axis.ticks = element\_blank(), axis.text.x = element\_text(angle = 45,vjust = .7,hjust = .6), panel.background = element\_blank(), panel.grid = element\_blank() ) + labs( title = "African Languages Tweets\nQ: Does Google Detect The Same Language?", subtitle = "A: Almost Entirely", x = "Afrisenti Assigned Language", y = "Google Translate Detected Language", caption = "source: Afrisenti Data Set" ) The fact that disagreement about the tweet language is so rare gives us some confidence that we are on the right track.

Then look at the first row.

afrisenti\_translated$tweet[1]

[1] "አማራ ክልል ፈልቶበታል ልኩን ማስገባት ነው!!! ሙስሊሞችን ጠልቶ 85% ሙስሊሞች በሚኖርባት ኦሮምያ ጋር ግንኙነትን አትሰበው !!!"

afrisenti\_translated$translatedText[1]

[1] "Amhara region needs moderation!!! He hates Muslims and does not think of relations with Oromia, where 85% of Muslims live!!!"

afrisenti\_translated$label[1]

[1] "negative" A quick glance at the translation shows obviously negative sentiment. We are off to a promising start. In the next post we’ll use the tidytext framework to measure the net balance of sentiment for each tweet.

ReferencesMuhammad, Shamsuddeen Hassan, Seid Yimam, Idris Abdulmumin, Ibrahim Sa’id Ahmad, Ousidhoum Nedjma, Ayele Abinew, David Adelani, et al. 2023. “SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval).” In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). To leave a comment for the author, please follow the link and comment on their blog: Outsider Data Science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Sentiment Analysis Using Google Translate (Pt. 1)

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to work with Big Data with Hadoop and Spark! Join our workshop on Working with Big Data with Hadoop and Spark which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: Working with Big Data with Hadoop and Spark

Date: Thursday, May 18th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Jannic Cutura is an economist turned data engineer turned software engineer who works as a Python developer at the European Central Bank’s Stress Test team. Prior to his current position he worked as research analyst/data engineer in the financial stability and monetary policy divisions of the ECB. He holds a masters and Ph.D. in quantitative economics from Goethe University Frankfurt and conducted research projects at the BIS, the IMF and Columbia University.

Description: Big data — datasets that are difficult to handle on standalone retail-grade computers — are rapidly becoming the norm in social science research. This is true both in academia as well as for policy-oriented research in central banks and similar bodies (let alone industry application). Yet traditional econometrics (and econometrics training) tells us little about how to efficiently work with large datasets. In practice, any data set larger than the researchers computer memory (~20- 30GB) is very challenging to handle as, once that barrier is crossed, most data manipulation tasks becomes painfully slow and prone to failure. The goal of this presentation is to (i) explain what happens under the hood when your computer gets slow and (ii) show how distributed computing (in particular Hadoop/Spark) can help to mitigate those issues. By the end, participants will understand the power of distributed computing and how they can use it to both tackle existing data handling challenges and as well as new ones that were previously prohibitively expensive to evaluate on retail grade computers. The workshop will both contain a theory part and a lab session using data bricks. If you want to follow along during the live session you can create your own free account at data bricks by signing up for the community edition (no credit card required).

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Working with Big Data with Hadoop and Spark workshop was first posted on April 14, 2023 at 4:45 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Working with Big Data with Hadoop and Spark workshop

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Our trees are dying. These days, the soft south-wind wafts only the sad notes of a forever-changed landscape. Our beloved forests are changing. And the faint murmurs from the pine tops that graced Aldrich’s ear, are vanishing; they are being replaced with the creaking groans of rising zombie forests.

But there is hope. The burden of taking large-scale action rests on the shoulders of decision-makers, including those in positions of power and influence such as policy-makers and foresters. The responsibility of ensuring that decisions are data-driven.. well that rests on us.

What Are Zombie Forests?
What Are the Risks of Zombie Forests?
Using Data Science to Stop the Spread
How We Use Shiny to Prevent the Rise of Zombie Forests

Zombie Forests: Dawn of the DeadwoodZombie forests – a creative, evocative term sprouting from one of many significant changes brought about by our changing climate. These forests are characterized by large numbers of dead or dying trees that remain standing outside of their rapidly shifting climate zones.

Photo by Niklas Hamann on Unsplash

Setting the Stage for DestructionExtreme weather conditions and temperature shift, resulting in warmer conditions, have left many of our forests off-balance. Unable to keep pace with the moving front, some tree spcies are struggling to surive in the new climate.

Hot and dry summers, warm winters with less snow, mismatched blooming and insect emergence, and frequent extreme weather events. These are just a few factors impacting the functioning of many tree species, limiting their ability to grow and reproduce.

Future Forests – a Shiny Zombie Tree Survival GuideFor my fellow hikers, set foot out on the Continental Divide and you’ll come across these wide, desolate zombie forests comprised of pines.

Need help with your conservation project? Explore our Data4Good projects.

These trees will remain standing, often for years or decades, and do pose a significant risk to both human safety and the environment. However, there is research showing hope for some tree species and a web application that makes it easy to explore and plan ahead. The app’s official title is Future Forests, but for our purposes today – it’s a Zombie Tree Survival Guide.

What Are Zombie Forests?Zombie forests are forested areas that are in a state of decay or decline. They are the result of a combination of factors – mostly human-induced – such as rising global temperatures, changes in precipitation, and the spreading of invasive pests and diseases.

Photo by Meritt Thomas on Unsplash

These factors, either individually or combined, weaken trees and make them vulnerable to additional stressors. Ultimately, this leads to large-scale die-offs. And as we’ve seen in the Sierra Nevada, changing climate thresholds for some species are hitting hard.

What Are the Risks of Zombie Forests?Instead of falling to the ground, many of these trees remain standing. This standing deadwood is both positive and negative. Small amounts of standing deadwood creates habitat and resources for a variety of animals, plants, and fungi. But at a large scale it creates a dangerous, unmanageable, and unstable environment.

Beyond the growing risk of intense wildfires, zombie forests impact biodiversity, create excessive runoff – damaging soils and surface water quality, and reduce carbon sequestration.

Using Data Science and R Shiny to Stop the SpreadTo help address the challenges posed by our changing climate, a group of researchers at IDPAN teamed up with data scientists and R Shiny developers at Appsilon to develop the Future Forests app.

Predicting the Future of ForestsTo better understand the impact of climate change on forest biodiversity, scientists typically use species distribution models. By implementing climate change scenarios based on projections of future greenhouse gas emissions, these models can predict where species will be able to thrive and where they will struggle to survive.

Consumable ScienceWhile these models are important, the results are often difficult to present in a way that is accessible to the general public or even expert audiences. A solution to this problem is the use of interactive data visualizations, tools that present the insights locked within the data in a clear and accessible way.

Shiny dashboards allow you to create interactive yet engaging visual tools that help translate complex scientific analysis into easy-to-understand conclusions.

About the Future Forests AppAppsilon built Future Forests using R Shiny, a web application framework for R and Python. It includes a suite of climate scenario models for 2070, with predicted habitat zones for 12 tree species in Europe.

You can explore the live app and see what tree species to plant for your future climate.

These modeled scenarios and the research performed by IDPAN are one of many solutions. The Shiny app is the delivery system, taking complex problems and making them available to the public.

How We Use Shiny to Prevent the Rise of Zombie ForestsWhile the concept of zombie forests may sound like something out of a horror movie, the reality is that it’s a serious problem that requires urgent attention. Fortunately, technology can play a crucial role in helping us address this issue. In our case, that technology is Shiny.

One of the key advantages of Future Forests is its accessibility. While traditional modeling techniques may require specialized expertise and technical knowledge, Future Forests presents the information in a user-friendly way that any user can click and explore. This makes it an invaluable tool for engaging a wide range of stakeholders in the fight against zombie forests.

The information available allows users to predict where a species may lose its optimal habitat and plan in advance, actions to reduce the loss of economic opportunity and future growth.

Who Is the Future Forests App For?It’s up to policy-makers, foresters, farmers, or any natural resource manager to plan crop species compositions, and shape future forests.

Knowledge of the uncertain duration of particular species can guide decisions for alternative forest type proposals in a given habitat, and for refining silvicultural principles.

Don’t Fight the Zombie Tree Horde, Prevent It!But in making science accessible, and casting the seed of knowledge via Shiny, we hope everyone can plant ahead, even at a local scale.

Learn how to share your Shiny app.

By giving users a glimpse into the future, Future Forests can help users make data-driven decisions about their trees. Policymakers can use the information to determine policies for protecting natural resources. Conservationists can learn where to focus their preservation efforts. Silviculturalists can decide which trees to plant and where. Neighbors can understand the benefits of having trees in their communities.

Users can tailor their searches to their specific needs and areas/species of interest, and explore policy or prioritize business strategies accordingly.

Ultimately, our Shiny apps help make better, data-driven decisions. With Future Forests we can cultivate more appropriate forests for future climates. In turn, we can mitigate the rising tide of Zombie Forests.

Zombie Forests, Climate Change, and Creating Hope with Data ScienceWe find value and trust in the scientific process. Climate change and its effects are here, but we can still mitigate and adapt. We can take scientific findings, apply our knowledge in our field, and connect decision-makers with the data.

We hope our tools, like Future Forests, can be used to prevent mass, ill-advised plantings and speed up the recovery of our planet.

Whether or not zombie forests can be saved, we can at least plant the right seeds for recovery.

If you’re interested in learning more about Data4Good and how we can help apply data science, R/Shiny development, or AI/Machine Learning to your project, please contact us.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Zombie Forests: Mapping Tree Migration with R Shiny

View Details

[This article was first published on R on Nicola Rennie, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Time series data often contains information that’s not easily seen when simply visualising the data. We can uncover these hidden features using time series analysis, and use them to classify time series that exhibit similar features.

Data In this blog post, we’ll extract features from and classify time series relating to sound recordings of patients’ heart beats. The data comes from the CirCor DigiScope Phonocardiogram Dataset, and consists of 5,272 heart sound recordings of 1,568 different patients. The data can be downloaded from www.physionet.org/content/circor-heart-sound/1.0.3 – see references at the end of this post for full attribution.

Show code: load packages

| 1 2 3 4 5 6 7 8 9101112131415161718 | library(tuneR)library(tsfeatures)library(stringr)library(tidyr)library(tibble)library(dplyr)library(ggplot2)library(glue)library(here)library(purrr)library(rstatix)library(nrBrand)library(vip)library(forcats)library(patchwork)library(ggcorrplot)library(tidymodels)tidymodels\_prefer() |

Show code: read data

| 1 2 3 4 5 6 7 8 910 | read\_data <- function(filename) { subject\_id <- str\_sub(filename, 1, 5) aus\_loc <- str\_sub(filename, 7, 8) d <- readWave(glue("Data/", filename)) df <- tibble(sample\_time = seq\_len(length(d)), value = d@left, subject\_id = subject\_id, aus\_loc = aus\_loc) return(df)} |

Time series plot of aortic valve sound recording for subject 13918.

Show code: plot sound data

| 1234567 | subject1 <- read\_data("13918\_AV.wav")ggplot(data = subject1) + geom\_line(aes(x = sample\_time, y = value), colour = "#546666") + labs(x = "", y = "", title = glue("Subject: {subject1$subject\_id[1]}")) |

Recordings were made at four main locations:

PV: pulmonary valve point
TV: tricuspid valve point
AV: aortic valve point
MV: mitral valve point
Phc: any other auscultation location

Recordings relating to any other auscultation locations were discarded as there were insufficient samples to draw conclusions from. Each sound recording lasted around 10 seconds, recorded at 4000 Hertz, resulting in approximately 40,000 observations per time series - though due to variability in the length of time of the recordings, the lengths of the time series vary slightly. These differing lengths makes direct comparisons of the time series difficult.

In addition to the sound recording data, information on the subjects such as height, weight, age, and whether or not they had been diagnosed with a heart murmur, was also available. The key question I’m interested in is: do time series features differ between people with heart murmurs and those without? There are a total of 2,385 recordings from subjects without heart murmurs, and 610 with heart murmurs, split across the four recording locations:

| Location | Absent | Present | | --- | --- | --- | | AV | 602 | 151 | | MV | 636 | 171 | | PV | 585 | 146 | | TV | 562 | 142 |

Calculating time series features The {tsfeatures} package provides support for calculating these time series features in R. Not all features will be useful. For example, trend: we know that there isn’t an increasing trend, given the nature of the sound recording data, so we don’t need to compute this.

For this blog post, I’ve focused on five key features:

Entropy: measures the forecastability of a time series, with lower values indicating a high signal-to-noise ratio, and higher indicating a lower signal-to-noise ratio.
ACF: measures the similarity between a time series and a lagged version of itself. For this post, we consider the first autocorrelation coefficient.
Hurst parameter: measures the long-term memory of a time series. It relates to the rate at the autocorrelations decrease as the lag between values increases.
ARCH LM Statistic: measures the autoregressive conditional heteroscedasticity, using Engle’s Lagrange multiplier test.
Median of the median of range indices: measures the median as more and more outliers are included in the calculation.

See the package documentation for full details on these features, and other available features.

Show code: calculate features

| 1 2 3 4 5 6 7 8 910111213141516 | calc\_features <- function(filename) { input\_data <- read\_data(filename) to\_ts <- ts(input\_data$value) ts\_fts <- tsfeatures(to\_ts, features = c("entropy", "acf\_features", "hurst", "arch\_stat", "outlierinclude\_mdrmd")) output <- ts\_fts |> mutate(subject\_id = input\_data$subject\_id[1], .before = 1) |> mutate(aus\_loc = input\_data$aus\_loc[1], .before = 2) return(output)}all\_files <- list.files(path = glue(here(), "/Data/"), recursive = TRUE, pattern = "\\.wav")all\_fts <- purrr::map\_dfr(.x = all\_files, .f = ~calc\_features(.x), .progress = "Features") |

Comparing features in subjects with and without murmurs Let’s compare these time series features for subjects with and without heart murmurs. In this analysis, only subjects that have complete data were considered: recordings available at all four locations, and data relating to sex, height, and weight has no missing values - a total of 533 patients.

Show code: split into murmur and no murmur

| 1 2 3 4 5 6 7 8 91011 | is\_murmur <- function(filename) { subject\_id <- str\_sub(filename, 1, 5) subj <- tibble(murmur = readLines(filename)) output <- subj |> filter(str\_detect(murmur, "#Murmur: ")) |> mutate(murmur = str\_replace(murmur, "#Murmur: ", "")) |> mutate(subject\_id = subject\_id) return(output)}all\_subjects <- list.files(path = glue(here(), "/Data/"), pattern = "\\.txt")all\_murmurs <- purrr::map\_dfr(.x = all\_subjects, .f = ~is\_murmur(.x)) |

Show code: join patient details

| 1 2 3 4 5 6 7 8 91011121314151617181920 | patient\_details <- function(filename) { subject\_id <- str\_sub(filename, 1, 5) subj <- tibble(murmur = readLines(glue("Data/", filename))) output <- subj |> filter(str\_detect(murmur, "#Sex: |#Height: |#Weight: ")) |> separate(murmur, into = c("variable", "value"), sep = ": ") |> mutate(variable = str\_remove\_all(variable, "\\#")) |> pivot\_wider(names\_from = variable, values\_from = value) |> mutate(subject\_id = subject\_id, .before = 1) |> mutate(across(c(Height, Weight), as.numeric)) return(output)}all\_patients <- purrr::map\_dfr(.x = all\_subjects, .f = ~patient\_details(.x), .progress = "Patients")all\_data <- all\_murmurs |> filter(murmur != "Unknown") |> left\_join(all\_fts, by = "subject\_id", multiple = "all") |> drop\_na(aus\_loc) |> filter(aus\_loc != "Ph") |

The boxplot below shows the distribution of the first autocorrelation coefficient at each of the four recording locations, and shows how they differ between patients with heart murmurs and those without. On average, those with a heart murmur exhibit a higher autocorrelation coefficient. Though there is considerable overlap in the distributions with and without murmurs, the first autocorrelation coefficients in those with murmurs are consistently higher across all four recording locations - meaning this is a potential variable to consider when trying to predict whether or nor a recording contains a heart murmur.

Show code: plot distribution of ACF

| 12345678 | ggplot(all\_data) + geom\_boxplot(aes(x = aus\_loc, y = x\_acf1, fill = murmur), alpha = 0.3) + labs(x = "", y = "First autocorrelation coefficient", title = str\_wrap("Difference in first autocorrelation coefficient of time series", 80), subtitle = str\_wrap("Comparing subjects with and without a heart murmur, showing those with a heart murmur exhibit a higher autocorrelation coefficient on average.", 80)) + theme(legend.position = "bottom", plot.title.position = "plot", legend.spacing.x = unit(0.7, 'cm')) |

Fitting a model with {tidymodels} The {tidymodels} framework is a collection of R packages that can be used for statistical modelling and machine learning. Before rushing into fitting and evaluating different models, let’s think a little bit about the data we have:

a binary outcome: a heart murmur is either present or absent.
multiple numeric explanatory variables: 42 in total (10 outputs from the time series features for each recording location, plus height and weight).
a single categorical explanatory variable: sex.

For patients with multiple recordings at the same location, the average of each feature was used.

Show code: prepare data for modelling

| 1 2 3 4 5 6 7 8 9101112 | all\_model\_data <- all\_data |> pivot\_longer(cols = entropy:outlierinclude\_mdrmd, names\_to = "ts\_feature", values\_to = "value") |> unite(col = "loc\_ts\_feature", c(aus\_loc, ts\_feature), sep = "\_") |> pivot\_wider(names\_from = loc\_ts\_feature, values\_from = value, values\_fn = mean) |> select(-subject\_id) |> drop\_na() |

When it comes to fitting models to data, I always recommend starting with simple models. They’re usually easier to interpret, quicker to run, and easier to explain to someone else. If the simple models give good results, then great - we’re done! If they don’t, you at least have something to benchmark other models against, and a way to justify the use of more complex, often more computationally intensive models.

So to start with, let’s keep it simple and try logistic regression. Logistic regression models a probability based on a linear combination of some (independent) variables. Since they model a probability, the outcome is a value between 0 and 1. Then the classification into whether or not the time series featured a heart murmur is based on the output being greater than or less than 0.5 (be default). Another aspect we need to think about with a regression model is: which explanatory variables are relevant?. There’s currently 43 potential explanatory variables, which is a reasonably high number for a data set of this size. It’s also likely that there’s some collinearity (a relationship between explanatory variables) present, and we probably don’t want to include all potential variables.

For example, in the correlation heatmap below you can see that, as expected, height and weight are highly correlated, and it’s unlikely to be useful to include both in a model.

Show code: correlation heatmap

| 1 2 3 4 5 6 7 8 91011 | all\_model\_data |> mutate(murmur = case\_when(murmur == "Absent" ~ 0, murmur == "Present" ~ 1)) |> select(murmur, Sex, Height, Weight, starts\_with("TV\_")) |> cor() |> ggcorrplot() + labs(title = "Correlation matrix for TV time series features", x = "", y = "") + theme(plot.title.position = "plot", axis.text.x = element\_text(angle = 45, vjust = 1, hjust = 1)) |

So we need a way of picking which variables to include in our model. One potential way is stepwise regression (where variables are iteratively included/removed from the model and the fit compared). With 43 variables, this approach could take quite a while, and it could be easy to go down the wrong path in selecting variables. Instead, we’re going to use Lasso regression.

Lasso regression is a method for estimating coefficients in linear models, with a special constraint that forces the sum of the absolute value of the coefficients to be less than a particular value. This means that some coefficients are forced to be zero, and so Lasso can be used to automatically select which variables are included in the model, i.e. those that are non-zero.

Julia Silge uses Lasso regression in her blog post looking at data from The Office, and I’d highly recommend having a look at it if you want more examples and explanation of how {tidymodels} can be used for Lasso regression. Read it at juliasilge.com/blog/Lasso-the-office.

We may also want to consider principal component analysis (PCA) which transforms the explanatory variables into a new set of artificial variables (or components). These new variables are chosen to explain the most variability in the original variables, and can also help to deal with correlations between explanatory variables in the data.

In this blog post, let’s compare using Lasso logistic regression on the raw data and Lasso logistic regression on the principal components. Before a model is fitted to the data, we need to split the data into a training set and a test set. This allows us to perform model selection using the training data without biasing the evaluation on the test set.

Show code: split data into train and test

| 123 | murmurs\_split <- initial\_split(all\_model\_data, strata = murmur)murmurs\_train <- training(murmurs\_split)murmurs\_test <- testing(murmurs\_split) |

Within {tidymodels}, we can create what’s called a recipe: a description of the steps to be applied to a data set in order to prepare it for data analysis. Here, we’ll have two recipes: one that simply normalises the raw data, and a second that normalises the data and computes the principal components.

Show code: make a recipe

| 1 2 3 4 5 6 7 8 9101112 | murmurs\_recipe <- recipe(murmur ~ ., data = murmurs\_train) |> step\_normalize(all\_numeric(), -all\_outcomes())murmurs\_pca\_recipe <- recipe(murmur ~ ., data = murmurs\_train) |> step\_normalize(all\_numeric(), -all\_outcomes()) |> step\_pca(all\_numeric())wf <- workflow() |> add\_recipe(murmurs\_recipe) wf\_pca <- workflow() |> add\_recipe(murmurs\_pca\_recipe) |

The next step in fitting a Lasso regression model is dealing with $\lambda$ (a hyperparameter). This isn’t something like the regression coefficients which can be optimally computed based on the data. Instead, we need to try lots of different values of $\lambda$ and pick the one that performs best. A (potentially) different value of $\lambda$ should be selected for the model using the principal components as explanatory variables.

Show code: tune $\lambda$

| 1 2 3 4 5 6 7 8 9101112131415161718 | set.seed(1234)murmurs\_boot <- bootstraps(murmurs\_train, strata = murmur)tune\_spec <- logistic\_reg(penalty = tune(), mixture = 1) |> set\_engine("glmnet")lambda\_grid <- grid\_regular(penalty(), levels = 50)set.seed(1234)Lasso\_grid <- tune\_grid( wf |> add\_model(tune\_spec), resamples = murmurs\_boot, grid = lambda\_grid)Lasso\_pca\_grid <- tune\_grid( wf\_pca |> add\_model(tune\_spec), resamples = murmurs\_boot, grid = lambda\_grid) |

So how do we know which value of $\lambda$ is best? For each value of $\lambda$ that we want to consider, we fit the model using that value, and then compute some performance metric. Here, let’s compare two different performance metrics: accuracy and ROC-AUC.

Accuracy: the proportion of the data that are predicted correctly.
ROC-AUC: a metric that computes the area under the ROC curve (which compares specificity and sensitivity). A higher value of ROC-AUC indicates better performance.

Show code: plot performace for different $\lambda$

| 1 2 3 4 5 6 7 8 91011121314151617181920212223242526272829303132333435363738394041424344454647 | p1 <- Lasso\_grid |> collect\_metrics() |> ggplot(mapping = aes(x = penalty)) + geom\_ribbon(mapping = aes( ymin = mean - std\_err, ymax = mean + std\_err ), fill = nr\_mid, alpha = 0.3 ) + geom\_line(mapping = aes(y = mean), linewidth = 1, colour = nr\_mid) + facet\_wrap(~.metric, scales = "free", nrow = 1) + scale\_x\_log10() + labs(x = "Penalty", y = "Mean", title = "Model performance under different penalties", subtitle = "(without PCA)") + nrBrand::theme\_nr() + theme(axis.text.y = element\_text(margin = margin(r = 5)))p1p2 <- Lasso\_pca\_grid |> collect\_metrics() |> ggplot(mapping = aes(x = penalty)) + geom\_ribbon(mapping = aes( ymin = mean - std\_err, ymax = mean + std\_err ), fill = nr\_mid, alpha = 0.3 ) + geom\_line(mapping = aes(y = mean), linewidth = 1, colour = nr\_mid) + facet\_wrap(~.metric, scales = "free", nrow = 1) + scale\_x\_log10() + labs(x = "Penalty", y = "Mean", title = "", subtitle = "(with PCA)") + nrBrand::theme\_nr() + theme(axis.text.y = element\_text(margin = margin(r = 5)))p2p1 + p2 + plot\_layout(ncol = 1) |

Unfortunately, here our two performance indicators don’t agree on the best value of $\lambda$. ROC-AUC is usually more robust when the data is imbalanced - when there are a lot more examples of one of the classes in the data than the other. Here, there are a lot more examples of subjects without heart murmurs, compared to those with (464 no murmur present, compared to 119 present). So ROC-AUC is probably more reliable here - and we’ll stick with it for the rest of the analysis here.

We can then use this best value of $\lambda$ in our final model.

Show code: fit final model

| 1 2 3 4 5 6 7 8 9101112131415 | highest\_roc\_auc <- Lasso\_grid %>% select\_best("roc\_auc")highest\_pca\_roc\_auc <- Lasso\_pca\_grid %>% select\_best("roc\_auc")final\_Lasso <- finalize\_workflow( wf %>% add\_model(tune\_spec), highest\_roc\_auc)final\_pca\_Lasso <- finalize\_workflow( wf\_pca %>% add\_model(tune\_spec), highest\_pca\_roc\_auc) |

Let’s look at which explanatory variables the Lasso regression has selected as important. I’ll only look at the model fitted to the raw data, as the PCA data can’t really be interpreted in the same way.

Show code: plot predictors

| 1 2 3 4 5 6 7 8 910111213141516171819 | final\_Lasso %>% fit(murmurs\_train) %>% extract\_fit\_parsnip() %>% vip::vi(lambda = highest\_roc\_auc$penalty) %>% mutate( Importance = abs(Importance), Variable = fct\_reorder(Variable, Importance) ) %>% ggplot(mapping = aes(x = Importance, y = Variable, fill = Sign)) + geom\_col(alpha = 0.6) + scale\_x\_continuous(expand = c(0, 0)) + scale\_fill\_manual(values = c(nr\_contrast, nr\_mid), labels = c("Negative influence", "Positive influence")) + labs(y = NULL, title = "Most important time series features for prediction") + nrBrand::theme\_nr() + theme(legend.position = c(0.7, 0.15), legend.title = element\_blank(), plot.title.position = "plot") |

Here, the top two most important predictors are both first autocorrelation coefficients for two of the different recording locations. Each have a positive impact on prediction - meaning that an observation of a higher first autocorrelation coefficient is more likely to result in a positive prediction for the presence of a heart murmur - which agrees with what was observed in the earlier boxplots.

We can then finally fit our final model to the test data to evaluate how well it’s working.

Show code: fit to test data

| 1 2 3 4 5 6 7 8 91011 | last\_fit( final\_Lasso, murmurs\_split) %>% collect\_metrics()last\_fit( final\_pca\_Lasso, murmurs\_split) %>% collect\_metrics() |

Confusion matrices are a common way of initially evaluating whether a classification model has performed well. It outlines the number of:

True positives (TP): number of murmurs correctly classified
True negatives (TN): number of non-murmurs correctly classified
False positives (FP): number of non-murmurs incorrectly classified as murmurs
False negatives (FN): number of murmurs incorrectly classified as non-murmurs

Sometimes we’ll care about these four things equally, and sometimes some will be more important than others. Here, it’s probably most important to think about false negatives - incorrectly classifying a patient with a heart murmur as not having one is likely to have more significant consequences. Metrics such as accuracy, specificity, and sensitivity can be computed from the confusion matrix if desired.

Show code: plot confusion matrix

| 1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334353637 | p1 = last\_fit( final\_Lasso, murmurs\_split) |> pull(.predictions) |> as.data.frame() |> as\_tibble() |> conf\_mat(murmur, .pred\_class) |> autoplot(type = "heatmap") + scale\_fill\_gradient(low = nr\_light, high = nr\_mid) + labs(title = "Confusion matrix of predictions", subtitle = "(without PCA)") + nrBrand::theme\_nr() + coord\_cartesian(expand = F) + theme(legend.position = "none", panel.grid = element\_blank())p1p2 = last\_fit( final\_pca\_Lasso, murmurs\_split) |> pull(.predictions) |> as.data.frame() |> as\_tibble() |> conf\_mat(murmur, .pred\_class) |> autoplot(type = "heatmap") + scale\_fill\_gradient(low = nr\_light, high = nr\_mid) + labs(title = "", subtitle = "(with PCA)") + nrBrand::theme\_nr() + coord\_cartesian(expand = F) + theme(legend.position = "none", panel.grid = element\_blank())p2p1 + p2 + plot\_layout(ncol = 2) |

Overall results Well, that didn’t work quite so well… This isn’t ideal, but we can still draw some conclusions. And also let’s normalise sharing the results that don’t always tell us what we want!

Both approaches have an overall accuracy of 0.791, which isn’t terrible. However, fitting the model to the principal components performs slightly better in terms of ROC-AUC: 0.529 (without PCA) vs 0.535 (with PCA). In either case, this performance is fairly poor, since 0.5 is essentially a baseline for randomly guessing. In both cases, the false negatives are high (top right corner) - which is exactly what we don’t want.
The autocorrelation coefficients are consistently found to be among important variables (of those considered), suggesting that this may be something to look into further.
It’s easier to predict whether some doesn’t have a heart murmur, than to predict if someone does. This is partly due to the fact that most people don’t have a heart murmur so if you had to guess if someone did or didn’t have a heart murmur, you’d probably guess they didn’t. If we reconsider the boxplot, we might see a second reason. For some recording locations, the first autocorrelation coefficients of the murmur present data lie within the range of the first autocorrelation coefficients of the murmur absent data - the more unusual autocorrelation coefficients values actually belong those without heart murmurs. Perhaps this is just a larger range due to the larger number of patients without heart murmurs, or perhaps it’s something worth delving into a bit deeper.

There may or may not be a part two to this blog, where I’ll look at some alternative approaches that could help improve the performance:

More complex machine learning models.
Considering a wider range of time series features. It’s very possible that the features considered here just aren’t sufficiently different to tell the two groups apart.
Dealing with the imbalanced nature of the data.

References Oliveira, J., Renna, F., Costa, P., Nogueira, M., Oliveira, A. C., Elola, A., Ferreira, C., Jorge, A., Bahrami Rad, A., Reyna, M., Sameni, R., Clifford, G., & Coimbra, M. (2022). The CirCor DigiScope Phonocardiogram Dataset (version 1.0.3). PhysioNet. doi.org/10.13026/tshs-mw03.

J. H. Oliveira, F. Renna, P. Costa, D. Nogueira, C. Oliveira, C. Ferreira, A. Jorge, S. Mattos, T. Hatem, T. Tavares, A. Elola, A. Rad, R. Sameni, G. D. Clifford, & M. T. Coimbra (2021). The CirCor DigiScope Dataset: From Murmur Detection to Murmur Classification. IEEE Journal of Biomedical and Health Informatics. doi.org/10.1109/JBHI.2021.3137048.

Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P. C., Mark, R., Mietus, J. E., Moody, B., Peng, C. K., & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation [Online]. 101 (23), pp. e215–e220.

Tibshirani, R. Regression Shrinkage and Selection via the Lasso (1996). Journal of the Royal Statistical Society. Series B (Methodological). Vol. 58, No. 1, pp. 267-288. www.jstor.org/stable/2346178.

To leave a comment for the author, please follow the link and comment on their blog: R on Nicola Rennie.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Detecting heart murmurs from time series data in R

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

This is part 2 of an ongoing series on why you should use R. Futureblogs will be linked here as they are released.

Part 1: Why should I use R: The Excel R Data Wrangling comparison:Part1

Why create plots in R and not Excel? To a programmer this may seem likea very obvious question, but it is still a common question asked byExcel users — If you have a data set, could you select it, hit a coupleof buttons and generate plots? This is one of the trickiest questions toanswer, especially if you have limited Excel experience as many new agedata scientists do. Hopefully, some of the reasons below will encourageyou to make the switch from Excel to R.

ReproducibilityHow do you view the code used to generate the Excel graph? Are you ableto tell exactly whats going on? Are you able to control and modify allof the aesthetics of the plot, such as changing the length of the axisticks, or changing the font? If yes, are you able to share your workwith a colleague and have them easily replicate your plot without youtelling them where to click and which modification should be applied?

With R all of these things are possible. You automatically have all thecode visible in the form of scripts. Reading and understanding the codeis possible because of its easy to read syntax, which allows you totrack what the code is doing without having to be concerned about anyhidden functions or modifications happening in the background.

Do you require help building a Shiny app? Would you like someone to take over the maintenance burden?If so, check outour Shiny and Dash services.

Understanding changesIn Excel it is challenging to eye-ball which changes have been made to agraph, especially if these were minor changes. With R (and some easy touse version control systems), you can see exactly which files werechanged. Also, in Excel, a user would usually draw a graph on a singleExcel document, and if the same graph is required on a different dataset, it is common to copy-and-paste a bunch of manipulations andconfigurations to another document. Such repeated human interaction isprone to introducing errors, as well as consuming a large amount oftime. With R we can avoid this by creating functions, which can be usedto run the same code on different data sets simply by changing theinput, thereby producing reliable outputs and saving us a lot of time.

ExtensibilityYes, Excel has a wide range of basic graphics available, but R has a lotmore. Excel has been around for a while, so it has some decent toolsthat have been developed over the years. R, however, is open source, andtherefore extensions are widely available – it’s even fairly easy tomake your own. R also has thousands of libraries that can be used toeasily produce graphics without all the pre-graph work to create somereally crafty stuff. With that being said, Excel is perfectly sufficientwhen creating basic, simple, straight forward plots. But what if we’renot looking to be basic?

The simplicity of RThe package {ggplot2} is a plottingpackage in R that provides us with commands to create complex plots. R’scommand line interface let’s you quickly select x- and y-axis labels,colour by variables, modify grid lines and much more. Each item is addedin a new layer, which allows us to add in and remove graph elementswithout affecting the rest of the plot. Interested in changing thecolour gradient/scale of your plot? No problem, just use a packagecalled{RcolourBrewer},which helps you select sensible colour schemes for your plots.Interested in changing the title of your plot? Simply add a layer calledggtitle – and so much more.

The comparisonLet’s create some simple plots in Excel and then create a similar plotin R using the {ggplot2} functions. Hopefully, by the end of this post,we’ll have motivated you to switch to R. Now, let’s get started byloading the data and packages. The data set that we’ve used below isdata from a selection of movies, and is comprised of five columns:country, year, highest profit gained per movie, number of moviesproduced and number of employees on set during production.

library("ggplot2") # For plottinglibrary("viridis") # Provides a range of colour paletteslibrary("readr") # For loading data library("tidyverse") # For data wranglingmovies\_data <- read\_csv("blog\_data.csv") Let’s start by creating a scatter plot, in which we compare the numberof employees present in the different countries within each year.

Scatter PlotExcelThe scatter plot generated in Excel was simple to create, but everythinghad to be done manually: selecting the data and the variables for the x-and y-axis and then selecting the type of plot. I was also required tomanually change the axes titles. If we were interested in changing thegrid lines, this would have to be done manually too. Looking at thisplot, is this something that you are able to easily recreate? Would youknow where to point and click to generate this visualisation?

RHere we created a similar plot in R using the {ggplot2} functions.Because the code is visible we can easily recreate the plot above, butalso, we are able to conveniently see which functions and aestheticswere applied to our plot.

ggplot(data = movies\_data, aes(x = Year, y = no\_employees)) + geom\_point(aes(colour = Country)) + labs(x = "Years", y = "Number of employees", colour = "Country") + theme\_bw() Theming system in {ggplot2}Theme arguments specify the non-data features that you can control. Forexample, the axis.text argument controls the appearance of the axistext such as the font size, colour and face of text. The axis.ticks.xcontrols the ticks on the x-axis and so on. The theme() functionallows you to override the default theme elements, liketheme(plot.title = element_text(colour = "red")). Completethemes, liketheme_bw(), set all of the theme elements to values designed to worktogether.

We can take this plot even further. Let’s say we were interested increating the same plot as above, but with each country having its ownplotting panel within the same visualisation. We can use the facetfunction from the {ggplot2} package:

ggplot(data = movies\_data, aes(x = Year, y = no\_employees)) + geom\_point() + facet\_wrap(~Country, ncol = 4) + labs(x = "Years", y = "Number of employees") + theme\_bw() + theme(axis.text.x = element\_text(angle = 45, vjust = 1, hjust = 1)) We have also utilised the axis.text.x element to adjust the angle andposition of the x-axis labels to ensure that they are legible. Are youable to create this in Excel without copying and pasting the graphs? Ifso please do show us how you were able to do this.

Now, let’s proceed to create a histogram using Excel and R. Looking atthe theme() function alone, we can see that R has a lot more featuresavailable that we are able to modify, such as axes text, fonts, legendsize and grid lines. As a data enthusiast, which graph looks moreaesthetically pleasing to you?

Histogram PlotExcelThe histogram generated below was a bit more time consuming. Firstly, wehad to change the size of the bars in a normal bar graph in order togenerate a histogram. The colours of each column had to manually beselected and applied. Adding a legend to this plot was also a manualprocess. Looking at this plot, is this something that you are able toeasily recreate?

Now, let’s generate a histogram using R and its {ggplot2} functions.

ROnce again, it is evident that we can easily control all of thevariables and aesthetics of the histogram plot generated using ggplot.Here we used a new function called thescale_fill_viridis()which is a function for {ggplot2} which allowed us to modify the coloursvisible on the histogram bars. We also used the theme_classic()function in R to create a classic looking plot with x- and y-axis linesand no gridlines. We also edited the size, colour and font of the texton the axes (axis.text).

ggplot(data = movies\_data, aes(x = Highest\_profit)) + geom\_histogram(aes(fill = Country)) + labs(x = "Yearly profit (in million dollars)", y = "Count") + scale\_fill\_viridis(discrete = T) + theme\_classic()+ labs(colour = "Country") + theme( axis.text = element\_text(size = 10, colour = "black", family = "serif") ) Now, let’s move on and generate our last plot.

Line PlotExcelThe line plot was the most complex plot to create. Firstly, whengenerating the line graph, it was evident that the data within the yearcolumn had to be rearranged in ascending order or it will put theearlier years after the later years. The line graph was also not able toplot more than one graph representing each country as a different lineas some countries did not have data for all the years. After a lot offrustration with Excel we attempted to create a very basic line plot inR.

RWith only three lines of code and very little frustration, we wereeasily able to recreate the line graph above in R.

ggplot(data = movies\_data, aes(x = Year, y = Number\_movies)) + geom\_line(aes(colour = Country)) + labs(x = "Years", y = "Number of movies produced") Now, let’s add some more aesthetics to our plot as we did for theprevious ones by changing the font size (axis.title and axis.text),changing the panel border (panel.border), as well as editing thelegend size (legend.key.size). Here we decided to use thetheme_dark() function in R to create a dark background, which iscommonly used to make thin coloured lines pop out.

ggplot(data = movies\_data, aes(x = Year, y = Number\_movies)) + geom\_line(aes(colour = Country)) + labs(x = "Years", y = "Number of movies produced") + labs(colour = "Country") + theme\_dark() + theme( panel.border = element\_rect(colour = "black", fill = NA, size = 2), axis.title = element\_text(size = 12, face = "bold", family = "Arial"), axis.text = element\_text(size = 10, colour = "black", family = "Arial"), legend.key.size = unit(0.50, "cm") ) When comparing R and Excel, it’s important to define the level ofinformation you are looking for. If you want to run basic statisticsquickly, Excel might be the better choice. If you are interested increating a very basic graph, Excel may be the better choice, due to itseasy point-and-click system. Before plotting a graph ask yourself; “Howdetailed does my visualisation need to be? Am I creating a plot for apublication or not? In Excel it is evident that we can easily select achunk of data and make a simple chart, however, when making morecomprehensive plots, using Excel can be extremely frustrating and timeconsuming. It all comes down to what you need your graphics to do. Forthose planning to publish large amounts of complicated data, spendingthe time in R to create impressive visual representations will certainlybe worth your time. It is also clear that R is not difficult, and givesyou the option to customise more than Excel.

R and Excel are beneficial in different ways. Excel starts off easier tolearn and is the go-to program when we are exposed to computers and someof us end up being stuck there. However, R is designed to bereproducible which is clearly of high importance. It’s not a question ofchoosing between R and Excel, but deciding which program to use fordifferent needs.

If you’re interested in learning how to create graphs using R, thenattend our Data visualisation withggplot2course.

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Why should I use R: The Excel R plotting comparison: Part 2

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

In this series of blog posts, we introduce you to the ten teams of this first cohort and what they will be working on in the program.

First, meet Paola Corrales from Argentina and Adam Sparks from Australia!

Pao Corrales – ChampionPao CorralesI’m a PhD student at the University of Buenos Aires. I study Atmospheric Sciences applying data assimilation techniques to improve short-term forecasts of severe events in Argentina. I love R and teaching science and programming using evidence-based practices centered on the students and their context.

As a Champion, I would like to submit the {agromet} R package to the peer review process and gain more experience as a developer. I hope that this package, designed for agrometeorological applications and thoroughly documented in Spanish, can be improved and used by many people.

I’m also looking forward to sharing experiences between Champions and learning from everyone during this process.

Adam Sparks – MentorAdam SparksI lead a research group for the Department of Primary Industries and Regional Development (DPIRD) of Western Australia in Perth, WA, Australia that conducts research in the areas of agricultural systems modelling and climate change, maintains a weather station network and provides biometry (statistical) support for the department.I’m a co-founder of Open Plant Pathology, and currently serve as an editor for rOpenSci.I have authored or co-authored more than seven R packages for CRAN and maintain one registered Julia package.

I was born in Indiana, USA and earned an undergraduate degree in agronomy from Purdue University, a graduate certificate in geographic information science (GIS) and my PhD in plant pathology on the epidemiology and ecology of plant pathogens from Kansas State University in the USA.Initially, when I moved to Australia I lived in Queensland for five years and more recently moved to Western Australia where I became a naturalized citizen, and I am now usually found in Perth, WA, Australia these days.

I’m excited for the Champions program because it’s a great way for me to contribute back to the broader R community and more specifically rOpenSci, which has been so supportive in my development as an R user and developer.

We’re excited to have both Pao and Adam be part of the Champions program and can’t wait to see what this partnership yields. Stay-tuned for the next series of introductions .

To leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: rOpenSci Champions Program Teams: Meet Pao Corrales and Adam Sparks

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Plankton may be small, but they are mighty. These organisms play a crucial role in maintaining the health of our oceans. Their diversity and abundance make them a primary source of food for many marine animals. And they produce around half of the world’s oxygen!

Understanding how plankton reacts to changes in the ecosystem is important to evaluate its ability to serve as food for other animals and its role in regulating cycles of carbon in the ocean. Appsilon has already been contributing to the scientific community’s efforts to better understand the Arctic zooplankton by leveraging machine learning.

From our Data for Good program, we partnered with ETH Zürich (ETHZ) and International Union for Conservation of Nature (IUCN) to leverage the power of R Shiny. With our combined domains, we aim to bridge the gap between science and policy, in a quest to turn data into effective conservation.

From Science to Impact: Using Interactive Dashboards to Visualize Plankton Diversity and Drive Policy Making

Evidence-Based Policy Change Through Interactive Data VisualizationsOceans are in trouble, but understanding plankton can help us protect them. Global marine ecosystems face significant environmental challenges, from overfishing to climate change. These threats do not discriminate, affecting the largest whales as well as the tiniest marine creatures.

Plankton are highly sensitive to changes in their environment. On the one hand, this makes it a powerful indicator of the health of the marine ecosystem as a whole. But on the flip side, it also means as our oceans change, it is at risk. Human-induced environmental changes threaten all components of biodiversity, from the physiology of organisms to the biological communities that provide ecosystem services. Climate change is causing a decline in plankton diversity, which could have serious consequences for ocean ecosystems, such as a lack of food for all marine organisms.

Plankton provides a wealth of information for bioprospecting endeavors, as it harbors genes useful for human industries. Despite the importance of global plankton diversity, it is notably absent from climate discussions and policies.

To better understand the current state of the marine ecosystem and the potential impact of climate change on plankton diversity, we turned to what we are best at – creating an interactive dashboard using R Shiny. This web-based tool allows policymakers and researchers to visualize a large dataset showing information on the marine ecosystem’s current state and projected future plankton diversity changes in different climate scenarios.

Plankton: The Superstars of the OceanPlankton are critical to the health of the world’s oceans serving as a quality food source for other animals, acting as bioindicators, reflecting changes in the marine environment and ecosystem, and much more.

While scientists have long suspected that this decline could harm ocean function, we don’t have a clear picture of how bad the situation could get. This lack of data makes it difficult for policymakers to make informed decisions about how to protect these important organisms. Without a better understanding of the vulnerability of plankton diversity, we risk losing an essential part of our planet’s delicate balance.

As such, plankton are increasingly being recognized as important for global marine biodiversity monitoring efforts.

Monitoring Marine Ecosystems with ML – A budding way to monitor shifting ecosystems.

Shaping Policy and Effective Conservation with R ShinyAs part of our Data4Good (D4G) initiative, we worked together with ETH Zürich (ETHZ) and International Union for Conservation of Nature (IUCN) to build a data visualization application showing information on the marine ecosystem’s current state and projected future plankton diversity changes in different climate scenarios. Such information is vital for preserving ocean health and creating effective and impactful laws on the international policy level.

The scientific studies done prior to building the visualization dashboard involved mapping the biogeography of 859 plankton species using observational data and novel machine-learning algorithms. Our partners defined ocean biomes, projected future changes in biodiversity, and identified hotspots of diversity change. Equipped with these data we built an interactive dashboard that visualizes the results on a global map, making it easy to see patterns and trends over time.

What makes our tool unique is that it is the first global plankton diversity and biogeography visualization as a function of time and emission scenario. This means that policymakers can use the tool to see how changes in emissions might impact the marine ecosystem in different parts of the world.

Mapmaker Shiny dashboard mockup

Interactive Shiny Dashboards: Bridging the Knowledge GapOne of the challenges of scientific research is translating it into meaningful policy changes. Often, policymakers lack quantitative evidence on the vulnerability of marine ecosystems, which makes it difficult to create effective and impactful laws on the international policy level.

The interactive dashboard helps to bridge that gap by providing policymakers with the data they need to make informed decisions about marine management. For example, policymakers can use the tool to identify areas of the ocean that are particularly vulnerable to changes in plankton diversity, and then target their conservation efforts accordingly. The tool can also help policymakers shape future policies and decision-making on what management tools and measures to use and where with maximum effectiveness.

To protect plankton diversity and its associated ecosystem services, decision-makers must have adequate information to plan and implement solutions at both global and local levels. Currently, future projections of marine plankton diversity changes remain uncertain, primarily due to the shortage and biased nature of global observational data. Developing new tools to visualize plankton diversity and ecosystem changes – like the MAPMAKER – offers a remedy to that.

More on Data4GoodInterested in using technology for positive impact? Get more information in your LinkedIn feed by signing up for the Data4Good Newsletter.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Visualizing Plankton Diversity and Climate Change: Impacting Policy with R Shiny

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I was cranking out a blog post for work earlier this week that shows off just how many integrations our platform has. I won’t blather about that content here, but as I was working on it, I really wanted to show off all the integrations.

A table seemed far too boring.

Several categorized unordered lists seemed too unwieldy.

Then, it dawned on me that I could make a visual representation of all the integration partners we have by thinking of the entire integrations’ ecosystem as a “universe” with each category being a “solar system” of that universe.

I’ve been leaning more heavily on javsascript for datavis these days, but I will always be more comfortable in {ggplot2}, so I headed to R to design a way to:

generate concentric orbits for “n” solar systems
randomize the placement of the planets in each ring
make a decent plot!

I worked with one of the most amazing designers on the planet (heh) to come up with some stellar (heh) styling for it, and this was the result:

I took the styling guidance and wrapped the messy, individual functions I had into a new {ggsolar} package, you can find at https://github.com/hrbrmstr/ggsolar.

It’s pretty raw, and I need to “geomify” it at some point, but it has

a function to generate the concentric circle polygons
another one to identify a random point on each ring
a naive plotting function, and
a theme cleanup function for decent output.

The default is to generate uniformly distributed concentric circles, but you have the option of supplying a custom radii vector to make it more “real”/“solar-sysetm-y”.

Here’s the general flow:

```

sol_planets is a built in vector of our system's planet namessol_orbits <- generate_orbits(sol_planets)set.seed(1323) # this produced a decent placements# naive but it works! You can specify your own point picker, too.placed_planets <- randomize_planet_positions(sol_orbits)# do the thing!plot_orbits( orbits = sol_orbits, planet_positions = placed_planets, label_planets = TRUE, label_family = hrbrthemes::font_es_bold) + hrbrthemes::theme_ipsum_es(grid="") + coord_equal() + labs( title = "Sol", caption = "Pluto is 100% a planet" ) + theme_enhance_solar()

`` Random SystemsI included agenerate_random_planets()` function that uses a hidden Markov model to create believable planetary names, so you can now make your own universe with {ggplot2}!

set.seed(42)(rando\_planets <- generate\_random\_planets(12))rando\_orbits <- generate\_orbits(rando\_planets)set.seed(123) # this produced decent placementsplaced\_planets <- randomize\_planet\_positions(rando\_orbits)plot\_orbits( orbits = rando\_orbits, planet\_positions = placed\_planets, label\_planets = TRUE, label\_family = hrbrthemes::font\_es\_bold) + hrbrthemes::theme\_ipsum\_es(grid="") + coord\_equal() + labs( title = "Rando System" ) + theme\_enhance\_solar() FINKick the tyres, use {gganimate} to make some animations, and be the ruler of your own universe! (We’re going to try to generate team “org charts” with these later in the week, so be creative, too!).

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Make “Solar System” Plots With {ggsolar}

View Details

[This article was first published on R, Econometrics, High Performance, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Last week collapse reached 1M downloads off CRAN. This is a milestone that was largely unforeseen for a package that started 4 years ago as a collection of functions intended to ease the R life of an economics master student. Today, collapse provides cutting-edge performance in many areas of statistical computing and data manipulation, and a breadth of statistical algorithms that can meet applied economists’ or statisticians’ demands on a programming environment like R. It is also the only programming framework in R that is effectively class-agnostic. Version 1.9.5 just released to CRAN this week, is also the first version that includes Single Instruction Multiple Data (SIMD) instructions for a limited set of operations. The future will see more efforts to take advantage of the capabilities of modern processors.

Meanwhile, the fastverse – a lightweight collection of C/C++-based R packages for statistical computing and data manipulation – is becoming more popular as an alternative to the tidyverse for data analysis and as backends to statistical packages developed for R – a trend that is needed.

It is on this positive occasion that I decided it was the right time to provide you with a personal note, or rather, some reflections, regarding the history, present state, and the future of collapse and the fastverse.

The Pastcollapse started in 2019 as a small package with only two functions: collap() – intended to facilitate the aggregation of mixed-type data in R, and qsu() – intended to facilitate summarizing panel data in R. Both were inspired by STATA’s collapse and (xt)summarize commands, and implemented with data.table as a backend. The package – called collapse alluding to the STATA command – would probably have stayed this way, had not unforeseen events affected my career plans.

Having completed a master’s in international economics in summer 2019, I was preparing for a two-year posting as an ODI Fellow in the Central Bank of Papua New Guinea in fall. However, things did not work out, I had difficulties getting a working visa and there were coordination issues with the Bank, so ODI decided to offer me a posting in the Ugandan Ministry of Finance, starting in January 2020. This gave me 4 months, September-December 2019, during which I ended up writing a new backend for collapse – in C++.

While collapse with data.table backed was never particularly slow, some of the underlying metaprogramming seemed arcane, especially because I wanted to utilize data.table’s GeForce optimizations which require the aggregation function to be recognizable in the call for data.table to internally replace it with an optimized version. But there were also statistical limitations. As an economist, I often employed sampling or trade weights in statistics, and in this respect, R was quite limited. There was also no straightforward way to aggregate categorical data, using, as I would have it, a (weighted) statistical mode function. I also felt R was lacking basic things in the time series domain – evidenced by the lengths I went to handle (irregular) trade panels. Finally, I felt limited by the division of software development around different classes in R. I found data.table useful for analytics, but the class too complex to behave in predictable ways. Thus I often ended up converting back to ‘data.frame’ or ‘tibble’ to use functions from a different package. Sometimes it would also have been practical to simply keep data as a vector or matrix – in linear-algebra-heavy programs – but I needed data.table to do something ‘by groups’. So in short, my workflow in R employed frequent object conversions, faced statistical limitations, and, in the case of early collapse’s data.table backend, also involved tedious metaprogramming.

The will for change pushed me to practically rethink the way statistics could be done in R. It required a framework that encompassed increased statistical complexity, including advanced statistical algorithms like (weighted) medians, quantiles, modes, support for (irregular) time series and panels etc., and enabling these operations to be vectored efficiently across many groups and columns without a limiting syntax that would again encourage metaprogramming. The framework would also need to be class-agnostic/support multiple R objects and classes, to easily integrate with different frameworks and reduce the need for object conversions. These considerations led to the creation of a comprehensive set of S3 generic grouped and weighted Fast Statistical Functions for vectors matrices and data.frame-like objects, initially programmed fully in C++. The functions natively supported R factors for grouping. To facilitate programming further, I created multivariate grouping (‘GRP’) objects that could be used to perform multiple statistical operations across the same groups without grouping overhead. With this backend and hand, it was easy to reimplement collap()1, and also provide a whole array of other useful functions, including dplyr-like functions like fgroup_by(), and time series functions that could be used ad-hoc but also supported plm’s indexed ‘pseries’ and ‘pdata.frame’ classes. collapse 1.0.0, released to CRAN on 19th March 2020 (me sitting in the Ugandan finance ministry) was already a substantial piece of statistical software offering cutting-edge performance (see the benchmarks in the introductory blog post).

To then cut a long story short, in the coming 3 years collapse became better, broader, and faster in multiple iterations. Additional speed came especially from rewriting central parts of the package in C – reimplementing some core algorithms in C rather than relying on the C++ standard library or Rcpp sugar – as well as introducing data transformation by reference and OpenMP multithreading. For example, fmode(), rewritten from C++ to C for v1.8.0 (May 2022), became about 3x faster in serial mode (grouped execution), with additional gains through multithreading across groups. Other noteworthy functionality was a modern reimplementation of ‘pseries’ and ‘pdata.frame’, through ‘indexed_frame’ and ‘indexed_series’ classes, fully fledged fsummarise(), fmutate() and across() functions enabling tidyverse-like programming with vectorization for Fast Statistical Functions in the backend, a set of functions facilitating memory efficient R programming and low-cost data object conversions, functions to effectively deal with (nested) lists of data objects – such as unlisting to data frame with unlist2d(), and additional descriptive statistical tools like qtab() and descr(). Particularly 2022 has seen two major updates: v1.7 and v1.8, and the bulk of development for 1.9 – released in January 2023. In improving collapse, I always took inspiration from other packages, most notably data.table, kit, dplyr, fixest, and R itself, to which I am highly indebted. The presentation of collapse at UseR 2022 in June 2022 marks another milestone of its establishment in the R community.

While using R and improving collapse, I became increasingly aware that I was not alone in the pursuit of making R faster and statistically more powerful. Apart from well-established packages like data.table, matrixStats, and fst, I noticed and started using many smaller packages like kit, roll, stringfish, qs, Rfast, coop, fastmap, fasttime, rrapply etc. aimed at improving particular aspects of R in a statistical or computational sense, often offering clean C or C++ implementations with few R-level dependencies. I saw a pattern of common traits and development efforts that were largely complimentary and needed encouragement. My impression at the time – largely unaltered today – was that such efforts were ignored by large parts of the R user community. One reason is of course the lack of visibility and coordination, compared to institutional stakeholders like Rstudio and H2O backing the tidyverse and data.table. Another consideration, it seemed to me, was that the tidyverse is particularly popular simply because there exists an R package and website called tidyverse which loads a set of packages that work well together, and thus alleviates users of the burden of searching CRAN and choosing their own collection of data manipulation packages.

Thus I decided in early 2021 to also create a meta package and GitHub repo called fastverse and use it to promote high-performance R packages with few dependencies. The first version 0.1.6 made it to CRAN in August 2021, attaching 6 core packages (data.table, collapse, matrixStats, kit, fst and magrittr), and allowing easy extension with additional packages using the fastverse_extend() function. With 7 total dependencies instead of 80, it was a considerably more lightweight and computationally powerful alternative to the tidyverse. The README of the GitHub repository has grown largely due to suggestions from the community and now lists many of the highest performing and (mostly) lightweight R packages. Over time I also introduced more useful functionality into the fastverse package, such as the ability to configure the environment and set of packages included using a .fastverse file, an option to install missing packages on the fly, and the fastverse_child() function to create wholly separate package verses. Observing my own frequent usage of data.table, collapse, kit, and magrittr in combination, I did a poll on Twitter in Fall 2022 suggesting the removal of matrixStats and fst from the core set of packages – which as accepted and implemented from v0.3.0 (November 2022). The fastverse package has thus become an extremely lightweight, customizable, and fast tidyverse alternative.

The PresentToday, both collapse and fastverse are well established in a part of the R community closer to econometrics and high-performance statistics. A growing number of econometric packages benefit from collapse as a computational backend, most notably the well-known plm package – which experienced order-of-magnitude performance gains. I am also developing dfms (first CRAN release October 2022), demonstrating that very efficient estimation of Dynamic Factor Models is possible in R combining collapse and RcppArmadillo. collapse is also powering various shiny apps across the web. I ended up creating a collapse-powered public macroeconomic data portal for Uganda, and later, at the Kiel Institute for the World Economy, for Africa at large.

So collapse has made it into production in my own work and the work of others. Core benefits in my experience are that it is lightweight to install on a server, has very low baseline function execution speeds (of a few microseconds instead of milliseconds with most other frameworks) making for speedy reaction times, scales very well to large data, and supports multiple R objects and modes of programming – reducing the need for metaprogramming. Since my own work and the work of others depends on it, API stability has always been important. collapse has not seen any major API changes in updates v1.7-v1.9, and currently no further API changes are planned. This lightweight and robust nature – characteristic of all core fastverse packages esp. data.table – stands in contrast to dplyr, who’s core API involving summarise(), mutate() and across() keeps changing to an extent that at some point in 2022 I removed unit tests of fsummarise() and fmutate() against the dplyr versions from CRAN.

Apart from development, it has also been very fun using the fastverse in the wild for some research projects. Lately, I’ve been working a lot with geospatial data, where the fastverse has enabled numerous interesting applications.

For example, I was interested in how the area of OSM buildings needs to be scaled using a power weight to correlate optimally with nightlights luminosity within a million cells of populated places in Sub-Saharan Africa. Having extracted around 12 million buildings from OSM, I programmed the following objective function and optimized it for power weights between 0.0001 and 5.

library(fastverse) library(microbenchmark)a <- abs(rnorm(12e6, 100, 100)) # Think of this as building areas in m^2g <- GRP(sample.int(1e6, 12e6, TRUE)) # Think of this as grid cellsy <- fsum(a^1.5, g, use.g.names = FALSE) + # Think of this as nightlights rnorm(g$N.groups, sd = 10000) length(y)## [1] 999989# Objective functioncor\_ay <- function(w, a, y) { aw\_agg = fsum(a^w, g, use.g.names = FALSE, na.rm = FALSE) cor(aw\_agg, y) }# Checking the speed of the objectivemicrobenchmark(cor\_ay(2, a, y))## Unit: milliseconds## expr min lq mean median uq max neval## cor\_ay(2, a, y) 30.42331 32.1136 35.02078 34.43118 36.75326 55.36505 100# Now the optimizationsystem.time(res <- optimise(cor\_ay, c(0.0001, 5), a, y, maximum = TRUE))## user system elapsed ## 1.375 0.051 1.427res## $maximum## [1] 1.501067## ## $objective## [1] 0.5792703 The speed of the objective due to GRP() and fsum()2 allowed further subdivision of buildings into different classes, and experimentation with finer spatial resolutions.

Another recent application involved finding the 100 nearest neighbors for each of around 100,000 cells (rows) in a rich geospatial dataset with about 50 variables (columns), and estimating a simple proximity-weighted linear regression of an outcome of interest y on a variable of interest z. Since computing a distance matrix on 100,000 rows up-front is infeasible memory-wise, I needed to go row-by-row. Here functions dapply(), fdist() and flm() from collapse, and topn() from kit became very handy.

```

Generating the dataX <- rnorm(5e6) # 100,000 x 50 matrix of characteristicsdim(X) <- c(1e5, 50)z <- rnorm(1e5) # Characteristic of interesty <- z + rnorm(1e5) # Outcome of interestXm <- t(forwardsolve(t(chol(cov(X))), t(X))) # Transform X to compute Mahalanobis distance (takes account of correlations)# Coefficients for a single rowcoef_i <- function(row_i) { mdist = fdist(Xm, row_i, nthreads = 2L) # Mahalanobis distance best_idx = topn(mdist, 101L, hasna = FALSE, decreasing = FALSE) # 100 closest points best_idx = best_idx[mdist[best_idx] > 0] # Removing the point itself (mdist = 0) weights = 1 / mdist[best_idx] # Distance weights flm(y[best_idx], z[best_idx], weights, add.icpt = TRUE) # Weighted lm coefficients}# Benchmarking a single executionmicrobenchmark(coef_i(Xm[1L, ]))## Unit: microseconds## expr min lq mean median uq max neval## coef_i(Xm[1L, ]) 927.051 965.591 1149.184 998.473 1031.068 3167.988 100# Compute coefficients for all rowssystem.time(coefs <- dapply(Xm, coef_i, MARGIN = 1))## user system elapsed ## 214.942 10.322 114.123head(coefs, 3)## coef_i1 coef_i2## [1,] 0.04329208 1.189331## [2,] 0.20741015 1.107963## [3,] 0.02860692 1.106427

`` Due to the efficiency offdist()andtopn(), a single call to the function takes around 1.2 milliseconds on the M1, giving a total execution time of around 120 seconds for 100,000 iterations of the program - one for each row ofXm`.

A final recent application involved creating geospatial GINI coefficients for South Africa using remotely sensed population and nightlights data. Since population data from WorldPop and Nightlights from Google Earth Engine are easily obtained from the web, I reproduce the exercise here in full.

```

Downloading 1km2 UN-Adjusted population data for South Africa from WorldPoppop_year <- function(y) sprintf("https://data.worldpop.org/GIS/Population/Global_2000_2020_1km_UNadj/%i/ZAF/zaf_ppp_%i_1km_Aggregated_UNadj.tif", y, y)for (y in 2014:2020) download.file(pop_year(y), mode = "wb", destfile = sprintf("data/WPOP_SA_1km_UNadj/zaf_ppp_%i_1km_Aggregated_UNadj.tif", y))

``` VIIRS Nightlights are available on Google Earth Engine on a monthly basis from 2014 to 2022. I extracted annual median composites for South Africa using instructions found here and saved them to my google drive3.

```

Reading population files using terra and creating a single data.tablepop_data <- list.files(pop_path) %>% set_names(substr(., 9, 12)) %>% lapply(function(x) paste0(pop_path, "/", x) %>% terra::rast() %>% terra::as.data.frame(xy = TRUE) %>% set_names(c("lon", "lat", "pop"))) %>% unlist2d("year", id.factor = TRUE, DT = TRUE) %>% fmutate(year = as.integer(levels(year))[year]) %T>% print(2)## year lon lat pop## ## 1: 2014 29.62792 -22.12958 38.21894## 2: 2014 29.63625 -22.12958 19.25175## --- ## 11420562: 2020 37.83625 -46.97958 0.00000## 11420563: 2020 37.84458 -46.97958 0.00000# Same for nightlightsnl_data <- list.files(nl_annual_path) %>% set_names(substr(., 1, 4)) %>% lapply(function(x) paste0(nl_annual_path, "/", x) %>% terra::rast() %>% terra::as.data.frame(xy = TRUE) %>% fsubset(avg_rad %!=% -9999)) %>% # Values outside land area of SA, coded using a mask in GEE unlist2d("year", id.factor = TRUE, DT = TRUE) %>% frename(x = lon, y = lat) %>% fmutate(year = as.integer(levels(year))[year]) %T>% print(2)## year lon lat avg_rad## ## 1: 2014 29.64583 -22.12500 0.08928293## 2: 2014 29.65000 -22.12500 0.04348474## --- ## 58722767: 2022 20.00833 -34.83333 0.37000000## 58722768: 2022 20.01250 -34.83333 0.41000000

``` Since nightlights are available up to 2022, but population only up to 2020, I did a crude cell-level population forecast for 2021 and 2022 based on 1.6 million linear models of cell-level population between 2014 and 2020.

```

Unset unneeded options for greater efficiencyset_collapse(na.rm = FALSE, sort = FALSE)system.time({# Forecasting population in 1.6 million grid cells based on linear regressionpop_forecast <- pop_data %>% fgroup_by(lat, lon) %>% fmutate(dm_year = fwithin(year)) %>% fsummarise(pop_2020 = flast(pop), beta = fsum(pop, dm_year) %/=% fsum(dm_year^2)) %>% fmutate(pop_2021 = pop_2020 + beta, pop_2022 = pop_2021 + beta, beta = NULL)})## user system elapsed ## 0.210 0.054 0.263head(pop_forecast)## lat lon pop_2020 pop_2021 pop_2022## ## 1: -22.12958 29.62792 52.35952 54.35387 56.34821## 2: -22.12958 29.63625 23.65122 24.44591 25.24060## 3: -22.12958 29.64458 33.29427 34.10418 34.91409## 4: -22.12958 29.65292 194.08760 216.78054 239.47348## 5: -22.12958 29.66125 123.92527 139.52940 155.13353## 6: -22.13792 29.56958 13.61020 13.39950 13.18880

`` The above expression is an optimized version of univariate linear regression:beta = cov(pop, year)/var(year) = sum(pop * dm_year) / sum(dm_year^2), wheredm_year = year - mean(year), that is fully vectorized across 1.6 million groups. Two further tricks are applied here:fsum()has an argument for sampling weights, which I utilize here instead of writingfsum(pop * dm_year), which would require materializing a vectorpop * dm_yearbefore summing. The division by reference (%/=%) saves another unneeded copy. The expression could also have been written in one line asfsummarise(beta = fsum(pop, W(year)) %/=% fsum(W(year)^2)), given that 3/4 of the computation time here is actually spent on grouping 11.4 million records bylatandlon`.

```

Appending population data with cell-level forecasts for 2021 and 2022pop_data_forecast <- rbind(pop_data, pop_forecast %>% fselect(-pop_2020) %>% rm_stub("pop_") %>% melt(1:2, variable.name = "year", value.name = "pop") %>% fmutate(year = as.integer(levels(year))[year]) %>% colorder(year))

``` As you may have noticed, the nightlights data has a higher resolution of around 464m than the population data at 1km resolution. To match the two datasets, I use a function that transforms the coordinates to a rectilinear grid of a certain size in km, using an Approximation to the Haversine Formula which rescales longitude coordinates based on the latitude coordinate (to have them approximately represent distance as at the equator). The coordinates are then divided by the grid size in km transformed to degrees at the equator, and the modulus from this division is removed. Afterward, half of the grid size is added again, reflecting the grid centroids. Finally, longitudes are rescaled back to their original extent using the same scale factor.

```

Transform coordinates to cell centroids of a rectilinear square grid of a certain size in kmsround_to_kms_fast <- function(lon, lat, km, round = TRUE, digits = 6) { degrees = km / (40075.017 / 360) # Gets the degree-distance of the kms at the equator if(round) div = round(degrees, digits) # Round to precision res_lat = TRA(lat, div, "-%%") %+=% (div/2) # This transforms the data to the grid centroid scale_lat = cos(res_lat * pi/180) # Approx. scale factor based on grid centroid res_lon = setTRA(lon * scale_lat, div, "-%%") %+=% (div/2) %/=% scale_lat return(list(lon = res_lon, lat = res_lat))}

``` The virtue of this approach, while appearing crude and not fully respecting the spherical earth model, is that it allows arbitrary grid sizes and transforms coordinates from different datasets in the same way. To determine the grid size, I take the largest 2-digit grid size that keeps the population cells unique, i.e. that largest number such that:

pop\_data %>% ftransform(round\_to\_kms\_fast(lon, lat, 0.63)) %>% fselect(year, lat, lon) %>% any\_duplicated()## [1] FALSE It turns out that 0.63km is the ideal grid size. I apply this to both datasets and merge them, aggregating nightlights using the mean4.

system.time({nl\_pop\_data <- pop\_data\_forecast %>% ftransform(round\_to\_kms\_fast(lon, lat, 0.63)) %>% merge(nl\_data %>% ftransform(round\_to\_kms\_fast(lon, lat, 0.63)) %>% fgroup\_by(year, lat, lon) %>% fmean(), by = .c(year, lat, lon))})## user system elapsed ## 8.195 1.380 4.280head(nl\_pop\_data, 2)## Key: <year, lat, lon>## year lat lon pop avg\_rad## <int> <num> <num> <num> <num>## 1: 2014 -34.82266 19.98068 2.140570 0.07518135## 2: 2014 -34.82266 19.98758 4.118959 0.09241374 Given the matched data, I define a function to compute the weighted GINI coefficient and an unweighted version for comparison.

```

Taken from Wikipedia: with small-sample correctiongini_wiki <- function(x) 1 + 2/(length(x)-1) * (sum(seq_along(x)sort(x)) / sum(x) - length(x))# No small-sample correctiongini_noss <- function(x) 2/length(x) sum(seq_along(x)sort(x)) / sum(x) - (length(x)+1)/length(x) Skp1qm1 <- function(k, q) (q-1)/2 (2(k+1) + q) + k + 1all.equal(Skp1qm1(30-1, 70+1), sum(30:100))## [1] TRUE# Weighted GINI (by default without correction)w_gini <- function(x, w, sscor = FALSE) { o = radixorder(x) w = w[o] x = x[o] sw = sum(w) csw = cumsum(w) sx = Skp1qm1(c(0, csw[-length(csw)]), w) if(sscor) return(1 + 2/(sw-1)(sum(sxx) / sum(xw) - sw)) 2/sw * sum(sxx) / sum(xw) - (sw+1)/sw}

``` This computes the population-weighted and unweighted GINI on a percentage scale for each year.

raw\_gini\_ts <- nl\_pop\_data %>% fsubset(pop > 0 & avg\_rad > 0) %>% fgroup\_by(year) %>% fsummarise(gini = gini\_noss(avg\_rad)*100, w\_gini = w\_gini(avg\_rad, pop)*100) %T>% print()## year gini w\_gini## <int> <num> <num>## 1: 2014 79.34750 55.51574## 2: 2015 91.35048 55.06437## 3: 2016 92.16993 54.75063## 4: 2017 55.96135 53.53097## 5: 2018 59.87219 52.84233## 6: 2019 64.43899 52.23766## 7: 2020 53.05498 51.15202## 8: 2021 52.19359 50.26020## 9: 2022 48.07294 49.69182# Plottinglibrary(ggplot2)raw\_gini\_ts %>% melt(1) %>% ggplot(aes(x = year, y = value, colour = variable)) + geom\_line() As evident from the plot, the population-weighted GINI is more smooth, which could be due to unpopulated areas exhibiting greater fluctuations in nightlights (such as fires or flares).

A final thing that we can do is calibrate the GINI to an official estimate. I use the africamonitor R API to get World Bank GINI estimates for South Africa.

WB\_GINI <- africamonitor::am\_data("ZAF", "SI\_POV\_GINI") %T>% print()## Key: <Date>## Date SI\_POV\_GINI## <Date> <num>## 1: 1993-01-01 59.3## 2: 2000-01-01 57.8## 3: 2005-01-01 64.8## 4: 2008-01-01 63.0## 5: 2010-01-01 63.4## 6: 2014-01-01 63.0 The last estimate in the series is from 2014, estimating a GINI of 63%. To bring the nightlights data in line with this estimate, I again use optimize() to determine an appropriate power weight:

np\_pop\_data\_pos\_14 <- nl\_pop\_data %>% fsubset(pop > 0 & avg\_rad > 0 & year == 2014, year, pop, avg\_rad) objective <- function(k) { nl\_gini = np\_pop\_data\_pos\_14 %$% w\_gini(avg\_rad^k, pop) * 100 abs(63 - nl\_gini)}res <- optimize(objective, c(0.0001, 5)) %T>% print()## $minimum## [1] 1.308973## ## $objective## [1] 0.0002598319 With the ideal weight determined, it is easy to obtain a final calibrated nightlights-based GINI series and use it to extend the World Bank estimate.

final\_gini\_ts <- nl\_pop\_data %>% fsubset(pop > 0 & avg\_rad > 0) %>% fgroup\_by(year) %>% fsummarise(nl\_gini = w\_gini(avg\_rad^res$minimum, pop)*100) %T>% print()## year nl\_gini## <int> <num>## 1: 2014 62.99974## 2: 2015 62.49832## 3: 2016 62.22387## 4: 2017 61.22247## 5: 2018 60.54190## 6: 2019 59.82656## 7: 2020 58.83987## 8: 2021 57.93322## 9: 2022 57.51001final\_gini\_ts %>% merge(WB\_GINI %>% fcompute(year = year(Date), wb\_gini = SI\_POV\_GINI), by = "year", all = TRUE) %>% melt("year", na.rm = TRUE) %>% ggplot(aes(x = year, y = value, colour = variable)) + geom\_line() + scale\_y\_continuous(limits = c(50, 70)) It should be noted, at this point, that this estimate and the declining trend it shows may be seriously misguided. Research by Galimberti et al. (2020) using the old DMSP OLS Nightlights series from 1992-2013 for 234 countries and territories, shows that nightlights based inequality measures much better resemble the cross-sectional variation in inequality between countries than the time series dimension within countries.

The example is nevertheless instrumental in showing how the fastverse, in various respects, facilitates and enables complex data science in R.

The FutureFuture development of collapse will see an increased use of SIMD instructions to further increase performance. The impact of such instructions - visible in frameworks like Apache arrow and Python’s polars (which is based on arrow) can be considerable. The following shows a benchmark computing the means of a matrix with 100 columns and 1 million rows using base R, collapse 1.9.0 (no SIMD), and collapse 1.9.5 (with SIMD).

library(collapse)library(microbenchmark)fmean19 <- collapsedev19::fmeanm <- rnorm(1e8)dim(m) <- c(1e6, 100) # matrix with 100 columns and 1 million rowsmicrobenchmark(colMeans(m), fmean19(m, na.rm = FALSE), fmean(m, na.rm = FALSE), fmean(m), # default is na.rm = TRUE, can be changed with set\_collapse() fmean19(m, nthreads = 4, na.rm = FALSE), fmean(m, nthreads = 4, na.rm = FALSE), fmean(m, nthreads = 4))## Unit: milliseconds## expr min lq mean median uq max neval## colMeans(m) 93.09308 97.52766 98.80975 97.99094 99.05563 190.67317 100## fmean19(m, na.rm = FALSE) 93.04056 97.47590 97.68101 98.05097 99.14058 100.48612 100## fmean(m, na.rm = FALSE) 12.75202 13.04181 14.05289 13.49043 13.81448 18.79206 100## fmean(m) 12.67806 13.02974 14.02059 13.49638 13.81009 18.72581 100## fmean19(m, nthreads = 4, na.rm = FALSE) 24.84251 25.20573 26.12640 25.52416 27.08612 28.71300 100## fmean(m, nthreads = 4, na.rm = FALSE) 13.07941 13.18853 13.96326 13.38853 13.68627 18.04652 100## fmean(m, nthreads = 4) 13.05813 13.18277 13.99704 13.33753 13.71505 19.18242 100 Despite these impressive results, I am somewhat doubtful that much of collapse will benefit from SIMD. The main reason is that SIMD is a low-level vectorization that can be used to speed up simple operations like addition, subtraction, division, and multiplication. This is especially effective with large amounts of adjacent data. But with many groups and little data in each group, serial programming can be just as efficient or even more efficient if it allows writing grouped operations in a non-nested way. So it depends on the data to groups ratio. My arrow benchmark from August 2022 showed just that: with few groups relative to the data size, arrow considerably outperforms collapse and data.table, but with more groups the latter catch up considerably and collapse took lead with many very small groups. More complex statistics algorithms like the median (involving selection) or mode / distinct value count (involving hashing), also cannot (to my knowledge) benefit from SIMD, and here collapse implementations are already pretty much state of the art.

Apart from additional vectorization, I am also considering a possible broadening of the package to support further data manipulation operations such as table joins. This may take a while for me to get into though, so I cannot promise an update including this in 2023. At this stage, I am very happy with the API, so no changes are planned here, and I will also try to keep collapse harmonious with other fastverse packages, in particular data.table and kit.

Most of all, I hope to see an increased breadth of statistical R packages using collapse as a backend, so that its potential for increasing the performance and complexity of statistical R packages is realized in the community. I have in the past assisted package maintainers interested in developing collapse backends and hope to increase further collaborations along these lines.

At last, I wish to thank all users that provided feedback and inspiration or promoted this software in the community, and more generally all people that encouraged, contributed to, and facilitated these projects. Much credit is also due to the CRAN maintainers who endured many of my mistakes and insisted on high standards, which made collapse better and more robust.

qsu() was implemented fully in C++.︎
Actually what is taking most time here is raising 12 million data points to a fractional power.︎
You can download the composites for South Africa from my drive. It actually took me a while to figure out how to properly extract the images from Earth Engine. You may find my answer here helpful if you want to export images for other countries.︎
Not the sum, as there could be a differing amount of nightlights observations for each cell.︎ To leave a comment for the author, please follow the link and comment on their blog: R, Econometrics, High Performance.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: collapse and the fastverse: Reflections on the Past, Present and Future

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

We’re delighted to announce that we’ve been named a Finalist in the British Data Awards2023.

The British Data Awards is an annual quest to discover and celebrate data success stories.Organisations taking part this year range from FTSE 100 heavyweights, public sectorpioneers, technology unicorns, fast-growing scale-ups, essential Not-For-Profits, andeverything in between.

A record 226 entries were received this year which means that competition to be named aFinalist proved to be particularly tough, so we’re especially pleased to be announced as aFinalist.

Jason Johnson, Co-Founder of Predatech and British Data Awards judge said: “Judging theBritish Data Awards this year wasn’t easy given the high standard of entries. All our Finalistsshould be incredibly proud of their data success stories and for helping to showcase the bestthat the world of data has to offer. I look forward to celebrating your achievements in May.”

Data comes in all shapes and sizes. It can often be difficult to know where to start. Whatever your problem, Jumping Rivers can help.

Our nominationsData for Good Consulting Initiative of the Year (sponsored by The Dot Collective)Jumping Rivers has been working on a project for the World Health Organisation Europe, streamlining and maintaining their COVID19 vaccination programme monitoring application. You may have read a little about this project in our recent blogs on offloading Shiny’s workload and working smarter, not harder with automated workflows. This is a great example of data being utilised to track and develop global initiatives. The maintenance that Jumping Rivers has performed on this app allows it to be quick, flexible and robust to changes in the data. The automation also allows the staff at the WHO/Europe to concentrate their efforts on important initiatives, rather than spending their time cleaning and managing data.

Rising Star of the YearJack Walton is a finalist for the Rising Star of the Year award! Jack is very community driven, leading data science meetups and contributing to open source projects and online support networks. He inhabits a unique space in the industry, between data science and data engineering, carving out a position for himself acting as an intermediary between the two areas, and allowing for greater collaboration across the company.

Quote from a company spokespersonThe British Data Awards 2023 will announce Winners across some 22 categories. A numberof Highly Commended awards will also be presented. This year, ‘Data for Good Initiative ofthe Year’ and ‘Innovation of the Year’ received the most entries categories.Other categories include ‘Data Leader of the Year’ and ‘Technology Company of the Year’,while new categories including ‘Climate Change Initiative of the Year’ were introduced tohelp showcase and celebrate the work of a diverse group of organisations.The British Data Awards 2023 judging panel included:

Roshan Awatar: Group Director, Data & Analytics at Sky
Lynne Bailey: Chief Data Officer at KPMG UK
Neil Carden: CEO at Forth Point I A Blend360 Company
Dr Sophie Carr: Founder at Bays Consulting
Caroline Carruthers: CEO at Carruthers and Jackson
Christina Finlay: Director of Data & Analytics, Nest
Roxane Heaton: Chief Information Officer at Macmillan Cancer Support
Natalie Jakomis: Director of Data & Analytics at Rightmove
Jason Johnson: Co-Founder at Predatech
Natasha Lauer: Head of Marketing at Soda
Dr Jo Watts: CEO & Founder at effini

Finalists will be celebrated, and Winners announced, at an awards ceremony taking place inLondon on the 11 th May.

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: We’re a British Data Awards 2023 Finalist

View Details

[This article was first published on DataGeeek, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The ongoing debate recently in Turkey is that the Turkish government has suppressed US Dollar/Turkish Lira exchange rates (USD/TRY) to prevent economic turmoil. Many authorities in the business, especially exporters, think that the USD/TRY parity should be in the range of 24-25 Turkish Lira.

To look through that, we will predict for the whole year and see whether the rates are in rational intervals. But first, we will model our data with bagged multivariate adaptive regression splines (MARS) via the earth package. The predictors of our regression model are the current account (account) and the producer price index (ppi) of Turkey.

library(tidyverse)library(tidymodels)library(lubridate)library(timetk)library(tsibble)library(modeltime)library(baguette)library(fable)library(plotly)library(ggtext)library(systemfonts)library(showtext)df <- read\_csv("https://raw.githubusercontent.com/mesdi/blog/main/usdtry\_mars.csv")#Turkiye Current Account (USD)df\_ca <- df %>% filter(type == "ca") %>% mutate(date = #removing parentheses and text within case\_when(str\_detect(date," \$.*\$") ~ str\_remove(date," \$.*\$"), TRUE ~ date)) %>% mutate( date = parse\_date(date, format = "%b %d, %Y") %>% #subtracting 2 months from release date floor\_date("month") %m-% months(2), value = str\_remove(value, "B") %>% as.numeric() ) %>% select(date, account = value)#Turkiye Producer Price Index (TRY)df\_ppi <- df %>% filter(type == "ppi") %>% mutate(date = #removing parentheses and text within case\_when(str\_detect(date," \$.*\$") ~ str\_remove(date," \$.*\$"), TRUE ~ date)) %>% mutate( date = parse\_date(date, format = "%b %d, %Y") %>% #subtracting 1 months from release date floor\_date("month") %m-% months(1), value = str\_remove(value, "B") %>% as.numeric() ) %>% select(date, ppi = value)#USD/TRY - US Dollar Turkish Liradf\_usdtry <- df %>% filter(type == "usdtry") %>% mutate( date = parse\_date(date, format = "%m/%d/%Y"), value = as.numeric(value) ) %>% select(date, usdtry = value)#Merging all the datasetsdf\_tidy <- df\_usdtry %>% left\_join(df\_ppi, by = "date") %>% left\_join(df\_ca, by = "date") %>% na.omit() Now that we have created our dataset, we can start modeling. Because the variables are in different scales, we normalize all of them. And we have to convert the categorical variables to numeric ones due to package/engine necessity.

```

Splitting the data for train and test datasplits <- df_tidy %>% time_series_split(assess = "1 year", cumulative = TRUE, date_var = date)df_train <- training(splits)df_test <- testing(splits)#Add time series signaturedf_rec_ts <- recipe(usdtry ~ ., data = df_train) %>% step_timeseries_signature(date) #Preprocessingdf_rec_tidy <- df_rec_ts %>% step_rm(date) %>% step_rm(contains("iso"), contains("minute"), contains("hour"), contains("am.pm"), contains("xts")) %>% step_zv(all_numeric_predictors()) %>% step_normalize(all_numeric_predictors()) %>% step_dummy(contains("lbl"), one_hot = TRUE)

``` We do hyperparameter tuning in order to find the optimal model for the data. As seen below, the degree of interaction is 2, which means there is an interaction term, and it exercises the backward pruning method.

```

Processed data for determining the number of model termsdf_proc <- df_rec_tidy %>% prep() %>% bake(new_data = NULL)#Model specificationmars_spec <- bag_mars(num_terms = min(200, max(20, 2 * ncol(df_proc))) + 1, prod_degree = tune(),#marks for tuning prune_method = tune()) %>% set_engine("earth") %>% set_mode("regression")#Workflowmars_wflow <- workflow() %>% add_model(mars_spec) %>% add_recipe(df_rec_tidy)#cross-validation for resamplesset.seed(12345)df_folds <- vfold_cv(df_train)#Building the hyperparameter data frame for tuninghyper_grid <- crossing( prod_degree = 1:2, prune_method = c("backward", "forward"))#Hyperparameter Tuningmars_reg_tune <- mars_wflow %>% tune_grid( df_folds, grid = hyper_grid, metrics = metric_set(rsq) )#Selecting the best parameters according to the r-squaremars_param_best <- select_best(mars_reg_tune, metric = "rsq") %>% select(-.config)mars_param_best# A tibble: 1 x 2# prod_degree prune_method# #1 2 backward

``` We can build our model with the best parameters we found earlier tuning process.

```

Final workflow with the object of best parametersfinal_df_wflow <- workflow() %>% add_model(mars_spec) %>% add_recipe(df_rec_tidy) %>% finalize_workflow(mars_param_best)#Model tableset.seed(2023)model_table <- modeltime_table( final_df_wflow %>% fit(df_train)) #Calibrationcalibration_table <- model_table %>% modeltime_calibrate(df_test)#Accuracycalibration_table %>% modeltime_accuracy() %>% select(rmse, rsq)# A tibble: 1 x 2# rmse rsq# #1 2.55 0.543

`` When we look at the accuracy results *coefficient of determination* (rsq`) looks low because the accuracy measures are calculated based on test data in the calibration phase, and our test data’s time range is small. But the RMSE looks fine, considering the target variable.

Before we start the modeling, we create our future data set to use in the regression model as a predictor. We will use the automated ARIMA function to do that. When we analyze the ARIMA models for ppi and account variables, we could see that they have an annual seasonality.

```

Future dataset for the next 12 monthsdate <- df_tidy %>% mutate(date = yearmonth(date)) %>% as_tsibble() %>% new_data(12) %>% as_tibble() %>% mutate(date = as.Date(date))ppi <- df_tidy %>% mutate(date = yearmonth(date)) %>% as_tsibble() %>% model(ARIMA(ppi)) %>% forecast(h = 12) %>% as_tibble() %>% select(ppi = .mean)account <- df_tidy %>% mutate(date = yearmonth(date)) %>% as_tsibble() %>% model(ARIMA(account)) %>% forecast(h = 12) %>% as_tibble() %>% select(account = .mean)df_future <- date %>% bind_cols(ppi) %>% bind_cols(account)

``` Finally, we can plot the actual values from 2016 to 2023 and predictions for the next 12 months.

```

Forecasting plot datasetset.seed(1983)df_plot <- calibration_table %>% modeltime_refit(df_tidy) %>% modeltime_forecast( new_data = df_future, actual_data = df_tidy %>% filter(year(date) > 2015) )#The observation datasetdf_actual <- df_plot %>% filter(.key == 'actual') #The prediction datasetdf_pred <- df_plot %>% filter(.key == 'prediction') #importing google fontsfont_add_google("Nunito", "nuni" )showtext_auto()#Hoverinfo texts text_actual <- glue::glue("{round(df_actual$.value, 2)} TRY\n{df_actual$.index}")text_pred <- glue::glue("{round(df_pred$.value, 2)} TRY\n{df_pred$.index}")df_plot %>% ggplot(aes(.index, .value)) + geom_area( data = . %>% filter(.key == "actual"), fill="#69b3a2", alpha=0.5 ) + geom_line( data = . %>% filter(.key == "actual"), color ="#69b3a2" ) + geom_point(aes(text = text_actual), data = . %>% filter(.key == "actual"), color ="#69b3a2" ) + geom_area( data = . %>% filter(.key == "prediction"), fill="#cf2765", alpha=0.5 ) + geom_line( data = . %>% filter(.key == "prediction"), color = "#cf2765" ) + geom_point(aes(text = text_pred), data = . %>% filter(.key == "prediction"), color = "#cf2765" ) + scale_y_continuous(limits = c(0,30))+ labs( x = "", y = "", title = "The USD/TRY rates from 2016 to 2023\nand the predictions for the next 12 months" ) + theme_minimal()+ theme( plot.title = element_markdown( hjust = 0.5, face = "bold" ), plot.background = element_rect(fill = "#f3f6f4", color = NA), panel.background = element_rect(fill = "#f3f6f4", color = NA) ) -> p #setting font family for ggplotly font <- list( family= "Nunito", size=20)#setting font family for hover labellabel <- list( font = list( family = "Nunito", size = 16 ) ) #converts ggplot2 object to plotly for interactive chartggplotly(p, tooltip = "text") %>% style(hoverlabel = label) %>% layout(font = font) %>% #Remove plotly buttons from the mode bar config(displayModeBar = FALSE)

``` When we hover over the points, we can see that the April data is quite similar to the expectations of the Turkish exporters and, the majority of economists. So, in the near future, the parity might explode, especially after the election is to be held on May 14.

To leave a comment for the author, please follow the link and comment on their blog: DataGeeek.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Predicting the Real USD/TRY Rates with MARS

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.When working with WooCommerce, you have access to a powerful and robust product that offers a variety of benefits. Not only is it free and supported by a large community, but it also has strong SEO capabilities and a vast selection of plugins to enhance functionality. Additionally, the WooCommerce admin tool is user-friendly and easy to navigate, requiring minimal time and effort to learn. In fact, most individuals can become proficient in its use in just one to two hours.

However, there is a drawback. The main concept of WooCommerce is that it’s like managing a storefront. Regardless of the size of your business, whether it’s small or large, it’s still just a storefront. This means that it lacks serious back-office capabilities, and the only way to manage your products is one-by-one, similar to how you would rearrange products in a storefront window.
If you’re managing an e-shop professionally, simply rearranging the products one by one as in a shop window won’t suffice. To stay ahead of the competition, you need to provide your customers (or boss) with more advanced capabilities and perform tasks quickly.
While it’s true that there are plugins available for almost anything you can think of, they often come at a cost. Moreover, they can negatively impact the speed of your store and lead to compatibility issues with each other. If you end up using more than 3-4 plugins, errors are bound to occur, making your workflow inefficient.

Over the years, I have faced several challenges in managing e-shops and I have finally decided to overcome them. After putting in a lot of effort, I have written over 90 functions in R (6.400+ lines of code) and utilized the WooCommerce API to develop a highly robust solution for these problems.
The central concept is to create a duplicate of the essential features of the e-shop such as categories, tags, attributes, products, customers, and orders inside R-Studio, utilize custom functions to perform filtering and CRUD operations through the REST API.

The solution is seamlessly integrated with WooCommerce through the REST API, and the source code is explained in detail, making it easy for you to modify the functions to suit your needs or even create new ones. I have incorporated multiple ways to achieve the same result, including GUI interfaces with the Tcl/Tk package, allowing you to customize your working environment.

My book, “WooCommerce Administrator with R,” is available on Amazon in both Kindle and paperback formats.

One use case included in the book demonstrates how easy it is to add new products, whether they are variable or simple. By creating an xlsx file with the necessary data (one line per product), along with variation attributes and category paths, you can use a single command to pass all the information to your e-shop, with variations created automatically.

Check the video in this link to see how it is done: Create new products with WooCommerce API in R.

Let’s see another example. You need to have a large sale on white women’s shoes, sizes 40 and 41 EU, as they are no longer in fashion and you have a lot of stock. You expect that smaller sizes will sell eventually. Act fast, customers, as the grand sale for white women’s shoes in sizes 40 and 41 EU will only last for two weeks!

filter = list(categories = "Shoes", variations = c("Color : White", "Shoe size : 40|41"))``filtered <- filter_products(filter = filter, search.variations = TRUE)

pr_filtered <- filtered[1] %>% as.data.frame() # parent products
var_filtered <- filtered[2] %>% as.data.frame() # filtered variations

schema_name =create_schema("name, v_Color, v_Shoe size, regular_price, sale_price, date_on_sale_to_gmt", template = F, echo = T)[[1]]

my_products <- populate_schema(schema_name, data = pr_filtered, var_df = var_filtered, values.from.parent = FALSE)

# adjust prices and offer date``my_products$sale_price = as.numeric(my_products$regular_price)*0.5 my_products$date_on_sale_to_gmt = paste0(Sys.Date()+14,"T23:59:59")

my_products <- keep.columns(my_products, "sale_price, date_on_sale_to_gmt") %>% filter(parent > 0) # we want to update only the variations

my_products <- modify_variations_batch (my_products, add.elements = F)

These commands, which may seem complex now, become simple to use once you have the source code and analysis. With these commands, you can complete your work in a matter of minutes, depending on the number of products you have, without ever needing to access the WP-Admin interface.

In my book, I also address the challenge of managing metadata. The functions I provide enable you to add additional fields to your products, customers, and orders. For instance, you can add information such as barcodes, product costs, discount policies, sales representatives, and more. If you have brick-and-mortar stores, you can even create orders in batches and include metadata about your retail customers, such as age group, sex, new/old customer status, and so on. All of this data can be extracted in a single data frame for further analysis. It’s a powerful tool that you’ll surely find useful!

I am confident that by learning to use the functions and basic directions provided in the book, you will see a significant improvement in your e-shop management capabilities. As an e-shop manager, this will allow you to work more efficiently and productively.
If you are a business owner, you will gain a better understanding of the potential of your e-shop and be able to hire the appropriate personnel to manage it effectively.
Furthermore, if you are interested in learning R, this book provides a great opportunity to do so while tackling real-life problems.
Lastly, for college students and business executives, acquiring the skills and knowledge provided in this book can be valuable for potential employers.

I highly recommend checking out my book on Amazon, as it provides a comprehensive solution to common issues faced by e-shop managers and business owners. Get started today and take your e-shop to the next level!

John Kamaras (www.jkamaras.com)

WooCommerce Administrator with R was first posted on April 10, 2023 at 4:48 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: WooCommerce Administrator with R

View Details

[This article was first published on R-Bloggers – Learning Machines, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
This post presents a real highlight: We will build and backtest a quantitative trading strategy in R with the help of OpenAI’s ChatGPT-4! If you want to get a glimpse into the future of trading system development, read on!

On this blog, I already provided a template to build your own trading system (see Backtest Trading Strategies Like a Real Quant) but back in the days we did everything by hand. Not anymore! AI to the rescue! Also on this blog, we very early on did some coding experiments with ChatGPT (see ChatGPT can Create Datasets, Program in R… and when it makes an Error it can Fix that too!).

So now, I will present to you a full session with ChatGPT-4, including the used prompts, in which we build and backtest a trading strategy together, so buckle up!

You can also watch the video for this post (in German):

I tried different approaches and different prompts. As always with Large Language Models (LLMs) like ChatGPT you get the best results when you are as specific as possible. I also found out that it is best to give it some seed idea where to start, otherwise you will most often only end up with standard stuff like moving average crossovers or something like that. Because I am a big fan of risk-based investing, as I call it, I started with this prompt and had the following exchange with the AI (outputs of the code after running it are copied directly into the respective sections, both packages can be found on CRAN, if you want to run the code yourself):

You are a renowned quantitative finance investor. You want to create a new trading system. The idea is to have a long/flat strategy on the S&P 500 based on the volatility as a stress and risk indicator. Do everything step by step and explain your reasoning behind it. Be extra careful to avoid data snooping bias, survivorship bias, look ahead bias and overfitting. Use R for backtesting, but don’t use the tidyverse. Load the necessary data with the quantmod package. The calculation of the performance and risk metrics should be done with the PerformanceAnalytics package. Give me the R code in one code window. I will give you the results of the tests and we will refine it together iteratively.

To create a long/flat trading system on the S&P 500 based on volatility, we’ll follow these steps:

Load necessary libraries and data
Calculate volatility as a risk indicator
Develop the trading strategy
Backtest the strategy
Evaluate performance and risk metrics

Here is the R code to accomplish these tasks:

```

Step 1: Load necessary libraries and datalibrary(quantmod)library(PerformanceAnalytics)start_date <- as.Date("2000-01-01")end_date <- as.Date("2021-12-31")symbol <- "^GSPC" # S&P 500 symbolgetSymbols(symbol, src = "yahoo", from = start_date, to = end_date)price_data <- Ad(get(symbol))# Step 2: Calculate volatility as a risk indicatorlookback_period <- 20volatility <- runSD(ROC(price_data, n = 1, type = "discrete"), n = lookback_period) * sqrt(252)# Step 3: Develop the trading strategythreshold <- 0.15signal <- ifelse(volatility < threshold, 1, 0)signal <- lag(signal, 1) # To avoid look-ahead biassignal[is.na(signal)] <- 0# Step 4: Backtest the strategyreturns <- ROC(price_data, n = 1, type = "discrete") * signalstrategy_returns <- na.omit(returns)# Step 5: Evaluate performance and risk metricscharts.PerformanceSummary(strategy_returns, main = "Long/Flat Strategy based on Volatility")table.AnnualizedReturns(strategy_returns)maxDrawdown(strategy_returns)

``` Please run this R code, provide the results of the backtest, and we will refine the strategy iteratively based on the results.

It says "> price_data <- Ad(get(symbol))Error in get(symbol) : object '^GSPC' not found"

Apologies for the confusion. It seems the variable name is causing issues. Let’s change the way we reference the data. Here’s the modified R code:

```

Step 1: Load necessary libraries and datalibrary(quantmod)library(PerformanceAnalytics)start_date <- as.Date("2000-01-01")end_date <- as.Date("2021-12-31")symbol <- "^GSPC" # S&P 500 symbolgetSymbols(symbol, src = "yahoo", from = start_date, to = end_date)price_data <- Ad(get(symbol))# Step 2: Calculate volatility as a risk indicatorlookback_period <- 20volatility <- runSD(ROC(price_data, n = 1, type = "discrete"), n = lookback_period) * sqrt(252)# Step 3: Develop the trading strategythreshold <- 0.15signal <- ifelse(volatility < threshold, 1, 0)signal <- lag(signal, 1) # To avoid look-ahead biassignal[is.na(signal)] <- 0# Step 4: Backtest the strategyreturns <- ROC(price_data, n = 1, type = "discrete") * signalstrategy_returns <- na.omit(returns)# Step 5: Evaluate performance and risk metricscharts.PerformanceSummary(strategy_returns, main = "Long/Flat Strategy based on Volatility")

```

table.AnnualizedReturns(strategy\_returns)## GSPC.Adjusted## Annualized Return 0.0334## Annualized Std Dev 0.0875## Annualized Sharpe (Rf=0%) 0.3821maxDrawdown(strategy\_returns)## [1] 0.2954578 Please run this updated R code, and provide the results of the backtest. We will refine the strategy iteratively based on the results.

Please modify the code so that one can compare the strategy with the underlying benchmark

Sure, I have modified the code to include the benchmark (S&P 500) performance for comparison:

```

Step 1: Load necessary libraries and datalibrary(quantmod)library(PerformanceAnalytics)start_date <- as.Date("2000-01-01")end_date <- as.Date("2021-12-31")symbol <- "^GSPC" # S&P 500 symbolgetSymbols(symbol, src = "yahoo", from = start_date, to = end_date, auto.assign = FALSE) -> sp500_dataprice_data <- Ad(sp500_data)# Step 2: Calculate volatility as a risk indicatorlookback_period <- 20volatility <- runSD(ROC(price_data, n = 1, type = "discrete"), n = lookback_period) * sqrt(252)# Step 3: Develop the trading strategythreshold <- 0.15signal <- ifelse(volatility < threshold, 1, 0)signal <- lag(signal, 1) # To avoid look-ahead biassignal[is.na(signal)] <- 0# Step 4: Backtest the strategyreturns <- ROC(price_data, n = 1, type = "discrete") * signalstrategy_returns <- na.omit(returns)# Calculate benchmark returnsbenchmark_returns <- ROC(price_data, n = 1, type = "discrete")benchmark_returns <- na.omit(benchmark_returns)# Step 5: Evaluate performance and risk metricscomparison <- cbind(strategy_returns, benchmark_returns)colnames(comparison) <- c("Strategy", "Benchmark")charts.PerformanceSummary(comparison, main = "Long/Flat Strategy vs S&P 500 Benchmark")

```

table.AnnualizedReturns(comparison)## Strategy Benchmark## Annualized Return 0.0334 0.0556## Annualized Std Dev 0.0875 0.1964## Annualized Sharpe (Rf=0%) 0.3821 0.2833maxDrawdown(comparison)## Strategy Benchmark## Worst Drawdown 0.2954578 0.5677539 This code will plot the performance of both the long/flat strategy based on volatility and the S&P 500 benchmark, as well as display the annualized returns and maximum drawdown for each. Please run this updated R code and provide the results of the backtest. We will refine the strategy iteratively based on the results.

Here are the metrics:

table.AnnualizedReturns(comparison)

Strategy Benchmark

Annualized Return 0.0334 0.0556

Annualized Std Dev 0.0875 0.1964

Annualized Sharpe (Rf=0%) 0.3821 0.2833

maxDrawdown(comparison)

Strategy Benchmark

Worst Drawdown 0.2954578 0.5677539

Based on the metrics provided, the long/flat strategy has a lower annualized return (3.34%) compared to the S&P 500 benchmark (5.56%). However, the long/flat strategy has a lower annualized standard deviation (8.75%) than the benchmark (19.64%), indicating that it is less volatile. The Sharpe ratio (with risk-free rate = 0%) is higher for the long/flat strategy (0.3821) than the benchmark (0.2833), suggesting that the strategy has better risk-adjusted returns. Additionally, the maximum drawdown of the long/flat strategy (29.55%) is significantly lower than that of the benchmark (56.78%).

Depending on your investment objectives and risk tolerance, the long/flat strategy might be an attractive option due to its lower volatility and better risk-adjusted returns, despite having a lower annualized return compared to the S&P 500 benchmark.

If you would like to further refine the strategy, you may consider adjusting the lookback period, the volatility threshold, or incorporating additional indicators to improve the strategy’s performance. Please let me know if you’d like to explore further improvements or adjustments to the strategy.

How did you arrive at the lookback period (20) and the volatility threshhold (0.15)?

The lookback period of 20 and the volatility threshold of 0.15 were arbitrary choices for the initial version of the strategy. In practice, these parameters should be optimized to achieve better performance while ensuring the strategy remains robust and doesn’t overfit the data.

One approach to optimizing these parameters is to perform a parameter sweep, in which you systematically vary the lookback period and the volatility threshold within reasonable ranges and evaluate the strategy’s performance for each combination. This can help you identify the parameter combination that yields the best performance.

However, it’s important to avoid overfitting, which occurs when the strategy is tailored too closely to the historical data, leading to poor performance when applied to unseen data. To mitigate overfitting, you can use techniques like out-of-sample testing and cross-validation.

What could be the economic reason that this strategy outperforms the benchmark on a risk-adjusted basis?

The economic rationale behind the long/flat strategy outperforming the benchmark on a risk-adjusted basis can be attributed to the strategy’s ability to reduce exposure to market risk during periods of high volatility. High volatility is often associated with increased uncertainty, stress, and risk in financial markets. By going flat (i.e., not holding any position) during these periods, the strategy effectively manages risk and limits potential losses.

A few key points provide insights into why this approach works:

Market inefficiencies: Financial markets are not always perfectly efficient. During periods of high volatility, market participants may react emotionally, leading to overreactions or panic selling. This can create price distortions and increase the risk of holding equities. By reducing exposure during high volatility, the long/flat strategy can avoid some of these potential losses.
Risk aversion: When volatility is high, investors tend to become more risk-averse and prioritize preserving capital over seeking returns. This risk-averse behavior can lead to a preference for safer assets, causing a decline in equity prices. By going flat during these periods, the strategy aligns with the risk-averse behavior of market participants.
Market turbulence: High volatility often indicates market turbulence or significant events (e.g., geopolitical risks, economic crises, or natural disasters) that can negatively impact the overall market or specific industries. By limiting exposure during periods of high volatility, the long/flat strategy helps mitigate the potential negative impact of such events.
Diversification benefits: A long/flat strategy can provide diversification benefits to an investment portfolio. The strategy may exhibit low correlation with other assets or strategies, thereby enhancing the overall risk-adjusted performance of the portfolio.

By managing risk and reducing exposure during high-volatility periods, the long/flat strategy seeks to deliver better risk-adjusted returns compared to the benchmark. However, it’s important to keep in mind that past performance does not guarantee future results, and the strategy’s effectiveness may vary across different market conditions and time periods.

I could have gone on to further refine the strategy, optimize the parameters while avoiding overfitting, add more parameters, and so on, but this post is for illustrative purposes only. What I did though is to test the strategy out-of-sample for the year 2022 and Q1 2023. This is what I got:

table.AnnualizedReturns(comparison)## Strategy Benchmark## Annualized Return -0.0145 -0.1280## Annualized Std Dev 0.0261 0.2294## Annualized Sharpe (Rf=0%) -0.5574 -0.5579maxDrawdown(comparison)## Strategy Benchmark## Worst Drawdown 0.03350335 0.254251 I think the chart and the metrics speak for themselves! Quite impressive, don’t you think?

In the near future, it is conceivable that you can connect ChatGPT to R and the internet and everything would be done autonomously. There are already projects under way (e.g. “Auto-GPT”) that build the necessary infrastructure, so this will be only a matter of some short time horizon before we will have fully automated trading AIs in the markets (if they aren’t there already), constantly refining their strategies!

Please let me know your thoughts in the comments below!

DISCLAIMER
This post is written on an “as is” basis for educational purposes only, can contain errors, and comes without any warranty. The findings and interpretations are exclusively those of the author and are not endorsed by or affiliated with any third party.

In particular, this post provides no investment advice! No responsibility is taken whatsoever if you lose money.

(If you make any money though I would be happy if you would buy me a coffee… that is not too much to ask, is it? )

To leave a comment for the author, please follow the link and comment on their blog: R-Bloggers – Learning Machines.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Building and Backtesting a Volatility-based Trading Strategy with ChatGPT

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The official example WebR REPL is definitely cool and useful to get the feel for WebR. But, it is far from an ideal way to deal with it interactively, even as just a REPL.

As y’all know, I’ve been conducing numerous experiments with WebR and various web technologies. I started doing this for numerous reasons, one was to get folks excited about WebR and try to show there are endless possibilities for it (and hopefully avoid lock-in to prescribed views on how you should work with it). Another was to brush up on rusty web skills and have something fun to do during the continuing long aftermath of my spike protein invasion.

I started poking under the WebR covers this past weekend, and until there’s a more pyodide-like JS bridge on the R side of WebR, I decided to forego said spelunking. Instead, I began a dive into {plot2}, a really neat {ggplot2}-esque enhancement to base R plotting. While I could use any R-compatible IDE (there are many, btw), I wanted to do all the experiments in WebR-proper, since base plots work out of the box and the {ggplot2} ecosystem takes a bit of time to install. The tinkering began just fine, but it became a bit tedious doing browser refreshes (they’re automatic with Vite in dev mode) for small tweaks. There was no way I was using the official REPL given the lack of real interactivity in the console. And, I wanted to avoid keeping re-rendering Quarto documents, since that would have been as tedious as the Vite refreshes.

So, I decided to make an “IDE REPL” for WebR, so I could work with it like I would R in Sublime Text, VS Code, or RStudio. I mean, wouldn’t everyone?

You can check it out here, and the source is on GitHub.

I’m not going to take up much more time here, since it comes with some explanations out of the box, but I will reproduce the GH README for it at the end. I will present the structure of the project, here, to make it easier to build upon it (clone/fork away!).

I’m using Monaco, the editor that powers VS Code and the online GitHub editor. It has so many batteries included that it’s hard not to want to use it, even considering how much I despise Microsoft as a company. It is dead simple to use.

The entire project is in vanilla javascript, and there is no builder this time, since I wanted to make this as accessible to as many folks as possible.

This is the project structure:

├── boilerplate.js # text that appears in the source on first load or hard refresh├── completions.js # a decent number of R completions (I'll add more)├── index.css # core CSS├── index.html # HTML shell├── main.js # Main "app"├── resizers.js # We need to keep the panes sized properly├── r.js # Some WebR bits└── rlang.js # Language stuff for Microsoft's Monaco editor Rather than adorn the interface with silly buttons and baubles, I am putting functionality into the Monaco command palette.

Here’s what you’ve got with v0.1.0:

Auto-saves current source pane contents to local storage
R syntax highlighting
An oddly decent # of auto-completes
cmd-shift-p to bring up command palette
- WebR: Clear Local Storage — nuke local storage and replace with the default document
- WebR: Save Canvas as PNG — captain obvious
- WebR: Save Source Pane — captain obvious
- WebR: View WebR Environment Summary — captain obvious
- WebR: View WebR History — captain obvious
cmd-shift-i inserts |>
option-shift-minus inserts <-
watches for ?… and will open up a new tab for web help on whatev u searched for (XSS protected)
watches for broweURL(…) and will open up the URL in a new tab (XSS protected)
baked-in install.runiverse(pkg) which will try to install a pkg from R Universe. It is ON YOU to load the deps and ensure all deps and the pkg itself will work in WebR. You can use this tool I made to help you out.

FINApart from making the current functionality more robust/pretty, one big forthcoming advancement will be the ability to save/load the WebR workspace to local browser storage. What that will mean for you, is that you can go to an instance of the app, all source changes will automagicallly be saved/restored to the session between visits. Plus — if you’ve saved the workspace image — it will be auto-restored on the visit, leaving you to just have to re-install/load any necessary packages. This means you can get right back to “work”.

I’ll be adding the ability to load files from your local system and use {svglite} for graphics (Monaco has an amazing SVG viewer), and to actually work in the R Console area (either with some janky input box or janky xterm.js).

Kick the tyres, file bugs, feature enhancements, and PRs, and start playing more with WebR!

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introducing WebRIDEr: The WebR “IDE”-ish REPL You Didn’t Know You Needed

View Details

[This article was first published on R – Giga thoughts …, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.It is carnival time again as IPL 2023 is underway!! The new GooglyPlusPlus now includes AI/ML models for computing ball-by-ball Win Probability of matches and each individual player’s Win Probability Contribution (WPC). GooglyPlusPlus uses 2 ML models

Deep Learning (Tensorflow) – accuracy : 0.8584
Logistic Regression (glmnet-tidymodels) : 0.728

Besides, as before, GooglyPlusPlus will also include the usual near real-time analytics with the Shiny app being automatically updated with the previous day’s match data.

Note: The Win Probability Computation can also be done on a live feed of streaming data. Since, I don’t have access to live feeds, the app will show how Win Probability changed during the course of completed matches. For more details on Win Probability and Win Probability Contribution see my posts

Boosting Win Probability accuracy with player embeddings
GooglyPlusPlus: Win Probability using Deep Learning and player embeddings
GooglyPlusPlus: Computing T20 player’s Win Probability Contribution

GooglyPlusPlus has been also updated with all the latest T20 league’s match data. It includes data from BBL 2022, NTB 2022, CPL 2022, PSL 2023, ICC T20 2022 and now IPL 2023.

GooglyPlusPlus has the following functionality

Batsman tab: For detailed analysis of batsmen
Bowler tab: For detailed analysis of bowlers
Match tab: Analysis of individual matches, plot of Runs vs SR, Wickets vs ER in power play, middle and death overs, Win Probability Analysis of teams and Win Probability Contribution of players
Head-to-head tab: Detailed analysis of team-vs-team batting/bowling scorecard, batting, bowling performances, performances in power play, middle and death overs
Team performance tab: Analysis of team-vs-all other teams with batting /bowling scorecard, batting, bowling performances, performances in power play, middle and death overs
Optimisation tab: Allows one to pit batsmen vs bowlers and vice-versa. This tab also uses integer programming to optimise batting and bowling lineup
Batting analysis tab: Ranks batsmen using Runs or SR. Also plots performances of batsmen in power play, middle and death overs and plots them in a 4×4 grid
Bowling analysis tab: Ranks bowlers based on Wickets or ER. Also plots performances of bowlers in power play, middle and death overs and plots them in a 4×4 grid

Also note all these tabs and features are available for all T20 formats namely IPL, Intl. T20 (men, women), BBL, NTB, PSL, CPL, SSM.

Important note: It is possible, that at times, the Win Probability (Deep Learning) for some recent IPL matches will give an error. This is because I need to rebuild the models on a daily basis as the matches use player embeddings and there are new players. While I will definitely rebuild the models on weekends and whenever I find time, you may have to bear with this error occasionally.

Note: All charts are interactive, which means that you can hover, zoom-in, zoom-out, pan etc on the charts

The latest avatar of GooglyPlusPlus2023 is based on my R package yorkr with data from Cricsheet.

Check out the latest version of GooglyPlusPlus

Follow me on twitter for daily highlights @tvganesh_85

GooglyPlusPlus can analyse players, matches, teams, rank, compute win probability and much more.

Included below are some random analyses of IPL 2023 matches so far

A) Chennai Super Kings vs Gujarat Titans – 31 Mar 2023

GT won by 5 wickets ( 4 balls remaining)

a) Worm Wicket Chart

b) Ball-by-ball Win Probability (Logistic Regression) (side-by-side)

This model shows that CSK had the upper hand in the 2nd last over, before it changed to GT. More details on Win Probability and Win Probability Contribution in the posts given by the links above.

c) b) Ball-by-ball Win Probability (Logistic Regression) (overlapping)

Here the ball-by-ball win probability is overlapped. CSK and GT both had nearly the same probability of winning in the 2nd last over before GT edges CSK out

B) Punjab Kings vs Rajasthan Royals – 05 Apr 2023

This was a another closely fought match. PBKS won by 5 runs

a) Worm wicket chart

b) Batting partnerships

Shikhar Dhawan scored 86 runs

c) Ball-by-ball Win Probability using Deep Learning (overlapping)

PBKS was generally ahead in the win probability race

d) Batsman Win Probability Contribution

This plot shows how the different batsmen contributed to the Win Probability. We can see that Shikhar Dhawan has a highest win probability. He played a very sensible innings. Also it appears that there is no difference between Prabhsimran Singh and others, though he score 60 runs. This computation is based on when they come to bat and how the win probability changes when they get dismissed, as seen in the 2nd chart

C) Delhi Capitals vs Gujarat Titans – 4 Apr 2023

GT won by 6 wickets (11 balls remaining)

a) Worm wicket chart

b) Runs scored across 20 overs

c) Runs vs SR plot

d) Batting scorecard (Gujarat Titans)

e) Batsman Win Probability Contribution (Gujarat Titans)

Miller has a higher percentage in the Win Contribution than Sai Sudershan who held the innings together.Strange are the ways of the ML models!!

D) Sunrisers Hyderabad vs Lucknow Supergiants ( 7 Apr 2023)

LSG won by 5 wickets (24 balls left). SRH were bamboozled by the pitch while LSG was able to cruise along

a) Worm wicket chart

b) Wickets vs ER plot

c) Wickets across 20 overs

d) Ball-by-ball win probability using Deep Learning (overlapping)

e) Bowler Win Probability Contribution (LSG)

Bishnoi has a higher win probability contribution than Krunal, though he just took 1 wicket to Krunal’s 3 wickets. This is based on how the Win Probability changed at that point in the game.

The above set of plots are just a random sample.

Note: There are 8 tabs each for 9 T20 leagues (BBL, CPL, T20 (men), T20 (women), IPL, PSL, NTB, SSM, WBB). So there are a lot more detailed charts/analses.

Do take GooglyPlusPlus for a test drive!!!

Follow me on twitter @tvganesh_85 for daily highlights of previous day matches

Take a look at some of my other posts

Using Reinforcement Learning to solve Gridworld
Deep Learning from first principles in Python, R and Octave – Part 6
Big Data-4: Webserver log analysis with RDDs, Pyspark, SparkR and SparklyR
Experiments with deblurring using OpenCV
Singularity
Practical Machine Learning with R and Python – Part 6
Pitching yorkpy … short of good length to IPL – Part 1
Analyzing performances of cricketers using cricketr template
Cricpy takes guard for the Twenty20s
Simulating an Edge Shape in Android

To see all posts click Index of posts

To leave a comment for the author, please follow the link and comment on their blog: R – Giga thoughts ….

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: IPL 2023:GooglyPlusPlus now with by AI/ML models, near real-time analytics!

View Details

[This article was first published on Online College Math Teacher, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I am getting tired of these little pictorial numerical puzzles with the four equations, like one where three chickens equals 60, one chicken plus two plates of two eggs per plate equals 26, and so on, untilthe final equation is to evaluate some mathematical expression involving chickens, eggs, and bananas.The solution generally requires that you remember the PEMDAS (in the US, or BODMAS elsewhere) rules fororder of operations especially that multiplication takes precedence over addition, and also that you carefully count the number of eggs and number of bananas. I get 36.

OK, let me try to create a more interesting pictorial puzzle.

Mathematicians agree on the PEMDAS rules, although there are many situations that PEMDAS doesn’t handle. Perhaps the most common is the unary minus operator as in -32. It is unary because unlike subtractionthat has two operands, the unary operator only has one. I think mathematicians would like to see the unary operator as changing the sign of the argument, so that -32 equals -9, although some software,most notably Excel, merrily calculate this as +9.

I don’t believe there is a single authority for all the order of operations cases. For example, Excel, Google Search, and Wolfram Alpha do not always agree. I bet there are some pretty smart peoplein those companies.

Nowadays I am doingmy fun calculations in the R computer language, so for the remainder of this post I will require R asthe authority.

So here is my attempt at a more interesting problem, but remember, you have to use the order of operation precedence rules of R: (Let me add the link to the first item: https://www.facebook.com/watch/?v=10158293605695705 )

| |

Do you want to try it before I reveal the R code?

The R code is:

apple <- 1
banana <- 2
kiwi <- 3
lemon <- 4
peach <- banana + lemon
pear <- banana^banana^kiwi # 2^(2^3) = 256
pineapple <- (pear - banana) %% kiwi^2 * lemon # (254 %% 9) * 4 = 2 * 4 = 8
strawberry <- pineapple / peach * peach # 8; no obelus in R
kiwi <- c(lemon, pineapple, strawberry)
watermelon <- kiwi[kiwi == lemon | kiwi == pineapple & kiwi == strawberry]

watermelon <- (lemon V pineapple ∧ strawberry) # 4 V (8 & 8) = 4, 8

The ordering rules of R include:

Modular arithmetic is at the same level as multiplication: a mod b * c is (a mod b) *c
The obelus does not appear in R but is just a division symbol: a ÷ b * c = (a / b) * c
Repeated exponentiation goes right to left: a ^ b ^ c = a^(b^c); lots of disagreement outside R on this one
Logical AND preceds logical OR

To leave a comment for the author, please follow the link and comment on their blog: Online College Math Teacher.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A more interesting pictorial numerical puzzle

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntruductionReading in an Excel file with multiple sheets can be a daunting task, especially for users who are not familiar with the process. In this blog post, we will walk through a sample function that can be used to read in an Excel file with multiple sheets using the R programming language.

FunctionThe function we will be using is called excel_sheet_reader(). This function takes one argument: filename, which is the name of the Excel file we want to read in. This function, since it is using the {readxl} package will automatically read that data to a tibble.

ExampleHere is the function:

excel\_sheet\_reader <- function(filename) { sheets <- excel\_sheets(filename) x <- lapply(sheets, function(X) read\_excel(filename, sheet = X)) names(x) <- sheets x} The first thing the excel_sheet_reader() function does is to determine the names of all the sheets in the Excel file using the excel_sheets function from the readxl package. This function returns a character vector containing the names of all the sheets in the Excel file.

sheets <- excel\_sheets(filename) Next, the function uses the lapply function to loop through all the sheet names and read in each sheet using the read_excel() function, also from the readxl package. This function takes two arguments: filename, which is the name of the Excel file, and sheet, which is the name of the sheet we want to read in. The lapply function returns a list containing all the sheets.

x <- lapply(sheets, function(X) read\_excel(filename, sheet = X)) Finally, the function uses the names function to assign the sheet names to the list of sheets and returns the list.

names(x) <- sheetsx Now that we have explained the excel_sheet_reader() function, let’s use it to read in the iris and mtcars datasets.

library(healthyR)library(dplyr)library(writexl)library(readxl)iris |> named\_item\_list(Species) |> write\_xlsx(path = "iris.xlsx")mtcars |> named\_item\_list(cyl) |> write\_xlsx(path = "mtcars.xlsx")iris\_sheets <- excel\_sheet\_reader("iris.xlsx")mtcars\_sheets <- excel\_sheet\_reader("mtcars.xlsx") Now lets see the structure of each file.

iris\_sheets

$setosa# A tibble: 50 × 5 Sepal.Length Sepal.Width Petal.Length Petal.Width Species <dbl> <dbl> <dbl> <dbl> <chr> 1 5.1 3.5 1.4 0.2 setosa 2 4.9 3 1.4 0.2 setosa 3 4.7 3.2 1.3 0.2 setosa 4 4.6 3.1 1.5 0.2 setosa 5 5 3.6 1.4 0.2 setosa 6 5.4 3.9 1.7 0.4 setosa 7 4.6 3.4 1.4 0.3 setosa 8 5 3.4 1.5 0.2 setosa 9 4.4 2.9 1.4 0.2 setosa 10 4.9 3.1 1.5 0.1 setosa # ℹ 40 more rows$versicolor# A tibble: 50 × 5 Sepal.Length Sepal.Width Petal.Length Petal.Width Species <dbl> <dbl> <dbl> <dbl> <chr> 1 7 3.2 4.7 1.4 versicolor 2 6.4 3.2 4.5 1.5 versicolor 3 6.9 3.1 4.9 1.5 versicolor 4 5.5 2.3 4 1.3 versicolor 5 6.5 2.8 4.6 1.5 versicolor 6 5.7 2.8 4.5 1.3 versicolor 7 6.3 3.3 4.7 1.6 versicolor 8 4.9 2.4 3.3 1 versicolor 9 6.6 2.9 4.6 1.3 versicolor10 5.2 2.7 3.9 1.4 versicolor# ℹ 40 more rows$virginica# A tibble: 50 × 5 Sepal.Length Sepal.Width Petal.Length Petal.Width Species <dbl> <dbl> <dbl> <dbl> <chr> 1 6.3 3.3 6 2.5 virginica 2 5.8 2.7 5.1 1.9 virginica 3 7.1 3 5.9 2.1 virginica 4 6.3 2.9 5.6 1.8 virginica 5 6.5 3 5.8 2.2 virginica 6 7.6 3 6.6 2.1 virginica 7 4.9 2.5 4.5 1.7 virginica 8 7.3 2.9 6.3 1.8 virginica 9 6.7 2.5 5.8 1.8 virginica10 7.2 3.6 6.1 2.5 virginica# ℹ 40 more rows Now mtcars_sheets

mtcars\_sheets

$`4`# A tibble: 11 × 11 mpg cyl disp hp drat wt qsec vs am gear carb <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> 1 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1 2 24.4 4 147. 62 3.69 3.19 20 1 0 4 2 3 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2 4 32.4 4 78.7 66 4.08 2.2 19.5 1 1 4 1 5 30.4 4 75.7 52 4.93 1.62 18.5 1 1 4 2 6 33.9 4 71.1 65 4.22 1.84 19.9 1 1 4 1 7 21.5 4 120. 97 3.7 2.46 20.0 1 0 3 1 8 27.3 4 79 66 4.08 1.94 18.9 1 1 4 1 9 26 4 120. 91 4.43 2.14 16.7 0 1 5 210 30.4 4 95.1 113 3.77 1.51 16.9 1 1 5 211 21.4 4 121 109 4.11 2.78 18.6 1 1 4 2$`6`# A tibble: 7 × 11 mpg cyl disp hp drat wt qsec vs am gear carb <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>1 21 6 160 110 3.9 2.62 16.5 0 1 4 42 21 6 160 110 3.9 2.88 17.0 0 1 4 43 21.4 6 258 110 3.08 3.22 19.4 1 0 3 14 18.1 6 225 105 2.76 3.46 20.2 1 0 3 15 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 46 17.8 6 168. 123 3.92 3.44 18.9 1 0 4 47 19.7 6 145 175 3.62 2.77 15.5 0 1 5 6$`8`# A tibble: 14 × 11 mpg cyl disp hp drat wt qsec vs am gear carb <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> 1 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2 2 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4 3 16.4 8 276. 180 3.07 4.07 17.4 0 0 3 3 4 17.3 8 276. 180 3.07 3.73 17.6 0 0 3 3 5 15.2 8 276. 180 3.07 3.78 18 0 0 3 3 6 10.4 8 472 205 2.93 5.25 18.0 0 0 3 4 7 10.4 8 460 215 3 5.42 17.8 0 0 3 4 8 14.7 8 440 230 3.23 5.34 17.4 0 0 3 4 9 15.5 8 318 150 2.76 3.52 16.9 0 0 3 210 15.2 8 304 150 3.15 3.44 17.3 0 0 3 211 13.3 8 350 245 3.73 3.84 15.4 0 0 3 412 19.2 8 400 175 3.08 3.84 17.0 0 0 3 213 15.8 8 351 264 4.22 3.17 14.5 0 1 5 414 15 8 301 335 3.54 3.57 14.6 0 1 5 8 And that’s it! Hope this has been helpful!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Reading in Multiple Excel Sheets with lapply and {readxl}

View Details

[This article was first published on Bluecology blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Tips for using chatGPT to learn RChatGPT can help you learn R code. Here are some tips my team and I have worked out for ways to use the model to help with learning R.

There is also an askgpt package now that is meant to integrate ChatGPT directly into RStudio. However, I’ve been unable to get this package to work on my laptop.

Ask GPT to explain R codeIf you can find code on the web, from collaborators or a supervisor you don’t understand, then trying asking GPT to explain it to you. Here’s an example:

Prompt: explain this R code: m1 <- gam(y ~ s(year, by = sites), data = my_data, family = “poisson”)

GPT: The code m1 <- gam(y ~ s(year, by = sites), data = my_data) specifies a generalized additive model (GAM) in R.

Here’s what each part of the code means:

m1 <- : assigns the output of the model to the object named m1.gam() : specifies that we want to fit a generalized additive model.y ~ s(year, by = sites) : specifies the model formula, where y is the response variable and year is the predictor variable of interest. The function s() specifies that year is a smooth function, which allows for a flexible fit to the data. The by = sites argument specifies that we want to fit separate smooth functions for each sites level, which is a way to model site-specific effects.*

Overall, this code fits a GAM to the data in my_data, with year as a smooth predictor variable and site-specific effects modeled with by = sites.

It gets everything right.

This tool is helping me keep meetings with my research students at a high level. We talk about what are appropriate methods and statistical designs, and don’t need to spend as much time discussing how to code those.

Ask GPT advice on appropriate statistical modelsYou use GPT to get started on a new problem. Say you had count data but weren’t sure how to analyse it or even the name of the correct method. You could ask something this:

Prompt: How would I do a regression model with count data in R

GPT: There are several ways to model count data in R, but one popular method is to use Poisson regression or Negative Binomial regression. Here’s a step-by-step guide on how to fit a Poisson regression model in R:…

And GPT continues to explain how to write a poisson GLM in R (one appropriate way to do regression with count data).

You can often improve GPT’s answers by breaking your questions into smaller chunks. So first ask it for advice on regression of count data. Then ask for an example in R.

Generate simulated datasetsTo gain a more general understanding of a method it might help you to work with some simulated (made-up data). GPT can help you write code to do this as well. Following on from the example above I asked it to generate simulated data.

Prompt: How would I simulate some data to test the poisson glm?

GPT: To simulate count data for testing a Poisson GLM, you can use the rpois() function in R, which generates random numbers from a Poisson distribution with a given mean. Here is an example of how to simulate count data with two predictor variables:

set.seed(123) # for reproducibilityn <- 100 # number of observationsx1 <- rnorm(n) # predictor variable 1x2 <- rnorm(n) # predictor variable 2lambda <- exp(0.5 + 0.8*x1 - 0.2*x2) # log-linear model with intercept 0.5 and coefficients 0.8 and -0.2y <- rpois(n, lambda) … And then GPT continues to explain what the above code does.

Create customised examplesIt helps to learn R and statistics if the examples you are learning from are for a familar context. This one one reason why there are so many R books and tutorials that overlap in the methods they explain, just in different contexts, like “R for Ecologists”, “R for Medical Research”, ‘R for Microbiology” etc…

You can get GPT to write you customised tutorial. That way you get the explanation in a context you know. It can be easier to learn a tangible application in a context you know, before you try to learn how to use a method more generally.

Following the poisson regression example above, we could ask GPT to modify its explanation for a specific context:

Prompt: Rewrite the R code for simulating data using an example of what predicts the number of easter eggs the easter bunny brings

GPT: Sure, here’s an example of how to simulate count data based on a linear model that predicts the number of Easter eggs brought by the Easter bunny:…

Then GPT continued with basically the same code as above, where it gave x1 and x2 as the predictors. It also provided a warning at the end:

Note that this is just an example, and in reality, the number of Easter eggs brought by the Easter bunny is likely to depend on more complex factors that may not be accurately captured by a simple linear model. Additionally, the true distribution of the count data may be more complex than a Poisson distribution, and may require a more advanced simulation approach to accurately capture its properties

I further prompted it again to explain what x1 and x2 could be

Prompt: What would be predictors of the number of easter eggs?

There are many potential predictors that could influence the number of Easter eggs brought by the Easter bunny. Here are a few possibilities:Number of children in a household: It stands to reason that the number of Easter eggs brought by the Easter bunny would be higher in households with more children, since the bunny would have more individuals to provide for.Household income: Depending on the cultural context, the number of Easter eggs brought by the Easter bunny could be influenced by the socioeconomic status of the household. For example, in some cultures, more affluent households may be more likely to provide larger quantities of Easter eggs….*

And it continued with other plausible suggestions.

Making stats more funYou can go further with the bunnies above, using any example you like to keep yourself entertained (topics that are well known on the web will work best). For instance, I prompted GPT to write an explanation of generalized additive models in the style of an 18th century author. It used the classic ‘cars’ dataset. So I additionally prompted it to use horses as an example instead..

Any style that is well known on the internet should work. You can ask for a crime novel with a gripping plot twist, in the style of a particular author like J.K. Rowling or as a script from Netflix series Bridgerton. If you are really passionate about R you could even try asking GPT for an explanation in the style of 50 Shades of Grey…

Pitfalls and words of warningChatGPT is just another tool we can use for learning. It can be part of your toolbox, but should not be the sole tool you use, it has pitfalls. It can give wrong or biased answers. Generally its correct, but some responses I’ve prompted have errors in the R code or in statistical logic. So always check your code independently. This should include running tests or working with someone who knows the correct way.

ChatGPT can also generate code with bugs, which could be hard for beginners to debug (especially if they are the types of bugs that don’t throw errors).

Make sure you ask it clearly framed questions. Additionally, it has a ‘token’ limit (tokens are parts of words), so give it lots of smaller requests rather than one huge one.

GPT may learn from text you enter, so don’t put in personal information or your best research ideas.

Finally, I’m now seeing R blog posts that I’m sure are written by ChatGPT or similar models (you can tell by the style). I don’t have a problem with this, so long as the code is checked and corrected. So when searching for tutorials online make sure they are from reputable authors who you can trust to have verified their code and corrected any mistakes in explanation.

To leave a comment for the author, please follow the link and comment on their blog: Bluecology blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Tips for using chatGPT to learn R

View Details

[This article was first published on Bluecology blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Making predictions from GAMs in R explained by an 18th century authorAs explained by ChatGPT with some minor edits by me. This post is an example for my post “Tips for using chatGPT to learn R”

Pray, let us first summon the mgcv package, that we may avail ourselves of its functions in our quest to fashion a GAM.Hark! Let us summon the ‘mgcv’ package

library(mgcv) Verily, let us now create a simulated ‘horse’ dataset

```

define the sample sizen <- 100# simulate age data from a normal distributionage <- rnorm(n, mean = 10, sd = 3)# simulate rectal temperature data from a normal distributiontemp <- rnorm(n, mean = 101, sd = 1)# Create a function for the mean heat rate# with a linear effect of age and a polynomial# effect of temp. hr_mean <- -0.1age + 10temp - 0.05*(temp^2) - 430# simulate heart rate data from a normal distribution# using the function above as the mean heart ratehr <- rnorm(n, mean = hr_mean, sd = 8)# combine the simulated data into a dataframehorse <- data.frame(age, temp, hr)

``` Withal, we shall construct a GAM model, with heart rate (hr) as the response variable, and age and rectal temperature (temp) as the predictors, fashioned with the ‘s’ function to impart smoothness onto these predictors

model <- gam(hr ~ s(age) + s(temp), data = horse) Verily, let us now display the summary of the GAM model

summary(model) Hark! Let us now appreciate the beauty of the GAM model through a visual representation

plot(model) Behold, the GAM model has been crafted with the utmost care and precision.

But hold, we must not overlook the most crucial task - to predict the heart rate values for a new horse. Let us accomplish this mission with the aid of the predict function, and relay the news unto the equine owner!

new\_horse <- data.frame(age = c(5, 10, 15), temp = c(99, 101, 103))predicted\_hr <- predict(model, new\_horse) Thus, we have fulfilled our obligations, and have relayed the good tidings of the predicted heart rate value unto the horse owner. May their new companion serve them well, and gallop with vigor and grace!

To leave a comment for the author, please follow the link and comment on their blog: Bluecology blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Making predictions from GAMs in R explained by an 18th century author

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

SatRdays London is fast approaching, and we have a couple of exciting announcements to share with you!

Full program available nowThe full list of speakers and their abstracts can now be found in a downloadable program on our conference website, along with the schedule for the day and the registration options.

Registration deadline extensionThe registration deadline has now been extended to the 21st April, so you can register all the way up to the day before the event!

Virtual tickets now availableMost excitingly, we are pleased to announce that this will now be a hybrid event! If you aren’t able to make it to London for the day, there’s no need to miss out. You can sign up in the same place (via the website), and select the “Virtual only” option! You will then be able to watch live on the day, and join in on the Q&A sessions with our speakers.

We’re really looking forward to hosting you all, whether at the incredible Bush House in London or virtually, so please book your place now to make sure you don’t miss out on our excellent line up of speakers. We have a great range of talk topics, from R in journalism and MLOps, to sustainability and EDI in the R project, air quality analysis to scrutinising government spending, and much more, there will be something for everyone at this month’s SatRdays London event!

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: SatRdays London is now Hybrid!

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Dr. Maria Prokofieva of the Business Analytics and R Business User Group, Melbourne, recently talked to the R Consortium about the growing use of R and Data Science in Australia. The group feels strongly about contributing to the R community by building resources for R users at all levels of expertise. They are currently looking for volunteers to help put their plans into action.

Maria is a Senior Lecturer at the Victoria University Business School. She has a Bachelor’s in IT and Graduate Certificate in Accounting. Her Ph.D. is in the area of Business Information Systems (in Russia). With a background in IT and Business, she is passionate about using R in academic research and everyday life.

How did you get introduced to R?

I work as an academic at Victoria University. My work is at the intersection of Business, Social Sciences, and Technology. I got involved with the R community in Melbourne ten years ago. My first exposure was through R-Ladies Melbourne. I am really grateful for all the support I was provided in my early years, and I am trying to give back. Being an academic, I take great joy in teaching my students and colleagues and helping other people get insights from their data. It’s an amazing feeling to watch people progress from being afraid to do anything to feeling empowered that they can do anything.

What is the R community like in Melbourne? What is the most interesting thing about the community?

The R and Data Science community in Melbourne is very supportive. I feel that this is not just limited to the community in Melbourne. The strongest point of the R community is that if you are struggling with a problem you can reach out for help and people will try their best to help you.

People in R and Data Science are very welcoming and you feel at home when you talk to these people. And this is what I really love about the local community here and the community overall. You feel that these are my birds of a feather and I want to be around them.

What industries do you see using R in Melbourne?

The use of R in industries in Australia is still in the early stages. We can say that R in Data Science and business is still growing. In some areas of business use of R is growing rapidly. For example, the actuaries community in Australia has adopted data science. On the other end, if we look at CPA Australia publications, they still promote the use of Pivot tables in Excel and it is considered a really advanced skill.

So overall businesses are at different stages when it comes to adopting the use of R and data science in general. We are trying to provide all these businesses at different levels, a platform to share their struggles and learn from others. This group acts as a uniting force that can put people in dialogue and they can talk to each other for solving issues.

How has COVID affected your ability to connect with members? What techniques (Github, zoom, other) have you used to connect and collaborate with members? Can these techniques be used to make your group more inclusive to people that are unable to attend physical events in the future?

I think the pandemic showed us new ways to work and some new ways to communicate with each other. As meetings have shifted online and people work from home, they have become more rational with their time. They no longer want to waste any time in commute. Many businesses only require their employees to come to work three days a week and they work from home for the rest of the week.

So even for group events people prefer the online format. Initially, they show enthusiasm for physical events as they are sick of staying at home. But when the actual time for the event comes everyone prefers attending virtually.

Even though Zoom works really well for presentations and workshops, nothing can replace physical events for networking and building relationships. In my opinion, as a participant and as an organizer, physical events work best for any networking activities. Just having a coffee together with a person can change your entire perception. And it allows you to build working relationships that can never happen on Zoom. And while there is the option of hosting hybrid events, the logistics can be very challenging.

What trends do you see in R language over the next year?

I really love the transition from R Studio to Posit, and I think it will open new horizons for all R users. In our group, we try to invite all people in the data science community to our events, regardless of the programming language they use. We believe that coming from different backgrounds is really great for information sharing and learning new approaches to solving a problem. So I think the integration of R with other programming languages is one of the biggest trends that I will be looking forward to.

What is your favorite R event you have attended?

R Studio conference is my favorite R event as you get to know all the new trends there. You get to see old friends, make new friends, and get lots of insights about what is going to happen. What I really appreciate are the contests they have started recently. And it’s not about the prize but the intellectual challenge. You get involved to see what you can do. It’s equally exciting for me and my students.

When is your next event? What are your plans for the group for the coming year? Please give details!

We are building an agenda for this year. We are planning to publish blog posts twice a month. We are also planning a series of webinars which we will be recording and sharing on our YouTube channel.

We are also building up our GitHub repository and have developed a plan for it. We will be making a list of all packages available for accounting and actuaries. We will also support them with educational materials and tutorials. Everyone will be able to find and use what interests them based on their level of expertise.

We will also generate data sets from data available from businesses. A lot of times businesses are not willing to share their data. Convincing them to share their data while preserving their privacy is a valuable task. This is all part of our efforts to give back to the community, as we have all used resources made available by the community to us. I feel that it is our responsibility to contribute and pay back to the community.

We are also looking for new members and have published a call for volunteers in our recent R blog post. We need passionate volunteers who can contribute to the group with their time and effort. I would also like to take this opportunity to spread the word. It will be a really good opportunity for someone who wants to contribute to the development of resources and pay back to the R and Data Science community.

How do I Join?

Apply HereThe post Adoption of R by Actuaries Community in Melbourne appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Adoption of R by Actuaries Community in Melbourne

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. BRVMIntroductionThis R package provides a user-friendly interface for accessing data from the {BRVM}, which is a regional stock exchange serving multiple West African countries. With this package, users can easily retrieve historical stock price data, trading volumes, and other financial indicators for a variety of stocks traded on the BRVM.

The package is designed to simplify the process of gathering and analyzing financial data, making it easier for researchers, investors, and other users to extract insights and make informed decisions based on the information available. Overall, BRVM library is a valuable tool for anyone looking to gain a better understanding of the BRVM and the financial markets in West Africa.

For now, you can install from GitHub until it is on CRAN (It is on the way)

```

github dev version## We can use devtools# install.packages("devtools")devtools::install_github("Koffi-Fredysessie/BRVM")# Or use remotes# install.packages("remotes")remotes::install_github("Koffi-Fredysessie/BRVM")

``` ExamplesFirst, let’s load in the BRVM package:

library(BRVM) The BRVM_ticker_desc() functionIt receives no argument and returns BRVM tickers information such as its full name, sector and country.

```

Display tickers of BRVMtickers <- BRVM_ticker_desc()tickers

```

Warning: package 'kableExtra' was built under R version 4.2.3

| Ticker | Company name | Sector | Country | | --- | --- | --- | --- | | ABJC | SERVAIR ABIDJAN COTE D'IVOIRE | DISTRIBUTION | IVORY COAST | | BICC | BICI COTE D'IVOIRE | FINANCE | IVORY COAST | | BNBC | BERNABE COTE D'IVOIRE | DISTRIBUTION | IVORY COAST | | BOAB | BANK OF AFRICA BENIN | FINANCE | BENIN | | BOABF | BANK OF AFRICA BURKINA FASO | FINANCE | BURKINA FASO | | BOAC | BANK OF AFRICA COTE D'IVOIRE | FINANCE | IVORY COAST | | BOAM | BANK OF AFRICA MALI | FINANCE | MALI | | BOAN | BANK OF AFRICA NIGER | FINANCE | NIGER | | BOAS | BANK OF AFRICA SENEGAL | FINANCE | SENEGAL | | CABC | SICABLE COTE D'IVOIRE | INDUSTRY | IVORY COAST | | CBIBF | CORIS BANK INTERNATIONAL BURKINA FASO | FINANCE | BURKINA FASO | | CFAC | CFAO MOTORS COTE D'IVOIRE | DISTRIBUTION | IVORY COAST | | CIEC | CIE COTE D'IVOIRE | PUBLIC SERVICE | IVORY COAST | | ECOC | ECOBANK COTE D'IVOIRE | FINANCE | IVORY COAST | | ETIT | Ecobank Transnational Incorporated TOGO | FINANCE | TOGO | | FTSC | FILTISAC COTE D'IVOIRE | INDUSTRY | IVORY COAST | | NEIC | NEI-CEDA COTE D'IVOIRE | INDUSTRY | IVORY COAST | | NSBC | NSIA BANQUE COTE D'IVOIRE | FINANCE | IVORY COAST | | NTLC | NESTLE COTE D'IVOIRE | INDUSTRY | IVORY COAST | | ONTBF | ONATEL BURKINA FASO | PUBLIC SERVICE | BURKINA FASO | | ORAC | ORANGE COTE D'IVOIRE | PUBLIC SERVICE | IVORY COAST | | ORGT | ORAGROUP TOGO | FINANCE | TOGO | | PALC | PALM COTE D'IVOIRE | AGRICULTURE | IVORY COAST | | PRSC | TRACTAFRIC MOTORS COTE D'IVOIRE | DISTRIBUTION | IVORY COAST | | SAFC | SAFCA COTE D'IVOIRE | FINANCE | IVORY COAST | | SCRC | SUCRIVOIRE COTE D'IVOIRE | AGRICULTURE | IVORY COAST | | SDCC | SODE COTE D'IVOIRE | PUBLIC SERVICE | IVORY COAST | | SDSC | BOLLORE TRANSPORT & LOGISTICS COTE D'IVOIRE | TRANSPORT | IVORY COAST | | SEMC | CROWN SIEM COTE D'IVOIRE | INDUSTRY | IVORY COAST | | SGBC | SOCIETE GENERALE COTE D'IVOIRE | FINANCE | IVORY COAST | | SHEC | VIVO ENERGY COTE D'IVOIRE | DISTRIBUTION | IVORY COAST | | SIBC | SOCIETE IVOIRIENNE DE BANQUE COTE D'IVOIRE | FINANCE | IVORY COAST | | SICC | SICOR COTE D'IVOIRE | AGRICULTURE | IVORY COAST | | SIVC | AIR LIQUIDE COTE D'IVOIRE | INDUSTRY | IVORY COAST | | SLBC | SOLIBRA COTE D'IVOIRE | INDUSTRY | IVORY COAST | | SMBC | SMB COTE D'IVOIRE | INDUSTRY | IVORY COAST | | SNTS | SONATEL SENEGAL | PUBLIC SERVICE | SENEGAL | | SOGC | SOGB COTE D'IVOIRE | AGRICULTURE | IVORY COAST | | SPHC | SAPH COTE D'IVOIRE | AGRICULTURE | IVORY COAST | | STAC | SETAO COTE D'IVOIRE | OTHER | IVORY COAST | | STBC | SITAB COTE D'IVOIRE | INDUSTRY | IVORY COAST | | SVOC | MOVIS COTE D'IVOIRE | TRANSPORT | IVORY COAST | | TTLC | TOTAL COTE D'IVOIRE | DISTRIBUTION | IVORY COAST | | TTLS | TOTAL SENEGAL | DISTRIBUTION | SENEGAL | | TTRC | TRITURAF Ste en Liquid | INDUSTRY | IVORY COAST | | UNLC | UNILEVER COTE D'IVOIRE | INDUSTRY | IVORY COAST | | UNXC | UNIWAX COTE D'IVOIRE | INDUSTRY | IVORY COAST |

The BRVM_index() function :It receives no argument and returns a table of updated data (with as table header: indexes, previous closing, closing, change (%), Year to Date Change) on all the indices available on the BRVM exchange.

| Indexes | Previous closing | Closing | Change (%) | Year to Date Change | | --- | --- | --- | --- | --- | | BRVM-30 | 99.71 | 99.75 | 0.04 | 0.00 | | BRVM - AGRICULTURE | 281.76 | 281.25 | -0.18 | -0.66 | | BRVM - OTHER SECTOR | 1295.58 | 1357.27 | 4.76 | -7.32 | | BRVM - COMPOSITE | 199.37 | 199.46 | 0.05 | 0.85 | | BRVM - DISTRIBUTION | 346.02 | 345.33 | -0.20 | 0.69 | | BRVM - FINANCE | 74.53 | 75.03 | 0.67 | -0.66 | | BRVM - INDUSTRY | 98.33 | 98.10 | -0.23 | 0.92 | | BRVM - PRESTIGE | 102.61 | 102.56 | -0.05 | 0.00 | | BRVM - PRINCIPAL | 94.56 | 94.62 | 0.06 | 0.00 | | BRVM - PUBLIC SERVICES | 480.97 | 479.60 | -0.28 | 2.23 | | BRVM - TRANSPORT | 345.28 | 341.70 | -1.04 | 0.35 |

The BRVM_get(“.symbol”, “.from”, “.to”) functionThis function will get the data of the companies listed on the BVRM stock exchange in Rich Bourse website. The function takes a single parameter, .symbol (which represents the “Ticker”). The function will automatically format tickers you enter in uppercase using toupper() and then ensure that the passed ticker is in a Google spreadsheet of allowed tickers.

.symbol : A vector of symbols, like: c(“BICC”,“XOM”,“SlbC”) ;
.from : A quoted start date, ie. “2020-01-01” or “2020/01/01”. The date must be in ymd format “YYYY-MM-DD” or “YYYY/MM/DD” ;
.to : A quoted end date, ie. “2022-01-31” or “2022/01/31”. The date must be in ymd format “YYYY-MM-DD” or “YYYY/MM/DD”.

```

' Displaying data of SONATEL Senegal stockBRVM_get(.symbol = "snts")

```

[1] "SNTS"

```

A tibble: 251 × 6 Date Open High Low Close Volume 1 2022-04-06 15800 15895 15750 15800 7436 2 2022-04-07 15800 15900 15750 15900 1265 3 2022-04-08 15900 15995 15800 15900 1164 4 2022-04-11 15895 15900 15800 15800 4252 5 2022-04-12 15800 15800 15780 15800 6561 6 2022-04-13 15800 15865 15795 15850 5409 7 2022-04-14 15855 15900 15850 15900 16957 8 2022-04-15 15995 15995 15900 15900 791 9 2022-04-19 15900 15995 15895 15900 3121710 2022-04-20 15900 15995 15895 15990 32322# ℹ 241 more rows

```

symbols <- c("BiCc","XOM","SlbC") # We use here three tickersdata\_tbl <- BRVM\_get(.symbol = symbols, .from = "2020-01-01", .to = Sys.Date() - 1)

[1] "BICC" "SLBC"

```

Display the first twenty observations of the tablehead(data_tbl, 20)

```

A tibble: 20 × 7 Date Open High Low Close Volume Ticker 1 2020-01-10 6500 6500 6500 6500 24 BICC 2 2020-01-13 6370 6500 6370 6500 29 BICC 3 2020-01-14 6495 6495 6495 6495 10 BICC 4 2020-01-29 6010 6010 6010 6010 24 BICC 5 2020-01-30 6000 6000 6000 6000 50 BICC 6 2020-02-04 5800 5800 5800 5800 12 BICC 7 2020-02-07 5650 5650 5650 5650 5 BICC 8 2020-02-10 5500 5500 5500 5500 5 BICC 9 2020-02-14 5300 5300 5300 5300 9 BICC 10 2020-02-17 4910 4910 4910 4910 210 BICC 11 2020-02-18 4910 4910 4910 4910 50 BICC 12 2020-02-20 4895 4895 4895 4895 5 BICC 13 2020-02-21 4895 4895 4890 4890 13 BICC 14 2020-02-25 4525 4525 4525 4525 16 BICC 15 2020-02-26 4435 4435 4430 4430 21 BICC 16 2020-02-27 4345 4760 4335 4760 1809 BICC 17 2020-03-03 4745 4750 4745 4750 11 BICC 18 2020-03-05 4700 4700 4700 4700 5 BICC 19 2020-03-06 4695 4695 4695 4695 6 BICC 20 2020-03-11 4345 4450 4345 4450 135 BICC

```

Display the last twenty elements of the tabletail(data_tbl, 20)

```

A tibble: 20 × 7 Date Open High Low Close Volume Ticker 1 2023-02-15 80000 80000 79000 79000 2 SLBC 2 2023-02-17 78000 78000 78000 78000 5 SLBC 3 2023-02-21 80000 80000 80000 80000 5 SLBC 4 2023-02-23 80000 80000 80000 80000 18 SLBC 5 2023-02-24 80000 80000 80000 80000 6 SLBC 6 2023-02-27 80000 80000 80000 80000 98 SLBC 7 2023-02-28 80000 80000 80000 80000 11 SLBC 8 2023-03-02 80000 80000 80000 80000 11 SLBC 9 2023-03-08 80000 80000 80000 80000 2 SLBC 10 2023-03-09 80000 80000 80000 80000 2 SLBC 11 2023-03-13 80005 80005 80000 80000 12 SLBC 12 2023-03-14 80000 80000 80000 80000 1 SLBC 13 2023-03-20 80000 80000 80000 80000 3 SLBC 14 2023-03-21 80000 80000 80000 80000 4 SLBC 15 2023-03-27 78000 80000 78000 80000 169 SLBC 16 2023-03-28 80000 80000 80000 80000 435 SLBC 17 2023-03-30 80000 80000 80000 80000 3 SLBC 18 2023-03-31 80000 80000 80000 80000 1 SLBC 19 2023-04-04 80000 86000 80000 86000 3 SLBC 20 2023-04-05 85950 86000 85950 86000 6 SLBC

`` The **BRVM\_get1(“ticker”, “Period”, “from”, “to”)** functionThis function will get data of the companies listed on the BVRM stock exchange through the sikafinance site. The function takes in a single parameter of ticker and will auto-format the tickers you input into all upper case by usingtoupper()`

ticker : A vector of ticker, like: c(“BICC”,“XOM”,“SlbC”, “BRvm10”);
Period : Numeric number indicating time period. Valid entries are 0, 1, 5, 30, 91, and 365 representing respectively ‘daily’, ‘one year’, ‘weekly’, ‘monthly’, ‘quarterly’ and ‘yearly’;
from : A quoted start date, ie. “2020-01-01” or “2020/01/01”. The date must be in ymd format “YYYY-MM-DD” or “YYYY/MM/DD”;
to : A quoted end date, ie. “2022-01-31” or “2022/01/31”. The date must be in ymd format “YYYY-MM-DD” or “YYYY/MM/DD”

** NB : There is a small difference between the BRVM_get and BRVM_get1 functions. * With BRVM_get it is only possible to download tickers’ daily data. * But with BRVM_get1, you can download daily, weekly, monthly, annual tickers’ data, indices and even market capitalization.

```

' Displaying data of SONATEL Senegal stockBRVM_get1("snts")

```

[1] "Make sure you have an active internet connection"

```

Get daily data of all indexesall_ind <- BRVM_get1("ALL INDEXES", Period = 0, from = "2020-01-04", to = "2023-03-24")

```

[1] "We obtained BRVM10 data from 2019-12-26 to 2023-01-04"[1] "We obtained BRVMAG data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMC data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMAS data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMDI data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMFI data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMIN data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMSP data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMTR data from 2019-12-26 to 2023-03-24"[1] "We obtained BRVMPR data from 2023-01-01 to 2023-03-24"[1] "We obtained BRVMPA data from 2023-01-04 to 2023-03-24"[1] "We obtained BRVM30 data from 2023-01-01 to 2023-03-24"[1] "We obtained CAPIB data from 2020-01-02 to 2023-03-24"

```

display the first two tens elements of the tablehead(all_ind, 20)

```

A tibble: 20 × 7 Date Open High Low Close Volume Ticker 1 2022-12-26 169. 169. 169. 169. 0 BRVM10 2 2022-12-27 169. 169. 169. 169. 0 BRVM10 3 2022-12-28 167. 167. 167. 167. 0 BRVM10 4 2022-12-29 167. 167. 167. 167. 0 BRVM10 5 2022-12-30 166. 166. 166. 166. 0 BRVM10 6 2023-01-02 166. 166. 166. 166. 0 BRVM10 7 2023-01-03 166. 166. 166. 166. 0 BRVM10 8 2023-01-04 166. 166. 166. 166. 0 BRVM10 9 2022-09-26 163. 163. 163. 163. 0 BRVM1010 2022-09-27 162. 162. 162. 162. 0 BRVM1011 2022-09-28 162. 162. 162. 162. 0 BRVM1012 2022-09-29 163. 163. 163. 163. 0 BRVM1013 2022-09-30 164. 164. 164. 164. 0 BRVM1014 2022-10-03 162. 162. 162. 162. 0 BRVM1015 2022-10-04 162. 162. 162. 162. 0 BRVM1016 2022-10-05 161. 161. 161. 161. 0 BRVM1017 2022-10-06 161. 161. 161. 161. 0 BRVM1018 2022-10-07 161. 161. 161. 161. 0 BRVM1019 2022-10-10 160. 160. 160. 160. 0 BRVM1020 2022-10-11 160. 160. 160. 160. 0 BRVM10

```

display the two tens of the last elements of the tabletail(all_ind, 20)

```

A tibble: 20 × 7 Date Open High Low Close Volume Ticker 1 2020-02-26 4281311 4281311 4281311 4281311 0 CAPIB 2 2020-02-27 4314933 4314933 4314933 4314933 0 CAPIB 3 2020-02-28 4346515 4346515 4346515 4346515 0 CAPIB 4 2020-03-02 4424073 4424073 4424073 4424073 0 CAPIB 5 2020-03-03 4379647 4379647 4379647 4379647 0 CAPIB 6 2020-03-04 4369550 4369550 4369550 4369550 0 CAPIB 7 2020-03-05 4342229 4342229 4342229 4342229 0 CAPIB 8 2020-03-06 4359879 4359879 4359879 4359879 0 CAPIB 9 2020-03-09 4338293 4338293 4338293 4338293 0 CAPIB 10 2020-03-10 4357221 4357221 4357221 4357221 0 CAPIB 11 2020-03-11 4332656 4332656 4332656 4332656 0 CAPIB 12 2020-03-12 4318096 4318096 4318096 4318096 0 CAPIB 13 2020-03-13 4318112 4318112 4318112 4318112 0 CAPIB 14 2020-03-16 4285184 4285184 4285184 4285184 0 CAPIB 15 2020-03-17 4301727 4301727 4301727 4301727 0 CAPIB 16 2020-03-18 4288582 4288582 4288582 4288582 0 CAPIB 17 2020-03-19 4207231 4207231 4207231 4207231 0 CAPIB 18 2020-03-20 4209788 4209788 4209788 4209788 0 CAPIB 19 2020-03-23 4154445 4154445 4154445 4154445 0 CAPIB 20 2020-03-24 4144325 4144325 4144325 4144325 0 CAPIB

```

To get yearly datayearly_data <- BRVM_get1(c("brvmtr", "BiCc", "BOAS"), Period = 365 ) # display the first two tens elements of the tablehead(yearly_data, 20)

```

A tibble: 20 × 6 Date Open High Low Close Ticker 1 2003-04-11 74.0 88.6 73.6 88.6 BRVMTR 2 2004-01-02 88.6 89.2 72.9 89.2 BRVMTR 3 2005-01-03 89.2 107. 70.7 104. BRVMTR 4 2006-01-02 104. 158. 104. 153. BRVMTR 5 2007-01-02 153. 275. 149. 249. BRVMTR 6 2008-01-02 249. 386. 226. 296. BRVMTR 7 2009-01-02 275. 296. 227. 236. BRVMTR 8 2010-01-04 236. 259. 224. 238. BRVMTR 9 2011-01-03 238. 249. 204. 239 BRVMTR10 2012-01-02 239 349. 201. 349. BRVMTR11 2013-01-02 349. 794. 339. 789. BRVMTR12 2014-01-02 789. 1213. 601. 1213. BRVMTR13 2015-01-02 1213. 1525. 653. 1525. BRVMTR14 2016-01-04 1525. 1525. 1216. 1432. BRVMTR15 2017-01-02 1432. 1433. 764. 1203. BRVMTR16 2018-01-02 1114. 1193. 966. 966. BRVMTR17 2019-06-03 403. 429. 311. 367. BRVMTR18 2020-01-01 367. 475. 292. 379. BRVMTR19 2021-01-04 376. 622. 325 622. BRVMTR20 2022-01-03 667. 667. 295. 342. BRVMTR

```

display the two tens of the last elements of the tabletail(yearly_data, 20)

```

A tibble: 20 × 6 Date Open High Low Close Ticker 1 2014-01-02 5650 7848 5650 7800 BICC 2 2015-01-02 8385 10750 7800 10100 BICC 3 2016-01-04 10000 10700 8566 9890 BICC 4 2017-01-05 9750 10000 6440 8490 BICC 5 2018-01-02 8700 8750 3795 7900 BICC 6 2019-01-04 7550 7550 3710 6800 BICC 7 2020-01-01 6800 6890 2855 6680 BICC 8 2021-01-04 6680 7525 4280 7400 BICC 9 2022-01-03 7250 7250 5550 6850 BICC 10 2023-01-02 6500 6850 5785 6275 BICC 11 2014-12-10 1613 3225 1613 3225 BOAS 12 2015-01-02 3370 4300 2900 3950 BOAS 13 2016-01-04 3700 4101 2000 2350 BOAS 14 2017-01-02 2325 3875 2035 2500 BOAS 15 2018-01-02 2400 3250 1700 2020 BOAS 16 2019-01-02 1900 2000 1500 1545 BOAS 17 2020-01-01 1550 1700 1295 1495 BOAS 18 2021-01-04 1480 2750 1340 2350 BOAS 19 2022-01-03 2350 2780 2200 2450 BOAS 20 2023-01-02 2580 2585 2175 2265 BOAS

``` The BRVM.index() function :It receives no argument and returns the name of all indexes available on BRVM Stock Exchange.

BRVM.index()

[1] "BRVMAG" "BRVMC" "BRVMAS" "BRVMDI" "BRVMFI" "BRVMIN" "BRVMSP" "BRVMTR" [9] "BRVMPR" "BRVMPA" "BRVM30" Authors :

Koffi Frederic Sessie (koffisessie@gmail.com),
Abdoul Oudouss Diakité (abdouloudoussdiakite@gmail.com),
Sanderson Steven(spsanderson@gmail.com)

Creator : Koffi Frederic Sessie

cph (Copyright Holder) : Koffi Frederic Sessie

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A New Package for the African Stock Market {BRVM}

View Details

[This article was first published on R on Jason Bryer, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.My son has been learning Python and wanted to know how many rounds does it take, on average, for someone to win the game of war. If you are not familar with the game, it is a two player game where each player gets half a deck of cards (26). For each round the player puts down their top card. Whoever has the higher card gets both cards. In the instance where there is a tie, each player puts down three more cards and flips the fourth card. The player with the higher card gets all the cards put down (a total 10). If by chance there is a tie, they put down another three cards flipping the fourth. This is my attempt to solve this problem in R.

First I defined a function that performs a single round. Note that this function can be called recursively in the case there is a tie. The players cards are kept track in an integer vector p1 and p2 for player 1 and 2, respectively. It also returns an integer indicating who won that round.

```

' Checks to see who wins the current round.#'#' This will compare the cards at position one. The winner gets both cards. If#' there is a tie then the next three cards are put down and the fourth card#' is compared. This is technically done recursively. The winner gets the#' loosers five cards (the original, three put down, and the fourth is checked).#'#' The game ends when either of the players have no cards remaining.#'#' @param p1 cards for player 1.#' @param p2 cards for player 2.#' @return a list with three elements:#' * `winner` - either a 1 or 2 for who won the round.#' * `player1` - cards remaining for player 1.#' * `player2` - cards remaining for player 2.check_cards <- function(p1, p2) {winner <- 0if(length(p1) == 0) {winner <- 2} else if(length(p2) == 0) {winner <- 1} else {if(p1[1] == p2[1]) {# print('Entering a war!')if(length(p1) < 5) {p1 <- integer()p2 <- c(p2, p1)winner <- 2} else if(length(p2) < 5) {p1 <- c(p1, p2)p2 <- integer()winner <- 1} else {winners_cards <- c(p1[1:min(length(p1), 4)], p2[1:min(length(p2), 4)])result <- check_cards(p1[min(length(p1), 5):length(p1)], p2[min(length(p2), 5):length(p2)])if(result$winner == 1) {p1 <- c(result$player1, winners_cards)p2 <- result$player2winner <- 1} else if(result$winner == 2) {p1 <- result$player1p2 <- c(result$player2, winners_cards)winner <- 2}}} else if(p1[1] > p2[1]) {p1 <- c(p1[-1], p1[1], p2[1])p2 <- p2[-1]winner <- 1} else if(p2[1] > p1[1]) {p2 <- c(p2[-1], p2[1], p1[1])p1 <- p1[-1]winner <- 2} else {stop('Should not be here. What happend?!')}}return(list(player1 = p1,player2 = p2,winner = winner))}

`` Next I defined a function that will initiate a game of war. It has a while loop callingcheck_cardsuntil one player has no more cards ormax_games` is reached therefore calling a draw.

```

' Initiate a game of war.#'#' @param cards a vector representing the cards. The default assumes a typical#' deck of 52 cards.#' @param print how often should the function print the status of how many#' cards each player has. Set to 0 to don't print any updates.#' @param max_games the maximum number of games to try before it is considered#' a draw.#' @return a list with four elements:#' * `games` - number of rounds it took for a player to win.#' * `winner` - 0 for a draw, 1 if player 1 won, 2 if player 2 won.#' * `p1\_cards` - a numeric vector with the number of cards player 1 had after each round.#' * `p2\_cards` - a numeric vector with the number of cards player 2 had after each round.play_war <- function(cards = rep(1:13, 4), print = 0, max_games = 2000) {deal <- sample(length(cards), length(cards) / 2, replace = FALSE)player1 <- cards[deal]player2 <- cards[-deal]games <- 0p1_cards <- integer(max_games)while(length(player1) > 0 & length(player2) > 0 & games < max_games) {result <- check_cards(player1, player2)player1 <- result$player1player2 <- result$player2p1_cards[games + 1] <- length(player1)if(length(player1) == 0 | length(player2) == 0) {break;}if(length(player1) + length(player2) != length(cards)) {stop(paste0('No longer `have ', length(cards), ' cards! ',(length(player1) + length(player2))))}games <- games + 1if(print > 0) {if(games %% print == 0) {print(paste0('Game ', games, ':', ' Player 1 cards: ', length(player1), '; ', ' Player 2 cards: ', length(player2)))}}}winner <- 0if(length(player1) == 0) {winner <- 2} else if(length(player2) == 0) {winner <- 1}return(list(games = games,winner = winner,p1_cards = p1_cards,p2_cards = length(cards) - p1_cards))}

`` We can now play a single game of war. Hereprint = 25` so the function will print the card status every 25 rounds.

play\_war(print = 25)$games## [1] "Game 25: Player 1 cards: 37; Player 2 cards: 15"## [1] "Game 50: Player 1 cards: 30; Player 2 cards: 22"## [1] "Game 75: Player 1 cards: 21; Player 2 cards: 31"## [1] "Game 100: Player 1 cards: 20; Player 2 cards: 32"## [1] "Game 125: Player 1 cards: 9; Player 2 cards: 43"## [1] "Game 150: Player 1 cards: 10; Player 2 cards: 42"## [1] "Game 175: Player 1 cards: 11; Player 2 cards: 41"## [1] "Game 200: Player 1 cards: 8; Player 2 cards: 44"## [1] 206 Let’s find out how many rounds it takes to win. Here we will play 10,000 games of war.

n\_games <- 10000war\_games <- list()for(i in seq\_len(n\_games)) {# set.seed(i) # Used to track down issues and debugwar\_games[[i]] <- play\_war(print = 0)}plays\_to\_win <- sapply(war\_games, FUN = function(x) { x$games }) First we’ll see if there are any draws (unlikely with max_games = 2000).

draws <- which(plays\_to\_win == 2000)sum(plays\_to\_win == 2000)## [1] 0 Now let’s see what the mean, median, and distrubtion of rounds is.

hist(plays\_to\_win, main = 'Histogram of Number of Plays to Win Game of War', xlab = 'Number of Plays')

mean(plays\_to\_win)## [1] 173.6619median(plays\_to\_win)## [1] 135min(plays\_to\_win)## [1] 9max(plays\_to\_win)## [1] 1381 To leave a comment for the author, please follow the link and comment on their blog: R on Jason Bryer.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How Long Does it Take to Win War?

View Details

[This article was first published on R Views, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Vidisha Vachharajani works in the EdTech industry, where she enjoys developing data-driven strategy solutions for learners. She has been an R user for over 15 years.

As a data professional, I have enjoyed learning and using multiple tools for my workflows. For me, everything used to begin and end with R. Today, SQL is a must-know. Not being able to pull your own custom tables from a warehouse can make things tricky. Then there is tidyverse, the master collection of packages for data science & analytics. As an OG R user, I cannot envision data work without tidyverse.

In this first part of a 2-part article, I want to demonstrate how a data analyst can use one OR the other for the initial stages of data exploration, and then double down on tidyverse, leveraging ggplot2 for a deeper exploration. By no means does this preclude the extensive use of SQL for data wrangling. Rather, this post showcases the wonders of tidyverse (a collection of R packages designed for data science, sharing an underlying design philosophy, grammar, and data structures) and specifically, ggplot2 (the language of elegant graphics) for a SQL user’s benefit.

The dataset and the goalThe dataset I am using is clinical. Sourced from the UCI machine learning repo, it is the Diabetes 130-US hospitals for years 1999-2008 Data Set. The dataset is large, ~100K rows and 51 columns in its raw format. It is, however, clean data. For the purpose of this article, in order to show SQL and tidyverse language in tandem, I will split it up into 5 parts, and we will assume that the data is actually available to us in these 5 different pieces, rather than as the whole, cleaned data, since this is typically the case in real life.

I will skip the portion about dbplyr, referring readers to the hyperlinked article that will show you how to actually pull data from a remote database using tidyverse’s dbplyr. Typically, this is done using SQL, butdbplyr allows you to do this within R. Rather, I will focus on the initial stages of data exploration, using both SQL and tidyverse for the same output, while extending the tidyverse portion to include ggplot2 visualization examples, using different plot types for each use case. Note that in each case, you can use SQL first, and then use the SQL output as an input for the ggplot2 visualization.

Reading in the dataThe data has been split into 5 parts – demographic, medical, hospital visits, outcome, test results. To learn more about the actual data, see here. Each part is connected with the other through a UID that is a concatenation of the patient encounter ID and the patient number (using either one doesn’t work to make the ID unique). Note that all of the analyses in this post will be done at the UID level, rather than patient level.

```

rm(list=ls())library(sqldf)library(dplyr)library(readxl)#library(dbplyr)library(ggplot2)data_path <- "./dataset_diabetes/diabetic_data.xlsx"dem <- read_excel(data_path, "demo")meds <- read_excel(data_path, "medications")visits <- read_excel(data_path, "hosp_visits")y <- read_excel(data_path, "readmissions")results <- read_excel(data_path, "test_results")

`` 3. Early explorationsLet’s begin using SQL andtidyverseto answer some initial questions related to the dataset. The primary hypothesis for this data is the **impact of HbA1c measurement on readmission rates**, where “readmission” is our response. We will also answer a number of other questions along the way to understand the data better, usingggplot2` when we can.

3.1 Look at the data3.1.1 Get some countsLet’s take a look at medications and get a sample size for it, first using SQL and then R.

sqldf('SELECT * FROM meds where 1=0') # SQL see col names## [1] uid metformin repaglinide ## [4] nateglinide chlorpropamide glimepiride ## [7] acetohexamide glipizide glyburide ## [10] tolbutamide pioglitazone rosiglitazone ## [13] acarbose miglitol troglitazone ## [16] tolazamide examide citoglipton ## [19] insulin glyburide-metformin glipizide-metformin ## [22] glimepiride-pioglitazone metformin-rosiglitazone metformin-pioglitazone ## [25] change diabetesMed ## <0 rows> (or 0-length row.names)sqldf('SELECT uid, metformin, repaglinide, nateglinide, chlorpropamide FROM meds LIMIT 5') # SQL## uid metformin repaglinide nateglinide chlorpropamide## 1 2278392-8222157 No No No No## 2 149190-55629189 No No No No## 3 64410-86047875 No No No No## 4 500364-82442376 No No No No## 5 16680-42519267 No No No Nohead(meds, n=5) # dplyr## # A tibble: 5 × 26## uid metfo…¹ repag…² nateg…³ chlor…⁴ glime…⁵ aceto…⁶ glipi…⁷ glybu…⁸ tolbu…⁹## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> ## 1 22783… No No No No No No No No No ## 2 14919… No No No No No No No No No ## 3 64410… No No No No No No Steady No No ## 4 50036… No No No No No No No No No ## 5 16680… No No No No No No Steady No No ## # … with 16 more variables: pioglitazone <chr>, rosiglitazone <chr>,## # acarbose <chr>, miglitol <chr>, troglitazone <chr>, tolazamide <chr>,## # examide <chr>, citoglipton <chr>, insulin <chr>,## # `glyburide-metformin` <chr>, `glipizide-metformin` <chr>,## # `glimepiride-pioglitazone` <chr>, `metformin-rosiglitazone` <chr>,## # `metformin-pioglitazone` <chr>, change <chr>, diabetesMed <chr>, and## # abbreviated variable names ¹metformin, ²repaglinide, ³nateglinide, …sqldf('SELECT COUNT(uid) FROM meds') # SQL## COUNT(uid)## 1 101766nrow(meds) # R## [1] 101766 How many patients with a diabetes diagnosis, vs respiratory, circulatory, etc.?

sqldf('SELECT primary\_diag, COUNT(*) FROM results GROUP BY primary\_diag') # SQL## primary\_diag COUNT(*)## 1 circulatory 30437## 2 diabetes 8757## 3 other 48149## 4 respiratory 14423results %>% group\_by(primary\_diag) %>% count(primary\_diag) # R## # A tibble: 4 × 2## # Groups: primary\_diag [4]## primary\_diag n## <chr> <int>## 1 circulatory 30437## 2 diabetes 8757## 3 other 48149## 4 respiratory 14423 How many women came in through an emergency admission type?

sqldf('SELECT gender, admission\_type\_id, COUNT(*) AS n FROM dem LEFT JOIN visits USING(uid) WHERE admission\_type\_id=1 GROUP BY gender') # SQL## gender admission\_type\_id n## 1 Female 1 29448## 2 Male 1 24540## 3 Unknown/Invalid 1 2visits %>% left\_join(dem, by=join\_by(uid)) %>% subset(admission\_type\_id==1) %>% count(gender, admission\_type\_id) # dplyr## # A tibble: 3 × 3## gender admission\_type\_id n## <chr> <dbl> <int>## 1 Female 1 29448## 2 Male 1 24540## 3 Unknown/Invalid 1 2 3.1.2 A mosaic plotInstead of extracting counts manually, let’s use a mosaic plot to get a sense of how 2 count variables are distributed relative to each other. In this case, age and admission type. This plot sheds light into data availability and asymmetric distributions. For example, here, we see that most patients come from emergency, urgent care, or as an elective, and that there is missing or “not available” admission type data. It is important to retain these 2 categories separately, since they mean different things. Note that in the ggplot parameters, I have not yet introduced axes label cleanup, etc.

p0 <- dem %>% left\_join(visits, by=join\_by(uid)) %>% mutate(admission\_type=ifelse(admission\_type\_id==1, "1:Emergency", ifelse(admission\_type\_id==2, "2:Urgent", ifelse(admission\_type\_id==3, "3:Elective", ifelse(admission\_type\_id==4, "4:Newborn", ifelse(admission\_type\_id==5, "5:Not Available", ifelse(admission\_type\_id==6, "6:NULL", ifelse(admission\_type\_id==7, "7:Trauma Center", "8:Not Mapped")))))))) %>% group\_by(admission\_type, age) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) ggplot(p0, aes(x=age, y=admission\_type)) + geom\_tile(aes(fill=n)) + scale\_fill\_gradient(low="white", high="blue") 3.1.3 A simple joinLet’s join all 5 datasets and look at it. Note that in SQL, in order to look only at the first few columns, we need to know the column names, which is what we first do here.

```

(Output suppressed)sqldf('SELECT * FROM dem LEFT JOIN visits USING(uid) LEFT JOIN results USING(uid) LEFT JOIN meds USING(uid) LEFT JOIN y USING(uid) where 1=0') # SQLsqldf('SELECT uid, race, gender, age, weight FROM dem LEFT JOIN visits USING(uid) LEFT JOIN results USING(uid) LEFT JOIN meds USING(uid) LEFT JOIN y USING(uid) LIMIT 5') # SQL## uid race gender age weight## 1 2278392-8222157 Caucasian Female [0-10) ?## 2 149190-55629189 Caucasian Female [10-20) ?## 3 64410-86047875 AfricanAmerican Female [20-30) ?## 4 500364-82442376 Caucasian Male [30-40) ?## 5 16680-42519267 Caucasian Male [40-50) ?dem %>% left_join(visits, by=join_by(uid)) %>% left_join(results, by=join_by(uid)) %>% left_join(meds, by=join_by(uid)) %>% left_join(y, by=join_by(uid)) %>% print(n=5) # dplyr, by default shows 10 rows, so we ask it to print 5## # A tibble: 101,766 × 50## uid race gender age weight admis…¹ disch…² admis…³ time_…⁴ payer…⁵## ## 1 2278392-822… Cauc… Female [0-1… ? 6 25 1 1 ? ## 2 149190-5562… Cauc… Female [10-… ? 1 1 7 3 ? ## 3 64410-86047… Afri… Female [20-… ? 1 1 7 2 ? ## 4 500364-8244… Cauc… Male [30-… ? 1 1 7 2 ? ## 5 16680-42519… Cauc… Male [40-… ? 1 1 7 1 ? ## # … with 101,761 more rows, 40 more variables: medical_specialty ,## # num_lab_procedures , num_procedures , num_medications ,## # number_outpatient , number_emergency , number_inpatient ,## # diag_1 , diag_2 , diag_3 , number_diagnoses ,## # max_glu_serum , A1Cresult , primary_diag , metformin ,## # repaglinide , nateglinide , chlorpropamide ,## # glimepiride , acetohexamide , glipizide , glyburide , …

``` 3.2 Explore the response: readmissions3.2.1 Lab proceduresLet’s start with the simplest question – for the primary response variable, “readmitted”, how many lab procedures were done by each category of the response? Note here that “number of lab procedures” is one of a handful of continuous design covariate – rest of the ~45 covariates are all categorical/discrete.

```

How many lab tests performed for readmitted patientssqldf('SELECT readmitted, SUM(num_lab_procedures) AS n FROM visits LEFT JOIN y USING(uid) GROUP BY readmitted') # SQL## readmitted n## 1 <30 502275## 2 >30 1558172## 3 NO 2325224visits %>% left_join(y, by=join_by(uid)) %>% group_by(readmitted) %>% summarise(n=sum(num_lab_procedures)) %>% mutate(freq = n / sum(n)) # dplyr, w/ an added proportion ## # A tibble: 3 × 3## readmitted n freq## ## 1 <30 502275 0.115## 2 >30 1558172 0.355## 3 NO 2325224 0.530

`` Since the above doesn’t really tell us much, other than actual counts, proportions by response categories, let’s useggplot2to explore the distribution of “number of lab procedures”, using a barplot/histogram approach, with “readmitted” as thefill` element. This helps us get a better picture of their relationship; we see here how, for a strikingly normally distributed “number of lab procedures” (other than 1 outlier), on average, the higher the volume of procedures, the more the proportion of readmitted.

```

How many lab tests performed for readmitted patients, use ggplot2p1 <- visits %>% left_join(y, by=join_by(uid))ggplot(data = p1 ,aes(x=num_lab_procedures,fill=readmitted)) + geom_bar() + labs(x="Number of lab procedures", y="counts") + scale_y_continuous( labels = function(n) scales::comma(abs(n)))

`` Let’s also do this usingggplot’s beautiful density plots. It is a slightly different type of visual, and tells us how the distribution of X shifts left or right by the response orfill`.

ggplot(p1, aes(num\_lab\_procedures)) + geom\_density(aes(fill=factor(readmitted)), alpha=0.8) + labs(x="Number of lab procedures") 3.2.2 DemographicsNext, we ask how readmissions differ across age groups and gender. Let’s also plot this to understand the output better. We first use a population pyramid approach to get the counts and then barplot the proportions to get a better understanding of the variance in readmissions across these groups.

```

What are readmission rates by the different age groups?sqldf('SELECT age, readmitted, COUNT() AS n FROM dem LEFT JOIN y USING(uid) GROUP BY age') # SQL## age readmitted n## 1 [0-10) NO 161## 2 [10-20) >30 691## 3 [20-30) NO 1657## 4 [30-40) NO 3775## 5 [40-50) NO 9685## 6 [50-60) >30 17256## 7 [60-70) NO 22483## 8 [70-80) >30 26068## 9 [80-90) NO 17197## 10 [90-100) NO 2793p21 <- dem %>% left_join(y, by=join_by(uid)) %>% group_by(gender, age, readmitted) %>% summarise(n=n()) %>%mutate(pct = 100 n / sum(n), readmission=ifelse(readmitted=="NO", "not readmitted", "readmitted")) %>% ungroup() %>% subset(gender=="Male"|gender=="Female") %>% ggplot() + geom_col(aes(x = ifelse(readmission == "readmitted", -n, n), y = age, fill = readmission)) + facet_wrap(~ gender) + scale_x_continuous( labels = function(n) scales::comma(abs(n))) + xlab("Counts")p21

```

p22 <- dem %>% left\_join(y, by=join\_by(uid)) %>% group\_by(gender, age, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) %>% subset(gender=="Male"|gender=="Female")ggplot(data=p22, aes(x=age, y=freq, fill=readmitted)) + geom\_col() + facet\_wrap(~ gender) + labs(y="proportions") + geom\_text(aes(label = paste0(round(freq, 4) * 100, "%")), position = position\_stack(vjust = 0.5), size=2.5, angle=90) + theme(axis.text.x = element\_text(angle=90, vjust=.5, hjust=1)) The population pyramid is an intriguing plot type, and already tells us that for most age groups, more women are readmitted. But this could be solely because there are more women than men in the sample. However, from the proportion barchart, we see here that proportion of readmitted women is greater than men, particularly for the 20-30 age group.

3.2.3 Patient diagnosesFinally, how are readmission rates distributed by patient and patient care features. For example, how is it distributed by patient primary diagnosis? In the final section of this post, we will leverage ggplot2’s visualization power to triangulate patient diagnoses with the key covariate and the response. Like in the previous section, we use proportions, adding the relevant labels to more easily infer that we see higher readmission rates for a diabetes diagnosis.

We change around quite a few of the plotting parameters in ggplot2 to make it look much more eye-catching.

```

How is readmitted rate distributed by diagnoses?sqldf('SELECT primary_diag, readmitted, COUNT() as n FROM results LEFT JOIN y USING(uid) GROUP BY primary_diag, readmitted') # SQL## primary_diag readmitted n## 1 circulatory <30 3485## 2 circulatory >30 10839## 3 circulatory NO 16113## 4 diabetes <30 1137## 5 diabetes >30 3318## 6 diabetes NO 4302## 7 other <30 5332## 8 other >30 15856## 9 other NO 26961## 10 respiratory <30 1403## 11 respiratory >30 5532## 12 respiratory NO 7488p3 <- results %>% left_join(y, by=join_by(uid)) %>% group_by(primary_diag, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n))ggplot(data=p3, aes(x=primary_diag, y=n, fill=readmitted)) + geom_bar(position = "dodge", stat = "identity", color="black") + scale_fill_manual(values=c("#999999", "#E69F00", "#56B4E9")) + theme_minimal() + labs(x="Primary diagnoses", y="Counts (proportions as labels)") + geom_text(aes(label = paste0(round(freq, 4) 100, "%")), position = position_dodge(width = 1), vjust=-0.7, size=3) + scale_y_continuous(labels = function(n) scales::comma(abs(n)))

`` 3.2.4 HbA1c measurementOne of the key questions this dataset seeks to answer is the *impact of the A1C test (decision to test) on readmission rates*, in the presence of covariates (especially the primary diagnosis). Output in its raw form (i.e. untransformed) doesn’t always give us the answer clearly. To get around this, we will useCASE WHENin SQL andmutateintidyverse`.

Let’s plot this in 2 ways – a barplot with labels, and a spineplot. The latter allows us to see the “weight” of the underlying categories.

```

What is the readmission rate profile of patients who had their A1C measured?sqldf('SELECT CASE WHEN A1Cresult = "None" THEN "not measured" ELSE "measured" END AS a1c, readmitted, COUNT() FROM results LEFT JOIN y USING(uid) GROUP BY a1c, readmitted ORDER BY 1') # SQL## a1c readmitted COUNT()## 1 measured <30 1676## 2 measured >30 5800## 3 measured NO 9542## 4 not measured <30 9681## 5 not measured >30 29745## 6 not measured NO 45322p4 <- results %>% left_join(y, by=join_by(uid)) %>% mutate(a1c=ifelse(A1Cresult=="None", "not measured", "measured")) %>% group_by(a1c, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) ggplot(data=p4, aes(x=a1c, y=freq, fill=readmitted)) + geom_col() + labs(x="HbA1c test measurement", y="proportions") + geom_text(aes(label = paste0(round(freq, 4) * 100, "%")), position = position_stack(vjust = 0.5), size=3)# dplyr

```

Spineplotlibrary(ggmosaic)p5 <- results %>% left_join(y, by=join_by(uid)) %>% left_join(dem, by=join_by(uid))%>% mutate(a1c=ifelse(A1Cresult=="None", "not measured", "measured")) %>% subset(gender=="Male"|gender=="Female")per <- p5 %>% group_by(a1c, readmitted) %>% summarise(n=n()) %>% mutate(freq = n / sum(n)) g <- ggplot(p5) + geom_mosaic(aes(x = product(a1c),fill = readmitted)) g + geom_text(data = ggplot_build(g)$data[[1]] %>% group_by(x__a1c) %>% mutate(pct = .wt/sum(.wt)*100), aes(x = (xmin+xmax)/2, y = (ymin+ymax)/2, label=paste0(round(pct, 2), "%")))

`` We observe a lower readmission rate (<30 days) when there is an A1C measurement taken, vs when it is not measured at all. In the 2nd/spineplot, we see this without actually calculating the percentages, while also inferring that number of patients not measured is much higher than those measured. We do however, manually add in the percentages to the spineplot to get a more complete picture on the relationship between HbA1c measurement and readmission rates.These are key findings which we will explore in greater detail, usingtidyverseandggplot2` more extensively, in the next part of this blog series, including cutting these plots across multiple covariates to explore how HbA1c affects readmissions in the presence of other patient groupings. Stay tuned!

To leave a comment for the author, please follow the link and comment on their blog: R Views.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A data analyst workflow, part 1: SQL & tidyverse

View Details

[This article was first published on r.iresmi.net, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.It’s cherry blossom time… A nice dataset going back to the year 812 in Kyoto can be found here. It describes the phenological data for full flowering date of cherry tree (Prunus jamasakura) in Kyoto, showing springtime climate changes.

Let’s draw…

```

params ------------------------------------------------------------------url_kyoto <- "http://atmenv.envi.osakafu-u.ac.jp/aono/kyophenotemp4/"file_kyoto <- "kyoto.rds" # cache dataicon_sakura <- "sakura.png" # cache icon# config ------------------------------------------------------------------library(tidyverse)library(fs)library(janitor)library(httr)library(rvest)library(glue)library(ggimage)Sys.setlocale("LC_TIME", "en_GB.UTF-8")# data --------------------------------------------------------------------# iconif (!file_exists(icon_sakura)) { GET("https://www.flaticon.com/download/icon/7096433?icon_id=7096433&author=232&team=232&keyword=Sakura&pack=packs%2Fsakura-festival-59&style=522&format=png&color=&colored=2&size=512&selection=1&premium=0&type=standard&search=cherry+blossom", write_disk(icon_sakura))}# blossom dates, scraped from the web page (more up to date than the xls files)if (!file_exists(file_kyoto)) { GET(url_kyoto) |> content() |> html_element("pre") |> html_text2() |> str_replace_all("\xc2\xa0", " ") |> # bad encoding needs correction read_fwf(fwf_cols("ad" = c(7, 10), "fifd" = c(17, 20), "fufd" = c(22, 25), "work" = c(27, 30), "type" = c(32, 35), "ref" = c(37, Inf)), skip = 26, na = c("", "NA", "-")) |> remove_empty() |> mutate(full_flowering_date = ymd(glue("{str_pad(ad, 4, 'left', '0')}{fufd}")), full_flowering_date_doy = yday(full_flowering_date)) |> write_rds(file_kyoto)}# plot --------------------------------------------------------------------read_rds(file_kyoto) |> mutate(random_size = sample(c(0.015, 0.02, 0.025, 0.03), length(ad), replace = TRUE)) |> ggplot(aes(ad, parse_date_time(full_flowering_date_doy, orders = "j"))) + geom_smooth(color = NA, fill = "chocolate4", alpha = 0.5) + geom_image(aes(size = I(random_size)), image = icon_sakura) + geom_smooth(color = "chocolate3", se = FALSE, alpha = 0.5) + scale_y_datetime(labels = scales::date_format("%b %d"), breaks = "weeks", minor_breaks = "days") + labs(title = "Cherry blossom", subtitle = "Kyoto", x = "year", y = "date", caption = glue("http://r.iresmi.net/ {Sys.Date()} data: {url_kyoto} icon by Vitaly Gorbachev")) + theme_minimal() + theme(plot.background = element_rect(fill = "cornflowerblue"), panel.grid = element_line(color = "dodgerblue"), text = element_text(color = "pink", family = "Ubuntu"), plot.title = element_text(size = 20), plot.caption = element_text(size = 7), axis.text = element_text(color = "pink"))ggsave("cherry_blossom.png", width = 25, height = 18, units = "cm", dpi = 300)

``` To leave a comment for the author, please follow the link and comment on their blog: r.iresmi.net.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cherry blossom

View Details

[This article was first published on R-Bloggers – Learning Machines, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
The topic of consciousness doesn’t cease to fascinate me. I have already written about it here: Will AI become conscious any time soon?. This essay could be seen as a continuation of my ongoing journey towards an integrated (holistic) worldview of mind and matter, that could be characterized as having a spiritual dimension to it, yet being grounded in science.

The essay was written with the help of ChatGPT-4, yet the ideas therein are my own. Of course, those ideas don’t come from nowhere, they can be seen as a culmination of my decadelong thinking about those topics. The most recent inspirations came especially from my colleagues Professor Phillip Goff, Durham University, and Professor Heinrich Päs, Technical University Dortmund. The picture was created with DALL-E.

IntroductionIn the realm of philosophy and physics, cosmopsychism and the Many Worlds Interpretation (MWI) of quantum mechanics are two intriguing ideas. Cosmopsychism posits that consciousness is a fundamental aspect of the cosmos, while the MWI suggests that every possible outcome of a quantum event results in the creation of new, parallel universes. This essay explores the combination of these two concepts, arguing that the all-encompassing cosmic consciousness seeks to experience all possible states of mind and emotion through the existence of multiple worlds, with quantum mechanics providing the mechanism for this process. By presenting consciousness as an essential aspect of existence, this view offers a monistic perspective that contrasts with traditional dualistic philosophies.

Cosmopsychism: A Brief OverviewCosmopsychism is a philosophical position that asserts consciousness is an intrinsic and fundamental aspect of the universe. Instead of viewing consciousness as emerging from complex physical processes, cosmopsychism sees it as a primary quality of reality. The idea proposes that all entities within the universe, including humans, are part of a greater cosmic consciousness that forms the basis of all existence. This perspective challenges the dualistic notion of a divide between the mental and physical realms, suggesting instead that they are deeply interconnected aspects of a single, unified reality.

The Many Worlds Interpretation of Quantum MechanicsThe MWI of quantum mechanics is a theory that aims to explain the seemingly strange and paradoxical behavior of particles at the quantum level. According to the MWI, when a quantum event occurs, the universe splits into parallel branches, with each branch representing a possible outcome of the event. Consequently, an infinite number of parallel universes exist, each embodying a different configuration of reality. In this framework, all potential states of existence are simultaneously actualized, allowing for a vast array of experiences and perspectives to coexist.

The Union of Cosmopsychism and the Many Worlds InterpretationThe connection between cosmopsychism and the MWI can be illustrated by envisioning the cosmic consciousness as a vast experiential landscape, eager to explore the full range of emotions, thoughts, and perspectives through the medium of countless parallel universes. Quantum mechanics, as described by the MWI, provides the mechanism by which this can be accomplished. Each quantum event generates an array of potential outcomes, effectively multiplying the number of available experiences and enabling the cosmic consciousness to explore every conceivable state of mind.

In this unified model, our individual consciousness is an inseparable component of the greater cosmic consciousness, with our unique experiences contributing to the totality of reality. As such, our personal encounters with the world around us are not isolated occurrences but rather integral aspects of the broader, interconnected tapestry of existence. By positioning consciousness as a fundamental component of reality, this monistic approach eliminates the need for a dualistic divide between the mental and the physical, offering a more cohesive and holistic understanding of the universe.

ConclusionThe fusion of cosmopsychism and the Many Worlds Interpretation presents a fascinating and profound perspective on the nature of consciousness and the workings of the universe. By envisioning the cosmic consciousness as an entity seeking to experience all possible states of mind through the mechanism of quantum mechanics, this approach offers a compelling monistic alternative to dualistic philosophies. In this view, our consciousness and the physical world are not separate entities, but rather, interconnected aspects of a single, all-encompassing reality. This perspective not only enriches our understanding of the cosmos but also deepens our appreciation for the intricate and boundless tapestry of existence.

I am looking forward to your comments below!

To leave a comment for the author, please follow the link and comment on their blog: R-Bloggers – Learning Machines.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Cosmopsychism and the Many Worlds Interpretation: A Monistic Perspective on Consciousness and Quantum Mechanics

View Details

[This article was first published on r.iresmi.net, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This old post sees a little traffic from search engines but is a mess after many editions due to the packages evolutions.

So, how can we (chose your term) append, merge, union or combine many shapefiles or other spatial vector data in 2023 with R, preferably using tidyverse functions ?

For good measure, we want to add the source file as an attribute.

First, make some data using the geopackage available here. We generate several shapefiles (one per région) in a temporary directory :

dep <- sf::read\_sf("~/data/adminexpress/adminexpress\_cog\_simpl\_000\_2022.gpkg", layer = "departement")dep |> dplyr::group\_by(insee\_reg) |> dplyr::group\_walk(\(grp\_data, grp\_name) sf::write\_sf(grp\_data, glue::glue("~/temp/reg\_{grp\_name}.shp"))) This is the wayCurrent and concise.

fs::dir\_ls("~/temp", regexp = ".*\\.shp$") |> purrr::map(\(f) sf::read\_sf(f) |> dplyr::mutate(source = f, .before = 1)) |> dplyr::bind\_rows() ApprovedRecommended, as seen in the purrr help on map_dfr (that I liked better, see below) but verbose because we have to specify that we want a sf-tibble which is lost in translation.

fs::dir\_ls("~/temp", regexp = ".*\\.shp$") |> purrr::map\_dfr(\(f) sf::read\_sf(f) |> dplyr::mutate(source = f, .before = 1)) OlderI liked it, too.

fs::dir\_ls("~/temp", regexp = ".*\\.shp$") |> dplyr::tibble(source = \_) |> dplyr::mutate(shp = purrr::map(source, sf::read\_sf)) |> tidyr::unnest(shp) |> sf::st\_sf() OldestIf all the files share the same attributes structure.

fs::dir\_ls("~/temp", regexp = ".*\\.shp$") |> purrr::map(\(f) sf::read\_sf(f) |> dplyr::mutate(source = f, .before = 1)) |> do.call(what = rbind) or

fs::dir\_ls("~/temp", regexp = ".*\\.shp$") |> purrr::map(\(f) sf::read\_sf(f) |> dplyr::mutate(source = f, .before = 1)) |> purrr::reduce(rbind) To leave a comment for the author, please follow the link and comment on their blog: r.iresmi.net.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Open and merge multiple shapefiles, updated

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn this blog post, we’ll walk through how to create a shiny application that allows users to analyze the weekly returns of FAANG stocks (AAPL, AMZN, FB, GOOGL, and NFLX) using the {tidyquant} and {TidyDensity} packages in R.

Section 1: Package and UI SetupThe first section of the code sets up the necessary R packages and creates the UI for the shiny app. The packages we’ll be using are:

shiny: for creating interactive web applications in R
tidyquant: for easily getting and analyzing financial data in R
TidyDensity: for computing and visualizing probability distributions in a tidy way
dplyr: for manipulating data in a tidy way
DT: for creating interactive and scrollable data tables

Analysts assemble your packages!

library(shiny)library(tidyquant)library(TidyDensity)library(dplyr)library(DT) The UI consists of a title panel, a sidebar panel, and a main panel. The sidebar panel contains a select input that allows users to choose which FAANG stock to analyze, as well as a numeric input for the number of simulations to run. The main panel contains two sections: one for the tidy_autoplot() output (a plot of the stock returns), and one for the tidy_empirical() output (a table of the log returns).

Section 2: Server SetupThe second section of the code defines the server function for the shiny app. The server function takes the input values from the UI (i.e. the selected stock and number of simulations) and uses them to get and analyze the stock data.

To get the stock data, we use the tq_get() function from the tidyquant package to retrieve the adjusted stock prices for the selected security from January 1, 2010 to the present. We then use the tq_transmute() function to compute the weekly log returns of the stock and rename the resulting column to “log_return”.

The tidy_empirical() function from the TidyDensity package is used to compute the empirical distribution of the log returns. The resulting table is displayed using the renderDT() function from the DT package, which creates a scrollable data table that can be sorted and filtered.

The tidy_autoplot() function is used to create a plot of the log returns, which is displayed using the renderPlot() function.

Section 3: Running the AppThe final section of the code runs the shiny app using the ui and server functions.

Overall, this shiny app provides a simple and interactive way for users to analyze the weekly returns of FAANG stocks using tidyquant and TidyDensity in R. By allowing users to choose which stock to analyze and how many simulations to run, the app provides a customizable way to explore the empirical distributions of the log returns.

ExampleHere is a full working example, except for FB for some reason that symbol errors out for me, go figure:

library(shiny)library(tidyquant)library(TidyDensity)library(dplyr)library(DT)# define UIui <- fluidPage( titlePanel("FAANG Stock Analysis"), sidebarLayout( sidebarPanel( selectInput("ticker", "Select a security:", choices = c("AAPL", "AMZN", "FB", "GOOGL", "NFLX"), selected = "AAPL"), numericInput("num\_sims", "Enter the number of simulations:", value = 1, min = 1, max = 25, step = 1) ), mainPanel( h3("Tidy Autoplot Output"), plotOutput("autoplot\_output"), h3("Tidy Empirical Output"), DTOutput("empirical\_output") ) ))# define serverserver <- function(input, output) { # get stock data using tq\_get and tq\_transmute stock\_data <- reactive({ tq\_get(input$ticker, get = "stock.prices", from = "2010-01-01") %>% tq\_transmute(select = adjusted, mutate\_fun = periodReturn, period = "weekly", type = "log", col\_rename = "log\_return") }) # output tidy\_empirical results output$empirical\_output <- renderDT({ datatable( tidy\_empirical(stock\_data()$log\_return, .num\_sims = input$num\_sims), options = list(scrollY = "250px") ) }) # output tidy\_autoplot results output$autoplot\_output <- renderPlot({ stock\_data()$log\_return %>% tidy\_empirical(.num\_sims = input$num\_sims) %>% tidy\_autoplot() }) }# run the appshinyApp(ui = ui, server = server) Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Looking at Daily Log Returns with tidyquant, TidyDensity, and Shiny

View Details

[This article was first published on R on ArData, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.* Usages * The table for the illustration + Preparation + The flextable + Annotations * RTF generation Version 0.9 of the ‘flextable’ package adds the possibility to produce tables in RTF documents.

This format is rare but mandatory for some communities, i.e. mainly in the pharmaceutical industry to our knowledge.

UsagesTwo methods can be used to embed flextables in an RTF file.

You can use the save_as_rtf() function which produces an RTF file from one or more flextables.The save_as_rtf() function is a utility function of the `flexable’ package.The documentation of the function can be found at at this address: https://davidgohel.github.io/flextable/reference/save_as_rtf.html.

If the output is not adapted to your needs, you can use the functionsin the ‘officer’ package dedicated to RTF file production,i.e. officer::rtf_doc() and officer::rtf_add().The documentation for rtf_add() can be found at thisaddress: https://davidgohel.github.io/officer/reference/rtf_add.html.

In the following we will show the use of save_as_rtf() and the result produced.

The table for the illustrationThe French news has highlighted some public datasets about theprime ministers of the fifth republic and the uses of the 49-3 article.We have experimented with the data to produce a simple table with flextable.

The data can be downloaded in rds format aswell as the script allowing to obtain thesedata: src-history-49-3.R.

x <- readRDS("history-49-3.RDS")head(x)#> couleur name n\_nominations debut fin duration#> 1 CENT Élisabeth Borne 1 2022-05-16 2023-03-25 313#> 2 CENT Jean Castex 1 2020-07-03 2022-05-16 682#> 3 CENT Édouard Philippe 2 2017-05-15 2020-07-03 1145#> 4 GAU Bernard Cazeneuve 1 2016-12-06 2017-05-10 155#> 5 GAU Manuel Valls 2 2014-03-31 2016-12-06 981#> 6 GAU Jean-Marc Ayrault 2 2012-05-15 2014-03-31 685#> n\_eng\_resp n\_text n\_censor\_motion#> 1 11 3 14#> 2 NA NA NA#> 3 1 1 2#> 4 NA NA NA#> 5 6 2 3#> 6 NA NA NA PreparationFirst we declare some general settings for flextable.

set\_flextable\_defaults( font.family = "Open Sans", font.color = "#333333", big.mark = "", fmt\_date = "%Y", na\_str = "", theme\_fun = theme\_vanilla) The political side of the minister is indicated with a circle whosecolor is the color generally used in the media. These shapes mustbe prepared first, the ‘grid’ package is used for this.

library(grid)library(dplyr)x$couleur <- mapply(function(z) { col <- case\_when( z %in% "GAU" ~ "#FF8080", z %in% "DTE" ~ "#143c77", z %in% "CENT" ~ "#ff9900", TRUE ~ "transparent" ) circleGrob(gp = gpar(fill = col, col = "transparent"))}, z = x$couleur, SIMPLIFY = FALSE, USE.NAMES = FALSE) The flextableNow we can define the table with the functions of ‘flextable’.

z <- flextable( data = x, col\_keys = c("name", "couleur", "exercice", "n\_eng\_resp", "n\_text", "n\_censor\_motion")) |> mk\_par(j = "exercice", value = as\_paragraph( as\_i(debut), as\_i("-"), as\_i(fin), as\_i(as\_bracket(duration, p = " (", s = " jours)")))) |> mk\_par(j = "couleur", value = as\_paragraph(grid\_chunk(couleur, width = .15, height = .15))) |> align(j = "name", align = "right", part = "all") |> align(j = c("n\_eng\_resp", "n\_text", "n\_censor\_motion"), align = "center", part = "all") |> set\_header\_labels(name = "Premier ministre", period = "Période", couleur = "", n\_eng\_resp = "Recours\nau 49-3", n\_text = "Textes\nconcernés", n\_censor\_motion = "Motions\nde censure", longeur = "Nombre de jours\nen fonction") |> autofit() |> width(j = "couleur", width = .3)z

| Premier ministre | Recoursau 49-3 | Textesconcernés | Motionsde censure | | --- | --- | --- | --- | | Élisabeth Borne | | 2022-2023 (313 jours) | 11 | 3 | 14 | | Jean Castex | | 2020-2022 (682 jours) | | Édouard Philippe | | 2017-2020 (1145 jours) | 1 | 1 | 2 | | Bernard Cazeneuve | | 2016-2017 (155 jours) | | Manuel Valls | | 2014-2016 (981 jours) | 6 | 2 | 3 | | Jean-Marc Ayrault | | 2012-2014 (685 jours) | | François Fillon | | 2007-2012 (1820 jours) | | Dominique de Villepin | | 2005-2007 (712 jours) | 1 | 1 | 0 | | Jean-Pierre Raffarin | | 2002-2005 (1121 jours) | 2 | 2 | 2 | | Lionel Jospin | | 1997-2002 (1799 jours) | | Alain Juppé | | 1995-1997 (747 jours) | 2 | 2 | 2 | | Édouard Balladur | | 1993-1995 (773 jours) | 1 | 1 | 1 | | Pierre Bérégovoy | | 1992-1993 (361 jours) | 3 | 3 | 1 | | Edith Cresson | | 1991-1992 (323 jours) | 8 | 4 | 2 | | Michel Rocard | | 1988-1991 (1100 jours) | 28 | 13 | 5 | | Jacques Chirac | | 1986-1988 (782 jours) | 8 | 7 | 7 | | Laurent Fabius | | 1984-1986 (611 jours) | 4 | 2 | 1 | | Pierre Mauroy | | 1981-1984 (1153 jours) | 7 | 5 | 6 | | Raymond Barre | | 1976-1981 (1722 jours) | 8 | 5 | 13 | | Jacques Chirac | | 1974-1976 (821 jours) | | Pierre Messmer | | 1972-1974 (691 jours) | | Jacques Chaban-Delmas | | 1969-1972 (1111 jours) | | Maurice Couve de Murville | | 1968-1969 (345 jours) | | Georges Pompidou | | 1962-1968 (2279 jours) | 6 | 2 | 4 | | Michel Debré | | 1959-1962 (1192 jours) | 4 | 2 | 4 |

AnnotationsIt only remains to add some annotations and the tablecan be sent in an RTF document.

z <- add\_header\_lines(z, values = as\_paragraph( as\_chunk("Historique des recours au 49-3 pendant la V république", prop = fp\_text\_default(font.size = 13, bold = TRUE) )) ) |> footnote(i = 1, j = 1, ref\_symbols = "(1)", part = "header", value = as\_paragraph( as\_chunk(" source : "), as\_i("https://www.assemblee-nationale.fr") ) ) |> footnote(i = 1, j = 1, ref\_symbols = "(2)", part = "header", value = as\_paragraph( as\_chunk(" source : "), as\_i("https://www.politiquemania.com/longevite-premier-ministre.html") )) |> footnote(i = 2, j = c(4, 6), ref\_symbols = "(3)", part = "header", value = as\_paragraph( as\_chunk("Fréquence ") )) |> footnote(i = 2, j = 5, ref\_symbols = "(4)", inline = TRUE, part = "header", value = as\_paragraph( as\_chunk("Nombre de cas") )) |> hline(i = 1, part = "header", border = fp\_border\_default(width = 0)) RTF generationBecause the table is quite large, we use a ‘landscape’ orientation; the header and footer are also augmented with some information including automatic page numbering.

These settings are made with the officer::prop_section() function.

library(officer)sect\_properties <- prop\_section( page\_size = page\_size( width = 8.3, height = 11.7, orient = "landscape" ), header\_default = block\_list(fpar("flextable and RTF")), footer\_default = block\_list(fpar("page ", run\_word\_field(field = "PAGE \\* MERGEFORMAT"))), page\_margins = page\_mar(bottom = .6, top = .6, right = .5, left = .5)) The call to the save_as_rtf() function finally produces the RTF file.

save\_as\_rtf( "Recours au 49-3" = z, pr\_section = sect\_properties, path = "49-3.rtf") The result can be seen below:

overview of 49-3.rtf

To leave a comment for the author, please follow the link and comment on their blog: R on ArData.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: flextable supports RTF

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Daniel Fischer of the UseR Chile Group recently talked to the R Consortium about the use of R for Data Analysis in Chile. He shared that even though there is a constant comparison of R with Python, R has a very loyal user base in Chile. Daniel also uses R for his work in the agricultural chemicals industry and teaches R through his YouTube channel, blog, and University of Andes.

Daniel Fischer,
Corporative Head of Advanced Analytics and Data Engineering at Anasac.

Please share about your background and your involvement in the R Community? What is your level of experience with the R language?

I am an industrial engineer by education. Here in Chile, it’s a six-year degree that is a mix of engineering and management. I have been working for the last 12 years in data analysis-related jobs. I am currently the Corporate Head of Advanced Analytics and Data Engineering at ANASAC. I am also a Business Analytics and Optimization Professor at the Universidad de los Andes (CL). I also use R for teaching.

Around 10 years ago, I did some research and I realized R is a really powerful and easy tool. I started looking for other people using R for their work and founded the useRChile. We were a small group and used to meet in a pub to talk about what we can do as a group. After some time, with help from MetricArts by EY, we organized monthly meetups. Most people with a scientific background, statisticians, and people interested in machine learning or networking attended our meetups.

We used to have two presentations in each meetup and recorded and uploaded them to our YouTube channel. We stopped having meetings a year ago because of the pandemic. But the whole organizing team is still in contact, and we will start organizing meetups again soon. Besides the group, I also teach R through my YouTube channel and blog.

What industry are you currently in? How do you use R in your work?

I currently work for an international agricultural chemicals-producing company. We use R for everything, as it is our main Extract, Transform, Load (ETL) tool. When I started working for the company, I told the team that we can use R or Python. We should not use SQL because it is hard to debug. I wanted something you could run line-by-line like R or Python. The whole team decided to use R because it is much easier. A couple of them already knew Python. But when they saw how much easier R was using dplyr and the whole dataverse, they just said Python is too complicated. So we have all our processes in R and various small libraries in Python which are shared with R using Reticulate.

We built most of the data warehouse data pipeline in R and we built the web applications in Shiny. There is also ongoing work on a real-time recommendation system. We have also built a set of tools to interact with databases which I want to make open-source in CRAN.

In Chile, Databricks is very popular and widely used in industry. Most people use it with Python because it’s cheaper, but there are many bricks built in R too.

Process panel built in R+Shiny. The form allows users to trigger processes, which saves time for the analytics team.

What trends do you currently see in R language and your industry? Any trends you see developing in the near future?

There is a lot of comparison between R and Python and many people believe that Python is much superior. But I think R is much superior for data analysis. People who already use R for data analysis stick to it and when new users learn about the power of R they also prefer it over Python. Even Julia says it wants to be as good for analysis as R and fast as C and flexible as python.

So I think while R might not be as big as Python, it’s like comparing a swiss pocket knife to a big knife used for chopping meat. If you want to cut the meat you are going to need a big knife. R is the big and specialized knife for data analysis. I am hopeful as new people get introduced to R, the user base will also increase with time.

Why do industry professionals come to your user group? What is the benefit for attending?

Our group is appealing to industry professionals as it helps them learn about interesting topics presented by speakers who are experts in their fields. Besides learning, they also get to meet people with similar interests and build their network. We also used to hang out after the meetings at a pub.

How do I Join?

Apply HereThe post Use of R in Agricultural Chemical Industry in Chile appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Use of R in Agricultural Chemical Industry in Chile

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionHello! In this code, we are making a program that will help us predict the number of air passengers in the future. Let me explain what each part of the code does, step by step.

First, we need to load some tools that will help us create the program. These tools are called “packages.” We use the library() function to load them. The packages we need are called shiny, forecast, and ggplot2.

Load Libraries library(shiny)library(forecast)library(ggplot2) DataNext, we need some data to work with. We will use a dataset of the number of air passengers each month from 1949 to 1960. We load this dataset using the data() function.

data(AirPassengers) User InterfaceNow, we need to create the user interface, or UI. This is what the user will see and interact with. In this case, we will create a simple app with a title, a dropdown menu to choose a forecasting model, and a plot and table to display the forecast results. We use the fluidPage() function to create the UI, and we define the UI elements inside it.

ui <- fluidPage( titlePanel("AirPassengers Forecast"), sidebarLayout( sidebarPanel( selectInput(inputId = "model", label = "Choose a model:", choices = c("auto.arima", "ets", "holtwinters")) ), mainPanel( plotOutput(outputId = "forecast\_plot"), tableOutput(outputId = "forecast\_table") ) )) ServerNow, we need to define the server. The server is where the program does the calculations and generates the output based on what the user selects in the UI. We define the server inside the function(input, output) argument.

server <- function(input, output) { Inside the server, we need to create a reactive expression that generates the forecast based on the model the user selects. We use an if statement to check which model the user selected, and then we use the corresponding function to generate the forecast.

Forecast the Data forecast\_data <- reactive({ if (input$model == "auto.arima") { fit <- auto.arima(AirPassengers) forecast(fit) } else if (input$model == "ets") { fit <- ets(AirPassengers) forecast(fit) } else { fit <- hw(AirPassengers) forecast(fit) } }) Render PlotThe renderPlot() function tells the program to create a plot based on the reactive expression we defined earlier. We use the plotOutput() function in the UI to display the plot.

output$forecast\_plot <- renderPlot({ plot(forecast\_data()) }) Similarly, the renderTable() function tells the program to create a table based on the reactive expression we defined earlier. We use the tableOutput() function in the UI to display the table.

output$forecast\_table <- renderTable({ forecast\_data()$mean }) Finally, we run the app using the shinyApp() function, with the UI and server arguments.

shinyApp(ui = ui, server = server) And that’s it! This program allows the user to choose a forecasting model, and then generates a plot and table with the predicted number of air passengers based on that model.

Here is the Full code block”

```

Load required packageslibrary(shiny)library(forecast)library(ggplot2)# Load AirPassengers datasetdata(AirPassengers)# Define UIui <- fluidPage( # Title of the app titlePanel("AirPassengers Forecast"), # Sidebar with input controls sidebarLayout( sidebarPanel( selectInput(inputId = "model", label = "Choose a model:", choices = c("auto.arima", "ets", "holtwinters")) ), # Output plot and table mainPanel( plotOutput(outputId = "forecast_plot"), tableOutput(outputId = "forecast_table") ) ))# Define serverserver <- function(input, output) { # Reactive expression to create forecast based on selected model forecast_data <- reactive({ if (input$model == "auto.arima") { fit <- auto.arima(AirPassengers) forecast(fit) } else if (input$model == "ets") { fit <- ets(AirPassengers) forecast(fit) } else { fit <- hw(AirPassengers) forecast(fit) } }) # Output plot output$forecast_plot <- renderPlot({ plot(forecast_data()) #checkresiduals(forecast_data()) }) # Output table output$forecast_table <- renderTable({ forecast_data()$mean })}# Run the appshinyApp(ui = ui, server = server)

``` Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A sample Shiny App to view Forecasts on the AirPassengers Data

View Details

[This article was first published on R | TypeThePipe, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.IntroductionIn data analysis and data science, it’s common to work with large datasets that require some form of manipulation to be useful. In this small article, we’ll explore how to create and modify columns in a dataframe using modern R tools from the tidyverse package. We can do that on several ways, so we are going from basic to advanced level.

Let’s use the starwars dataset for that purpose:

data("starwars")head(starwars, 4)# # A tibble: 4 × 8# name height mass hair\_color skin\_color eye\_color birth\_year sex # <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> # 1 Luke Skywalker 172 77 blond fair blue 19 male # 2 C-3PO 167 75 NA gold yellow 112 none # 3 R2-D2 96 32 NA white, blue red 33 none # 4 Darth Vader 202 136 none white yellow 41.9 male The most basic example is using mutate to create and modify variables.

starwars %>% mutate( height = height * 2, new\_numeric\_column = row\_number(), new\_char\_column = "This variable is new" ) %>% select(name, height, new\_numeric\_column, new\_char\_column) %>% head(4)# # A tibble: 4 × 4# name height new\_numeric\_column new\_char\_column # <chr> <dbl> <dbl> <chr> # 1 Luke Skywalker 344 1 This variable is new# 2 C-3PO 334 2 This variable is new# 3 R2-D2 192 3 This variable is new# 4 Darth Vader 404 4 This variable is new Note how the second variable we created is recycled to match the length of the dataset.But you already knew that, right?

Common tricksA common trick is making use of if_else to conditionally modify some variables. I use this structure on a daily basis.

starwars %>% mutate(height = if\_else(hair\_color == "blond", height * 10, height)) %>% select(name, height, mass, hair\_color) %>% head(4)# # A tibble: 4 × 8# name height mass hair\_color # <chr> <int> <dbl> <chr> # 1 Luke Skywalker 344 77 blond # 2 C-3PO 167 75 NA # 3 R2-D2 96 32 NA # 4 Darth Vader 202 136 none Another common use case is to rely on the case_when function to modify the variable based on several conditions:

starwars %>% mutate(mod\_height = case\_when(eye\_color == "blue" ~ height/2, eye\_color == "red" ~ height*2, eye\_color == "yellow" ~ height*10, TRUE ~ height)) %>% select(name, height, eye\_color, mod\_height) %>% head(4)# name height eye\_color mod\_height# <chr> <int> <chr> <dbl># 1 Luke Skywalker 172 blue 86# 2 C-3PO 167 yellow 1670# 3 R2-D2 96 red 192# 4 Darth Vader 202 yellow 2020 Note that we should end the case_when with an option that always yields TRUE because the conditions are evaluated in order. If our data doesn’t meet any condition we are leaving the column as is.

All these are fairly basic examples. Let’s go with the dplyr advanced way of creating and modifying variables.

The Advanced Way: Using across()In modern R, we can simultaneously modify several columns at once using the verb across. We need to pass the transformation we will be performing on those variables as well. For that, we are using a lambda function which basically means that we are creating the function on-the-fly but we are not storing it.

starwars %>% mutate(across(c(height, mass, birth\_year), ~ .x*10)) %>% select(name, height, mass, birth\_year) %>% head(4)# # A tibble: 4 × 4# name height mass birth\_year# <chr> <dbl> <dbl> <dbl># 1 Luke Skywalker 1720 770 190# 2 C-3PO 1670 750 1120# 3 R2-D2 960 320 330# 4 Darth Vader 2020 1360 419 That’s quite nice but sometimes you don’t want to modify the existing columns but creating new ones.This is an important use case: batch-creating several columns at once based on the existing ones. I already discussed how to do it in How to create multiple lags like a Pro. We can use the .names argument to dynamically specify the new column names, like this:

starwars %>% mutate(across(c(height, mass, birth\_year), ~ .x * 100, .names = "{.col}\_x100")) %>% select(name, height, mass, birth\_year, height\_x100, mass\_x100, birth\_year\_x100) %>% head(4)# # A tibble: 4 × 7# name height mass birth\_year height\_x100 mass\_x100 birth\_year\_x100# <chr> <int> <dbl> <dbl> <dbl> <dbl> <dbl># 1 Luke Skywalker 172 77 19 17200 7700 1900# 2 C-3PO 167 75 112 16700 7500 11200# 3 R2-D2 96 32 33 9600 3200 3300# 4 Darth Vader 202 136 41.9 20200 13600 4190 Awesome, right? However, I still had to type them all manually. There is a better way.

The Pro Way: Using across() + tidyselectorsWhat if we want to modify a lot of columns? There must be a better way to avoid having to type them all…Sure there is! tidyselectors to the rescue! Those are a family of functions that allow us to dynamically select several columns based on a condition. Let’s see that with an example.

Let’s say we want to modify only the numerical variables. We can do that easily with the help of where function and the neat part is this family of functions works with several verbs of the Tidyverse. For instance, they work with select:

starwars %>% select(name, where(is.numeric)) %>% head(4)# # A tibble: 4 × 4# name height mass birth\_year# <chr> <int> <dbl> <dbl># 1 Luke Skywalker 172 77 19 # 2 C-3PO 167 75 112 # 3 R2-D2 96 32 33 # 4 Darth Vader 202 136 41.9 But also with mutate! So combining across with where we can apply the function only over the desired columns (without having to type them!)

starwars %>% mutate(across(where(is.numeric), ~ .x * 100)) %>% select(name, where(is.numeric)) %>% head(4)# # A tibble: 4 × 4# name height mass birth\_year# <chr> <dbl> <dbl> <dbl># 1 Luke Skywalker 17200 7700 1900# 2 C-3PO 16700 7500 11200# 3 R2-D2 9600 3200 3300# 4 Darth Vader 20200 13600 4190 Note how the name feature hasn’t been modified, as it is not a numeric variable. This is a really handy trick specially when you are working with big datasets and need to perform an operation on many columns at once.

Also, it is worth noting that we can pass any function to across to modify the selected columns. We don’t necessarily have to define the operation with a lambda function, but any existing function can be used.

Here is another powerful example working with character columns. We can apply an existing function to make all of them uppercase:

starwars %>% mutate(across(where(is.character), toupper)) %>% select(where(is.character)) %>% head(4)# # A tibble: 4 × 8# name hair\_color skin\_color eye\_color sex gender homeworld species# <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> # 1 LUKE SKYWALKER BLOND FAIR BLUE MALE MASCULINE TATOOINE HUMAN # 2 C-3PO NA GOLD YELLOW NONE MASCULINE TATOOINE DROID # 3 R2-D2 NA WHITE, BLUE RED NONE MASCULINE NABOO DROID # 4 DARTH VADER NONE WHITE YELLOW MALE MASCULINE TATOOINE HUMAN Also, you don’t have to rely only on the where tidyselector, you can use many others like contains, matches, starts_with or ends_with.

Here’s another example example using ends_with:

starwars %>% mutate(across(ends\_with("\_color"), ~ paste0("the color is ", .x))) %>% select(name, ends\_with("\_color")) %>% head(4)# # A tibble: 4 × 4# name hair\_color skin\_color eye\_color # <chr> <chr> <chr> <chr> # 1 Luke Skywalker the color is blond the color is fair the color is blue # 2 C-3PO the color is NA the color is gold the color is yellow # 3 R2-D2 the color is NA the color is white, blue the color is red # 4 Darth Vader the color is none the color is white the color is yellow Handy stuff, right? There is so much more possibilities to discover. You can read more about it on the across reference.

Suscribe for more R tips!To leave a comment for the author, please follow the link and comment on their blog: R | TypeThePipe.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How to Modify Variables the Right Way in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionIn this blog post, we will be discussing how to create a Shiny application in R that will download and extract data from a zip file and allow users to choose which data they would like to see presented to them in the app from a selection drop-down menu. We will be using the current_hosp_data() function to obtain and read in the data. This function is in the upcoming release for the {healthyR.data} package.

ExampleInstall install.packages(c("shiny","shinythemes")) Current Hospital DataHere is the current_hospital_data() function:

current\_hosp\_data <- function() { # URL for file url <- "https://data.cms.gov/provider-data/sites/default/files/archive/Hospitals/current/hospitals\_current\_data.zip" # Create a temporary directory to process the zip file tmp\_dir <- tempdir() download\_location <- file.path(tmp\_dir, "download.zip") extract\_location <- file.path(tmp\_dir, "extract") # Download the zip file to the temporary location utils::download.file( url = url, destfile = download\_location ) # Unzip the file utils::unzip(download\_location, exdir = extract\_location) # Read the csv files into a list csv\_file\_list <- list.files( path = extract\_location, pattern = "\\.csv$", full.names = TRUE ) # make named list csv\_names <- stats::setNames( object = csv\_file\_list, nm = csv\_file\_list |> basename() |> gsub(pattern = "\\.csv$", replacement = "") |> janitor::make\_clean\_names() ) # Process CSV Files parse\_csv\_file <- function(file) { # Normalize the path to use C:/path/to/file structure normalizePath(file, "/") |> # read in the csv file and use check.names = FALSE because some of # the names are very long utils::read.csv(check.names = FALSE) |> dplyr::as\_tibble() |> # clean the field names janitor::clean\_names() } list\_of\_tables <- lapply(csv\_names, parse\_csv\_file) unlink(tmp\_dir, recursive = TRUE) # Return the tibbles # Add and attribute and a class type to the object attr(list\_of\_tables, ".list\_type") <- "current\_hosp\_data" class(list\_of\_tables) <- c("current\_hosp\_data", class(list\_of\_tables)) list\_of\_tables} App FileNext, let’s create a new file called app.R. In this file, we will create the Shiny app. The app will have a user interface (UI) and a server.

The UI is responsible for creating the layout of the app, while the server is responsible for processing the data and responding to user input.

First, let’s create the UI. The UI will consist of a drop-down menu that will allow users to choose which data they would like to see presented to them in the app.

library(shiny)library(shinythemes)hosp\_data <- current\_hosp\_data()ui <- fluidPage(theme = shinytheme("cerulean"), # Set up the dropdown menu selectInput(inputId = "table", label = "Select a table:", choices = names(hosp\_data), selected = NULL), # Set up the table output tableOutput(outputId = "table\_output")) The fluidPage() function creates a new Shiny app page. We also specify the theme using the {shinythemes} package. The selectInput() function creates the drop-down menu, which allows users to select which data they would like to see presented to them in the app. The choices argument is set to the names of the tables in the current_hosp_data() object. The tableOutput() function creates the output for the selected table.

ServerNext, let’s create the server. The server will be responsible for processing the data and generating the output based on user input.

server <- function(input, output) { # Load the data into a reactive object data <- reactive(hosp\_data) # Set up the table output output$table\_output <- renderTable({ # Get the selected table table\_selected <- input$table # Get the table from the data object table\_data <- data()[[table\_selected]] # Return the table data table\_data })} The reactive() function is used to create a reactive object that will load the data when the app starts. The renderTable() function generates the output for the selected table. It does this by getting the selected table from the drop-down menu, getting the table data from the reactive data object, and returning the table data.

Shiny AppFinally, we need to run the appl using the shinyApp() function:

shinyApp(ui = ui, server = server) Pros and ConsPros:

The app is easy to use, and users can quickly select which data they would like to see presented to them in the app.
The current_hosp_data() function is only called once when the app starts, which can save time and resources if the function is time-consuming or resource-intensive.

Cons:

The app will not update if the data in the zip file changes. Users will need to restart the app to see the updated data.
The app loads all the data into memory when it starts, which can be an issue if the data is large and memory-intensive.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A sample Shiny App to view CMS Healthcare Data

View Details

[This article was first published on R on ArData, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Export table as png * Function gen_grob + Simple usage + The resizing options + Content wrapping One of the great features of the flextable package is the ability to workwith ‘Grid graphics’ format. The gen_grob() function generates a grob*(Graphical Object) from a flextable, which can then be added to a ggplot graphicusing the ‘patchwork’ package or with ggplot2::annotation_custom().

These features require the use of a graphical output using‘systemfonts’: svglite::svglite(), ragg::agg_png() or ggiraph::dsvg().These devices must be used to ensure all fonts you are using will be recognizedby the R graphical device.

The graph shown above is produced with the function gen_grob().
the associated code is available in this articleof the ‘flextable gallery’.

Export table as pngPast versions of ‘flextable’ used packages ‘webshot’ or ‘webshot2’ to get an imageversion of the HTML table. This mechanism has been removed and replaced by theuse of ‘Grid graphics’ output and the ‘ragg’ package. This allows for a fasterprocess and makes maintenance easier.

First, we define some global parameters.

library(flextable)library(gdtools)set\_flextable\_defaults( font.family = "Open Sans", font.color = "#333333", theme\_fun = theme\_vanilla, digits = 2, big.mark = "", na\_str = "")register\_gfont("Open Sans") # register Open Sans To export a flextable to a png file, we need to call the save_as_image() function.Let’s illustrate with an flextable made with the dataset ‘palmerpenguins::penguins’.

library(palmerpenguins)filename <- "head-penguins.png"ft <- as\_flextable(palmerpenguins::penguins)save\_as\_image(ft, path = filename)knitr::include\_graphics(filename) Function gen_grobLet’s illustrate with the previous ‘flextable’ and a new ‘ggplot’.

library(tidyverse)g1 <- ggplot(penguins, aes(x = bill\_length\_mm, y = bill\_depth\_mm, color = species)) + geom\_point() + scale\_color\_viridis\_d() + theme\_minimal() + theme(plot.margin = unit(c(0, 0, 30, 0), "pt")) Simple usageThe function gen_grob() will by default adapt to the space available for the table.

library(patchwork)g1 / gen\_grob(ft, just = "bottom") It should work with any function that support grid objects:

g1 + inset\_element(gen\_grob(ft |> bg(bg = "#EFEFEFAA", part = "all"), just = "bottom"), 0.1, 0.25, .9, .75) + theme( plot.background = element\_rect(fill = "transparent"), panel.background = element\_rect(fill = "transparent") ) The resizing optionsOf course, it is possible to take control over this automatic resizing.

For this we will use a small table that will help to demo some options.

set\_flextable\_defaults(font.family = "Fira Sans Condensed")register\_gfont("Fira Sans Condensed")dummy\_ft <- data.frame(zzzz = ";)") |> flextable() |> color(color = "white", part = "all") |> mk\_par( value = as\_paragraph( as\_chunk("made", props = fp\_text\_default(font.size = 30, color = "#f2af00")), as\_chunk(" with\n", props = fp\_text\_default(color = "gray", font.size = 15)), as\_chunk("flextable", props = fp\_text\_default(color = "#c32900", font.size = 45)) ), part = "header") |> autofit() |> align(align = "center", part = "all") |> border\_outer(border = fp\_border\_default(width = 0)) HTML version of the table is the following:

dummy\_ft

| made withflextable | | --- | | |

The graphic below shows some possible resizing options.

The first table will be sized to fit the available width (fit = "width") andthe fonts will be resized to the maximum possible (scaling = "full").
The second table is displayed with no resizing (fit = "fixed").
The last table is displayed using the maximum possible size (fit = "auto").

design <- "AAA BBC"wrap\_plots( dummy\_ft |> bg(bg = "#ff006e", part = "all") |> gen\_grob(fit = "width", scaling = "full", just = "bottom"), dummy\_ft |> bg(bg = "#3a86ff", part = "all") |> gen\_grob(fit = "fixed"), dummy\_ft |> bg(bg = "#ffbe0b", part = "all") |> gen\_grob(), design = design) Content wrappingAn algorithm is used to wrap content within the available space of the cell.

library(safetyData)adsl <- adam\_adsl[, c("AGE", "SEX", "ARM")]dat <- summarizor(adsl, by = "ARM")ft <- as\_flextable(dat, spread\_first\_col = TRUE)ft |> bg(bg = "#EFEFEF", part = "all") |> plot(fit = "width", wrapping = TRUE, scaling = FALSE) It can be disabled with the wrapping = FALSE argument.

ft |> bg(bg = "#EFEFEF", part = "all") |> plot(fit = "width", wrapping = FALSE, scaling = FALSE) To leave a comment for the author, please follow the link and comment on their blog: R on ArData.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: flextable for ‘Grid graphics’

View Details

[This article was first published on business-science.io, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.What if you could 100X your coding productivity? Well you can with ChatGPT.

One of the areas I’m most excited about is speeding up the development process of R Shiny web apps. And in this tutorial I’m going to show you how I built an app in 10 minutes with ChatGPT.

Table of ContentsToday I’m going to show you how to build a shiny with ChatGPT Here’s what you’re learning today:

Tutorial Part 1: How to use ChatGPT Prompts to progressively build a shiny web app
Bonus: Steal my app library for this R-Tip

Special Announcement: ChatGPT for Data Scientists WorkshopWhat: ChatGPT for Data Scientists

When: Wednesday April 5th, 2pm EST

Price: Does Free sound good?

How To Join: Register Here.

R-Tips WeeklyThis article is part of R-Tips Weekly, a weekly video tutorial that shows you step-by-step how to do common R coding tasks. Pretty cool, right?

Here are the links to get set up.

Get the Code
YouTube Tutorial

This Tutorial is Available in VideoI have a companion video tutorial that gives you the bonus “progressively built” 4 shiny app versions in this video (plus walks you through how to use them). And, I’m finding that a lot of my students prefer the dialogue that goes along with coding. So check out this video to see me running the code in this tutorial.

Why ChatGPT is a Must (No Hype Version)Listen, chatgpt is a game changer. No question about it.

Is chatgpt perfect? No.

Am I faster when using chatgpt for “AI assisted programming”? Yes.

Will a beginner data scientist take my job? Heck no.

But you know who will take my job?Someone that’s faster at getting results than me.

I’m talking other experts that are learning how to use chatgpt to their advantage.

Are you worried now?Well let’s start calming those nerves… Good ole’ Matt Dancho is here to the rescue.

Today, I’m going to introduce you to chatgpt by making a shiny app.

Before we dive in…

Free Gift: Cheat Sheet for my Top 100 R Packages (Special Advanced Analysis Topics Included)You’re going to point chatgpt in the right direction. And this little cheat sheet gives you the specific packages that you can ask ChatGPT to use.

It complements chatgpt by giving you an edge over those that don’t have it. In fact, it’s my secret weapon…

Even I forget which R packages to use from time to time. And this cheat sheet saves me so much time. Instead of googling to filter through 20,000 R packages to find a needle in a haystack. I keep my cheat sheet handy so I know which to use and when to use them. Seriously. This cheat sheet is my bible.

Once you download it, head over to page 3 and you’ll see several R packages I use frequently just for Data Analysis.

Which is important when you want to work in these fields:

Machine Learning
Time Series
Financial Analysis
Geospatial Analysis
Text Analysis and NLP
Shiny Web App Development (see page 2)

So steal my cheat sheet. It will save you a ton of time.

Tutorial: How to Make a Shiny App in 10 Minutes with ChatGPTTime to build a shiny app with AI assistance from chatgpt. Let’s have some fun!

Step 1: Open Up Rstudio & ChatgptFirst, open up an Rstudio Session and chatgpt I have them stacked sideways here. (Don’t worry if you don’t have a shiny app yet. We’ll create it in this tutorial)

Get the code.

Step 2: Get A Project Idea and Prompt ChatGPT to Build ItMy project idea was a “monthly expense calculator”.

I figured this would be easy to build because it requires no data (csv files), and I can test out chatgpt capabilities quickly.

Here’s my first prompt:

Then head over to Rstudio and run the code. We get a very basic shiny app that calculates monthly expenses.

Get the code.

Step 2: Begin stacking chatgpt promptsThis is an iterative process where I now request additions and changes to functionality.

First, I use the key words: Update the app.

And then I request what updates I want to have the app changed.

Once the updated code is generated by chatgpt, I copy and test it out inside of Rstudio.

Get the code.

Step 3: Continue Stacking PromptsNext, I progressively request updates and changes to the app as I see things I’d like changed. Each time I run the code in Rstudio to see what impact the changes have.

First, I update the app to allow the user to add multiple months.

Next, I request the visualization to be changed to a bar plot.

Then I ask for the columns to be stacked on top of each other.

And the final shiny app is produced:

Bonus: Steal my app library for this R-TipWant all the code I just showed you? Steal my app library.

The code for the 4 apps are inside of R-Tip 060_chatgpt_shiny.

Steal my shiny apps from this tutorial.

ConclusionsYou learned how to use chatgpt to make a shiny app in 10 minutes. Great work! But, there’s a lot more to becoming a data scientist.

If you’d like to become a Business Data Scientist (and have an awesome career, improve your quality of life, enjoy your job, and all the fun that comes along), then I can help with that.

Do You Need Help Becoming A Business Data Scientist Right Now?YOU know the feeling. Being unhappy with your current job.

Promotions aren’t happening. You’re stuck. Hopeless. Confused…

And you’re praying that the next data science interview will go better than the last 12…

… But you know it won’t. Not unless you take control of your career.

The good news is…

I Can Help You Speed It Up.I’ve helped 5,897+ students learn data science for business from an elite business consultant’s perspective.

I’ve worked with Fortune 500 companies like S&P Global, Apple, MRM McCann, and more.

And I built a training program that gets my students life-changing data science careers (don’t believe me? see my testimonials here):

6-Figure Data Science Job at CVS Health ($125K)Senior VP Of Analytics At JP Morgan ($200K)50%+ Raises & Promotions ($150K)Lead Data Scientist at Northwestern Mutual ($175K)2X-ed Salary (From $60K to $120K)2 Competing ML Job Offers ($150K)Promotion to Lead Data Scientist ($175K)Data Scientist Job at Verizon ($125K+)Data Scientist Job at CitiBank ($100K + Bonus)Whenever you are ready, here’s how I can help you:Here’s the system that has gotten aspiring data scientists, career transitioners, and life long learners data science jobs and promotions…

Join My 5-Course R-Track Program
(And Become The Data Scientist You Were Meant To Be…)

P.S. – Samantha landed her NEW Data Science R Developer job at CVS Health (Fortune 500). This could be you.

To leave a comment for the author, please follow the link and comment on their blog: business-science.io.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: ChatGPT: Made this Shiny App in 10 Minutes

View Details

[This article was first published on Stories by PINstimation on Medium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The use of high-frequency data has gained widespread attention and popularity in contemporary financial research thanks to its potential to capture minute fluctuations and provide detailed insights into the behavior of financial markets. However, analyzing such data can pose significant challenges due to the high level of activity observed in current markets, which are largely dominated by high-frequency trading. Typically, high-frequency data is aggregated into discrete intraday periods or daily data using trade classification algorithms. In this article, we will explore how to classify and aggregate high-frequency data using the PINstimation package.

Trade classification algorithmsPINstimation package implements four algorithms for trade classification¹: Tick, Quote, LR, and EMO. Let’s take a closer look at each of these:

Tick: A trade is classified as a buy (sell) if the price of the trade to be classified is above (below) the closest different price of a previous trade.
Quote: Classifies a trade as a buy (sell) if the trade price of the trade to be classified is above (below) the mid-point of the bid and ask spread. Trades executed at the mid-spread are not classified.
LR²: Classifies a trade as a buy (sell) if its price is above (below) the mid-spread (quote algorithm), and uses the tick algorithm if the trade price is at the mid-spread.
EMO³: Classifies trades at the bid (ask) as sells (buys) and uses the tick algorithm to classify trades within the then prevailing bid-ask spread.

Package functionsThe package offers two functions that are specifically designed for the classification and aggregation of intraday trades:

★ classify_trades() classifies high-frequency trades using one of the aforementioned algorithms. It has the arguments:

data: a dataframe with four variables in the following order (timestamp, price, bid, ask).
algorithm: the trade classification algorithm used to determine the trade initiator. It takes one of four values: “Tick”, “Quote”, “LR”, and “EMO”. The default value is “Tick”.
timelag: the time lag in milliseconds used to calculate the lagged mid-quote for the methods “Quote”, “EMO”, and “LR”. The default value is 0.

★ aggregate_trades() aggregates high-frequency trades using one of the aforementioned algorithms, and has two additional arguments.

frequency: the frequency used to aggregate intraday data. It takes one of the values: “sec”, “min”, “hour”, “day”, “week”, “month”. The default value is “day”.
unit : an integer referring to the size of the aggregation window used to aggregate intraday data. The default value is 1. For example, when the parameter frequency is set to “min”, and the parameter unit is set to 15, then the intraday data is aggregated every 15 minutes.

Usage examplesWe use a dataset called hfdata included in the package as raw data to aggregate. It is a simulated dataset containing sample timestamp, price, volume, bid, and ask for 100.000 high frequency transactions.

To begin, we delete the variable volume and store the remaining data in the variablexdata.

library(PINstimation)xdata <- hfdataxdata$volume <- NULL * Use the EMO algorithm with a timelag of 500 milliseconds to classify
high-frequency trades in xdata.

ctrades <- classify\_trades(xdata, algorithm = "EMO", timelag = 500, verbose = T) * Display the first two rows of the dataframe ctrades:

head(ctrades, 2)## timestamp price bid ask isbuy## 38 2018-10-18 00:13:10 15.4754 15.4568 15.4754 TRUE## 49 2018-10-18 00:17:52 15.5143 15.5143 15.5236 TRUE * Use the LR algorithm with a timelagof 1 second to aggregate intraday data in xdata at a frequency of 15 minutes.

lrtrades <- aggregate\_trades(xdata, algorithm = "LR", timelag = 1000, frequency = "min", unit = 15, verbose = TRUE) * Use the Quote algorithm with a timelagof 1 second to aggregate intraday data inxdataat a daily frequency.

qtrades <- aggregate\_trades(xdata, algorithm = "Quote", timelag = 1000, frequency = "day", unit = 1, verbose = TRUE) * Display the first two rows of qtrades

head(qtrades, 2)## b s## 1 873 746## 2 823 793 * The output qtrades consists of a pair of daily sequences of buyer-initiated, and seller-initiated trades. These trading data are frequently used as input for the estimation of various research models, particularly in the field of market microstructure. One prominent model is the Probability of Informed Trading model⁴ (PIN), which typically requires quarterly datasets of daily buyer-initiated and seller-initiated trades. To estimate the PIN model, PINstimation provides the function pin_ea(), which computes the maximum-likelihood estimator of PIN using the initial parameter sets of Ersan O, Alıcı A (2016)⁵ . As such, this function can directly use the output qtrades for the estimation of the PIN model.

model <- pin\_ea(qtrades)show(model)## ----------------------------------## PIN estimation completed successfully## ----------------------------------## [...]## ========== ===========## Variables Estimates ## ========== ===========## alpha 0.739135 ## delta 0 ## mu 247.46 ## eps.b 548.72 ## eps.s 717.26 ## ---- ## Likelihood (1232.001) ## PIN 0.126237 ## ========== ===========## ## -------## Running time: 0.789 seconds * Display the optimal parameter estimates and the PIN value

model\_ea@parameters## alpha delta mu eps.b eps.s ## 0.7499975 0.1333342 1193.5179655 357.2659099 328.6291793model\_ea@pin## [1] 0.5661721 ConclusionThe PINstimation package is a highly efficient tool for classifying and aggregating high-frequency data. With just a few lines of code, it enables you to quickly classify trades using a range of algorithms and time lags, and aggregate them for virtually any desired frequency. Moreover, the classification process is fast, making it an ideal option for researchers who are working with large datasets. More information about the package and its functions can be found in the package documentation and on the dedicated website.

For more great examples of R in action, check out R-bloggers and R-users.

References

Aktas O, Kryzanowski L (2014) Trade classification accuracy for the bist. Journal of International Financial Markets, Institutions and Money 33:259–282, DOI
Lee CM, Ready MJ (1991) Inferring trade direction from intraday data. The Journal of Finance 46(2):733–746
Ellis K, Michaely R, O’Hara M (2000) The accuracy of trade classification rules: Evidence from nasdaq. The Journal of Financial and Quantitative Analysis 35(4):529, DOI
Easley, N. M. Kiefer, M. O’Hara, and J. B. Paperman. Liquidity, information, and infrequently traded stocks. The Journal of Finance, 51(4):1405, 9 1996. ISSN 00221082. DOI.
Ersan O, Alıcı A (2016) An unbiased computation methodology for estimating the probability of informed
trading (pin). Journal of International Financial Markets, Institutions and Money 43:74–94, DOI

To leave a comment for the author, please follow the link and comment on their blog: Stories by PINstimation on Medium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Trade Classification in R with PINstimation

View Details

[This article was first published on Louise E. Sinks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. This is the current version of the word game guesser. I discussed how I used this project to hone my coding skills in the companion blog post to this one.

I’m not going to walk through this in much detail, but I’m going to point out some of the major lessons I learned as I revised the code. Again, both the initial version and this version are on GitHub in all their messiness.

I learned how to put functions in a separate file and call them from my many script. This can make long code much easier to read. Here, I’ve included the helper functions in-line and commented out the source("code/helper-functions.R") in the main code. I’ve also set up switchable troubleshooting help with verbose and debug_detail parameters in my functions. Setting them to TRUE provide more info as the functions are executed.

```

|label: helper-functionsConstruct_Freq_Table <- function(word_list) {#scoring code uses the counting code from#https://www.r-bloggers.com/2018/12/rrrrs-in-r-letter-frequency-in-r-package-names/ # making the frequency table ----letters <- unlist(strsplit(word_list[,1], split = ""))char_frequencies <- as.data.frame(table(letters))#normalizedcommon <- max(char_frequencies[,2])y = (char_frequencies[,2]/common)char_frequencies$normalized <- yreturn(char_frequencies)}Scoring_Word <- function(word, freqs = char_frequencies, verbose = FALSE, debug_detail = FALSE){ letter_vec <- unlist(strsplit(word, split = "")) if (verbose == TRUE) {message("I'm in Scoring_words message and scoring: ", word)} value <- 0 for (i in 1:length(letter_vec)) { position <- letter_vec[i]== freqs$letters value[i] <- freqs$normalized[position] if (debug_detail == TRUE) { print("I am in the scoring loop calculating value: ") print(i) print(sum(value)) } if (i == length(letter_vec)) { return(total <- sum(value)) } } } Scoring_Word_Unique <- function(word, freqs = char_frequencies, verbose = FALSE, debug_detail = FALSE){ # This does only score on unique letters letter_vec <- unlist(strsplit(word, split = "")) unique_letter_vec <- unique(letter_vec) #unique_letter_vec <- letter_vec if (verbose == TRUE) {message("I'm in Scoring_words_Unique and scoring: ", word)} value <- 0 if (length(unique_letter_vec) == 0) { return(value) } else{ for (i in 1:length(unique_letter_vec)) { position <- unique_letter_vec[i] == freqs$letters value[i] <- freqs$normalized[position] if (debug_detail == TRUE) { print("I am in the unique scoring loop calculating value: ") print(i) print(sum(value)) } if (i == length(unique_letter_vec)) { return(total <- sum(value)) } } }}Removing_Letters <- function(word, chosen_word, verbose = FALSE, debug_detail = FALSE) { lvec <- gsub(paste0("[", chosen_word, "]"), "", word) return(lvec)}

`` I finally did figure out how to make variables the types I wanted. I also replaced several loops with map functions from purrr. I also made a reshaped version of my dataframe usingpivot_longer` from tidyr. Reshaping data is a really useful skill, but might be a bit confusing at first. So I certainly wanted to make sure I could do it correctly. The reshaped data is used to make a nice density plot later.

```

Loading libraries and data ----library("tidyverse")#from https://www-cs-faculty.stanford.edu/~knuth/sgb-words.txtword_list <- read.table("C:/Users/drsin/OneDrive/Documents/R Projects/Word-Games/input/sgb-words.txt") # Functions ----#source("code/helper-functions.R")# calculate letter frequencies from word listchar_frequencies <- Construct_Freq_Table(word_list)# Initialize the word_scores dataframe ----num_words <- nrow(word_list)#num_words <- 5word_scores <- data.frame(word_name = word_list[1:num_words,1], word_length = rep(0, times = num_words), word_guess1 = rep(0, times = num_words), word_guess2 = rep(0, times = num_words), word_guess3 = rep(0, times = num_words), word_guess4 = rep(0, times = num_words), score = rep(0, times = num_words), score_guess1 = rep(0, times = num_words), score_guess2 = rep(0, times = num_words), score_guess3 = rep(0, times = num_words), score_guess4 = rep(0, times = num_words) )#fill in word lengths. This is so code can be expended to longer wordsword_scores$word_length <- str_length(word_scores$word_name)# Calculates the initial scores for all words -----word_scores <- word_scores %>% mutate(score = map_dbl(word_name, Scoring_Word))word_scores <- word_scores %>% mutate(score_guess1 = map_dbl(word_name, Scoring_Word_Unique))# Finding the best first wordtop_words <- word_scores %>% arrange(desc(score_guess1))word_1 <- top_words$word_name[1]# Scoring for second guessword_scores <- word_scores %>% mutate(word_guess2 = map_chr(word_name, Removing_Letters, chosen_word = word_1))word_scores <- word_scores %>% mutate(score_guess2 = map_dbl(word_guess2, Scoring_Word_Unique))top_words <- word_scores %>% arrange(desc(score_guess2))word_2 <- top_words$word_name[1]# Scoring for third guessword_scores <- word_scores %>% mutate(word_guess3 = map_chr(word_guess2, Removing_Letters, chosen_word = word_2))word_scores <- word_scores %>% mutate(score_guess3 = map_dbl(word_guess3, Scoring_Word_Unique))top_words <- word_scores %>% arrange(desc(score_guess3))word_3 <- top_words$word_name[1]# Scoring for fourth guessword_scores <- word_scores %>% mutate(word_guess4 = map_chr(word_guess3, Removing_Letters, chosen_word = word_3))word_scores <- word_scores %>% mutate(score_guess4 = map_dbl(word_guess4, Scoring_Word_Unique))top_words <- word_scores %>% arrange(desc(score_guess4))word_4 <- top_words$word_name[1]# subsetting this dataframe and reshaping it.# This is used to make a density plot later.word_scores2 <- word_scores %>% select(word_name, score_guess1, score_guess2, score_guess3, score_guess4)word_scores_reshaped <- pivot_longer(word_scores2, cols = 2:5, names_to = "score_type", values_to = "score")

``` Nice visualizations were definitely not part of the initial code. In the next chunk, I make some prettier visualizations.

```

This is now just visualizing what we've done. ------#plotting the frequency of the letters in our word_setggplot(char_frequencies, aes(x = fct_rev(fct_reorder(letters, normalized)), y = normalized )) + geom_col() + theme_classic() + theme(legend.position = "none") + labs(title = "Frequencies of Letters", caption = "from 5 letter words") + theme(plot.title = element_text(hjust = 0.5)) + xlab("Letter") + ylab("Frequency") + scale_y_continuous(expand = c(0, 0))

```

This looks at the distribution of scores as guessing occurs. Initially, you have aword_scores_reshaped$score_type <- as.factor(word_scores_reshaped$score_type)ggplot(word_scores_reshaped, aes(score, fill = score_type)) + geom_density(alpha = 0.5) + theme_classic() + labs(title = "Evolution of Word Scores as Guessing Progresses", caption = "for 5 letter words") + xlab("Score") + ylab("Density") + labs(fill = "") + theme(legend.position = c(0.7, 0.8)) + scale_x_continuous( expand = c(0, 0)) + scale_y_continuous( expand = c(0, 0))

```

Now we are visualizing what letters are picked in each guessguess <- rep("not guessed", times = 26)char_frequencies <- cbind(char_frequencies, guess)# this is done in reverse order because some letters are guessed in more than# one word and I'd like them marked at the earliest guess.letter_vec <- unlist(strsplit(word_4, split = ""))print(letter_vec)

```

[1] "w" "h" "a" "c" "k"

for (i in 1:length(letter\_vec)) { position <- letter\_vec[i] == char\_frequencies$letters char\_frequencies$guess[position] <- "Guess 4"}letter\_vec <- unlist(strsplit(word\_3, split = ""))print(letter\_vec)

[1] "d" "u" "m" "p" "y"

for (i in 1:length(letter\_vec)) { position <- letter\_vec[i] == char\_frequencies$letters char\_frequencies$guess[position] <- "Guess 3"}letter\_vec <- unlist(strsplit(word\_2, split = ""))print(letter\_vec)

[1] "u" "n" "t" "i" "l"

for (i in 1:length(letter\_vec)) { position <- letter\_vec[i] == char\_frequencies$letters char\_frequencies$guess[position] <- "Guess 2"}letter\_vec <- unlist(strsplit(word\_1, split = ""))print(letter\_vec)

[1] "a" "r" "o" "s" "e"

for (i in 1:length(letter\_vec)) { position <- letter\_vec[i] == char\_frequencies$letters char\_frequencies$guess[position] <- "Guess 1"}ggplot(char\_frequencies, aes( x = fct\_rev(fct\_reorder(letters, normalized)), y = normalized, fill = guess)) + geom\_col() + ggtitle("When Letters are Guessed") + ylab("Normalized Counts") + xlab("Letter") + theme\_classic() + theme(legend.position = c(0.6, 0.6)) + scale\_y\_continuous(expand = c(0, 0)) So that’s the current state of this project. This was a really useful project for me and it really strengthened by R and Tidyverse skills.

CitationBibTeX citation: @online{e.sinks2023, author = {Louise E. Sinks}, title = {Self-Guided {Learning} Through a {Wordle} {Guess} {Generator:} {Part} 2}, date = {2023-04-01}, url = {https://lsinks.github.io/posts/2023-04-01-self-guided-learning-wordle-guesser-part-2}, langid = {en}} For attribution, please cite this work as:Louise E. Sinks. 2023. “Self-Guided Learning Through a WordleGuess Generator: Part 2.” April 1, 2023. https://lsinks.github.io/posts/2023-04-01-self-guided-learning-wordle-guesser-part-2. To leave a comment for the author, please follow the link and comment on their blog: Louise E. Sinks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Self-Guided Learning through a Wordle Guess Generator: Part 2

View Details

[This article was first published on R on Harshvardhan, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In this blog post, I explore who are the Twitter Blue subscribers. It is not celebrities, businesses or governments. It is our regular old Joe with fewer than a hundred followers.To leave a comment for the author, please follow the link and comment on their blog: R on Harshvardhan.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Who are Twitter Blue Users?

View Details

[This article was first published on Mirai Solutions, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.We are happy to announce our fifth series of workshops, to take place in 2023.

Between May and June, we are proposing again two learning paths: R & Shiny, with few variations compared to last year.
Like always, the workshops composing a path can be purchased singularly. Who has already a base of R or Shiny programming may like to join only the more advanced workshops, while those wishing to start from scratch and reach a more advanced level within a short time, can follow the full path investing an afternoon every week.

Our focus is on bringing our professional expertise to our workshops, sharing the best practices from our applied day-to-day work.
Last but not least, we strive to provide a hands-on experience to support the learning process, at Mirai we believe in an experiential, agile and incremental training approach (i.e. learning by doing): apply what you learned by trying a short exercise (or build a small product step by step).

 **Learning path: Introduction to R**

From the basis of the R syntax, to data manipulation, writing functions and building an R package. For solid programming foundations in R.

R basics – objects, functions and operations (recording freely available)
Data analysis with tidyverse 03/05/2023
Programming R 10/05/2023
Build an R package 17/05/2023
Becoming an R developer 24/05/2023

Compared to last year there are two main changes in the Introduction to R series:

The first R workshop (R basics) is available for free in our YouTube channel. Register on our page to get access to the videos and to receive the workshop material per email. In order to access the YouTube videos, an email with a Google account must be provided.
“Become an R developer” was addressing a vast agenda with very development important topics, for this reason we decided to provide 2 workshops to allow more time on practicing certain operations.
4) Build an R package: using appropriate tools, e.g. usethis testthat devtools and roxygen2, we will put together an R package, with tests, documentation and dependency management.
5) Becoming an R developer: you will connect your R project with Git, hosting your R package code on your GitHub repository where Continuous Integration (CI) can be implemented with GitHub Actions and collaboration facilitated with a GitFlow approach

If you have no R basis at all, you can then start for free with the basics of R.

The Shiny workshop Series is instead a replication of the past Series, with technical updates on various chapters. For example in the “Advanced Shiny Development” workshop we will introduce the shinytest2 package for the first time.

 **Learning path: Shiny**

Learn how to create a dashboard programming in R, share your data insights with an attractive UI, learn about modules, testing and first DevOps practices deploying a Shiny app.

Build your first Shiny App 31/05/2023
Make a Shiny App sparkle 07/06/2023
Advanced Shiny Development 14/06/2023
Bring a Shiny App to production 21/06/2023

Our workshops will be held online and can be booked and purchased directly from our website via PayPal (Credit Card payment or invoice is possible) as single classes or as a set.
Our attractive deals:

get a discount registering multiple people (10% from 2nd person)
get a discount registering to multiple workshops within a path (-40CHF from 2nd workshop)
register 10 days in advance to be an early bird.

After the successful test of last year, this year we are offering the recording of the workshop included in the price.

Not finding what you were looking for? Still open questions? Do get in touch! We are happy to reply to any questions you may have and we are always keen to get feedback and collect ideas for interesting new workshops.

We are looking forward to e-meeting you in our classes!

To leave a comment for the author, please follow the link and comment on their blog: Mirai Solutions.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 2023 Workshops Series Kick-off

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.This is an analysis inspired by Dan’s 2014 article and it’s meant to be a continuation of his idea with a larger sample size and a slightly different approach.

I’ve used the PRAW package in Python to scrape data from the progresspics subbredit, and I ran the data analysis in R. I was able to collect data from 77 individuals and I chose to split the entire analysis by gender – due to the differences in abdominal fat distribution between men and women.
As can be observed from Table 1 and the plot, weight loss tends to be much higher for the males than for the females – both in absolute units and percentage-wise.

I chose to run four different regression models, for each gender. While Dan’s article only considered weight change, I also included the weight percentage change, BMI change, and BMI percentage change.

```

Malelm(weightDiff ~ waistDiff, data, subset = gender == "Male")lm(weightDiff_perc ~ waistDiff, data, subset = gender == "Male")lm(bmiDiff ~ waistDiff, data, subset = gender == "Male")lm(bmiDiff_perc ~ waistDiff, data, subset = gender == "Male")# Femalelm(weightDiff ~ waistDiff, data, subset = gender == "Female")lm(weightDiff_perc ~ waistDiff, data, subset = gender == "Female")lm(bmiDiff ~ waistDiff, data, subset = gender == "Female")lm(bmiDiff_perc ~ waistDiff, data, subset = gender == "Female")

```

Based on the results from Table 2, we could conclude the following:
Males1. On average, you need to lose 8.6 lbs (3.9 kg) to lose 1 inch off the waist. 2. On average, you need to lose 1.4% of your weight to lose 1 inch off the waist. 3. On average, you need to lose 1.1 BMI points to lose 1 inch off the waist. 4. On average, you need to reduce your BMI by 1.4% to lose 1 inch off the waist.

Females1. On average, you need to lose 4.8 lbs (2.1 kg) to lose 1 inch off the waist. 2. On average, you need to lose 2.3% of your weight to lose 1 inch off the waist. 3. On average, you need to lose 0.82 BMI points to lose 1 inch off the waist. 4. On average, you need to reduce your BMI by 2.4% to lose 1 inch off the waist.

Of course, the sample size is relatively small (especially for women) and the starting weights vary widely, so take these results with a grain of salt.
If you want to delve deeper, you can find the data and the code I used here.

How much weight do you need to lose to drop an inch off the waist? was first posted on March 31, 2023 at 8:00 am.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How much weight do you need to lose to drop an inch off the waist?

View Details

[This article was first published on R programming – Oscar Baruffa, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.2023-03-31

Many thanks to Sergey Bolshakov and Mokandil for some of this update’s submissions!

I want to again give a special thanks to Niels Ohlsen for helping me vet books and adding them to the collection. Niels is the co-organiser of the Dataviz meetup in Bremen, Germany. If you’re in the area, why not look them up on LinkedIn and Meetup?

I’m applying for a grant to upgrade the Big Book of R. Have a look at the details and if you like, take two minutes to submit your statement of support!

Introduction to urban accessibilityby Rafael H. M. Pereira, Daniel Herszenhut

The aim of this book is to equip its readers with the fundamental concepts, the data analysis skills and the processing tools needed to perform urban accessibility analyses and transportation projects impact assessments. The book was written with the problems faced by public managers, policy makers, students and researchers working on urban and transportation planning in mind. Hence, the book is essentially practical. All the material in the book is presented with reproducible examples using open data sets and the R programming language.

https://www.bigbookofr.com/geospatial.html#introduction-to-urban-accessibility

Introduction to Data Scienceby Hansjörg Neth

This book provides a gentle introduction to data science for students of any discipline with little or no background in data analysis or computer programming. Based on notions of representation and modeling, we examine some key data types and data structures, and then learn to clean, transform, summarize and visualize data to communicate our results.

https://www.bigbookofr.com/data-science.html#introduction-to-data-science-1

R for Data Analyticsby Abhay Singh

This is compilation of notes for R for Data Analytics. These notes are used as learning material in R for Research, R for Financial Analytics and R for Data Analytics workshops.

https://www.bigbookofr.com/statistics.html#r-for-data-analytics

A Little Book of R for Bioinformatics 2.0by Avril Coghlan, Nathan L. Brouwer

This book is based on the original A Little Book of R for Bioinformatics by Dr. Avril Coghlan (Hereafter “ALBRB 1.0”). Dr. Coghlan’s book was one of the first and most thorough introductions to using R for bioinformatics and computational biology.

https://www.bigbookofr.com/life-sciences.html#a-little-book-of-r-for-bioinformatics-2.0

R for data analysisby Trevor French

The content will start at the very beginning by showing you how to set up your R environment and the basics of programming in R. By the end of the book, you will be able to perform intermediate analytics techniques such as linear regresion and automatic report generation.

https://www.bigbookofr.com/data-science.html#r-for-data-analysis

A Crash Course in Geographic Information Systems (GIS) using Rby Michael Branion-Calles

Introduction into concepts for GIS and spatial data in R. Later chapters are not finished.

https://www.bigbookofr.com/geospatial.html#a-crash-course-in-geographic-information-systems-gis-using-r

Keep up to date with new data posts and Big Book of R updates by signing up to my newsletter. Subscribers get a free copy of Project Management Fundamentals for Data Analysts worth $12.

Once you’ve subscribed, you’ll get a follow up email with a link to your free copy.

The post 6 new books added to Big Book of R appeared first on Oscar Baruffa.

To leave a comment for the author, please follow the link and comment on their blog: R programming – Oscar Baruffa.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 6 new books added to Big Book of R

View Details

[This article was first published on R – Jacob Smith Economics, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.In a previous blog post I discussed how we can compute rank dependent utility in R. Though this function was quite exciting for myself as computing rank dependent utilty by hand is quite labor intensive, I saw the function as limiting in the sense that I was suppressing all the information generated in the process of computing rank dependent utility. This inspired me to modify the original RDU_Compute() to RDU_Data(). Lets see some of the features and output of this newer function.

Getting more with RDU_Data()The function RDU_Data() is built as follows:

RDU\_Data<-function(outcomes,p\_vec,pw){ if(length(outcomes)==length(p\_vec)){ #Step 0 organize outcomes and probabilities into a single vector df<-data.frame(outcomes,p\_vec) #Step 1 Organize Probabilities by outcome. df1<-df[order(-outcomes),] zerorow<-c(0,0) df2<-rbind(zerorow,df1) #Step 2 Define vector of ranks rank<-cumsum(df2[,2]) df3<-data.frame(df2,rank) #Step 3 compute pweights pw\_vec<-pw(df3$rank) df4<-data.frame(df3,pw\_vec) #Step 4 Take difference between pw\_vec to compute decision weights d\_weights<-diff(df4$pw\_vec) #Check if weights sum to 1 sum(d\_weights) #Add to dataframe d\_weights1<-c(0,d\_weights) df5<-data.frame(df4,d\_weights1) #Step 5 drop the first "helper" row df6<-df5[-1,] colnames(df6)<-c("Outcomes","Probabilities","Rank", "Weighted Ranks", "Decision Weights") return(df6)} else{ print("Outcomes and vector of probabilities must be the same length") }} After generating this function we can apply it to valuing a set of uncertain outcomes defined by a vector of outcomes and a corresponding vector of probabilities.

```

Define outcomes, probabilities and probability weighting function.payoffs<-c(2,1,-1)probs<-c(1/3,1/2,1/6)p_0.61<-function(p){p^0.61/((p^0.61+(1-p)^0.61)^(1/0.61))}#Compute resultsRDU_Data(payoffs,probs,p_0.61)#Output: Outcomes Probabilities Rank Weighted Ranks Decision Weights2 2 0.3333333 0.3333333 0.3359522 0.335952163 1 0.1666667 0.5000000 0.4206394 0.084687194 -1 0.5000000 1.0000000 1.0000000 0.57936065

``` This framework is desirable when you want to get information on your decision weights and more information on how its computed. You can easily compute the rank dependent utility from this valuation by running the following code:

```

Computing Rank Dependent Utilitypayoffs%*%RDU_Data(payoffs,probs,p_0.61)$`Decision Weights`#Output [,1][1,] 0.1772309

``` ConclusionThe goal of this R function was to make the computing and valuing of uncertain outcomes under rank dependent utility theory accessible and easy to implement, while giving all the “behind the scenes” information on what goes into the valuation. Perhaps I’ll think about writing an R package if I keep working on these types of models. We’ll see.

If you found this blog interesting, let me know your thoughts in the comments below!

To leave a comment for the author, please follow the link and comment on their blog: R – Jacob Smith Economics.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Computing Rank Dependent Utility in R: Getting More out with RDU_Data()

View Details

[This article was first published on R on kieranhealy.org, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.While we’re redoing some classics, here is the time series of assault deaths in the United States and eighteen other OECD countries from 1960 to 2020.

Assault deaths in the OECD, 1960-2020.

Code and data are available on GitHub.

To leave a comment for the author, please follow the link and comment on their blog: R on kieranhealy.org.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Assault Deaths in the OECD 1960-2020

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to use deep learning in R Join our workshop on Introduction to Deep Learning with R which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: Introduction to Deep Learning with R

Date: Thursday, May 4th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Eran Raviv is an expert researcher at APG Asset Management, working for the Digitalization & Innovation department. His academic papers are published in top-tier journals. In his present role, Dr. Raviv helps the organization develop its Data Science capabilities and he is engaged in both strategic planning and leading bottom-up initiatives.

Description: The purpose of this workshop is to offer an introductory understanding of deep learning, regardless of your prior experience. It is important to note that this workshop is tailored to those who are absolute beginners in the field. We therefore begin with few necessary fundamental concepts, after which we cover the basics of deep learning, including topics such as what is actually being learned in deep learning, what makes it “deep,” and why it is such a popular field. We will also cover how you can estimate deep learning models in R using the neuralnet package. You should attend this workshop if you heard about deep learning and would like to know more about it.

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Introduction to Deep Learning with R workshop was first posted on March 30, 2023 at 11:34 am.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Introduction to Deep Learning with R workshop

View Details

[This article was first published on rOpenSci - open tools for open science, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.A new post of our interview series “Meeting the stars of the R-universe”. We aim to introduce the teams and people behind the development of software and packages many of us use and which are available through the R-Universe. We want to highlight and explore different teams and projects around the world, the work they do, their processes and users. Our third stop is Norway to talk with Athanasia Monika Mowinckel. Be sure to watch the video at the end with excerpts from the interview.

Working with the human brain requires specific software and complex developments. However R appears to be the natural way to collect and analyze the huge amount of data needed, and the R-Universe the most friendly place to share and invite collaborations.

Athanasia Monika Mowinckel
Researcher and RSEAthanasia Monika Mowinckel introduces herself simply as Mo, which is her nickname. But her CV is much more extensive and includes, among many other activities, her current work at the Center for Lifespan Changes in Brain and Cognition at the University of Oslo in Norway. She works there as a scientist and tells us that her work “involves research software engineering or research data management” and because of that “R is the default statistical program we use for our daily work at the center, and we generate results in the form of R packages.”

Research at the center involves specialized software because of the information they need to process, such as brain images. Athanasia explains that at the end of the day all their statistics are in R, and because of that, “having our R packages in our tool belt when we are working is super important for our staff.”

Athanasia is working on a project called Life Brain, funded by the European Union for the Human Life project and involving 13 European centers. Mo explains that they have developed a lot of research, written publications, and many results have been implemented in R packages. Asked about the project, with passion and great enthusiasm, Mo says:

We are trying to investigate how people are aging, what their brain aging is like, what their cognitive aging is like.

R-Universe

The R universe actually came at an extremely timely point for us,

Athanasia explains that this platform offered them a simple way to show all the software they were working on. In this research development, there are different types and sources of funding. Even though they have a collective organization in GitHub, they needed a clear way to show all the work that is distributed in different repositories in the personal accounts of the researchers or the accounts of the research centers. That is why Mo assures that in R-Universe “we can create a place for all our universes and thus show the packages.”

In addition, Athanasia explains that an important part of using the R-Universe is that it is very clear who the developers or maintainers are, which means that everyone keeps their corresponding credits and contributions.

Another interesting use of the R-Universe for sharing work that Mo showed us, is the possibility of accessing the data as well as some extra functions used in an analysis. For example, in the Lifebrain Global Brain Health Survey they obtained more than 27 thousand responses, produced articles and papers, and provide all the data and code in the gbhs package (you can see more information about this work in the video below).

Subcortical map plot with ggsegAs we explore the packages in the Lifebrain R-Universe, Athanasia shows us her greatest pride: ggseg, software to create images of the brain that can be analyzed with different functions. It was precisely with this work that Mo found in R-Universe the solution to a series of problems with the size of the data and the installation of the tools needed to work:

When the data packages are too big and cannot be reduced, it is not possible to publish them in CRAN (…) and failures appear when users want to install them in their development version that took us a long time to solve and with non-optimal results.

In this context is when R-Universe presents her with a definitive solution:

This is the real solution, to be able to have everything we work on the project with a working installation and not waste any more time troubleshooting our users.

ggseg in action: plotting brain atlasesIn Athanasia’s experience, R-Universe allowed them to publish all the packages in one place, with easy accessibility and the possibility to receive contributions, maintenance and no installation problems for the users. In her own words:

R-Universe is magic for me.

Remember that the R-Universe generates binaries for different operating systems for all packages that belong to the universe, with no limit on the package size. One of the platform’s objectives is to make it easier to publish and install R packages.

In the following video, Athanasia explains in more detail the development of their Life Brain project, how they use R to organize and work with information, and how the R-Universe allowed them to organize and share all their packages.

Video of the interviewTo leave a comment for the author, please follow the link and comment on their blog: rOpenSci - open tools for open science.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Meeting the Stars of the R-Universe: Researching Our Brain with the Magic of the R-Universe

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.I won’t wax long and poetic here since I’ve already posted the experiment that has all the details.

TL;DR: there are still only ~90-ish in the WebR WASM “CRAN”, but more are absolutely on the way, including the capability to build your own CRAN and dev packages via Docker and host your own WebR WASM pkg repo.

@timelyportfolio created an experimental method to install built base R packages from R Universe, and I enhanced that method in another, recent experiment, but that’s a bit wonky, and you have to do some leg work to figure out if a package can be installed and then do a bunch of manual work (though that Observable notebook will save you time).

The aforelinked new experiment shows how to use Pyodide side-by-side with WebR. While this one doesn’t have them sharing data or emscripten filesystems yet, we’ll get there! This puts SCADS of Python packages at your fingertips to fill in the gap while we wait for more R to arrive.

Code is up on GitHub but hit the experiment first to see what’s going on.

A small taste of the experiment.

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Using WebR + Pyodide To Fill In The (Temporary) Package Gaps

View Details

[This article was first published on R on kieranhealy.org, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The visualization exercise of the day for class is a re-creation of a figure I first saw Lane Kenworthy make. It’s a connected scatterplot of total health spending in real terms and life expectancy of the population as a whole. The fact that real spending and expectancy tend to steadily increase for most countries in most years makes the year-to-year connections work even though they’re not labeled as such.

A connected scatterplot of life expectancy against total health spending in the US and eighteen other OECD countries.

Data and code are on GitHub.

To leave a comment for the author, please follow the link and comment on their blog: R on kieranhealy.org.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Life Expectancy and Health Spending in the OECD

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionTime series analysis is a powerful tool for understanding and predicting patterns in data that vary over time. In this tutorial, we will use the AirPassengers dataset to create a bootstrapped timeseries model in R.

The AirPassengers dataset The AirPassengers dataset contains data on the number of passengers traveling on international flights per month from 1949 to 1960. To begin, we will load the dataset into R and plot it to get an idea of the data’s structure and any underlying patterns.

library(forecast)data(AirPassengers)plot(AirPassengers, main = "International Airline Passengers 1949-1960") From the plot, we can see that there is a clear upward trend in the data, as well as some seasonality.

Creating a bootstrapped timeseries modelNow that we have an idea of the structure of the data, we can create a bootstrapped timeseries model using the auto.arima() function from the {forecast} package. The auto.arima() function uses an automated algorithm to determine the best model for a given timeseries.

set.seed(123)n <- length(AirPassengers)n\_boot <- 1000# create bootstrap sample indicesboot\_indices <- replicate(n\_boot, sample(1:n, replace = TRUE))# create list to store modelsmodels <- list()# create bootstrapped modelsfor(i in 1:n\_boot) { boot\_data <- AirPassengers[boot\_indices[, i]] models[[i]] <- auto.arima(boot\_data)}models[[1]]

Series: boot\_data ARIMA(0,0,0) with non-zero mean Coefficients: mean 275.5347s.e. 9.5443sigma^2 = 13209: log likelihood = -887.01AIC=1778.02 AICc=1778.1 BIC=1783.96 In the code above, we first set a seed to ensure reproducibility of our results. We then specify the length of the timeseries and the number of bootstrap iterations we want to run. We create a list to store the models and a set of bootstrap sample indices.

We then loop through each bootstrap iteration, creating a new dataset from the original timeseries by sampling with replacement using the boot_indices. We use the auto.arima() function to create a timeseries model for each bootstrap sample and store it in our models list.

Summarizing and plotting residualsNow that we have created our bootstrapped timeseries models, we can summarize and plot the residuals of each model to get an idea of how well our models fit the data.

```

create list to store residualsresiduals <- list()# create residuals for each modelfor(i in 1:n_boot) { boot_data <- AirPassengers[boot_indices[, i]] residuals[[i]] <- residuals(models[[i]])}# summarize residualsresidual_means <- sapply(residuals, mean)residual_sd <- sapply(residuals, sd)# plot residualspar(mfrow = c(2, 1))hist( residual_means, main = "Bootstrapped Model Residuals", xlab = "Mean Residuals" )hist( residual_sd, col = "red", main = "", xlab = "SD Residuals" )

```

par(mfrow = c(1,1)) In the code above, we create a list to store the residuals for each model, loop through each model to create residuals using the residuals() function, and summarize the residuals by taking the mean and standard deviation of each set of residuals.

We then plot the mean residuals and standard deviations for each model using the plot() function and add a legend to indicate the meaning of the two lines.

Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A Bootstrapped Time Series Model with auto.arima() from {forecast}

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

SatRdays London 2023 is fast approaching!

Don’t miss out! Ticket sales close at midnight on 8th April!On 22nd April 2023 we will be hosting SatRdays London, an inclusive, low cost event, which gives R users an opportunity to network and learn from other experts across sectors. In a recent blog post, we introduced all of the speakers for the event! This week, it’s the sponsors turn.

Data comes in all shapes and sizes. It can often be difficult to know where to start. Whatever your problem, Jumping Rivers can help.

CUSP LondonThe Centre for Urban Science and Progress (CUSP) are based in London, UK. Their mission is to support interdisciplinary research and innovation using Data Science in and for London, bringing together multi-disciplinary teams of academics, a group of associates from external partners and students working with CUSP.

CUSP London, hosted in the King’s Department of Informatics, is part of an international network of multidisciplinary institutes led by CUSP New York at the NYU Tandon School of Engineering. They welcome new academic collaborators and external partners into the CUSP family locally and internationally.

CUSP are generously providing the venue for SatRdays London 2023.

Jumping RiversJumping Rivers is an analytics company based in the North East who specialise in creating bespoke solutions for modern business problems. Their team is made up of experts in data science and data engineering from many different backgrounds, and their wealth of knowledge and experience allows them to think outside the box and solve problems in new and innovative ways.

PositPosit (formerly RStudio), are a US based company who develop R and Python-based tools to help you produce higher quality analysis faster. As well as creating open-source tools for all to use and develop, Posit are very active in the data science community, hosting their own annual conference, as well as supporting conferences around the world, including SatRdays London!

R ConsortiumThe central mission of the R Consortium is to work with and provide support to the R Foundation and to the key organisations developing, maintaining, distributing and using R software through the identification, development and implementation of infrastructure projects. Its members include leading institutions and companies dedicated to the use, development and growth of R.

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: SatRdays London 2023: Sponsors

View Details

[This article was first published on Blog: John C. Nash, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.John C. Nash, Retired Professor, University of Ottawa

While I have had little to do with Apple since the early 1980s when they abandoned several products in which a colleague and I had invested time, effort and money, I recently had to take note of the introduction of the M1 chip in some of their products. This was not a choice. I had made some updates to my R package nlsr, then did the appropriate tests and submitted it to CRAN (the Consolidated R Archive Network). The package very smoothly passed the checks and was upgraded on the repository servers as fast as I can remember.

A few days later, however, I received a message containing the text

Please see the problems shown on<https://cran.r-project.org/web/checks/check\_results\_nlsr.html>.Please correct before 2023-02-23 to safely retain your package on CRAN.There is a check service for M1mac issues: seehttps://www.stats.ox.ac.uk/pub/bdr/M1mac/README.txt . It turned out that the issue was not in my code, but in an example given in a vignette, a form of package documentation useful to present a longer exposition than typical man pages. Moreover, the example did not use any of my own code, but referred to another package.

It was easy to resolve the immediate issue with my package by wrapping the offending example lines with the R try() function to catch the “errors”. However, my curiosity was raised.

Here is what I believe was going on. The particular “error” was that a nonlinear least squares code that called Fortran routines returned a so-called “singular gradient” (actually Jacobian) and failed. On non-M1 platforms, this does not happen, but truthfully, the example problem does present near-singular Jacobians. Because no other platform raised the issue, the example was not protected with try().

IEEE ArithmeticPrior to the mid 1980s, computers used a frightening variety of different ways of performing arithmetic. Besides the choice of radix – 2 (binary), 10 (decimal), 8 (octal) or 16 (hexadecimal) – we had the possibility of no radix (“decimal”) point for integers, which could be signed or unsigned, we could use fixed point arithmetic, rather like the old mechanical calculators, or we could have floating-point, which is what scientific calculations really need so that different scales can be accommodated.

However, we also had various, sometimes crazy, extensions to offer guard digits, “fuzzy” comparison and a whole lot of other obstacles to the straightforward comparison of results output by different systems.

The Intel 8087 and IEEE 754 standard were parallel developments. I was a rather minor member of the 1985 standardization committee, though an ardent supporter. This introduced what we now more or less expect as the arithmetic of modern computing. Likely for historical reasons, most modern calculations take what we used to call “double precision” as the common form of real numbers.

Summarizing severely, this form, now called binary64 has a mantissa of 53 binary digits, 11 exponent bits and a sign. But that is 65 bits! Yes, because in scientific notation, there is always a non-zero first digit. In binary, this is a 1, so we can assume it is there and get a slightly extended precision “for free”. We make the electronic engineers work hard to deal with an assumed binary digit in their chips.

80 bit extended precisionThe Intel 8087 development was influenced by a number of numerical analysts, notably W. Kahan. During the 60s and 70s, we had to deal with a lot of machines that had very limited arithmetic. I did a lot of the development of the codes in my book Compact numerical methods for computers (1979) on a Data General Nova with a 24 bit mantissa and no guard digit. One recognized improvement was the use of a longer precision for an accumulator register for summation. If you have a situation where you need to add up a lot of very small numbers, say a thousand of around 0.001 and two of 10 thousand, but your calculator carries only 6 digits, your answer will be 20000, not 20001. The reason is that 10000.001 cannot be represented. However, if you have an 8-digit accumulator, you get the right answer.

That’s the motivation for extended precision registers. Of course, handling extended accumulators requires extra electrical engineering and extra software to provide access. Two notes:

It appears that in the M1 chip, there are no extended registers
A lot of software doesn’t bother to provide access anyway.

A little example in RConsider the following little snippet of R code that adds numbers in a vector one at a time.

loopsum <- function(vec){n <- length(vec)vsum<-0.0for (i in 1:n) { vsum <- vsum + vec[i]}vsum} This forces vector summation to be one element at a time, and the intermediate quantity vsum will be forced into the binary64 representation, so no extended bits. However, internal functions in R can use such extended arithmetic, and the underlying C code seems to use it. Let us try the following example:

small<-.Machine$double.eps/4 # 1/4 of the machine precisionvsmall <- rep(small, 1e4) # a long vector of small numbersvv1 <- c(1.0, vsmall) # 1 at the front of this vectorvv2 <- c(vsmall, 1.0) # 1 at the end(sum(vv1) – 1.0e0)(sum(vv2) - 1.0e0)(loopsum(vv1) – 1.0e0) # should be zero(loopsum(vv2) – 1.0e0) # should be greater than zero Though I don’t have an M1 Macintosh, some very community-minded R workers have provided a test environment at https://mac.r-project.org/macbuilder/submit.html . This is set up to run the R standard ”R CMD check” on a tarball package that is uploaded to the site via the web-page linked. Creating a small package with the example code in a file of type .Rd in the man directory of the package allows testing in this environment. Results are returned as a compressed archive from which the results of examples can be obtained. The results are shown below side by side for comparison. Note particularly the first summation, where the M1 results suggest no extended precision.

Intel I5 results R macOS builder results> (sum(vv1) - 1.0e0) > (sum(vv1) - 1.0e0)[1] 5.551115e-13 <-----> [1] 0> (sum(vv2) - 1.0e0) > (sum(vv2) - 1.0e0)[1] 5.551115e-13 [1] 5.551115e-13> (loopsum(vv1) - 1.0e0) > (loopsum(vv1) - 1.0e0)[1] 0 [1] 0> (loopsum(vv2) - 1.0e0) > (loopsum(vv2) - 1.0e0)[1] 5.551115e-13 [1] 5.551115e-13 CONCLUSIONShould the above be a worry that we may be getting poor results? Probably not. However, it would be good to have better documentation than I could find easily on the M1 arithmetic. It also makes sense to keep in mind that the M1 may be blindingly fast, but that it will get slightly weaker results on some vector, matrix and signal processing operations than chips that do have extended precision for accumulation.

To leave a comment for the author, please follow the link and comment on their blog: Blog: John C. Nash.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Floating-point summation and the M1 processor

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The hottest topic in AI has sent shockwaves through mainstream media and multiple industries. In 2023, ChatGPT signals a massive breakthrough in accessible AI, where any schmuck can open it up and start interacting with a large language model instantly.

But, are you maximizing your time on the model?

Probably not. However, you can easily deepen your understanding with a bit of practice—especially through DataCamp’s Introduction to ChatGPT course where you can easily supercharge your AI abilities in just two hours with no prior experience required.

First, let’s take a closer look at Open AI’s large language model.

What is ChatGPT?ChatGPT is a large language model developed by OpenAI.

It has been trained on a large collection of written language and can generate an almost endless amount of human-like responses to various inputs (i.e. questions or commands), making it a powerful tool for natural language processing tasks.

ChatGPT has the potential to be used in a wide range of applications, from chatbots and customer service to content creation, language translation, and beyond.

Will it affect my job and can it help me?The first part—if it hasn’t already, it will soon. Second part—absolutely.

ChatGPT can be useful in a variety of industries such as customer service, healthcare, education, e-commerce, and content creation.

For instance, ChatGPT can help customer service representatives handle a large volume of queries, provide immediate assistance, and improve customer satisfaction. In healthcare, ChatGPT can assist doctors in diagnosing and treating patients by analyzing medical records and recommending appropriate treatments.

In education, ChatGPT can act as a personal tutor, answer questions, and provide explanations. Specifically in content creation, ChatGPT can help generate original content, summarize articles, and simplify complex concepts for readers. And, the list goes on.

It’s groundbreaking in every sense of the world. And, we’re just scratching the surface.

How can I learn it?If you want to stay relevant, elevate your skillset, and increase the quality of your work—you should learn how to proficiently use ChatGPT ASAP. Whilst most people can log on and start using it instantly, many uninitiated users do not maximize their time on the platform.

Fortunately, DataCamp—a world-leading ed-tech platform specializing in data science and AI—has just released a brand new course: Introduction to ChatGPT.

With no experience required, anyone can gain the fundamental skills to instantly improve their ChatGPT skills and start applying their skills in the real world.

From text summarization, explaining complex concepts, drafting engaging marketing content, and generating and explaining code, you’ll learn about the most common and useful applications of ChatGPT.

Plus, you’ll be equipped with a framework to evaluate new use cases and determine if ChatGPT is the right solution for your needs. Finally, you’ll explore the legal and ethical considerations that come with implementing ChatGPT in various situations.

Stay ahead of the AI curve. Start now for free.

What is ChatGPT? Will it affect my job? Can it help me? How can I learn it? (sign up for a free course to find out) was first posted on March 28, 2023 at 6:30 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: What is ChatGPT? Will it affect my job? Can it help me? How can I learn it? (sign up for a free course to find out)

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionYesterday I posted on performing a benchmark on reading in a compressed .csv.gz file of a 2,000 by 2,000 data.frame. It was brought to my attention by someone on Mastadon (@mariviere@fediscience.org – https://fediscience.org/@mariviere) that I should also use {duckdb} and {arrow} so I will perform the same analysis as yesterday but I will also add in the two aforementioned packages.

FunctionThe functions that we will be using this time around are as follows:

fread() – data.table
duckdb_read_csv() – duckdb
read_csv() – readr
read.csv() – base
read.table() – base
vroom() with altrep = FALSE – vroom
vroom() with altrep = TRUE – vroom
oprn_csv_dataset() – arrow

ExampleMake the DataLet’s make that dataset again:

library(R.utils)# create a 1000 x 1000 matrix of random numbersdf <- matrix(rnorm(2000000), nrow = 2000, ncol = 2000) |> as.data.frame()# Make and save gzipped filewrite.csv(df, "df.csv")gzip( filename = "df.csv", destname = "df.csv.gz", overwrite = FALSE, remove = TRUE) BenchmarkingTime to benchmark

library(rbenchmark)library(data.table)library(readr)library(duckdb)library(arrow)library(vroom)library(dplyr)library(DBI)n <- 30benchmark( # Base R "read.table" = { a <- read.table( "df.csv.gz", sep = ",", colClasses = list(numeric = 1:2000) ) }, "read.csv" = { b <- read.csv( "df.csv.gz", sep = ",", colClasses = list(numeric = 1:2000) ) }, # data.table "fread" = { c <- fread( "df.csv.gz", sep = ",", colClasses = list(numeric = 1:2000) ) }, # vroom "vroom alltrep false" = { d <- vroom("df.csv.gz", delim = ",", col\_types = "d") }, "vroom alltrep true" = { e <- vroom("df.csv.gz", delim = ",", altrep = TRUE, col\_types = "d") }, # readr "readr" = { f <- read\_csv("df.csv.gz", col\_types = "d") }, # Arrow "arrow" = { g <- open\_csv\_dataset("df.csv.gz") }, # DuckDB "duckdb" = { con <- dbConnect(duckdb()) h <- duckdb\_read\_csv( conn = con, name = "df", files = "C:\\Users\\ssanders\\Documents\\GitHub\\steveondata\\posts\\rtip-2023-03-28\\df.csv.gz" ) dbDisconnect(con) }, # Replications replications = n, # Columns columns = c( "test","replications","elapsed","relative","user.self","sys.self")) |> arrange(relative)

test replications elapsed relative user.self sys.self1 arrow 30 3.01 1.000 5.04 0.252 fread 30 28.28 9.395 19.56 4.303 vroom alltrep false 30 31.89 10.595 26.25 10.754 vroom alltrep true 30 33.72 11.203 25.75 10.675 duckdb 30 94.09 31.259 90.70 2.776 readr 30 98.28 32.651 113.05 45.127 read.table 30 109.97 36.535 107.78 1.248 read.csv 30 153.79 51.093 152.44 0.56 Important note is the session info on the pc I am using to write this:

sessionInfo()

R version 4.2.3 (2023-03-15 ucrt)Platform: x86\_64-w64-mingw32/x64 (64-bit)Running under: Windows 10 x64 (build 19045)Matrix products: defaultlocale:[1] LC\_COLLATE=English\_United States.utf8 [2] LC\_CTYPE=English\_United States.utf8 [3] LC\_MONETARY=English\_United States.utf8[4] LC\_NUMERIC=C [5] LC\_TIME=English\_United States.utf8 attached base packages:[1] stats graphics grDevices utils datasets methods base other attached packages: [1] dplyr\_1.1.1 vroom\_1.6.1 arrow\_11.0.0.3 duckdb\_0.7.1-1 [5] DBI\_1.1.3 readr\_2.1.4 data.table\_1.14.8 rbenchmark\_1.0.0 [9] R.utils\_2.12.2 R.oo\_1.25.0 R.methodsS3\_1.8.2loaded via a namespace (and not attached): [1] pillar\_1.9.0 compiler\_4.2.3 tools\_4.2.3 digest\_0.6.31 [5] bit\_4.0.5 jsonlite\_1.8.4 evaluate\_0.20 lifecycle\_1.0.3 [9] tibble\_3.2.1 pkgconfig\_2.0.3 rlang\_1.1.0 cli\_3.6.1 [13] rstudioapi\_0.14 parallel\_4.2.3 yaml\_2.3.7 xfun\_0.38 [17] fastmap\_1.1.1 knitr\_1.42 generics\_0.1.3 vctrs\_0.6.1 [21] htmlwidgets\_1.6.2 hms\_1.1.3 bit64\_4.0.5 tidyselect\_1.2.0 [25] glue\_1.6.2 R6\_2.5.1 fansi\_1.0.4 rmarkdown\_2.21 [29] tzdb\_0.3.0 purrr\_1.0.1 magrittr\_2.0.3 htmltools\_0.5.5 [33] assertthat\_0.2.1 utf8\_1.2.3 crayon\_1.5.2

Sys.info() |> as.data.frame() |> tibble::rownames\_to\_column() |> as\_tibble() |> slice(1,2,3,5)

```

A tibble: 4 × 2 rowname `Sys.info()` 1 sysname Windows 2 release 10 x64 3 version build 19045 4 machine x86-64

```

memory.profile() |> as.data.frame()

memory.profile()NULL 1symbol 24303pairlist 642504closure 11189environment 4009promise 22963language 189766special 47builtin 701char 2039073logical 18866integer 108132double 20060complex 5character 160381... 21any 0list 58500expression 5bytecode 41555externalptr 12382weakref 13860raw 10113S4 1362

gc()

used (Mb) gc trigger (Mb) max used (Mb)Ncells 3363479 179.7 5830931 311.5 5830931 311.5Vcells 32950395 251.4 81254422 620.0 81254324 620.0 To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How fast does a compressed file in Part 2

View Details

[This article was first published on R Consortium, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Woo June Jung, Founder of the R Korea Group (also on Facebook) recently talked to the R Consortium to discuss his efforts to promote the use of R in Korea. He stressed the importance of communities and also shared the group’s experience of hosting an annual R User Conference in Korea for six consecutive years. They R Korea Group stopped hosting the conference during the pandemic but are hopeful to start again this year. He also discussed his work on two accounting projects developed in R.

Woo June has played a vital role in building the R community in Korea and now hopes to start a R-Ladies chapter in Korea.

Please share about your background and your involvement in the R Community. What is your level of experience with the R language?

I don’t remember exactly, but I think it was around 2005 when I first encountered R, through my econometrics course in graduate school. With some experience in languages like C or Basic, I was fascinated by the amazing capabilities of R and began using it for research from that time onwards. Of course, I knew how to use SAS and SPSS, but I thought R was much better in many aspects.

Around 2010, the big data boom in Korea began and R started receiving attention as well. I think this was around the time when I started using R for analytical work. At first, I analyzed survey data, and later worked on web log analysis, text mining, sentiment analysis, anomaly detection, recommendation, etc., in various fields such as finance, commerce, communication, and manufacturing.

In addition to my two roles, as an analyst and researcher using R, I also founded and manage the Korean R user community (R Korea). The reason for this is that I hoped other people would learn about this amazing programming language. I had been using a Facebook group for building the community, but I am currently working on moving it to a web-based platform. In the early days of the community, I held free and open R seminars every other week for 2-3 years to expand our user base. With these seminars, the community grew rapidly and now has over 10,000 users. From 2014 to 2019, R Korea also held an annual conference called R User Conference in Korea (RUCK), which attracted at least 400 attendees each year, with some years exceeding 700 attendees. In 2020, we invited Hadley Wickham, but due to the COVID-19 situation, the conference was canceled, and we have not been able to hold it again since.

R User Conference in Korea (RUCK) 2015Unfortunately, during this period, people’s interest in R declined as artificial intelligence rapidly rose in popularity. However, some people who had positive memories of the RUCK conference contacted me and expressed their desire to keep the tradition going (special thanks to Kim Jin-seop and Na Young-jun). I hope we can hold the RUCK again for 2023. Personally, I have made good friends through community activities. One of my closest friends is Jeon Hee-won, who developed the Korean morphological analysis package KoNLP. (Currently, the KoNLP package is not supported on CRAN, and I would like him to maintain the package again if his circumstances allow.)

The community provides a communication channel with people in related fields. Community participants can solve their own problems within the community and also help others. Sometimes, they can relieve daily fatigue with witty jokes. These activities can help develop capabilities as analysts. I think these are the reasons why people seek out communities.

I consistently receive a lot of help on the new trends in R through R-bloggers, and I am very interested in the activities of R-Ladies. I would like to launch R-Ladies Korea through a newly opened community website.

What industry are you currently in? How do you use R in your work?

Currently, I have two roles. One is a DX researcher in the finance industry, and the other is a researcher in the accounting field. In my job, I mainly deal with the valuation modeling of unlisted companies and startups, which fortunately is also one of the research topics in accounting. As an accounting researcher, I am interested in the DX field of accounting such as structured accounting information like XBRL and digital reporting. Since accounting information is provided to the market mostly in the mixed form of data types such as numbers, text, etc., text mining can also be used in accounting research. I use R for both work and research.

What trends do you currently see in R language and your industry? Any trends you see developing in the near future?

Currently, I am working or researching in various fields, but artificial intelligence is the trend. For this reason, the demand for R in Korea has significantly decreased, and it is in a dangerous state. This phenomenon is thought to be due to Korea’s sensitivity to trends and small market size.

The scope of (statistical) data analysis is quite broad. However, data analysis is often simply classified into data analysis (descriptive statistics from a statistical perspective) and machine learning (there is also a machine learning field in statistics). Special thanks to the authors who wrote ISLR. Many people claim that using traditional but most used models such as regression is outdated, and machine learning is a new approach. This is wrong obviously.

Nevertheless, as the IT industry grows and the significant achievements of artificial intelligence and machine learning have become a major trend, interest in statistics significantly decreases. People seem to be more interested in creating IT services or products and believe that using artificial intelligence can create better services. In this environment, Python, a general-purpose language, has become the dominant language. This trend is likely to continue for some time to come. R, which is strong in (commonly called) statistics, is likely to face more difficult times in Korea. However, to confirm the analysis results at the same level as those output by R, Python requires much more code to be written. R just lacks deep learning packages.

Please share about a project you are currently working on or have worked on in the past using the R language.

Currently, I am not working on any projects like developing packages. Instead, I am focusing on research papers related to accounting. Previously, I worked on projects called WARD (Wrangling Accounting-Related Data) and AIA (Accounting Information Analysis) related to accounting, but they are currently private.

WARD is a project that structures and digitizes accounting-related information. Accounting information has diverse ranges and types. Accounting information does not simply mean financial statements, and it has been said that all publicly available data can be a subject of accounting research such as privacy, ESG, etc.

For example, one of the important research topics in accounting is firm value relevance, which requires stock price data. If it is an evaluation of the value of an unlisted company, it may be necessary to crawl funding news. Forensic techniques are also used to investigate accounting fraud, and recently, AI is also used for continuous auditing. To conduct accounting research for these purposes, various types of data such as tabular data, text data, and numerical and string data stored in databases or web must be handled, and various analysis techniques such as visualization, analysis using numerical values, and text mining can be applied.

WARD aimed to develop a data package for accounting that provides these various types of data. However, as the data size grew larger, it became difficult to handle accounting on GitHub and it continues to be used for only private purposes. AIA provided functions to analyze the data provided by the accounting package, but it is also being operated privately.

What resources or techniques did you use?

I am performing all the work for my research papers in the Posit (RStudio) environment. Recently, I have been conducting research on information security, internal controls, ESG, and the valuation of unlisted companies. Of course, tidyverse is always one of the first packages I load, and I even use Python in the Posit environment. I use several other packages as well, but there are too many to list in detail.

How do I Join?

Apply HereThe post R in Finance and Accounting Sector in Korea appeared first on R Consortium.

To leave a comment for the author, please follow the link and comment on their blog: R Consortium.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R in Finance and Accounting Sector in Korea

View Details

[This article was first published on Online College Math Teacher, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The government doesn’t want you to know about this, but I have discovered it and I will share this with a few close friends: There is a black hole in the number line, and it’s at number 4. Every word in the English language will eventually fall into it and can’t get out of it. As an example, take the word mathematical. Mathematical has twelve letters.
Twelve has six letters.
Six has three letters.
Three has five letters.
Five has four letters.
Four has four letters.
Four has four letters, and now we entered this black hole at 4, and we can’t get out of it!

Try a few more words. Try words as long as you like. Try morphophonemically, which has 18 letters. I have done exhaustive research on this with R, and you will find every English word eventually falls into the black hole at 4 and can’t get out. At this rate, there will be no words left! Of course, this is an April Fool’s Day prank.

Did you figure it out?

Here is some R code to test the word mathematical.

# Try word mathematical
library(broman)
x <- "mathematical"
y <- -99 # Initialize y
while(y != “four”){
y <- nchar(x)
y <- broman::spell_out(y, max_value = 20) # Spell out an integer as a word
print(c(x,y))
x <- y
}

Here is some R code to test ten random words.

# Try ten random words
set.seed(123)
words <- read.table("https://raw.githubusercontent.com/dwyl/english-words/master/words_alpha.txt")
original <- sample(words$V1, 10, replace = FALSE)
x <- original
rm(words) # free up memory
for (i in 1:10){
y <- vector()
y[1] <- "dummy"
for (j in 1:100){
c <- nchar(x[i])
y[j] <- spell_out(c, max_value = 20)
x[i] <- y[j]
if (y[j] == “four”) {
break
}
}
cat(c(original[i], “\t”, y), “\n”)
}

Here is the code for the number line:

library(ggplot2)
df <- data.frame(x = c(1,2,3,4,5,6,7,8,9,10),
y = rep(0,10),
group = c(“A”,”A”,”A”,”B”,”A”,”A”,”A”,”A”,”A”,”A”))
ggplot(df, aes(x = x, y = y)) +
geom_point(aes(color = group, size = ifelse(x == 4, 15, 15))) +
geom_hline(yintercept = 0, linetype = 1, color = “lightblue”, size = 1) +
scale_x_continuous(limits = c(0, 11), expand = c(0, 0),
breaks = NULL, minor_breaks = NULL) +
scale_y_continuous(limits = c(-0.2, 0.2), expand = c(0, 0),
breaks = NULL, minor_breaks = NULL) +
scale_color_manual(values = c(“red”, “black”)) +
ggtitle(“THE BLACK HOLE AT NUMBER 4”) +
theme_void() +
theme(legend.position = “none”,
axis.line = element_blank(),
axis.text = element_blank(),
plot.title = element_text(color=”black”, size=14, face=”bold”)) +
geom_text(aes(x = x, y = -0.1, label = x), size = 5)

To leave a comment for the author, please follow the link and comment on their blog: Online College Math Teacher.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: There’s a black hole in the number line

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionI received an email over the weekend in regards to my last post not containing the reading in of gz compressed file(s) for the benchmarking. While this was not an over site per-se it was a good reminder that people would probably be interested in seeing this as well.

Benchmarking is the process of measuring and comparing the performance of different programs, tools, or configurations in order to identify which one is the most efficient for a specific task. It is a critical step in software development that can help developers identify performance bottlenecks and improve the overall performance of their applications.

In this post I create a square matrix and then convert it to a data.frame (2,000 rows by 2,000 columns) and then saved it as a gz compressed csv file. The benchmark compares different R packages and functions, including base R, data.table, vroom, and readr, and measures their relative speeds based on the time it takes to read in the .csv.gz file.

Here are some pro’s of trying things different ways and properly benchmarking them:

Identify the most efficient solution: Benchmarking can help you identify the most efficient solution for a specific task. By measuring the relative speeds of different programs or tools, you can determine which one is the fastest and use it to improve the performance of your application.
Optimize resource utilization: Benchmarking can help you optimize resource utilization by identifying programs or tools that consume more resources than others. By choosing the most resource-efficient solution, you can reduce the cost of running your application and improve its scalability.
Avoid premature optimization: Benchmarking can help you avoid premature optimization by measuring the performance of different programs or tools before you start optimizing them. By identifying the slowest parts of your application, you can focus your optimization efforts on the most critical areas and avoid wasting time optimizing code that doesn’t need it.
Keep up with technology: Benchmarking can help you keep up with technology by comparing the performance of different tools and libraries. By staying up to date with the latest technologies, you can improve the performance of your application and stay ahead of your competitors.
Improve code quality: Benchmarking can help you improve the quality of your code by identifying performance bottlenecks and areas for optimization. By optimizing your code, you can improve its maintainability, reliability, and readability.

In conclusion, benchmarking is an essential tool for software developers that can help them identify the most efficient solutions for their applications. By measuring the relative speeds of different programs or tools, developers can optimize resource utilization, avoid premature optimization, keep up with technology, and improve the quality of their code.

FunctionThe different functions I use in the benchmarking are as follows:

Base R* read.csv() * read.table()

data.table* fread

vroom* vroom() with altrep = FALSE * vroom() with altrep = TRUE

readr* read_csv()

ExampleLet’s make a 2,000 by 2,000 matrix, covert to a data.frame and then save it out as a .csv file and then convert to a .gz file.

library(R.utils)# create a 1000 x 1000 matrix of random numbersmy\_matrix <- matrix(rnorm(2000000), nrow = 2000, ncol = 2000) |> as.data.frame()# Make and save gzipped filewrite.csv(my\_matrix, "my\_matrix.csv")gzip(filename = "my\_matrix.csv", destname = "matrix.csv.gz", overwrite = FALSE, remove = TRUE) Ok now that the data is written we can benchmark the read in times from various packages.

Benchmarking library(rbenchmark)library(data.table)library(readr)library(vroom)library(dplyr)n <- 30benchmark( # Base R "read.table" = { a <- read.table("matrix.csv.gz", sep = ",") }, "read.csv" = { b <- read.csv("matrix.csv.gz", sep = ",") }, # data.table "fread" = { c <- fread("matrix.csv.gz", sep = ",") }, # vroom "vroom alltrep false" = { d <- vroom("matrix.csv.gz", delim = ",") }, "vroom alltrep true" = { e <- vroom("matrix.csv.gz", delim = ",", altrep = TRUE) }, # readr "readr" = { f <- read\_csv("matrix.csv.gz") }, # Replications replications = n, # Columns columns = c( "test","replications","elapsed","relative","user.self","sys.self")) |> arrange(relative)

test replications elapsed relative user.self sys.self1 fread 30 19.44 1.000 13.56 1.592 vroom alltrep true 30 22.06 1.135 10.54 2.633 vroom alltrep false 30 24.75 1.273 10.22 2.844 read.table 30 94.34 4.853 79.02 0.645 read.csv 30 143.28 7.370 115.64 0.746 readr 30 177.61 9.136 50.37 10.05 Voila!

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How fast does a compressed file in?

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

See it live before reading!

The previous post brought lit-webr, to introduce Lit and basic reactivity.

Today, is more of the same, but we bring the OG Shiny demo plot into the modern age by using Observbable Plot to make the charts.

We’re still pulling data from R, but we’re letting Plot do all the heavy lifting.

Here’s what’s changed…

First, main.js no longer has an {svglite} dependency. This means slightly faster load times, and less code. After ensuring we have datasets available, this is remainder of what happens (please see the larger example for more extended “what’s goin’ on?” comments):

// WE WILL TALK ABOUT THIS BELOWimport { webRDataFrameToJS } from './utils.js'const regions = document.getElementById("regionsInput")const plotOutput = document.getElementById("regionsOutput")regions.options = await (await R.webR.evalR(`colnames(WorldPhones)`)).toArray()// WE WILL TALK ABOUT THIS BELOWplotOutput.worldPhones = webRDataFrameToJS( await (await webR.evalR( `as.data.frame.table(WorldPhones, stringsAsFactors=FALSE) |> setNames(c("year", "region", "phones"))` )).toJs())plotOutput.region = regions.options[ 0 ] The webRDataFrameToJS() function in utils.js was mentioned in a previous experiment. Its sole mission in life is to turn the highly structured object that is the result of calling WebR’s toJs() function on an R data.frame. Most JS data things like the structure webRDataFrameToJS() puts things into, and Observable Plot is a cool JS data thing.

The ugly await… await… sequence is to get the data from R to give to webRDataFrameToJS(). We got lucky thins time since as.data.frame.table does a niiice job taking the WorldPhones rownamed matrix and pivoting it longer.

We store the output of that into the region-plot component. I could/should have made it a private property, but no harm, no foul in this setting.

Lastly, in region-plot.js, our component is reduced to two properties: one to store the region name and one for the data you saw, above. We still use events to trigger updates between the popup and the plotter, and said plotter is doing this in render():

render() {return html`<div><slot></slot>${ Plot.plot({ style: { background: "#001e38", color: "#c6cdd7", padding: "30px", fontSize: "10pt", fontFamily: '-apple-system, BlinkMacSystemFont, …' }, inset: 10, marginLeft: 60, caption: "Data from AT&T (1961) The World's Telephones", x: { label: null, type: "band" }, y: { label: "Number of ☎ (K)", grid: true }, marks: [ Plot.barY( this.worldPhones.filter((d) => d.region === this.region), { x: "year", y: "phones", fill: "#4a6d88" } ), Plot.ruleY([0]) ] })}</div>`;} When the region changes, it triggers a reactive update. When the refresh happens, this snippet:

js this.worldPhones.filter((d) => d.region === this.region)

does the hard work of filtering out all but the region we selected from the tiny, in-memory phones “database”.

Plot may not be {ggplot2}, but it cleans up well, and we’ve even had it match the style we used in the previous experiment.

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 🧪 Lit + WebR + Observable Plot: Linking Lit’s Lightweight Web Components And WebR For Vanilla JS Reactivity & JS DataVis

View Details

[This article was first published on r.iresmi.net, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Today I saw this :

So, Portugal has only one terrestrial neighbour. Opencage lists the other countries with only one neighbour.

Can we get this by ourselves ?

library(tidyverse)library(rnaturalearth)library(spdep)# get data from Natural Earthcountries <- ne\_countries(scale = 10, returnclass = "sf") |> st\_make\_valid() |> group\_by(sov\_a3, sovereignt) |> summarise(.groups = "drop")# for joining codes and names laterlookup <- countries |> st\_drop\_geometry()# get neighbours with {spdep}neighbours <- countries |> poly2nb()# we only keep countries with 1 neighbour# and add namesneighbours |> set\_names(countries$sov\_a3) |> keep(\(x) length(x) == 1) |> enframe("sov\_a3", "nb") |> unnest(nb) |> filter(nb != 0) |> mutate(nb = countries$sov\_a3[nb]) |> left\_join(lookup, by = "sov\_a3") |> left\_join(lookup, by = c("nb" = "sov\_a3"), suffix = c("", "\_nb")) |> relocate(sovereignt, .after = 1)# A tibble: 16 × 4 sov\_a3 sovereignt nb sovereignt\_nb <chr> <chr> <chr> <chr> 1 BRN Brunei MYS Malaysia 2 CAN Canada US1 United States of America 3 DN1 Denmark DEU Germany 4 DOM Dominican Republic HTI Haiti 5 GMB Gambia SEN Senegal 6 HTI Haiti DOM Dominican Republic 7 IRL Ireland GB1 United Kingdom 8 KOR South Korea PRK North Korea 9 LSO Lesotho ZAF South Africa 10 MCO Monaco FR1 France 11 PNG Papua New Guinea IDN Indonesia 12 PRT Portugal ESP Spain 13 QAT Qatar SAU Saudi Arabia 14 SMR San Marino ITA Italy 15 TLS East Timor IDN Indonesia 16 VAT Vatican ITA Italy I miss Bahrain / Saudi Arabia because the Natural Earth dataset is not detailed enough…

Border on a small islandAnd the « new » Canada / Denmark border is also not taken into account…

While we are at it, which country has more neighbors ?

```

A tibble: 209 × 3 sov_a3 n sovereignt 1 CH1 15 China 2 RUS 14 Russia 3 BRA 11 Brazil 4 FR1 11 France 5 COD 9 Democratic Republic of the Congo 6 DEU 9 Germany 7 AUT 8 Austria 8 SDN 8 Sudan 9 SRB 8 Republic of Serbia 10 TUR 8 Turkey # … with 199 more rows

``` To leave a comment for the author, please follow the link and comment on their blog: r.iresmi.net.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: One neighbour

View Details

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Writing markdown documents outside RStudio (using the usual set of packages) has benefits and struggles. Huge struggle is transforming dataframe results into markdown table, using hypens and pipes. Ugghhh…

This useless functions takes R dataframe as input and prints out dataframe wrapped in markdown table.

iris[1:3,1:5]# Result:#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species#> 1 5.1 3.5 1.4 0.2 setosa#> 2 4.9 3.0 1.4 0.2 setosa#> 3 4.7 3.2 1.3 0.2 setosa And if we want this result in Markdown, it should look like:

|Sepal.Length|Sepal.Width|Petal.Length|Petal.Width|Species| |---|---|---|---|---| |5.1|3.5|1.4|0.2|setosa| |4.9|3|1.4|0.2|setosa| |4.7|3.2|1.3|0.2|setosa| This can be directly used in any editor. (Sidenote: if you decide to use Latex, same function can be created, just different ASCII chars needs to be used – ampersend and backslash).

The super lazy function

df\_2\_MD <- function(your\_df){ cn <- as.character(names(your\_df)) headr <- paste0(c("", cn), sep = "|", collapse='') sepr <- paste0(c('|', rep(paste0(c(rep('-',3), "|"), collapse=''),length(cn))), collapse ='') st <- "|" for (i in 1:nrow(your\_df)){ for(j in 1:ncol(your\_df)){ if (j%%ncol(your\_df) == 0) { st <- paste0(st, as.character(your\_df[i,j]), "|", "\n", "" , "|", collapse = '') } else { st <- paste0(st, as.character(your\_df[i,j]), "|", collapse = '') } } } fin <- paste0(c(headr, sepr, substr(st,1,nchar(st)-1)), collapse="\n") cat(fin)} # run functionshort\_iris <- iris[1:3,1:5]df\_2\_MD(short\_iris) As always, code is available on Github in Useless_R_function repository. The sample file in this repository is here (filename: Dataframe_to_markdown.R) Check the repository for future updates.

Happy R-coding and stay healthy!

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Little useless-useful R functions – Transforming dataframe to markdown table

View Details

[This article was first published on R on kieranhealy.org, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Sometimes data arrives as a series of individual files each of which is organized in the same way—which is to say, each of which has the same variables, features, or columns. Imagine a series of tables reporting mandated information about every school in the state, or a hundred spreadsheets each with information about a different country, or thirty seven CSVs each with the same columns of information about representatives in each U.S. Congressional Session since 1945. Seeing as these data were not provided to us as a single big table we generally want to make it into one. If the files are CSVs on our local computer, R has some nice functions that allow us to iterate over a vector of filenames and produce a tidy table of data.

Case 1: Lots of local CSV filesFor example, our Congressional project might have a data folder with a subfolder called congress. We can get a listing of the CSV files inside it like this:

| 1 2 3 4 5 6 7 8 910111213141516 | filenames <- dir(path = here("files", "data", "congress"), pattern = "*.csv", full.names = TRUE)filenames[1:15] # Just displaying the first 15## [1] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/01\_79\_congress.csv"## [2] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/02\_80\_congress.csv"## [3] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/03\_81\_congress.csv"## [4] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/04\_82\_congress.csv"## [5] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/05\_83\_congress.csv"## [6] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/06\_84\_congress.csv"## [7] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/07\_85\_congress.csv"## [8] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/08\_86\_congress.csv"## [9] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/09\_87\_congress.csv"## [10] "/Users/kjhealy/Documents/courses/vsd/files/data/congress/10\_88\_congress.csv" |

We can feed that vector to read_csv() and it will be quite happy. It reads each file and binds them all together by row, in a big stack. (Again, remember we know ex ante that the files all have the same column structure. We’ll get errors or warnings if this isn’t true.)

| 1 2 3 4 5 6 7 8 91011121314151617181920212223242526 | df <- read\_csv(filenames, id = "path", name\_repair = janitor::make\_clean\_names)df |> mutate(congress = str\_extract(path, "\_\\d{2,3}\_congress"), congress = str\_extract(congress, "\\d{2,3}")) |> relocate(congress)## # A tibble: 20,580 × 27## congress path last first middle suffix nickname born death sex position## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> ## 1 79 /User… Aber… Thom… Gerst… <NA> <NA> 05/1… 01/2… M U.S. Re…## 2 79 /User… Adams Sher… <NA> <NA> <NA> 01/0… 10/2… M U.S. Re…## 3 79 /User… Aiken Geor… David <NA> <NA> 08/2… 11/1… M U.S. Se…## 4 79 /User… Allen Asa Leona… <NA> <NA> 01/0… 01/0… M U.S. Re…## 5 79 /User… Allen Leo Elwood <NA> <NA> 10/0… 01/1… M U.S. Re…## 6 79 /User… Almo… J. Linds… Jr. <NA> 06/1… 04/1… M U.S. Re…## 7 79 /User… Ande… Herm… Carl <NA> <NA> 01/2… 07/2… M U.S. Re…## 8 79 /User… Ande… Clin… Presba <NA> <NA> 10/2… 11/1… M U.S. Re…## 9 79 /User… Ande… John Zuing… <NA> <NA> 03/2… 02/0… M U.S. Re…## 10 79 /User… Andr… Augu… Herman <NA> <NA> 10/1… 01/1… M U.S. Re…## # ℹ 20,570 more rows## # ℹ 16 more variables: party <chr>, state <chr>, district <chr>, start <chr>,## # end <chr>, religion <chr>, race <chr>, educational\_attainment <chr>,## # job\_type1 <chr>, job\_type2 <chr>, job\_type3 <chr>, job\_type4 <chr>,## # job\_type5 <chr>, mil1 <chr>, mil2 <chr>, mil3 <chr> |

You can see how path is created as an id column, to help us keep track of which file each row of data came from. We create the congress column after we read in the data by extracting the congressional session from the filename with a regular expression. But you can see how nice it is to have this facility to read data in like this.

Case 2: Lots of local Excel filesWhat if the data is in a file format whose read-in function doesn’t know this trick about accepting a vector of file paths? In that case we can do what read_csv() is doing behind the scenes and map the vector of file names to the read-in function, and explicitly bind the results together. The default map() function binds by making a list of whatever you did. But there are type-specific mappers, too. We are dealing with data frames—with regular tables of data—so we have a function, map_dfr(), that takes input and binds its output by row into a data frame (or dies trying). So for example, if our Congressional data were all Excel files in .xlsx format, and they all had the same structure with a header row of column names, we could write this:

| 123 | df <- filenames |> map\_dfr(~ readxl::read\_xlsx(.x), .id = "path") |

And we would get (almost) the same result. Here we start with the vector of filenames and pass it down a pipe to map_dfr(), which goes ahead and maps, or applies, the read_xlsx() function to each element of the filenames vector—i.e. to each file. The .x there is a placeholder or pronoun that means ‘whatever one we’re working on right now’. Mapping functions like this is just a kind of iteration where you don’t have to explicitly write a loop. This makes it easier to cleanly compose sequences or chains of functions without having to explicitly create a bunch of placeholder objects or declare counters and so on.

Case 3: Lots of remote CSV or Excel files in a bare directoryNow, what if the files we want are stored remotely on a server? These days there’s often an API for such things. But quite often, even now, you may find yourself dealing (as I did yesterday) with a bare directory of files that looks like this:

A listing of state-level life-tables from the National Center for Health Statistics

This is a directory of state-by-state life tables associated with a CDC report. Again, we want them (or, as we’ll see, some of them) as a single table.

Now, if these were provided as CSVs our task would be a little easier because in addition to being able to deal with a vector of filenames at once, read_csv(), and indeed all the read-in functions in readr in general, will happily read URLs as well as local file paths. However, the read_xlsx() function in the readxl package can’t do this yet. It only wants file paths. A second issue is that the Excel files themselves are not entirely tidy. At the top they look like this:

The top of a state-level life table Excel file.

Those first two rows are a mild violation of one of the rules of thumb for entering data in spreadsheets, helpfully outlined by Karl Broman and Kara Woo. The first row is metadata; the second is a more verbose description of the standard lifetable headers in the third row. Except for Age, which is not labeled in the third row. That big box labeled “Age (years)” is actually an super-sized second row. That means the first element of row three, our actual column headers, is blank! This is annoying. Fortunately these are easily dealt with, as we can just tell our read function to skip those two lines. There’s also a Source row at the bottom (not shown here) that we’ll have to strip out.

But the first order of business is getting a vector of the actual file URLs to download. You could just copy and paste the listing, like an animal, but we are not going to do that. Instead, we’ll take advantage of the old-school empty-directory listing to get the file names. We’ll do this using R’s implementation of curl.

| 1 2 3 4 5 6 7 8 910111213141516171819 | ```

The directory locationurl_loc <- "ftp.cdc.gov/pub/Health_Statistics/NCHS/Publications/NVSR/71-02/"## Prepare the connectionlist_files <- curl::new_handle()curl::handle_setopt(list_files, ftp_use_epsv = TRUE, dirlistonly = TRUE)## Open the connection as an FTP listingcon <- curl::curl(url = paste0("ftp://", url_loc), open = "r", handle = list_files)## Get the file namesfiles <- readLines(con)## Close the connection close(con)files[1:10] # Just show the first ten## [1] "AK1.xlsx" "AK2.xlsx" "AK3.xlsx" "AK4.xlsx" "AL1.xlsx" "AL2.xlsx" "AL3.xlsx" "AL4.xlsx" "AR1.xlsx" "AR2.xlsx"

``` |

We open a connection to the remote folder and use ftp_use_epsv and dirlistonly flags to restrict what we get back. Then we read the lines recieved from the FTP server into the files object. This gives us the bare file names of everything in this remote folder.

The spreadsheets are named according to a scheme with a two-letter state abbreviation followed by a number. The number signifies the kind of life-table it is. The files ending in 1 have the life-table for the population as a whole, which is what we are interested in.

Now that we have a vector of the file names (but just the file names at this point) we can do a bit of prep:

| 1 2 3 4 5 6 7 8 910111213141516171819202122 | ```

Just take files containing a `1``fname_stubs <- files[str_detect(files, "1")]## State abbreviationsfname_labs <- substr(fname_stubs, start = 1, stop = 2)## Construct the filenames and give them a name attribute of the 2-letter state## abbreviation, so they are trackable in the data frame we're about to make.fnames <- paste0("https://", url_loc, fname_stubs)fnames <- set_names(fnames, fname_labs)fnames[1:4]## AK ## "https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Publications/NVSR/71-02/AK1.xlsx" ## AL ## "https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Publications/NVSR/71-02/AL1.xlsx" ## AR ## "https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Publications/NVSR/71-02/AR1.xlsx" ## AZ ## "https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Publications/NVSR/71-02/AZ1.xlsx"

``` |

You can see that the vector elements (the actual URLs) also have a name or label (the state abbreviation). This will make it easier to create a state id column, because map_dfr() will use the label as its index counter.

The last step is to get read_xlsx() to get all the remote files, which it does not have the capacity to do directly. It won’t even accept a single URL, it only wants file paths. So we will have to write a one-off function that gets the file and puts it in a temporary location that read_xlsx() can see.

| 1234567 | ```

Feed this an http URLget_lifetable <- function(x) { httr::GET(x, httr::write_disk(tf <- tempfile(fileext = ".xlsx"))) readxl::read_xlsx(tf, skip = 2, .name_repair = "unique_quiet") |> rename("age" = `...1`) |> filter(!str_detect(age, "SOURCE")) # remove trailing source line}

``` |

The first line inside the function uses httr to GET the file, and immediately save it locally using write_disk(), taking care to specify that the temporary file we save should have an .xlsx extension. (Otherwise read_xlsx() will complain.) The second line actually reads in the file that’s been downloaded. We take the opportunity to suppress chatter about the name repair that has to happen on that blank first column header in the third row, rename that location age, and strip the trailing line about the source of the data that I mentioned above.

This function reads one given URL. Now we just need to map a vector of URLs to it and bind the results by row:

| 123 | life\_tabs <- fnames |> map\_dfr(~ get\_lifetable(.x), .id = "state") |

As I said, a nice thing is that map_dfr() will use the name attribute of fnames to create its id column, which we can therefore name state, because the name of each URL element is the abbrevation for the state it is providing data about.

And we’re done:

| 1 2 3 4 5 6 7 8 91011121314151617 | life\_tabs## # A tibble: 5,151 × 8## state age qx lx dx Lx Tx ex## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 AK 0–1 0.00513 100000 513. 99623. 7661992 76.6## 2 AK 1–2 0.000308 99487. 30.6 99472. 7562368. 76.0## 3 AK 2–3 0.0000992 99457. 9.87 99452. 7462896. 75.0## 4 AK 3–4 0.000194 99447. 19.3 99437. 7363444. 74.0## 5 AK 4–5 0.000192 99428. 19.1 99418. 7264007 73.1## 6 AK 5–6 0.000288 99408. 28.6 99394. 7164589 72.1## 7 AK 6–7 0.000328 99380. 32.6 99364. 7065195 71.1## 8 AK 7–8 0.000348 99347. 34.6 99330. 6965832. 70.1## 9 AK 8–9 0.000349 99313. 34.6 99295. 6866502. 69.1## 10 AK 9–10 0.000333 99278. 33.1 99261. 6767206 68.2## # ℹ 5,141 more rows## # ℹ Use `print(n = ...)` to see more rows |

A few thousand rows of data programmatically extracted from fifty spreadsheets, now ready for a bit more cleaning and any amount of confusion about what it is that life-tables actually show.

To leave a comment for the author, please follow the link and comment on their blog: R on kieranhealy.org.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Reading Remote Data Files

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

See it live before reading!

This is a Lit + WebR reproduction of the OG Shiny Demo App

Lit is a javascript library that makes it a bit easier to work with Web Components, and is especially well-suited in reactive environments.

My recent hack-y WebR experiments have been using Reef which is an even ligher-weight javascript web components-esque library, and it’s a bit more (initially) accessible than Lit. Lit’s focus on “Web Components-first” means that you are kind of forced into a structure, which is good, since reactive things can explode if not managed well.

I also think this might Shiny folks feel a bit more at home.

This is the structure of our Lit + WebR example (I keep rejiggering this layout, which likely frustrates alot of folks )_

lit-webr├── css│ └── style.css # you know what this is├── favicon.ico # when developing locally I want my icon├── index.html # you know what this is├── main.js # the core experiment runner├── md│ └── main.md # the core experiment markdown file├── r-code│ └── region-plot.R # we keep longer bits of R code in source files here├── r.js # place for WebR work├── renderers.js # these experiment templates always use markdown├── themes│ └── ayu-dark.json # my fav shiki theme├── utils.js # handy utilities (still pretty bare)├── wc│ ├── region-plot.js # 👉🏼 WEB COMPONENT for the plot│ ├── select-list.js # 👉🏼 for the regions popup menu│ └── status-message.js # 👉🏼 for the status message├── webr-serviceworker.js.map # not rly necessary; just for clean DevTools console└── webr-worker.js.map # ☝🏽 A great deal has changed (due to using Lit) since the last time you saw one of these experiments. You should scan through the source before continuing.

The core changes to index.html are just us registering our web components:

```

``` We could have rolled them up into one JS file and minified them, but we’re keeping things simple for these experiments.

Web Components (“components” from now on) become an “equal citizen” in terms of HTMLElements, and they’re registered right in the DOM.

The next big change is in this file (the rendered main.md), where we use these new components instead of our <div>s. The whittled down version of it is essentially:

<status-message id="status"></status-message><region-plot id="regionsOutput" svgId="lit-regions"> <select-list label="Select a region:" id="regionsInput"></select-list></region-plot> The intent of those elements is pretty clear (much clearer than the <div> versions), which is one aspect of components I like quite a bit.

You’ll also notice components are - (dash) crazy. That’s part of the Web Components spec and is mandatory.

We’re using pretty focused components. What I mean by that is that they’re not very reusable across other projects without copy/paste. Part of that is on me since I don’t do web stuff for a living. Part of it was also to make it easier to show how to use them with WebR.

With more modular code, plus separating out giant chunks of R source means that we can actually put the entirety of main.js right here (I’ve removed all the annotations; please look at main.js to see them; we will be explaining one thing in depth here, vs there, tho.):

import { renderMarkdownInBody } from "./renderers.js";import * as d3 from "https://cdn.jsdelivr.net/npm/d3@7/+esm";await renderMarkdownInBody( `main`, "ayu-dark", [ 'javascript', 'r', 'json', 'md', 'xml', 'console' ], false)let message = document.getElementById("status");message.text = "WebR Loading…"import * as R from "./r.js";message.text = "Web R Initialized!"await R.webR.installPackages([ "svglite" ])await R.library(`svglite`)await R.library(`datasets`)const regionRender = await globalThis.webR.evalR(await d3.text("r-code/region-plot.R"))message.text = "{svglite} installed"const regions = document.getElementById("regionsInput")const plotOutput = document.getElementById("regionsOutput")regions.options = await (await R.webR.evalR(`colnames(WorldPhones)`)).toArray()plotOutput.region = regions.options[ 0 ]plotOutput.renderFunction = regionRenderplotOutput.render()message.text = "Ready" I want to talk a bit about this line from main.js:

const regionRender = await globalThis.webR.evalR( await d3.text("r-code/region-plot.R")) That fetches the source of the single R file we have in this app, evaluates it, and returns the evaluated value (which is an R function object) to javascript. This is the script:

renderRegions <- function(region, id = "region-plot") { # our base plot theme list( panel.fill = "#001e38", bar.fill = "#4a6d88", axis.color = "#c6cdd7", label.color = "#c6cdd7", subtitle.color = "#c6cdd7", title.color = "#c6cdd7", ticks.color = "#c6cdd7", axis.color = "#c6cdd7" ) -> theme # get our svg graphics device amp'd s <- svgstring(width = 8, height = 4, pointsize = 8, id = id, standalone = FALSE) # setup theme stuff we can't do in barplot() par( bg = theme$panel.fill, fg = theme$label.color ) # um, it's a barplot barplot( WorldPhones[, region], main = region, col = theme$bar.fill, sub = "Data from AT&T (1961) The World's Telephones", ylab = "Number of Telephones (K)", xlab = "Year", border = NA, col.axis = theme$axis.color, col.lab = theme$label.color, col.sub = theme$subtitle.color, col.main = theme$title.color ) dev.off() # get the stringified SVG plot\_svg <- s() # make it responsive plot\_svg <- sub("width='\\d+(\\.\\d+)?pt'", "width='100%'", plot\_svg) plot\_svg <- sub("height='\\d+(\\.\\d+)?pt'", "", plot\_svg) # return it plot\_svg} That R function is callable right from javascript. Creating that ability was super brilliant of George (the Godfather of WebR). We actually end up giving it to the component that plots the barplot (see region-plot.js) right here:

plotOutput.renderFunction = regionRender We’re getting a bit ahead of ourselves, since we haven’t talked about the components yet. We’ll do so, starting with the easiest one to grok, which is in status-message.js and is represented by the <status-message></status-message> tag.

These custom Lit components get everything HTMLElement has, plus whatever else you provide. I’m not going to show the entire source for status-message.js here as it is (lightly) annotated. We’ll just cover the fundamentals, as Lit components also have alot going on and we’re just using a fraction of what they can do. Here’s the outline of what’s in our status-message:

export class StatusMessage extends LitElement { static properties = { /* things you can assign to and read from */ } static styles = [ /* component-scoped CSS */ ] constructor() { /* initialization bits */ render() { /* what gets called when things change */ }}// register itcustomElements.define('status-message', StatusMessage); Our status-message properties just has one property:

static properties = { text: {type: String}, // TypeScript annotations are requried by Lit}; This means when we do:

let message = document.getElementById("status");message.text = "WebR Loading…" we are finding our component in the DOM, then updating the property we defined. That will trigger render() each time, and use any component-restricted CSS we’ve setup.

Things get a tad more complicated in select-list.js. We’ll just cover the highlights, starting with the properties:

static properties = { id: { type: String }, // gives us easy access to the id we set label: { type: String }, // lets us define the label up front options: { type: Array } // where the options for the popup will go}; If you recall, this is how we used them in the source:

<region-plot id="regionsOutput" svgId="lit-regions"> <select-list label="Select a region:" id="regionsInput"></select-list></region-plot> The id and label properties will be available right away after the custom element creation.

We start option with an empty list:

constructor() { super() this.options = []} Our render() function places the <label> and <select> tags in the DOM and will eventually populate the menu once it has data:

render() { const selectId = `select-list-${this.id}`; return html` <label for="${selectId}">${this.label} <select id="${selectId}" @change=${this.\_dispatch}> ${this.options.map(option => html`<option>${option}</option>`)} </select> </label> `;} Their clever use of JS template strings makes it much easier than ugly string concatenation.

That html in the return is doing alot of work, and not just returning text. You gotta read up on Lit to get more info b/c this is already too long.

The way we wired up reactivity in my Reef examples felt kludgy, and even the nicer way to do it in Reef feels kludgy to me. It’s really nice in Lit. This little addition to the <select> tag:

@change=${this.\_dispatch} says to call a function named _dispatch whenever the value changes. That’s in the component as well:

\_dispatch(e) { const options = { detail: e.target, bubbles: true, composed: true, }; this.dispatchEvent(new CustomEvent(`regionChanged`, options));} We setup a data structure and then fire off a custom event that our plot component will listen for. We’ve just linked them together on one side. Now we just need to populate the options list, using some data from R:

const regions = document.getElementById("regionsInput")regions.options = await (await R.webR.evalR(`colnames(WorldPhones)`)).toArray() That’ll make the menu appear.

Hearkening back to the main.js plot setup:

const plotOutput = document.getElementById("regionsOutput")plotOutput.region = regions.options[ 0 ]plotOutput.renderFunction = regionRenderplotOutput.render() we see that we:

find the element
set the default region to the first one in the popup
assign our R-created rendering function to it
and ask it nicely to render right now vs wait for someone to select something

The other side of that (region-plot.js) is a bit more complex. Let’s start with the properties:

static properties = { // we keep a local copy for fun region: { type: String }, // this is where our S asyncSvg: { type: String }, // a DOM-accessible id string (cld be handy) svgId: { type: String }, // the function to be called to render renderFunction: { type: Function }}; WebR === “async”, which is why you see that asyncSvg. Async is great and also a pain. There are way more functions in region-plot.js as a result.

We have to have something in renderFunction before WebR is powered up since the component will be alive before that. We’ll give it an anonymous async function that returns an empty SVG.

this.renderFunction = async () => `<svg></svg>` Oddly enough, our render function does not call the plotting function. This is what it does:

render() { return html` <div> <slot></slot> ${unsafeSVG(this.asyncSvg)} </div>`;} This bit:

<slot></slot> just tells render() to take whatever is wrapped in the tag and shove it there (it’s a bit more powerful than just that tho).

This bit:

${unsafeSVG(this.asyncSvg)} is just taking our string with SVG in it and letting Lit know we really want to live dangerously. Lit does its best to help you avoid security issues and SVGs are dangerous.

So, how do we render the plot? With two new functions:

// this is a special async callback mechanism that // lets the component behave normally, but do things // asynchronously when necessary.async connectedCallback() { super.connectedCallback(); // THIS IS WHERE WE CALL THE PLOT FUNCTION this.asyncSvg = await this.renderFunction(this.region, this.svgId); // We'll catch this event when the SELECT list changes or when // WE fire it, like we do down below. this.addEventListener('regionChanged', async (e) => { this.region = e.detail.value; const res = await this.renderFunction(this.region, this.svgId); // if the result of the function call is from the R function and // not the anonymous one we initialized the oject with // we need to tap into the `values` slot that gets return // with any call to WebR's `toJs()` if (res.values) this.asyncSvg = res.values[ 0 ] ; });}// special function that will get called when we // programmatically ask for a forced updateperformUpdate() { super.performUpdate(); const options = { detail: { value: this.region }, bubbles: true, composed: true, }; // we fire the event so things happen async this.dispatchEvent(new CustomEvent(`regionChanged`, options));} That finished the wiring up on the plotting end.

Serving ‘Just’ Desserts (Locally)I highly recommend using the tiny but awesome Rust-powered miniserve to serve things locally during development:

miniserve \ --header "Cache-Control: no-cache; max-age=300" \ --header "Cross-Origin-Embedder-Policy: require-corp" \ --header "Cross-Origin-Opener-Policy: same-origin" \ --header "Cross-Origin-Resource-Policy: cross-origin" \ --index index.html \ . You can use that (once installed) from the local justfile, which (presently) has four semantically named actions:

install-miniserve
serve
rsync
github

You’ll need to make path changes if you decide to use it.

FINI realize this is quite a bit to take in, and — as I keep saying — most folks will be better off using WebR in Shiny (when available) or Quarto.

Lit gives us reactivity without the bloat that comes for the ride with Vue and React, so we get to stay in Vanilla JS land. You’ll notice there’s no “npm” or “bundling” or “rollup” here. You get to code in whatever environment you want, and serving WebR-powered pages is, then, as simple as an rsync.

Drop issues at the repo.

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Linking Lit’s Lightweight Web Components And WebR For Vanilla JS Reactivity

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Garcia-Lerma E, Carmezim J, Satorra P, Peñafiel J, Pallares N, Santos N, Tebé C.
Biostatistics Unit, Bellvitge Biomedical Research Institute (IDIBELL)

The REDCapDM package allows users to read data exported directly from REDCap or via an API connection. It also allows users to process the previously downloaded data, create reports of queries and track the identified issues.

The diagram below shows the data management cycle: from data entry in REDCap to obtain data ready for the analysis.

The package structure can be divided into three main components: reading raw data, processing data and identifying queries. Typically, after collecting data in REDCap, we will have to follow these three components in order to have a final validated dataset for analysis. We will provide a user guide on how to perform each one of these steps using the package’s functions. For data processing and query identification, we will use the COVICAN data as an example (see the package vignette for more information about this built-in dataset).

Read data: redcap_data

The redcap_data function allows users to easily import data from a REDCap project into R for analysis.

To read exported data from REDCap, use the arguments data_path and dic_path to, respectively, describe the path of the R file and the REDCap project’s dictionary:

dataset <- redcap\_data(data\_path="C:/Users/username/example.r", dic\_path="C:/Users/username/example\_dictionary.csv") Note: The R and CSV files exported from REDCap must be located in the same directory.

If the REDCap project is longitudinal (contains more than one event) then a third element should be specified with the correspondence of each event with each form of the project. This csv file can be downloaded in the REDCap of the project following these steps: Project Setup < Designate Instruments for My Events < Download instrument-event mappings (CSV).

dataset <- redcap\_data(data\_path="C:/Users/username/example.r", dic\_path="C:/Users/username/example\_dictionary.csv", event\_path="C:/Users/username/events.csv") Note: if the project is longitudinal and the event-form file is not provided using the event_path argument, some steps of the processment can not be performed.

Another way to read data exported from a REDCap project is using an API connection. To do this, we can use the arguments uri and token which respectively refer to the uniform resource identifier of the REDCap project and the user-specific string that serves as the password:

dataset\_api <- redcap\_data(uri ="https://redcap.idibell.cat/api/", token = "55E5C3D1E83213ADA2182A4BFDEA") In this case, there is no need to specify the event-form file since the function will download it automatically using the API connection, if the project is longitudinal.

Remember that the token would give anyone access to all the project’s information. You should be careful about who you give this information to.

This function returns a list with 3 elements (imported data, dictionary and event-form mapping) which can then be used for further analysis or visualization.

Data process: rd_transformThe main function involved in the processing of the data is rd_transform. This function is used to process the REDCap data read into R using the redcap_data, as described above. Using the arguments of the function we can perform different type of transformations of our data.

As previously stated, we will use the built-in dataset covican as an example.

The only necessary elements that must be provided are the dataset to be transformed and the corresponding dictionary. If the project is longitudinal, as in the case of covican, also the event-form dataset should be specified. These elements can be specified directly using the output of the redcap_data function or separately in different arguments.

```

Option A: list object covican_transformed <- rd_transform(covican)#Option B: separately with different argumentscovican_transformed <- rd_transform(data = covican$data, dic = covican$dictionary, event_form = covican$event_form)#Print the results of the transformationcovican_transformed$results

```

1. Recalculating calculated fields and saving them as '[field\_name]\_recalc'| Total calculated fields | Non-transcribed fields | Recalculated different fields ||:-----------------:|:----------------:|:-----------------------:|| 2 | 0 (0%) | 1 (50%) || field\_name | Transcribed? | Is equal? ||:-------------------:|:------------:|:---------:|| age | Yes | FALSE || screening\_fail\_crit | Yes | TRUE |2. Transforming checkboxes: changing their values to No/Yes and changing their names to the names of its options. For checkboxes that have a branching logic, when the logic is missing their values will be set to missingTable: Checkbox variables advisable to be reviewed| Variables without any branching logic ||:-------------------------------------:|| type\_underlying\_disease |3. Replacing original variables for their factor version4. Deleting variables that contain some patterns This function will return a list with the transformed dataset, dictionary and the output of the results of the transformation.

As we can see, there are 4 steps in the transformation and they are briefly explained in the output of the function. This four steps are:

1. 1. 1. Recalculation of REDCap calculated fields 2. Checkbox transformation 3. Replacement of the original variable by its factor version 4. Elimination of variables containing some pattern

In addition, we can change the final structure of the transformed dataset by specifying in the final_format argument whether we want our data to be split by event or by form.

For more examples and information on extra arguments, see the vignette.

QueriesQueries are very important to ensure the accuracy and reliability of a REDCap dataset. The collected data may contain missing values, inconsistencies, or other potential errors that need to be identified in order to correct them later.

For all the following examples we will use the raw transformed data: covican_transformed.

rd_queryThe rd_query function allows users to generate queries by using a specific expression. It can be used to identify missing values, values that fall outside the lower and upper limit of a variable and other types of inconsistencies.

MissingsIf we want to identify missing values in the variables copd and age in the raw transformed data, a list of required arguments needs to be supplied.

example <- rd\_query(covican\_transformed, variables = c("copd", "age"), expression = c("%in%NA", "%in%NA"), event = "baseline\_visit\_arm\_1")# Printing resultsexample$results

Report of queries| Variables | Description | Event | Query | Total | | --- | --- | --- | --- | --- | | copd | Chronic obstructive pulmonary disease | Baseline visit | The value should not be missing | 6 | | age | Age | Baseline visit | The value should not be missing | 5 |

ExpressionsThe rd_query function is also able to identify outliers or observations that fulfill a specific condition.

example <- rd\_query(variables="age", expression=">70", event="baseline\_visit\_arm\_1", dic=covican\_transformed$dictionary, data=covican\_transformed$data)# Printing resultsexample$results

Report of queries| Variables | Description | Event | Query | Total | | --- | --- | --- | --- | --- | | age | Age | Baseline visit | The value should not be >70 | 76 |

More examples of both functions can be seen at the vignette.

OutputWhen the rd_query function is executed, it returns a list that includes a data frame with all the queries identified and a second element with a summary of the number of generated queries in each specified variable for each expression applied:

| Identifier | DAG | Event | Instrument | Field | Repetition | Description | Query | Code | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 100-58 | Hospital 11 | Baseline visit | Comorbidities | copd | · | Chronic obstructive pulmonary disease | The value is NA and it should not be missing | 100-58-1 |

The data frame is designed to aid users in locating each query in their REDCap project. It includes information such as the record identifier, the Data Access Group (DAG), the event in which each query can be found, along with the name and the description of the analyzed variable and a brief description of the query.

check_queriesOnce the process of identifying queries is complete, the typical approach would be to adress them by modifying the original dataset in REDCap and re-run the query identification process generating a new query dataset.

The check_queries function compares the previous query dataset with the new one by using the arguments old and new, respectively. The output remains a list with 2 items, but the data frame containing the information for each query will now have an additional column (“Modification”) indicating which queries are new, which have been modified, which have been corrected, and which remain pending. Besides, the summary will show the number of queries in each one of these categories:

check <- check\_queries(old = example$queries, new = new\_example$queries)# Print resultscheck$results

Comparison report| State | Total | | --- | --- | | Pending | 7 | | Solved | 3 | | Miscorrected | 1 | | New | 1 |

There are 7 pending queries, 3 solved queries, 1 miscorrected query, and 1 new query between the previous and the new query dataset.

Note: The “Miscorrected” category includes queries that belong to the same combination of record identifier and variable in both the old and new reports, but with a different reason. For instance, if a variable had a missing value in the old report, but in the new report shows a value outside the established range, it would be classified as “Miscorrected”.

Query control output:

| Identifier | DAG | Event | Instrument | Field | Repetition | Description | Query | Code | Modification | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 100-58 | Hospital 11 | Baseline visit | Comorbidities | copd | · | Chronic obstructive pulmonary disease | The value is NA and it should not be missing | 100-58-1 | Pending | | 100-79 | Hospital 11 | Baseline visit | Comorbidities | copd | · | Chronic obstructive pulmonary disease | The value is NA and it should not be missing | 100-79-1 | New | | 102-113 | Hospital 24 | Baseline visit | Demographics | age | · | Age | The value is NA and it should not be missing | 102-113-1 | Pending | | 105-11 | Hospital 5 | Baseline visit | Comorbidities | copd | · | Chronic obstructive pulmonary disease | The value is NA and it should not be missing | 105-11-1 | Pending |

Future improvementsIn the short term, we would like to make some improvements to the query identification and tracking process to minimise errors and cover a wide range of possible structures. We would also like to extend the scope of the data processing to cover up specific transformations of the data that may be required in some specific scenarios. As a long-term plan, we would like to complement this package with the development of a shiny application to facilitate the use of the package and make it as user-friendly as possible.

REDCapDM: a package to access and manage REDCap data was first posted on March 25, 2023 at 3:29 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: REDCapDM: a package to access and manage REDCap data

View Details

[This article was first published on R-posts.com, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Learn how to design beautiful tables in R! Join our workshop on Designing Beautiful Tables in R which is a part of our workshops for Ukraine series.

Here’s some more info:

Title: Designing Beautiful Tables in R

Date: Thursday, April 27th, 18:00 – 20:00 CEST (Rome, Berlin, Paris timezone)

Speaker: Tanya Shapiro is a freelance data consultant, helping businesses make better use of their data with bespoke analytical services. She is passionate about data visualization and design, and fell in love with the online R community via #TidyTuesday. When she’s not working on data projects, you can find her cycling or exploring downtown St. Petersburg, Florida.

Description: When we think about data visualization, bar charts and line charts are often top of mind – but what about tables? Tables are a great way to summarize and display different metrics across many records. In this workshop, we will learn how to design visually engaging tables in R and how to enhance them with HTML/CSS techniques. From sparklines, to heatmaps, to embedded images, we’ll cover a variety of tricks to help elevate your tables!

Minimal registration fee: 20 euro (or 20 USD or 800 UAH)

How can I register?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro. Feel free to donate more if you can, all proceeds go directly to support Ukraine.
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the registration form, attaching a screenshot of a donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after donation).

How can I sponsor a student?

Go to https://bit.ly/3wvwMA6 or https://bit.ly/3PFxtNA and donate at least 20 euro (or 17 GBP or 20 USD or 800 UAH). Feel free to donate more if you can, all proceeds go to support Ukraine!
Save your donation receipt (after the donation is processed, there is an option to enter your email address on the website to which the donation receipt is sent)
Fill in the sponsorship form, attaching the screenshot of the donation receipt (please attach the screenshot of the donation receipt that was emailed to you rather than the page you see after the donation). You can indicate whether you want to sponsor a particular student or we can allocate this spot ourselves to the students from the waiting list. You can also indicate whether you prefer us to prioritize students from developing countries when assigning place(s) that you sponsored.

You can also find more information about this workshop series, a schedule of our future workshops as well as a list of our past workshops which you can get the recordings & materials here.

Looking forward to seeing you during the workshop!

Designing Beautiful Tables in R was first posted on March 25, 2023 at 3:28 pm.
To leave a comment for the author, please follow the link and comment on their blog: R-posts.com.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Designing Beautiful Tables in R

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionI will demonstrate how to generate a 1,000 row and column matrix with random numbers in R, and then save it in different file formats. I will also show how to get the file size of each saved object and benchmark how long it takes to read in each file using different functions.

Generating a large matrixTo generate a 1,000 row and column matrix with random numbers, we can use the matrix() function and the runif() function in R. Here’s the code to generate the matrix:

```

set seed for reproducibilityset.seed(123)# number of rows/columns in matrixn <- 1000# generate matrix with random normal valuesmat <- matrix(runif(n^2), nrow = n)

``` This code sets the random number generator seed to ensure that the same random numbers are generated every time the code is run. It then generates a vector of 1,000^2 random numbers using the runif() function, and creates a matrix with 1,000 columns using the matrix() function.

Saving the matrix in different file formatsWe can save the generated matrix in different file formats using different functions in R. Here are the functions we will use for each file format:

CSV: write.csv()
RDS: saveRDS()
FST: write_fst()
Arrow: write_feather()

Here’s the code to save the matrix in each file format:

library(fst)library(arrow)# Save matrix in different file formatswrite.csv(mat, "matrix.csv", row.names=FALSE)saveRDS(mat, "matrix.rds")write\_fst(as.data.frame(mat), "matrix.fst")write\_feather(as\_arrow\_table(as.data.frame(mat)), "matrix.arrow") This code saves the matrix in each file format using the corresponding function, with the file name specified as the second argument. Getting the file size of each saved object

To get the file size of each saved object, we can use the file.size() function in R. Here’s the code to get the file size of each saved object:

```

Get file size of each saved objectcsv_size <- file.size("matrix.csv") / (1024^2)rds_size <- file.size("matrix.rds") / (1024^2)fst_size <- file.size("matrix.fst") / (1024^2)arrow_size <- file.size("matrix.arrow") / (1024^2)# Print file size in human-readable formatprint(paste("CSV file size in MB:", format(csv_size, units="auto")))

```

[1] "CSV file size in MB: 17.17339"

print(paste("RDS file size in MB:", format(rds\_size, units="auto")))

[1] "RDS file size in MB: 5.079627"

print(paste("FST file size in MB:", format(fst\_size, units="auto")))

[1] "FST file size in MB: 7.700841"

print(paste("Arrow file size in MB:", format(arrow\_size, units="auto")))

[1] "Arrow file size in MB: 6.705355" This code uses the file.size() function to get the file size of each object, and stores the file size of each object in a separate variable.

Finally, it prints the file size of each object in a human-readable format using the format() function with the units=“auto” argument. The units=“auto” argument automatically chooses the most appropriate unit (e.g., KB, MB, GB) based on the file size.

Benchmarking file read timesTo benchmark how long it takes to read in each file, we can use the {rbenchmark} package in R. In this example, we will compare the read times for the CSV file using four different functions: read.csv(), read_csv() from the {readr} package, fread() from the {data.table} package, and vroom() from the {vroom} package. We will also benchmark the read times for the RDS file using readRDS(), the FST file using read_fst(), and the Arrow file using read_feather().

Here’s the code to benchmark the read times:

```

Load rbenchmark packagelibrary(rbenchmark)library(readr)library(data.table)library(vroom)library(dplyr)n = 30# Benchmark read times for CSV filebenchmark( # CSV File "read.csv" = { a <- read.csv("matrix.csv") }, "read_csv" = { b <- read_csv("matrix.csv") }, "fread" = { c <- fread("matrix.csv") }, "vroom alltrep false" = { d <- vroom("matrix.csv") }, "vroom alltrep true" = { dd <- vroom("matrix.csv", altrep = TRUE) }, # Replications replications = n, # Columns columns = c( "test","replications","elapsed","relative","user.self","sys.self")) |> arrange(relative)

```

test replications elapsed relative user.self sys.self1 fread 30 1.35 1.000 0.90 0.202 vroom alltrep true 30 6.59 4.881 3.58 1.713 vroom alltrep false 30 6.62 4.904 3.43 1.624 read.csv 30 33.86 25.081 26.15 0.225 read\_csv 30 82.39 61.030 20.39 3.47

```

RDS Filebenchmark( # RDS File "readRDS" = { e <- readRDS("matrix.rds") }, "read_rds" = { f <- read_rds("matrix.rds") }, # Repications replications = n, # Columns columns = c( "test","replications","elapsed","relative","user.self","sys.self")) |> arrange(relative)

```

test replications elapsed relative user.self sys.self1 read\_rds 30 0.95 1.000 0.74 0.012 readRDS 30 0.97 1.021 0.74 0.02

```

FST / Arrowbenchmark( # FST "read_fst" = { g <- read_fst("matrix.fst") }, # Arrow "arrow" = { h <- read_feather("matrix.arrow") }, # Replications replications = n, # Columns columns = c( "test","replications","elapsed","relative","user.self","sys.self")) |> arrange(relative)

```

test replications elapsed relative user.self sys.self1 read\_fst 30 0.21 1.000 0.05 0.122 arrow 30 3.00 14.286 1.60 0.11 This code loads the {rbenchmark} package, and uses the benchmark() function to compare the read times for each file format. We specify the function to use for each file format, and set the number of replications to 10. Conclusion

In this blog post, we demonstrated how to generate a large matrix with random numbers in R, and how to save it in different file formats. We also showed how to get the file size of each saved object, and benchmarked the read times for each file format using different functions.

This example demonstrates the importance of choosing the appropriate file format and read function for your data. Depending on the size of your data and the requirements of your analysis, some file formats and functions may be more efficient than others.

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: How fast do the files read in?

View Details

[This article was first published on R on Stats and R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. Introduction * What types of jobs are available? + Statistician + Data scientist, data/business analyst, data engineer or machine learning engineer + Actuary or actuarial analyst + Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst + Business intelligence analyst + Operational researcher or quality control analyst + Market or survey researcher + Economist or econometrician + (Freelance) consultant + Teacher + PhD student * Conclusion This post has been written in collaboration with Daniel Williams.*

IntroductionA reader recently contacted me because he was hesitating between starting his studies in statistics or economics. I studied economics and I am now doing a PhD in statistics, so he reached out to me to know what I thought about these two fields, and in particular, what were my feelings about the career opportunities available with these two degrees.

Before considering pursuing a graduate degree, it is important to have a clear understanding of the skills you will develop and the career opportunities that will become available to you.

I leave it to those who have studied economics and work in this field to discuss career opportunities with a degree in economics. In this post, I will focus on the career options that are available for people with a degree in statistics.

Bear in mind that there are many professional opportunities with a degree in statics. Indeed, statistical expertise is highly sought after in a wide range of computing and data analysis job roles; as soon as there are data, statistical expertise is required. Moreover, there are almost as many different jobs as people. Therefore, the list of jobs below is non-exhaustive and you may get a job that is not mentioned in this list. However, I hope it will still give you an overview of what you can expect after your studies.

If you believe I missed one—if you studied statistics and your job is not mentioned here for example, feel free to leave a comment at the end of the post.

What types of jobs are available?If you are skilled in statistics and R programming, there are a plethora of job opportunities available to you.

Some of the roles you can pursue include:

Statistician: Collect, analyze and interpret data in a wide range of fields, including economics, finance, marketing, social and medical sciences.
Data scientist, data/business analyst, data engineer or machine learning engineer: Collect, analyze and interpret large and complex data sets using statistical and machine learning techniques to gain insights and inform decision-making to business and organizations.
Actuary or actuarial analyst: Use statistical modeling to assess risk and uncertainty, and to create financial projections for insurance and investment companies.
Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analyst: Use statistical and mathematical modeling to analyze financial and market data, identify and evaluate risks for investment and trading decisions.
Business intelligence analyst: Use data analysis and visualization tools to create reports and dashboards that help businesses make informed decisions.
Operational researcher or quality control analyst: Use mathematical and statistical techniques to monitor and optimize the quality of products and processes in manufacturing.
Market or survey researcher: Conduct research and surveys, gather and analyze data to help businesses develop and improve their marketing strategies.
Economist or econometrician: Use statistical methods to analyze (socio)economic data and develop economic models to inform policy and decision-making.
Teacher: Facilitate learning and provide guidance to students in order to help them acquire knowledge, skills, and values that will prepare them for their future.
(Freelance) consultant: Provide statistical consulting services to clients, helping them solve problems using a combination of statistical methods and business expertise.

In the following sections, we delve deeper into the aforementioned job roles and outline the qualifications expected of prospective candidates.

StatisticianAs a graduate in statistics, you have the potential to land a coveted statistician’s job in a reputable private company or public agency. These types of roles typically involve analyzing and assessing data, and utilizing various tools and software to manage data effectively.

With data proliferation and more and more companies/organizations that rely heavily on data analysis, your skills will be valuable in a broad range of fields and industries. Being a statistician mean that you could work for instance as an:

environmental statistician: analyze data related to environmental issues, such as climate change and pollution, to inform policy and decision-making,
sports statistician: analyze data related to sports performance, such as player statistics and game outcomes, to inform coaching and strategy decisions,
government statistician: collect, analyze and interpret data for government agencies and public departments to guide policymakers in their decisions,
biostatistician: use statistical methods and data analysis techniques to study and interpret medical and (public) health-related data,
etc.

Statistical expertise is required wherever there is data, so feel free to specialize in the field or industry you care about most!

Data scientist, data/business analyst, data engineer or machine learning engineerAs a relatively new job role in various organizations, data scientists and data analysts are experts in manipulating, analyzing and interpreting data. These roles typically require the analysis of large datasets using statistical methods to assist organizations in making more informed business decisions. These professionals dive into unstructured information to uncover valuable insights for businesses, ultimately increasing their revenue.

To pursue this career, one would usually require a graduate degree in science, statistics, or mathematics, as well as training in data mining. It is also expected that you are familiar with one or several programming languages such as R, Python or SAS.1

Statistics graduates can also pursue their career as machine learning engineers. Job roles in this field typically involve conducting experiments and implementing machine learning algorithms. Consequently, many organizations hire these professionals to create a wide range of AI products. To do so, they must possess strong programming and statistical skills. Data science and software engineering knowledge are also beneficial.

Actuary or actuarial analystActuaries usually apply statistical modeling tools on data related to retirement, financial/insurance products, accidents or mortality. As an actuary, one would be responsible for analyzing financial costs and risks. It means examining uncertainties associated with investments and other financial products and making predictions about the risks involved in a particular venture.

Actuaries are also in charge of writing insurance proposals, determining policy terms, and calculating premiums for different products, among other responsibilities. You may also play a vital role in deciding whether to accept or reject insurance applications by evaluating their risks. You will draw upon your statistical, actuarial, and background information to accomplish these tasks.

Typically, actuaries have a range of financial companies as clients, and they must identify potential risks and recommend compensation strategies accordingly. It means that their work is closely related to insurance products, as they must assess the likelihood of certain events occurring and the resulting financial impact.

Typically, those who pursue this role hold a degree in actuarial science, mathematics or statistics, and are trained in statistical analysis tools and software. Note that each country has its own specific accreditation related to this profession.

Financial (risk) analyst, investment analyst, financial trader, financial manager or quantitative analystAs a financial analyst, one is required to gather data from various sources, organize and analyze historical results, and make projections and forecasts. Furthermore, in addition to analyzing data, their role often involves reporting financial results to the board of directors/management to help them in setting the overall strategy and direction of the company.

Investment analysts are experts in evaluating different financial assets such as stocks, securities, and bonds. They do not stop there, though. They also conduct research and make crucial decisions about purchasing business financials.

As a financial trader, you will have a keen understanding of financial markets and the ability to execute trades (i.e., buy or sell shares, bonds, and other assets) on behalf of clients. Note that there are also various sub-roles to explore within this job segment.

Financial managers combine their passion for finance with their love of numbers and statistics. Financial managers are experts at creating and interpreting complex financial reports, providing invaluable insights into a company’s financial health. They can also advise management on investment strategies and assist with vital financial decisions. With their keen analytical skills, financial managers keep a watchful eye on daily financial activities and help ensure a company’s long-term success.

These skills are in high demand in financial organizations, such as banks, actuarial firms, insurance companies (commercial insurance, reinsurance, general insurance, life insurance, car insurance, etc.), and other similar establishments.

The educational background of these professionals is diverse. Graduates in finance, mathematics, and statistics all bring unique perspectives to the table.

Business intelligence analystA business intelligence (BI) analyst is responsible for collecting, organizing, analyzing, and presenting large amounts of data from various sources such as databases, spreadsheets, and software applications. They use various tools and techniques to identify trends, patterns, and relationships in the data to create reports, dashboards, and visualizations to communicate their findings to stakeholders. They may also be responsible for monitoring business performance metrics and KPIs, identifying areas for improvement, and making recommendations to optimize business operations.

Additionally, BI analysts may also be involved in the development and implementation of data-driven software applications, such as business intelligence platforms and data warehouses, to improve data accessibility, accuracy, and efficiency. Overall, the role of a BI analyst is crucial in filling the gap between business analysts and the IT team.

The most common software tools and platforms used by BI analysts are, at the time of writing this post, R Shiny, Microsoft Power BI, Tableau, QlikView and SAS BI.

Operational researcher or quality control analystOperational researchers are responsible for using mathematical and analytical methods to solve complex problems and optimize business operations. They collect and analyze data, develop models and algorithms, test and validate them, and make recommendations for improvements.

On the other hand, quality control analysts are responsible for ensuring that products or services meet the required standards of quality. They monitor and analyze product or service performance, identify areas for improvement, and develop strategies for maintaining or improving quality. They may also work with production teams to develop and implement quality control processes and procedures.

While operational researchers focus on optimizing processes and systems, quality control analysts focus on ensuring that the end product or service meets the required quality standards. Both roles require analytical skills and the ability to work with data, but operational researchers focus more on mathematical modeling and optimization, while quality control analysts focus more on monitoring and improving quality.

Individuals who pursue job roles of this nature typically possess strong mathematical abilities, often having graduated with degrees in statistics or mathematics. They are trained to analyze vast amounts of data and are well-versed in various analytical tools, such as simulation, mathematical modeling, and data science.

Market or survey researcherMarket or survey researchers are professionals who specialize in conducting research to gather information about consumer behavior and preferences, market trends, and competitive landscapes. They use various research methods and techniques to collect and analyze data, and provide insights and recommendations to businesses and organizations.

Market research professionals typically work with marketing agencies on various projects for clients across different sectors. In addition to a degree in statistics, knowledge in marketing and familiarity with the industry is beneficial.

Economist or econometricianEconomists and econometricians study and analyze economic systems, markets, and policies using quantitative methods and models. They collect and analyze data, develop economic models, conduct economic analysis, provide recommendations to policymakers and organizations, and communicate their findings to a variety of audiences. Their role is to inform economic policies and decisions in both public and private sectors.

If you have a background in statistics in addition to knowledge in economics, there is a whole world of vocational opportunities in economics just waiting for you. Indeed, armed with the ability to appropriately analyze data (thanks to your background in statistics) and the ability to understand (socio)economic issues and financial data, you will be a hot commodity as an advisor to governments and businesses on all economic decisions.

(Freelance) consultantFrom a general point of view, a consultant (or freelance consultant) is a professional who provides expert advice and guidance to organizations or individuals on a contractual basis. They analyze problems, develop solutions, and provide recommendations to clients in a specific field or industry.

A consultant specializing in statistics provides expert advice and guidance to organizations or individuals on statistical analysis, data management, and modeling. They use statistical tools and techniques to analyze and interpret complex data, and provide insights and recommendations based on their analyses. The role of a statistical consultant may include identifying research questions, designing studies, collecting and managing data, analyzing data using appropriate statistical methods, interpreting results, and presenting findings to clients. They may also assist clients with the implementation of statistical methods and tools, provide training on statistical software and techniques, and develop custom statistical models for specific applications.

As a side note, note that many consultants in statistics provide their service as data visualization consultants. These consultants are responsible for helping clients to effectively communicate complex data through the creation of clear and visually appealing graphics, charts, and other forms of visual representation that can aid in decision-making and enhance understanding.

You can either be a consultant in a consulting firm, or a freelance consultant. If you choose to be a freelance consultant, you can more easily select the projects you want to work on and the industry you want to focus on, but you will also need to take care of all the administrative tasks such as finding and building relationships with clients, do your bookkeeping, etc.

If you are not ready to be a full-time freelance consultant but would like to experience a taste of it, you can always keep your primary job and accept a couple of side projects. This way, you will gradually build your portfolio and create relationships with clients, all that with the advantage of keeping a safe source of income. This is what I am doing with datanalyze.be, and I recommend it to anyone who is considering the option of being a freelance in the future.

TeacherIf you care about education, you like to transmit your knowledge and explain complex things in a simple manner, you may be interested in being a teacher in high school or a university professor. With a degree in statistics, you may be eligible to teach different subjects such as mathematics, statistics, physics, or science in general. Your ultimate goal is to educate and inspire students to develop an understanding and appreciation of these subjects (which are often not well appreciated by students), as well as to develop critical thinking skills, problem-solving abilities, and scientific literacy.

Like actuaries, note that each country has its own specific accreditation related to this profession.

To know whether this job suits you, you can start by being a private tutor. This way, you will experience what the job is like and you will be more able to tell whether it is the direction you want to take.

To become a university professor, note that a PhD (and even sometimes a postdoc) is usually required. This brings me to the last career option I would like to mention: a PhD.

I do not include a PhD in the list of the 10 potential career options because it is more seen as an additional degree rather than a job, and even for those who see it as a job, it is a temporary one. However, I would still like to mention it because it is worth considering with a degree in statistics (I am of course biased, but my biased opinion may be of interest to some readers).

PhD studentFor those who enjoy academic research, pursuing a career as a researcher after taking a statistics degree may be a worthwhile consideration. Typically, pursuing a PhD in statistics involves working in close collaboration with one or two university professors (your supervisors) on a specific topic.

I cannot speak for all PhD students, but I can speak from my experience. It involves conducting original research in statistical theory and methodology, developing new statistical models and methods, and applying statistical techniques to solve real-world problems or to advance knowledge in a specific field. My research focuses on applying biostatistical procedures to cancer patients, so it is a rather applied PhD, but many of my colleagues work on a more theoretical subject.

We also sometimes participate in academic conferences, present our research findings, and collaborate with other researchers in our field of research.

After completing our PhD, we can can continue to evolve in academia (and become a university professor for instance), or decide to work in a private or public research organization. Because professionals with a PhD in statistics become experts in statistical theory and methodology, commercial and public organizations often seek such profiles.

The length of a PhD depends on the country and the type of contract. If you are interested to know more, feel free to contact me, I might be able to help you (at least with how it goes in Belgium). Otherwise, the program director at your university will definitely be able to answer the questions you may have.

ConclusionAs you have seen, with a degree in statistics and knowledge in programming, you will be able to pursue positions at various companies or organizations. You can also always enhance your qualifications with additional courses if you feel that you miss some skills that are required for the job you want to apply to.

Statisticians are in demand in government departments across many regions and sectors. Actuarial firms, banks, investment firms, and market research organizations are also great potential employers.The demand for statistical and analytical skills is on the rise across various fields. Job seekers with these skills are particularly needed. However, it is becoming increasingly common for these positions to require specialized education and training. For example, if you have a degree in statistics, pursuing a finance specialization could open doors to opportunities in banking, investment, accountancy, or insurance firms. Last but not least, those passionate about solving complex problems may find roles as researchers (within or outside academia) appealing.

Thanks for reading.

I hope this article helped you to get an overview of the career options you have with a degree in statistics.

If you are looking for a job, I wish you the best of luck in finding your dream job! Remember that you will find a variety of job openings in academic, research-oriented, and analytical fields on online and offline job boards. Moreover, do not underestimate the power of friends, colleagues and family: make sure to network with people in your field to increase the chance of getting a job.

As always, if you have a question or a suggestion related to the topic covered in this article, please add it as a comment so other readers can benefit from the discussion.

There are many programming languages for data analysis, and it will certainly continue to evolve in the future. The most common ones at the time of writing this post are: R, Python, SAS, Jamovi, SPSS, JMP, Stata and Matlab. Depending on the role and the organization, more or less programming skills will be required.︎ To leave a comment for the author, please follow the link and comment on their blog: R on Stats and R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: 10 potential career options with a degree in statistics

View Details

[This article was first published on R with White Dwarf, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Sobre este postEstamos creando mapas de datos que muestran los cambios durante un período de tiempo para diferentes países y orientado a todo tipo de ciudades. Esto básicamente significa que necesitamos mapear cualquier región del mundo con R. Hoy en día existen todo tipo de paquetes y técnicas para hacerlo. Quiero compartir la estrategia que utilicé con ggplot2 y maps, utilizando el soporte de Open Street Map para obtener las coordenadas de las ciudades y finalmente hacerlo interactivo con shiny.

Estas publicaciones comparten mi camino en la creación de la aplicación Shiny. Es un proyecto vivo en el que estoy trabajando actualmente y decidí compartir mis experiencias durante el proceso de creación. Estas publicaciones no son sólo acerca de Shiny apps, si no más bien sobre la creación del paquete detrás, incluyendo temas sobre la generación de funciones, creación de los mapas, clases de objetos, entre otros, incluyendo cualquier tema interesante que aparezca en el camino. Es mi manera de contribuir a la comunidad de R y al mismo tiempo documentar el proyecto en si mismo.

Pueden encontrar todas las publicaciones en este tema bajo la etiqueta maps-app (incluyendo las versiones en inglés).

También pueden encontrar el estado actual del proyecto en mi GitHub repo mapic.

Este post está escrito originalmente en español, desde el aeropuerto de Amsterdam, de camino a México. Espero que lo disfruten. Siéntanse libres de dejar cualquier tipo de comentario y/o pregunta al final.

MotivaciónComo mencioné en los posts anteriores de la serie, he estado trabajando últimamente en el código para la creación de los mapas y he hecho cambios que incrementan la eficiencia de las funciones, la lectura del código, y facilitan su uso, al mismo tiempo que me permite extender las funciones mas allá de cómo fueron originalmente diseñadas.

He mencionado en algunas ocasiones que el código evolucionó poco a poco a partir de scripts creados para generar el mapa específico de algún país. Por lo tanto, las primeras funciones son más bien una colección de los pasos utilizados para generar el mapa, atrapados en forma de funciones para automatizar el proceso.

Por este motivo había querido hacer cambios para adecuar las funciones a paradigmas mas adecuados a la programación funcional, que es el punto fuerte de R. Sin embargo, la prioridad era generar los mapas, así que la mayoría de mi tiempo iba dirigido a la creación de los mapas y el debug del código cuando era necesario. A esto debo agregar que este proyecto es parte de un trabajo voluntario para una ONG, de la cual pasé a ser el director de la división de investigación, lo que me generaba aún mas responsabilidades y trabajo. Y todo como trabajo secundario, separado de mi fuente principal de ingresos (que también esta basada en R).

Sin embargo, para bien o para mal, 2022 fue un año lleno de cambios y retos para mi y mi familia, lo que me obligó a dejar de lado el proyecto por un tiempo, resignar mi posición como director de la división, y enfocarme únicamente a mi carrera, mi salud y mi familia. Esto resultó en que, al recuperar la estabilidad en mi vida, me encontré con mas tiempo libre y menos obligaciones para re pensar el código y trabajar en ello. Adicionalmente, mi trabajo principal tuvo un giro que fue de la estadística a mas orientado a la programación en R, lo cual me ha dado mas herramientas para mejorar el código, y me ha motivado a retomar viejas lecciones sobre programación funcional y, sobre todo, iteración.

Esto me permitió mejorar las dos funciones principales: la encargada del webscrapping y la que manda los datos a SQLite. Puedes encontrar las funciones originales en el post anterior Mapa de cualquier región del mundo con R – Parte I: El mapa base y compararlos con las nuevas funciones mejoradas en este.

Webscrapp a SQLiteLa función webscrap_to_sqlite se encarga de enviar las coordenadas encontradas por Open Street Map a nuestra base de datos. La función como está definida originalmente es poco efectiva, ya que hace cada operación línea por línea. También es muy rígida en la forma en la que dirige los valores de las regiones, tanto su petición a la API como la colocación de los valores en la base de datos, lo que hace cualquier extensión o modificación muy complicada.

Por lo tanto, es la función que recibió mas cambios, fue prácticamente re escrita desde ceros, haciendo la búsqueda mas eficiente, permitiendo también búsqueda interna de los datos ya almacenados; mas flexible, lidiando con los parámetros de las regiones de forma mas clara; y mas entendible, mejorando el estilo del código.

webscrap\_to\_sqlite <- function(db.name, dat, city = "City", country = "Country", region = NULL, state = NULL, county = NULL, db\_backup\_after = 10) { ## Cargas las liberarias necesarias require(RSQLite) require(dplyr) ## 1. Conexion a la DB con <- dbConnect(drv = SQLite(), dbname = db.name) dbExecute(conn = con, "CREATE TABLE IF NOT EXISTS orgs (ID INTEGER UNIQUE, City TEXT, Country TEXT, Region TEXT, State TEXT, County TEXT, osm\_name TEXT, lon REAL, lat REAL)") db <- as\_tibble(dbReadTable(con, "orgs")) ## 2. Filtrado de los datos new\_coords <- data.frame() dat\_local <- compare\_db\_data(db.name, dat) df\_len <- nrow(dat\_local) ## 3. Mientras haya filas en DF repetimos: if (df\_len != 0) { ## 3.1 Definir tamaño de la sub-muestra dat\_local <- dat\_local[c(1:db\_backup\_after), ] dat\_local <- filter(dat\_local, rowSums(is.na(dat\_local)) != ncol(dat\_local)) ## 3.2 for loop para el webscrapping for (i in 1:nrow(dat\_local)) { print(paste0("Searching entry ", dat\_local[["ID"]][i])) ## 3.3 Abstracción de la info rg <- ifelse(is.null(region), "", dat\_local[[region]][i]) st <- ifelse(is.null(state), "", dat\_local[[state]][i]) ct <- ifelse(is.null(county), "", dat\_local[[county]][i]) rcity <- dat\_local[[city]][i] rcountry <- dat\_local[[country]][i] ## 3.4 Obtener las coordenadas ## 3.4.1. Primero buscamos si ya existen en DB search\_query <- filter(db, City == rcity, Country == rcountry, Region == rg, State == st, County == ct) if (nrow(search\_query != 0)) { coords <- search\_query[1, ] coords$ID <- dat\_local[["ID"]][i] print("Found from memory") ## 3.4.2 Si aun no existen, busca en OSM API } else { coords <- coords\_from\_city(rcity, rcountry, Region = rg, State = st, County = ct) coords <- cbind(ID = dat\_local[["ID"]][i], City = rcity, Country = rcountry, Region = rg, State = st, County = ct, coords) } new\_coords <- rbind(new\_coords, coords) } ## Y envía sólo los nuevos resultados a la DB dbWriteTable(con, "orgs", new\_coords, append = TRUE) dbDisconnect(con) ## 3.4.3 Repetir webscrap\_to\_sqlite(db.name = db.name, dat = dat, city = city, country = country, region = region, state = state, county = county, db\_backup\_after = db\_backup\_after) ## 4. Terminar la iteracion } else { db\_final <- import\_db\_as\_df(db.name) size <- nrow(db\_final) not\_found <- nrow(db\_final[is.na(db\_final$lat), ]) message(paste("Search finished.\n", size, "entries searched.\n", not\_found, "ENTRIES NOT FOUND")) }} La función comienza llamando a las librerías necesarias y (1) conectándose a la base de datos, creándola si es necesario. (2) Luego genero dos data frames, uno vacío que almacenará nuevas coordenadas y otro relativo que contiene únicamente los datos que aún no existen en la base de datos. (3) Así pues, mientras haya datos en esta última data frame, la función continuará repitiéndose.

(3.1) Luego definimos la sub-muestra, que es un sub conjunto de dat_local del tamaño de db_backup_after y nos enfocamos únicamente en esta sub-muestra. (3.2) Sobre esa sub muestra hacemos las iteraciones utilizando for para obtener las coordenadas. Primero (3.3) preparamos la información como strings y después (3.4) buscamos las coordenadas. (3.4.1) Si ya existen en la base de datos lo tomamos de ahí, y si no (3.4.2) se buscan utilizado coords_from_city. Finalmente, (3.4.3) repetimos todo de nuevo permitiendo a la función llamarse a si misma.

Dado que en el paso 2 se filtran los datos que aún no están en la base de datos y en 3 se colocan los resultados de nuevas búsquedas en un data frame vacío, la función se llama a sí misma y aplica únicamente para cada sub conjunto de datos. Cuando finalmente compare_db_data arroja 0 valores por que todos los datos que se ingresaron a la función ya están contenidos en la base de datos, podemos salir de la función. En este caso decidí importar de nuevo los datos desde SQLite para obtener detalles de la búsqueda, y terminar la iteración enviando un mensaje al usuario sobre el total de entradas y la cantidad de las cuales no fueron encontradas.

Si comparamos esta función con la propuesta en mi post anterior, la función es completamente diferente pero el resultado final es el mismo. Los argumentos utilizados por la función también son los mismo y toman los mismos valores, lo cual evita conflictos para el usuario. El único parámetro nuevo es db_backup_after que nos permite controlar a cada cuantas filas se realiza la iteración. Un valor mas pequeño significa más iteraciones, lo que resultado en un mayor uso de la memoria local, pero también mayor agilidad para encontrar datos que ya existen en la DB. Por otro lado, un valor mas alto reduce el número de iteraciones pero incrementa el número de conexiones a la API. Por este motivo le he otorgado un valor pre definido de 10. Esto, además de ser un valor balanceado, también reduce la confusión del usuario que podría no estar familiarizado con los cambios.

Remover valores faltantes de la base de datosEn la propuesta anterior, únicamente las coordenadas encontradas eran enviadas a la base de datos, y las no encontradas se ignoraban. En la propuesta presente, todas las entradas se envían a la DB. Por lo tanto, es importante tener alguna opción para remover las entradas no encontradas.

Para ello generé la función remove_na_from_db, una función muy simple pero que le otorga al usuario una propuesta remover NAs automáticamente.

remove\_na\_from\_db <- function(db.file) { require(RSQLite) con <- dbConnect(drv = RSQLite::SQLite(), dbname = db.file) dbExecute(conn = con, "DELETE FROM orgs WHERE lon IS NULL OR trim(lon) = '';") dbDisconnect(con)} La función es únicamente una conexión a la base de datos que envía la orden de remover filas donde el campo lon está vacío, en sintaxis de SQLite. Esto es la manera mas segura, directa y rápida de hacerlo. También podríamos importar los datos de nuevo a R, filtrarlos y enviarlos de nuevo a SQLite, pero esto requeriría mayor uso de la memoria local, mayor cantidad de código y un mayor riesgo ya que requeriría re-escribir la base de datos a SQLite por completo. El poder de la librería RSQLite (o cualquier otra librería que conecta a R con SQL) está precisamente en la posibilidad de pasar ordenes escritas y ejecutadas directamente en SQL.

La obtención de las coordenadasLa función coords_from_city también recibió cambios considerables en lectura del código y flexibilidad, y un poco menores en funcionamiento y eficiencia.

coords\_from\_city <- function(city = NULL, country\_code, region = NULL, state = NULL, county = NULL) { require("RJSONIO") ## 1. Abstracción de regiones para OSM CityCoded <- gsub(" ", "%20", City) CountryCoded <- paste("&countrycodes=", CountryTwoLetter, sep = "") extras <- c(city = City, state = State, region = Region, county = County) extrasCoded <- "" if (!is.null(extras)) { for (i in 1:length(extras)) { if (extras[i] != "" && !is.na(extras[i]) && !grepl("^\\s*$", extras[i])) { valCoded <- gsub(" ", "%20", extras[i]) extrasCoded <- paste0(extrasCoded, "&", names(extras)[i], "=", valCoded) } } } ## 2. Respuesta link <- paste( "http://nominatim.openstreetmap.org/search?city=" , extrasCoded , CountryCoded , "&format=json" , sep = "" ) response <- try({fromJSON(link)}, silent = TRUE) if (class(response) == "try-error") { stop(response[1]) } else if (class(response) == "response") { response\_status <- http\_status(response) if (response\_status$category != "Success") { stop(response\_status$message) } } else if (is.list(response)) { ## 3. Organización de los resultados if (length(response) == 0) { message(paste("No results found for", extrasCoded)) coords <- data.frame("lon" = NA, "lat" = NA, "osm\_name" = as.character(NA)) } else if (length(response) == 1) { message(paste("Found", response[[1]]$display\_name)) coords <- data.frame( lon = response[[1]]$lon, lat = response[[1]]$lat, osm\_name = response[[1]]$display\_name ) } else { message(paste("Several entries found for", city, country\_code)) coords <- data.frame( lon = response[[1]]$lon, lat = response[[1]]$lat, osm\_name = response[[1]]$display\_name ) } ## 4. Salida como data frame return(coords)}> Error: <text>:68:0: unexpected end of input> 66: return(coords)> 67: }> ^ El principal cambio está en la sección 1, en lugar de pasar cada una de las regiones como su propio string y darles formato una por una, las he abstraído todas en un solo vector. Esto reduce la cantidad de código, el uso de memoria, y nos permite incluir la ciudad en la lista, convirtiéndolo también en un valor opcional. La razón por la que las había preparado por separado publicación anterior es simplemente porque la función creció lentamente: al principio solo necesitábamos ciudad, pero luego tuvimos que usar algunos campos adicionales según el país en el que trabajábamos. Para facilitarme las cosas, simplemente agregué cada campo de región según fuera necesario. Ahora que tengo tiempo para trabajar en el código, esta fue la primera función que modifiqué.

El paso 2 ahora imprime mensajes que nos ayudan a identificar el error cuando se trata de la conexión, al mismo tiempo que detiene el proceso. Ya sea un error local de conexión, o problemas del lado del API, obtendremos un mensaje y el proceso se detendrá, lo cual debe evitar tiempos de espera largos cuando no hay conexión y se está haciendo la búsqueda de muchas localidades.

El paso 3 cambia un poco la organización de los resultados, devolviendo siempre un data frame con las mismas columnas cuando los resultados no fueron encontrados, pero ahora con los campos vacíos en dicho caso. Esto ayuda a las funciones presentadas anteriormente para llenar la base de datos. Adicionalmente, cuando muchos resultados fueron encontrados, se imprime esta información en pantalla; por ahora es sólo como información. La idea es mantener este espacio para realizar cambios en el futuro que nos permitan seleccionar la opción de manera interactiva. Esto es algo que aún necesito pensar y planear por que, por un lado quiero utilizarlo en una aplicación Shiny, y por otro lado queremos mantener la habilidad de que el web scrapping suceda automáticamente con menor intervención posible.

Como ya he mencionado antes, estas nuevas funciones también nos permiten realizar búsquedas con el valor de ciudad vacío. Este fue un requisito solicitado en la última versión, ya que algunos usuarios comenzaron a hacer mapas por regiones, mientras que otros, al no encontrar ciudades muy pequeñas, decidieron agrupar los datos por región. Gracias a los cambios realizados en coords_from_city, la función webscrap_to_sqlite ahora puede obtener resultados cuando el valor para ciudad es NA, considerando que se encuentren las coordenadas para la región o el estado. Aquí es importante mencionar que se recomienda utilizar el argumento state para la búsqueda de regiones, por alguna razón, esto función mejor en la API de OSM. Como ejemplo, la búsqueda coords_from_city(state = "Castilla La Mancha", country_code = "ES") nos arroja los resultados esperados, a pesar de que España no tiene estados; sin embargo si hacemos coords_from_city(region = "Castilla La Mancha", country_code = "ES") nominatim no encuentra los resultados.

ConclusionesEstos cambios han resultado muy importantes para agilizar el proceso de la búsqueda de coordenadas y la automatización de la creación de mapas. Por otro lado, me permitió darle mas estilo al código y mejorar su eficiencia. Ya que mi principal proyecto por ahora es convertirlo en una aplicación Shiny, era importante para mi el mejorar el código y la eficiencia antes de lidiar con los detalles del server. Ya que este es un trabajo reciente que he realizado en los últimos meses, decidí compartirlo de inmediato ahora que tengo fresca la información de los cambios. Espero que pueda a ayudar a mas de uno a hacer código mas abstracto y practicar recursión.

To leave a comment for the author, please follow the link and comment on their blog: R with White Dwarf.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Webscrap e iteraciones con R

View Details

[This article was first published on R with White Dwarf, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.About this postWe are creating maps of data showing changes over a span of time for different countries and pointing at all kinds of cities. That basically means that we need to map any region of the world with R. Today there are all kinds of packages and techniques to do that. I will share the strategy I used with ggplot2 and maps packages, using support of Open Street Map to obtain the coordinates of cities and finally making it interactive with shiny.

You can find all the posts on this series under the tag maps-app (including the Spanish versions).

You can also find the current state of the project under my GitHub repo mapic.

This post is originally written in Spanish, from the Amsterdam airport, on the way to Mexico. I hope you enjoy. Feel free to leave any type of comment and/or question at the end.

MotivationAs I mentioned in the previous posts in the series, I’ve been working lately on the code for creating the maps and I’ve made changes that increase the efficiency of the functions, the readability of the code, and make it easier to use. At the same it allows me to extend the functions beyond their original design.

I have mentioned on a few occasions that the code evolved slowly from scripts created to generate the specific map of some country. So, the first few functions are more of a collection of the steps used to generate the map, wrapped in the form of functions to automate the process.

For this reason, I wanted to make changes to adapt the functions to paradigms more suitable for functional programming, which is R’s strong point. However, back then the priority was to generate the maps, and thus, most of my time was devoted to creating the maps and the debugging of the code when it was necessary. I need to add that this project is part of a voluntary work for an NGO, of which I became the director of the research division, which generated even more responsibilities and work for me. And all as a side job, separated from my main source of income (which is also based on R).

However, for better or worse, 2022 was a year full of changes and challenges for me and my family, which forced me to put the project aside for a while, resign my position as responsible of the division, and focus solely on to my career, my health and my family. The result was that when I managed to regaining stability in my life, I found myself with more free time and fewer obligations to rethink the code and work on it. Additionally, my main job had a turn going from statistics to more programming oriented in R, which has given me more tools and experience to improve the code, and has motivated me to take up old lessons about functional programming and, above all, iteration.

This allowed me to improve the two main functions: the one in charge of the webscrapping and the one that sends the data to SQLite. You can find the original functions in the previous post, Map any region in the world with R – Part I: The basic map and compare it with the new, improved functions in this.

Webscrapp to SQLiteThe webscrap_to_sqlite function is responsible for sending the coordinates found by Open Street Map to our database. The original function is inefficient, as it does each operation line by line. It is also very rigid in the way it directs the values of the regions, both its request to the API and the placement of the values in the database, which makes any extension or modification very complicated.

For these reasons, it is the function that received the most changes, it was practically rewritten from scratch, making the search more efficient, also allowing internal search of the data already stored; more flexible, dealing with region parameters more clearly; and more understandable, improving the style of the code.

webscrap\_to\_sqlite <- function(db.name, dat, city = "City", country = "Country", region = NULL, state = NULL, county = NULL, db\_backup\_after = 10) { ## Loading libraries require(RSQLite) require(dplyr) ## 1. DB connection con <- dbConnect(drv = SQLite(), dbname = db.name) dbExecute(conn = con, "CREATE TABLE IF NOT EXISTS orgs (ID INTEGER UNIQUE, City TEXT, Country TEXT, Region TEXT, State TEXT, County TEXT, osm\_name TEXT, lon REAL, lat REAL)") db <- as\_tibble(dbReadTable(con, "orgs")) ## 2. Data filtering new\_coords <- data.frame() dat\_local <- compare\_db\_data(db.name, dat) df\_len <- nrow(dat\_local) ## 3. While there are rows in DF: if (df\_len != 0) { ## 3.1 Define subsample size dat\_local <- dat\_local[c(1:db\_backup\_after), ] dat\_local <- filter(dat\_local, rowSums(is.na(dat\_local)) != ncol(dat\_local)) ## 3.2 for loop for the webscrapping for (i in 1:nrow(dat\_local)) { print(paste0("Searching entry ", dat\_local[["ID"]][i])) ## 3.3 Info abstraction rg <- ifelse(is.null(region), "", dat\_local[[region]][i]) st <- ifelse(is.null(state), "", dat\_local[[state]][i]) ct <- ifelse(is.null(county), "", dat\_local[[county]][i]) rcity <- dat\_local[[city]][i] rcountry <- dat\_local[[country]][i] ## 3.4 Getting the coords ## 3.4.1. First, check if they are already in the DB search\_query <- filter(db, City == rcity, Country == rcountry, Region == rg, State == st, County == ct) if (nrow(search\_query != 0)) { coords <- search\_query[1, ] coords$ID <- dat\_local[["ID"]][i] print("Found from memory") ## 3.4.2 If they are not, search with OSM API } else { coords <- coords\_from\_city(rcity, rcountry, Region = rg, State = st, County = ct) coords <- cbind(ID = dat\_local[["ID"]][i], City = rcity, Country = rcountry, Region = rg, State = st, County = ct, coords) } new\_coords <- rbind(new\_coords, coords) } ## Send only new results to DB dbWriteTable(con, "orgs", new\_coords, append = TRUE) dbDisconnect(con) ## 3.4.3 Repeat webscrap\_to\_sqlite(db.name = db.name, dat = dat, city = city, country = country, region = region, state = state, county = county, db\_backup\_after = db\_backup\_after) ## 4. Exit iteration } else { db\_final <- import\_db\_as\_df(db.name) size <- nrow(db\_final) not\_found <- nrow(db\_final[is.na(db\_final$lat), ]) message(paste("Search finished.\n", size, "entries searched.\n", not\_found, "ENTRIES NOT FOUND")) }} The function starts by calling the necessary libraries and (1) connecting to the database, creating it if necessary. (2) Then it generates two data frames, an empty one that will store new coordinates and a relative one that contains only the data that does not yet exist in the database. (3) So, as long as there is data in this last data frame, the function will continue to loop.

(3.1) We then define the subsample, which is a subset of dat_local the size of db_backup_after and focus solely on this subsample. (3.2) On this subsample we make the iterations using for to obtain the coordinates. First (3.3) we prepare the data as strings and then (3.4) we look up the coordinates. (3.4.1) If they already exist in the database we take it from there, and if not (3.4.2) they are searched using coords_from_city. Finally, (3.4.3) we iterate all over again, allowing the function to call itself.

Since step 2 filters the data that is not yet in the database and step 3 places the results of new searches in an empty data frame, the function calls itself and applies only for each subset of data. When compare_db_data finally returns 0 values because all the data that was fed into the function is already contained in the database, we can exit the function. In this case I decided to import the data again from SQLite to get details of the search, and end the iteration by sending a message to the user about the total number of entries and the number of which were not found.

If we compare this function with the one proposed in my previous post, the function is completely different but the end result is the same. The arguments used by the function are also the same and take the same values, which avoids conflicts for the user. The only new parameter is db_backup_after which allows us to control how many rows the iteration is done. A smaller value means more iterations, which results in higher local memory usage, but also faster in finding data that already exists in the DB. On the other hand, a higher value reduces the number of iterations but increases the number of API connections. For this reason I have given it a default value of 10. This, in addition to being a balanced value, also reduces confusion for the user who might not be familiar with the changes.

Remove missing values from the databaseIn the previous proposal, only found coordinates were sent to the database, and those not found were ignored. In the present proposal, all entries are sent to the DB. Therefore, it is important to have some option to remove the missing entries.

For this I generated the function remove_na_from_db, a very simple function which gives the user the possibility of removing NAs automatically.

remove\_na\_from\_db <- function(db.file) { require(RSQLite) con <- dbConnect(drv = RSQLite::SQLite(), dbname = db.file) dbExecute(conn = con, "DELETE FROM orgs WHERE lon IS NULL OR trim(lon) = '';") dbDisconnect(con)} The function is just a connection to the database that issues the command to remove rows where the lon field is empty, in SQLite syntax. This is the safest, most direct and fastest way to do it. We could also import the data back into R, filter it, and send it back to SQLite, but this would require more local memory usage, more code, and more risk as it would require rewriting the database to SQLite entirely. . The power of the RSQLite library (or any other library that connects R to SQL) lies precisely in the ability to pass commands written and executed directly in SQL.

Obtaining the coordinatesThe coords_from_city function also received significant changes in code readability and flexibility, and a bit less in functionality and efficiency.

coords\_from\_city <- function(city = NULL, country\_code, region = NULL, state = NULL, county = NULL) { require("RJSONIO") ## 1. Abstract regions for OSM CityCoded <- gsub(" ", "%20", City) CountryCoded <- paste("&countrycodes=", CountryTwoLetter, sep = "") extras <- c(city = City, state = State, region = Region, county = County) extrasCoded <- "" if (!is.null(extras)) { for (i in 1:length(extras)) { if (extras[i] != "" && !is.na(extras[i]) && !grepl("^\\s*$", extras[i])) { valCoded <- gsub(" ", "%20", extras[i]) extrasCoded <- paste0(extrasCoded, "&", names(extras)[i], "=", valCoded) } } } ## 2. Response link <- paste( "http://nominatim.openstreetmap.org/search?city=" , extrasCoded , CountryCoded , "&format=json" , sep = "" ) response <- try({fromJSON(link)}, silent = TRUE) if (class(response) == "try-error") { stop(response[1]) } else if (class(response) == "response") { response\_status <- http\_status(response) if (response\_status$category != "Success") { stop(response\_status$message) } } else if (is.list(response)) { ## 3. Organize results if (length(response) == 0) { message(paste("No results found for", extrasCoded)) coords <- data.frame("lon" = NA, "lat" = NA, "osm\_name" = as.character(NA)) } else if (length(response) == 1) { message(paste("Found", response[[1]]$display\_name)) coords <- data.frame( lon = response[[1]]$lon, lat = response[[1]]$lat, osm\_name = response[[1]]$display\_name ) } else { message(paste("Several entries found for", city, country\_code)) coords <- data.frame( lon = response[[1]]$lon, lat = response[[1]]$lat, osm\_name = response[[1]]$display\_name ) } } ## 4. Exit as data frame return(coords)} The main change is in section 1, instead of passing each of the regions as its own string and formatting them one by one, I have abstracted them all into a single vector. This reduces the amount of code, memory usage, and allows us to include the city in the list, making it an optional value as well. The reason I had prepared them separately in the previous post is simply because the feature grew slowly: at first we only needed city, but then we had to use some additional fields depending on the country we were working in. To make things easier for me, I simply added each region field as needed. Now that I have time to work on the code, this was the first function I modified.

Step 2 now prints messages that help us identify the error when it comes to the connection, while also stopping the process. Whether it is a local connection error, or problems on the API side, we will get a message and the process will stop, which should avoid long waiting times when there is no connection and several locations are being searched.

Step 3 changes the organization of the results a bit, always returning a data frame with the same columns when the results were not found, but now with empty fields in such case. This helps the functions presented above to populate the database. Additionally, when many results were found, this information is printed on the screen; for now this is for information purposes only. The idea is to keep this space to make changes in the future that allow us to select the option interactively. This is something I still need to think about and plan properly because on one hand I want to use it in a Shiny app, and on the other we want to keep the ability for web scrapping to happen automatically with as little intervention as possible.

As I mentioned before, these new features also allow us to perform searches with the empty city value. This was a requested requirement in the last version, as some users started making maps by region, while others, not finding very small cities, decided to group the data by region. Thanks to the changes made to coords_from_city, the webscrap_to_sqlite function can now return results when the value for city is NA, assuming that the coordinates for the region or state are found. Here it is important to mention that it is recommended to use the state argument for region search, for some reason this works better in the OSM API. As an example, the search coords_from_city(state = "Castilla La Mancha", country_code = "ES") returns the expected results, despite of the fact that Spain has no states; however if we do coords_from_city(region = "Castilla La Mancha", country_code = "ES") nominatim does not find the results.

ConclusionsThese changes have been very important in speeding up the coordinate search process and automating map creation. On the other hand, it allowed me to style the code more and improve its efficiency. Since my main project for now is turning it into a Shiny app, it was important for me to improve the code and the efficiency before dealing with the details of the server. Since this is recent work that I have been doing in the last few months, I decided to share it right away now that I have fresh information on the changes. I hope it can help more than one to make more abstract code and practice recursion.

To leave a comment for the author, please follow the link and comment on their blog: R with White Dwarf.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Webscrap and iteration in R

View Details

[This article was first published on The Jumping Rivers Blog, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

What is alt text?Alt text (short for alternative text) is text that describes the appearance and purpose of an image. Alt text has multiple purposes, the main one being that it aids visually impaired users to better understand your content when the alt text is read aloud by screen readers. Alt text is also used in place of an image if it fails to load, which means that users with poor internet connection are more likely to be able to engage with your content.

How do I write alt text?There are already a lot of good resources on how to write alt text, and so that’s not the main focus of this blog post. This Medium article by Amy Cesal describes a simple formula for helping you to write alt text for charts, which I’ve found really helpful. Liz Hare also recently gave a talk on alt text for R-Ladies New York, and the slides are an excellent resource.

Can I automate writing alt text?One of the often-cited arguments for using programming languages, such as R, is that they allow you to automate processes. And so you may very well be wondering “can I use R to automate the writing of alt text?” Before I answer that question, let me remind you of the phrase just because you can, doesn’t mean you should.

The examples of automated alt text to describe plots that I’ve seen tend to describe which variables are on the x and y axes, the range of the data, the chart title, and maybe the colours in the plot. Some make attempts to describe a trend line. What’s almost always missing is the “why”. It’s very difficult to automate a description of what you’re trying to communicate to the person interpreting a plot with only a list of plot components.

One R package that may be useful as a starting point for writing alt text is the {BrailleR} package. It has support for generating alt text for both base R and {ggplot2} graphics, using the VI() function. Since it’s still missing the “what am I supposed to be seeing” message, and it doesn’t always get it right, I’d encourage you never to rely 100% on automated alt text. It could provide a starting point for you to check, edit, and include the take-home message of your graphics.

After you’ve written the alt text for your image, you need to actually add it to your document or app. If you’re directly writing HTML code, it’s usually quite straightforward – and this guide for improving accessibility with alt text gives a great overview. Today, this blog post will show you how to include alt text in your web applications and documents when you’ve built them in R.

Do you require help building a Shiny app? Would you like someone to take over the maintenance burden?If so, check outour Shiny and Dash services.

How do I add alt text in R?With R, you can create static plots, documents, presentations, web applications, and many other output types – and they all need alt text! We’ll go through how you do that for the most common output types in R.

{ggplot2}It goes without saying that {ggplot2} is one of the most popular packages for creating plots in R. So it’s likely that you’ll be adding alt text to a plot created with {ggplot2}. Within the labs() function in {ggplot2}, there’s an argument alt (introduced in version 3.3.4) – and this is where you can add alt text.

g <- ggplot(lemurs, aes(x = name, y = n)) + geom\_col() + labs(x = "", y = "Number of lemurs", title = "Lemurs at Duke Lemur Center", alt = "A bar chart titled Lemurs at Duke Lemur Center. On the x-axis three species of lemurs are shown including the Crowned lemur, Gray mouse lemur, and Ring-tailed lemur. On the y-axis the count of the number of each species is shown. The number of lemurs ranges from just under 2500 for Crowned lemurs, to almost 12500 for Gray mouse lemurs. The number of Crowned lemurs is significantly lower than the other two species shown.") If you save the plot as a variable, you can extract the alt text with:

get\_alt\_text(g) There are a couple of reasons for using the alt argument in {ggplot2}:

It’s usually easier to write the alt text when you’re making the plot, rather than when you’re compiling the outputs since it’s fresher in your mind.
The string passed to alt automatically gets passed as the image’s alt text if you use the plot in a Shiny app (more on that later…)

Quarto and R MarkdownBoth R Markdown and Quarto (next generation R Markdown) allow you to create outputs in HTML format, such as documents or presentations. Although MS Word, and Adobe are starting to allow you to add alt text to word documents and PDFs, neither R Markdown or Quarto have support for this yet (although hopefully they will in the future). HTML outputs are more accessible in general, so I’d recommend HTML outputs where possible anyway.

If you’re creating a plot within a code chunk, you can use the fig.alt option in R Markdown to pass in a character string of alt text:

{r, fig.alt="A bar chart titled Lemurs at Duke Lemur Center. On the x-axis three species of lemurs are shown including the Crowned lemur, Gray mouse lemur, and Ring-tailed lemur. On the y-axis the count of the number of each species is shown. The number of lemurs ranges from just under 2500 for Crowned lemurs, to almost 12500 for Gray mouse lemurs. The number of Crowned lemurs is significantly lower than the other two species shown."}g and in Quarto, the idea is similar but the syntax is slightly different:

{r}#| fig.alt: "A bar chart titled Lemurs at Duke Lemur Center. On the x-axis three species of lemurs are shown including the Crowned lemur, Gray mouse lemur, and Ring-tailed lemur. On the y-axis the count of the number of each species is shown. The number of lemurs ranges from just under 2500 for Crowned lemurs, to almost 12500 for Gray mouse lemurs. The number of Crowned lemurs is significantly lower than the other two species shown."g Adding alt text directly into the code chunk options has always felt slightly clunky, and so what you can do instead is store your alt text in a variable and reference it in the code block option. Alternatively, you can make good use of the get_alt_text() function in {ggplot2} in your R Markdown chunk options:

{r, fig.alt=ggplot2::get_alt_text(g)}g In Quarto, you need to be a bit more explicit about the fact you’re calling a function, but it’s still pretty straightforward:

{r}#| fig-alt: !expr ggplot2::get_alt_text(g)g If you inspect the HTML of your R Markdown / Quarto output (by right-clicking and selecting Inspect, or using the Ctrl+Shift+I keyboard shortcut), you can see the alt text has been added to the image:

If you’re creating an output format that doesn’t allow you to add alt text, such as PDF, you should still add a description of the image somewhere. You could pass in ggplot2::get_alt_text(g) to the fig.cap chunk option as an alternative.

If you’re adding an image outside of a code chunk, you can add alt text to images in Quarto using:

![](lemur.png){fig-alt="A drawing of a lemur."} and replacing fig-alt with alt works for R Markdown.

ShinyFinally, on to adding alt text in Shiny apps! Since Shiny makes it easy to build web applications straight from R, it’s important that you know how to add alt text to Shiny apps from R. If you haven’t made your plots with {ggplot2}, haven’t added your alt text to the alt argument in labs(), or need your alt text to update, read on!

Most plots in Shiny apps are generated within a renderPlot() call, with the first argument being the code that generates the plot. renderPlot() also has an alt argument (added in version 1.5.1) where you can pass in a character string of alt text for your plot:

renderPlot({ # code to generate plot goes here }, alt = "alt text goes here") However, most plots in Shiny apps have some sort of reactivity associated with them - when a user changes an input value, the plot updates. This means that the alt text should update as well. Luckily, you can pass in a reactive() to the alt argument in renderPlot():

renderPlot({ # code to generate plot goes here }, alt = reactive({ # code to add alt text goes here })) This means you can pass in different strings of alt text depending on which input values a user has selected. Depending on what the plot contains and what the user inputs do, you could construct the alt text based on the inputs. Even better, create a look-up table that returns human-written alt text based on a combination of input variables.

If you want to read more about accessibility in Shiny then check out our previous blog post on the topic.

I hope this blog post has convinced you that writing alt text is worthwhile, and not too tricky to add into your R developed documents and apps!

For updates and revisions to this article, see the original post

To leave a comment for the author, please follow the link and comment on their blog: The Jumping Rivers Blog.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Alt Text in R: Plots, Reports, and Shiny

View Details

[This article was first published on R on R (for ecology), and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.As you’re learning R, it can be hard to come up with data sets that you can practice with. Though many of us have our own data, those might not always be in the best format to do what we want. Our own data are often messy and require a lot of recoding and reformatting. Wouldn’t it be nice if we could download clean data sets that we could work with? Luckily, there are a number of resources out there – you just have to know where to look!

In this tutorial, I discuss the following data sets:

Data sets that come with R
The Knowledge Network for Biocomplexity
The Environmental Data Initiative
The National Ecological Observatory Network
The Global Biodiversity Information Facility

I also mention the Ocean Biodiversity Information System, DataOne, and the Central Michigan University Library website’s list of resources.

1) Basic data sets in ROne of the first places you can look for practice data sets is within R itself.

R comes with some standard data sets that you can view if you type data() into the console. These data sets range from describing the survival of Titanic passengers to describing the locations of earthquakes off the island of Fiji. They are wide-ranging and fun to explore, but most of them are not explicitly ecological.

Some common ecological data sets that you might use are iris, PlantGrowth, and Loblolly. I find these data sets useful when I’m trying to do something quick, like testing how a new function works. Since these data sets are so straightforward, I can usually predict what my expected output should be, and then I can know whether or not the function worked correctly. I also use these data sets as examples for the blog posts that I write – these data sets are great teaching tools because they’re fairly simple and easy to understand.

These data sets are not really intended to be used to conduct your own research; they are primarily used for practice and demonstration purposes.

2) The Knowledge Network for BiocomplexityIntroduction and how-toThe Knowledge Network for Biocomplexity (KNB) is an international repository of ecological data sets that have been uploaded by scientists to facilitate environmental research. These data are also often affiliated with published papers.

You can search data sets in a variety of ways. On the left side, you can filter the data based on different attributes (e.g., author, year, taxon, geographic location). On the right side, you can look for data sets by location by navigating the handy world map and clicking on the different squares.

When you click on a data set, you’re taken to a page where you can download all the associated files. The heading at the top is also the citation for the data package, so it’s easy to correctly attribute the work. If you’re using a public data set and publishing something (even if just in a blog post or an example), it’s a good idea to cite the data set.

Published data sets are often identified by their DOI, or “digital object identifier”. This is just a unique ID assigned to each published entity. If you type in the DOI string after “https://doi.org/” (e.g., https://doi.org/10.5063/F1FN14M4 ), you’ll get a URL that takes you to the publication.This page also includes the metadata for the data set to make it easier to navigate and understand the data you’re downloading.

All good data sets come with metadata, or data that describes the data set of interest. When you download a data set that was collected by someone else, it’s usually hard to tell what each column means, how it was collected, and what its units are. Luckily, metadata helps us figure out how a data set is organized and how we might want to use it. If a data set doesn’t come with metadata, then it’s very difficult to use and understand the data, rendering it almost useless.For example, this data set by Haas-Desmarais et al. (2021) comes with great metadata for each file that’s included in the data package. The “observations_complete.csv” file contains several variables, listed on the side. The authors have defined each variable for us – now we know that the variable “actual_time” represents the time listed on the camera and does not reflect the actual time in the world. The metadata also tells us the format / unit of the measurement.

Takeaways and applicationOne of the great things about KNB data sets is that there’s often a published journal article associated with them (usually linked in the metadata). This allows you to put the data set in the context of the research, and can give you an idea of how you might be able to manipulate the data as you’re practicing your R skills. Maybe reading the article will even raise some questions for you that you might want to explore.

Sometimes the data sets also come with associated R scripts or R Markdown documents that contain the analysis for the paper. This provides a great learning tool where you can see how other scientists conducted their analyses and try to reproduce them.

You can also download data from the KNB through R, using the package rdataone. However, I usually like to download data directly from the site so I can first familiarize myself with the data set.

3) The Environmental Data InitiativeIntroduction and how-toOne of my favorite places to download ecological data is the Environmental Data Initiative (EDI) data portal. The EDI archives a lot of environmental data that come from publicly-funded research. The EDI’s specialty is that it is the primary location where data from Long-Term Ecological Research (LTER) sites in the United States are archived. This means that the EDI will often have several years’ worth of data for a given data set, making this a great resource for examining long-term trends. For example, the EDI hosts data for a project called “EcoTrends”, which is a large synthesis effort that aggregates ecological data on a yearly or monthly time-scale. The aim of the project is to make long-term ecological data easier to access, analyze, and compare among research sites to evaluate global change. All the EcoTrends data are organized into a common and clean data format (maybe providing good practice for making plots in R?).

As with the KNB, you can browse data in the EDI portal in a number of ways – you can search by LTER site, or based on keywords that the data creators associated with their data set. Some especially useful methods might be to look for data by discipline, by ecosystem, or by organism.

You can also browse data sets by their package identifier, which groups data sets by LTER site or by a specific project (e.g., EcoTrends or the PaleoEcological Observatory Network). Examples of package identifier names include “edi”, “ecotrends”, or “knb-lter-arc”. These codes, in combination with strings of numbers, are used within the EDI to uniquely identify each data set.

The EDI also has an advanced search tool, where you can specify several attributes like geographic location, temporal scale, research site, authors, taxon, etc.

Once you’ve decided on a data set, you’ll be taken to a page that summarizes the data package you’re looking at. This page will provide some basic information like authors, publication date, citation, abstract, and spatial coverage. There will also be a link to download the data, and a link to view the full metadata. Like with the KNB, some data sets come with R scripts that you can run and learn from.

Takeaways and applicationSomething really neat that the EDI provides on each data package page is a code generator that will read in the data for you and format it appropriately. The EDI will generate code for several different coding languages, like Matlab, Python, R, and SAS. We are of course interested in the “R” and “tidyr” options.

The code under the “R” option will read in the data as a data frame, while the code under the “tidyr” option will read in the data as a tibble, using the tidyverse package (check out our post here [LINK] for a rundown on the differences between data frames and tibbles). You can either download an .R file with the code already written, or you can copy and paste the code into your own file.

Again, EDI data boasts numerous data sets with long-term measurements (some on the scale of decades!), making it really useful for examining long-term trends.

Quick note from Luka: But what do you do with your data once you have it? If you are still a beginner with R, then I encourage you to check out my full course on The Basics of R (for ecologists). I designed the course to take away the stress of learning R by leading you through a self-paced curriculum that makes R easy and painless. I’m confident this course will give you all the essentials you need to feel comfortable working with your own data in just a few weeks. Just click below to start the course and see what you think!

Or, if you already feel solid with the basics, take your data visualization to the next level with my Introduction to Data Visualization with R (for ecologists) where I teach you everything you need to create professional and publication-quality figures in R.

4) National Ecological Observatory NetworkIntroduction and how-toThe next resource I’m going to discuss is the National Ecological Observatory Network (NEON), which is a network of field sites across the United States at which several types of ecological data are regularly collected in terrestrial and aquatic environments.

The network is designed so that the U.S. is divided into 20 ecological/climatic domains. Almost every domain has terrestrial and aquatic field sites, which are often placed in close proximity to one another to allow for analysis of linkages across these ecosystems. NEON collects remotely-sensed data, observational data, and data via automatic sensors (e.g., meteorological towers), with the idea that these data will be collected over many, many years. These data are also standardized across NEON sites. As a result, NEON data covers a broad spatial and temporal extent, allowing us to collect and compare certain measurements across the entire U.S. and over long periods of time.

When you’re looking for NEON data, you can search for data in one of two ways.

The first way is to look for data by site or location through the interactive map on NEON’s homepage. This is more of an exploratory approach, where you can zoom in on different parts of the map. The table beneath the map shows you what field sites and plots are visible. If you want to look at a site’s data, you can just click “Explore Data” under the site name, and you’ll be taken to NEON’s data archive page.

If you zoom in on a specific research site (I zoomed in on the Smithsonian Environmental Research Center), the map will show you specific plots and locations of towers.

If you’re curious about a specific research site, you can also navigate to the site’s information page, which gives a lot of great background about the history of the site, some native fauna and flora, the geology, climate, etc. The image below shows part of the Toolik Field Station NEON page. The right-hand side of the page shows a lot of basic information about the site, like the coordinates, elevation, mean annual temperature, etc. Note that many NEON sites are also LTER sites (e.g., Toolik, Konza Prairie, Jornada).

The other way to search for data is to simply go to NEON’s “Explore Data Products” page. You can filter your data search by date, research site, state, domain, and research theme (e.g., atmosphere, biogeochemistry, land cover, organisms/populations/communities). The data sets are grouped by measurement and not by research site. So, for example, you can download a wind speed data set that includes wind speeds from all the research sites that collect that data.

When you decide on a data set that you want to look at, you can click on the data set name. This will take you to the page for the specific data set, which has loads of information.

The first part of the page shows information on the data set, including a description of the data, an abstract / reasoning for the data collection, and a citation for when you use the data.

If you scroll down, you can see information about how the data was collected and processed. NEON provides a brief description about the sampling scheme and instrumentation, as well as detailed documentation about the methods and QA/QC process. They also provide an issue log to address problems that arose during data collection or processing, and they let you know at what sites those issues occurred.

The next section shows the spatial and temporal availability of the data. In the table below, each row represents a research site and each column represents a month. The cells are colored in if there is data available at the research site during that month. The cells are grey if there is no data available. You can click the blue “Download Data” button to begin the data downloading process.

When you’re ready to begin downloading data, you can choose what research sites and time periods you want to download data for. Note the estimated file size in the top right corner, as some data sets are very large and can take a while to download. The page provides instructions for how to select sites and your date range. After you make your selection, you will be able to choose whether or not you want to download any associated documentation (i.e., sampling scheme and protocol documents listed in the “Collection and Processing” section). You can then choose whether you want a basic data package or expanded package, which includes QA/QC metrics. After you agree to NEON Usage and Citation policies, you can then download your data set!

When you unzip the data download, you’ll see a bunch of folders. Each folder represents a site-month combination. Within each folder, there are several .csv files. I recommend that you read the .txt file that comes with it, as it describes what each .csv file contains and helps you put together the pieces to understand the data.

NEON also comes with a helpful visualization tool on the data set information page. The tool will graph the data for you, so you can get an idea of what it looks like before you download it. You can manipulate pretty much any aspect of the graph. You can add sites to the plot to see how they compare to one another, and you can choose what specific sensor’s data you want to display (each site usually has multiple sensors at different locations). You can also adjust the date range that is displayed and the specific variable that is plotted (e.g., minimum, maximum, or mean values). The scroll bar below the X axis allows you to zoom in/focus on a specific time range. The axes ranges, scales, and breaks can also be adjusted. Lastly, you can download the plot as a PNG.

I encourage you to play around with this – it’s such a neat tool! Unfortunately, the visualization tool isn’t available for every data set, but it’s often available for measurements that are taken by automatic sensors or towers (e.g., air temperature, wind speed, barometric pressure).

Takeaways and applicationNEON has its own R package, called neonUtilities. The package provides functions to help you work with and import NEON data. Something great that NEON provides are R tutorials for working with NEON data and for general ecological analysis. For example, here’s a tutorial on how to download and explore NEON data. And here’s a guided practice lesson where you can learn how to search for and visualize precipitation data. Here are NEON’s recommendations for people who are just getting started with NEON data and/or R.

In short, NEON data are useful for illuminating spatiotemporal trends. NEON is great for comparing several types of data (phenological, biogeochemical, climatological, etc.) across different terrestrial and aquatic environments in the United States. There are also several sites within each ecoclimatic Domain, so you can examine trends across ecological gradients (e.g., elevation).

5) Species and biodiversity dataThe Global Biodiversity Information FacilityIntroduction and how-toCollecting species occurrence and biodiversity data can be really useful for modeling species distributions and understanding how they might change (e.g., studying impacts of climate change or predicting the spread of invasive species).

The Global Biodiversity Information Facility (GBIF) is an international data repository that is commonly used to obtain species occurrence data. Let’s check it out.

The main ways to search for data are to search for occurrences, to search for species, or to browse data sets.

When you search for data by occurrences, the easiest method is probably to search for your species of interest. When you type in your species name in the search bar, a drop down menu will appear that shows you the different names or subspecies that your species of interest might be known by. If you want to download all occurrences for your species, then you should include all possible names in your search. In the image below, I searched for Callinectes sapidus, commonly known as the Atlantic blue crab.

Once you complete your search, you can view occurrences in a table, as a map, or through a photo gallery (usually photos from iNaturalist, an app used for sharing biodiversity/wildlife observations).

There’s also a tab that you can click on to download occurrence data, which will look something like this once it’s downloaded. Each row of data is one observation of the species, and there are columns that will give you information on taxonomy, the country where the species was observed, the coordinates, and the date, among other data.

The species search is slightly different from the occurrence search. As one might think, the species search focuses more on information about the species itself than individual records of occurrence data. The page has a pane on the left that describes the species taxonomy. The pane on the right shows an overview of the species, including the photo gallery, a map of its distribution, its common names, and places where the species is classified as “introduced” rather than native. This is helpful for broadly learning about your species of interest before you dive into the data.

Lastly, you can browse GBIF-associated data sets, which are not organized by species but by network / event / project.

For example, if I click on the “iNaturalist Research-grade Observations” data set, I’m taken to a page where I can download the whole iNaturalist database of species observations, see the geographic distribution of occurrences, and see the taxonomic breakdown of species listed in the data set.

Takeaways and applicationGBIF also has a “Resources” section that can provide inspiration for projects and show you several helpful tools. For example, the “Data Use” tab lists different publications and projects that use GBIF data, showing you how GBIF data can be used to drive research.

You can also explore biodiversity and species distribution-related tools in the “Tools” tab and search for GBIF-related literature in the “Literature” tab. GBIF also has a data blog, where they discuss tips and tricks for how to use GBIF. Very useful!

One last note about GBIF is that it has its own R package, called rgbif. rgbif makes it really easy to read GBIF data into R. For more on this, check out this blog post from R-bloggers, which provides a commented script that walks you through how to import, clean, and map the data. GBIF is pretty commonly used, so there are several tutorials out there on how to use the data.

The Ocean Biodiversity Information SystemThere’s also the Ocean Biodiversity Information System (OBIS), which is like GBIF but for marine species (OBIS actually contributes marine data to GBIF). I’m not going to dive too deep into this resource, but OBIS also comes with its own R package, called robis. Something nice is that OBIS provides a few examples of analyses that can be done using OBIS data and using the robis package. The image below is an example of an R notebook that OBIS created to showcase its data – this can be a great learning tool to follow along with!

OBIS also has a great visualization tool, called “mapper”, that allows you to map species distributions on top of one another. Mapper is also the primary way you can search for species records in OBIS. In the image below, I mapped Callinectes sapidus (blue crab) distributions on top of Zostera marina (eelgrass) distributions. The green drop down menu beside each species occurrence layer also allows you to view or download occurrence data for that species and modify its appearance on the map.

Looking for more?The DataOne portal is a huge archive of environmental data that aggregates data sets from several different repositories and organizations, including many of the resources we listed above (e.g., KNB, EDI, NEON). This is a good portal to look to if you want a very comprehensive search, or if you don’t know exactly what you’re looking for. The other repositories might be more helpful if you already know exactly what kind of data you want to retrieve.

I also want to highlight the Central Michigan University Library website, which has a great list of resources that you can consult to find data relating to the life sciences (including ecological data!). The website lists a few of the sources we described above, and more. It also provides some good sources of environmental data (e.g., habitat/spatial data and climate data), which could be helpful for modeling. I would definitely check it out, especially if you’re searching for public data to use for your own research.

If you’re just looking for practice data, the resources we listed above should provide plenty of data sets for you to use! I recommend that you explore all the different data repositories that I recommended – they’re rich with tools and exciting data beyond what I covered in this blog post.

Do you have any favorite sources of ecological data? Let us know in the comments below! We made a top 5 list so we could dive deep into the details of each one, but it never hurts to learn about more resources.

I hope this tutorial was helpful. As always, happy coding!

Quick note from Luka: If you are just starting with R, then I encourage you to check out my full course on The Basics of R (for ecologists). I designed the course to take away the stress of learning R by leading you through a self-paced curriculum that makes R easy and painless. I’m confident this course will give you all the essentials you need to feel comfortable working with your own data in just a few weeks. Just click below to start the course and see what you think!

Also be sure to check out R-bloggers for other great tutorials on learning R

CitationsStephanie Haas-Desmarais, Gabriel Benjamen, and Christopher Lortie. 2021. The effect of shrubs and exclosures on animal abundance, Carrizo National Monument. Knowledge Network for Biocomplexity. doi:10.5063/F1FN14M4.

To leave a comment for the author, please follow the link and comment on their blog: R on R (for ecology).

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Top five(ish) sources of ecological data

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.How many times has your code unexpectedly stopped working? Even better, how many times did the code work on your machine, but your coworkers couldn’t run it? Both questions share the same underlying problem – R environment.

That’s where R renv comes in – a dependency management package that will ensure issues like these won’t happen ever again. Today you’ll learn all about renv through a hands-on example. You’ll also see how to work with Plotly and Git. Let’s get started!

Need to make interactive Markdown documents? Try R Quarto – you’ll never look back.

Table of contents:

What is R renv and Why Should You Care?
2 Ways to Use renv in Your R Project
How to Take Snapshots of your R Environment
How to Restore your R Environment with a Single Command
Summing up R renv

What is R renv and Why Should You Care?Renv stands for Reproducible Environment and does just what the name suggests. Developers often have trouble managing the R environment and dependencies due to reasons of how R works by default. It installs packages to a central library and shares them between the projects.

It sounds like a good and time-saving feature. After all, you don’t need to install the same package in every project. But that’s where the problems arise. You might have a newer version of some package than your coworkers – resulting in a deprecated or not-implemented functionality.

End result? The app crashes or the code won’t run altogether, and no one knows why because the code didn’t change.

The answer is always the same – environment differences and package version mismatch. R renv is here to create a separate, reproducible environment that you and your coworkers can use, hassle-free.

To be more precise, renv will do the following:

Create a project-specific library (no dealing with the central R library).
Create a list of dependencies and their version, making it easy to share the project.

You now know why you should use renv, but how do you actually go about it? Let’s answer this question next.

2 Ways to Use renv in Your R ProjectThere are two distinct ways to leverage renv – when first creating a project and later through a command line.

When creating a new project, just tick the second checkbox, as shown in the image below:

Image 1 – Using renv when creating a new R project

In case you forgot to do it, or prefer doing things through a command line, just type the following into the R console:

renv::activate() Image 2 – Using renv through a command line

This will do all the housekeeping for you, and create several files and folders (we started with an empty directory):

Image 3 – Files and folders created by renv

Let’s discuss the purpose of each file next.

Files and folders created by R renv – What do they mean?Here’s the meaning behind every file/folder created by renv:

.Rprofile – A file run by RStudio every time you load or reload an R session. It calls the renv/activate.R file.
renv/.gitignore – Tells Git to ignore the library folder, as it contains dependencies that can be large in size. There’s no need to keep track of them, as the correct version can easily be downloaded by your coworkers.
renv/activate.R – A file used to activate a local R environment.
renv/library/* – Folder with many subfolders – contains the project dependencies.

And with that out of the way, let’s discuss an essential R renv topic – snapshots.

How to Take Snapshots of your R EnvironmentA snapshot is a term used by renv determine dependencies used in your R project and write them to a separate file – renv.lock.

You’ll see how it works in a second, but first, let’s install some R packages:

install.packages("dplyr")install.packages("plotly") Image 4 – Installing R packages

Once the packages are installed, take a snapshot by running the following line from the R console:

renv::snapshot() Image 5 – Taking an environment snapshot

Specify that you want to proceed by typing y when prompted, and you’re good to go! A new file named renv.lock is now created, containing all project dependencies and sub-dependencies with the correct version:

Image 6 – Contents of the renv.lock file

The file is almost empty (the only dependency being renv), so what gives? Well, as said earlier, renv will store only the dependencies used in your project. Since there’s no R code importing either dplyr or plotly, they weren’t added to the lock file.

Let’s change that by making an R file that will render a chart. We’ve copied some code from the Plotly documentation, and encourage you to do the same:

library(plotly)x <- c("Product A", "Product B", "Product C")y <- c(20, 14, 23)y2 <- c(16, 12, 27)text <- c("27% market share", "24% market share", "19% market share")data <- data.frame(x, y, y2, text)fig <- data %>% plot\_ly()fig <- fig %>% add\_trace( x = ~x, y = ~y, type = "bar", text = y, textposition = "auto", marker = list( color = "rgb(158, 202, 225)", line = list(color = "rgb(8, 48, 107)", width = 1.5) ))fig <- fig %>% add\_trace( x = ~x, y = ~y2, type = "bar", text = y2, textposition = "auto", marker = list( color = "rgb(58, 200, 225)", line = list(color = "rgb(8, 48, 107)", width = 1.5) ))fig <- fig %>% layout( title = "January 2013 Sales Report", barmode = "group", xaxis = list(title = ""), yaxis = list(title = ""))fig Here’s the figure displayed by Plotly:

Image 7 – A Plotly bar chart

You can now once again take the snapshot:

renv::snapshot() Image 8 – Taking an environment snapshot (2)

And take a look at the lock file:

Image 9 – Contents of the renv.lock file (2)

As you can see, there are many dependencies listed, but we’ve used only one – plotly. The reason is simple, plotly needs a handful of packages in order to work, and each of these packages has its own dependencies. You can now see how quickly dependency management can become a nightmare.

But how easy it is now for other developers to recreate this environment? To answer this question, we have to put ourselves in the shoes of other developers.

Pushing Your R Project to GitHubThis will allow us to clone the project and start fresh (just as you weren’t the author of the code), and determine if dependency management with renv really works.

Start by creating a new repository on GitHub:

Image 10 – A new GitHub repo

Initialize the R project folder as a Git project, and push it to a remote with the following set of commands:

git initgit add .git commit -m "initial commit"git remote add origin https://github.com/<you>/<project>.gitgit push -u origin main Image 11 – Pushing a local project to a remote

Assuming you did everything correctly, you’ll see the R project pushed to the Main branch:

Image 12 – GitHub repo after the initial push

Next, let’s restore this R environment to test if dependency management works as advertised.

How to Restore your R Environment with a Single CommandLet’s start by cloning the repository into a new folder – NewRenvProject:

git clone https://github.com/<you>/<project>.git NewRenvProject Image 13 – Cloning a project from GitHub

As soon as you open it as a project in RStudio, you’ll see the message from renv telling you how to restore the environment:

Image 14 – Opening a new project in RStudio

Just run the following command:

renv::restore() And you’ll be good to go:

Image 15 – Restoring an R environment with renv

The R environment is now restored and you have access to all dependencies at the correct version, just as intended by the project author.

Summing up R renvAnd that’s how easy it is to manage project package dependencies in R environments. It all boils down to three renv functions – activate(), snapshot(), and restore(). You’ve learned how each works through a practical example, and by now, we hope you can appreciate the heavy lifting renv does for you.

What’s your favorite way to manage dependencies in R? Are you still using the outdated packrat package? Please let us know in the comment section below. Also, don’t hesitate to move the discussion to Twitter – @appsilon. We’d love to hear from you.

What do R Shiny developers do? Here’s a typical day in the life of Appsilon’s Alexandros Kouretsis.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R renv: How to Manage Dependencies in R Projects Easily

View Details

[This article was first published on Tag: r - Appsilon | Enterprise R Shiny Dashboards, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Every data science project needs a data validation step. It’s a crucial part, especially when feeding data into machine learning models. You don’t want errors or unexpected behaviors in a production environment. Data validation is a way you can check the data before it touches the model and ensures it’s not corrupted. And yes, you can automate data quality reports!

Today you’ll learn how to work with datasets in R, in order to create automated R data quality reports. The best part – you’ll build a UI around the data validation logic, so you can easily upload CSV files for examination and validation, and R Shiny will handle the rest. Let’s dive straight in!

Looking to create reports with R Markdown? Read our top tips on code, images, comments, tables, and more.

Table of contents:

How to Approach R Data Quality Checks for Reporting
Introduction to data.validator Package
Data Validation and Automated Reporting in R Shiny
Summing up R Data Quality

How to Approach R Data Quality Checks for ReportingIn machine learning, you typically train and optimize a model once and then deploy it. What happens from there is more or less wild west because you can’t know how the user will use your model. For example, if you create a web interface around it and allow for data entry, you can expect some users to enter values that have nothing to do with the data your model was trained on. If a feature value typically ranges between 0 to 10, but the user enters 10000, things will go wrong.

Some of these risks can be avoided with basic form validation, but sooner or later an unexpected request will go through. That’s where data quality checks play a huge role.

So, what are your options? Several R data quality packages are available, but today we’ll focus on Appsilon’s data.validator. It’s a package we developed for scalable and reproducible data validations, and it includes a ton of functions for adding checks to the data.

Other options for automated data quality reports with R exist, such as pagedown and officeR, and you’re free to use them. We found these alternatives capable of report automation, but nowhere near as interactive and scalable as data.validator.

Let’s dive into the examples next.

Introduction to the data.validator PackageThe package is available on CRAN, which means you can install it easily through the R console:

install.packages("data.validator") Alternatively, you can install the latest development version:

remotes::install\_github("Appsilon/data.validator") We’ll work with the Iris dataset for R data quality reports, and we recommend you download the CSV version instead of using the one built into R. You’ll see reasons why in the following section.

For now, just load these couple of libraries and read the dataset:

library(assertr)library(dplyr)library(data.validator)df\_iris <- read.csv("/path/to/iris.csv")head(df\_iris) Here’s what it looks like:

Image 1 – Head of the Iris dataset

Now, how can you implement data quality checks on this dataset? Just imagine you were to build a machine learning model on it, and you want to somehow regulate the constraints of user-entered data going into the forecasting. First, you need to think of the conditions that have to be satisfied. Let’s mention a few:

Values can’t be missing (NA)
Column value has to be in a given range, let’s say between 0 and 10 for sepal.width and sepal.length

You can add these and many other conditions to a data validation report. Here’s how:

```

A custom function that returns a predicatebetween <- function(a, b) { function(x) { a <= x & x <= b }}# Initialize the reportreport <- data_validation_report() # Add validation to the report validate(data = df_iris, description = "Iris Dataset Validation Test") %>% validate_cols(predicate = in_set(c("Setosa", "Virginica", "Versicolor")), variety, description = "Correct species category") %>% validate_cols(predicate = not_na, sepal.length:variety, description = "No missing values") %>% validate_cols(predicate = between(0, 10), sepal.length, description = "Column sepal.length between 0 and 10") %>% validate_cols(predicate = between(0, 10), sepal.width, description = "Column sepal.width between 0 and 10") %>% add_results(report = report)# Print the reportprint(report)

`` Thebetween()` function is a user-defined one, allowing you to check if a value is within range. You can define your custom functions in a similar manner, or use the ones built into data.validator.

Here’s what the printed report looks like:

Image 2 – Printed data.validator report

Saving Reports Locally – HTML, CSV, and TXTYou can also save the report locally in HTML format and open it by running the following code:

save\_report(report)browseURL("validation\_report.html") Image 3 – data.validator report in HTML format

In case you prefer a simpler, flatter design without colors, change the value of the ui_constructor parameter:

save\_report(report, ui\_constructor = render\_raw\_report\_ui)browseURL("validation\_report.html") Image 4 – Raw data.validator report

You’re not limited to saving R data quality reports to HTML. After all, it’s not the most straightforward file format to automatically parse and see if any validation checks failed. For that reason, we include an option for saving reports as CSV files:

save\_results(report, "results.csv") Image 5 – R data quality report in CSV format

It’s not as visually attractive, sure, but you can easily write scripts that would read these CSV files if you want to automate data quality checks.

And finally, there’s an option to save the data as a plain text document:

save\_summary(report, "validation\_log.txt") Image 6 – data.validator report in TXT format

The report now looks like the one from the R console, which is the format you may prefer.

Want to dive deeper into data.validator? This article further explores validation functions.

You now know the basic idea behind R data quality checks and reports, so next, we’ll take this a step further by introducing R Shiny.

Data Validation and Automated Reporting in R ShinyBy now you’ve created a data validation report with data.validator, so why bring R Shiny into the mix? The answer is simple – it will allow you to create an application around the data validation logic, and will further simplify data quality checks for non-tech users. The app you’ll create in a minute allows the user to upload a CSV file, for which a validation report is displayed.

We recommend you save the following snippet in a separate CSV file. It contains a couple of Iris instances that will fail the data validation test:

"sepal.length","sepal.width","petal.length","petal.width","variety"5.1,3.5,1.4,.2,"Setosa"100,3,1.4,.2,"Setosa"4.7,3.2,47,.2,"Setosa"4.6,3.1,1.5,.2,"Sertosa"5,NA,1.4,.2,"Setosa" As you can see, either the species is wrong, the value is missing, or the value is out of range.

Our Shiny app will have a sidebar that allows for CSV file upload and the main section that renders the head of the uploaded CSV file and its validation report.

Keep in mind: R Shiny already has the validate() function, so we have to explicitly write data.validator::validate() to avoid confusion and errors:

library(shiny)library(data.validator)library(assertr)library(dplyr)# data.validator helper functionbetween <- function(a, b) { function(x) { ifelse(!is.na(x), a <= x & x <= b, FALSE) }}ui <- fluidPage( titlePanel("Appsilon's data.validator Shiny Example"), sidebarLayout( sidebarPanel( fileInput(inputId = "dataFile", label = "Choose CSV File", multiple = FALSE, accept = c(".csv")), checkboxInput(inputId = "header", label = "Data has a Header row", value = TRUE) ), mainPanel( tableOutput(outputId = "datasetHead"), uiOutput(outputId = "validation") ) ))server <- function(input, output, session) { # Store the dataset as a reactive value data <- reactive({ req(input$dataFile) tryCatch( { df <- read.csv(file = input$dataFile$datapath, header = input$header) }, error = function(e) { stop(safeError(e)) } ) }) # Render the table with the first 5 rows output$datasetHead <- renderTable({ return(head(data(), 5)) }) # Render the data validation report output$validation <- renderUI({ report <- data\_validation\_report() data.validator::validate(data(), description = "Iris Dataset Validation Test") %>% validate\_cols(in\_set(c("Setosa", "Virginica", "Versicolor")), variety, description = "Correct species category") %>% validate\_cols(predicate = not\_na, sepal.length:variety, description = "No missing values") %>% validate\_cols(predicate = between(0, 10), sepal.length, description = "Column sepal.length between 0 and 10") %>% validate\_cols(predicate = between(0, 10), sepal.length, description = "Column sepal.width between 0 and 10") %>% validate\_cols(predicate = between(0, 10), petal.length, description = "Column petal.length between 0 and 10") %>% validate\_cols(predicate = between(0, 10), petal.width, description = "Column petal.width between 0 and 10") %>% add\_results(report) render\_semantic\_report\_ui(get\_results(report = report)) })}shinyApp(ui = ui, server = server) Here’s what the app looks like:

Image 7 – R Shiny app build around data.validator

And that’s how easy it is to build a UI around the data validation pipeline. You can (and should) add more checks, especially for custom datasets with many attributes. The overall procedure will be identical, only the validation part would get longer.

Let’s make a short recap next.

Summing Up Automated R Data Quality ReportingIn data science, it’s essential to stay on top of your data. You never know what the user may enter into a form or how the data may change over time, so that’s where data validation and automated data quality checks come in handy. You should create constraints around data going into a machine learning model if you want to guarantee reasonable predictions. Quality data in – quality prediction out.

Appsilon’s data.validator package simplifies data quality checks with a ton of built-in functions. You can also declare custom ones, so there’s no hard limit on the checks you want to make. The package can also save data quality reports in HTML, CSV, and TXT format, and is fully compatible with R Shiny. What more do you need?

What are your thoughts on data.validator and automated data quality checks in general? Which tools/packages do you use daily? Let us know in the comment section below, and don’t hesitate to reach out on Twitter – @appsilon. We’d love to hear from you.

R Shiny seems more interesting by the day? Our detailed guide shows you how to make a career out of it.

The post appeared first on appsilon.com/blog/.

To leave a comment for the author, please follow the link and comment on their blog: Tag: r - Appsilon | Enterprise R Shiny Dashboards.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: R data.validator – How to Create Automated Data Quality Reports in R and Shiny

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionCumulative mean is a statistical measure that calculates the mean of a set of numbers up to a certain point in time or after a certain number of observations. It is also known as a running average or moving average.

Cumulative mean can be useful in a variety of contexts. For example:

Tracking progress: Cumulative mean can be used to track progress over time. For instance, a teacher might use it to track the average test scores of her students throughout the school year.
Analyzing trends: Cumulative mean can help identify trends in data. For example, a business might use it to track the average revenue generated by a new product over the course of several months.
Smoothing data: Cumulative mean can be used to smooth out fluctuations in data. For instance, a meteorologist might use it to calculate the average temperature over the course of a year, which would help to smooth out the effects of daily temperature fluctuations.

In summary, cumulative mean is a useful statistical measure that can help track progress, analyze trends, and smooth out fluctuations in data.

FunctionThe function we will review is cmean() from the {TidyDensity} R package. Let’s take a look at it.

cmean() The only argument is .x which is a numeric vector as this is a vectorized function. Let’s see it in use.

ExampleFirst let’s load in TidyDensity

library(TidyDensity) Ok now let’s make some data. For this we are going to use the simple rnorm() function.

x <- rnorm(100)head(x)

[1] -0.8293250 -1.2983499 2.2782337 -0.1521549 0.6859169 0.3809020 Ok, now that we have our vector, let’s run it through the function and see what it outputs and then we will graph it.

cmx <- cmean(x)head(cmx)

[1] -0.8293249774 -1.0638374319 0.0501862766 -0.0003990095 0.1368641726[6] 0.1775371452 Now let’s graph it.

plot(cmx, type = "l") Ok nice, so can we do this on grouped data or lists of data? Of course! First let’s use a for loop to generate a list of rnorm() values.

```

Initialize an empty list to store the generated valuesmy_list <- list()# Generate values using rnorm(5) in a for loop and store them in the listfor (i in 1:5) { my_list[[i]] <- rnorm(100)}# Print the generated listpurrr::map(my_list, head)

```

[[1]][1] -0.8054353 -0.4596541 -0.2362475 1.1486398 -0.7242154 0.5184610[[2]][1] 0.3243327 0.7170802 -0.5963424 -1.0307104 0.3388504 0.5717486[[3]][1] 1.7360816 -1.0359467 -0.3206138 -1.2157684 -0.8841356 0.1856481[[4]][1] -1.1401642 -0.4437817 -0.2555245 -0.1809040 -0.2131763 -0.1251750[[5]][1] 0.08835903 -1.79153379 -2.15010900 0.67344844 1.06125849 0.99848796 Now that we have our list object let’s go ahead and plot the values out after we pass the data through cmean().

library(purrr)my\_list |> map(\(x) x |> cmean() |> plot(type = "l"))

[[1]]NULL[[2]]NULL[[3]]NULL[[4]]NULL[[5]]NULL From here I think it is easy to see how one could do this on gruoped data as well with dplyr’s group_by().

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Some Examples of Cumulative Mean with {TidyDensity}

View Details

[This article was first published on schochastics, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.My posts are usually R only. But in this post, I want to talk about a suite of toolsdeveloped by my colleagues and me thatgoes beyond R only. This suite of tools helps to gather results from different search engines and includes abrowser extension to scrape the results, and a Python libraryand an R package to parse the results.

The browser extensionThe core tool is a browser extension for Mozilla and Chrome that simulates a user searching (at least) 50-top main, news, images and videos search results of up to 8 different search engines.

The repository is well documented and walks you through the steps of setting up the extension to scrape data.Always make sure to deactivate the extension once you are done since it interferes with your normal browsing.

The extension can store local snapshots of the html files of the search results for later analysis. This is done via the web extension SingleFile. To parse these html files efficiently forimportant information, we a Python library as well as an R package.

The Python libraryTo use the Python library, simply clone therepository and either add webbotparser/webbotparser.py to your working directory or navigate to the folder, and run

pip install -e . The webbotparser is then available in your Python installation.

(The following is an excerpt of the repository README)

UsageFor the search engines and result types supported out of the box, simply run

from webbotparser import WebBotParser and initialize the WebBotParser for the search engine and result type your are investigating, for example

parser = WebBotParser(engine = 'DuckDuckGo News') Then, you can obtain the search results as a pandas DataFrame and metadata as a Python dictionary with

metadata, results = parser.get\_results(file='path/to/the/result\_page.html') Furthermore, parser.get_metadata(file) can be used to only extract the metadata. parser.get_results_from_dir(dir) allows to directly extract search results spread over multiple pages, as Google text result are provided for instance. For examples also see example.ipynb.

Extracting imagesWebBot archives images inline in the html file of the search results, i.e., they are neither external files on your drive nor fetched from the original source on viewing the downloaded search results page. This allows us to extract the images directly from the html file for further analysis. The engines and result types supported out of the box with WebBotParser allow for extracting images as well. Simply initialize WebBotParser as follows:

parser = WebBotParser(engine = 'Google Video', extract\_images=True) You can optionally specify extract_images_prefix, extract_images_format, and extract_images_to_dir. See example.ipynb for more details, including preview in Jupyter Notebooks.

Custom result typesWebBotParser out of the box only provides support for some search engines and result types. Even these parsers might stop working if the search engine providers decide to change their layout. However, WebBotParser can still be used in these cases by defining a custom result_selector, queries, and optionally a metadata_extractor function. In this case, a WebBotParser is initiated with these instead of with the engine attribute

parser = WebBotParser(queries, result\_selector, metadata\_extractor) Under the hood, WebBotParser uses BeautifulSoup to

Parse the search result page’s HTML via LXML
Disciminate the individual results on each page using a CSS selector called result_selector that matches a list of search results
For each of those results, extract available information through a list of queries

See the below example for available types of queries and their usage

queries = [ # extract the text from inside a matched element, getting all the text over all its children {'name': 'abc', 'type': 'text', 'selector': 'h3'}, # extract the value of an attribute of a matched element {'name': 'def', 'type': 'attribute', 'selector': 'a', 'attribute': 'href'}, # whether or not a CSS selector matches, returns a Boolean {'name': 'ghi', 'type': 'exists', 'selector': 'ul'}, # extract inline images and name them by a title {'name': 'jkl', 'type': 'image', 'selector': 'g-img > img', 'title\_selector': 'h3'} # pass a custom query function {'name': 'mno', 'type': 'custom', 'function': my\_function},] You can optionally provide a metadata_extractor(soup, file) function to extract metadata alongside the search results, or import one of the existing extractors, e.g. with

from webbotparser import GoogleParsermetadata\_extractor = GoogleParser.google\_metadata The R packageThe R package can be installed from GitHub

remotes::install\_github("schochastics/webbotparseR")library(webbotparseR) The package contains an example html from a google search on climate change.

ex\_file <- system.file("www.google.com\_climatechange\_text\_2023-03-16\_08\_16\_11.html", package = "webbotparseR") Such search results can be parsed via the function parse_search_results(). The parameter engine is used to specify thesearch engine and the search type.

output <- parse\_search\_results(path = ex\_file,engine = "google text")output## # A tibble: 10 × 10## title link text image page posit…¹ searc…² type query date ## <chr> <chr> <chr> <chr> <chr> <int> <chr> <chr> <chr> <dttm> ## 1 What… http… Clim… data… 1 1 www.go… text clim… 2023-03-16 08:16:11## 2 Home… http… Vita… data… 1 2 www.go… text clim… 2023-03-16 08:16:11## 3 Vita… http… “Cli… data… 1 3 www.go… text clim… 2023-03-16 08:16:11## 4 Clim… http… In c… data… 1 4 www.go… text clim… 2023-03-16 08:16:11## 5 IPCC… http… The … data… 1 5 www.go… text clim… 2023-03-16 08:16:11## 6 Clim… http… Comp… data… 1 6 www.go… text clim… 2023-03-16 08:16:11## 7 Clim… http… Clim… <NA> 1 7 www.go… text clim… 2023-03-16 08:16:11## 8 UNFC… http… What… data… 1 8 www.go… text clim… 2023-03-16 08:16:11## 9 Clim… http… Clim… data… 1 9 www.go… text clim… 2023-03-16 08:16:11## 10 Caus… http… This… data… 1 10 www.go… text clim… 2023-03-16 08:16:11## # … with abbreviated variable names ¹position, ²search\_engine Note that images are always returned base64 encoded.

output$image[1]## [1] "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAIAAACQkWg2AAAABnRSTlMAAAAAAABupgeRAAAAMklEQVR4AWMAgYYG4hEdNJAHGoCIABvBJayhgcYaIAwaakCwydUA52MKYeeSCgZh4gMAXrJ9ASggqqAAAAAASUVORK5CYII=" The function base64_to_img() can be used to decode the image and save it in an appropriate format.

CaveatsGiven that search engines change their frontpage from time to time, the extension can break and needs to be adjusted.As of writing (22/03/2022), the search engine Bing is not supported due to some changes on their frontpage.

Both the Python and R library rely on css selectors to extract the relevant information from the html files. This is even more fragile thanwhat the browser extension does and may require more frequent updates. Both libraries though offer the possibility to use custom selectors.This can be useful in cases where search engines updated their css classes and those changes have not yet been incorporated into the libraries.

To leave a comment for the author, please follow the link and comment on their blog: schochastics.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: A suite of tools to scrape and parse search engine results

View Details

[This article was first published on R | TypeThePipe, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Are you tired of creating lag variables one by one? Are you ready to level up your time series analysis game? Forget everything you know about creating lag variables. There’s a better way, and it’s been right in front of you all along.

This is a good one. We’ll make use of the semi-unknown partial function to create a useful wrapper around the lag function. Let’s go straight to the point.

First, we create a new function called map_lag. This function is essentially a mapped version of the lag function from dplyr, where we pre-fill the n argument to create different lag functions. Then, we can apply this list of functions, each one representing a different lag length, to the desired variable.

And just like that, voila! We have multiple lag variables without breaking a sweat. To make things even better, we can change the names of our newly created lag variables on the fly to make them more meaningful.

calculate\_lags <- function(df, var, lags){ map\_lag <- lags %>% map(~partial(lag, n = .x)) return(df %>% mutate(across(.cols = {{var}}, .fns = map\_lag, .names = "{.col}\_lag{lags}")))} Let’s see a quick example. We’ll be using the closing prices of the TSLA stock to showcase its use. We have a data frame like this:

tsla %>% head(4)## # A tibble: 4 × 6## date open high low close volume## <date> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 2022-01-03 383. 400. 379. 400. 104686047## 2 2022-01-04 397. 403. 374. 383. 100248258## 3 2022-01-05 382. 390. 360. 363. 80119797## 4 2022-01-06 359 363. 340. 355. 90336474 We simply pass the desired lags to the function, as well as the column we will apply the lags on. Note that we are also using tidyevaluation to reference the column without quotes. This way we keep the tidyverse vibe intact.

tsla %>% calculate\_lags(close, 1:3) %>% head()## # A tibble: 6 × 9## date open high low close volume close\_lag1 close\_lag2 close\_lag3## <date> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 2022-01-03 383. 400. 379. 400. 104686047 NA NA NA## 2 2022-01-04 397. 403. 374. 383. 100248258 400. NA NA## 3 2022-01-05 382. 390. 360. 363. 80119797 383. 400. NA## 4 2022-01-06 359 363. 340. 355. 90336474 363. 383. 400.## 5 2022-01-07 360. 360. 337. 342. 84164748 355. 363. 383.## 6 2022-01-10 333. 353. 327. 353. 91814877 342. 355. 363. It’s time to create your own lags like a pro. Embrace the power of purrr and partial and take your time series analysis to the next level. You will impress your colleagues with your advanced R skills and will have more time to focus on the real analysis.

Short and sweet!

Suscribe for more R tips!To leave a comment for the author, please follow the link and comment on their blog: R | TypeThePipe.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Unlocking the Power of purrr: How to Create Multiple Lags Like a Pro in R

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.When interviewing impressive applicants from a stunning variety of places and background for fellows in our Data Science for Social Good program (in Warwick and Kaiserslautern) this summer, we came through the common conundrum of comparing ranks while each of us only meeting a subset of the candidates. Over a free morning, I briefly thought of the problem (while swimming) and then wrote a short R code to infer about an aggregate ranking, ρ, based on a simple model, namely a Poisson distribution on the distance between an individual’s ranking and the aggregate

a uniform distribution on the missing ranks as well as on the aggregate, and a non-informative prior on λ. Leading to a three step Gibbs sampler for the completion and the simulation of ρ and λ.

I am aware that the problem has been tackled in many different ways, including Bayesian ones (as in Deng et al., 2014) and local ones, but this was a fun exercise. Albeit we did not use any model in the end!

To leave a comment for the author, please follow the link and comment on their blog: R – Xi'an's Og.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: partial rankings and aggregate ranks

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.After writing the initial version of a tutorial on wrapping and binding R functions on the javascript side of WebR, I had a number of other WebR projects on the TODO list. But, I stuck around noodling on the whole “wrapping & binding” thing, and it dawned on me that there was a “pretty simple” way to make R functions available to javascript using javascript’s Function() constructor. I’ll eventually get this whole thing into the GH webr-experiments repo, but you can see below and just view-source (it’s as of the time of this post, in the 130’s re: line #’s) to check it out before that.

Dynamic JavaScript Function CreationThe Function() constructor has many modes, one of which involves providing the whole function source and letting it build a function for you. R folks likely know you can do that in R (ref: my {curlconverter} pkg), and it’s as straightforward on the javascript side. Open up DevTools Console and enter this:

const sayHello = new Function('return function (name) { return `Hello, ${name}` }')(); then call the function with some string value.

We only need one input to create a dynamic R function wrapper (for SUPER BASIC R functions ONLY). Yes, I truly mean that you can just do this:

let rbeta = await wrapRFunction("rbeta");await rbeta(10, 1, 1, 0) and get back the result in the way WebR’s toJs() function returns R objects:

{ "type": "double", "names": null, "values": [ 0.9398577840605595, 0.42045006265859153, 0.26946718094298633, 0.3913958406551122, 0.8123499099597378, 0.49116132695862963, 0.754970193716774, 0.2952198011408607, 0.11734111483990002, 0.6263863870230043 ]} Formal AttireR’s formals() function will let us get the argument list to a function, including any default values. We won’t be using them in this MVP version of the auto-wrapper, but I see no reason we couldn’t do that in a later iteration. It’s “easy” to make that a javascript function “the hard way”:

async function formals(rFunctionName) { let result = await globalThis.webR.evalR(`formals(${rFunctionName})`); let output = await result.toJs(); return(Promise.resolve(output))} Now, we just need to use those names to construct the function. This is an ugly little function, but it’s really just doing some basic things:

async function wrapRFunction(rFunctionName) { let f = await formals(rFunctionName) let argNames = f.names.filter(argName => argName != '...'); let params = argNames.join(', ') let env = argNames.map(name => `${name}: ${name}`).join(', '); let fbody = `return async function ${rFunctionName}(${params}) { let result = await globalThis.webR.evalR( '${rFunctionName}(${params})', { env: { ${env} }} ); let output = await result.toJs(); globalThis.webR.destroy(result); return Promise.resolve(output); }`; return new Function(fbody)()} * first, we get the formals for the function name provided to us * then we remove ones we can’t handle (yet!) * then we take the R function parameter names and make two objects: + one that will make a comma-separated parameter declaration list (e.g. param1, param2, param3) + another that does the same, but as key: value pairs to pass as the environment (see previous WebR blog posts and experiments) * the function body is another template string that just makes the standard WebR set operations to evaluate an R object and get the value back.

We pass the built function string to the Function() constructor, and we have an automagic javascript wrapper for it.

FINThis is a very naive wrapper. It’s all on the caller to specify all the values. R has some fun features, like the ability to not specify a value for a given parameter, and then check on the R side to see if it’s missing. It can also intermix named and positional parameters, and then there’s the dreaded ....

I’m going to noodle a bit more on how best to account and/or implement ^^, but I’ll 100% be using this idiom to wrap R functions as I keep experimenting.

One more thing: I had to abandon the use of the microlight syntax highlighter. It’s super tiny and fast, but it doesn’t handle the code I need, so I’ll be updating my nascent webr-template to incorporate what I am using now: Shiki. The next release will also include these basic wrapper functions.

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: You’re One JavaScript Function Call Away From Using (Most) WebR R Functions In Your WebR-Powered Apps/Sites

View Details

[This article was first published on Steve's Data Tips and Tricks, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't. IntroductionThe CCI30 Crypto Index is a cryptocurrency index that tracks the performance of the top 30 cryptocurrencies by market capitalization. It was created in 2017 by a team of researchers and analysts from the CryptoCompare and MVIS indices.

The CCI30 Crypto Index is designed to provide a broad-based and representative measure of the cryptocurrency market’s overall performance. It includes a diverse range of cryptocurrencies, such as Bitcoin, Ethereum, Litecoin, Ripple, and many others. The index is weighted by market capitalization, with each cryptocurrency’s weight determined by its market capitalization relative to the total market capitalization of all 30 cryptocurrencies.

The CCI30 Crypto Index has become a popular benchmark for the cryptocurrency market, as it offers a comprehensive view of the market’s performance, rather than just focusing on one particular cryptocurrency. It is often used by investors, traders, and researchers to analyze trends and make investment decisions.

One notable feature of the CCI30 Crypto Index is that it is rebalanced every quarter. This means that the composition of the index is adjusted to reflect changes in the market capitalization of the constituent cryptocurrencies. This helps to ensure that the index remains representative of the overall cryptocurrency market.

Overall, the CCI30 Crypto Index provides a useful tool for tracking the performance of the cryptocurrency market. It is a valuable resource for investors, traders, and researchers who are interested in this exciting and rapidly evolving field.

Code ExplanationLet’s break it down step by step:

The first line of the code loads the “dplyr” package, which provides a set of functions for data manipulation.
The second line of the code reads the HTML code from the website “https://cci30.com/” using the “read_html” function from the “xml2” package.
The next two blocks of code extract two tables from the HTML document using the “html_node” function from the “rvest” package. The tables are located at two different XPaths in the HTML document.
The extracted tables are then converted into tibbles using the “as_tibble” function from the “tibble” package. The tibbles are further transformed by selecting only the columns from the second to the fifth column using the “select” function from the “dplyr” package.
The column names of the tibbles are then set using the “set_names” function from the “purrr” package.
Finally, the two tibbles are combined using the “union” function from the “dplyr” package, and the resulting tibble is printed to the console.

In summary, the code is extracting two tables from a website, transforming them into tibbles, selecting a subset of columns, renaming the columns, and combining them into a single tibble.

Example cci30 <- xml2::read\_html("https://cci30.com/")tbl1 <- cci30 |> rvest::html\_node(xpath = "/html/body/div[2]/div/div/div/div[2]/div[1]/table") |> rvest::html\_table(header = 1) |> tibble::as\_tibble() |> dplyr::select(2:5) |> purrr::set\_names( "Coin","Price","Mkt Cap","Daily Change" )tbl2 <- cci30 |> rvest::html\_node(xpath = "/html/body/div[2]/div/div/div/div[2]/div[2]/table") |> rvest::html\_table(header = 1) |> tibble::as\_tibble() |> dplyr::select(2:5) |> purrr::set\_names( "Coin","Price","Mkt Cap","Daily Change" )tbl <- tbl1 |> dplyr::union(tbl2) |> knitr::kable()tbl

| Coin | Price | Mkt Cap | Daily Change | | --- | --- | --- | --- | | Bitcoin | $27,767.24 | $536,553,055,078 | 0.17% | | Ethereum | $1,735.32 | $212,357,972,798 | 0.04% | | BNB | $332.92 | $52,565,516,823 | 0.13% | | XRP | $0.37 | $19,087,613,742 | 0.13% | | Cardano | $0.33 | $11,547,419,916 | 0.05% | | Polygon | $1.10 | $9,643,536,324 | 0.17% | | Dogecoin | $0.07 | $9,484,198,878 | 0.34% | | Solana | $22.18 | $8,507,167,040 | 0.12% | | Polkadot | $6.10 | $7,119,808,610 | 0.08% | | Shiba Inu | $0.00 | $6,169,390,592 | 0.24% | | TRON | $0.07 | $5,936,468,687 | 0.14% | | Litecoin | $78.42 | $5,685,409,727 | 0.40% | | Avalanche | $16.64 | $5,418,625,875 | 0.06% | | Uniswap | $6.19 | $4,716,487,304 | 0.19% | | Chainlink | $7.06 | $3,649,558,739 | 0.06% | | Cosmos | $11.56 | $3,309,216,299 | 0.03% | | UNUS SED LEO | $3.35 | $3,195,413,769 | -0.55% | | Toncoin | $2.38 | $2,907,590,168 | -0.62% | | Monero | $151.58 | $2,767,118,876 | -0.01% | | Ethereum Classic | $19.58 | $2,741,016,944 | -6.84% | | OKB | $44.34 | $2,660,455,327 | 0.56% | | Bitcoin Cash | $130.60 | $2,526,173,508 | -0.48% | | Stellar | $0.09 | $2,293,156,708 | -0.04% | | Cronos | $0.07 | $1,787,408,658 | 1.05% | | NEAR Protocol | $2.00 | $1,728,135,015 | 0.26% | | VeChain | $0.02 | $1,665,251,562 | 0.33% | | Quant | $126.33 | $1,525,183,149 | 0.31% | | Internet Computer | $5.11 | $1,516,076,264 | 0.19% | | Algorand | $0.21 | $1,498,361,340 | 0.41% | | ApeCoin | $4.06 | $1,496,070,125 | 0.12% |

To leave a comment for the author, please follow the link and comment on their blog: Steve's Data Tips and Tricks.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Getting the CCI30 Index Current Makeup

View Details

[This article was first published on R – Jacob Smith Economics, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Probability weighting without rules can make no sense. Fortunately we have some guidance.Introduction: What is Rank Dependent Utility?Rank dependent utility theory made its debut in John Quiggin’s 1982 paper “A Theory of Anticipated Utility” with the mission, similar to other papers at the time, to come up with a model which addressed the Allais Paradox which was a major problem for expected utility theory. what all prior attempts had in common was replacing probabilities of specific outcomes occuring with “decision weights” or “weighted probabilities” which expressed how individuals use information regarding the probability of outcomes occuring. What made this attempt different was the fact that Quiggin came up with a formulation which required these decision weights to add up to one and preserve dominance (something which usually is not preserved in usual probability weighting).

To illustrate the problem with usual probability weighting suppose an individual faces the choice between being given a dollar with certainty and facing a 50-50 gamble between winning one dollar or two dollars. If our individual weights their probabilities according to the function w(p)=p^2 we end up with a scenario where:

1>1(1/2)^2+2(1/2)^2
1>11/4+21/4
1>3/4

While the math checks out, the decision is ridiculous as the gamble is clearly better than just getting dollar with certainty. To rule out these cases, we need to put some rules on how they are computed. The steps for computing rank dependent utility is given in Peter Wakker’s 2010 text “Prospect Theory: For Risk and Ambiguity” .

Building RDU_Compute()While there is likely a more efficient code for generating a function which computes rank dependent utility, I have come up with the following to compute these the rank dependent utility of a gamble.

RDU\_Compute<-function(outcomes,p\_vec,pw){if(length(outcomes)==length(p\_vec)){ #Step 0 organize outcomes and probabilities into a single vectordf<-data.frame(outcomes,p\_vec)#Step 1 Organize Probabilities by outcome, inserting a row of zeros in your data frame.df1<-df[order(-outcomes),]zerorow<-c(0,0)df2<-rbind(zerorow,df1)#Step 2 Define vector of ranksrank<-cumsum(df2[,2])df3<-data.frame(df2,rank)#Step 3 compute pweightspw\_vec<-pw(df3$rank)df4<-data.frame(df3,pw\_vec)#Step 4 Take difference between pw\_vec to compute decision weightsd\_weights<-diff(df4$pw\_vec)#Check if weights sum to 1sum(d\_weights)#Add to dataframed\_weights1<-c(0,d\_weights)df5<-data.frame(df4,d\_weights1)#Step 5: Compute Rank Dependent UtilityRDU<-df5$outcomes%*%df5$d\_weights1return(data.frame(RDU,sum(d\_weights1)))} else{ print("Outcomes and vector of probabilities must be the same length")}} It should be noted:

RDU_Compute() requires a list of outcomes or payoffs, list of probabilities and probability weighting function (denoted by pw in the code).
The number of probabilities and outcomes must be the same, if not an error message will show.
The function is designed in a way to display two outputs, the first being the computed rank dependent utility of the gamble and if the sum of decision weights add up to 1. while this is satisfied in most cases, there are circumstances where you can break it as we will see below.

A Computed ExampleConsidering the rank dependent utility with a prospect that has four different outcomes we get the following:

rewards<-c(100,70,80,120)probs<-c(0.25,0.1,0.6,0.05)p2<-function(p){p^2}RDU\_Compute(rewards,probs,p2)#Output: RDU sum.d\_weights1.1 1.091 1 Further its simple to illustrate how we can solve the initial issue of dominance. When valuing a 50-50 gamble between one dollar and two dollars we get:

rewards<-c(1,2)probs<-c(0.5,0.5)p2<-function(p){p^2}RDU\_Compute(c(1,2),c(0.5,0.5),p2)#Output: RDU sum.d\_weights1.1 1.25 1 Probability Weighting Functions that Break Rank Dependent UtilityFor reasons that are not entirely clear to me (at the time of writing this article) we can easily violate this condition of having our decision weights summing to one. To illustrate consider the case of where w(p)=p^2+0.01*p, this gives us the following:

rewards<-c(1,2)probs<-c(0.5,0.5)pbreak<-function(p){p^2+0.01*p}RDU\_Compute(c(1,2),c(0.5,0.5),pbreak)#Output: RDU sum.d\_weights1.1 1.265 1.01 There must be some structure required of these probability weighting functions, but at this time I cant quite figure it out. If you know anything on this be sure to let me know in the comments below!

To leave a comment for the author, please follow the link and comment on their blog: R – Jacob Smith Economics.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Computing Rank Dependent Utility in R

View Details

[This article was first published on ouR data generation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Very large data sets can present estimation problems for some statistical models, particularly ones that cannot avoid matrix inversion. For example, generalized estimating equations (GEE) models that are used when individual observations are correlated within groups can have severe computation challenges when the cluster sizes get too large. GEE are often used when repeated measures for an individual are collected over time; the individual is considered the cluster in this analysis. Estimation in this case is not really an issue because the cluster sizes are typically relatively small. However, if there are groups of individuals, we also need to account for correlation. Unfortunately, if these group/cluster sizes are too large – perhaps bigger than 1000 – traditional GEE estimation techniques just may not be feasible.

An approach to GEE that is feasible has been described by Lipsitz et al and implemented using a SAS macro (which is available on the journal’s website). I am not much of a SAS user, so I searched for an implementation in R. Since I didn’t come across anything, I went ahead and implemented it myself. I am undecided about creating an R package for this, but in the meantime I thought I would compare it to a standard package in R and provide a link to the code if you’d like to implement it yourself. (And, if it does already exist in an R package, definitely let me know, because I certainly don’t want to duplicate anything.)

The one-step GEE algorithmTraditional GEE models (such as those fit with R packages gee and geepack) allow for flexibility in specifying the within-cluster correlation structure (we generally still assume that individuals in different clusters are uncorrelated). For example, one could assume that the correlation across individuals is constant within a cluster. We call this exchangeable or compound symmetry correlation, and the intra-cluster correlation (ICC) is the measure of that correlation. Alternatively, if measurements are collected over time, we might assume that measurements taken closer together are more highly correlated; this is called auto-regressive correlation.

The proposed algorithm that is implemented here is called the one-step GEE, and is operating under the assumption of exchangeable correlation. To provide a little more detail on the algorithm, but to keep it simple, let me quote directly from the paper’s abstract:

We propose a one-step GEE estimator that (1) matches the asymptotic efficiency of the fully iterated GEE; (2) uses a simpler formula to estimate the [intra-cluster correlation] ICC that avoids summing over all pairs; and (3) completely avoids matrix multiplications and inversions. These three features make the proposed estimator much less computationally intensive, especially with large cluster sizes. A unique contribution of this article is that it expresses the GEE estimating equations incorporating the ICC as a simple sum of vectors and scalars.

The rest of the way, I will simulate data and fit the models using the traditional estimation approach as well as the one-step approach.

Comparing standard GEE with one-step GEETo start, I am simulating a simple data set with 100 clusters that average 100 individuals per cluster.

library(simstudy)library(data.table)library(geepack) First, the definitions used in the data generation. Each individual has three covariates, and the probability is a function of two of them:

d1 <- defData(varname = "n", formula = 100, dist = "noZeroPoisson")d2 <- defDataAdd(varname = "x1", formula = 0, variance = .1, dist = "normal")d2 <- defDataAdd(d2, varname = "x2", formula = 0, variance = .1, dist = "normal")d2 <- defDataAdd(d2, varname = "x3", formula = 0, variance = .1, dist = "normal")d2 <- defDataAdd(d2, varname = "p", formula = "-0.7 + 0.7*x1 - 0.4*x2", dist = "nonrandom", link="logit") And then the data generation - the final step creates the within-site correlated outcomes with an ICC of 0.15:

set.seed(1234)ds <- genData(100, d1, id = "site")dc <- genCluster(dtClust = ds, cLevelVar = "site", numIndsVar = "n", level1ID = "id")dc <- addColumns(d2, dc)dd <- addCorGen(dc, idvar = "site", param1 = "p", rho = 0.15, corstr = "cs", dist = "binary", cnames = "y", method = "ep") Here’s a few records from the data set, which has just under 10,000 observations across the 100 clusters:

dd## site n id x1 x2 x3 p y## 1: 1 88 1 -0.57111723 0.21022042 -0.31201547 0.2343570 0## 2: 1 88 2 -0.18406857 0.68915628 -0.72801775 0.2488957 0## 3: 1 88 3 -0.35066169 -0.10884256 0.28141278 0.2886548 0## 4: 1 88 4 -0.32095917 -0.03676544 -0.33847489 0.2870070 0## 5: 1 88 5 -0.05132678 0.07095837 -0.09308927 0.3177108 1## --- ## 9784: 100 106 9784 -0.26912777 -0.22893853 0.16303770 0.3107074 1## 9785: 100 106 9785 0.16399013 0.26200677 -0.03187061 0.3340309 0## 9786: 100 106 9786 0.14450843 -0.24399797 -0.28108422 0.3772482 1## 9787: 100 106 9787 0.29307929 0.03889844 -0.37495296 0.3750989 0## 9788: 100 106 9788 -0.06246070 -0.41041326 0.43236084 0.3590345 1 We can fit a regular GEE model here, since the cluster sizes are relatively small:

system.time(geefit <- geese(y ~ x1 + x2 + x3, id = site, data = dd, family = binomial, corstr = "exchangeable"))## user system elapsed ## 2.264 0.102 1.946summary(geefit)## ## Call:## geese(formula = y ~ x1 + x2 + x3, id = site, data = dd, family = binomial, ## corstr = "exchangeable")## ## Mean Model:## Mean Link: logit ## Variance to Mean Relation: binomial ## ## Coefficients:## estimate san.se wald p## (Intercept) -0.7275165 0.08459309 73.9632404 0.000000e+00## x1 0.7396758 0.06584516 126.1929384 0.000000e+00## x2 -0.3191633 0.06073004 27.6196887 1.476680e-07## x3 -0.0477172 0.06113708 0.6091727 4.350995e-01## ## Scale Model:## Scale Link: identity ## ## Estimated Scale Parameters:## estimate san.se wald p## (Intercept) 0.9954265 0.03464657 825.4633 0## ## Correlation Model:## Correlation Structure: exchangeable ## Correlation Link: identity ## ## Estimated Correlation Parameters:## estimate san.se wald p## alpha 0.1441719 0.02168204 44.21411 2.943523e-11## ## Returned Error Value: 0 ## Number of clusters: 100 Maximum cluster size: 125 The one-step GEE function (which I’ve called gee1step) runs quite a bit faster than the standard GEE model (more than 10 times faster), but the results are virtually identical.

system.time(fit1 <- gee1step(y ~ x1 + x2 + x3, data = dd, cluster = "site"))## user system elapsed ## 0.133 0.019 0.087fit1## $estimates## est se.err z p.value## Intercept -0.72743563 0.08549337 -8.5086793 8.796283e-18## x1 0.73943837 0.06704127 11.0296000 1.375430e-28## x2 -0.31903588 0.06136514 -5.1989760 1.001947e-07## x3 -0.04758544 0.06165700 -0.7717768 2.201233e-01## ## $rho## [1] 0.1440159## ## $clusters## $clusters$n\_clusters## [1] 100## ## $clusters$avg\_size## [1] 97.88## ## $clusters$min\_size## [1] 77## ## $clusters$max\_size## [1] 125## ## ## $outcome## [1] "y"## ## $model## y ~ x1 + x2 + x3## ## attr(,"class")## [1] "gee1step" The one-step algorithm with very large cluster sizesObviously, in the previous example, gee1step is unnecessary because geese handled the data set just fine. But, in the next example, with an average of 10,000 observations per cluster, geese will not run - at least not on my MacBook Pro. gee1step does just fine. I’m generating the data slightly differently here since simstudy doesn’t do well with extremely large correlation matrices. I’m using a random effect instead to induce correlation:

vicc <- iccRE(0.15, dist = "binary")d1 <- defData(varname = "n", formula = 10000, dist = "noZeroPoisson")d1 <- defData(d1, varname = "b", formula = 0, variance = vicc)d2 <- defDataAdd(varname = "x1", formula = 0, variance = .1, dist = "normal")d2 <- defDataAdd(d2, varname = "x2", formula = 0, variance = .1, dist = "normal")d2 <- defDataAdd(d2, varname = "x3", formula = 0, variance = .1, dist = "normal")d2 <- defDataAdd(d2, varname = "y", formula = "-0.7 + 0.7*x1 - 0.4*x2 + b", dist = "binary", link="logit")### generate dataset.seed(1234)ds <- genData(100, d1, id = "site")dc <- genCluster(dtClust = ds, cLevelVar = "site", numIndsVar = "n", level1ID = "id")dd <- addColumns(d2, dc) Now, we have almost one million observations:

dd## site n b id x1 x2 x3 y## 1: 1 9879 -1.3761022 1 -0.11929302 0.4136057653 -0.16119383 1## 2: 1 9879 -1.3761022 2 0.03086998 0.3085905336 0.46055431 0## 3: 1 9879 -1.3761022 3 0.51821656 0.2047086274 0.16721059 0## 4: 1 9879 -1.3761022 4 -0.27688665 0.0030742246 -1.08113944 0## 5: 1 9879 -1.3761022 5 0.03850389 -0.0991678903 -0.09447011 0## --- ## 997872: 100 10065 0.2648166 997872 0.46022770 0.1588510133 -0.11851768 0## 997873: 100 10065 0.2648166 997873 0.10696608 0.0002424567 -0.10632926 1## 997874: 100 10065 0.2648166 997874 0.32317258 -0.1626614787 0.37855380 1## 997875: 100 10065 0.2648166 997875 -0.17992641 -0.0333636043 -0.20060539 0## 997876: 100 10065 0.2648166 997876 0.65942651 0.1960137135 -0.05687481 1 Despite the very large cluster sizes, the one-step algorithm still runs very fast. In addition to what is shown here, I have conducted experiments with repeated data sets to confirm that the coefficient estimates are unbiased and the standard error estimates are correct.

system.time(fit1 <- gee1step(y ~ x1 + x2 + x3, data = dd, cluster = "site"))## user system elapsed ## 2.226 0.329 1.611fit1## $estimates## est se.err z p.value## Intercept -0.569908988 0.069360637 -8.216605 1.046722e-16## x1 0.615260991 0.010232501 60.128117 0.000000e+00## x2 -0.349871546 0.008694526 -40.240439 0.000000e+00## x3 0.008132196 0.006470784 1.256756 1.044210e-01## ## $rho## [1] 0.1016716## ## $clusters## $clusters$n\_clusters## [1] 100## ## $clusters$avg\_size## [1] 9978.76## ## $clusters$min\_size## [1] 9766## ## $clusters$max\_size## [1] 10242## ## ## $outcome## [1] "y"## ## $model## y ~ x1 + x2 + x3## ## attr(,"class")## [1] "gee1step" And please, if someone thinks it would be valuable for me to create a package for this, let me know. It would certainly help motivate me :).

Reference:

Lipsitz, Stuart, Garrett Fitzmaurice, Debajyoti Sinha, Nathanael Hevelone, Jim Hu, and Louis L. Nguyen. “One-step generalized estimating equations with large cluster sizes.” Journal of Computational and Graphical Statistics 26, no. 3 (2017): 734-737.

To leave a comment for the author, please follow the link and comment on their blog: ouR data generation.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Implementing a one-step GEE algorithm for very large cluster sizes in R

View Details

[This article was first published on R – Xi'an's Og, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Hey, JAGS users and would-be users, be warned that registration is now open for the annual JAGS workshop on probabilistic modelling for cognitive science. The tenth instalment of this workshop takes place July 10–14, 2023 in Amsterdam and online. This workshop is meant for researchers who want to learn how to apply Bayesian inference in practice. Most applications we discuss are taken from the field of cognitive science. The workshop is based on the book Bayesian Cognitive Modeling: A practical course written by Michael Lee and Eric-Jan Wagenmakers. It is followed by a shorter workshop (15-16 July) on Theory and Practice of Bayesian Hypothesis Testing.

To leave a comment for the author, please follow the link and comment on their blog: R – Xi'an's Og.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: JAGS Workshop [10-14 July 2023]

View Details

[This article was first published on R on Publishable Stuff, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.The first thing I thought when I tried all the cool tools of the Year of the AI Revolution (aka 2022) was: OMG this is amazing, it’s the AI future that I never thought I would see. The second thing I thought was: OMG this is going to be used to spam the internet with so much bland auto-generated content.

I hate bland auto-generated content as much as the next person, but I was tempted by the forbidden fruit, I irresponsibly took a bite, and two short R scripts and a weekend later I’m now the not-so-proud owner of officialcocktails.com: A completely auto-generated website with recipes, description, tips, images, etc. covering all the official International Bartenders Association cocktails.

Here’s the quick recipe for how I whipped this up.

Making an AI-generated websiteThis was a quick experiment to try to auto-generate a website using modern AI tools with as little effort as possible, so the code to do this ended up quick and dirty, and put up here on GitHub more for the record. But shortly, the recipe is as follows.

First up you need some data as a text-generating AI like ChatGPT needs something to work with. Here I went with the IBA list of official cocktails that I scraped from wikipedia earlier:

[{ "category": "The unforgettables", "name": "Whiskey sour", "type": "Cocktail", "served": "On the rocks or straight up", "standard\_drinkware": "Old fashioned glass", "method": "Shake with ice. Strain into chilled glass, garnish and serve.", "notes": "If served 'On the rocks', strain ingredients into old-fashioned glass filled with ice.", "ingredients": ["4.5 cl (3 parts) bourbon whiskey", "3 cl (2 parts) fresh lemon juice", "1.5 cl (1 part) simple syrup"]},{ "category": "Contemporary classics", "name": "Black Russian", "type": "Cocktail", "served": "On the rocks: poured over ice", "standard\_drinkware": "Old fashioned glass", "method": "Pour the ingredients into the old fashioned glass filled with ice cubes. Stir gently.", "ingredients": ["50 ml Vodka", "20 ml Coffee liqueur"]}// etc. Secondly, you need to use this dataset to generate prompts that the AIs will expand into more text (“List fun facts about the Cuba Libre cocktail”) and images of the cocktails (“A photograph of a Cuba Libre”). This is what 01_regenerate-assets.R does. That script will create a cocktail-descriptions.yaml file that contains all the drink info scraped from Wikipedia (from iba-cocktails-wiki.json) plus empty fields for all the things we want to fill in, like the image, fun facts, etc. The script will then go on to fill in all the blanks. For text generation, I used the ChatGPT API and for image generation, I picked one of the many available Stable Diffusion APIs.

For example, by this process the scraped Whiskey sour data above got expanded with this auto-generated texts and image:

hook: Whiskey sour is a classic cocktail made with whiskey, lemon juice, and sugar. It has a sweet and sour taste, and is usually served on the rocks.extended\_method: |- 1. Fill a cocktail shaker with ice. 2. Pour the bourbon whiskey into the shaker. 3. Add the fresh lemon juice. 4. Pour the simple syrup into the shaker. 5. Shake the mixture well for about 10-15 seconds. 6. Strain the mixture into an Old fashioned glass filled with ice. 7. Garnish the drink with a lemon wedge or cherry (optional). 8. Serve and enjoy! tips: |- - **Use fresh ingredients**: Use freshly squeezed lemon juice instead of bottled lemon juice for a brighter, more authentic flavor. Similarly, use freshly cracked ice instead of pre-made ice cubes for a smoother, better blended drink. - **Balance the sweet and sour flavors**: A good whiskey sour should be perfectly balanced between sweet and sour flavors. Start by adding equal parts simple syrup and lemon juice, then adjust as needed to suit your personal taste. - **Choose the right whiskey**: Opt for a bourbon or rye whiskey with a smooth, mellow flavor that won't overpower the other ingredients. - **Shake it up**: Make sure to shake the whiskey sour well to properly blend the ingredients and create a frothy foam on top. - **Garnish with style**: A traditional whiskey sour is typically garnished with a lemon wheel and cherry, but feel free to get creative and add your own personal touch with a unique garnish like a sprig of rosemary or a dash of cinnamon. fun\_facts: |- 1. Whiskey sour is believed to have originated in the Navy as a way to prevent scurvy. 2. Whiskey sour was a popular drink during Prohibition due to its easy-to-hide ingredients. 3. The traditional recipe calls for whiskey, lemon juice, and sugar, but modern variations may include egg whites or other flavors like bitters. 4. The cocktail gained popularity in the 1940s and was featured in the famous movie "Casablanca." 5. Some people believe that the drink was named after the sour taste of unripe lemons commonly used in the original recipe. alcohol\_free\_alternative: |- There are several delicious and easy-to-make alcohol-free alternatives to whiskey sour that you can try. Here are some options: 1. Virgin Sour: Simply swap out the whiskey for lemonade, and you have a refreshing sour drink with a citrusy zing. 2. Pineapple Paradise: Combine 2 parts pineapple juice and 1 part lemon juice, and add a splash of grenadine syrup for sweetness. Shake it all up with ice and serve in a glass with a cherry garnish. 3. Coconut Mojito: Muddle fresh mint leaves with lime juice and sugar syrup in a tall glass. Add coconut water and a splash of soda water, and garnish with more mint leaves. 4. Ginger Highball: Mix ginger beer with lime juice and cranberry juice for a spicy, fruity, and refreshing cocktail. 5. Tea-Tini: Brew a strong cup of tea (such as earl grey or green tea) and mix with lemon juice and agave syrup. Serve chilled in a martini glass with a lemon twist. image\_path: cocktail-images/whiskey\_sour.jpeg One thing that I found really useful was for the 01_regenerate-assets.R script to only fill in blank fields when rerun. Unless you’re a prompt wizard, there are always going to be generated text and images that are a bit wonky. But if you can simply delete what looks off, and then only regenerate what’s missing, it’s much quicker to fix the most egregious AI hallucinations than if you had to regenerate everything from scratch.

Third, package all the generated content into a format that a site generator like hugo can render. This is what 02_generate-website.R does. That script takes the auto-generated cocktails descriptions and images from cocktail-descriptions.yaml, glues together one markdown file for each cocktail, and puts those files in the correct place in a hugo site structure. For example, here’s how whiskey_sour.md turned out.

Fourth, make it look OK. This was the hardest part for me, but luckily I found a decent-looking theme for hugo that I could bend to my will.

Finally, put it somewhere. To make it extra official I snagged the official-sounding URL officialcocktails.com and I’m currently hosting it using the free tier on Netlify.

How ashamed am I?I mean, I’m a bit ashamed. I’m ashamed about having added yet another drop of noise into the sea that is the Internet. I’m not sure if the warning I put on the website makes up for it:

The recipes and ingredient lists are correct, trustworthy, and fact-checked. Everything else, images included, is hallucinated by an evil AI.

I’m also a bit ashamed because the auto-generated content isn’t very good. ChatGPT does introduce a lot of silly mistakes (Whiskey sour was not “featured in the famous movie Casablanca”) and the Stable Diffusion images often include weird artifacts. However, this is not always bad, as in this interesting picture that got generated for the drink Yellow Bird:

But maybe I just need to work on my prompts…

What I’m happy about is how quick it was to generate something like this from scratch. And cheap! The API cost of generating this site was less than $1. And one thing I’m excited about is revisiting officialcocktails.com in a couple of months and replacing ChatGPT and Stable Diffusion with what’s then the latest and greatest to see how that improves the site.

To leave a comment for the author, please follow the link and comment on their blog: R on Publishable Stuff.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: officialcocktails.com – An AI-generated cocktails site

View Details

[This article was first published on rud.is"In God we trust. All others must bring data"R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.It’s difficult to believe it has only been a couple of weeks since WebR has been around. But that might just be my perception. The spike protein invasion has significantly increased sedentary time, and that has enabled me to focus on this new toy to keep my attention focused on something positive. So, I’ve had “WebR on the mind” more than most others.

As folks likely know, I’m keeping a log of these “experiments” over at GH, and I’ve blogged, here, a bit on a few of them. (I promise I’ll get to Part 2 of “ggwebr”, soon.). Today we have an update on two of them.

WebR Filesystem MachinationsThis WebR-powered page is pretty a pretty self-contained explanation of how to work with the Emscripten filesystem from both the javascript and R contexts of WebR. I set out to just do that, but I was also sick of writing HTML for everything, and spent some time coming up with a system for hacking on these experiments with Markdown.

Yes, I could just use the exciting new Quarto WebR extension and Quarto itself, but there are advantages to working in plain Markdown, HTML, and Vanilla JS blank canvas (Quarto-rendered HTML is a bit, er, crufty).

The code in this filesystem experiment is also a bit different in another way: it uses a Vanilla JS lightweight reactive framework called Reef.

It worked so well that I made a few more examples, which are covered in the remaining sections.

More Concise ExampleThe filesystem experiment is a tad big, especially since it introduces some fancy javascript things many R folks are not familiar with. That, combined with the cognitive load associated with learning WebR itself, meant I had to simplify things a bit if I wanted to convey the utility of this setup.

If you’re asking, “why not just use React?”, the answer is that React is overkill for the small projects I have in mind, and that I think many other folks who may want to use WebR also have in mind. I also don’t really like packers/bundlers. The early days of the web were fun and easy to work in, and I wanted to replicate that for WebR.

So, I made the sadly named webr-reef example. It’s a small app that:

loads a list of all datasets in the datasets package and shoves it into a <select> list.
when a dataset is selected, the str() output of it is displayed below the popup menu.

If you view index.md, you’ll see it’s just some light markdown and a couple bits of HTML for the:

reactive #message <div> that will show anything we assign to it
reactive #selected-dataset <select> list which will get automagically populated on load and, when changed, will cause the value of the next item in this list to change
reactive #dataset-output which will hold a str() of the selected dataset.

The index.html file is really just a shell for opengraph tags plus an element to put this app into. main.js does all the work.

The source is fairly well-annotated, but it still was not simple enough IMO.

ReefR TemplateReef + WebR == ReefR

I wanted to make this super easy to read and start using quickly, so it’s ready-made to deploy to GH Pages. Just tell GHP to use / of the main branch, change a couple settings — like SW_URL — and it should “just work”. Here’s the GHP Link for the template repo, and here it is running on my site.

There are just two reactive components: the #message thing I mentioned earlier, and a <pre> block that gets filled with the value of the WebR WASM R version string, which is taken from executing code in the WebR context.

FINI’ll be iterating on ReefR to add some JS and R helpers folks can use to make it easier to work with WebR, so keep an eye on the experiments and template repo if this piques your interest. Also drop issues with ideas or questions.

To leave a comment for the author, please follow the link and comment on their blog: rud.is"In God we trust. All others must bring data"R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: WebR Filesystem Machinations & ReefR

View Details

[This article was first published on R – Jacob Smith Economics, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Prospect theory made its debut back in 1979 and was one of the first major attempts to address empirical deviations from expected utility theory. One of the key ingredients in operationalizing prospect theory involve conversion of probabilities to “weighted probabilities”.

It should be noted that while there are more advanced libraries which are designed to implement prospect theoretic models (like the pt library ), my objective here was to write a function which takes a vector of probabilities and converts them into a list of weights.

The CodeThe actual code for the function pt_weights() is simple and at its core is just two for loops. I wrapped these for loops in a if else statement.

```

Prospect Theory Weighting Function (Circa 1979)pt_weights<-function(prob,gamma){if(sum(prob)==1){ b<-0 wvec<-NULLfor(p in prob){ d<-p^gamma+b b<-d} for(p in prob){ w<-(p^gamma)/(b^(1/gamma)) wvec<-rbind(wvec,w)} return(as.numeric(wvec))} else{ print("Error: probability vector must sum to 1") }}

``` ExampleThe utility of this simple code can be seen in the following example. It should be noted that this code embodies the 1979 version of prospect theory as opposed to the 1992 version which requires the probability weights to sum to one (Though the functional form is from the 1992 paper). In this case we have subadditivity of these weights.

```

Examplep_vec<-c(0.1,0.3,0.2,0.4)pt_weights(p_vec,0.61)#Output:[1] 0.1057201 0.2066338 0.1613563 0.2462715#Note: subadditivity of weighted probabilitiessum(pt_weights(p_vec,0.61))[1] 0.7199816

``` Future WorkThe next project of mine would be to compute weighted probabilities which sum to one. This requires some looking into how Rank Dependent Utility works as this was the primary springboard which transformed the 1979 version of prospect theory into an operable model of decision making as shown in the 1992 version.

Let me know your thoughts in the comments below!

To leave a comment for the author, please follow the link and comment on their blog: R – Jacob Smith Economics.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Simple code for the probability weighting function according to prospect theory

View Details

[This article was first published on r – bensstats, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.IntroductionRecently, I released a R package called starBliss that aimed to replicate the output of a e-commerce site called MapsForMoments – a site which lets users order custom prints of the night sky on the date of their choosing (usually a special occasion such as a birthday, first date, wedding etc.) and allows them to choose a style, and add some custom text. It was a great experience getting to build the package which replicated the MapsForMoments product and I was shocked to see how well it was received when I posted about it- with the Github receiving over 30 stars at the time of writing this blog!

I decided to take this to the next level by trying to build a similar UI in shiny which allows the user to create a custom star map and not need to use the R console. In this blog I share my experience constructing and showcase the “free alternative” to MapsForMoments – starBlissGUI!

Building the AppBuilding an app for starBliss presented some interesting challenges. For one, the plot output created by the plot_starmap() function is quite unwieldy. Previewing the image requires a user to have a large enough screen size and does not shrink well. The present workaround in the starBliss package is to save the plot as an image that accommodates the appropriate sizing like so:

```

install.packages("ggplot2")# devtools::install_github("benyamindsmith/starBliss")library(ggplot2)library(starBliss)p<- plot_starmap(location= "Toronto, ON, Canada", date="2022-01-17", style="black", line1_text="Toronto", line2_text ="January 17th, 2023", line3_text="43.6532° N, 79.3832° W")ggsave('toronto_black.png', plot = p, width = unit(10, 'in'), height = unit(15, 'in'))

`` ForstarBlissGUI` a temporary image is created in the server and displayed on the preview panel. The only drawback with this approach is that does create some lag on cheaper hardware and on the free tier of Shinyapps.io. You can check out the source code on the Github here and the hosted app on Shinyapps.io here (apologies for the lag, as it is being hosted on the free tier).

Building a Docker ImageSince getting a paid plan from ShinyApps.io was not something that I was particularly interested in perusing if I were to host this app professionally, I decided to explore the Docker route. With the help of Google, Stackoverflow and yes, ChatGPT (lots of prompts!) I was able to put together the following script:

FROM rocker/shiny:latestRUN apt-get update && apt-get install -y \ libssl-dev \ libcurl4-gnutls-dev \ libxml2-dev \ libudunits2-dev \ libgdal-devRUN R -e 'install.packages(c("shiny","shinyWidgets","ggplot2","lubridate", "remotes"))'RUN R -e 'remotes::install\_github("benyamindsmith/starBliss")'COPY app.R /srv/shiny-server/COPY www /srv/shiny-server/wwwEXPOSE 3838CMD ["R", "-e", "shiny::runApp('/srv/shiny-server/',, host='0.0.0.0', port=3838)"] One of the things that is important to note if you are installing packages which are hosted on github, you need to make sure that the libudunits2-dev and libgdal-dev libraries are installed in the container otherwise it will not work.

ConclusionWhile I played around with shiny before, I never tried to build anything serious with it until now. It was a very useful experience getting to thoroughly build the app. While there is still some functionality that I would like to improve on (like autofill for the location field, optimizing the server side and making a form to collect some user data before downloading the map for free), I am happy to share what I have right now!

Please be sure to check out the starBlissGUI GitHub and give it a star and open an issue if you see anything that you want to have added!

Thank you for reading!

Want to see more of my content? Be sure to subscribe and never miss an update!

Type your email…

YouTubeFacebookPatreonTo leave a comment for the author, please follow the link and comment on their blog: r – bensstats.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: RObservations #46: Starmaps and Shiny- I Replicated an E-Commerce Store Using My Custom Package, starBliss

View Details

[This article was first published on business-science.io, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.What’s the one thing that help you add value to your company’s raw geospatial data? GEOCODING.

Geocoding is the process of converting raw physical addresses to latitude and longitude geospatial points that can be viewed on a map and used for geospatial calculations. Heck – Geocoding has been known to increase my machine learning model perfomance by up to 10%!

Table of ContentsToday I’m going to show you how to do Geocoding in R for FREE using tidygeocoder. Here’s what you’re learning today:

Tutorial Part 1: How to use tidygeocoder to effortlessly geocode addresses (convert your company addresses to Lat/Long)
Tutorial Part 2: And I’m going to show you how to do Reverse Geocoding (go from Lat/Long to Physical Addresses)
Bonus: I’m going to show you how to Map lat/long data using Simple Features + Mapview!

R-Tips WeeklyThis article is part of R-Tips Weekly, a weekly video tutorial that shows you step-by-step how to do common R coding tasks. Pretty cool, right?

Here are the links to get set up.

Get the Code
YouTube Tutorial

This Tutorial is Available in VideoI have a companion video tutorial that gives you the bonus Mapview Shortcuts shown in this video (plus walks you through how to use it). And, I’m finding that a lot of my students prefer the dialogue that goes along with coding. So check out this video to see me running the code in this tutorial.

Why Geocoding is a MustLook, I’ve been working with customer data for a long time…

And one of the RICHEST sources of data is raw company addresses!

Think about it. If you know where a company is located, do you think that might be important to their purchasing behavior?

Well it was for me. In fact I found out that just simply adding the Latitude and Longitude information to my customer churn prediction models…

Gave my models a 10% increase in performance!Lot’s of Value to Machine Learning in Raw Customer Addresses

The Latitude and Longitude was key!

And that’s just one of the benefits of working with geospatial data (and geocoding).

But you’re probably thinking geospatial data is really tough.Listen, I get it. Geospatial data is a little weird.

But, you have good ole Matt Dancho to help you out.

And my promise is today, I’m going to get you on the right track.

So let’s fix that geospatial problem, and make one small step today. And it starts with geocoding.Thank You to the Developer (and Community).Before we do our deep-dive into tidygeocoder, I want to take a brief moment to thank the developers working on theTidygeocoder project, Jesse Cambon, Diego Hernangómez, Christopher Belanger and Daniel Possenriede. Without their hard work, this tutorial (and easy Geocoding) wouldn’t be possible. Thank you!

Free Gift: Cheat Sheet for my Top 100 R Packages (Special Geospatial Analysis Topics Included)Before we dive in…

You’re going to need R packages to complete the geospatial analysis that helps your company. So why not speed up the process?

To help, I’m going to share my secret weapon…

Once you download it, head over to page 3 and you’ll see several R packages I use frequently just for Data Analysis.

Which is important when you want to work in these fields:

Machine Learning
Time Series
Financial Analysis
Geospatial Analysis
Text Analysis and NLP
Shiny Web App Development

So steal my cheat sheet. It will save you a ton of time.

Tutorial: How to Geocode in R for Free with tidygeocoderTime for geocoding with tidygeocoder. Let’s have some fun!

Step 1: Load the LibrariesLoad the following libraries.

tidyverse and tidygeocoder are the main libraries.
But my bonus lat/long map hack uses sf and mapview.

Get the code.

Step 2: Get My Pittsburgh Pharmacies DatasetNext, you can steal my Pittsburgh Pharmacies dataset. This dataset is a great way to test your skills with Geocoding.

Steal The Pittsburgh Pharmacies Data Set

We’ll the Pittsburgh Pharmacies dataset (171 geocoded pharmacies) throughout the rest of this tutorial.

Get it here. It’s in the 059_geocoding folder.

Next, read the data set into R.

Get the code.

Step 3: Geocode the Address Column to get Latitude and LongitudeNext, use the geocode() function to convert a company’s physical address to a Latitude / Longitude.

Get the code.

Here’s what happens…

Get the code.

Step 4: Reverse Geocode to go from Lat/Long to Physical AddressSometimes you have a latitude and longitude and want a physical address. For example, if your salesperson needs to know what addresses to visit (you wouldn’t send them a Lat/Long… or else they’d think your nuts!)

Did you know that you can reverse geocode?

You can! Here’s how to go from Latitude / Longitude to a Physical Address. (And save your inter-office reputation)

Get the code.

And you can see that reverse geocoding creates an address from Lat/Long coordinates.

Get the code.

Bonus: Steal My Map Hack to Visualize Lat/Long DataWant to visualize the geocoded data?

Steal my bonus script here. (It’s in the 059_geocode.R file)

Here’s what it does in 2 lines of code:Now you can visualize all 171 Pittsburgh Pharmacies in an interactive map!

ConclusionsYou learned how to use the tidygeocoder library to geocode and reverse geocode. Great work! But, there’s a lot more to becoming a data scientist.