[This article was first published on Econometrics and Free Software, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Data scientists, statisticians, analysts, researchers, and many otherprofessionals write a lot of code.
Not only do they write a lot of code, but they must also read and review a lotof code as well. They either work in teams and need to review each other’s code,or need to be able to reproduce results from past projects, be it for peerreview or auditing purposes. And yet, they never, or very rarely, get taughtthe tools and techniques that would make the process of writing, collaborating,reviewing and reproducing projects possible.
Which is truly unfortunate because software engineers face the same challengesand solved them decades ago. Software engineers developed a set of projectmanagement techniques and tools that non-programmers who write a lot of codecould benefit from as well.
These tools and techniques can be used right from the start of a project at aminimal cost, such that the analysis is well-tested, well-documented,trustworthy and reproducible by design. Projects are going to be reproduciblesimply because they were engineered, from the start, to be reproducible.
But all these tools, frameworks and techniques boil down to two acronyms that Ilike to keep in my head at all times:
DRY WIT: by systematically avoiding not to repeat yourself andby writing everything down, projects become well-tested, well-documented,trustworthy and reproducible by design. Why is that?
DRY: Don’t Repeat YourselfLet’s start with DRY: what does it mean not having to repeat oneself? It means:
The most widely used programming languages for data science/statistics, Python and R,both have first-class functions. This means that functions can be manipulated likeany other object. So something like:
Reduce(`+`, seq(1:100))## [1] 5050
where the function +() gets used as an argument of the higher-order Reduce()function is absolutely valid (and so is Python’s equivalent reduce fromfunctools) and avoids having to use a for-loop which can lead to other issues.Generally speaking, the functional programming paradigm lends itself verynaturally to data analysis tasks, and in my opinion data scientists andstatisticians would benefit a lot from adopting this paradigm.
Literate programming is another tool that needs to be in the toolbox ofany person analysing data. This is because at the end of the day, the resultsof an analysis need to be in some form of document. Without literate programming,this is how you would draft reports:
But with literate programming, this is how this loop would look like:
Quarto is the latest open-source scientific and technicalpublishing system that leverages Pandoc and supports R, Python, Julia andObservableJs right out of the box.
Below is a little Quarto Hello World:
---output: pdf---In this example we embed parts of the examples from the\texttt{kruskal.test} help page into a LaTeX document:{r}data (airquality)kruskal.test(Ozone ~ Month, data = airquality)which shows that the location parameter of the Ozonedistribution varies significantly from month to month.Finally we include a boxplot of the data:{r, echo = FALSE}boxplot(Ozone ~ Month, data = airquality)
Compiling this document results in the following:
Example from Leisch’s 2002 paper.Of course, you could use Python code chunks instead of R, you could also compilethis document to Word, or HTML, or anything else really. By combining code andprose, the process of data analysis gets streamlined and we don’t need to repeatourselves copy and pasting images and tables into Word documents.
Finally, treating code as data is also quite useful. This means that it ispossible to compute on the language itself. This is a more advanced topic, butdefinitely worth the effort. As an illustration, consider the following R toy example:
show\_and\_eval <- function(f, ...){ f <- deparse(substitute(f)) dots <- list(...) message("Evaluating: ", f, "() with arguments: ", deparse(dots)) do.call(f, dots)}
Running this function does the following:
show\_and\_eval(sqrt, 2)## Evaluating: sqrt() with arguments: list(2)## [1] 1.414214show\_and\_eval(mean, x = c(NA, 1, 2))## Evaluating: mean() with arguments: list(x = c(NA, 1, 2))## [1] NAshow\_and\_eval(mean, x = c(NA, 1, 2), na.rm = TRUE)## Evaluating: mean() with arguments: list(x = c(NA, 1, 2), na.rm = TRUE)## [1] 1.5
This is incredibly useful when writing packages (to know more about thesetechniques in the R programming language, read the chapter Metaprogramming fromAdvanced R).
WIT: Write It DownNow on the WIT bit: write it down. You’ve just written a function. To see ifit works correctly, you test it in the interactive console. You execute thetest, see that it works, and move on. But wait! What you just did is called aunit test. Instead of writing that in the console and then never use it everagain, write it down in a script. Now you’ve got a unit test for that functionthat you can execute each time you update that function’s code, and make surethat it keeps working as expected. There are many unit testing frameworks thatcan help you how to write unit tests consistently and run them automatically.
Documentation: write it down! How does the function work? What are its inputs?Its outputs? What else should the user know to make it work? Very often,documentation is but a series of comments in your scripts. That’s already nice,but using literate programming, you could also turn these comments into properdocumentation. You could use docstrings in Python or {roxygen2} stylecomments in R.
Another classic: you correct some data manually in the raw dataset (very often a.csv or .xlsx file). For example, when dealing with data on people, sex issometimes “M” or “F”, sometimes “Male” or “Female”, sometimes “1” or “0”. Youspot a couple of inconsistencies and decide to quickly correct them by hand.Maybe only 3 men were coded as “Male” so you simply erase the “ale” and go onwith your project. Stop!
Write it down!
Write a couple of lines of code that does the replacement for you. Not only willthis leave a trace, it will ensure that when you get an update to that data inthe future you don’t have to remember to have to change it by hand.
You should aim at completely eliminating any required manual intervention whenbuilding your project. A project that can be fully run by a machine is easier todebug, its execution can be scheduled and can be iterated over very quickly.
Something else that you should write down, or rather, let another tool do it foryou: how you collaborate with your teammates. For this, you should be usingGit. Who changed what part of what function when? If the project’s code isversioned, Git writes it down for you. You want to experiment with a newfeature? Write it down by creating a new branch and going nuts. There’s somethingwrong in the code? Write it down as an issue on your versioning platform (usuallyGithub).
There are many more topics that us disciplines of the data could learn fromsoftware engineers. I’m currently working on a free ebook that you can readhere that teaches these techniques. If this postopened your appetite, give the book a go!
Hope you enjoyed! If you found this blog post useful, you might want to followme on Mastodon or twitter for blog post updates andbuy me an espresso or paypal.me, or buy my ebooks.You can also watch my videos on youtube.So much content for you to consoom!
Buy me an Espresso
To leave a comment for the author, please follow the link and comment on their blog: Econometrics and Free Software.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: Software engineering techniques that non-programmers who write a lot of code can benefit from — the DRY WIT approach