Skip to content
Episode 41: Conversations with an R Jedi episode artwork

Jun 21, 2016 20 min

Episode 41: Conversations with an R Jedi

from Naturally Speaking Hosted by The Naturally Speaking Editors

FEATURED: Our most visited post of 2016 – In this episode of Naturally Speaking Shorts Laurie Baker speaks to R-guru Hadley Wickham, Chief Scientist at RStudio, about his work to make programming and data analysis in the statistical programme R less painful.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Episode 41: Conversations with an R Jedi

0:00 20:56
of matches

Transcript Auto-generated, may contain errors

Naturally speaking. Hello and welcome to Naturally speaking Shorts. I'm Loey Baker and it's my great pleasure to welcome Hadley Wickham, chief scientist at RStudio, and developer of several famous and widely used R packages, including Ggplot2, Deplyr, TidyR, and Ggbiz. So how do you describe R and what you do to people outside your field?

I hope the comparison is not terribly good at that, but it's becoming increasingly easy. As people now know, I've heard of data science and big data, and so what I say now is I write software that helps people make sense of big data. I read that you come from a family of statisticians. What was it like growing up in that environment?

Yeah, it's kind of funny. So both my sister and I, like, full-on statisticians and my dad is in D and he just eliminated things we didn't like and ended up in statistics. I think one thing that did really influence me was just because my dad had a dentist in the US. That would just seem like an obvious thing to do.

Most people don't think I'll go to do it at all, let alone in another country, but if your parents have done it, it seems like a really normal thing. Did you ever think that you'd be a teacher? Not really. I've always like helping people.

I had this experience in medical school in our population health class. There was this one assignment required you to do this age-weighted, get estimated Excel, and I was the only person in the class. It was so obvious to me how you did it, and then I basically taught everyone else in the class how to do it. I always enjoyed that kind of stuff.

I like helping people learn new things and solve new problems. Why should people use R and what role does it serve in the wider group of programming languages? I think R is a really great environment for making sense of data. Some people can call R a domain-specific language, but I don't think it's true.

It's a general-purpose language. You can use R to do anything, but it's particularly well-suited for data analysis. Partly, some of it's by design. Missing values are really important in statistics and data analysis, and that's baked into the language, in most states, not the case in any other programming language.

But it's also part of it is just a product of the community. There's lots and lots of people using R to do data analysis, so the chances are that whatever you want to do, someone's already done it before, and that makes it much easier. I think that's like when you compare R to other programming languages, or we compare programming languages in general, the technical merit to the language, only if there's only a small part, that's really the community of people who have built up around the language. The vast majority of languages, you can use them to do anything, but it's just kind of like, are you fighting?

Do you have to do everything yourself? What can you rely on what other people have done already? The great thing about R is if you're doing data analysis, then lots and lots of people have done lots and lots of stuff before, and you can use some of the 8,000 packages on Crayon to get what you want to be. As a developer, why did you choose to work in R instead of another language?

It's really like, part of the reason was when I started 12 years ago, there really was only one game in town. There were no other programming languages that were doing data analysis. Another product was that I did do my undergraduate at the University of Auckland, which is the home of R. So obviously, every statistics course that Auckland pretty much uses R, so it's exposed to it from a pretty early age.

There's certainly a lot of frustrating things about R. But the more I learn about the language, the more I appreciate the original vision of John Chambers and Rossi Haka and Rob Gentleman, just in terms of, I think they're being incredibly prescient. Many of the things that are becoming super popular in programming languages now were built into R, like these things like immutable data structures and generic functions. The goal of R has never been to be the be-all and end-all.

The goal of R has always been to glue together different pieces. The original goal of S was basically to help people glue together these C and 4 transcripts. So I think that that's really good, because these days you can't do data analysis and isolation. You're always talking about web APIs, or maybe your data is in a database, or maybe you need some high performance code from C++.

So R is really great at bringing all those things together in one place. The other thing I think is really neat about R is just to create a flexible language so you can write new tools, you can invent new ways of interacting with data. That may be, that sort of introduces maybe some ambiguity and a little bit of magic, but most of the time allow you to get your ideas out of your head and into the code. So a lot of your work focuses on making data manipulation, visualization, and writing code a lot easier and fun.

Why are these areas so important to you? I think some of it was sort of motivated when I did my PhD at Iowa State. My assistantship was doing consulting, so I would, PhD students from other departments would come to me with their statistics problems and I would try to help them. One thing I discovered there is that it seemed like the majority of my efforts were not kind of statistics.

They weren't fitting models and doing regression diagnostics. They were actually just getting the data into the right form to be able to do analysis. And that made me realize this is a huge struggle that being a little formal research work done on. But it's a huge roadblock.

I found I could often look at a data set and say, I know how to put this in the right form. And at that time the tools to get it in the right form were pretty challenging. And that led me to create the reshape package. But I could look at a data set into it, but I couldn't explain what I was doing and I couldn't really teach it very effectively.

It took much longer for me to realize, oh, there's actually these really simple principles. Now that I know what they are, I'm like, why are they taking six years to figure it out? But this idea that you put your variables in the columns and the observations in the row, it does not seem like it should take you five years to realize that. But I think these things that you're sort of so deeply internalized can be really, really difficult.

And I don't think I'm unique in that. I think most statisticians will probably internalize that, but to pull it out and express it explicitly and build tools around it. And so that's what led to the development of the tidy package and then just this framework of tidy data and thinking about how do you get whatever data you've got in your form that's going to make your life easier. And so I really like that aspect of empowering people to do stuff with their own data.

And there's just so many people collecting so many interesting data sets that it's currently, it feels like it's too hard for them to get the understanding out of their data. And I really enjoy making better tools for that. And why is it important for people to write tidy and reproducible code? I think it's really important for good science.

Code reproducibility is so much lower standard to be able to repeat an entire experiment. But if you can't start with the same data and get to the same conclusions, you're building on a really, really shaky foundation. And I think that's important for science, obviously. There's a similar thing in industry writing code and production where you've read a model or something and it's going to run every day now and you want to make sure it's going to run correctly every day and it's not just going to break or it's not going to suddenly give you the wrong results.

So I think that it's really, really important because it gives you this solid foundation. So that's a reproducible code. I think it's also important to think of code as a medium of communication. Because it's fine.

If your code works now, but you can't understand it, the chances are, some of the features, some of the requirements are going to change. You need to modify it. And if you don't know how it works, then the chances are when you modify it, you're going to break it. You might not realize that.

So I think that that's really important. So your code should not only be correct, but you should be able to read it and kind of understand the reasoning behind it. And that's really hard to do, obviously. And often requires it.

Some of it I think is just like when you're writing a paper. Like you understand to write a paper, you have to rewrite some of it again and again and again until you can figure out how to explain things clearly. And sometimes you need to do that with code as well. You might have to rewrite it a few times before you get to the essence that really conveys what it's doing.

You work with some really cool data sets for your teaching. Where do you find them and what are the most interesting and unusual ones that you've come across? A lot of it is just, I don't know, I kind of keep my eyes open. I read a lot of blogs and I'm pretty active on Twitter.

So I kind of see things and kind of note them down. I work a lot with US data because I live in the US and that kind of relates to students best. One problem with the US is there's a lot of privacy laws which is good and bad. So one thing I discovered recently is there's some, like, datasets from Mexico.

And there's a dataset from Mexico that is basically every death in Mexico along with the location and the cause of death and a lot of demographic information. So there's no way you could ever release that in the US because if you know where someone died and when someone died, you can find out what they died of which could be potentially kind of embarrassing or overdose or something. And so that's kind of like, I don't know, it makes for a much more interesting dataset because you have this much more fine grained spatial temperate data along with the cause of death. So that's kind of makes it fun to work with.

On the other hand, it's kind of like used to worry a little bit about the privacy concerns. The other one that I found recently that's really fun to work with is a sample of all emergency room admissions in the US. And the thing that's really neat about it is every, as well as these codes to tell you what caused the, you know, what was the problem and what body part and what products are involved. There's also this free form text and there's just like so many funny little stories.

Like one of them is like ran out of a burning house. House was marijuana grower. It's like, there's just really funny little stories in these 75 characters. And that's like really fun.

Like it's fun to kind of, it's a fun challenge for like learning regular expressions and string manipulations, but it's also just kind of fun to look at this data. In ecology, many of us work on dynamical systems, including population dynamics, animal movement and disease spread. And these systems develop through time and space and depend on the state of the system in the previous time step. So in the case of animal movement, the animal's next location depends on its previous location.

In programming, we often use four loops to model these data. I was just wondering if there's more computationally efficient way to model these recursive data. Yeah, I think that's like pretty much the canonical use of a four loop. It's because it's very, because the next state depends on the previous state, you can't use either built in apply or map functions.

One technique that is possible to use sometimes is to like solve the recurrence relation explicitly. And sometimes you can get a close form relationship, but it's often pretty challenging to do that. I think the other thing that's worth looking at for that problem is like writing the loops and see your C++ instead of R. And I think for simple stuff like that, you could often spend like a couple of days learning some C++ and then have code that runs much, much, much faster.

And now with the RCP package, it's really, really easy to call the C++ code from R. RStudio also has a lot of tools, so you can get some text hiding of C++ and auto completion. And I think it's worth investing at least a little time in to see if that's worthwhile, because you can easily get like 100 fold speedups to code like that. What do you have coming up that you were really excited about in the next few years?

This is three bigger things that I've been working on. So one is this framework for dealing with lots of models. So the basic idea is that in the data frame in R, you can have a column that's a list. And you can put anything in a list.

And so that means you could have, for example, a column that is a list of data frames. So it seems kind of crazy to have a data frame that contains a list that contains other data frames. But I think that actually often makes a lot of sense in some cases, because you'll have like, if you want to fit lots of models, you want to break your data up and fit one model per individual or per country or something, you can kind of think of having one data set per country. And you'll have one data set per country and you'll have one model per country.

And the chances are you won't just fit one model, but you have multiple models. And then you might have residuals from those models and other summaries from those models. So putting them in, it makes a lot of sense to put them in a data frame, because then you keep like the data and the models and the model summaries all connected together. So rather than having these kind of vectors that are just floating around in your workspace, and when you subset one, you have to remember to subset every single other one to make sure they stay in sync.

You can just put them in a data frame. So to get this all working is involved like some work in DeepLire. The tidy R package now has these nest and nest functions which allow you to go from a regular data frame to a data frame where some of the columns are nested in this list format. And then this new package could per which provides tools.

They basically sort of generalize the apply functions in base R, make them consistent, fill in the view gaps and a few nice shortcuts. And that I think is a really nice framework for working with these collections of models. And the other package is really helpful is Broom by David Robinson, which makes it easy to extract tenured models into tidy data sets. And once you've got a tidy data set, then you can visualize it and then use all the other tools that you have.

Another thing that I've been working on is this sort of non-standard evaluation, which is one of the things that makes our special and unusual and as well, kind of powers Ggplot and DeepLire. Basically you type the name of the variable and Ggplot it. DeepLire kind of looks inside the data frame instead of using the normal rules. So one problem with that is that if you want to write a function, for example, where one of the arguments to the function is a variable you want to plot.

And you want to use Ggplot too. It's kind of surprisingly difficult to do that for yourself. So one thing I've been trying to work on is a consistent framework that provides tools that you can write functions that work like Ggplot to and DeepLire. And it's not super complicated.

Where should undergraduate and postgraduate college students focus their time in learning programming? Generally, I think you're better up being really good at one thing rather than being mediocre at a couple of things. Or in other words, it's better to master something and then move on and master something else rather than trying to learn two things at the same time because there's sometimes kind of a lot of confusion that can occur. So I think if you're using multiple programming languages, it's worth saying, I'm going to invest my time for the next year and I'm going to be really good at one of them.

And I think like for R in particular, I think one thing that's really valuable is learning this functional programming tool. So basically instead of writing for loops, use L-apply every base R or one of the map functions from per. Some of it is you're sort of moving up this tower of abstraction. So rather than writing a full loop yourself, you're using a function that writes a full loop for you.

And that takes a little while to get your head around. But in the long run, it makes it easier for you to write code and your code's more likely to work the first time. And the whole class of areas, you just can't make any more because someone else has written that for you and that done it correctly every time. So I think that that's very valuable.

I think learning how to write functions is really important. And learning how to write, like once you've kind of started master functions that sort of work, the next step is to kind of think when you write a function, well how might the inputs to this function cause it to fail and then kind of proactively check them and throw errors and stuff. So I think like functions and like for loops and iteration and functional programming, I think those are like the main tools for getting a replication in your code. And a really good rule of thumb is as soon as you have copied something, copy and paste something more than twice.

Like if you've got more than three copies, it's time to start thinking like how can I do a real loop application? Like can I write a function? Can I write a full loop or use a map function? Once you get good at that, you'll start to get to situations where like writing a function or writing using a map function doesn't get a little duplication, doesn't solve that problem.

And then like that's, that means it's a signal to kind of learn something more to learn some new programming taking that does that you get a relatupcation. Because anytime you've got duplication in your code, that's just a chance for like bugs to come in because you know you change one and you forget to change another one, you get inconsistencies and gets really frustrating to work with. So now we have a quickfire round. What is the strangest piece of fan mail that you received?

I think that the weirdest thing that I've asked is someone asked me to sign there a wooden giraffe. So I've had a couple of people like request a signed photograph for me because they want to give them a sort of gag gift to someone at their wedding. That's quite funny. And do you ever receive hate mail?

I suppose you mean make people's lives a lot easier so it's probably infrequent. Yeah, I don't know hate mail, but some people do feel like pretty entitled. I make all this stuff for free, but some people are pretty high expectations about what I should be doing for them, but by and large you're just kind of ignoring all those people. That's fine.

If you don't like it, I'll give you your money back. So if you weren't a software developer, what would you do? I don't know, I think my hobby is like baking cocktails, so I think maybe like a bathtub or a baker or something. Excellent.

Well, thank you so much for joining us, sadly. It's been a pleasure. Thanks for listening to The Naturally Speaking Podcast. If you've got any ideas for future shows or want to get involved, then please contact us.

We'd love to hear from you. Our email address is naturallyspeakingpodcast.com. Naturally speaking is a podcast from the Institute of Diversity, Animal Health, and the University of Glasgow. For more information you can find us on Facebook.

You can find us on Twitter at theNSPodcast.com. And you can download future episodes and find more information at our blog at natterysspeakingpodcast.webpress.com.

Never Alone Infertility Morgan and Madison Never Alone is a podcast for women navigating infertility, loss, and everything in between. We’re two friends whose paths intertwined during our own walks through infertility, and we understand how isolating and overwhelming the process can be. From the uncertainty of IVF to the heartache of miscarriage and the patience required in moments of waiting, we’re here for you.Join us each week as we have real and raw conversations about the challenges and emotions of walking through infertility and loss – both from our own experiences and speaking with other women. We’re glad you’re here, and hope you feel seen, supported and most of all know that you’re never alone. The Regular Man Podcast The Regular Man Podcast Welcome to the Regular Man Podcast, where we celebrate God's gift of masculinity in the life of the Regular Man. Here, there are no pastors, no pretense, and no pandering. Just interesting conversation, realistic expectations, and practical advice for the regular man. Steve Cruz is the host of The Regular Man Podcast, created specifically for the regular, hard-working, blue-collar man. He’s a husband and father who believes men should embrace God’s gift of masculinity and live a life of Christian virtue. Steve learned the value of hard work, integrity, and respect through an upbringing of cars, construction, and a cadre of strong hardened men. As a military veteran and law enforcement officer, Steve understands the value of speaking directly and plainly on subject matters that are, far too often, nuanced to death by the modern-day Christian Church. As the host of The Regular Man Podcast, Steve has down-to-earth

Frequently asked questions

How long is this episode of Naturally Speaking?

This episode is 20 minutes long.

When was this Naturally Speaking episode published?

This episode was published on June 21, 2016.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Naturally Speaking episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.