Hey, hey. What's up, crew? Welcome back. Or if this is your first time listening and I anticipate this episode may attract a few first-time listeners, then welcome. It's nice to have you tuned in. I'm your host, Aaron Fifield.
For this episode, I got to speak with Hugo Bown Anderson, who, as far as I know, has never traded a single share in his life, but he is a brilliant programmer. More specifically, Hugo is a data scientist at Datacamp, which is an online educational platform for learning data science. And Hugo also hosts Data Framed, which is a podcast by Datacamp. So I guess the idea for asking Hugo to appear on this episode was essentially... to chat about learning to code. Because for some traders, having the ability to write code can have great advantages, such as having the ability to collect statistics on market behavior, perform research in a robust data-driven way, visualize large amounts of data, backtest and analyze trading ideas, and also to go as far as implementing algorithmic strategies, et cetera, et cetera.
As some of you will already be aware, more professional trading firms and even broader finance-related positions are requiring applicants to have some programming skills. And this spreads across all industries too, which I guess should be no surprise considering a recent IBM study revealed that about 90% of the world's data has been created in the last two years alone. So some of the topics we dig into during this episode include when should someone actually consider learning to code, how to determine what's relevant for you, how long until you'll become somewhat fluent in a programming language, and how to work with new data sets, what to be wary of when using predictive models. And just for the fun of it, I also asked Hugo, as someone who's a non-trader, how he'd go about creating a basic quant strategy. Now, if you like the episode, and if you're keen to learn more about this type of thing, I do have an affiliate link for Datacamp, which you can use.
That is chatwithtraders.com slash datacamp. Unfortunately, you won't get a special discount for using this link, but you will be helping to support Chat With Traders. You know, I myself, I've been a paying subscriber and a user of Datacamp for the past couple of years. So I've got no hesitation recommending this resource to anyone else who wants to learn how to code. So again, that's chatwithtraders.com slash datacamp. Thank you very much. And I think that's everything. So here is Hugo Bown Anderson for episode 166.
Well, obviously, most of the things we're going to be chatting about on this particular podcast are going to be around programming. But that aside, I'm interested to hear a little bit about your own personal background. I know you're at DataCamp now. What did you do leading up to this? Yeah, that's a good question because it does – my passage to programming, I think, is one of many routes that people take these days. So my background is actually in science and humanities, science and arts. So I did my undergrad. I did a combined Bachelor of Arts, Bachelor of Science, and I majored in – I had a double major in pure and applied maths. So I finished the science part, and then I went to grad school. So this was at University of Sydney. Then I went to grad school to do a PhD at the University of New South Wales in pure maths. Tried to finish my arts degree at the same time, but they kicked me out because purportedly you can't be enrolled in two different degrees at different institutions, and I wasn't aware of that. So I got kicked out of my second undergrad.
When I finished my pure maths PhD, which was in an area called algebraic geometry, I realized I wanted to get more into the applied mathematics, applied math, or applied maths, as we say. And so I actually started working in biology. I started a postdoc in Germany, in Dresden. There's a big Max Planck Institute for cell biology there, doing applied mathematics, working in a biology lab, thinking about cell growth, cell dynamics, the physics, the biophysics behind these types of complex systems. And this is the first time I started working with a lot of data. And I actually started calling it data then because people...
You know, my American colleagues thought I was talking about a surrealist movement when I said data too often. But I started working with a lot of data at this point, and biologists kept asking me the same questions about statistical tests, about cleaning their data, about learning R and Python and this type of stuff. So I essentially taught myself data analysis, statistics, a bunch of machine learning in order to stay ahead of the curve working in these fields. I then wrapped up my postdoc. My postdoc boss accepted professorship in the U.S. at Yale University in New Haven, Connecticut. So I spent two years there doing research.
And I was doing a lot of teaching, a lot of education there. Because biologists kept coming and asking me the same questions, I started to run what I called practical statistics and data science workshops. At that point, I met the DataCamp team, who was around eight or nine people then. And they were really trying to get moving on their Python curriculum. So, you know, they put the hard sell on me and I jumped on board to come and build kind of online education systems to teach in browser experiences for learning data science, essentially. So that's really how I ended up at DataCamp. A lot has happened since then, though. Okay. And how long have you been with DataCamp for now? So I've been here for two and a half years now. And we've grown from 10 people to 75 in that time and moved from teaching...
essentially R and Python to now teaching R, Python, SQL, command line stuff, Git, which we might get to, which is version control. So kind of versioning your code for reproducibility and spreadsheets now as well. So we've got our first several spreadsheets courses, which we're really excited about. Yeah, I did say that, which is kind of interesting. Just going back to your background, it kind of sounds as though, and I might be wrong here, but it sounds as though you kind of took it upon yourself learning how to code. Like it wasn't something that was expected of you in your role. You just sort of saw it as something that would be very beneficial to what you were doing. Would that be a fair statement?
Yeah, that's absolutely right. And so you hit the nail on the head. My job was to do applied mathematical modeling and whatever that may involve. And I realized at that point that it actually involved learning a lot about the data I was trying to model, learning a lot about the data generation process, thinking about it statistically. And the way we do that these days is using programming languages. I mean, you know, essentially people used to flip coins and use pen and paper. But if you want to deal with large amounts of data, complex data from different sources these days, you need to do it.
using using programming so that's i was i did self-start in that way i think that's a lot of people's journey to coding as well i mean a lot of a lot of data scientists and programmers and coders don't have computer science or software engineering degrees they did learn on a job i'm a little bit surprised that you didn't learn any of this type of thing with the studies you were doing at university yeah yeah that's a that's a good point and i think that definitely in pure maths i did a bit of um a small amount of matlab and a bit of a language called maple, but very minimal in the applied math stuff. It was really more focused around differential equations, real analysis, which kind of leans towards applications. But this is something that, you know, it's a big question in how we're teaching people mathematics and statistics now, in that maybe we haven't necessarily done our job with too much of a focus on calculus, for example, and not enough on, you know, getting people analyzing data and simulating stuff and looking at distributions and all of this fun stuff. Well, let's kind of turn the conversation now to maybe help.
some of the people who are listening to this podcast. So most of the people naturally who tune into this podcast are traders and investors, et cetera, who are participating and betting in financial markets. When or why should someone consider learning to code? So I think firstly, I want to say that one of the alternatives... is spreadsheets. And I think spreadsheets are amazing in a lot of ways. I mean, tens of millions, if not hundreds of millions of people are using spreadsheets to do their job. I do think there are challenges involved in using spreadsheets. And I'll kind of discuss this in a variety of ways, but there are even online resources as to the types of mistakes people have made with spreadsheets. We don't have access to all of this information, but I've met a significant number of people in finance working for big companies who've lost a significant amount of money due to spreadsheet errors, for example. So if you're working in spreadsheets, I definitely think it can serve you well.
But essentially, in today's job market, and even for a lot of traders, if you want to have an edge or a differentiating factor for your practice and the type of work you do, I think programming is definitely a very viable option. And I think, you know, we see now companies and organizations like Citigroup and Bank of America and JP Morgan, I mean, all of these places put Python first now. And I think, who was it? It was Robin Wigglesworth, right, in the Financial Times a few years back, wrote that. It used to be traders who were first-class citizens of the financial world.
But it's technologists now who are the priority and people who can work robustly with large amounts of data coming from disparate sources. And the way to do this now is with programming. Whether that's Python or R or databases, I'm not really... You know, we can have that conversation, but I think the real conversation needs to revolve around whether you can do this stuff in a GUI, a graphical user interface, which, you know, Excel or spreadsheets is an example of, or you want to do it in a programming language. And if I may, I think I'd like to just give you a few things about programming that I think really help with this type of process. The first is that it's reproducible. So, you know, when I'm using a spreadsheet or some sort of GUI, I do a bunch of clicks and I can send you my results and you can't reproduce what I did. All you have is the results. Whereas if it's text-based, I can share it with you.
straight away um so it's reproducible in that sense you can reproduce it on your own operating system it's readable um and on top of that it's what what we call diffable so you can see the diffs and what i mean by that is if you write some code to automate a process and then you change it because it's plain text we can see exactly what the changes um what the changes are that have been made and and we can see that the workflow and the process there um On top of that, one of the strongest aspects of coding and programming is that it automates stuff. And I've hinted at this throughout what I've said already. But if you're doing pivot tables a few days a week or whatever, and you can write three lines of code that do it for you, that's just common sense to me. And pointing and clicking isn't scalable in that way. So I think these provide general arguments for why.
why programming is good we can talk about why open source programming in particular is good depending on on on how you'd like to frame the rest of this conversation as well okay yeah cool i'll make note of that and we might get to it um just right now i'd like to ask you you spoke there about spreadsheets and i presume you're talking mostly about excel right Yeah, or Google Sheets. I mean, these are the two big ones that I know of. Okay. So it's probably fair to say that there's a large majority of people who are very familiar with Excel or Google Sheets and spreadsheets in general. When is it beneficial for someone like that to actually take the next step and learn an actual programming language? I think it's beneficial when they're doing the same stuff all the time. So doing pivot tables was one example. But if someone finds themselves in Excel on Monday, Tuesday, Wednesday with different data sets doing a similar thing, it makes sense for them to learn.
To write some code in order to automate that, I think. when it wouldn't be beneficial for them is if it's like a one-off job doing a bunch of data entry, for example. But in terms of automating, time-consuming, road tasks, it definitely makes sense then. It also makes sense when they want to do more robust modeling and understand why their models are saying what they're saying. I mean, you can model stuff in Excel. Excel is relatively powerful with basic modeling. But if you want to build models that you can dig into and find out why they're doing what they're doing, I think both Python and R are exceptional at this as well. Another aspect, I think, is if you want to share.
your workflow with other people as i said you're doing a lot of pointing and clicking um collaborators and colleagues can't really get inside into what you're doing whereas if you're working on a team of people writing code together you can see exactly each step especially when it's well commented right which i'm a huge fan of so these are these are several reasons um i would also say one of the problems with spreadsheets is that your data source and your logic and your functions and formatting are all intertwined. I mean, you see nightmare spreadsheets where people highlight a row in order to mean something. And I think separation of data from this type of logic and from formatting is incredibly important to do robust data analysis and data science in the end. I like that last point you gave there. I hadn't really considered that, actually. Now, you mentioned Python and you've mentioned R. Obviously, there's many programming languages which exist and which are open source, which anyone can access easy enough. How do they decide?
which one they're going to focus their time and energy on to learn as a trader? Yeah. So I think this is a deeply personal question. And I need to be as sensitive as possible because there are very strong sides on each side of the argument. But I think – so if your main interest is in doing statistics and exploratory data analysis and data visualization and these types of things, R is a really good place to do this.
Particularly in the way there's a set of packages referred to as the tidyverse, which by Hadley Wickham, who's from New Zealand as well. And he and his colleagues have developed this kind of set of packages, which allow for very, I think, thoughtful exploration of data and data visualization. And one of the great things is they've written these packages that you can, you write code kind of the way you think about the data. And they refer to not as functions in these packages, but as verbs. So you'll filter something, then select something, then arrange something. And it's very easy to read in that respect.
So for these types of things, I think R is incredibly strong. I do think Python, if you want to do machine learning on large data sets in production and serious automation of tasks, which I think perhaps is more what traders are interested in, Python is probably a win there, particularly in the machine learning and deep learning space, although R is seeing strides there also. I would also add that I personally... The barrier to entry for Python is slightly higher, I think, but I like to refer to Python as the Swiss Army knife of programming languages because rarely will it be the best tool for the job, but you can do anything with it. Yeah, okay. So what are some of the limitations of R which you're going to be able to do with Python? Obviously, you mentioned machine learning there. Is that kind of the key one or are there a few other things which separate the two?
Yeah, I think probably productionized machine learning models, as opposed to, you know, getting a machine learning model working on your laptop, which you can submit to, you know, an online machine learning competition, getting something productionized. Python is definitely, so getting it, you know, coming in with your streaming data source and getting all the results out there or, you know, feeding it into another API or something like that. Python is a lot stronger. I do think, though, we are seeing massive strides on that side. But in all honesty, I think the deep learning packages...
are probably more mature on Python and the way that they all interact as well, kind of the way these packages all interface to each other. So I think those are the places that Python win. Having said that, the open source software community on both fronts, they're moving so much more quickly, so quickly that what I say could be outdated tomorrow. Yeah, well, that's where we live in, right? Exactly. So let's say that someone wants to begin learning R or Python. They've never looked at a programming language before. They've never written any code, maybe some basic Excel. Where should they even begin?
to start out, like what are some of the best first steps you can take when learning to code? Right. So I'm definitely biased. I would say Datacamp is a fantastic place. And I'll tell a few others afterwards. But one of the reasons I initially joined Datacamp and one of the reasons that I'm still here is Datacamp has helped to lower the barrier to entry for so much of the learning process, in particular, the initial steps, right? So when you come to Datacamp, you don't need to install anything locally on your machine. We spin it up for you. It's all in browser. You can start learning and writing Python code straight away and feel functional straight away. And that's one of the most important things for our learners and for learners in general is to feel like you're doing something, not like spending the first 90 minutes.
installing something and then having an error returned, right? Exactly. We've all been there, right? So I definitely think that's very helpful and motivating in all honesty. I also think, so for people interested in finance, we have finance curriculum here. I also think, you know, you've spoken with Eve Hillpish in the past and his quant stuff is also his Python quant stuff. I think he's really, really interesting. And he has online courses for those very interested in the finance stuff.
But also, depending on whether you've chosen R or Python, figure out where you want to write your code. And what I mean by that is, you know, there are these things called IDEs, integrated development environments, and that sounds like a mouthful. All it is is like a piece of software that you'll click on and open, and it's where you write code, save it, execute it, this type of stuff. For Python in data science, and for you can do R and Julia and all of these things in this infrastructure as well, there's an incredible project called Project Jupyter, and they're Jupyter Notebooks, and now they're JupyterLab, are really good for getting started running code, because it's kind of like an interactive data science notebook, essentially, where you can write text, store images, videos, write code, execute it in your browser, and it's all going straight away. On top of that, on the internet, there are lots of galleries of interesting Jupyter notebooks that you can get up and running with. There's one actually called a gallery of interesting Jupyter notebooks, actually, where there's a whole bunch of finance, among other things.
The two other things I'd really suggest is to read as widely as possible. You know, blogs. We've got, you know, a lot of blogs and tutorials on our Datacamp community. Fast Forward Labs has a lot of great stuff on data science. ODSC, which is a conference, the open data science conference. Got a lot of finance stuff as well. And wherever you are, go to meetups and meet people and hackathons. Because in all honesty, doing this after work, like in your own bedroom, you know, it can get hardcore. And, you know, we've all been there. But seriously, coding with other people and pair programming, the learning curve there, makes it a lot more fun. And the last piece of advice I'd probably give.
is try to get your day job to give you time and or money to do this. So for example, if you want to subscribe to DataCamp, we've got a whole bunch of free stuff that you can check out first. Tell your boss that this is your learning stuff that's going to help your job. We'll try to get Friday afternoons from your work to invest in your future to make you more efficient at your job and work at that because wherever you're working is going to win in the end, in all honesty as well. Yeah, that's a really cool tip actually. I like it. Yeah, and obviously I...
recommend data camp as well because you know that's one of the reasons i got you on i mean i'm a paying subscriber of data camp uh i'm always on there checking out new courses and material etc so i mean it's certainly helped my learning curve and learning how to program as well so obviously that's why it'd be really cool to get you on the podcast because you know i've seen some of the courses which you've been a lecturer of Appreciate that. And I would be, one thing I'd like to add is, as you know, I host a podcast called Data Framed, which we put out weekly episodes for Data Camp. And I'd urge you to listen to that as well, because, you know, the premise of this of Data Framed is to interview thought leaders and working data scientists from all over the place, whether it be finance, tech. The most recent one was health. It was data scientists from Doctors Without Borders. So I think this type of stuff will definitely give you insight into how people think about data science and what's kind of really current and modern in the space at the moment. Yeah. And I said this to you when we spoke.
which it was a few months ago when we first made the connection, but that... You know, I thought it was really cool when you guys actually started a podcast because prior to that, I hadn't really found any great, and I'm not saying they don't exist, but I just hadn't really come across any other sort of really cool podcasts on programming. So I was kind of excited to see yours pop up. Yeah, I appreciate that. Now, obviously, there's a lot that you can explore. There's plenty of subjects and areas that you can go into. And I guess it's very easy to get sort of led astray in some ways when you're new to programming because you don't really know what's relevant. You don't really know what you should be learning. You know, as a trader who's...
learning to code, you're mostly going to be wanting to learn to code for the purpose of exploring data, getting some sort of statistics and probabilities, visualizing data, backtesting certain strategies and trade ideas and market behavior. How do you determine what's relevant? Like whether this is something I should be learning at this stage early on in my, you know, while learning to program, or is this something more of an advanced topic that would be nice to know a little further down the track? Like how do you determine what's relevant? Yeah, that's such a good question, Aaron, because on top of every, all the good reasons you've just said, if you go and look at like, the amount of python packages out there for you know time series and data analysis and this type of stuff it is so overwhelming that you can you know you can just freak and probably like run away forever so i definitely think that's this type of conversation is super super valuable and the way i generally approach this is take a bunch of courses sure and get introduced to the main players in in the space of python packages um and for those who haven't heard the term package before a package is really something that's developed that's you know a set of functions and utilities that will help you to do stuff right
And of course, you'll come across the main ones like Pandas, which is seminal now and absolutely undeniably necessary for all data science, all financial data analysis in Python. And it's really a way of structuring data in order to facilitate analysis. So the basic unit in Pandas is a data frame, which kind of looks like a matrix of numbers or whatever with column names, essentially. But that having been said, besides learning courses and becoming familiar with the main players in the space, I always advise learners to do a project on something they're interested in. And of course, in this case, if you've got...
a trading project that you want to do, definitely jump in. And what will happen then is you'll learn all the tools you need for that particular project, right? And you'll be drawing on a lot of stuff that will be relevant in future for you, whether it's, you know, Pandas or some nice visualization libraries, or if you start doing statistical modeling packages around that, or time series analysis, or even, you know, machine learning, right? You'll make scikit-learn among other packages then. So if you do a project from go to woe, I'm not saying it's going to be easy. It'll actually probably at some point be super frustrating. But if you're invested in the goal, right, you'll...
you'll learn what you need to. The last thing you want is to do a project that you're kind of half-assed about, right? And you get to a roadblock and you're like, oh, I don't care about it so much. So if you pick something that you're super passionate about and get to that end, and by the end, I don't only mean getting out a result or an actionable, I also mean talking about it with mates or giving a talk about it at a meetup or writing a blog post about it. I think these types of things definitely get you far more acquainted with everything you've used. And on top of that, it doesn't only have to be a trading project. I mean, I encourage people to get out of their professional zone when doing this. So I met a student who...
um is all about their fitness tracker um you know it wasn't a fitbit or something something like that and i was like dude just get your get all your fitness tracker data um and you know analyze that and see if you can predict you know what's what's going to happen in the next three months and that type of stuff and they learned a whole bunch of uh cool tools around that if you're a foodie you know you can scrape yelp reviews right and do some analysis there um i met a law student who wanted to get into data science and they did a project of um It was something like, I'm going to bugger this up, but it was something like looking at Supreme Court judgments that were handed down and trying to cluster them according to whether the judge was like Republican or Democrat or in the middle or whatever it was, right? So find something you absolutely love and get in there and get into it. You said something really key in your answer there that there are going to be points when you try to step out on your own and do an exercise that it's going to be very frustrating and you're going to get stuck. What's the best thing to do when you do get stuck? Because as you said earlier, if you're just at home and you're trying to work this out, and it's not something you've done before. And like I've had it before and where I've gotten stuck and I've literally been stuck on it for two days. I just can't work out why this won't work. And it just keeps spinning out the same error. And often some of the errors that you get in Python, especially for new programmers, are quite confusing and not always very clear. What's the best thing to do if you're in that situation where you're trying something, you get stuck, you don't really know where to go. Where do you go? Yeah. So firstly, I agree completely the amount of frustration.
you can experience with like really idiosyncratic, esoteric error messages is wild. And I do think open source package developers are kind of getting onto this and they realize that if they want more users, they need to provide more helpful feedback messages. One example is Keras, which is a deep learning library developed by Francois Chalet, who's a Google. And it provides some error messages that are along the lines of, hey, you use this function, but why don't you try this instead, right? So it's as opposed to error, syntax, error, backtrack, whatever it is, right? So if you're getting wacky error messages that you don't understand.
my number one answer to that question is google it google google knows or your search engine of choice i don't know if many people use ask jives anymore but i would i would google it and most a lot of the time you'll end up on a website called stack overflow which is essentially a q a a forum for uh answering these types of questions so if you copy and paste your error message into google a lot of the time you'll find something you know you might end up in a in a stack overflow question that links to an issue on um you know the repository, which is a repository is where you store code for the particular package. And then you need to go through a few other things, but definitely search engines will be your best friend, I think. Yeah. Yeah. Agreed. I know this is going to depend kind of person to person. It's again, a very personal thing, but I know there will be some people listening to this who are kind of wondering, what is a reasonable timeframe for someone to learn the basics of programming? Let's say in Python, for example. It depends what...
what you want to do um but if you want to get a sense of how to analyze data of how to get a result um and of the basic packages you need in a space and you work full time let's say you've got a couple of nights a week let's say five hours a week or something like that i'd say six months you could get get up and running pretty well Yeah. Maybe less, but I would say six months is pretty good if you're efficient and scrappy and resourceful yourself. But it's hard. Just always remember that it's hard work as well. And it's, as we said, frustrating. It can be infuriating at points, in all honesty, particularly when you work full-time and you've got a family and all this stuff. You're taking a lot of time out of your full-time life in order to learn something new. But just remember that this has a pretty serious payoff as well. And I do think that...
You know, we all know this, right? But computation is becoming so much more important and the ability to code and have these conversations will make you a far more valuable member of the workforce, for example. And I think, you know, there are really big wins to be made if you put in the time here. Yeah. And on the flip side, when you do get some code working exactly how you wanted it to, it's also very rewarding. Like, it feels awesome. It's beautiful. Couldn't agree more. Yeah. I'd like to ask you a couple of questions, Hugo, about...
maybe what questions to ask your data. So I think this is interesting because obviously you're not a trader. You don't come from a trading background. You come more from a, you know, you're a data scientist. I'm interested to sort of hear a little bit about your thoughts about, you know, how you perceive some of the challenges that traders may come up against.
So I guess maybe one of the first questions on this topic would be, how do you think about what questions to ask your data? Because essentially that's what you're doing as a data scientist. You get a data set and then you're trying to use that to solve problems or to answer certain questions. You're trying to find answers within the data. How do you think about what questions to ask that data? Yeah, that's a really interesting question. And there are actually several ways to approach this. A very common way is actually to flip that entire process. And a lot of the time you ask the question first before you even have the data.
One great example is, I think, you know, in kind of the modern age of data science emerged in tech, right? So LinkedIn is a really good example where they, one of their first MVPs of data science was they were trying to figure out how to get more people active on LinkedIn and how to get more connections made. And they realized, for example, if you and I are connected on LinkedIn and you're connected with someone else on LinkedIn, then it would make sense for them to suggest that I connect with that other person on LinkedIn, right? And so then they decided that the data that they needed to work with was a network graph of people, a social network or professional network of nodes of people and edges as connections, professional connections between people. So all this is to say that their question existed pre any data and then they figured out.
what was that they needed to look at. So essentially in this framework, the data science question emerges from a real world business question. And then you kind of transition down to this level of data science and data. And then to get the business solution, you transition back up to this world of business questions. Another prevalent example is fraud detection for credit cards where, you know, you can ask a question straight away like, how do we detect whether a particular credit card transaction is fraudulent right and that doesn't involve any data at all and you're like oh okay damn it we need to look into the data and figure out some patterns and do some prediction and wait that's machine learning that's probably supervised learning fraud or not um a binary classification problem and suddenly you're speaking the language of data and data science and you can answer it in that realm and then you transition back up to this business question of you know is this fraudulent or not and the reason i like to frame um at least part of data science this way is because it actually speaks to this huge challenge of getting decisions made in organizations, right? Because people hire data scientists to turn data into insight. But in the end, if you're not making business decisions around it, how do you see the value of it? And who makes business decisions? It's rarely a data scientist who makes business decisions. It's usually someone that they report to or a team of people they need to convince, whether they're C-level or management or whatever it may be, right? So in the end, data scientists need to get non-technical people on board. So the point there is that the question can pre-exist before data is even involved. Having said that, of course,
you know there is a lot of data already existing within tech companies for example so figuring out which questions to ask from them definitely you know exploring it so an example that we're having in in data camp at the moment um is we can look at uh this is something i'm actively thinking about we can look at um learner activity right so we can think about how often someone logs onto data camp how many exercises they do and and how often do they sorry how long the duration of each each visit to data camp right and suddenly If you start looking at this data, you start to see clusters emerge. You see, oh, the students who come on once a week for a long time, and then you see the learners who come on every day for half an hour, right? And so I didn't ask a question there at all to start, but suddenly I'm seeing some sort of segmentation emerge, and you can then ask questions like, oh, maybe would different parts of our product and our platform be more beneficial to some students than others, for example? And I think these two approaches are definitely complementary, and a lot of...
kind of the most thoughtful data scientists, I think, work in both across many organizations, if that makes sense. Yeah, no, it does. And I imagine this is probably quite an important step that you actually do think about what kind of questions you want answered from your data instead of just sitting down and then getting lost in the data. Yeah, absolutely. And as I said before, like...
companies are writing serious paychecks to data scientists, right? So they need to see the value. They need to have their business questions answered and they need it early on as well. I mean, to get a company to invest in data science and analytics as a defining aspect of their business, they need to see value demonstrated early on, right? Hugo, I want to ask you this question. It might be a little bit of a curveball, but I think it'd be a little bit of fun. So just answer it the best you can. And given the fact that you're not someone who is a trader, but let's say,
you were throwing some stock data. Let's just say open, high, low, close data for the stock price. And you were challenged to develop a strategy or to gather some insight from that, preferably a strategy or something. So where you would buy, where you would sell, et cetera. How would you go about that?
Right. So I like this because I actually have, I would have no idea how to do this, but the way I'd probably think about it is I'd get the data, I'd load it into Python in a Jupyter notebook using the pandas package, which as we discussed, I think was, you know, literally developed for these types of financial questions by Wes McKinney. And then I start to visualize it. I mean, I'd literally start eyeballing, you know, the time series of what's happening here to just get a sense of what it looks like. And I think, you know, I don't know a lot about, you know, trading and financing in this sense but when thinking about time series um i suppose the three things that i usually think about are um trend to see if there's any general upward trend or downward trend um seasonality so looking at those types of fluctuations and then whatever random noise is is associated um And then I think after doing that, I'd probably try to develop some sort of basic model to predict what's going to happen in the future to see when to buy and sell. And the two approaches to that, which in time series forecasting, which is essentially what we're talking about, I think, the two, I think, orthogonal approaches to that are some sort of like...
a REMA model, like autoregression moving average stuff. And once again, I don't know whether this is used a lot in trading, but to predict what's going to happen in the future with these time series. The other approach, which is becoming a lot more prevalent, is using machine learning approaches. So getting a whole bunch of machine learning models, having your in-sample training and your out-of-sample testing, and seeing which models do the best there with their prediction, and then basing my decisions around what I predict the movements will look like in the future. And just a fun fact, you mentioned Wes McKinney, who created the Pandas package. He actually works at Two Sigma.
You know, some listeners will know that's a serious, a major quant fund. So, yeah, I think that's kind of cool, actually. Absolutely. And in fact, Tuso, he actually, I think he may have left Two Sigma very recently, but he still collaborates with them. He started his own company called Ursa Labs to develop open source software. But Two Sigma also...
works with and pays a number of other developers of pandas and the open source community so two sigma in open source land they do a lot of a lot of good work to support open source and this is one of the other things you know like we haven't talked a lot about open source stuff but if you've got organizations like two sigma realizing the power of open source software development i mean that's the writing's on the wall if you ask me yeah two sigma also did um what's that website where they run a lot of machine learning competitions uh kaggle Yeah. Yeah. Two Sigma, I don't know how, it was a little while ago, ran a competition on Kaggle as well. Yeah, that's right. And in fact, another fun fact, Two Sigma in the West Village in downtown Manhattan hosted our launch party for the Data Framed podcast in January this year. With the PyData NYC meetup as well. Yeah, okay. Oh, very cool. Yeah. What are some of the common problems that data scientists encounter?
I know that's kind of a bit of a vague question. I guess one of the things I have in mind around that is actually the data itself, being able to clean data, access data, et cetera. Are there any efficient data cleaning methods which... you know you can suggest or even like i said broader common problems that data scientists encounter oh dude this is this is an absolute nightmare this is this is you know a minefield of pain and and suffering i mean there's the old joke that you know 80 of you know data analysis is is cleaning and importing and that type of stuff um i actually saw a while ago there's um there's a twitter account and we should include this in the show there's a twitter account called big data borat and it's like borat's take on stuff and i'm not going to try to do the accent but big data borat says you know 80 80% of time is cleaning data, 20% of time is complaining about 80% of time being cleaning data. So no, you're absolutely right. And even cleaning, missing values, the fact that you'll have data corrupted, the fact that sometimes you'll get a data source, which for some reason, some of the values have like three more zeros on the end, it'll be the millions instead of the hundred thousands. And I think there aren't necessarily a lot of good automated ways to deal with this.
Yeah, there are some things in development, but essentially a lot of data scientists are siloed in different companies and have reinvented the wheel. I mean, everyone writes their own scripts to deal with this type of stuff. I mean, Pandas, for example, and a bunch of packages in both Python land and the R ecosystem deal with this, but you need to kind of hand code it yourself, essentially. The other thing, though, that I find super cool... is something we get from engineering, which are testing principles. So you've got software testing, right? You can write tests to make sure your software or your code that you're writing does what you think it does. Now, you can also do this with data. So if you load a data set into your Python environment, you can write a set of tests to make sure that the values of your price column are actually numeric, right? And you can flag it if not. So having these types of automated tests, which once again, you would hand code yourself, but you can use them across a variety of projects, will allow you to do this type of stuff. We are seeing increasing automation around this stuff, though, and I do think that...
in the next, this is pretty broad, pretty vague, in the next two to five to 10 years, what data scientists do on a daily basis will be a fundamentally different thing. And I don't necessarily know the time scale. Definitely in the next 10 years, a lot of the cleaning, the basics of writing statistical models and selecting features and tuning hyperparameters, which happens in machine learning a lot, all of this stuff will be automated and it'll free up a lot of our time for fun or creative, interesting work. How about on the subject of creating predictive models? What are some of the pitfalls that...
uh data scientists or traders who are learning to code could fall into uh like is there anything you need to be mindful of when you're working with data and you're trying to create predictive models that i'm sure that if you're not careful you can get results that are actually going to trick you and and going to have very little value when you actually try to implement that in the real world yeah yeah there are and i think um the two big ones are if you have a small a small amount of data You'll be prone to overfitting. And what that means is every data set, if you're trying to model it or something, you'll be trying to model some sort of signal and there'll be noise. There'll be fluctuations around it. And overfitting means that you're really modeling the noise in your data. And so if you try it on new data, you won't signal. You'll be getting noise as well. So it won't generalize well to new data sets. So when you've got a small amount of data, you may be overfitting quite a bit.
If you've got a large amount of data, this won't be a problem, though, because essentially the task there is to see how well it performs by building the model, otherwise known as training, on one subset of your data set and then testing it to see how well it performs on this, what's called an outer sample or holdout test. And you've had people on the podcast before who've explained this far better than I have. But that's the basic idea there. So in the small sample size, overfitting can be an issue. I personally think one of the biggest issues in the machine learning and the deep learning space now is model interpretability.
uh and what i mean by that is we discussed before this idea that you know as data scientists we need to communicate our findings to non-technical stakeholders whether they be managers decision makers citizens um if we're influencing you know legislation or you know parole hearings right like literally there are black box models so i'll explain that term in a second and then give that example but the point is that when you have a model that predicts something um there'll be people who want to know why it predicts or what it does so if i'm telling my manager that we should make a business decision based around the model he or she may want to know why it makes that prediction and what the drivers are there so this idea of model interpretability as opposed to black box models and what a black box model is and deep learning models are a famous example of these or you just can't peer into it you can't see why the prediction is what it is right um and so the famous example there's a there's a fan they're all there a series of fantastic books there's one in particular called weapons of math destruction which is not only a great title but um a great book about you know black box models among others, they're having deleterious effects on society. And I think one of the most telling examples is in the United States, in the justice system, there's a model that's used in parole hearings to predict the recidivism rate. So how likely the person on parole is to re-offend, right? And this is a black box model and the judge uses the output of this model to influence their decision on whether to grant parole or not, right? So this is an obvious case in which we would want to be able to peek into this model to see the inner workings, to understand what's happening there.
So I think this is one of the biggest challenges moving forward, which in all honesty speaks to a far greater challenge of developing a system of ethics around the practice of data science as well. Yeah, it's funny you mentioned that title, actually, because I have read that book. I think that's about it. I don't know. Is there anything else that you wanted to discuss?
Not really. I just want to encourage people. I want to reiterate that I understand. We all work more than we should. We all got so many obligations in the world. So take your time. Have some fun. Do stuff you enjoy with coding. If you've got kids and you want to learn to code with your kids, that's also awesome. Also, the other thing about learning to program that I haven't mentioned yet is that Twitter is an incredible place to ask questions and to receive answers. So R in particular, the hashtag R.
And it's the letter R. I know because I'm Australian, it sounds like what the doctor makes me say. But, you know, it's hashtag RRStats. You know, a lot of people will jump on board and answer your questions there. But feel free to ping me on Twitter. It's at Hugo Bowne, B-O-W-N-E for Edward. And, you know, be part of the community as well because it's getting more and more welcoming. I know that there are some, you know, there are some gatekeepers out there who will, you know, shake their fists at you. But in the end, we're all trying to help each other. Yeah, too right. And I will throw it out there as well. I'm an affiliate for Datacamp.
So if you do want to sign up and check out the courses, obviously, as Hugo's already mentioned, some of the courses are free. There's also some paid courses as well if you want to take a deeper dive. So chatwithtraders.com slash datacamp. I don't think it gives you any special discounts or whatever, but if you do sign up, please use that link because it will obviously help to support chatwithtraders. So yeah, chatwithtraders.com slash datacamp. Yeah, I appreciate it if you could use that link.
Now, Hugo, you've already given out your Twitter handle. You also mentioned earlier that you do host a podcast of your own for Datacamp, and that's called Dataframed. Where's the best place to listen to that? Probably wherever they're listening to this podcast, right? Yeah, exactly. You can, you know, it's on iTunes and Google Play and all of that. You know, it'll be on all your apps. You can also check it out at datacamp.com slash community slash podcast. We host it on our website there. But yeah, Dataframed, and it's one word.
And it'd be great to, you know, if you checked it out and gave some feedback as well, we always welcome as much critical feedback as possible. Cool. Okay. Well, let's leave it there. I'm glad we could tee this up, Hugo. I know it's been a couple of months in the making, so I appreciate you making the time to make it happen. Oh, look, this has been a load of fun, Aaron, and I really appreciate the invitation. Cool. Thank you very much. You've reached the end of this episode of Chat with Traders, but rest assured, there are more episodes loaded with real market insight and zero hype on the way soon. So to stay updated with each great new release, subscribe to the podcast on iTunes, and we'd love it if you'd leave a rating and review. We'll catch you next time on Chat with Traders.