Journey to TimeScaleDB [Mike Freedman]

Listen to the full interview on SEDaily: https://softwareengineeringdaily.com/2021/06/28/timescale-time-series-databases-with-mike-freedman/We originally created Timescale, really from our own need. Around thattime, 2014-2015, my co-founder and I, Ajay Kulkarni, who we go back many years, we resyncedup and we started thinking about it was a good time for both of us to think about what the nextchallenges are that we want to tackle. It seemed to us that there was this emerging trend ofnow, people talk about the digitization, or digital transformation. It feels like somewhat of ananalyst term, but I think, it's really responsive of what's happening, in that if you think about thelarge, big IT revolution, it was about changing the back office. What was used to be on paperwas now in computers.What we saw was somewhat the same thing happened to basically, every industry, from heavyindustry, to shipping, to logistics, to manufacturing, both discrete and continuous and home IoT.Sometimes this gets blurred under IoT, but we also think about it more broadly as operationaltechnology, those which are not necessarily bits, but atoms. A big part of that was actuallycollecting data of what those systems were doing. It's about sensors and data and whatnot.When we do Initially looked at this problem, we were thinking about a type of data platform wewould want to build, to make it easy to collect and store and analyze that type of data. I thinkthat's a way that we're slightly different, or why our – what we ultimately built as our databaseended up being fairly different than a lot of other so-called time series databases. That'sbecause many of them arose out of IT monitoring, where they were trying to collect metrics fromservers, where we were originally thinking about collecting data more broadly from all these typeof applications and devices around your world.When we started building it, it was originally focusing mostly on IoT. We quickly ran into thisproblem that the existing databases out there and the time series databases out there were notreally designed for our problems. They were often much more limited, because they werefocusing on this narrow infrastructure monitoring problem, where the data maybe wasn't asimportant. It was only a very specific type. Let's say, they stored only floats. They didn't have tohave extra metadata that they wanted to enrich their data to better understand what was goingon, like through joins.After, basically working on this platform for about a year, we somewhat came to the conclusionthat we actually need to build somewhat of our own time series database that was focusing onthis more broad type of problem, and so that's what we do. That's what led the development ofwhat became Timescale.JM: Today, what are the most common applications of a time series database?Like and speak mostly about obviously, TimescaleDB, rather than – as I wasalluding to before, a lot of the other time series databases are much more narrowly focused onIT monitoring, or observability. We really see our use cases across the field. We certainly seecases of observability. In fact, we have subsequently built actually a separate product on top of Timescale called Promp scale, that is really used for initially Prometheus metrics, but morebroadly, to make it easier to store observability data with TimescaleDB.We see still a lot of IoT. We see a lot of logistics. We see financial data and crypto data. We seeevent sourcing. We see product and user analytics. We see people collecting data about howusers are using their SaaS platforms. We see gaming analytics, where companies are collectinginformation about how people's virtual avatars are actually playing within the games. We seemusic analytics. We like to think of the old way, used to find the pop stars, you went down to thesmoky club. Now you collect SoundCloud and Spotify streams, and you use that to identify whothe next breakout artist is going to be.All of these are example of time series data. It's really what's so exciting to us as is it's such abroad use case, so horizontal, because basically, it's all about collecting data at the finestgranularity you can.Tell me about the initial architecture for TimescaleDB. You’re based off ofPostgresSQL. What was the reasoning around that decision?I think, as you point out, Timescale is actually implemented as an extension onPostgresSQL. Starting maybe 10 or 15 years ago, PostgresSQL started exposing low-levelhooks throughout its code base. This is not a plugin where you're running a little JavaScriptcode. We have function pointers into – we get function hooks into the C. PostgresSQL is writtenin C, and so TimescaleDB is, for the most part written in C. We have hooks throughout the codebase at the planner, at sometimes in the storage, at the execution nodes. We are able to insertourselves and do Lot of optimizations as part of the same process.You could ask the question of why not just implement a new database from scratch? Why buildit on top of PostgresSQL? I think this really gets to that, we always viewed ourselves as, and wehear this from our users and community all the time that we are – they are storing critical datainside TimescaleDB, and they need it to, A, work and be reliable. They also need it to be – theyhave a lot of use case requirements. It’s not this, again, narrow thing where you're collectingone metrics, and all you're asking to do is figure out the min-max average of a certain metric.You want to do fancy analysis. You want to do joins. You want to do sub queries. You want to docorrelations. You want to have views. You want the operational maturity of a database. You wanttransactions, backup, and restore, and all of the replication and all of the above. Some peoplesay, it takes maybe 10 years, at least, to build a reliable database. We thought this was a greatway in order to immediately gain that level of reliability, we ourselves are huge fans ofPostgresSQL. It has such a great community. It also has such a large ecosystem.The idea is that effectively, that entire ecosystem would work from us on day one. That means,all of the tooling, all of the ORMs, all of your libraries would just work. If we support full SQL, notSQL-ish. If you know how to use SQL, you could start using – and if your tools speak SQL, ifyou're running Tableau, if you're running Power BI if you're running Grafana, if you're runningSuperset, those all just start working on day one.Now, the second part of it is, well, what does that mean to build a time series database on top ofPostgresSQL, which clearly was designed more as a traditional transactional database, OLTPengine? Sometimes they talk about you think about this architecturally. What I mean by that isyou somewhat think about what your workloads look like and what that would mean from asoftware architecture. Maybe I'll give you a very concrete example. Starting maybe 10 or 15years ago, if you look at traditional databases, you started seeing the growth of what peoplecommonly now called as log structured merge trees, LSMs.This is a data structure that goes back to the mid-90s, but I think you first saw Google, JeffDean and Sanjay Ghemawat built something called LevelDB. The whole idea of an LSM treewas, if you look at a workload that has a lot of updates, so with a lot of e-commerceapplications, with a lot of social networks, you're constantly updating things. Traditionaldatabase, if you think about a disk...