Recorded 21-03-2022. Published 30-06-2022.
Ryan Trinkle is interviewed by Joachim Breitner and Niki Vazou. Ryan Trinkle co-founded Obsidian Systems, a company that uses not only Haskell but also even more exotic tech such as Functional Reactive Programming (FRP) and Nix. Ryan shed some light on the business side of Haskell and we get to hear that hiring for Haskell is actually excellent.
Joachim Breitner: Welcome to this Haskell Interlude episode, today Niki Vazou and I interview Ryan Trinkle, Ryan has co-founded Obsidian Systems a company that not just uses Haskell as the secret button but even more exotic tech that as functional reactive programming and Nix. Ryan shed some light on the business side of Haskell and we get to hear that hiring for Haskell is actually excellent. Stay tuned, especially if you want to hear this story of the supersitious garbage collector that doesn’t like number 13. Hi Ryan, good to have you on the podcast.
Ryan Trinkle: Hi Joachim, I’m very glad to be here with you and Niki. It’s a great opportunity. Thank you.
Niki Vazou: Hello everyone.
JB: Yeah, I must say I’m particularly excited about this particular episode because when I was relatively new to Haskell and I started this impression that has this academic language only for teaching, only for language teachers, I was always very excited when I heard that somebody’s actually building a company on top of Haskell and solving real world problems for people that probably don’t care about Haskell and when I heard about Obsidian Systems that you co-founded back then using Haskell and even more interesting tech that we’ll get into. I was was following this with… with great interest. So I’m very curious Ryan, how did it all started? What came first? Did you want to build the company? Did you want to use Haskell? We’re just in the need of money? Can you tell us the story?
RT: Sure, so I’ve been working with my business partner Ali Abrar for about 15 years now, we really started when we met in law school and we actually tried to start another company using Haskell right after we finished law school making iPhone games and we were, you know, we were not successful in that company but we did end up contributing a whole lot of ARM and mobile stuff back to GHC which was a really nice result, and then, you know, we worked in a lot of different startups, both during law school and also after iPhone Studios which was the company that was doing iPhone games. And what we saw repeatedly as we worked in lots of different startups was the same sort of mistakes would get made over and over and it’s understandable, I mean, people start a company, they may not have experience building a product or whatever else and they had have to bring it the first time a lot of the time, the founders are not highly technical, and so if you’re not a highly technical person, how do you even know whether the engineers you hire are doing the job or not. It’s just really difficult to understand. So we decided that, we thought that having been around the block a few times we could do considerably better than that. And we wanted to turn that into a consultancy that could reliably deliver results for our clients. And so the key with that was really that we wanted to break out of this paradigm of just trying to pick the most popular technologies with the most sort of justification, we were just, it was just the 2 of us and so we could make decisions that were a little bit more bold. Obsidian Systems doesn’t have investors so Ali and I just have sort of a lot of latitude in doing what we think is right and in a lot of technical cases that means choosing a technology that isn’t highly popular like Haskell, like Nix and NixOS which we also use, but a lot of the time it does mean choosing something popular like Postgres, for example, with a database I’m not looking for a ton of features or anything fancy, I just want something that’s going to hold on to my data and never lose it and be a reliable source of something for my application. So we tried to really dig into things and and make the right decision. Obviously a lot of our clients have not heard of Haskell before they start talking to us, especially the ones who maybe just they’ve gotten some sort of funding and they want to build something and they hear about us through usually word of mouth. For a long time we didn’t even really have a marketing site, it was just sort of our phone number on a web page, what was actually kind of embarrassing, um… but it didn’t really matter because when people are deciding who to hire for… you know, really staking the life of their startup on it. They usually are not looking for a fancy website, they’re usually looking for a friend or somebody trusted who can vouch for us.
JB: So when you say clients, what kind of field are we talking about, what kind of clients or products you were catering for?
RT: Early on, um… the vast majority of our clients were startups and they were folks who had really nothing to do with Haskell or even necessarily high tech, we did one that was a logistics startup, it was just managing trucking. We we did another one that was trying to streamline some things in the finance industry with mortgage applications and things like that. Really, these are not companies that are trying to achieve some spectacular engineering or technical goal. They’re just trying to build a reliable useful application where the business value is in something else, right? It’s like oh we want to make it really easy for people to get a mortgage or or to refinance their student loans or something like that. So they didn’t care that we were using Haskell. Sometimes they would be a little skeptical like I’ve never heard of this, why don’t you use Python or whatever. But then once they saw us starting to deliver results then they would usually be very quickly won over. So yeah.
NV: And now that Haskell is getting more popular does it affect your clients, I mean you said they didn’t care, now do you even like get clients because of Haskell?
RT: Yeah, yeah. I think that’s happened a little bit all along, but it has definitely happened more recently. So we’ve had some some clients that, for example, we’re known for expertise in functional reactive programming (FRP), that a couple of clients who say you know we’re doing Haskell, we want to use FRP, maybe you can help us with that.
NV: So can you tell us a little bit what is a FRP?
RT: Yeah, absolutely. Functional reactive programming (FRP) is a way of describing interactive systems using only pure functional semantics. So in most programming including in Haskell, but especially outside of Haskell, when you want to describe an interactive system which is a system that gets some sort of input and then produces output and then gets more input and produces more output, like a user interface or really frankly most applications. When you want to build something like that usually the way it’s done is with imperative style coding, you will say sort of a list of things that need to happen in a certain order and we do that in the IO monad in Haskell but you know, we also do it in C and it really the C and the Haskell look pretty similar when you’re writing that kind of code. And the reason that we write things like that even in Haskell is that we need some way of specifying like when the data is coming in when it’s going out. And traditional pure functional programming doesn’t really give us a vocabulary for that, it will sort of take all of its data as input at the beginning like maybe a compiler right? You give it a bunch of source files and then it does a whole lot of computational work and then it spits out an executable or something like that, of course real compilers need to worry about memory management and stuff. But the basic idea is that’s a very clear pure function, you take a bunch of stuff and you produce an output and what happens in between is sort of side effect free, but a graphical user interface is quite the opposite of that, almost every interesting thing a user interface does is like okay the user clicked a button now we need to refresh the screen and we need to pull up some data and maybe take some sort of action and then we wait for the user to do something else, and maybe we wait for data to come in so that we can put it on the screen and the user sees it updating live. So the problem is writing these things in imperative style even within Haskell loses a lot of the benefit of pure functional programming because you get back into this mode of I’ve got mutable variables, I’ve got often sort of like poorly scoped state things that are a little bit too like global variables or even just variables living in a very large area in your program and that makes it very difficult to reason about what your code is going to do, and it gets you far away from the whole “if it compiles it works” thing, which is I mean never strictly true, but it is surprisingly often true in Haskell I think most Haskell developers have had this feeling that if it compiles it works way more often than it feels like it should, so functional reactive programming gives us a totally pure vocabulary for describing these interactive things, like buttons on a user interface, a button for example might produce an event and an event is a pure value, but it represents a potentially endless stream of clicks on the button.
NV: So when you say pure value, you say a monadic computation, no?
RT: No, I really mean a pure value here.
NV: The button is a pure value?
RT: The event is semantically pure and what I mean by that is you cannot affect anything by just holding on to this event. So I don’t know if there is academic theory that backs me up here, but this is the sort of mental model I’ve arrived at. Basically the fundamental thing that makes let’s say a pure integer easier to deal with than say an IORef is that when an IORef is being used in many places in your software they can all sort of have arbitrary impacts on it and you get into this situation where you can’t know what the IORef does unless you read through the whole codebase. And even if you do read it you probably won’t be able to understand it all. It’s just too big for the human brain to hold. But if you’ve got an integer, well you pass it into all kinds of places and who cares what they do with it because it’s not going to come back and change the value of the integer or anything like that. And event in FRP works the same way even though it represents something that’s changing through time, the people who you send it to the functions that you give it as an argument to can’t have any impact at all on what happens so you can kind of think of the event as though it’s an infinite timeline stretching into the past and the future and you can only observe it at the current time so you can’t necessarily, you definitely can’t see the future of the event but you also can’t affect the future of the event. So, basically this has sort of brought me around to the idea that mutability and purity are actually separate properties of a programming semantics.
JB: So maybe another way of looking at this if I compare it with maybe more traditional way of doing UI, so you have handlers on events then you can compose these handlers and but often what we see I guess in most of these libraries is that they bubble down and up some kind of chain of event handlers and you can disable the surrounding event handlers by returning falses or depending or calling some disabled default and suddenly you have this kind of action at a distance whereas with the pure event from a button you’re still handling an event in a way if you want to think about it morerationally which maybe I am still doing too much here. But at least I know that nothing else can interfere with how I’m handling this particular events, that…
RT: Yeah, absolutely, and if you sort of look at the standard for how these things are done in most of the industry which of course has been making improvements with things like React, but you know when you’ve got a bunch of callbacks, you’ve first of all callbacks have a type signature that’s something like A to IO unit. So by definition they’re pretty much just doing side effects. So we all sort of in the Haskell community know that can get you into trouble, it’s very difficult to reason about, but even more so, you know, if you have a single event source and you add on several different callbacks to it, those callbacks can easily interfere with each other and the order is actually important at least sometimes and at the same time you don’t usually have very good control over what the order of execution is, so he just kind of toss them into a bucket and hope for the best and then this gets even worse when you have chains of callbacks like if you have one callback from a certain source of data and then you use that to update another source of data and then that triggers another callback that goes somewhere else. Ah, it sort of very quickly becomes impossible to understand what the system’s really going to do.
NV: Y’re using a monad basically to encapsulate this side effect that you’re talking about, right?
RT: No, um… we do use a monad in Reflex for certain things but not for the the sort of basic stuff. For example, merging two events together, that’s completely pure, there’s no monad involved.
NV: But when you say event, the event is not a monad itself?
RT: No, it’s not. It actually can’t be because you can’t write return, but at a sort of so, okay, so there’s the technical level where it can’t be a monad because it’s impossible to write return, because that would be an event that’s always firing and that actually turns out to sort of break everything in FRP, but you could write bind for it but it’s not doesn’t wind up being very useful, so it could be a semi-monad, if you really wanted it to be, but it’s better to think of an event as essentially a container like an infinite list, except that with a list you have a bunch of items that are all existing at the same time but in different places in memory, with an event you have a bunch of different items that are all existing at the same place but at different times.
JB: So it’s like applying transpose to a list and then excellently getting the dimensions wrong and turning into time rather than space.
RT: Exactly, exactly. Yeah, so… so the semantics are just like any other pure container you can fmap over it. You can’t traverse it because if you, it would take an infinite amount of time for your traverse to return. But you could sort of like you can scan over it, you can fold over it, you can do those sorts of things and they all just sort of perfectly make sense. The only place that a monad is actually necessary in Reflex, which is the FRP system that I built, is when you need to know what the current time is. For example, when you want to create state in an FRP system, you need to… let’s say you had an event and you wanted to count how many times the event actually occurred, like a button click. Well you need to know when you’re starting from, because as I said before the event logically speaking stretches infinitely into the past and infinitely into the future. So despite that you sort of get the event at a certain time, right? So you can’t look infinitely into the past, don’t retain history for any of this stuff because it would be a memory leak, so logically speaking we do have a monad that says like okay if I’m going to start counting when do I start counting? And semantically it’s just a reader monad, operationally it’s extremely different from a reader monad, but it’s a way of sort of saying this is when we’re starting. But other than that nothing else uses monads, um… well, internally but the overall semantics is pure with the exception of that.
JB: You… The FRP you’re using is use it only for the UI of your application, so does it actually like permeyate to the backend then you can program everything and across a stack in one single programming model.
RT: Well, I would love for that to be the case. But right now we do primarily use it for the user interface side of things. We also have a terminal library that lets you build like curses style user interfaces using FRP and Reflex itself has no opinion about what you’re using it for, it doesn’t… it doesn’t know anything about user interfaces, we have a Reflex DOM library which allows you to do DOM manipulation in the browser, and then other people have built libraries for this for like native iPhone and I think Android and various things like that. So it can be used for a bunch of things people have also built like web servers that use Reflex as their sort of basic model, the only problem with that at the moment is that Reflex’s implementation like the operational core is all single-threaded, which means that you can instantiate it multiple times so you can run multiple threads but those different threads can’t talk to each other in FRP, you have to have them communicate via traditional imperative techniques. So each FRP domain is just sort of its own single-threaded thing, so that’s not ideal, we’ve kept our backends a little bit more traditional for the time being, but I would certainly look forward to at some point having, you know, FRP be the way that we do backends as well.
JB: So, the UIs you are building are Web, iPhone, Android and terminal applications?
RT: Yeah, we have not built any terminal applications for clients, that has not been a big request, but we’ve built internal tools that way. Our clients, yeah, we build web, mobile and also desktop apps for some clients and they all build with the same, you know, semantics the same Reflex DOM based core.
NV: And how are you generating your backend, are you… are you using GHC?
RT: Yeah, the backends that we build are typically built with Snap or WAI sometimes and all GHC-based, you know, we certainly appreciate the high degree of parallelism and everything else that GHC provides, I mean it’s a wonderful backend language, I don’t think that we’re doing anything that special I think a lot of Haskell companies get a great benefit out of using Haskell on their backend.
JB: So if you’re so heavily invested into to, like GHC and Haskell, how… when you look at the current evolution of Haskell, like what’s happening at the moment with language extensions with Haskell Foundation (HF) of course on the governance level, also technically, what do you think, is it going in a good direction, is it going in a bad direction, is it and going no direction and that’s good or bad or?
RT: I think it is going in a good direction and you know I’m really excited about what HF is doing, so I was asked to be part of the initial creation of the HF and now I’m on its board and I’m the treasurer and I think that the board has been assembled from a really excellent, very representative group of people, I’ve learned a lot from speaking with the members of the board and also members of some of the task forces, like the technical task force and I think it is really going to be very helpful to Haskell going forward I think it’s already accomplished a good amount of stuff and fundamentally to me Haskell is like this still mostly hidden gem and Haskell has been the recipient of so much academic work of such high quality, but a lot of that work has not really made its way into production. I mean as you said you know back in 2015 like there were really not that many people doing commercial Haskell, and now there are a lot more but it’s still I think a lot less than it ought to be, so we need to improve things like the way that we… we use improve a lot of the incidental stuff I think that the core of the language is excellent and of course we’ll keep improving it I mean there’s a lot of really brilliant people working on things, but it’s really making it accessible to business so that we can take those academic accomplishments and start to really reap the value out of it. One of the reasons that I’m in industry and a practitioner myself rather than trying to personally go an academic route is that I just felt like there was a big need to get these high-tech really smart things actually being used, I think that almost nobody would be happy to invent an amazing thing and then just have nobody else use it. You know, people want their ideas to be used and that’s how the real value for society gets generated. So yeah.
NV: And do you have any suggestions on how this can happen, I mean maybe using your personal experience right? Because you finished law and like how did you get into Haskell?
RT: Well, so I got into Haskell actually a little bit before law school, I was… I did Computer Science undergrad and I at that time, you know I got into law school and I wanted to make myself a more well-rounded person. So that’s why I wanted to go to law school I wanted to really get the education, a lot of people say that law school is like a applied philosophy degree and I think that’s reasonably accurate. I certainly learned a lot about how to speak and how to think about things that were not just technical and so I really appreciate that education, I never took the bar, I never practiced law but I still am very glad that I that I went and got that education. But going back to your question I think that’s really impacted me to take this sort of look of like okay well not just how do we build this amazing technical thing, but how do we give people the trust they need to actually use it. Databases for example are an area where I skew very conservative because it’s like to me if somebody has a brilliant database, they just made yesterday even if it’s awesome like I don’t know how it’s going to respond when like a disk fails in a certain weird way, or something like that you know whereas with Postgres I know it’s been put through its paces for you know, 20 years or more, and so that’s… I don’t know that’s sort of part of how I think about this stuff.
NV: So the industrial setting is not trusting Haskell because it’s still like a bunch of academics playing with…
RT: Well, I don’t think the academics are doing something wrong, yeah, you say playing with it I… I think it’s really substantial work. But I do think it’s a different skill set to translate that into something that is easy for businesses to adopt. You know, that means we need to have things like better education which everybody knows about, but we also need things like, clearer decision making, right? Like there’s a lot of stuff… Haskell is a lot like… you know, choose your own adventure book, right? Like there’s a lot of excellent things but…
JB: Especially if your adventure is dealing with dependencies and package installations then you really have the best game you can have.
RT: Yeah, yeah, like you know when when somebody starts using Ruby on Rails which was so popular like back when I was like starting to get into the tech industry I read a book on Ruby on Rails because everybody was saying like this is the way you should build a website. And I tried it and I didn’t like it very much and I ended up switching to Haskell but, but I see the appeal because it’s sort of like this big set piece thing and you don’t have to go in and research every little component and you know when, typically building stuff in Haskell you need a lot of knowledge of how to assess which dependency to use, like there’s just not a lot of stuff that’s like prefabricated for you, and industry probably needs more of that kind of stuff in order to make it effective. There’s also a lot of, in industry sort of trepidation about having, finding enough engineers you know and this is something that I don’t think we can directly solve because it’s a chicken and egg problem. We’re going to have to sort of gradually grow our way out of it, but you know, business people are afraid that you know if they’re dealing with a very small community then what if their business gets big which of course everybody wants their business to get big in at least in the startup world. And then what are they gonna do, you know? So I think it’s a valid concern and there’s things we can do about it. But it’s not gonna be any kind of silver bullet.
JB: How bad is it or how good is it like you, I guess you’re in the position of somebody who occasionally you’re often has to hire somebody who knows Haskell or wants to learn Haskell so how’s your experience?
RT: Well, our experience has been excellent. So for us hiring Haskell folks is really great and it’s nice because people who use Haskell have typically taught themselves. So they’re pre-selected for having a good amount of motivation, um… There are a lot of benefits to hiring people in Haskell I think but it’s certainly true that if I wanted to go out and hire a hundred developers over the next two months I wouldn’t know how to do that and it might be possible with enough money I think IOHK has done that kind of thing in the past like the very well funded Haskell shops have maybe done big lifts like that. But I can see how it would be a challenge. For us on the other hand we focus a little bit more on you know how can we make the maximally effective team and frankly, we’ve found that we can do a lot more with you know, 4 engineers than a lot of other teams can do with 20 and so that’s really a huge benefit. Not just because obviously from a business perspective it’s more cost effective but also because you don’t have these diseconomies of scale that usually crop up in large teams. For example, like you can’t really have a single manager for 20 people you need to have multiple managers which means now you probably need somebody to manage all those managers and that’s going to make all your communication, a whole lot slower. So yeah having small teams of developers, where each developer is extremely able to make changes in the system. Like for example, we do everything full stack, we don’t divide our engineers into backend at frontend. We always work throughout the whole stack, of course people may have different specialties and things like that. But then usually they’ll pair or they’ll sort of just talk amongst themselves. But we like to give people responsibility for whole features and this is way better because it means that people can sort of do everything they need to do to get things done. They don’t have to send a request to the backend team and wait three days to hear back that oh no that’s a terrible idea. You know and it’s it works so much more nicely and by the way, the fact that we can cross compile Haskell to everything that we do, browser, backend, mobile, whatever that is really a key part of enabling us to work full stack because if we had to have somebody who was an expert in Swift and also an expert in Haskell and also an expert in Kotlin or whatever. There aren’t that many people who are very good at all those things.
JB: So the model you describe with having a small very effective team I think it definitely sounds great. I wonder does it did do you sometimes hit roadblocks or maybe not roadblocks but where which just doesn’t scale anymore because you have a project that’s too big for even for 5 x programmers to manage where you say okay this is no, no longer a project for us and you need to go to a company that has these hundreds of programmers available or did it never come up?
RT: Well typically what we do as our companies, as our clients grow, they will start taking over more and more of the project. So you know usually what we’ll do is we’ll sort of kick off the project, get it launched, you know, get it to be somewhat successful and then we’ll help train internal developers as our clients hire. And then at some point we’ll hand over the project fully. And that seems to be a model that works pretty well. You know, clearly as well… I guess this is probably not common knowledge in the engineering community but in the business communities it’s pretty widely accepted that service businesses are relatively slow growth and that’s what we are as a consultancy. Whereas things like startups are obviously trying to be extremely high growth and so it’s not very realistic for us to keep up if one of our clients really you know takes off. Of course we would try to do everything we can to be there for them and make it happen but generally speaking as that kind of growth picks up, people want to start having in-house development teams and take more ownership of that.
JB: But that’s intriguing that means you have companies who’ve never heard about Haskell before come to you, you build them something that they like and and they’re actually now they’re suddenly becoming a Haskell shop themselves, because…
RT: Yeah, yeah, absolutely.
JB: That’s like viral.
RT: Yeah I hope so, I mean and the thing is it’s not… It’s not like we’re trying to sell them on the sort of theory and like oh man, don’t you, don’t you love purity and strong types like the business people are not getting down to that level and they care that the bug counts are lower. They care that we’re better at shipping on a schedule and by the way that’s very related to the bug counts right? Bugs, you can’t really schedule how long it’s going to take to fix a bug and if most of your time is spent debugging it really wrecks your ability to plan anything on a schedule. So you know we can build things with less bugs, we can build it faster, we can typically build it with a smaller team and the business people see this and you know after a while it can really sink in. So, yeah I mean that’s definitely something we’ve been looking to do, you know… we’ve always wanted to take these technologies and use them to produce the things that our clients want to produce rather than trying to seek out clients specifically who just want the technologies that we are expert in.
NV: But then the hiring problem that we discussed before becomes more interesting, right? Because you are asking from business people that don’t even know what is Haskell to hire Haskell experts.
RT: Yeah that’s true, it does… it doesn’t become a challenge, I’m not sure that it’s actually a worse challenge than they have in other languages and the reason is that hiring Haskell developers you may have a little bit less volume, like you’re not going to get a hundred resumes for a Haskell job, typically. But with… like I’ve you know I’ve had to hire Javascript developers and you do get a hundred resumes and then you have to spend a lot of time going through them and looking for somebody who’s actually good and that can be a real challenge. So I think that you know for our clients with our help they’ve always been able to hire who they need to hire. I definitely think that you know even, even with explosive growth you don’t end up needing an enormous number of developers typically with a Haskell shop I mean, I think like some of our clients have had like 60 or 70 Haskell engineers. So it’s you know, I guess IOHK has also been a client I think they probably have more than that but I wasn’t involved in their hiring. So yeah, I guess it’s the hiring thing has always been scary but I’ve never seen it be a really big problem. You know, its bark is much worse than its bite.
JB: But it’s not just that you not just Haskell that you’re betting on and betting on for your customers and then your customers seem to be happy, but it’s still a bet, you’re actually there’s more interesting non-standard tech that you’re using. So for I mean, I guess next one would be GHCJS which is this GHC fork I guess I could still say that compiled Haskell to Javascript which I believe is I maybe may correct me if this is the wrong, wrong way of phrasing it but it was an impressive feat of work by very few people, mostly Luite Stegeman and I always wondered is it very risky to have such a very rarely I mean it’s okay, Haskell is rarely used but GHCJS is even less used component is so central to your business, so…
RT: Yeah I certainly understand where you’re coming from on that and it is a project with relatively few maintainers, it’s not just Luite these days but, yeah, it’s not a huge group. But we have not found this to be really a big problem, really what it’s meant is that a few times we Obsidian have had to step in and just do the maintenance. You know whether it’s tracking down some obscure bug or you know helping update to a newer version of GHC and this has been a cost that we’ve just sort of taken on ourselves and done as part of our Open Source work. We do quite a lot of open source work that’s not built to any client and it’s just sort of what we think is necessary to move the ecosystem forward and GHCJS is part of that. I would say that the benefit we’ve received from GHCJS has been like orders of magnitude higher than the maintenance costs that we’ve had to pay. I don’t know exactly how much it’s been probably like a couple of… maybe like two engineer years have been spent on that over the last seven years by us and it’s just not a very large fraction of our total open source work even, so yeah, it’s I think it’s another thing it’s a very reasonable kind of fear but in practice for us it has not been a problem.
JB: And you never had like horrible bugs that you were unable to fix the reasonable end of time or like times where you thought oh my God we made this was a mistake to use this tech.
RT: No, I’ve never had a time when I felt that way, there was one really crazy bug very early on, right after GHCJS was released and it was getting in the way of one of the client projects and I ended up having to spend just like a ton of personal time and it ended up being this like really obscure little thing where if you have a heap object with 13 or more slots in it, like references to other objects, then one of those slots would not be traversed by the garbage collector and so if in that situation you had an object on the heap where the only reference was from one of these missing slots then it would get collected…
NV: Why 13?!
RT: Yeah but only if it had some sort of finalizer or something were you really in trouble because this is Javascript based, so you’d actually still have the data, it’s just that your finalizer would run at the wrong time that was a very painful thing to track down it ended up being a one character change I think changing a 2 to a 1 or something in GHCJS, but yeah, so that was very tough that was really, just when Obsidian was getting started, we didn’t have any employees at that time, Luite just released GHCJS like maybe six or eight months before so we were completely prepared for it to be still buggy and having issues. But yeah I don’t… I don’t really think there’s a realistic way that we could do the kind of stuff that we do without having something like GHCJS, I’m very excited about the WebAssembly progress that’s being made within the GHC community I really hope that we have the ability to have WebAssembly as a first class compilation target in near future.
JB: But in summary I think it’s worth pointing out that despite the impression that one might get that. Maybe I also had at some point of GHCJS being looking like a dangerous technology. It works very well and if you get your way around of actually installing it and using it then yeah, you can use it.
RT: Yeah, absolutely, you know I think that’s to a certain extent this comes down to non-Haskell stuff. For example, we use Nix for all of our dependency management. And we have a rule at the company that every repo has to build in one step via Nix and when we hand things off to QA for example, um… and by the way not every software company believes in QA these days but we certainly do, um… and when we hand things off to QA we give them a git hash, we don’t give them built artifacts. We just give them a git hash and they are able to nix-build from that and this gives us a high degree of confidence that we are testing the right thing and this also includes the ability to pin down things like GHCJS like if we have to run GHCJS with a small patch because you know we’ve found an issue and we fixed it and we’ve contributed upstream but maybe it hasn’t gotten merged or released yet, Nix makes it essentially trivial to work on these minor forks of things and this really gives us a huge ability to operate in the open source world as good citizens where we will build patches and we will fix things in place rather than making workarounds, this also saves our clients a ton of money by the way because when you do workarounds in the downstream codebase, well now you’re on the hook for maintaining those workarounds forever. It’s actually a surprisingly high maintenance cost whereas if you just go into the open source codebase and fix it. You know assuming it doesn’t take too long to fix it, it’s so much cheaper. So Nix gives us the tools we need to be able to pin things so that once we QA it like I don’t really care if there’s a bug in some version of GHCJS that we’re not using. We’re not going to accidentally upgrade to it. You know we’re gonna upgrade to it. We’re gonna put it through QA and we’re gonna be sure that it really works. So the the sort of the flexibility to do patching and the rigidity to know exactly what we’re building and not have accidental upgrades and things like that Nix enables us to really use things that are active open source projects like this without any real downside to us or to our clients and then the second thing is just you know being serious about QA and about process, you know, we don’t really believe in just you know, putting things out and seeing how users respond I’ve certainly worked with plenty of startups who kind of put the QA on their users and I understand the sort of hesitance to budget out a bunch of money for QA but every case where I’ve seen it’s totally worth it. So you know the fact that we do a lot of that kind of testing really helps us work with technologies where we might be a little bit fearful about the maintenance.
JB: So with user interfaces I’m always kind of interested, how do you actually go about effectively testing and like do you do you have actually humans sitting there clicking through the thing for every version or is it scripted as it’s Selenium. Do you test below the UI. What is your key strategy?
RT: We do both or all 3 of the things that you just mentioned. Um… so we have a team of in-house Quality Assurance people and their job is to sort of be the hunter killers of the of the process. So they go into the app and they’re not just running test scripts, right? They’re not.. they’re not just, they do have checklists to make sure they don’t miss things but they’re not just sort of like going through a fixed thing, they’re really trying to break the app, you know, they’re… looking for ways that they can get into a weird state or whatever and they’re very good at it, um… and…
JB: That sounds slightly annoyed.
RT: Yeah, no, no, no they, they should be.. It’s actually excellent I mean we’ve been put on the spot so many times by our clients you know CEO of the client comes in and wants us to demo something and then and then they’ll like specifically say like can you show me that thing which looks like it’s rarely used you know and you know and our QA team has gone through it and and and it’s solid, um… So it gives yeah gives me a lot of confidence in demos actually, but then so we also do scripted like automated user interface testing. For example, Reflex-DOM has a fairly large user interface test suite that is Selenium-based and headless Chrome based and for us the big question there is a sort of cost or like ah is there’s like a time cost tradeoff kind of thing, like it takes quite a while to write Selenium test scripts. So usually makes sense to start with manual QA on things and then once something is stable at the UI level like if you have something that hasn’t changed for six months or a year um it’s not getting redesigned and it’s also like a somewhat important part of the app or whatever then it makes a lot of sense to start doing a lot of automation but we don’t jump straight into automation because you know with startups a lot of the time you know we operate on a two week iteration cycle so we go from releasable thing to releasable thing every two weeks and it’s not uncommon for major user interfaces to be getting redesigned every two weeks and at the beginning of a project. You know, you release something to your users and you see that like you know there’s a bunch of complaints like hey I really want to be able to do this thing or this thing is too hard to get to or I don’t understand this and so then you get a redesign and we want to be very responsive to that. That’s how the company can succeed, but if we were taking the time to write manual or automated test scripts for every single UI that would just cost way more with no grow benefit.
JB: So you said you have, um… also testing people is it are they like all sitting next door and in the office or is your company more distributed than that I think you’re based in New York City that correct?
RT: We used to be based in New York City and yeah, our QA team did work out of our New York City office um we’ve always had a principle as a company that we just want to hire the best people wherever they are. We don’t do any kind of cost driven offshoring or anything like that. And so we actually found it was extremely useful to be sitting with our QA guys and be able to work through them work through things with them. Now when COVID hit we did make the decision early on to close down the office and then a little bit later that we would make it permanent, um… now we already had at that time about a third of our people remote, we’ve always hired internationally as I said but… yeah, we had two thirds of our people in the New York City office and it was quite a big transition to go fully distributed but it has also had a lot of benefits. The biggest one being that it makes sure that the people who are remote have equal footing with people who are in the office. You know, one of the problems that I think many companies have with remote work is if you’ve got people you know, get to know each other really well in the office then people remote can really feel excluded. So I think that we were probably affected by that as well and so I’m really glad that now we’ve got a system where you know everybody is able to interact we use basecamp as our primary interaction tool. And we do a lot of video chats. You know that kind of thing like a lot of people are doing now. And the overall effect I think yeah has been really positive, now it is a little bit of a challenge sometimes to work across many time zones. But we have primarily dealt with that by you know, ensuring that, ah… we, well… we always have project managers take point on things and they are typically in a time zone that’s very close to the client that they’re working with and so when engineers are talking to the client it’s usually a more special kind of situation, but the product managers are able to gather requirements report back to the client, work with the engineers because we now have a fully distributed asynchronous process, um… we don’t have a lot of standing meetings at the company. Especially ones that involve engineers, we like to keep engineers schedules as clear as we absolutely can, um… you know, so the time zone shift between PM and engineer are not that problematic it still you know still can be a little bit of a challenge but, um… but you know, all in all being globally distributed enables us to hire the best people and that’s a benefit that I think outweighs any of the challenges associated with it.
JB: So maybe to wrap this up if let’s come back to Haskell a little bit again and there are so many things going on with Haskell right now, both organizational, technical, there’s extensions to the type system in the making, um… if you could make one thing happen for… for… free you know like to get a fairy tale 1 wish 1 thing about Haskell either something that is already maybe happening or something that’s completely, um… maybe, maybe not, not not on the rise and even yet but you could make it happen, um.. do you even sense what that would, that would be?
RT: I think the biggest thing if I had if I had a magic wand I would want there to be a multibillion dollar startup, a unicorn, as they call them, that is primarily Haskell focused and I think that would do a huge, um… favor to us as a community and just demonstrate that you know it can be used to make money. That’s… that I think it certainly can be, this is we know that it can be used to make money because that’s what we do every day but it’s not always as visible as we’d like you know we have the great work out of ah Facebook now meta, with Simon Marlow and you know we have some other high profile users. But we don’t have like a champion in the business community and I would really like to see us have that and yeah, and I think that means that we as the Haskell community, um… if we want to get that kind of thing we need to make sure that we’re delivering the value of the language to businesses, so that it can be you know making it to that kind of scale and, and… you know, again like making a big impact on people’s lives. You know, we’ve made this great thing, we need to see it in action.
JB: All right! I think that’s a very, very inspiring answer. Thanks a lot for this interview. I enjoyed it a lot, I hope our listeners will too…
RT: Yes me too. Thank you so much.
NV: Thank you Ryan.
JB: I hope all the people who want to work for a cool Haskell using company will now pay attention to the job page on your website. All right, hope to see you at um ZuriHac or ICFP or any of these and yep, thank you.
RT: Yeah, I’m looking forward to it.
