Building the Lucky Web Framework in Crystal with Paul Smith
Play • 1 hr 9 min
Paul Smith is a Software Engineer at GitHub and the creator of the Lucky web framework. He previously worked at heroku and thoughtbot and has experience building applications using Rails and Phoenix. He's also the creator of the Bamboo e-mail package and the co-creator of the ExMachina test data package for Elixir.

We discuss:
  • The tradeoffs of object oriented and functional programming
  • How a lack of compile time guarantees slow down ruby and elixir development
  • Creating conversational error messages
  • Ways fast languages can change how you write applications
  • Writing templates with Crystal instead of HTML
  • Choosing what to include in a web framework
  • The Crystal community and ecosystem
Related Links:This episode originally aired on Software Engineering Radio.

Transcript:

You can help edit this transcript on GitHub.

Jeremy: Today I'm talking with Paul Smith.

Paul is the creator of the lucky web framework and he currently works at GitHub. Today, we're going to talk about the crystal programming language and the lucky web framework. Paul, welcome to software engineering radio. 

Paul: Thank you so much. Happy to be here.

Jeremy: There are a lot of languages for software developers to choose from. What excited you about crystal? 

Paul: Yeah, that's really interesting because when I first saw Crystal, I actually was not interested at all. it basically looked like Ruby to me. And so I just think, okay, so it's a faster Ruby. And typically if I want to learn a new language and want something that feels really different, that pushes the boundaries on things.

I started getting more interested in compile time guarantees. I worked at thoughtbot previous to github and previous to Heroku and people were starting to get really into typed languages. Um, some people were starting to get into Haskell, which is like, you know, the, the big one that, I guess is probably one of the more type safe, but also hard to use languages.

Um, but also Elm, which has a good focus on developer happiness and productivity and explaining what's going on. And as they were talking about, how they were writing fewer tests and it was easier to refactor, uh, it started becoming clear to me that that's something I want. Um, one of the things somebody said was, if the computer can check the code for you let the computer do that rather than you, or rather than a test. so I started to get really interested in that. I was also interested in elixir, um, which is another fantastic language. I did a lot of work with elixir. I built a library called bamboo, which is an email library. And another called ex machina, which is what a lot of people use for creating test data. Um, so I was really into it for awhile.

And at first I'm like, wow, I love functional. And then I realized like. I can do a lot of, like a lot of the stuff I like about this I can do with objects. I just need to rethink things so that it uses objects rather than whatever random DSL

Jeremy: Cause I mean, when you think about functions, right? Like you've got this big bucket of functions and you got to pass in all the parameters right? Whereas, you know, in a lot of cases, I feel like if you have those instance variables available in the object, then the actual functions can be a lot simpler in some ways.

Yeah.

Paul: Totally. That's like a huge focus and making the object small so that it. It doesn't have too much, but that's how I began to feel with elixir is that I'm like, I just have 50 args and most of them I don't care about. Like I want to look at what's important to this method, to this method.

It's, you know, this argument, but with functions you're like, which things important. Is the first thing? Probably not. That's probably just the thing I'm passing everywhere. And so I liked that ability to kind of focus in and know like, this object has these two instance variables everywhere.

Jeremy: Yeah. It's kind of interesting to get your perspective because, it seemed like you were pretty deep into elixir if you had created, bamboo and ex machina and stuff like that, so it's kind of

Paul: Yeah. I was like way gung ho and, and then I started missing objects. And luckily with crystal and ruby, you still get a lot of the functional stuff. Like you can pass blocks around. Um, that's functions. You can use functions. But it's not the other way in Elixir, you can't use objects. It just doesn't exist.

And then the type safety. I'm just like, I still run into so many errors and it was so frustrating. I don't want to do that.

The main benefit I got out of elixir compared to rails, um, which is what I had been using and still use a lot of, was speed. That was really big. Um, in terms of bugs caught about the same, mostly because it's still for the most part dynamically typed language with very few compile time guarantees. Um, so I'd still get the nil errors. I'd still mess up calls to different functions and things like that. And so that's where I ran into crystal. It has the nice syntax. I like from elixir and Ruby. It's also very, very fast. Faster than go in some benchmarks.

So it's quick. Plenty fast for what I need. 

And it has those compile time guarantees, like checking for nils. That's a huge one. and it also makes the type system very friendly. So it does a lot of type inference. And very powerful macros so you can reduce some of the boiler plate.

And so that's when I kind of started getting into crystal was seeing Elixir  I still got a lot of these bugs that I was running into with rails, but I liked the speed but I don't want to use Haskell and Elm doesn't exist on the backend. so I started looking at crystal.

Jeremy: And so it sort of sounds like there's this spectrum, right? You have Ruby and you have, elixir, where you don't necessarily specify your types so the compiler can't help you as much. And then you've got Haskell, which is very strict, right? You have a compiler that helps you a lot. Um, and then there's kind of languages inbetween Like. For example, Java and C and things like that. They've been around for quite some time. how does crystal sort of compare to languages like those ? 

Paul: Yeah, that's a great question cause I did look at some of those other ones. TypeScript for examples is huge. Kotlin was another one that I had looked at because it's Java but better basically. That's the way it's pitched. And so far everyone that's used it has basically said that.  And also looking at rust, what it came down to was how powerful was the type system. So crystal has union types, which can be extremely helpful, um, and it catches nil. Java does not have a good way to do that. Um, Kotlin does. But also boiler plate and the macro system crystal's is extremely powerful. Elixir also has a very powerful macro system.

But crystal's is type safe, which is even more fantastic. So basically what that let me do with lucky, it was build even more powerful type safe programs. And we can kind of get into that once we, we talk about lucky and how that was designed. Um, but basically with these other languages, a lot of what we do in lucky just simply wouldn't be possible or wouldn't be possible without a significant amount of work and duplication.

Jeremy:  You covered a few things there. One of the things was, macros, what are are macros? 

Paul: Yeah. This is like a confusing thing. It took me a while to, to get, um, what it is. But, uh, in Ruby, for example, they have ways of, of metaprogramming. That are not done at compile time for most compile time languages, compiled languages, I should say.  You need macros to de-duplicate thing, and basically what a macro does is it generates code for you.

The way I think about it is basically you've got a method or a macro, but it looks like a method. It has code inside of it. And it's like you're copy pasting, whatever's inside of that macro into wherever you called it from. So in other words, rails has a, has many, like has many users, has many tasks that's generating a ton of code for you.

So that's how Ruby does it. Um, and crystal has many would be a macro and it would literally  generate a ton of code. And copy paste that into wherever you called it. Um, so it's just a way to reduce boilerplate.

Jeremy:  So in the case of dynamic languages, like Ruby, when you talk about Metaprogramming, that's having I guess, a function that is generating code at runtime, right? And the macro is sort of doing something similar except it's generating that code at compile time. Is that kind of the distinction? 

Paul: That's the way I look at it. there are people much smarter than me that probably have a more specific answer about what the differences are, but in my mind and in practical usage, that's what it comes down to in my mind.

Jeremy: Let's say there's a problem in that code, what do you get shown in the debugger? 

Paul: Debugging macros is definitely harder than debugging your regular code for that exact reason. it is generating a code. So what crystal does, uh, there's different ways of doing this, but I like Crystal's approach. It'll show you the final result of the code and it'll point to the line in the generated code that caused the issue and tell you which macro generated it. Now, it's still not ideal because that code isn't code you wrote, it's code that the macro generated, but it does allow you to see what the macro generated and why it might be an issue.

Part of that can be solved by writing error messages and error handling as part of the macro. So, in other words, making sure, if you're expecting a string literal, you can have a check at the top that checks for it to be a string literal. I wouldn't use them by default, but it's great for, I think a framework where you have a lot of boiler platey things that you're literally typing in every single model or every single controller, and that people kind of get used to. It's well tested. It has nice error messages. In my own personal code though, I pretty much never used macros. They're only in the libraries that I write.

Jeremy: Another thing you mentioned is how crystal helps you detect Nils or nulls. Um, how does, how does the language do that?

Paul: It actually uses union types for that, some languages that have this, they'll have an optional type, which is basically a wrapper around whatever real type, like an optional string, optional int, and you have to unwrap it. The way crystal does it is you would say string or nil, and there's a little bit of syntactic sugar.

So you can just say string with a question mark at the end. But that gets expanded to string or a nil type. Um, so then within that method, the compiler knows that this could be a string, could be a nil, and there's a little bit of sugar there where the compiler, if you say, if whatever variable you have, it's going to know that within that, if it is not nil and in the else it is.

So there's a little bit of sugar there as well. Um, but that's basically how they handle it. And there are ways to force the compiler, uh, just say, Hey, this thing is not nil you can call not nil on it. That's a little, I would avoid that because maybe the compiler's right. And it really is nil. Or maybe you change the method later and then it can become nil and you're going to get a runtime error there.

But it does have those escape hatches. Cause sometimes you just need the quick and dirty and you can, if you need to.

Jeremy: As long as you don't tell the compiler that, then you will actually have a compiler error. If you have a method that takes in, let's say some type of object or a, a, nil. And then you don't account for the fact that like it could be nil. Then the compiler actually won't let you compile, is that correct?

Paul: That is correct. So for example, if you just had a method that's like, print. email and it accepts a user or nil, now, I'm not saying I would do that, but let's say that it does. And you just tried within that method to do user.email to print the user's email. Um, it's going to fail and tell you that nil does not have the method, email.

And so you need to handle that. And then, yeah, you're forced to either do an if, or for example, you can use try, which is basically a method that says call call a method on this object. Unless it's nil, if it's nil, just return nil. But yes, it kind of forces you to do that.

Jeremy: And in crystal, how do you handle errors? Because a lot of different languages, they'll have things like exceptions or they may have result types.  What's sort of the the main way in crystal?

Paul: I'd say I'd group it into two types of errors where. You have runtime exceptions still because things do break. Not everything is in a perfect world. Inside your type system, databases go down, you know, redis falls over or whatever. So you still have runtime exceptions and then you have the compile time errors, which we kind of just talked about.

But in terms of how those runtime exceptions are handled it's I don't want to say exactly the same as Ruby, cause there probably are some subtle differences, but extremely similar to Ruby and that you're not passing around errors. It's so, it's not like go where you are explicitly handling errors at every step.

Um, you raise it and you can rescue that error kind of like a try catch in other languages and you can also just let it bubble up and rescue at a higher level, which I personally prefer. Because not every air is something that I care about and kind of forcing me to handle every single error everywhere means that it is harder as a reader of the code to tell which errors I should care about because they're all treated as equal.

So I like that in crystal, I can say this particular error, this particular method I want to handle in a special way. And somewhere up above the stack. I can just say anything else. Just print a 500 log it, send it to Sentry.

Jeremy: Yeah, so it's very similar to, like you said, Ruby, or any other language that primarily relies on exceptions. Like I think Java for example, probably falls into the same category.

Paul: probably. I haven't used it in quite some time, but I imagine it would be similar.

Jeremy: You had mentioned that that crystal is like pretty, pretty fast compared to other languages.  what are the big. benefits you've gotten from that raw speed?  

Paul: The biggest benefit I would say is not having to worry so much about rendering times, and rails for example. You can spend a ton of time in the view, even though everyone says databases are slow, they're not that slow in something like rails active record takes a huge amount of time to instantiate every single record.

So how does this play out in real life? You could, for example, in lucky if you wanted to load a thousand records and print them on the page and probably do that in. a couple hundred milliseconds maybe, which is a totally reasonable response time. Same thing in rails would be many seconds, which is not reasonable in my opinion.

And this can be really helpful, partly because it just means your apps are faster, people are getting the response as quickly. But also because you have a lot more flexibility. I've built internal tools where they want to have the ability to search all of the inventory or products or whatever else and they want to have like a select all or be able to select everything.

And in rails, you can't just render all 1000 products cause it basically falls over and you can try and cache stuff. But then that gets complicated. Um, so you kind of have to paginate. But when you paginate that makes it hard to select things across multiple pages, it's then you need some kind of JavaScript to remember which ones you selected across pages, and it just balloons the complexity, right?

If you know, Hey, we only have eight or 900 products, we're not going to suddenly have 20,000 in lucky. You just render them all, put them all on the same page, give them all check boxes, and it's. In the user's hands in 200 milliseconds and you're done. You just removed most of that complexity. So those are some of the ways that that speed is playing out. And I think one key difference there is some people think speed is just about scalability. How many people can be using this? The speed improvements I care about are the ones where even if you have one request per day, I want that request to be insanely fast. and so that's kind of what you're getting with lucky and crystal.

Jeremy: When you talk about web applications, you know, with lucky being a web. Framework. A lot of people point out that a lot of the work being done is IO, right? It's talking to the database, it's making network calls. But I guess you're saying that  rendering that template, those are things that actually having a fast language, it really does make a big difference.

Paul: It does. Yeah. I, I think the whole database IO thing, a lot of times that's what people say when they're working with a slow language. If you have a fast one. It's not as big of a deal. Cause this was the same with Phoenix and Elixir. Like I loved, how quickly it could render HTML. That was huge.

Jeremy: And like you said, that opens up options in terms of, not having to rely on caching or pagination or things like that.

Paul: Yeah. This is huge. I mean, an example from work. We just announced github discussions. Um, and I'm on that team. And one of the big things we were trying to get working was, was performance of the discussions show page. You could have hundreds of comments on that page. And we were finding that most of the time taken was actually spent rendering the views and calling methods on the different objects to render things differently in the seconds. And we can't cache those reliably because there are so many different ways to show that data. If you're a moderator, you get certain buttons. If you're an unverified user, like someone who just signed up, you see a different thing. If you're not signed in and you see a different thing, and so you can't reliably cache those, and we had a lot of cool techniques to kind of get that down, but this is something that if this were written in lucky, it just would not have been an issue.

Jeremy: And github in particular is written in Ruby, is that correct?

Paul: It is. Yeah. It's using Ruby on rails, and I'm not trying to knock rails. I, I really love rails. I mean, I've been using it for 12 years. Um, I like Ruby. Uh, but Hey, if there's something that could be even better, I'm open to that.

Jeremy: For sure. You have used Rails for 12 years. how would you say that your productivity compares in Ruby versus in crystal?

Paul: I think that's tricky. It's kind of better and worse. And what I mean by that is. I think crystal, I am. I'm more productive. And crystal, you do have compile times and we can talk about that. They're not the fastest, they're not the slowest, but I do find that I can write more code and then compile once, and it kind of just tells me where the problems are and I have a lot more confidence and I spend a lot less time banging my head on like, why isn't this thing working?

And it's because I passed the wrong type somewhere. however, Ruby has a massive ecosystem, so there are things that exist in Ruby that I would have to rewrite and crystal. and so that for sure, no matter how productive I am in crystal, is not as productive as requiring the gem and then just using it.

So the hope with lucky though, is that we're building up enough things that. You don't have to be rewriting everything. And the community is also really stepped up and writing a number of, libraries that are super helpful for web development. Um, for example, somebody just wrote web drivers.cr, which makes it so that it can automatically install the version of Chrome driver that matches the version of Chrome that you have installed.

So you don't have to manage that at all. That's something that was in Ruby for awhile, and will be in lucky, probably in the next release. So yeah, I think it's better. It's one of those things that will get better with time.

Jeremy: So in terms of the actual language, productivity, crystal, it sounds like basically a net positive, but it's more in the the community aspect and how many libraries are available. that's where a lot more time, but it's taken.

Paul: I think so. And then just the initial ramping up, uh, it is a new language and so there aren't as many stack overflow questions and answers and there aren't as many tutorials. So there's definitely some things there. But like I said, those are things we're working on, especially for one out of lucky. Try and make sure we have really good guides, uh, really good error messages.

We tried to borrow a little bit from Elm. Not specific error messages, but just the idea that an error message should raise something human readable and understandable, and if possible, help guide them in the right direction of what they probably want to do, or at least point them to documentation to make it easier.

So we're trying to help with that as much as, as we can.

Jeremy: I kind of want to move into next more into your experience. building lucky. you know, you were a rails developer for many years, and  are there any like specific major pain points, I guess, in rails or in your previous web development experience that you wanted to address with lucky?

Paul: Yeah. There were, um, some more specific than others. Um, some easier to solve. In the sense that the solution is like it works or it doesn't. And others that are a little bit more abstract. So I'll talk about some of the specific things. I often said that I'm into type safety. I don't think that is quite true, and I think it. Especially if you haven't used lucky, it just doesn't click what that means or why it matters. Cause you just think like, Oh, so you know, don't tell me if I pass an integer instead of a string. Like who cares? I'm not seeing those kinds of errors.

What I'm most interested in is compile time guarantees, whether that's with a type or some other mechanism. and that's there, not just to prevent bugs, but to help you as a developer to spot problems right away. And give you a nice error so you know what to do about it. So, for example, one of the things that I've seen in basically every framework I've ever used, regardless of whether it is type safe or not, is that you need to use an HTTP method, a verb and a path.

So, for example, if you want to delete a user, you would have forward slash users forward slash one to be the ID. The tricky part is you have to have the HTTP method delete for it to do the delete action. But sometimes you forget that you use a regular link and you wonder why the heck it just keeps showing you this thing instead of deleting it or the particularly insidious one is when you have a update and a create. One uses post one uses put, if you have an update form and you forget to put the method put, you get all kinds of routing errors cause it says, Hey, this doesn't exist. And you went, well why? Why doesn't this exist? I can see it right here. I've got the route, I've got everything.

Oh it's cause I forgot to put the HTTP method is a PUT. And it just waste time. So that's one of those things where we wanted to compile time guarantee and lucky. And so I don't want to go too in depth here, but basically what we did was we made every controller into a single class that handled the routing and also the response.

Jeremy:  If I understand correctly, when you have a page. And you want to link to a specific user, on that page. Then you would use this function link to, and you would pass in the class that corresponds to showing a user, and then you would pass parameters into that function. Like, for example, the id of the user.

And if you didn't do that. Then you would have an error at compile time, not

Paul: correct.

Jeremy: you. You wouldn't need to like start the website and then go to the page and have it, basically explode, which I guess is typically what you would expect from most web frameworks.

Paul: Or what's worse, it wouldn't explode. It would just generate the wrong link and you would have to remember to click that link or write an automated test that clicks that link. And so it's really easy for bugs to sneak in, and this just completely prevents that class of bug. As well as just makes life easier because if you forget a parameter while you're developing from the start, instead of just generating something with like a nil ID, it's going to say, Hey, you forgot this.

It just saves a lot of debugging time, and I think it's also more intuitive if you've ever used rails helpers or Phoenix, help any of these man the conventions. Like it's a singular, isn't plural, is it? Does it have the namespaces and not have the namespace in lucky that it's gone. You just call the action, the one that you created, you call that exactly as is.

Jeremy: It sounds like this is maybe a little more explicit, I guess? 

Paul: Yeah, it's a little more explicit, but I hesitate. I've heard a couple of things in the programming community. Um, one, the rails started as convention over configuration, which that was huge because you had to learn the convention, but at least once you did, you knew how about other rails projects were. And then another one I hear is explicit over implicit.

I don't buy into either of those in particular. Um, because sometimes implicit is better, sometimes explicits better. I mean, for example, it was a quick example. I don't hear anyone arguing to bring back the old objective C where you had to manually reference and dereference memory that is technically more explicit.

But does anyone want to do that? No. So I don't think explicit over implicit, you have to think about it. Everything needs to be judged, in its own context. And what I think is even better than convention over configuration is intuitive over inventions. Meaning you don't even think about it.

You don't even need, there doesn't need to be a convention. Because you're literally just calling the thing that you created like anything else, there's nothing special about that. It's a class just like any other class and you call a method on it, just like any other method.

I think it's tricky because I think it's also easy to say explicit over implicit and make your code super hard to follow. And it's like, yes, it's more explicit, but also I just wrote 20 lines of code instead of one. And those 20 lines could differ because I do it differently than the other guy or girl.

Jeremy: Another thing about lucky that's a little different is that for templating, instead of having somebody write HTML and embedding language code in it, uh, you instead have people write crystal code.

So could you kind of explain sort of why you made that decision and what the benefits are.

Paul: Yeah, sure. So a lot of things actually with, lucky. Kind of I did not want to do, or were definitely not how I started doing things. And it just kind of moved in that direction. based on the goals. And I think that's part of what makes lucky, different is that we don't say, here's how I want to do it.

We say, here's what I want to do and I want it to be easy, simple, and bug free. So. What we started with was using templating languages, just like you'd use in almost any, anything where you write your HTML and then you interpolate values in it. At the time I wrote lucky, and this may be changed now. you could not use a method that accepted a function or a block is what it would be called and crystal, and have that output correctly in the template. I think it just blew up. I don't remember, this was two years ago, three years ago. The other problem I was having was, it's not just a template. Any bigger size framework also has partials or you know, fragments or includes or whatever you want to call it. It also has layouts where you can inject different HTML in different parts of your HTML layout, and those are all things that a person has to learn when they're learning your framework. What are these methods called for.

Generating a partial for calling a partial or injecting stuff in different layers of the layout. And it's also more stuff that I have to write. And with lucky, like there was already a lot to write. They were building the ORM and the automated test drivers and the router and like everything. So I can't afford to just do stuff like everyone else does it if it's not pulling its weight.

So eventually. I started experimenting with building HTML, using classes and regular Crystal methods. Some of the requirements, um, for me when I was building it was it had to match the structure of HTML  and it had to be very easy to refactor. Meaning I can pull something out into a new method and it just works.

So easy refactoring. And then I also need to be able to do layouts with it. The reason for that is Elm also uses, code to generate HTML. However, it is not approachable to a newcomer. if for example, you have a designer and they pull up in an and try and look at what that, what that generates.

No way. I mean. I'm a programmer, I still don't know what it generates without really looking through Elm. And that's partly because you are generating data objects. So arrays of arrays. Or maps or whatever else. so I didn't want that. It has to be approachable to people and look and be structured like HTML.

And so we were actually able to do that. I don't know if I need to go into huge detail, but basically you can say, Hey, I want to div. Inside of that, I want an H1 underneath that. I want another div. And you're not building arrays and maps and anything else. What that provides is actually a lot of things that I did not think of.

One super easy refactoring. If you have a link in a particular page and you don't want to copy that over and over and over, extract a method and you call it like any other method, there's nothing to learn. It's just a method. Like anything else, it can accept arguments just like anything else. Your conditionals work.

Um, you can extract that into a component, which is basically another class and it tells you explicitly here's what I need to run. And it renders the thing. Um, you always have the correct closing tag. I have been bitten so many times by shifting stuff around. And forgetting a closing tag and my whole page looks wonky and I have to go through layers of indentation.

That just doesn't happen if you forget an end so you would have a do end when you're creating these blocks, it blows up. It's like, Hey, you're missing one. And the coolest part is you just add an end in there and you've run the crystal formatter and it re indents everything perfectly. And then on top of that, it's, if that wasn't enough.

Like I just loved how easy it was to refactor and use. you don't have to split up your code from your template. Like in rails, you would have a helper. So you've got like, here's your template, but then you might have a helper, a totally separate file. If you've got something that pertains to just that page, you can just extract a method.

It's right there. But this also made it so we can do layout without any special work. Your layout is basically a class. You would say, here's my class with the head. It renders the head renders HTML body or whatever. And then it calls a content method or a sidebar method or whatever else, and your page.

So if you wanted to render a list of users inherits from that class and implement a content method or a sidebar method. And so when that's rendered out, it just calls those methods. So we got all of that for free. If you look at our view rendering code, it's 50 lines. because basically we use a macro and give it a list of tags, like, you know, paragraph H1 H2 whatever, and generate a bunch of methods.

And that's basically it. So from an implementation perspective, it's extremely simple. Plus, you get all these niceties around  refactoring is super easy. It's super easy to tell what a page needs to render at the top of the page. You just say, you know, I need a user. I need a paginator. I need a current user.

So you know what that page needs. You don't get that with a template. and you get all the power of crystal for rendering layouts however you want. that all basically came for free. So it was kind of a happenstance that templates weren't working and this has worked out better people, a lot of people when they see this, they're like, what the heck is this?

I hate it. And I always just say, just give it a try. Just give it a try for a little bit. So far. One person has said like, okay, I don't like it, and you can use templates if you want. We've actually built that in, but everybody else is like, now that I've used it, I love it.

Jeremy: What it sounds like is in a lot of, JavaScript frameworks, for example, like react, there's this concept of components, right? And so you can create, what looks like new HTML tags, but really has. some other HTML in it like let's say you have a a list of, uh, businesses and maybe you have a component that would have, all the business details in it. it sounds like in the case of lucky, you kind of can do the same thing. It's just that your component would be in the form of a crystal class. And so there isn't any new syntax. and you're not mixing, different languages. Like you're not mixing HTML and JavaScript. Instead, everything is just using crystal.

Paul: exactly. you have two options. You can extract a private method cause sometimes it's just a small thing you want to extract only used by one page. Just do a method. If not. Uh, extract a class. And the cool part about all of this is that you don't need to restructure anything. Meaning you can start with everything in one method, in your content method, and then you can pull out just a little bit into a private method.

And then if that's not enough cool, pull that out into a class so you're not forced into just pulling out classes all over the place if you don't need one.

It really worked out kind of really well because it also makes testing easier. You can pull out a class component that just does one thing and you can instantiate just that component and test just that HTML. And once again, this is very easy because it's a class you call it and run it like any other class.

And so that's been a big goal of Lucky is try to reduce, and this also comes down to the whole like convention over configuration is how do we just make it so there is no convention. It's just intuitive. Like if you know how to extract and refactor a crystal class, you know how to extract and refactor stuff for a page in lucky automatically. Um, and I mean, of course there's still some degree of learning and experimentation, but it's the same paradigms. if you want to include methods in multiple pages, use a module just like any other module. So that was very much a goal. 

And that's part of, uh, other parts of lucky, for example, querying in something like rails. The model is for creating, updating, reading, everything. In lucky you do create a model and we use macros to actually generate other classes for you, but you have a query object that is a class. 

Jeremy: What am I passing into my query object what does that look like? 

Paul: Let's say you have a user by default, it generates a User::Base query. So basically you have this new object namespace under the model. And by default, the generators generate an another file.

And basically what that does is it creates a new class called user query. That inherits from that user based query class. What you would do in your controller action or anywhere, uh, say user query dot new by default. That just gives you a new query that would query everything in the database. Unless of course you overrode initialize and did something else. Then it would use that scope. so if you want him to further filter down, you would call, for example, if you wanted the name to be Paul, it would be user query dot.

new.name parens Paul as a string. Because lucky generates methods for every column on the model with compile time guarantees. So if you typo that method, it's going to blow up. If you've renamed the column later, it's going to blow up. if you accidentally give it nil, it's going to blow up and tell you to use something else, but that's how you would do it.

You say dot. Name is Paul. Or, uh, we also have type specific criteria and methods. You can do things like dot age. Dot. G T for greater than 30. And so you have this very flexible query language that's all completely type safe. So in your scopes, if you wanted to do something like recently published for a post, inside that method, you would do something like published at dot gt at.gt for greater than one dot week dot ago.

And you can chain that. So  you could do post query.new dot. Recently published dot, authored by Paul or whatever. So that's basically how it works. Um, you just have these methods that are chained, that you can build upon in pretty much any way you want.

Jeremy: In a lot of applications now, people use JavaScript frameworks, whether its react or Vue or angular, what does integrating with JavaScript libraries and frameworks look like in lucky?

Paul: I think easier than a lot in the sense that you can generate a lucky project with. different modes. So when you initialize a project, you can use just the command line with some flags, or the default is to walk you through a wizard, which will say, do you want API only? In which case, you know, it won't even have HTML pages or the default, which is a full app.

What that does is it generates Webpack config for you. Um, it sets up your public assets and images so that they can be copied and fingerprinted. and so out of the box that already has a basic web pack set up for you that handles CSS. Um, it handles most of your ES6, JavaScript type stuff that people typically like.

That's just handled out of the box. if you want to include react or vue. You would include that just like any other Webpack project in terms of building it. Um, and it's actually a little simpler. We use Laravel mix on top of Webpack, which is basically a thin JavaScript layer that calls Webpack underneath the hood.

If you want a full single page app. That's also totally supported. Um, you would basically have just one HTML page that, you know, has the basic HTML and body tags and within that Mount to your app. So whatever that is for your language in vue, it might be, just a tag that's like main app.

And then in your JS you would initialize, um, that tag with your app. And we have fall back routing so that you can do client side routing if you want. It's not particularly well-documented, which is the biggest problem. Um, some people are helping with that cause a number of people have done react and view.

And so, um, hopefully those will be fleshed out a little bit more, but it's totally supported. in the longterm though, we've got plans to make it, so you don't even need those types of frameworks quite as much. since we already have class components and a bunch of other things, uh, I'm working on a way to add type safe interactivity to HTML.

So you're not writing the Javascript, you're writing crystal for the most part, and it can interface with Javascript and you can run, you know, use react and vue inside of it. But a lot of your simple open close, if anything like that is going to be handled client side, but written with crystal and server interactions will also, those will be sent over an Ajax request, but will also be typed safe when you call the actions and do all the HTML rendering similar to live wire for Laravel or live view by Phoenix. But with some. Some differences that's not done yet, but it will be, and I think it's going to be really exciting. I've got a proof of concept up locally and uh, it's really awesome.

Jeremy: We had a previous episode on live view and I think the possibilities of things like that are really interesting of, of being able to not have to have this sort of separation between your JavaScript front end, and your server backend yet still be able to have, the kind of interactivity people expect.

Paul: Yeah, I think it could be cool. and that's also where speed comes into play. When you're doing interactions like that, you don't want to wait even a hundred, even 50 milliseconds. Is noticeable for those types of interactions. And so Phoenix also fast, really fast, template language. Uh, basically it gets compiled down to elixir, and so that helps a lot.

Um, I do think there's some big flaws that I've seen in some other implementation. Well, I don't want to say flaws, that sounds a little overly harsh, but things that I personally, are just deal breakers for me. And one of those is some clientside interactions have to be instantaneous. I just have to be, if I click on my avatar on the top right, I expect the menu that has settings and log out to be instant.

If there's any kind of latency in the network and it takes 200 milliseconds, even. That's going to be a weird interaction and it's going to feel like your app is broken. And of course that's exacerbated by people, not in your country. This is another problem. People are doing these things, deploying servers in their own country.

Put a VPN in front of your computer in Australia or even the UK, 400 milliseconds. That's just, you can't do that for a settings menu or for opening a modal. And so there needs to be some way to do those interactions instantaneously. Live wire by Laravel, the same guy that wrote it, built our Alpine JS.

Which is kind of, it looks a little bit like vue, but it doesn't have a virtual, DOM it operates with the Dom that you generate. That's what it uses for client side interactivity. So you can do the server side stuff, which I mean, if latency's there, you're, if you're submitting a comment, look, there's no way around it.

You've got to hit the server. But if you're opening and showing something at a menu, a tab, a modal. That's instantaneous and is handled by Alpine. So lucky actually going to use that along with our own server rendered stuff to do client side interactions instantaneously.

Jeremy: So Alpine, it's a JavaScript front end framework, you said, similar to vue. without the virtual Dom, and it sounds like what you're planning is to be able to write crystal code and have that generate Alpine code. Is that right? 

Paul: That's correct. Cause it's mostly in line and it can't do everything. But most of what I want from client side interactions are typically super simple things. I want to open and close something. I want to show tabs. And those are things that Alpine's incredibly good at because you don't need a separate JavaScript file.

We can just generate something that says, it uses X as it's kind of modifier X dash click toggle the thing. True or false, toggle open to true or false and X if or X show and then if it's open or not. Those are things that we can very easily generate on the backend and make type safe because we can say, you know, this has to be a boolean and here's the action.

And all those things are then type safe, but you can still do JavaScript if you want, so you can still use JavaScript functions in there with your Alpine if you need to.

Jeremy: Yeah. That just sounds like the distinction between that and like a live view or a live wire is that my understanding is with those solutions you're shipping over basically diffs in your HTML, and that's how it's determining what to change. Whereas you're saying like, you may still have some of that but there's certain interactions where you just want to run JavaScript locally on the person's client, and you should still be able to do that even if you are doing this sort of sending diffs over the wire, for other things.

Paul: Yeah. Exactly. Alpine's made for that. The biggest key differentiator between Livewire live view is the type safety, all those nice things that you get in lucky you're going to get also for your client side interactions. So if you have an action and you have a typo or something, it's going to blow up.

It's going to tell you if you forget something, if you've missed the wrong type. I mean, and this is something that's very hard in the front end world because you either have to run an automated test to make sure you catch these or the worst. You have to open up the console. Because like, why isn't this working?

I don't know. Now I have to dig into the console. It's not even where you typically want to see logs, and so being able to shift that to where you're used to seeing errors and before you even have to open the browser, I think that's going to be a huge deal.

Jeremy: I think on the server side, testing is pretty well understood in terms of, you know, especially if you have API end points, or you have  just regular server code, like people know how to test that. But on the client side, there's like so many different ways of doing it.

It feels like, and a lot of them involve spinning up browsers and, um, it can get kind of complicated and so, yeah, it'll be interesting to see if you can shift more of that to the, the server environment that a lot of people are used to.

Paul: Yeah, I think it will be cool. We'll see how it goes and yeah, I do think there's definitely complexity that comes with moving it to Javascript, especially if you have a single page app cause then you need to spin up an API. You need the the server and an API. When you use your Cypress tests or whatever, or a lot of people mock the API, which sometimes is fast, but can get out of sync, in which case you lose confidence in your tests.

So having it in one spot, is I think really great. And we do have the capability to run browser tests that's built into lucky because I think it is still good to have at least a couple smoke tests for your critical paths. To test the happy path. Um, but I mean if you can write fewer of those, that's great cause they take forever to run.

Jeremy: For sure. Yeah. Um. In lucky, there's a lot of features that in other frameworks would be not usually be included. Like for example, there's authentication. you have this setup check script to see if your app has all of its dependencies, things like that.

I wonder if you could sort of explain sort of how you decided what sorts of features should exist in the framework versus being something that you'd leave to the user to decide.

Paul: I think things If there's no downside for one thing, if there's no downside, only upside and almost everyone would benefit from it, I want to include it. So that's, for example, the system checks script. Um, we also have a setup script and that's what we tell people to use. Instead of saying like, first installed yarn and then run your migrations and blah, blah, blah.

No our documentations don't even mention that. It's like run script set up. Um, and the idea there is, it serves as kind of a best practice. It kind of pushes you into things to say like, Hey, put stuff that you need in here. Then we lay it on the system check, which also runs before setup. And also every time you boot the development environment, um, where it'll check, Hey, do you have a process manager?

Which you need. It'll check whether Postgres is installed and running, because that's required. so if you go back to kind of that criteria, it's useful to pretty much everyone. Meaning like, if Postgres isn't running and the app's not going to work, everyone would need to know that. Um, and it doesn't really have a downside.

if you don't want it for whatever reason, you just delete it. Or stop running it, that's not a huge downside. That's like, you know, one click. So that's part of why that's included. I don't like spending time on things that aren't delivering actual real value.

So I don't like spending time figuring out why my local environment is not working or why it was working and now suddenly isn't. And with something like a system check that makes teams happier in the sense that, let's say all of a sudden ads somebody adds a new search capability and it requires elastic search, and I do git pull from master, do my feature as soon as I boot the app, if they've added something to system check that says, Hey, you need elastic, it's going to tell me it's not going to just blow up.

It'…
Search
Clear search
Close search
Google apps
Main menu