Web Apps in Clojure and ClojureScript with Pedestal Brenton Ashworth

hello my name is Brenton Ashworth I work for cognate echt formerly relevance and this talk is about web applications ring web applications in closure and closure script with pedestal I apologize for the really long title and the ambiguity of the title when I submitted this talk I wasn't quite sure where pedestal would be and it's changed quite a lot since since I submitted the talk so just to make it clear where we're going I'm I'm pedestal has two parts it has a service part and a client part during this talk we're gonna focus on the client side of pedestal and it's designed to build a certain kind of application we're going to take a look at the kind of application that it will help you build and we're gonna look at an example application that we built for a client and then after seeing that and having a good example in our heads we'll take a look at how we use pedestal to build that example and then we're gonna take a look at the future there are a lot of interesting things happening right now and we're trying to incorporate a lot of new ideas into pedestal so if I could pick one word to describe the kind of application that pedestal is designed to help you build it would be interactive applications and specifically this definition that you see up here this partial definition applications which require that there be a continuous two-way transfer of information a lot of the applications that we've worked on at relevance in the last two years that I've been there have been this kind of application and we found that we need something to help us out if you decide to make this kind of application there's a lot of implications for how you need to think about developing it and even the artifacts that you'll create we can no longer just assume that the person sitting in front of the screen using the application is going to be the only source of events that we need to respond to or react to in the user interface we receive inputs and events from multiple sources and we need to somehow coordinate all this information into a coherent picture and display it reasonably to the user and this requires that we do coordination on the client that we used to just do on the server and so we need something to help us out there when we do this on the server we have a database and we usually do all our coordination there and when we have a lot of state and a lot of events coming into the client we need something to kind of play that role these applications don't really have like a clear page reload boundary so they run for a long time that means they're usually larger and and large from a code perspective they do more and it's a it's a lot more code so we need to actually apply real engineering effort into making these applications to make sure that they run well and and don't die on us so why in the world are we creating something new we need something besides just the language languages are important and we like closure script for these kinds of applications but in any like endeavor we usually use libraries to help us out and we think that something is needed here and we later in the talk you'll see some of the things that I think we need and how we implement those in pedestal and also there are a lot of existing libraries that we could use but I think that from our standpoint they either aren't well-suited for building this kind of application or they rely heavily on mutation and they have a lot of object-oriented concepts which I think unnecessarily complicate things so we want to build something that is more functional and will allow us to program in the way that we're used to in closure even when we're in in the browser and making large applications in the browser so the first thing we're going to do is take a look at an example application something that we built for a client we had a client that came to us and asked us to build a way to visualize their sales funnel and they wanted to see in near-real-time how people are progressing through the funnel and how and if there's any problems how they could they would be able to see how they could deal with those problems in real time and the system needs to support receiving about a thousand events per second from from the system that it lives within and it even though it's getting a thousand events per second it's it's updating the browser about twice a second so this is a very high-level overview of the architecture of the whole system I just wanted us to see how the thing that I'm going to show you fits into the rest we have a system that we deployed on AWS if you go from left to right we have collectors which are polling or having data pushed to them they're putting that data on a queue we have aggregators pulling data off the queue and doing some processing and sticking that data into day Tomic and then we have an analyzer which is being notified whenever a new data arrives in day Tomic and then pulling the new data out rolling it up and sending updates to the browser twice a second and so I'm going to show you a screenshot of the application and explain what all the pieces are and then we're gonna run it so hopefully you can see that and what we have is we have the states that people could be in and each state the size of the circle representing the state represents the number of people in the state around each one we have a donut that tells us how many people are instantaneously arriving in the state versus leaving the state the paths tell us how people Traverse between states and the bubbles are going along the paths are telling us how how many people right now are traveling between the states and then line pads on the top above the circles represent forward motion through the states and then calves on the bottom represent backward motion and if you look over there's one state that's got like a little bit of blood dripping out of it and that represents people dropping out of that state so we've also got some charts that show us the number of users currently in the funnel and how many how much inventory were selling and airs and then at the bottom we have the history of each state so I'm going to show you two versions of this one that works and one that doesn't work so well so we're feeding data into the visualization and we're feeding data at once every five seconds to start with and we're gonna ramp up to once or twice a second so you can kind of see how it how it works as we as we progress so it will order order the states as data arrives in the way that makes the most sense this is a pretty simple flow everybody's just moving through the states an order and we can see that the thicknesses of the lines represent how many people are actually traveling between each state and what we're gonna see here is that the if you can see the labels the shipping State is actually getting larger it's getting clogged and we can see that there's a big blue doughnut and a little bit of orange that means that there's a lot of people arriving in that state and not a lot of people leaving and so maybe the person watching this would get on the phone and somebody would push some buttons and somehow solve this problem we can also see that the sales is kind of suffering up there we've dropped off because not a lot of people are making it clear through to the confirmation state and now we've we've added some new ways to traverse the through the state's people can skip over shipping now so anyway that's just a simulation to show you how it works don't be afraid of all those airs up there it's it's just a simulation normally that much wouldn't be going wrong so I want to show you one version of this that doesn't work so well and we'll see if we can see how it doesn't work so it's the same exact code it's just running in a different way and we're all we're going to start sending messages once every 5 seconds and then ramped up to twice a second can you see this by the way okay so what you see here is there's a pretty significant pause and that pause is happening whenever we get an update and we have to process the data and figure out what to do with it so I don't know how many of you have run into this kind of problem when you have a lot of animations in the browser and you're trying to do other work it can be kind of tough to figure out how to solve it and sometimes we have to resort to breaking up our code into all these pieces with timeouts between them in order to make this run smoothly so that's the demo so this is what we're trying to build and there's really two hard problems here how do we draw this stuff that's hard and pedestal doesn't help you with that and then how do we control it there's a lot of state that we're getting in the browser a lot of messages and there's a lot of things to control on the screen even what's visible on the screen and how do we do that and that's the part that pedestal helps us with so what we did on this project is we divided the work into these two main categories we we use Rafael for the for the animations and we used highcharts for some of the charts and we created widgets we wrapped these things in two self-contained pieces of code that we could add to the page and remove and each one had a simple API so we could send data to it control it for example we could set we could tell the big chart you know make this state a certain size or this path a certain size and that all that code is actually written in JavaScript we had people working on this project with different skill sets and so the people that were helping some of the people helping with with the widgets I didn't know closure script so we just wrote these in JavaScript so we have widgets and we can we can they can draw the things that we want to draw but none of these things have any kind of application state in them they just draw draw things on the screen so we need to be able to control them we need to be able to determine when these things should be displayed and and that's what pedestal helps us do so pedestal at least in the context of this project provides four things that helped us out it provides a clear separation of concerns it provides an information model we're gonna look at these one at a time it provides an information model a state transition model and data flow so what do we mean by separation of concerns the thing that pedestal does that I think is very helpful is that it it clearly separates these three things rendering from your information model and logic and services and what I mean here by services is the part of your application that does communication with the backend so this might be the part that's got like an SSL can ssed connection or a WebSockets or as making HTTP requests to the backend we want to isolate that because it's it could be very complex and if you let that get intermixed with everything else you're gonna have problems and then the same with rendering we want to keep that isolated as much as possible these two places rendering and services this is where all the complexity of programming in the browser exists so we want to keep these things keep those isolated and then in our information model and logic that is where all that the application logic for our application lives that's where all the state lives and that's really very simple it's it's data it's data transformation and this is the kind of this is the kind of thing the closure is really good at we can we can run this code on in the browser we can run it on it on enclosure to test it that makes testing very easy so anyway how do we how do we separate these things we separate them with messages and queues so there's no way to what you can probably call across them but if you want to use pedestal you communicate between these parts by using by sending messages and you send messages on queues we don't make direct function calls across these boundaries so messages just give us a way to communicate without making function calls and then queues sever the direct connections so even if we if we didn't have queues but we had messages we'd still be determining when things run queues give us the ability to kind of step away from that and these cues are not real cues they're JavaScript style cues they're kind of simulated cues basically an array and an asynchronous processing of that array cues also give us the ability to independently process the data on the queue so we can take a look at our application and if the queue backs up we can decide we're going to start dropping stuff or we can maybe compact things and it's it's very nice to have them I just wanted to show you an example of the the kinds of messages that we would use I'm not going to go into the details of these messages but I just wanted to give you an idea of what they look like the top message is the kind of thing that we would send into our application and the bottom two are the kinds of things that might go out see the renderer and we would these would be mapped to functions which performs some small change and what we're in what we're rendering and so in this application we use widgets the functions that receive these messages would maybe make an API call to a widget to make it to make a small change notice that there is a similarity between these messages even though they're slightly different they all have an operation something we want to do they have a target the thing that we want to do it to and then they have arguments and in the future this is actually going to get standardized a bit so next we'll look at the fact that pedestal has an information model and what I mean by this specifically is is that we have a standard way to organize and store data if you if you didn't have this you could you could store data however you'd like or organize your data however you'd like but if you do that you can't really leverage leverage that you can't build things on top of it so having some kind of standard helps us out it's kind of like in a database you have schemas and you talk about types and that allows the database to do things more efficiently so the way that we store data is in a tree and you can just think of this as nested Maps that tree contains the basis information that we might receive from other sources that talk about things that we know these the facts the we that we have learned about we have derived information things that we derive from the basis information and we have some information that supports the UI so that's what we put in our data model or our information model so this is what it might look like this is a true representation of that it's a tree we see the nodes but we don't really see the values in the tree the thing that this allows us to do is talk about talk about nodes in the tree using paths so the one that you see on the bottom there is a path to a specific node and we might use that to say I want to update that subtree and the top message has a wild card in it we might use that to say that we would like to be notified if one of the children of the device's node is changed these are just the way that we can refer to nodes in the tree and this is an example of one thing that we can have if we assume that there's a certain structure to our data so another thing that we have are state transitions a state transition model and what I mean by this is that there's just there's some way to have an orderly succession of states our application state is stored in one one big tree and we want to know that when there's an input we're gonna have a new tree a new state and we want to know that we're progressing between States in an orderly way we're not updating things in a random fashion or out from under us we also want to be able to associate cause and effect so something happened we now have a new state I want to be able to associate those two things here's the cause there's the message that came in here's the new state that that produced that's extremely valuable for testing so if you've written applications in closure script you may have you may have used an atom for this type of thing it's it's well-suited for this and some of the implementations of our transition model in Adam if you're not familiar with closure and Adam is a reference type it allows you to do your F it and get a value and a mutable value and you can send functions to it perform state transit transitions those are atomic state transitions you can also add a watcher to the atom and then the watcher function will be called whenever the atom changes and to get the old and new value and so we can use this to store state and be notified when that state changes but one of the problems that you run into is you start to ask yourself how many atoms do I need to use it's nice to have everything in one atom because you have all your state there and you have consistent updates but as that data gets larger and larger and larger it's harder to tell what you've changed so if you get a big if you get you change one thing you have a lot of data now you have to figure out where's that one thing that I changed so we would like to have the best of both worlds and that's what pedestal gives us it gives us a way to consistently move between states and we have fine grained change reporting so you can keep all your state and one thing and you can watch for specific parts of that thing and be notified when those when that part changes so data flow is another thing that pedestal provides it allows us to disconnect functions from how they're executed and what this means is that we we can declaratively say here are the inputs to this function and here's what should happen with the output and then the engine that's running our application can decide when that function needs to be run it will do that based on what has changed in the data so we specify declaratively the positions in the data that we were actually watching and when those positions change when those nodes change the function is called and we get an update so this allows us to atomically propagate changes through our automatic sorry-sorry automatically propagate changes through through our data model this is all based on the data model so we're not actually tying together function inputs and outputs we change the data model we call a function that updates the data model and then another function may be called if it's watching the output of the previous function so this is all centered around the data model itself and it allows us to avoid writing explicit pub/sub code or big hairy conditionals so some of the advantages of dataflow is that it's a great way to encode the dependencies between data in UIs and UI applications these can be quite complex something like I click a button I want a menu to be a menu item to be disabled and I want maybe some other menu items to show up we can we can do all that with dataflow and it's actually a nice way to do it it also promotes adding code rather than updating code which i think is nice you can you usually write very small focused functions that are that are part of the data flow and if you need to do something new instead of finding the conditional that you need to update and going in and updating some big nasty confusing thing you can add a new a new function to the data flow so what we're gonna do now is walk through how the data flow would actually work to give you a better idea of how this works in the upper left hand corner we have our view this is going to something's going to happen in the view to generate an event and then the callback for that event is going to transform that event into a message and put it on a queue so that green arrow is a message on a queue and then what we do is we find a function that handles that that particular message that function runs and it updates a part of our subtree and even though it operates on a specific sub tree the thing that it changes might just be the phone node of that selected subtree and then the visible node the blue one down at the bottom it depends on selected so if that selected changes we want the visible subtree to change so the data flow engine will will know that that changed and know that that function depends on it and that function has two inputs it cares about both the selected subtree and the device's subtree and those will be passed into this function it will produce a new value for the visible subtree and what that will do is update the review node and so on we update the order node and then finally what we end up with is these are the things in the tree that we actually changed and in between each of these steps we generate a change report that tells us what what actually changed at the end of this process we would have something that might look like this these are things that actually changed in the tree and then we can use that to match match patterns later on and find functions that may care about that or or when we're finished with the complete state transition we would get this report out and we can do something with it so another way to look at this is to think about what a round trip looks like if something happens in the view how does that work its way through the system and actually update the view so again we have an event and we have a message on a queue and then this green box represents finding a function to handle that message and then the black arrow there is basically taking the return value of that function that we found and applying it or using that to replace some part of our information model and then the date the info model has changed so we get a change report which is the orange dotted line there and the same thing we find a we find a function that cares about that the return value updates the data model and these run until we're finished and then when we're finished we have a change report coming out the other side and we find a function that is going to generate the data that we're going to send up for rendering and we have a renderer that guy is going to going to find a function that handles each message and make a change to the view so that might be like a fine grained change to the or it might be a call to an API so in this case we're controlling widgets so we're calling some function on our widget API so the reason why the in the example that I showed you there were two versions of the application one of them that didn't work so well and one that did and the reason why the one that did work so well worked is because because we've we've separated rendering from all of our application logic we can run them in different places so I can run all of the application logic and the data flow I can run it in closure for testing I can run it in a web worker and we already have a way to communicate across the across the boundary between between the web worker thread and the main JavaScript thread and so that's how that application runs everything about the application runs in a web worker and rendering happens on the main JavaScript thread and so I have to say it was pretty nice being able to program in the browser with real parallelism so that's how that works that's at least an overview of how it works so then while we were working on this I think when this project started the project that we did for the client that was about the exact same time that core async was started and it finished about the time when we finished this project and if you're ever working on something and rich Hickey starts working on something related to what you're doing you should probably just wait and see how that's going to affect what you're doing because because it can disrupt you so we have all these things how many people here went to Rich's talk and know what core ASIC is ok so almost everybody core async I'm not going to really cover all the all the things the core Eason gives you but basically it provides channels so these are like more like real cues not like the fake ones that we're using and it allows us in the browser to program as if we had threads so we can have real processes that are communicating with each other and we can block which is amazing in the browser you can actually sit there block on a channel and it's not going to keep the rest of your code from running and so there are some huge advantages here and we want to have these advantages as well as some of the ones that we already have in in pedestal so if you think about some of the diagrams that I showed you there's a couple problems in those diagrams a couple problems that we have in pedestal right now one of them is that if you look at the way we process messages going from some kind of event happening or something changing to some value that might change our information model or affect something else we have this basic pattern that we that we go through we have a change come in some sort of novelty either a change to our info model or a new message so it's something new and then we have a way to take a map of patterns to functions and find a function that handles one of those changes or cares about one of those changes we we call that function and then we get something we get a return value or we get a message that we put on some kind of a queue and so we're doing things in a couple of different ways which could be unified and the way that we plan to unify them with core async is to think about having two different kinds of channels a channel that we'll call an in-form channel that carries messages which describe something something about the source that changed so this is a way for a source to say I have changed in this way so we'll send a message and inform message and that will have an identifier for the source in some kind of state and then we can do the same thing that we're doing now we can have patterns that match those changes find functions to handle them or they care about them call those functions and instead of returning whatever they want to return they will return transform messages so transform messages are very much like the messages that I showed earlier they have an operation they have a target and they have arguments and a transform message is saying here's how I would like you to change so we have these two sides right here's how I changed and here's how I'd like you to change and so I want to walk through how we can make pedestal from these kinds of channels so again we're gonna do we're going to go through the whole process we have a view an event occurs what we're gonna do is take that callback and that when we run that callback we're gonna take the event and turn it into an inform message so that message is going to tell us something about how the thing changed that we care about so this is our inform channel in blue and then we're going to find a function based on that inform message that cares about that and can transform that into a transform message on the data model and instead of actually finding a function here we're going to directly apply it to the data model if we know the the path to the thing we want to change we know a way to automatically find the function that we're referring to and we know the arguments we can just apply that with update in to the data model so we remove a step there and then when we talk about the data flow portion of state transitions we can do that with these same channels so we can have an informed channel that tells us here's how the information model changed and we have a function that we found which generates transforms that and those get applied to the information model once this is finished we'll have the final inform channel coming out that that tells us how the info model changed as we've gone through the entire data flow and then what we can do is we can find here just like our emitter in the previous example we can find a function which will generate the transformations that must happen to the user interface and then finally this is just like our renderer we can find the functions that actually make the changes to the user interface based on those transformations if you think about this it's the same separation except now we're using channels and we've got a few we've got fewer parts so another advantage that this gives is that it allows us to use we can encapsulate changes in widgets so if we would like to not do the manual message transformation and finding functions or or manually trance handling transform message messages by updating the UI we can encapsulate that in widgets and so in the application that we built we could either write our own widgets that was actually a nice way to handle it or we could have reusable widgets and I don't know if any of you have looked at the work that David Nolan has been doing like his example of the autocomplete field that he created that's the kind of thing that if we if we made it aware of transform messages and inform messages we could just use that in our application as long as we have a sufficiently good way to to customize it and another thing too to know about widgets is that they may not just be valuable for the UI maybe we could have some that help us with parts of our of the service so that's pretty much it thank you check out pedestal if you're interested if anybody would like to talk to me about this I'd be happy to talk to you if anybody thinks this is a good or bad idea just come and talk to me I would love to especially here bet people who think this is a bad idea so are there any are there any questions yes well if I think we want to make some kind of API so that you could have JavaScript widgets that would that would actually plug into this I don't think that's actually gonna be a problem we want to support that definitely yes you don't because the functions themselves wouldn't really be aware of the cory singh stuff so the engine is wiring up the functions and the engine is responsible for creating like the pipeline that's going to run your code so the functions themselves wouldn't necessarily need to be aware of core async unless you wanted to have a function that returned a channel which is actually one of the advantages of this you could have a function in your data flow that if you if you had a sufficiently fast like API that you wanted to call it's much easier to write the straight line code that calls that API and blocks and you could just return a channel that would then when when you read from the channel it would get the response the result instead of returning immediately so that's actually one of the advantages of this the functions normally shouldn't shouldn't have to no yes I don't think it changes at all because we still have messages that we that define all the changes that we're making and we can store those messages and replay them later with with a part of our application I didn't really show that at all we because we're sending data sending messages to a renderer to render we can actually record those messages and play them back through the renderer and that is very helpful when you're making super complex stuff in the UI and you have to maybe have like five people simulating like acting on one thing it's easy to like you can do that simulation one time record the data and then run it over and over again to get the rendering just right but I think we'll still be able to do that kind of stuff yes it does actually yeah that web workers are actually super hard to work with and normally you have to decide ahead of time which code is actually running in the web worker and there's we added some support so that you can configure your application to say here's the stuff that I want to run in the web worker and then when you compile it will automatically compile that stuff into like its own file and load it in the web worker and there's even stuff for like making the chana going across the boundary like the channels go across the thread boundary yep but it is still a little bit hard to use if you want to know about that check out the tutorial the very last section of the pedestal tutorial has has all the documentation for how to how to do that yes so in the current in the current model we can actually figure that out ahead of time by looking at the inputs and outputs we we can figure what are the inputs and what are the outputs of each function that we run and so we can order them and then run them in order one time so we know we're not going to get in an infinite loop in the new version we would do like a fixed point so we have to be careful we'll have to be careful to just not run forever and maybe kind of throw some air if people do something wrong right but it'll be like a fixed point set up anything else okay yes and the choice of closure make this project easier well the whole project all of the ideas in it are based on like the ideas in closure so it's kind of a hard question to answer because we wouldn't even have thought of thought this if we weren't using closure and so because we we are so absorbed into those ideas this is just this actually feels kind of natural so I guess that's the advantage as that closure makes it feel natural a lot of people think this is very confusing but I think for people who are really into closure and doing stuff on the UI I think that it is actually it feels quite natural anybody else yes in the back which which one yeah we haven't looked into that in detail yet so that's something that we're going to be doing in the future so now if the answer to your question is no I haven't looked into it okay one more question and then we'll wrap it up yep yeah right so actually they're both they're both actually the same they just seem different you're always in the in the first model we are saying here's how I want you to change sometimes we talk about it like here's a change react to it but really we're saying here's how I want you to change right yes so it's I think what we're doing is we're more we're being more clear about what the roles of each part are so all right thank you [Applause]

from infoq