Goodbye YAML: Infrastructure as Code in Clojure - Eno Compton & Tyler van Hensbergen

so this is goodbye yeah Mille infrastructure as code and closure before we start though I'd like to quickly thank Kim Foster and Alex Miller for doing all the work of organizing this conference then to a number of conferences and I've never been to a closure conscious before this is my first conch and I'd have to say I really like the tone it's a very thoughtful audience very thoughtful conference so thanks a lot to Kim and Alex for making it possible and of course all the people at cognate echt for sharing this great language with us it's brought me a lot of joy in my life so my name is Eno joining me on stage is a little bit is going to be Tyler there's our email addresses if you want to a ping us later for questions this talk is basically in two parts it's a story of a library and it's a demo I'm gonna be telling the story of where this library came from and Tyler's gonna give you a feel for what it's like to use the library so the story starts at steady it's an early-stage startup located in Boulder Colorado we've just raised our series a of about 14 million dollars in funding and it's steady we're building a global network for b2b structured communications that's kind of a dense sentence with a lot of meaning and a lot of words packed into very few letters so if you go look at our website it says this the structured messaging platform for b2b trade and it helps I think to kind of unpack this because that will help to understand where this library came from and also you know just what this company is doing in general so if any of you have ever worked in purchasing or fulfillment or you have a friend or you know somebody who's doing that work then you probably know that a lot of the business processes that people are using to do purchase orders invoices ship notices you know all kind of things that businesses any kind of structured communication that one business is sending to another the systems that are facilitating that communication are incredibly archaic often painful tedious and you know what we're talking about here is it's a trillion dollar market any kind of like physical item in the world is touched by these archaic processes you know the clothes you put on this morning the Breck you ate laptop you put in your bag everything is touched by these these business processes and even in the most extreme cases what we're talking about is people bringing home you know giant stacks of paperwork with a ruler and a red pen and literally going line by line by line to make sure an invoice is properly fulfilled or a purchase order is fulfill or whatever so it's steady you know we're trying to make this a we're trying to fix this trying to make it better so in effect what we're building is you know digital mailbox for any kind of structured communication between businesses we're doing it 100% enclosure and we're using AWS service to do it so like I said studies located in Boulder Colorado which in my opinion is one of the best places to be writing software on earth we have a 12 person team and we're growing so if you'd like to hear more about that Tyler and I will hang out afterwards we're happy to talk with you so this is basically the bread and butter of what we do we're using closure to solve real business problems and we're deploying it on AWS and this is basically what we do every day now just as rich shows Java for closure we like to think that we chose AWS for infrastructure so you know the whole question of multi cloud and all the related problems that go with that I'm just gonna put that aside today and I'm gonna be focusing squarely on AWS we've adopted AWS and you know our whole world is AWS so that's going to be the focus of this talk nonetheless you know some of the things I say I think will be relevant for you know the experiences of deploying on other clouds so like I said we like to use closure to solve real business problems and then deploy it on AWS and in particular what that means are you know not just AWS in general but we're using AWS lambda which is AWS is service offering and has all sorts of interesting characteristics that kind of change the way you think about problems you know instead of having these long-running JVM processes we have these short-lived processes and and cold starts and all sort of new things that make a lot of new things possible now one of the things that were we're using to do all this infrastructure is this tool called AWS CloudFormation if you're not familiar with AWS tools you might have heard of terraform it's in effect AWS is version of terraform built in-house AWS describes it as a common language to describe and provision infrastructure and that language is written either in yamo or json so for the stock I'm gonna be talking about yeah Mille in particular but every time I say yeah Mille you could very well think I said JSON and the same will apply now the nice thing about this tool is at any moment a developer can open a text file and look at what the production infrastructure is we've been very rigorous it's steady that we want to have all of our infrastructure codified in cloud formation and that personally helps me sleep well at night because I know there's not any kind of state lurking out there it's all described in these nice files that we can change and you know make new change make new infrastructure tear down old and it's all codified in a single place so it's a great tool and so we've been using it we've been writing some closure we deployed on land uh you know and things are good then we did that some more and then we did that a lot you can see closure script is sprinkled in here and there we're particularly concerned about cold starts and along the way we've learned a lot about what it means to take closure and deploy it on lambda so this is this is kind of the world we lived in initially it was great we wrote some closure had a great time put it on lambda solving real business problems you know when we wrote some yeah Mille to to do that and some time went on and then we were at some Auriemma brought some closure deployed it we wrote some more yeah Mille some closure deployed it then we had a lot of Gamal you know it's nicely organized but there's a lot of you know then we kind of had a big pile of the animal and then it felt like we had this huge thicket of the animal you know there's the closure in there somewhere but it felt like to get to the closure we had to fight away through a lot of yeah Mille and that's not a great experience for some reasons that I'll talk about so when we realized that we had this view mess of yam all describing all our infrastructure and when we realized that we just weren't you know we weren't totally happy with that experience we stopped and looked at our tools so the first thing we looked at was closure and we thought well closure is great we actually really like closure we feel like we're productive we enjoy using it we looked at closure script same kind of thing we liked it we're productive we enjoy using it in fact if you go look at the closure webpage you'll see this quote from rich where he says I hope you find closures combination of facilities elegant powerful practical and fun to use and in our experience closure and closure script have been exactly that elegant powerful practical and most importantly well you know for the selfish developer and me fun to use it's fun to write closure all day and then we looked at Yamma which is the other thing that we were using and in a lot of ways yama was just the complete opposite the experience was completely different the feeling of using it was completely different such that you could even graph it like this you know you have all that all the fun stuff in the upper-right fun to use productive and then you have a yamo in the lower left which is you know I I don't personally enjoy using it and we found that when we wanted to make a small change or maybe add some new infrastructure it would be a day several days a week and you know there's just this feeling inside of not being super happy when you're doing that and a big part of that is you know syntax errors it's a white space dependent language which I'm generally terrible at but more importantly you know you we'd make a mistake we'd push it out we'd wait five or ten minutes for CloudFormation to deploy the m4 strike and say hey you got a mistake you forgot this one required property or you got one space and I need to hear and that's a really terrible feedback loop and that just kind of compounds you know as you're trying to work through a problem this this slow feedback loop just keeps getting bigger and bigger bigger which is why yeah Moe is not as fun to use as closure in closure scripts so if you think about this for a moment it's because yeah Moe has an effect become the assembly language of cloud infrastructure if you're putting anything on the cloud today I'm fairly confident to say that you're probably using the amyl in some capacity or to change the wording to put it in JVM nomenclature you can say that ya know has become the bytecode of cloud infrastructure now if a person came to you and said I'm gonna write all my application code in byte code you'd probably say hmm I'm not sure that's the best idea and you know the reason for that is that we're working with two layers of two layers of abstraction here you know Yama was kind of a lower level concern and closure is a much higher level concern and you know all the properties of those two layers are with that so for example when you're working in the animal you're almost certainly having to be much more verbose and you know have to write a lot more lines to get stuff done whereas if you're working in closure you're able to get wonderful things done in a very concise way which i think is a large part of wide closure such a joyful language to use so this is the world that I think a lot of developers are living in we at study we're certainly living in it for a while and that we wanted to be solving business problems and deploying it on lambda and yet you know we had this thicket of yamo in front of us that you know it's kind of hard to see what it is that we're excited about anymore this is the world that we want to live in I think we want to be focusing on solving our business problems with closure and then getting it out to our customers so there's also a bigger problem here though I mean there's the there's the problem of the levels of abstraction and you know you know some people don't love yamo but there's actually a bigger problem here and that is how are we going to make infrastructure reusable and shareable across teams across deployments across companies even while maintaining the elegant and powerful facilities of closure so in the process of building all these systems using closure and deploying it on lambda we we've learned some things in fact along the way we realized that there were some patterns emerging in our application infrastructure so let me walk you through what I mean by that and give you an example so one of the patterns that we've used in our infrastructure that we found is very useful as having a lambda that consumes events and take some kind of action based on that event so we have some closure we have a lambda we have an S qsq that the lambda is consuming events from we have some source of events cloud watch events for instance that are originating elsewhere in our systems not important we're just that's what's filling up the queue and as responsible engineers we've also deployed some alarms and monitoring such that if for instance a message gets stuck in the queue there's an alarm that will go off that will send a notification to a mess and that's topic and then we'll go to an alerting system to get developers involved and of course you know for anybody who's worked on AWS you know that you have to have all these iam policies to get the permissions just right and as you can imagine it takes a little while to get all this right there's a lot of moving pieces here it's not just a matter of like you know putting together 15 lines of code and you're done you know you have to kind of like iterate to get all the pieces of the configuration right and once you've done that you want to reuse it you don't have to do it again you don't want to be copy pasting the amyl you want to have this robust way of you know I've done the work I built the infrastructure and now I want to reuse this event consumer thing across the board without writing more yam oh that's that's the really important part so the first question we asked ourselves was how does cloud formations support reuse you know the cloud formation is a powerful tool that people build it or who built it are very thoughtful so surely they've run into this problem and yes they have in fact there are a number of tools that CloudFormation makes possible or makes reuse possible nested stacks more yamo template snippets more llamó custom resources I'm guessing you can see the pattern here Mori ammo and then even service application model which has done a fantastic job at making resource declarations much more concise but nonetheless means that you're still living in this amel world with all the problems that come with the ammo so at steady given the choice between the ammo and closure we like closure we want to write closure all day so we had to broaden our question you know maybe cloud formation wasn't the right tool for us is what we thought and we said well what are the alternatives in the AWS ecosystem you know maybe there's a better tool out there that we could be using that will help us solve their problems you want to do in a way that we want to do it well it turns out about a year ago AWS released something called the cloud developer kit or cdk or AWS cdk is sometimes called it's written in typescript and it's this fantastic tool that allows you to be provisioning infrastructure without all the ammo so if you go look at Amazon's AWS cdk webpage they give you this example to deploy a container onto a Fargate service which is an on a far gate which is Amazon's container service just to contain it like a hello world example we're talking about this much Hamel plus four hundred lines that I'm not even going to show you or this much typescript that's a huge difference we're talking about 500 lines versus 20 lines to do exactly the same thing now the way the cdk team has done this is that they've gone and looked at all the AWS service offerings and they've created what they call the AWS construct library which amounts to a lot of sane defaults rich primitives that solve the problems that you know most of us are working with in one way or another without having to worry about all the fiddly bits that CloudFormation who makes possibly configure and the construct library is basically an assemblage of infrastructure constructs so this thing that I showed you earlier is also an infrastructure construct and what's really powerful about that is you spend the time to build the infrastructure that is meaningful to your business or the problem that you're solving you do that once and then you can reuse it over and over and over and over and over because it is exactly this infrastructure construct so not only are you getting these constructs this construct library that the cdk team is built you're also able to define your own infrastructure constructs as makes sense so earlier I said the library was written in typescript some of you might be thinking he's about to recommend that I should take on my amal and rewrite it in typescript and honestly you could do that but you know I feel typescript is definitely an improvement but I want to be writing closure all day well it turns out in addition to AWS cdk there's this very interesting library that ships with it called J SII that allows the cdk team to choose their language of choice you know they're working in typescript but to be releasing bindings in Python c-sharp and Java now as soon as I said Java a lot of you are probably thinking well why not closure and that's what we thought too so that's what we did we open sourced a library called CDK clj about two months ago it's been a fantastic experience and I could continue talking about it but I think at this point well first of all I should tell you it's 100 percent closure which is great but at this point I think it's really better just to show you what it looks like to interact with it and how to use and so on so joining me on stage right now is Tyler he's gonna walk you through a demo all right thanks you know live infrastructure demo in the repple what could go wrong all right so my job is to give you all a sense for what replicas and infrastructure looks like and can be and and to do that we're going to go through the process of deploying a lambda and that's gonna run some closure code it's gonna be connected to a s3 bucket so we designed this from the ground up to be a like first-class repla experience we wanted to make it as easy as possible to get documentation make it integrate with dock strings you know make it so that you get fast feedback when there's problems with your infrastructure so that you don't have to go through that long feedback cycle of making a bunch of Yamla changes sending it off to cloud formation and then just hoping alright so the first command I want to show you is the cdk browse command which is how you navigate to the documentation so calling it without any arguments is going to take you directly to the opening page for the AWS cdk documentation so this is primarily an interrupt library so this is something very useful to have at hand so this is gonna have the developer guide which gives you a bunch of background information that's super useful and working with cdk and it's gonna give you the api reference which is going to give you basically the holistic view of all of the packages and constructs that are available to you so constructs compose lower-level CloudFormation resources so there's always an escape hatch to using the lower level resources but these are going to give you documentation for both those high-level constructs and lower-level CloudFormation resources okay so let's start to build our infrastructure and all infrastructure in cdk is a tree of constructs with a app construct at the root so the other command or the other function is the import function so this Maps since this is an interrupt library to the JavaScript import command so running this basically we're pulling out the app class from the AWS cdk core construct library and when we do this we get it does it does a couple of things one is it makes the class resolvable in the local namespace and then the other thing that it does is it makes the symbol an alias for the namespace that gets created behind the scenes for app so app is going to have a set of functions around it things that you can do with Apps and those are going to be in a namespace that gets created as a part of this import command so if I evaluate app you can see now that this is a resolving to a.j SII class so JSI is that low-level Interop library that's enabling us to do this and in essence this becomes our constructor to create an app instance and this is going to form the root of our infrastructure tree so I'm gonna go ahead and evaluate that and give it an out there and and at the lowest level this is generating cloud formation so this is just saying where where to put the cloud formation at the end of the day so it's gonna go in the cdk dot out directory and because we've done this in closure and ratified all of these or interned all of these as bars and namespaces we get things like docstrings so we get the documentation for this construct in your apple so that you can use it to make sure that you're doing the right thing so I mentioned that that import command does two things so one is it gives us this class the next thing that it does is it turns or it converts the app symbol into an alias for the namespace for app so we can see that the app namespace is going to have a few functions and then these are a couple of examples of that so we can grab doc strings for these functions as well so you have since it's an interrupt library you have both static and instance methods so this is an example of a static method you can tell that because it doesn't take a this as an argument and this is an example of a instance method which is going to take a this is the first argument it's sort of similar to the Java Interop in closure and so another useful way you can use this browser command is to jump to the documentation directly from a class so I can pass in that class to that browse command and I can jump straight into the documentation if you need to figure out how to use these things so a app without any stacks or other constructs attached to it isn't particularly useful so let's attach a stack to the app and we're gonna import that from the cdk core package again and the way that we connect this to app is we call the constructor with app as the first argument so basically child components attached to their parents so we can call this constructor and we get a instance of a stack and so now we've connected these two things these two pieces of infrastructure together so on these resources that we've wrapped we actually implemented the I look up protocol on it so you can use keyword look up with these things which is going to come in handy later when we want to link all our resources together so a stack is the minimum deployable unit of CloudFormation but by itself isn't particularly interesting so let's add a bucket to it and while we're adding a bucket I want to show you that some of the examples of the feedback you get we have put specs on all of these things so if you try and call one of these things with incorrect information you're going to get a a hard failure right away with like a spec assertion telling you exactly what you did wrong so in this case it takes a ID as a second argument and it's requiring that to be a string so you don't have to send this off to confirmation wait 10 minutes to find out that you can find it out right away so let's do this correctly now and give an actual ID and buy into a bucket so at this point we have a bucket attached to a stack attached to an app that's still not that interesting so let's add a lambda because they're all the rage these days so we're gonna have to import a few constructs to do that so we're gonna import duration from the cdk norms cdk core namespace code function and runtime from the AWS lambda package namespace and then we have a little bit of boilerplate here because we need to make a jar for this in order to deploy it so let me just evaluate these and while that's building I'll jump over to the application code that we're going to deploy and so it's a really basic function we're going to make it's going to be a web handler and that's why we're returning a status code here at the bottom with a body we're gonna put an object on s3 and then read it immediately back it's not particularly useful or interesting but I want to show how we set up things like permissions and all of that and we're also going to pass in a secret because I saw Jean Kim's talk this morning and one of the bullet points was that doing secrets right is hard so I want to show how that works in CDK clj so let's connect our function together here so I'm gonna import one more library we are one more contract which is secret value which is how we're gonna bind the secret and we're gonna set up our function here so the way we do that is first we have to tell it where the code is so above and beyond CloudFormation cdk is gonna take care of things like asset management so we have to tell it where our code is so it knows what to upload to s3 so that it's resolvable next we tell it the handler we want to use so because we're using the Lambada library they have this def lambda function that when we compile is going to generate a class named like this that implements the and protocols or interfaces rather we have to pass in the handler name and that just has to map directly to the handle their name over here or the class that will get generated next we have to tell it the run time so there's this run time class that has all of the different runtimes as static properties on it from the Interop so we just want to use the Java eight run time so I'll just show a quick example of how you get feedback with this here and so like this is not a valid run time and you're gonna get a hard failure if you try and do that and it's going to tell you you know this is the set of things you can do so you can use this to deploy Python node in this case we want Java so next I want to show you how we link resources so we were writing to and then reading from a bucket in that code so we need to pass in the bucket name where it's ideal to pass in the bucket name because it's going to be dynamically generated we didn't name it explicitly ourselves so we passed that in as a reference here and we can do that just using the keyword lookup and in addition to that we're gonna pass in a secret so this is a secret that I uploaded ahead of time and this is just a reference to that that's going to be passed into that lambda and made available as an environment variable the last couple of things are just setting a timeout and memory size so that we don't run into a cold start issue or minimize the impact of a cold start issue so let's evaluate that connect it to our construct tree next we have to give that permission so right now we don't have permission to do anything with that bucket or that function doesn't have anything permission to do anything with that bucket I think this is one of the places that cdk really shines I mean I don't know how many times I've seen or done myself set star star permissions on like a cloud formation resource saying I'll come back to that later after I fix this and then never do this there's no excuse now if you're using CDK clj you know we should be granting fine-grained permissions to get grant lee's privilege privilege to follow the best practice so we do this with just the grant function here and if you actually take a look at the bucket name space you know we get autocomplete for all of the options in there so this is another reason why it was nice to build this from the ground up for closure specifically and not to use the Java version having this autocomplete functionality makes everything very discoverable so lastly we're going to just make this available by via REST API through API gateway and we do this with a lambda REST API construct which is very handy I don't know if anybody has ever tried to set up API gateway with CloudFormation by itself it's not fun I'm getting the deployments right and validating them pushing it up again getting all of the methods and resources correct is not a fun activity to go through the last thing that we do is we call absinthe which is going to synthesize this into a CloudFormation template so I just evaluated the whole buffer there so that we pull in all of these changes and that's going to dump it into a cdk out directory which is going to take care of our assets and it's going to take care of building our template so you can see it JSON version of the CloudFormation template it generates here so I'm gonna jump over the command line now and start the riskiest part of this demo where we're going to try and deploy it so before I do that I want to run the synth command because this is going to show us all the ammo we didn't have to write I I don't know about you but I'd prefer not to write this this size a yamo file I have I don't want to do it again it's not fun and we get that leverage of having it build all of these resources for us through cdk so this is gonna take care of policies roles methods all of the things that you'd rather not think about if you don't have to the API gateway deployments and generate it in a way that if it evaluates in the repple it should deploy because of that fast err feedback so let's give that a and see if we can actually do it so this is another nice value add so cdk the the tooling around it there is a command line a program that they give you and it's going to take care of your deployments packaging your infrastructure and all of that based off of your cdk declarations one of the nice things it does is it gives you this nice review page for all of the permissions that you set up so you can make sure that you didn't put star star in there out of frustration so I'm gonna go ahead and say yes I didn't intend to make those changes and this is gonna deploy and while it's doing that I want to jump over to a larger example just to show you some of the patterns and what this looks like at scale unfortunately this is not a working example yet it is a deployable example but the code sight doesn't work yet I got carried away with the infrastructure I want to show off all the different things you could do so you can see here examples of doing things like cron jobs by using cloud watch events you can see examples of using state machines and hooking those up so we're pulling in a lot of resources here to show you examples of how to hook all of these things up so I'm gonna jump down to the bottom to show you some of the patterns that we use perhaps the most useful is just regular closure functions it's very nice to wrap up these complicated constructs as simple closure functions so in this case we've wrapped up this entire stack app stack as a closure function and this is very handy for things like environments so right here we have a stack for dev and we have a stack for prod and we can pass in different configurations for death and prod and use that data to basically make decisions about what infrastructure we want to turn on or turn off so in this case you might want to turn a timer off in a dev environment because you don't want to run that all the time you want them running manually when you're testing it the other pattern that I want to show here is like wrapping up a common construct that you're going to use in multiple places so Lambada function carries all of the common configuration it's used a few times in this namespace and this is something that could be put in a adepts project and shared across teams or even open sourced so if you have common build tasks or if you have common pieces of infrastructure that are general concepts you know you can just turn them into a function put them in a package and share them let's jump back to our deployment and it looks like we've been lucky and everything deployed so you know this took care of uploading our jar connecting it to our lambda function it created all of our resources for us and we can verify that and it spits out a nice URL for the API that it created and we can just curl that and this is going to be going through API gateway to lamda putting something in s3 reading it from s3 and saying hello conch 2019 [Applause] you could of course done that with the amal but i i think i don't know i prefer this way alright so with that i'm gonna pass this back to you know to bring us home that was great so once we started using this internally we started seeing PRS land like this massive deletions look at that these kind of pr's bring joy to my heart because we're doing exactly the same thing in fewer lines of code we're making our code more understandable more changeable and i like that so if you go look at what the cdk team has to say about the library and its benefits they'll say you know you can define high level abstractions you can organize your project into logical modules namespaces functions you know whatever you can share and reuse your infrastructure which is a huge point you spend the time to build it right once and then you share it across whatever unit you'd like to you're getting code completion with your IDE that's CD k CD k CL j adds yet another bunch of great things to it in my opinion you're working with your infrastructure in the repple the environment that we know and love with a fast feedback you're getting spec checking at AWS CD k boundary so if you forgot it what required parameter you pass the wrong kind of parameter whatever you're gonna get a nice spec error right there when you evaluate it in your repple not waiting 10 minutes to find out you screwed something up and then finally you're able to think about your infrastructure in the same language that you're thinking about your business problems which to me is a huge benefit like I said this is up on github steady ink that's a little bit hard to spell but steady ink /gd k LJ the demos that are the examples that Tyler walk through are up there you know try it out pull it down let us know what you think if you don't like something you'd like to change something we're open to features or pull requests issues you know all that stuff or sorry not pull requests we're not ready for pull requests but definitely issues and feature requests so I'm state our world work study my name is Eno this is Tyler thanks a lot everybody [Applause]

When building systems in Clojure, we have a great choice for backend applications (Clojure or ClojureScript), frontend applications (ClojureScript), and the CLI (Clojure or ClojureScript). When it comes to creating infrastructure, though, we must put down our favorite language and turn to CloudFormation and YAML, or as an alternative, we might use tools like AWS Cloud Development Kit (CDK) to write infrastructure as code, but we would be using another language and not Clojure. At Stedi, we found ourselves wanting to spin up new Clojure applications without having to resort to endless lines of YAML or another language. This talk introduces a newly opened-source library we created in Clojure, cdk-clj (https://github.com/StediInc/cdk-clj) that allows us to write all our infrastructure in Clojure in the familiar REPL-based development environment, without having to give up the power of CloudFormation or CDK while enabling all the patterns of re-use inherent to Clojure, especially given the recent release of tools.deps. This talk will introduce the library within the context of its alternatives, then briefly go over the library’s implementation, and finally end with a short demo of building a serverless GitHub bot using a Clojure lambda that demonstrates the power and simplicity of cdk-clj.