Starting a fresh Clojure project with Neovim REPL driven development

all right hi so i'm going to be setting up this uh closure project this is a project i've been meaning to start for a while uh this is going to be a debug adapter protocol system for closure and it's going to talk with uh and the cider mripple debugger middleware uh so this is gonna enable uh editors like neorim uh to use their debugger adapter protocol support and plugins uh so you've got like mvim dap ui uh you'll be able to use this sort of thing to connect to uh closure dap and that will connect onwards to your n-repel enabled closure server and that means that you can live debug a running closure application using the same memorable connection that you're using with your repo tooling this is being developed alongside conjure my neovim ripple system uh but it doesn't depend on it it's it's completely separate to it uh it's gonna pair really well with it but you can use this with any uh dap supporting editor where dap is i don't know if they have a link to it uh dap debugger a uh thing i think yeah the spec is designed by microsoft uh it's a bit like language server protocol um yeah yeah let's go to the other camera good point um so uh the debug adapter protocol is uh kind of similar to language server protocol and i think lsp is also designed by microsoft so this is a simple sort of thing but it allows you to introspect live running programs rather than a static analysis of your code so i want to enable the bridge between that world the closure nripple and live replied experience being able to use that debugger through the dap protocol but today we're going to set up this closure project from scratch so it's only going to be hello world still it's not going to do anything special we're not going to actually implement this yet because that's going to take me a long time i have to work that out but i thought i'd show how i set up a few parts of this so some basic stuff like hey you got to pick a license um as well as your folder structures because that's something that can actually be kind of confusing to a new closure programmer uh we're gonna then set up some scripts so we can run the project really easily uh we want to be able to start uh enriple servers so that we can connect our editor to our closure program be able to uh send code to it while it's running and modify it and we're going to be able to want we're going to want to be able to run the tests uh either in a one-off mode in ci or in a watch mode locally and then we're going to want to dockerize it because that just makes it easy for people to run they don't need a specific closure version that sort of thing and then run it in ci on github actions which i've never done before might not get to that part and if we do i might struggle on that bit for a bit so we'll leave that until last so first of all let's just grab a license uh i always use the unlicense i'm going to continue to use that i just take it from an existing project i've got and let's just drop down here so we just do analyze paste done hey and we can commit that and this program uh that i'm managing my git with is called lazy git uh and i am one uh get being a insult in the uk lazy git is very very good uh i highly recommend it so we're gonna commit this actually there's a trailing line there get rid of that you can actually like get into your editor from the lazy get interface which is really nice so we'll commit that a license lovely uh so we're on the main branch we have two commits here we have initial commitment we have adolescence lovely uh our next step let's check that off small victories you know uh we're gonna want an initial closure project structure so a closure project is normally split into two directories we have our src source and test uh and you normally add an extra layer underneath source uh to kind of separate your files from other libraries and different parts of uh the system that's running because in the jvm you're loading all of the classes into one big space and they all share the same name space or the same um the same class path and you don't want your class path to collab with somebody else's so you want a unique prefix to your files which is why enclosure projects would do this so i'm going to do makedow.p p is going to create intermediate directories i want a source directory and under here i want closure dap that is going to have to be underscore yes confusing myself here so java does not allow hyphens in names of the jvm does not allow hyphens of the names of things or in the paths so closure uh will accept a hyphen but turn it into an underscore at the file system level so we have to create it with another school and inside source closure dap i'm going to create uh normally i go with main here as an entry point so i'm going to go with that so let's go with that main.crj and that's my editor has actually created the namespace for that and we're going to create a function in here um what do we call this we're actually going to when we run this we're going to run a specific function so we're not going to say run the main file we're going to say run this function under main which means that you can actually have multiple entry points you can have like um a user entry point for when users run it and you could also have a a dev entry point and would be able to pick and choose which function we want to run which is very handy it used to be that you could only say run the main namespace and it would run a function called main under the namespace you specify that's no longer the case we can now pick and choose which is good so i'm just going to go with um what do we call it i'm going to go main for now because i don't know what it's going to need to do but we'll go with that for now and it's actually going to give given um an argument which is going to be a an options uh map so we'll go with opts for now and we're going to do a println hello world and we'll print the ops because could come in handy uh dot p print we'll use closure dot p prime for this let's require actually require closure dot p print as pp and then ppe slash p print ops so this will print the options it receives and then print hello world nothing too fancy but it's a good start i kind of want to give this something different let's go with run yeah we'll go main run that's that's good i like that okay so we have a main function uh we're also going to need some tests radio p test closure dap and i'm going to create a test file in here closure dap main test dot coj again this is going to be main hyphen test in the code main underscore test in the file system um so we require a few things here i'm not going to write any tests really but we'll do a sample one uh so i'm going to include closure.test as t and uh what do we have closure hyphen dap can you can you see here it's it's put hyphens in it knows it should be hyphens even though the file name has underscores and the path has underscores so the file system is always underscores the closure code is always hyphens just because jvm legacy reasons so we're going to include closure dot main as main and what are we going to do with that i can't load it right now i try to evaluate it but that won't work we'll define a test def test hello we'll just have something here that makes sure things work so t testing math still works t is uh two plus one one we always write it backwards like this in all these test frameworks they expect that the uh expected value is on the left and the right hand side is your actual thing you're testing so we've got that there we have an explanation so if this test failed a our computer is broken and we've been hit by a cosmic ray and b we would see this explanation and if you nest a bunch of these testings like this you would actually get these combined so the error message would include all of this which is quite helpful you can tell like what level of nesting the error came from uh gives it a bit of context so now we have a test we have no way to run it yet we can't just say like closure test closure isn't that easy closure gives you all the building blocks to do whatever you want but it doesn't do it out the box you've got to do a little bit extra to get the stuff you need it's only one step away which means that you have a lot of flexibility but it isn't um like with cargo with rust or you can do like cargo test i think and it just runs your test which is really good but you've only got one choice then um so we have some test files that's d3 take that in in all of its glory so what's next on our readme we have some initial closure project structure good uh i'd argue let's make a script directory as well just because i like putting my scripts into a directory uh and i think we're done with that one so let's mark that off and let's commit those so main coj main test commit them and we'll say add initial hello world code and test cool yeah lovely so next up is being able to run it how do we run disclosure program well i'm going to use the closure cli so uh the closure cli can execute any kind of closure or fetch all of our dependencies uh it is very confusing uh especially to new people like this stuff i don't think this is particularly helpful to a beginner at least uh to someone who kind of already knows how to use it it might be more useful um but to an absolute beginner this is quite scary and you should really like you want to find some examples uh or like recipes of here's how you run a project so here's how you run a project we want to run that uh start function inside main so we're gonna run closure uh there's some interesting differences here where closure and clj are essentially the same except clj uses a program called rlwrap which adds kind of linewise movements so the ability to move your your cursor and edit in line like that isn't a given so clj wraps the closure program so it's uh kind of essentially doing rl wrap closure is the same as clj so if you ever try and download the closure cli and just run clj and it says hey i can't do that i don't have rl wrap that's why because rl wrap is a separate program and it's just a short hand to run it i'm not going to use that here we'll just use closure so closure dash x is how we run a function you can see some history of mine in my fish shell there we're going to give it the name which is closure dap dap dot main slash start did i call it start or run run and it executed it ran so that printed our hello world pretty quickly as well a little bit slow because it's creating a jvm and everything and we've got a nil here and that nil is the options so there's currently nothing there however if we give it something say um n repeal true it now gives it a map of options where the key and rebel is set to true so you can give it any number of of key value pairs and it will pass those into your function so we're actually going to use that to set flags in our system early on so rather than having to include a library to do like cli passing that sort of thing we'll just rely on this for now because it allows us to set stuff on and off um so we can set a flag on and off useful i'm actually going to put this in a script so we're going to have a script slash let's call it closure dap i guess um i don't know i don't know whether we should put this on top level or not i'm just going to run for now we might change that in the future uh so bin bash no bin user um user bin m bash is that how you do it yeah that's the portable way to uh declare what you want your script to run with uh i'm going to run this exact thing here and we're going to do a dollar app on the end this means we can do script run and it will run the hello world and we can do foo true and it route with food true cool so we can run our script nice and easy we can run our tool nice and easy so we have a one liner so developer tooling how do we run this with a rebel so this is where we start to get to the point where we can interactively change our program so first of all let's commit that script add a simple script to run the program so now we want uh to start an m-rebel within our system if a certain flag is set uh or at least we want a script to be able to run it in dev mode um so how we're going to do that we're going to have um script dev perhaps user bin m mash and i'm going to go look up the cider and ripple documentation because there is a really good example over here there is a one lineup basically uh do you want to use the api usage so there's all these different things you can do with that uh we don't want to use line it is i didn't add a depth eden file yet so we'll add that as well um sync this one yes uh should we go with that one [Music] yeah we'll go with the command line one so do we i think we do yeah normally when i write a closure program i write it so that when you start in development mode you also start the real program so if i'm writing a server that is pulling events from a stream and pushing them off somewhere else or to a database or something i write it so that my development setup starts a database it starts a stream it starts everything in docker it starts the program and it starts running and reading from something and writing somewhere and then connects my repl into the middle of that so that i have a running system that i can change and modify while it's running uh and see the effects um i think for this system because it involves connecting to a running n rebel and requires a bit more setup uh i'm not gonna do that it's like a lot of this stuff is on a case-by-case basis on this case uh i don't want my program to do anything i want it to turn on load the code but don't actually begin running uh i don't want it to try and run a server on a port for example i just want to load the code with an mreppel and then i'll start working on it so we're going to copy what they have here so for this we need a depth.eden and this is how you declare all of your dependencies and your kind of aliases uh for your closure project and those aliases can contain different dependencies so in javascript in npm you have dependencies and dev dependencies in depth.eden and closure's dependency system you have as many as you want you can combine them and mix and match them as well so in here i need a depth section uh i'll put some depth in there in a sec i need some aliases and in here cider coj is going so we'll do uh should we we could call it cider clj why not i i'm just going to call it mrevel yeah i'm going to call it mrapple um and they're specifying a closure version here which we don't need to do we can do that over here so do that uh paths uh so we're gonna specify bars which will be um what's in our paths source and test yeah we'll just do both of those we could make it so that the test paths aren't uh loaded initially and we only load them while we're writing our tests but i don't think it's worth it like there's just no point this makes it easier for us to do our development so i'll keep it this way and it won't impact any users either i don't think i'm going to go to closure.org and check the latest version i think it's 111 something 111 so one level one there's our closure version and if you haven't written any closure before uh you might find it weird that we specify our language dependency inside our depths uh if you've never been able to experience that before it's kind of a game changer because it means that you are not relying on the system-wide version of some language that's installed you can have per project in different directories across your file system each one can specify a different closure version that it depends on this wouldn't even be a problem if we couldn't do this really because closure is scarily backwards compatible uh it is never introducing breaking changes i can't think of one maybe there have been like a long time ago um but it only accretes and adds things um there is no culture of yeah we'll just break stuff and people will update it will be fine um the culture enclosure is don't break things only improve um but despite that we can still specify a closure version uh on a project per project basis which is very handy so we're depending on side of ram rebel but only when the alias nrepo is enabled um so this will uh when it's invoked uh run the module and rebel command line with the middleware middleware so these are like command line options pre-configured and i actually think that's out of date and we'll throw a warning and i'll have to tweak that slightly but we will see so this is how they run it they run it with clj dash a slider clj so i'm going to go into my dev file but down here i'm going to just use closure instead of clj as well and i called my alias n rebel so script dev see if this starts it might throw an error oh no it's up cool so what i can do now is come down into uh close your dap open up the source file so here we have main.coj and you can already see my repple has connected to port 545 107 which is 45107 and if we list the sessions now we are connected to the norwegian london that's just conjure it makes up names um so we are connected to this running program now this terminal on the left is a running program terminal on the right is the source code so i can go in here and i can run hello world and we see hello world here now i'm just going to move this other terminal over to another window so it's not blocking let's make this a bit bigger and i can evaluate run so i just reloaded the run function so i can actually say well what happens when we run the run function oh we need to give an argument give an argument it prints an empty map and it prints hello world okay let's go redefine it hello world uh hello twitch and youtube i just reloaded that function and now i print something different so we're gonna be able to do this like the entire time i developed this project in that one line in the depth.eden and in the uh dev script which really didn't need to be a script because it's so simple we have started the program with a a hole in the in the side that my editor my tooling can plug into and modify live and this is how i do all of my development but this is the first thing i want set up before i work on a project this isn't something i set up later when it gets a bit hard i set this up straight away and this even works for tests so let's uh i'm going to leave that code like that for now um let's go to our test i can load the test file so i just did comma ef and we can see over here it evaluates to the file and i can run this test success test was okay how about we break it reload the test run the test nope i expected three and three does not equal two good glad math still works and run the test again it's all good so we have live evaluation uh plugged into our program already in one line excellent that's useful more than useful that is what i base my career on so we have developer tooling i'm going to mark that one as done i think that's good enough uh let's commit it oh this dot cp cache is from lsp tooling for closure so the stuff that is giving me some my auto completion go to definition that sort of thing is going to generate these files you don't want to commit those so what i'm going to do is add them to ignore so slash dot cp cache and let's commit so we'll ignore that oh and the end report we don't want that either so slash dot n rep or port cool ignore some files add dev tooling um say hi to cool viewers there we go so next up is testing so we are going to integrate coucher which is my favorite test runner and it's a very similar story so we go and find uh in there in the documentation here's the alias so it's got test extra depths main ops looks very very similar make sure i grab the right curly brace so we're going to put that in our depths where are you there we go and i'm indenting here i'm telling my editor to do that but there's so many ways i can do this uh yeah uh so we have coucher we have the latest version because i hope that they're keeping their documentation their versions up to date here so that's 1069. they're on 1069. lovely and i actually recommend that you do this they create a bin directory and they put this script inside it which really just does this so what i normally do is take that go over to my script directory create a counter file hashing switch mark user bin m bash and put that there if you don't write this this way by the way uh it's not just for no reason it actually breaks some operating systems like uh nixos it works fine on arch but if you're using nixos and you do like uh bin bash that won't work so it won't work on some operating systems so pro tip do that um that is the most portable way to reference bash uh so we're doing dash m test that means we can do script capture and it takes a second starts at jvm all those wonderful things oh we want to put configuration file ran one test cool what you can do is say watch and now it's going to start uh the system and it is going to continually rerun the test when i change a file so we go back into closure dap pick a source file we're going to go one and let's just change the file there and it re-ran and that takes like less than 50 milliseconds so once it's up and running it's fine that's how i normally use my test uh so it did warn us that we don't have any config file so i'm just going to go and grab a default config file because that normally helps out in the future where's the default one i know there's a default configuration file somewhere ah this will do here's an example yeah and that goes in tests.eden so we paste this in um i'm not sure about that formatting but you know uh so counter report dots um what other reporters do they have could we could we make this pretty we'll just leave it like this for now no point wasting time on it so we've got randomized filter capture output profiling random seed let's put change that slightly profiling count so it's going to tell us what tests are slow test paths are just test ns patterns are anything ending in test looks great let's just change this indentation because i don't like it too much this is a huge mistake okay better oh no more i'll do these ones a bit more manual cool there we go so now when i run coucher it shouldn't complain and it might look prettier maybe good enough good enough for me okay so we have tests let's commit those so we have coucher we have added to depths and we have a test.eden file add couch slash test runner just in case people don't know what it is um so what are we up to now we are going to wrap this in docker quickly so when i wrap closure programs in docker what i do is go to here so the closure hub and i like to find the latest light one so there's an alpine build so what one have we got tamarind eight no 18 alpine two steps alpine i think one of these will do yeah okay don't do so we'll do from and it's going to be this one from oh you have to do like closure colon that yes um so i think all we need to do is uh add our source code so let's just look at our source so we're going to add um well we're also going to do uh dot dot ignore which is really important so if you ignore your get and your dot cp cache it will improve your docker build and load times and dot co j condo yeah let's uh ignore cj condo as well dot co j condo these are just more linting tools that i have you might not have those slash clj condo let's ignore that so you should always docker ignore stuff that isn't essential to um your uh dockerfile so we're going to add uh add dot now i think yeah i'm pretty sure we can add dot we'll do source source add test test add script script and i think that's everything oh no we're going to also add depths.eden and tests.eden um right i think it's good um lsp oh yeah that's a good point now you make a very good point i might actually have that in my global git ignore which is probably why it's not showing up but yep good catch yeah i think i have a few things in my global get ignore so i'm not going to notice those too often um okay how do you do the run this is something that i always forget how entry point works let's look through my github i normally pick from something that's already pretty good i know what we can use i know what you can use where is it there it is so this is still a private repo and i never finished it wait is it private maybe it's not no um so i started building a discord uh chatbot that uses voice and i do have one that works but i shut it down because it was not too great and i started building a new one which is also all enclosure so let's copy from this which has a very clever docker system where it has multiple layers but we're not gonna bother with that so what i need is command that's what it's looking for so it's cmd script slash run so we should be able to do docker build dot tag uh closure dap i think i normally use docker compose or that sort of thing to run this as well like to do my builds and execute things but we'll do it the manual way i don't want to have to go into docker compose right now we've kind of already covered all of the closure stuff now i just want to get a simple docker container working and i want to get that running the tests in github actions so that's all that is outstanding really um and i should be able to get that done and if you're not interested in that part no problem um so now i want to run i never run um docker stuff manually docker run think it's that dash dash rm no i don't need ti but rm so how you do it r wrap not found of course uh so where is our wrap trying to be used um maybe in our dockerfile we can install that so how do we get ro wrap uh alpine linux fetch rlwrap i think it's a what's their package manager called is it apk i think it's apk um apk ad maybe yeah apk update apk ad so run apk update run apk add uh rl wrap see whether this works hopefully and normally in my dockerfiles i'll put a little time and date stamp as well uh just above the uh the update command so i'll put a little time and date here and i'll update that whenever i need the docker container to cache new dependencies and update uh which is kind of handy but i'm not gonna bother with that right now so let's see if our run command works ah yep let's remove the rm put it here and the rm should remove the container when it's done i think hey there we go we've run it inside docker excellent um so i should be able to tell this what script to run as well how do you specify the entry point um that might just be script slash coucher i was right so what i actually want to do is make sure this dockerfile caches the uh testing tools here run the tests and it told us what ones were slow so slowest test was this thing and it took 0.05 seconds i'm okay with that so we're going to make our doc file uh just cache the tests and cache our dependencies and we're going to do that by saying run closure dash p and we want to do that with uh so when you check the closure command line arguments we need to tell it to use our alias but we don't want it to you can use different parts of aliases so one of them is just use the class path one of them will do the main function as well i think it's dash which one is it maybe there's a if i do closure dash a test n no without n rebel dash p does that just fetch dependencies no uh [Music] he said may not a is deprecated use dash instead dash m no yeah i don't know how you do that okay we won't do that but what i wanted to do was make sure that it would pre-fetch the dependencies for that alias so it would only download coucher um or it would pre-download capture while we were building the docker container but we won't bother with that okay so we have a dockerfile and we have some more ignores so let's commit those and add our basic doc file and we now want to get this in in a github action so we have a dockerfile which should be enough to get us going i think all we need to do is to find an action in github um so how do we do that they define the action here so when they create a new action.yaml file inside the action directory you created above um do i have to call it action.yaml maybe i do that'll be annoying let's go look at github actions here can we do it through the ui can we do docker so run a docker image build a docker image deploy run our push or registry i don't want to have to run my tests inside github's vms continuous integration docker image i mean that's kind of what i want almost container now run docker build yeah this is doing run docker build okay why not be able to do this part just yet but the goal and what i do with every other ci system is set up a docker file that can capture all of my dependencies so a closure version or something like that and then can be executed by any ci service so circle ci uh for example we'll just take a container and run it for you and just tell you how it went um i was hoping github actions would do the same but weirdly they they seem to want you to kind of use their setup system so they're kind of almost their own equivalent of docker in some ways and i don't really want to do that because that means using ubuntu which is notoriously out of date for things like closure packages to install a closure package which i really don't want to do that's it's why i want to run it inside uh inside closure inside uh docker so they're putting it under dot github slash workflow slash something so if i create uh make do dash p dot github slash work flows slash uh test.yaml test.yaml i like four letter yaml oh no i didn't make that a directory armed github workflows test.yaml mvim so we're going to edit this file and let's go grab something how about one of these we'll just we'll try one of these uh maybe we just put this action in the root could do i don't like it though i might have to leave that as an exercise the viewer so cheery says you could run all of the closure stuff yeah on a docker container and use conjure outside the container a dev environment not only run tests so do you mean um for ci testing or running your development environment and your kind of n-reply inside a docker container um yeah and ripple and stuff inside doc container yeah you totally could do that i think we can actually do um docker run script dev and as long as we uh yeah so as long as we um forward this port outwards the mrapple port so you'd need to um specify the mr port ahead of time so we wouldn't know that this is port 39017 a lot of the closure work i do is inside docker containers and i will forward the ports out of those but you have to know the port ahead of time so you have to start n replica and tell it what port to use and bridge that through docker and say hey docker pass this port outwards and then i will edit outside mount my source code into the container and run the n-repo inside the container and then open the port up in the dock container and i'll connect to that and control it that way so you can keep everything dockerized but still do interactive development with it that's uh yeah very doable it doesn't want me to control see it huh oh because i ran it without dash ti yeah uh docker ps we need to i need to kill that out from outside there we go um that's why you run it with ti so i'm not going to set up the ci right now i'll do that another time but the goal would be use a docker container to run your tests and run that under ci which is what i do with conjure but under circle ci not under github actions but i'll probably get that working at some point so let me just make sure this is pushed and then anyone can come and grab this commit as well so push these commits so if you go to oh not that one where are we i know what happened if you go to oligal slash closure dap there you go and you go to the first commit you will be able to get this exact state so uh doc file uh license test scripts entry points dev stuff all those things um so if you're working on this theoretically you'd come down into the repo i'd open two terminals i'd have my source um in one of them one of my terminals i'd uh start my tests over here open another terminal so you want three and i'll do my script dev so now i have my tests running in uh one terminal i have my entrapple server started here i can connect to it over here now i can evaluate things live i can go and change test i can go and change test and break it it breaks i can fix the test it's fixed and i can evaluate stuff live and run the tests in my editor so this is the bare bones this is the absolute like minimum that you would need um the next step on from this is obviously ci but that really depends on the ci system you want to use and i haven't set up ci in like two years so uh i'm not fresh on that but i'll probably do another tutorial at some point and show how to actually get the ci working but that's not my expertise and there's people out there that are better than me at that um but this is how i recommend setting up a closure project it's how i recommend working with one um you can even um because we're using cider if you're using something like conjure or you're using any good ripple tooling uh you should be able to use uh well for me it's comma rr to refresh every file in the project as well so you can actually go through and make changes to loads of files hit comma rr and it will reload anything that changed so a bit like how your tests are reloading that can be a really nice workflow as well um yeah it works cool okay i'm gonna stop the video now but uh hopefully that's helpful um and yeah leave comments with questions and things like that and come and look at the readme and the repo if you want to grab this repo in its current state and then modify it yeah cool

Let's learn how to set up a fresh Clojure project with a beautiful test runner and a working CIDER nREPL for iterative REPL driven development. Sorry it gets very messy when I try to do CI! I was just completely lost but I thought I'd leave my confusion in for your amusement / interest. I hope you like the first section at least and got to learn something. Recorded live at https://www.twitch.tv/olicaluk Repository: https://github.com/Olical/clojure-dap Specific commit from the end of this video: https://github.com/Olical/clojure-dap/commit/0736b92ef878770076784f6ac33e9793b9b27370