CLIs Serving UIs

over the years I've had a lot of luck building uis into my CLI tooling perhaps the most successful example I've seen in my career was the ton Zoo CLI provided by VMware the cool thing about this CLI is that rather than remembering a bunch of configuration options and flags for creating clusters the user had a UI option that would pop up a dialogue talk to the cloud provider and through these dialogues help them set those different options behind the scene the UI was just creating that same yaml file in those same Flags to do the creation for the user but they didn't have to understand all that complexity they just had to choose options based on what was available to them and while this example is really good I think there's actually a case to be made for a far simpler approach where we're not using JavaScript Frameworks or anything really too complex the example I'm going to go over today is a utility called Proctor now Proctor is a command line utility and it enables us to do a couple things one of the common things we'll use for Proctors to list out process details kind of like PS along with process details it'll give us some information like what the path of that process is additionally it will give us things like the Sha sum of that binary and as a tool to explore processes you'll typically find a process you're interested in here and then you'll run a command similar to this get command with an ID provided I'll ask for the output in Json and pretty print it with JQ and with this output in place I now get a bunch of details about the process from the Kernel's perspective things like the virtual memory space the resonance set size and all other types of settings another really common use for Proctors to understand the hierarchical relationship of processes so let's say that I want to run the tree command and the tree command is effectively going to show what my processes I search for so 58354 or Go please and then all the processes that were used to launch it so in this case we can go all the way down to the init system we can see that was using sshd this then launched through bash vs code server then launched and that's what called go please which is the go language server so this is a case where vs code is remotely pulling into a system to do coding and these are the different processes that are kicking things and owning things to actually give us language server functionality so Proctor is a command line like any other it has Flags it has commands but what if we want to provide an easier on-ramp for interacting with Proctor or perhaps just a different user experience what if we could just go in and say pseudoproctor UI and then this would serve a user interface that we could load into our browser and get the similar result sets that we were looking at before additionally what if it was navigatable so we could go in and refresh and get a list of the new processes and their IDs we could find the process we were looking for before this time it's go please looks like the ID has maybe changed click on go please get information about all of the different kernel level details and then view that process hierarchy again in a cool tree view with again hyperlinks that allow us to find some other process click on it and navigate to it and the key thing here being can we do all this with minimal overhead using the same libraries and data structures that we were using in our CLI utility and that's what we're going to talk about today how we can turn this Proctor UI command into something where it serves up an experience similar to that of the CLI let's take a look at the architecture that can facilitate this type of UI workflow I think one of the key things when building out CLI tools assuming they're not super trivial is to think about the composition of our packages my focus area when I think this through is really considering what these underlying libraries should be and a lot of times I don't know out of the gate I just kind of start coding everything in a main file and then breaking them out as we go through but in my case plib handles processes Source lib handles source code GitHub handles metadata about GitHub these are all features that are sort of baked into Proctor and by pulling out these packages I now have the ability to bring a CLI on top of it and consume these underlying libraries eventually this will also enable us to bring a UI in and consume those same libraries and there could even be a future State up here which could be consuming these libraries to create services in a larger platform and so on so just by generating these constructs we have a lot of different ways where we can kind of think about what the import Cycles will look like and how we can ensure that introducing something like a UI is really not recreating the wheel of of our application to start us off here we are going to dig into creating that UI package so if we look into the packages that are available out of the gate we can see a couple p-lib Source platforms and so on and as the diagram suggested we're going to start off with a UI package now we're also going to make a new file and this is going to be inside of UI called ui.go and this will be the definition of the package UI now to get us started off we want to create a struct that we can instantiate from the command line that holds details about the UI so this will be a type it'll be called UI it will be a struct and it will have a few different parameters inside of it the first parameter it's going to have is going to be an inspector and this inspector is going to come from a library inside of Proctor an existing one called p-lib and it's going to have a inspector pleb.insector type so if we just take a quick look at inspector this is something that's going to load processes and gather all of the available process information from a host this will be a very key thing to key keep an instance of inside of the UI so along with inspector we're also going to hold something called the data struct and it'll make a little bit of sense later why we hold data this will be things we get out of the inspector you can think of it like we're caching results so we can easily feed it back to the UI without having to call the inspector every time but we'll get to the data definition in a moment then we're also going to create a refresh lock and this is going to be part of the sync.mutex package inside of go sync.mutex is going to provide Mutual exclusion locks so that we can ensure if we have something like a refresh that could be updating processes if we're doing a read we don't step on each other's toes and end up with potentially bad data coming back to our client so this is the construct of the UI that we'll be creating instances of the next thing we want to do is create the data struct so this will be data and inside of data since I know Proctor already I know that I'm going to want to keep track of a couple things the first thing I'm going to keep track of is the last re be fresh time so refresh this will just be a time.time object in go and will allow us to keep track of a structured date format that we could present in a multitude of ways over time and then finally I'm going to keep track of a variable called PS and PS is going to be that cache if you will of the processes that we get from the inspector so the type that we're going to grab here is going to be ps.processes and this is actually funny enough just a map where the key is the PID and then it points to another type in p-lib called a process and you could probably imagine a process holds all the different fields that the kernel knows about relating to that specific process all right now we're looking pretty good what we're going to do next is we're going to create a Constructor for this UI struct so we need to be able to create a new instance of this so what we'll do for the Constructor is I'll bring it up to the top here we'll do a funk and this will be new since it's in the UI package just calling new would make sense that it creates a new UI it'll return a pointer to that UI and we're going to do quite a few things in this initial Constructor so the first thing I'm going to do is declare a error variable because it'll be multiple times where I'm overwriting this and what we're going to do is start off by creating a new inspector that we store in this new inspector could potentially be an error because the Constructor will be calling is plibib dot new inspector okay so pleb.new inspector creates an inspector takes an optional config and could potentially return an error with this new inspector created we're now ready to set up the new UI so the new UI is going to equal a instance of a UI so that will be this and we'll close it out here and we can use the fill UI to get all the different fields inside of here so for the new UI we're going to obviously set new inspector to new inspector we will be setting data to empty and then we're going to create that mutex as a brand new mutex as well new UI is all looking good and for now if there's an error from this inspector here we probably should do something with it and just to make sure in fact I don't even think we need this far error so we'll get rid of that if error does not equal nil so does not equal nil we'll just go ahead and panic in this case with the error because it couldn't spin up the web server we can get the full stack Trace all of that good stuff so new UI is right here and in fact instead of storing new UI let's just go ahead and return the UI just like that and this will need to be of course a reference to that UI and with all that in place we've now got the Constructor let's make sure you can see the whole thing which will look something like this so new UI creates a new Constructor holds that instance of the inspector and initializes some default values with the Constructor in place I know the next thing I'm going to want to do is set up kind of a run command that can kick off an HTTP server to feed our UI so let's just think about how the composition of this API is going to look I know that I'm going to have a root path and I'm going to want the root path to give a list of all of the processes I also know that I'm going to want to offer a refresh command that can go out to the system and get an updated view of the that so this will update and then we probably just want after the update for it to Circle back to the list all processes so kind of like a redirect I also know that I want to offer a process command here so process slash and this will be a arbitrary PID that gets put in and this will give us the process details and then finally after we get process details I also want the ability to get that hierarchy so tree right tree for PID and I want this to be able to go out and give the tree for the processes so parent processes all the way down to the init system like we saw in that initial example so terrible handwriting aside let's go in and start off by just setting some cons up that will store these paths so at the top of the file I'm going to go ahead and grab a new const block so these will be constant variables that do not change and inside of this const block we'll do a couple first I'm actually going to capture the port because I know that I will be needing that the port we'll expose on will be 8080 probably something we'd want to make configurable at some point we'll do that refresh path which we know is just going to be forward slash refresh we will do the process path so process path and process path will be the process forward slash where the ID will come after and then we'll do the process tree path or we'll just call it tree path I guess to keep it simple tree path will be right here this will be called tree similar idea and now we've got these consts that we can throw together now if we go to the bottom of the file for a moment I'm going to set up the first method on the UI struct so to set that up this will be a funk it will be UI which is a pointer to UI or attached to a pointer of UI this will then just be called run and run will run the UI as you could likely imagine so when we call run we will go down and actually to keep this consistent with my blog post I'll call it run UI and now we're going to set up a bunch of different handlers so handlers are basically a way for you to map a request that comes in with a function that can get called based on that request so using the HTTP package we can declare the handle Funk handle Funk will ask us again what that path is and then what is the thing that we want to call based on the request coming in on that so we'll do a UI dot so this will be a new method that we will be implementing soon and we'll call this handle all processes okay so this is our first one then inside of here we are going to do the refresh path which we will say handle refresh and then along with handle refresh let's go ahead and do the process path const and this will be UI dot handle process we'll call it details and then as the final one as you could guess is going to be the pros actually just the tree path I think is what we called it right tree path yep which will be handle process tree all right so we've got handle process details handle process tree handle process refresh and handle processes all set and good to go of course these aren't implemented yet but this is what will be redirected into now we'll just set up the actual listener itself and we'll maybe just do a quick log statement that says log printf and we will say that we are serving at and we'll make sure we print out the port here in a string format so we're serving here and we will do this at the Port constant all right and since this is a log F let's make sure we do that great we'll do a quick import of log and finally we will do the serve itself so one cool way that we can kind of make sure we understand if something blows up is we can wrap this in a panic and we're going to do HTTP listen and serve and then listen and serve is just going to take the argument of the port okay so this will be port and then let's see what the I know there's a second argument on this the Handler if we wanted to provide a uh alternative Handler which we're not going to so this will be nil so now we've got run UI set up and we've got all of our mappings inside of the UI package for what we want to do when it's time to hit or satisfy these requests as a Next Step let's code these handle functions the second argument in the handle function is this Handler and Handler expects an implementation that will be a function where there are two arguments one is the response writer and then two is the HTTP request itself so to set these up since they're attached to UI we're just going to do Funk we're going to put it inside of UI or call it UI and make it a pointer to UI this will be handle all processes great handle all processes is again going to accept that writer which we'll say is W so HTTP dot response writer and then for R we will do H pointer to http pointer to http dot request okay so this is our initial setup for the handle processes we can see up here it's looking pretty good and now we can really just duplicate this for all of our other ones like refresh all process details and tree so let's bring this to the top and yes we will start right here so we will do handle refresh we will do handle details and then we will do handle tree okay so with this in place we now have all of these set up and registered although we need to now Implement them to put some logic in place in this video I'm going to focus on the implementation of just one of these handlers primarily because refactoring everything and building it out will take a really long time and I've got the source code linked in my blog that you can check out but we'll focus on handle all processes so that we can understand the patterns and mappings that will be going on and then you can check out some of these other ones at a later time again via my site so handle all processes how are we going to set this up well the first thing I know out of the gate is that when we do these different function calls there's certainly going to be mutation of data via refresh Gathering of data through that cache and we want to make sure that we're not stepping on the toes of different processes so the first thing we'll do is we'll grab the refresh lock and the lock command uses that Mutual exclusion to ensure that we can either block or gather the lock and then using go we can Leverage The defer keywords that when we exit this function we release the lock and that could be done via the refresh lock unlock command all right now that we've got this all set up we are ready to start getting the processes themselves so inside of UI data there is that PS field that I had mentioned before and PS is going to allow us to Cache a list of processes and the call I'm going to make I know might also return an error so this is going to be on top of UI dot inspector specifically the API call for inspector which is get processes so get processes is going to either using an existing cache on the host or by actually accessing the proc FS file system is going to get process information and then return that back to us so we're storing that in the UI struct as a local cache if the error does not if the error does not equal nil here we are going to eventually return a error so we'll do to do return error here so this will probably be a status code to the client but we'll come back to that in a bit and then last but not least if we look inside of the UI inspector we'll notice that there's also a get last load time function in here too or method rather and get last load time will tell us when the last time that process information was gathered this will be helpful in our UI so we will capture this in ui.data in the last refresh field and store it there okay so this all looks pretty good PS is obviously not a new variable so we're going to do this and say VAR air error to capture that and make Go Happy okay so we're capturing all the processes we are getting the last refresh and the next thing that we want to look at is the templating itself to get templates set up I'm going to start off by creating a new file and I've actually pre-created this in the UI package called static.go so this file doesn't have much fancy going on it's a bunch of variables that are storing static HTML there are a couple ways to do this and go there's a way to actually statically compile external assets into the binary but in this case it's actually easy to use these variables they work just great so you'll see a pattern here we'll start off with UI header which is just the beginning of the HTML page along with a bunch of style sheet information as we move beyond that there's also a footer so this will be placed at the very bottom closing some tags out and as you could probably imagine imagine what will happen when we create our template is we'll start with the header we'll start out with the custom content for that page or template for that page and then we'll append the footer to the end as well the one page I'll be focusing on coding out with you today is going to be the all process view now this requires some knowledge of go templates but I'll do I'll try to do a little bit of it justice by explaining it here when you use a go template what happens is you ask it to render the information and a lot of this is just static info that will render as is but then you'll see this kind of mustache or curly brace notation here what happens when we run render is we're going to be able to pass an arbitrary struct into the render function and when we render that we're going to be able to access fields on that so in this case I can actually look up the last refresh data field in fact if you go back to our ui.go for a moment and go to the very top you might remember in the data struct there is the last refresh field so you can kind of hint off of where we're headed we'll have eventually be passing this full data struct in and as long as those are publicly accessible methods or sorry Fields rather we'll be able to access them here and then we'll get down into probably the more important and complex aspect which is the ability to iterate over that struct so similar to how we were accessing a field another field we can access is PS and PS inside of here is going to give us this process list which is effectively a map so we can iterate over maps by looking at keys and values and if we look here we're capturing key and value just like that so we're going to be printing out an HTML table that has the key right here and then next to the key and by the way the key is like the PID or the process name or I guess PID in this case the key and then we'll be able to access the actual fields on that on the value so things like command name and binary Shaw are the things that we will print out in our HTML table so it will effectively range through end here and just continue to print row and row and row in row based on which which struct we send into it which will effectively be data so we've got static.go good to go let's now take a moment to look at the UI implementation inside of here back inside a UI we can create the template so to set up the template this is going to be using the template dot new function in go and this is going to start off with just naming the template so we'll call this View all right and then when we've created the new template there's a couple different things that we're going to do with it the first thing we're going to do is tell it what we specifically want to parse so these are going to be the variables that I had mentioned before so we are going to start off with UI header we know that we want to concat that to some other template which is going to be the view all processes or all process View and then we want to append that with the UI footer as well so this will be the section of HTML pieces that get all brought together when this gets parsed and obviously all process view is the one with those go templates that we're going to be rendering as we go through so once we create this new template we're going to end up with a template that we'll store here or potentially an error and if error does not equal nil we will once again actually we'll just copy this from here we are once again going to have some error case we need to respond to the client with and we'll deal with that in a moment okay so this is sort of our data Gathering piece of the puzzle right here and then this is our template rendering piece of the puzzle that we're going to want to look at in a moment as well okay so the template's good to go now let's talk about doing the actual not parsing but I guess in this case the rendering of the template if you will okay so with that template coming back the next thing that we can do is on the template we can call the execute command so execute is basically going to say give me something to write to so this could be like a file in our case it's going to be an HTTP response and then give me some data as well that I'm going to render against that data is the struct I was telling you about the specific struct that the go template will expect to render so for the writer here we are going to push ourselves into W which is what we captured in the function as the HTTP response writer and then along with that we are going to ask to give it UI dot data which will have the data fields inside of it reminder these data fields are last refresh and PS which is the cache of processes Okay so we've got T dot execute being called here this can also return an error so we will do error equals and then if error does not equal nil we will once again want to do something bad here so we'll return that error great okay so now we're executing the template and in theory this should be the response we want to write back to the client it should work without issue let's now test this out by doing a simple command creation where we can run something like Proctor space UI and kick off this process now if I run a make build in the Proctor project this will build the binary out and if we go to the build location out Proctor and run it these are the current commands that are available so again what we're looking for here is to add a new available command that is called UI and there are a couple places using the Cobra library for this project that we will wire this in all right so to wire in that CLI command we are going to hop back in and in this project there is a package called CMD which hosts most of the information for how the command line utility Taps into those underlying packages so we can add the UI command it's going to be a command on the root command so this will be add command and we will call it UI CMD now obviously UI CMD is not defined so let's go to the definition of process CMD copy that and set up our UI CMD inside of here so this is a cobra command Cobra is a framework that lets us easily create a command line utilities with flags and arguments and all kinds of stuff the use is going to be UI we won't have any shortcuts or aliases and then this will be create a web based UI for Proctor okay and then the command that we're going to run will be run UI which obviously does not exist either basically when this command calls this will be the specific command that Cobra should run which will eventually kick off our UI package so to set up run UI let's now go into something like run process here and do something similar so if I copy run process and paste this here for run process we're going to call this run UI okay now run UI is going to be pretty simple for us in fact we don't even need any of this stuff in the Middle Run UI is just going to call that UI package so if you remember we have UI dot new new is going to create a new iteration of the UI and then we can call run UI which will register our handlers and run that server so this is basically all we need to do to set our Command up and get it moving and I will just delete the comment for now you can clean that up if you're writing your own and you know put something more descriptive in so now we've wired this in we've created a run UI function we have set up a command in Cobra that describes how it should be used and we've registered that to the root command so we now if we exit out should be able to see that UI command let's do that now so if I make build and looks like in commandfs.go I did mess something up so let's see this real quick command defs.go run UI okay we did not specify run UI let's see if there's a typo run UI which is ah we probably should keep this consistent right so let's call this run UI all capital and now we should be good to go so let's try this again we will make build run it out Proctor or sorry out Proctor and now we have a UI command so that UI command should be able to set up a user interface we'll do out Proctor UI and now we can see it's serving at 8080 and we should be ready to check out the interface opening up a browser will check out the final output here and you can see that we now have a full table view rendered from that go template with all of our different processes the checksum of the binary and they're all hyperlinked so while this hyperlink won't do anything yet you can see it is uh well you can assume that it's going to be navigating to that rest API that we have been setting up and I'll show you kind of the finished piece here and then we've also got a refresh button in the top that can eventually once we implement it can actually go in and refresh the process list and so on so with this now rendering and being served up in its own little command let me fast forward to the fully baked solution and talk about all the different things it can be doing back in the UI package I've got everything set up and implemented so I'll just kind of talk through all the different pieces that are going on here so we still got our handlers registered and good to go and handle all processes is still here but it's a little bit more refactored so what we've got going on here is when there are failures in fact I probably need to put this failure up here to be honest with you I have a right failure function when this happens so whenever right failure needs to occur I send a internal server error back to the client and then I have an error view template that I parse but other than that for the most part we are still rendering as is for this process there's also a create template helper function here that Returns the template back to the caller this just abstracts that whole idea where we are bringing together the UI header template and UI footer additionally if we go down to the handle refresh here handle refresh acquires a lock it clears the process cache and then it actually redirects the client back to that initial root page if you will and then handle process details probably the most interesting one 1. handle process details is giving us a full list of the process details themselves so basically you can think about it like everything that the Linux kernel knows about the process details now what ended up happening with the process details if you dig into the source code that you might find a little interesting is handle process details required using reflection inside of go to find the fields on the struck and print those back so if I look over at get process details in fact I'll kind of scroll down into where the reflection's happening this is a function that's going to go through and understand every field name in the struct and return a detailed key value so the key will be the field the value will be the actual kernel value so this is kind of a cool little trick it can be dangerous to do Reflections you got to kind of know what you're doing but effectively we are pulling all of that together and if you do dig into the source code what ends up happening from a template perspective is that we end up registering let me find it in here we end up registering that function of get process details as a function that we can call inside of the go template so this is kind of an interesting cool thing you can actually take an arbitrary name of something and then map it to a function in a go template and then you can call that function and render the results that come back from it inside of your template so P Deets represents calling get process details in fact I'll just show you inside of that UI static file real quick if we go down to PD right here you can see that inside of one of the go templates I am calling pdets this is where I actually give it a process that's what the dot is and then pdets is returning again that field and then the value and then I'm creating a full table out of that so there's a lot of stuff in here but largely the key thing that you want to understand from a pattern perspective before you get two in the Weeds about some of the go nuances is that all we're doing at a high level is rendering a bunch of HTTP handlers to handle a bunch of paths and we're calling those underlying libraries to make that possible so if we take one more look at what the end State should look like we'll make build in here and then we will do a proctor out again for UI we'll go back to our browser and refresh it here okay and now if we refresh and click on some different processes inside of here we can actually see a big process table that's going on this is for process one two two zero here so let's click on um let's see what we've got we'll do network manager applet we can see a bunch of details this all happen through reflection to give us this fully rendered table of all the different details about the process we can click this process hierarchy button which will actually show us the details around that tree in fact what's kind of cool about this tree is it's respecting permissions so if we did a pseudo out of Proctor UI real quick so let's do that I bet you will get even more processes because we should be able to see all the way down to the init system so there we go now we can see NM applet xfce light DM light DM and system D things that even our own user can't see inside of here and since these are all hyperlinked in the template we can click on them we can get their details and so on so this concludes baking a UI inside of our CLI although we got into the weeds of some code hopefully you can see how the patterns here are conceptually not that complex and it's something that you can set up as well and next time you're building a CLI it might also remind you that it would be worth considering some of the details around how you structure your packages because again this CLI was built fully with the green box pulling into these pink libraries and then when it came time to have the UI consume them they were all set up to fire through and gather that information as well so I hope you found this interesting something to consider for your future CLI creations and I'll see you in the next one

Blog post: https://joshrosso.com/c/cli-ui Countless times I’ve found myself shipping CLI tooling amongst teams or co-workers. The challenge of onboarding users to a new CLI is that there’s a whole new set of commands, subcommands, flags, and arguments for the user to work with. While my daily workflows are essentially built on CLIs, there’s no doubting that a UI can ease the on-ramp and usability of your tooling. Overview: 00:00:00 Architecture: 00:03:14 Data Struct: 00:04:24 Path & URIs: 00:09:15 HTTP Handlers: 00:11:24 Implementation: 00:15:33 CLI Wiring: 00:24:01 Full Solution: 00:28:09 Closing: 00:32:43