The ULTIMATE homelab setup

Hey folks. So, after the past couple of years, I think I've finally perfected my home lab setup, and I want to share it with you. Now, it's probably famous last words, but um I will say I've been home labbing for the past decade, which is scary to say. And over that decade, I've used pretty much everything. I've used uh Naked Linux with just straight up systemd or with containers. I've done Proxmox. I've done OpenStack. I've done Kubernetes-based dros. I've done all sorts of things and used all sorts of automation approaches because I'm sure like many of you I would gladly spend dozens if not hundreds of hours to save myself the effort of ever manually doing something a second time. And so what I've found is Nyx OS along with a tool called Kolmina have really provided the most reliable resilient environment that I've had where I'm able to configure things super easily as we'll see in a moment here. But I also know that it just works. I can go away for a long time. I can come back and it just keeps on chugging along. Um, now if you're not too familiar with Nyx OS, it is kind of a confusing thing at first. You know, if we're on their website, we see declarative builds and deployments. Okay, it's reproducible, declarative, reliable. Got 120,000 packages and all sorts of cool stuff you can do. But this really, I don't think, sells it strongly enough. Once you actually start diving in, you'll start to see all that it can do. But Nyx is a functional programming language that was built for the Nyx package manager, which the Nyx package manager can run on any Linux system as well as Mac. Nyx OS happens to be a distribution based on the package manager and it goes deep. And so the idea is the entire system can be defined declaratively as code which for someone who loves automation this is a dream come true. And just to preface this so for the past 2 years I've been daily driving Nyx OS on my workstations as well as my home lab servers. And I will say my needs for my home lab have kind of adjusted over the years. So initially it was a playground. Now it's moved more towards being a an actual self-hosting environment where where I run dozens of services that I actually use all the time. I even have my own gy instance where I store all my source code and things like that. So for me it's a very important tool to have be reliable. I'm past the days where I want to scrap stuff all the time. Okay. Well, let us get into it. So I put together a simpler version of my own home lab setup. Um, I've actually put this together as a GitHub repo as a starter that any of you can use if you want. Um, in a little bit we'll actually go through how to use it, how to add stuff, but I first want to show you why Nyx OS is so cool and combined with Kina, which is the deployment manager. So, first things first, if you come down here, we'll see I have this set of three different hosts. Um, don't worry about the syntax. A lot of this is going to start to make sense as we get more and more through it. Um, but I have this Alpha, Bravo, and Charlie server. Now, these are just VMs on my machine. Um, we'll take a look at how this works shortly, but we have this structure here where I have hosts which have the configurations for each of these servers and then I have modules which are for all the different services we have defined. So, let's crack one of these open. So, first let's take a look at the alpha server. All right. So, this is going to be a lot to take in at first if you're not familiar with Nyxos, but what it essentially does is it provides all the different types of configuration options you'd want as properties that you can define. So, we have like networking and the host name. And so the way I like to think of this is if you compare this to more of an imperative system like Ansible. With Anible, you're going to have some explicit task that says, "Hey, go set the host name to alpha." In this case, with a declarative system, we're not telling it how to do that. We're telling it to do that. We're saying, "Hey, the host name should be this. We're basically giving it the blueprint, and it's going to ensure that that happens. We don't really care how it happens. In fact, we don't even know how it happens. We just know that at the end of the day, it is happening. And if we come over to the browser here, there's a really cool utility website called mynixOss.com. And what you can do is go to the options page. And you can see every single configuration option you have available in your system. So we might want to look for let's say uh containers. So if we're going into a container, we have all sorts of configuration options that we could have on said container. We have networking can set up firewall rules. We can see how that works on different interfaces. And for each option here, we can actually see what the defaults are if there are any. We can see the example and some details about it. And so immediately where this is really useful is if we come here and say I want to run a new service. Um, we can actually go to the options and go to services or we can just search for things. Um, here you can see we have 1,400 different services. So these are the types of services that we can actually start up that are prepackaged. Um, we'll see later that we can also do any sort of Docker container as well or Podman and all of those will work just fine as well. But if you find something you like, all you have to do um, let's just find something like Jupyter Hub and we can see. Okay, so this is a service. If we jump to the Jupyter Hub top level here, and we can find a bunch of configuration options. And so in this case, we know we'll want to enable it, of course. Um, but we'll also want to maybe set a custom port. It's on port 8000 right now. Um, we'll put on a custom port. So, if we switch back to the code here, um, we can maybe put this on Charlie. And what I can do is just say, all right, services. JupyterHub. We'll put it in squiggly brackets. And then we'll say enable equals true. and we'll say port is let's just say 444 and then down below here where we have the firewall um we can say all right we want port 44 444 amazing all right so with that um in fact just to prove this if we come back here and we go to charlie.loc local and we can see port 4444 does not resolve. So at the moment this is not in use. If we come to our terminal now what we can do is use kulina. So this is a deployment tool and so I can say kina apply on Charlie. So it's going to pick up the changes that we made and it's going to tell us hey this is actually you know you've made some changes they haven't been committed yet. So it's just letting us know that we can see it's gone through it's evaluated and it has pushed the system closure which is basically a fancy word for the entire configuration of the system that's been built has been pushed over. And if we come back to the browser now we can actually see the browser was still attempting to connect and finally it's now able to. Very cool. Now I don't know what the credentials are by default. So I don't know how to log into this but um you can see with almost no effort whatsoever literally what four lines here and then we added a new firewall rule. We now have a new service and if we wanted to remove it delete that and we can go remove that. If we switch back over here now and we reapply it we'll see now it has succeeded at that. And if we switch back to the browser try to refresh that now it's going to sit there forever. All right. now. So, as you can see, it's pretty straightforward to add new services. That's one of the beautiful things about using Nyx OS. Okay, so the real power of all of this though, um, it's nice to be able to add services like that. And that in and of itself is like a real game changer for me. But what I find even more powerful is the module system. We can create wrappers for everything. So if we go to we can say take a look at this. Right now we have this fresh RSS service. If I come here I have all of these options available. So I'm making it something that's enable. And then I'm setting things like okay what port do we want? Where do we want this to be dumped? Uh what time zone? What user? What group? All of this stuff AI actually turns out to be fantastic at because it's all purely declarative. So, actually one thing I've done in this repo is I've added a cloudMD, an agents.mmd, and an MCP.json that has an MCP server for Nyxos. So, it's very easy to start scaffolding stuff out. If you want to challenge run this, you know, you can do it all yourself, but for me, I just want this stuff to work. And ultimately using AI to speed up my process with working with my home lab or I should really say my home servers has really improved things. In any case, so here's how an actual module looks. So we can configure anything that we want to be reusable. So here we're creating a new system user. The services that we might look up in like the myixos.com. So the ones that come from Nyx packages like JupyterHub, those already package all this type of effort and so you can just, you know, wrap around that directly. But in this case, we're spinning up a container. And so we're going to want to create a system user so we can kind of isolate things that way. We can set up a data directory typically going to be by default in a varlib path here. And then we're actually defining the container itself. So it uses virtualization annoyingly with an s. It's a peculiarity of Nyx OS, but then we use OCI containers containers and we're defining a container called fresh RSS. We're giving it this image. We're telling it what port to bind and we're passing that in as a parameter. Getting some environment variables, doing the volume binding and adding some options. And we also set the firewall rule um as well as the systemd services. So what's brilliant about this type of setup is we configure this once and now we could choose all right now it's in the Charlie server but if I also wanted an instance of this on the Bravo server I don't need to think about how to move it around I can just copy and paste this little block over or I could have it on both this really takes advantage of the programming ideology around encapsulation right all the magic happens behind the scenes and we just expose a very simple interface where we have some configuration options Um, one useful thing to note though is we can see we have this empty list here for the containers with this map actually. Um, and same thing here with the firewall rules, these TCP ports, we have just an empty array. And one of the really cool things about this is all this will get merged automatically. So by virtue of us using this fresh RSS module that we built, we're actually automatically pulling in the TCP port that we defined at that level. So if we go back down here, we can see it's using whatever was passed in as the configuration which was by default 8080. And so all of that just magically works. And the way this works, again going back to the comparison to Ansible, where Anible you run a playbook and that goes through step by step and if something breaks halfway through, it's going to error out. But then you've actually got a system that's half broken or it's half with the new configuration, half with the old configuration, and you've got some error in the, you know, middle of it. With Nixos, what's cool is when you actually go to build it, it will ensure that everything is configured in a buildable form, if the build fails, then you know, it's going to crap out and say, "All right, here's the issue." But it's not going to push that change. And so there's this concept of a generation which is the entire encapsulation of all of the configurations all into a single what we saw there the closure and that gets passed along to the target system. And what's really cool about this is and so this catches all sorts of issues at build time. Now there are configuration options that you can create that will break at runtime um for a given application. It's not an infallible system because you're relying on thousands of other services potentially out there and each one of those has its own configuration options and all applications are breakable. So you still run into those problems. But what's really cool and you know maybe I'll do a full video at some point about Nixos as a workstation. What's amazing is what you can do is if you make a configuration change that ends up breaking your whole system, you can actually just roll back to the previous generation because it tracks multiple different generations. And so you can roll back and be back to the exact state you're in prior to whatever broke it. Anyway, pretty cool. So let's fly through kind of the rest of this situation here. So I've got a bunch of different services. We've got a gy one. In this case, we're wrapping one of the official services. So, services.git is one that's in Nyx packages. But you can see there's a lot of configuration options. So, I don't want to expose this every single time that I'm going to use it. I want to be able to encapsulate that and provide a standard way to interface with it. Same thing with Jenkins. This one's a bit simpler because Jenkins is a lot simpler. Doesn't have databases or anything like that. Here's another cool one. So, we're setting up Grafana. And with Graphfana, typically what you're going to do is you're going to have all sorts of configuration options like connecting to your data source. We can actually do that directly in here as code. So I'm passing in our Prometheus instance that we'll set up. And that's actually passed in again as a parameter. We're telling it where the dashboards are. And in this case, we're actually populating it with a node exporter in a default dashboard. And if I come back over here to Prometheus, we can see we have some default scrape configurations like it's going to scrape itself, but we'll be able to merge that with any custom ones we add in. And another beautiful part of encapsulation, we look at the node exporter. Um, this is one where every single system you're probably going to want to add this on. And so rather than having to define this each time, we can tell it, all right, on all of them, we want this default list of the collectors to be enabled. And we want all the you know the ports need to be enabled and here's the service and then this ends up getting used again each of our different hosts will have its own configuration options but notice here we are importing from this common.nix N where the common configurations will live that will go across all of them. I should say this is not some sort of Nyx magic. This is very explicit imports. So you're able to structure things exactly the way that you want. And you know we're able to configure things like our Nyx configuration can add a user here. I have an authorized key. But we can have all these default options like okay we want to make sure that every single host is going to have port 22 available so we can SSH into it and we want open SSH to be enabled. All that makes sense. Okay. So I think at this point hopefully you kind of understand the power of this. Now what I think will be useful is if we actually go ahead and set up a new server. So right now we have these three servers. So I'm going to come over here and so I'm using virtual machine manager. So what I'll do is I will create a new virtual machine. And so if you want to kind of do this same approach, you go to nixos.org/d download. Um you can choose to use the minimal ISO image if you want. But honestly for something like this, if you've actually if you're using VMs or if you've got KVM access, it's just simpler to use the ISO. You need to prove how hardcore you are, you know, go for it. But for me, I want whatever the simplest path is that gets me up and running instantly. So we'll download that. Okay. Now that we have that selected, we can go ahead and move forward. Can set that to whatever we want. And we can just give it some amount of storage. And we'll call this one, I suppose, delta. Let's make sure that's using, yeah, the default. That should be fine. All right. And so we'll just go through the installer. And so just go through the install process. Pretty straightforward. As this is a server, obviously we don't want a desktop. And we'll just go through use the defaults. And that'll just take a moment. All right, there we go. So, let's go ahead and hit restart and should take just a moment. All right, this is where it gets fun. So, if you've never used Nixos before, this is where it'll get a little bit tricky. So, first let's just go into root and we're going to cd into Etsy Nixos. And if we ls, we can see there's two files. So, it's the hardware configuration and this is something that's autogenerated. This is something that you're not going to need to change, but we will actually need to grab this. In fact, we'll grab both of these files. What we're going to do is we're going to modify configuration.nix. So, what you can do is run nyx-shell-p and then neoim or whatever your preferred editor is because doesn't come with that by default. Um, it doesn't even have vim by default annoyingly. But we can say nvim configuration.nix. And here we're going to want to do a few things. First, we want to set the host name. So, in this case, delta. We're going to want to jump down here and we will add that system package. Doesn't really matter. Um, but we want to do here is enable open SSH. And I think that's all we need to do for now. So, again, that was just update the host name. And then what we need to do is just make sure that Open SSH is enabled. So, let's save that. And then here's the important part. So we can say nixos-reu and then switch. And so what this is going to do is rebuild the configuration of the system, create the new generation, and then it will switch over to this new generation. All right. And so now if we take a look and see system control and we can say status SHD, we can see that is now running and it has been for 8 seconds. So see that just started. Last thing we need to do, be sure to grab that IP address. So that is 1 192.168.122.125. And just to confirm that, let's hop over to our terminal here. We'll SSH192.168.122.125. And we're in. There's actually one extra configuration option we're going to want. I forgot about this. So let's go back into the root. Let's go to Etsy Nyx OS. It's good practice to understand how this all works. We'll need to um open up that configuration file again. And now what I want to do is it doesn't actually matter where you put it, but I'm going to put in this security pseudo.wheel needs password is false. This way I'm actually able to remotely so when I'm shing into my user I don't need to type my password in order to elevate or you know become rude. This is important um when using these like automated deployment tools. So let's go ahead and do nxos rebuild switch. Very good. Now I come back here and I say uh let's just say pseudo ls. We can see no password. All right. Last thing I want to do is just reboot the system. that is to ensure that everything like the actual host name gets applied and all that. So we'll let that go. In the meantime, what we'll do is we'll come back here. So we're going to need to add a new host section here. So we'll add a folder called delta. And what we'll also do is go to this hosts.nix file. Now this is just kind of an organizational file that I created. This is not like a Nixxos specific thing or a Kina thing. This is just an organization thing. So this was 125. Why are these tags all control plane? I'm not actually using those. Um, so let me remove those. Anyway, not important. So delta is on that port. And then if we come down to flake.nix, we need to add a new section here. Delta. And then we'll change all the Charlie's. And we'll change that to delta. Oh, Charlie. Delta. There we go. And if we come up to the top here and so basically what this is doing is just defining within coina each of the different servers. So it knows the key alpha what the configurations are again hosts alpha IP. These are getting pulled from this hosts file. This is just an easier way. I don't like to repeat myself. I don't like to have the same IP defined in multiple places. So I find having something like this is just very handy. Okay. So that's all good. Um, by the way, so this approach we have here, if you have like hundreds of servers, would get annoying. And so what you could do is make a little function like this, this make host, and just have this loop over or have a map over each of the hosts that you're importing from that list. And as long as the configuration like that the same structure applies for all of them, um, you can do that and define it once and you can save yourself from all this repetition. Um, that being said, there are some advantages to this approach, too, because in some of these you might want to pass in additional things. So, pros and cons. In any case, next we need to go to Delta. And so, we're going to do here is I'm going to SCP all of the uh files from our Nixos directory into our Delta directory. And while we're at it, what I'm going to do is SSH copy ID over to that host. So, I don't need to type the password anymore. um that will of course get replaced as soon as we apply this change. So we had our host key set up here but for the first run this will make things simpler. All right. Now what we can do again as I said the hardware configuration you don't need to touch this. In fact it tells you do not touch this. And the reason for that is every system is going to have its own like device IDs and all this. So you don't want to mess with that. But what you can do is we can clean this up. And so we can get rid of a lot of this and we can import the common.nix which is going to save us from needing to do a lot of this stuff over and over again. So let's take out um let's just keep what we want. So we want the host name, we want the bootloader configurations because these are all VMs. They happen to be the same. But in most cases if you have a variety of different systems, you know, you might end up with different configurations here. So we'll keep those. All of this stuff again is going to be common across the systems. So we can just delete all of this. Open SSH. Yeah, all of this. The only thing that we need to keep is the system state version. So this is what we need. Um, now if you saw in the other configurations, I like to have these section headers. I think they're useful just to understand where everything is. It keeps them more organized. So you could go through and add all of those if you want. Um, the important thing is we have these value set and let's just come up with an example of what we want this to be. Really quick, one thing I do want to show is just how good AI is with kind of dealing with this stuff. So, if we jump back over here for a moment, I'm going to open up Claude Code and what I'll tell it is update the Delta configuration the same same organization style as the other hosts. And there we go. So, it's going to automatically do all this. Of course, you could have just copied this over as well. But let's get back to the code now. We can see got a much nicer setup. It's even added a message of the day. So, we can have that as well as the TCP ports. Very good. So, I was looking at Linux server.io, which is, you know, it's an organization that just makes well-designed containers for all sorts of different applications. And so, I found this open VS Code server. So, I wanted to see do we have on Nixos a pre-made service for this? And it turns out we do. So, what I want us to do now is to create a new module that will actually create this. Um, again, we could go the container approach if we wanted with Linux server, but I actually want to do this as a native service because it is an option. All right, so I've gone ahead and just added this quick service here, this open VS Code server. I've enabled it. adding this on a port and adding a couple extra packages so we have some runtime environments to work with. So let's see if this works. So what we can do is go ahead and run colina apply on delta and we're going to actually see an error here. But this is a very important error to understand because you'll run into it a lot and it's often very confusing and you'll see all this stuff and you're going to think what in the world you know config deployment like what's the issue but the issue is right here. This file does not exist. This is because if we look at git status, we're actually working with some files that don't yet exist in the context of git. So let's add all those. And now if we run the same command, we'll see it's going to go through and it will apply it. But it's not happy with that. So let's remove that. Ah, here's another issue. Forgot about this. So on the first run, basically some of our configuration from common.nix isn't over there yet. Um, so what we need to do is actually run this again. We can say build on target. Um, this is an option you actually might want to do quite a bit. If you're on like a laptop, when you're trying to deploy to five different servers at once, it can make sense that you actually want each of the servers to do the build rather than your laptop. Because right now if you just do clamina apply without the build on target it builds it locally and then as soon as that's done it passes it over which is kind of the good default if you're using like production servers. Obviously you don't want your production apps competing with the build for the new version. That would be a very strange approach if that was the case. But in home labs you that can make a lot of sense. In any case that has worked. So what we can do is let's just quickly grab from our host file. I already forgot the IP. So grab that and then we'll go back to the browser 122.125 and then port 3333. Ah again I've made a classic blender. So if we look at this ISD and we can see it's available on local host. That's not what we want. We want the host to be set to 0.0.0.0. So it's actually accessible externally. So let's go ahead and do this again. This time we don't need to build on target which by the way the reason for that I don't like to leave things unexplained. So basically what we have here is this trusted users setting and we've got my user in the wheel group. So this trusted users this wasn't configured on the initial setup but once we applied the configuration with the common.nix mix. Now I'm able to actually send stuff over um via that user. Anyway, that's how that works. We've rebuilt this. So in theory, now if we go back to the browser, we can see well we've got progress. However, what we need to do is actually I'm going to just grab in the logs. There was a I just want this part. So there's a token. And there we have it. We have VS Code in our browser connected to the Delta server. And through here, we could add a new file. What do we want? I think we only added Go. So, let's just see main.go. Sure. Uh, install that. Let's do it. Install Vim. Fantastic. All right. Whatever. You know, this isn't really an ideal setup here. We're missing all of our tools. But in any case, we're off to something. we have a usable starting point. All right, last thing I want to go into is kind of the setup that we have here in terms of um so we've got these different servers here. One of the challenges that you often face with home lab setups is how you deal with actual traffic getting into them because I mean realistically as cool as it is to go to 1 192.168.122.125 3333, it's a lot nicer to be able to go to jenkins.local local or if we want we can go to we can go to graphfana.local local and we can see all that cool stuff that we set up with the node exporter. Again, all of this as code, not configured in the UI and all the stuff just works. But it's a lot cooler to have that as an actual host name and one that has HTTPS rather than having the certificate issues all the time. Now, in my own home lab, I actually have a real domain and I use let's encrypt. I use the DNS approach to uh generating the searchs, but we actually can just generate our own internal CA. And so I've done that in this setup for you. So if you want to just take this and try to run it, here's how this works. So first we have on alpha, we have uh this reverse proxy. And if we go to the reverse proxy module, um there's a lot of stuff here where we're basically wrapping around engine x. So, we've got the service engine X. We're adding the default configurations. And then what we're doing is for the virtual hosts, we're actually mapping over all these things we're passing in. And so, we're able to take things like our SSL certificates, locations, configuration options like the the headers, all that stuff. And then we expose all of this so that wherever we want to set up a reverse proxy like on the alpha server all we have to do is set up a map where we have all right Jenkins.local local is going to target the alpha server on port 8080. Graphana is going to be on alpha on 3000 and so on. And with this we have a very scalable approach. And then we have these uh this self-signed CA that I've set up. And again this is nice and encapsulated. The actual logic behind this is quite ugly. Go in here. Take a look at the self-signed CA.nix. We can see we have some options. And then we go through here and we have this system B service. And so this is of type oneshot. So this means it's not like a server that's running forever. This is just run this and then it's done. But what this does is it goes through and it'll set up um the private keys and all this. And you can read through here if you want, but it just sets everything up and then it goes through and actually enables this uh CA trust on this environment. Again, not very pretty, but what it gives us is now self-signed, which again, if if you actually want to use this, um, I put together this little guide here where again, this is kind of hard-coded for this particular setup. But, um, basically what you can do is just take the certificate that's generated and then load it into your browser. So, when you're hitting it like we have here, we can see connection is secure, verified by fleet internal CA. So, this is one that has been added by me. And with all of that, I think this is probably a good place to wrap it up. We've gone through a whole lot. Again, I'm going to link this down below. So, when you look at something like Nix OS, where there's a ton of different things that you don't really understand fully, I just want you to consider that when you first started using Linux, there's probably like 90% of the system that you don't touch and you don't really understand. Like, if you look at the root file path, all the directories there, like did you have any idea what any of those did? No. but doesn't actually prevent you from using the system. So I think having a system like this where you're able to just get started very quickly with the configuration side of things. You don't need to worry about the fancy aspects of the language, especially if you're using AI, it can help you with a lot of that. The beauty is it's declarative. So if the syntax is wrong, it'll just fail. You're not going to be stuck in a halfway complete state. In any case, yeah, just if you want to go down this path, try it out. Start with some VMs. It's a lot simpler that way. And over time you'll start to understand more and more of the system. Um a last thought. So if you go to my Nyx OS and if you go to anything. So let's say we got Jupyter Hub here. If you click into this service, we actually see this declarations here. And this brings us into the Nyx code. And so we can see literally the exact code for this service. So if you want to build your own official service and you can check out what these look like. Uh but beyond that, if we go to say like networking.hostname hostn name. You can go in here and we can see this is part of the network interfaces and we can see how all of this works under the hood as well. This is where when you start to dive deep and you want to understand fully how all this stuff comes together, you know, the source code is all here. You're able to dig in and do whatever you want. All right. Well, with that, I think it's time to wrap. So, hope this has been helpful and I will see you in the next

IMO NixOS is the perfect distro for homelabbing/self-hosting these days. The starter template: https://github.com/logandonley/fleet Colmena: https://github.com/zhaofengli/colmena