7 Important Things To Know About Proxmox! (esp. for VMWare Refugees)

Hello and welcome VMware refugees. V VMware refugees. Yes, VMware refugees. If you're doing your homework on Proxmox or even if you're not a Proxmox refugee, I've got some hard one knowledge to share about Proxmox. And if you're coming from VMware, it's going to be even more like a bonus round because Proxmox is going to feel familiar in all the right ways, but it'll also be a little bit dangerous for you in some new interesting ways. Dangerous because freedom. That'll make sense by the end. Hang in there. Uh today I want to talk about seven things that I've noticed newcomers from VMware consistently trip over in our user community. Some of the things not from VMware just in general. And so this isn't really beginner stuff, but I'm going to try to keep it fairly high level. If you're getting started with Proxmox, I might suggest checking out the video that I just did with Open Metal on setting up a Proxmox cluster. uh it's aimed at setting it up on their dedicated server cloud instances, but it applies even if you're just setting it up locally pretty much except for the networking part. So if you're doing experiments or testing Proxmbox, all of that will make sense from the other video. going through the motions and setting up a cluster is good for you because there are architectural differences with what you may already know and your mental model and that's probably the part that's going to need updating now because Proxmox isn't VMware. Uh it's, you know, you just don't think of it as like a cheaper VMware. Like I I I don't know. Let's just get into it. [music] Storage. Storage. This is the number one thing. If you migrate and it feels slower than VMware, something is wrong with the migration. So, there's a lot to talk about there, but look, here's a screenshot of 9 GB per second in a Windows VM. And this is coming over NFS RDMMA. Yeah, storage can be fine. Okay, so VMware in my opinion trained us to think in terms of data stores. And this is true of other platforms as well. In the past almost since the beginning of the uh of time the migration away from mainframes when we talk about storage and the storage subsystem the storage system itself is responsible for its own integrity and reliability and performance consistency for VMware VMFS it's a file system abstraction that deliberately hides the storage stack VMware assumes that the array RAID controller or VSAN handles the integrity and the caching and the failure semantics and all of that. Prox um doesn't hide this at all. Storage usually means ZFS, ZFS data sets, ZV volumes, you know, ZFS stuff or LVM and software raid with LVM or SEF or NFS and ICE or some combination. The flexibility here is incredible, but it comes with landmines or rakes scattered about that you might step on. And it's important for you to understand that at a fundamental level, Proxmox does not own these storage technologies. VMware, VMFS, HyperV has NTFS and storage spaces. ZFS here is the same ZFS you get from Sun Micro on Sun Micro systems at the turn of the 20th century, I'm sorry, 21st century. 21st century ZFS is uh a common recommendation with Proxmox. Linux's LVM is a close second. But understand ZFS isn't just a file system. It's a storage system. It provides end to-end check sums for both data and metadata. It scrubs and healing with redundancy works differently than RAID controllers. And it's actually good. RAID 5 when it encounters an inconsistency like on a modern LSI hardware raid controller, it is a coin flip as to whether you're going to get the good data or just it's going to spread the corruption. And that sounds crazy, but I've been through the code snapshots and replication bakedin replication is super handy for sync across nodes. Uh more about that in a second. VMFS, uh VMFS does none of this. VMF VMFS assumes that the lower tier storage layer like the underlying storage is going to handle all that. Okay. But VMFS did add some primitives for copy on write and some transactional stuff to help with VSAN but mostly it's not really comparable to the features that you get with something like ZFS. ZFS has one other killer feature it's important to understand and as the superstar administrator that you are aspiring to be that you should understand. ZFS is insanely flexible. As a practical matter, data sets are subvols. And when you have your ZFS storage pool, you can tune the specific geometry of ZFS features like DDUP, record size, compression, and much more. And these tunables can be tuned to match the workload that you're running. And these tunables have an outsized impact on performance. And you can do that with data sets inside of a ZFS volume. You can pick if you want a little bit of a right buffer for synchronous writes to improve performance. Are you comfortable with 5 seconds of data being inflight, 30 seconds of data? You're in control. It's you. That's a U choice. Foreshadowing. So yeah, ZFS has a lot of overhead. Modern NVME is so fast it doesn't even make sense to offload that overhead to a PCIe card in most scenarios because the NVME drives are faster than the PCIe card can be. And I did some videos on that a while back, too. So let's look at a practical example. We have a VM in VMware and we we searched the internet to find out how to get from here to there. One tutorial would suggest you convert your VMDK to Qcow 2. Qcow 2 is the QMU copy on write file format. QMU is the name of the virtualization subsystems primitives that exist in Linux and cow is literally copy onrite. But the storage system that we're using here is already copy onrite. The copy on mechanism is critical for snapshotting and systems to work with you know that sort of thing. And Proxmark is smart enough to use the ZFS snapshot stuff when ZFS is the underlying storage mechanism. But the disk format for virtualizations in that scenario should be raw instead of Qcow 2. So you see Qcow 2 is copy on right on top of copy on write if you're using it on ZFS. And that means for small write bursts, you're more than doubling the amount of writes that you have to do. That's right amplification. This graph here in a VM in Proxmox can show IO right pressure and write stalls which is a health metric you can keep an eye on with your VMs and with the Proxmox backend but that's you know one of the places where it comes from. The other problem that you can run into when you are converting a VMDK to RAW or QA2 is sector alignment. you know your underlying file system it's like 16 kilobytes 32 kilobytes 64 kilobytes and if that misalign with the ZFS data set like you get half of a you know 8 kilobytes from this cluster and 8 kilobytes from that cluster you'll have to do twice as many so you want to ensure that all of that aligns you want the virtual disc to align with the underlying IO subsystem and so there are some tools in the thread in the forum that I've created to help you test that ZFS also supports creating data sets as I said before and they will show up as raw block devices on the host rather than files and so we can create a raw volume and the little guide in the forum will walk you through that so that you can take your VMDK and apply it and if you get it wrong you can run some tests on it and it's like oh I got this wrong the alignment here is not correct and you can use qmu IMG to convert from VMDK to raw and then you're converting it directly to a ZFS volume that you created before that hopefully has the same parameters and uh you know this is something you should keep in mind for your mental model how when you're virtualizing how the virtualization happens but close your eyes see how the VM is going to do the right it's got a 16k file system page when that reaches through it the file system in the VM to the underlying storage system we don't want that to be significantly amplified it's pretty easy to use basic tools like crystal dismark in Windows and just run the basic tests You you can also create a new ZFS data set if you think you got it wrong with new parameters. And then when you copy your storage or migrate your storage with the VM turned off from the old data set to the new data set, that act of rewriting the data will create the new data set with the new parameters. And so you change the geometry of how it's stored and you want to do extra testing and extra performance analysis. That's how you do it. Now I will also mention database servers. If you're running an an SQL workload and you want to understand how to optimize your your I/O, look at the page size your database uses and set your ZFS data set up to do that. I've gone over this at a very high level, but there's even more details and there's details in other videos that I've done in the past, but this is how you come to terms with ZFS. Another killer feature unlocked by ZFS is ZFS replication. And replication is awesome in terms of recovery. Is very worth it. If you have questions about that, let me know. uh because we can, you know, do future content based on that. What I want to do here is try to help you understand, not just walk you through the actual steps. And so with ZFS snapshots, you can set up Proxmox to do ZFS replication every 15 minutes. And so your VM, a 15-minute old version of it can exist on another node in your Proxmox cluster, which is pretty awesome. Now, what about things outside of ZFS? Well, there's also LVM, Linux volume manager. You can skip ZFS and use LVM. LVM works with a traditional hardware RAID controller. So if you're rocking a SAS RAID controller or a SATA RAID controller, this might make more sense for you. Software RAID in LVM, if you're not running a hardware RAID controller, it's built on the underlying Linux MD admin codebase, but it doesn't use the Linux MD admin utilities, but it's also very fast and very tried and true as a codebase. Oh, Intel V-Rock will get a special mention here because Intel and VMware at the last minute there partnered and you could run V-Rock arrays on VMware which is the fastest NVME RAID option for VMware that you can get. With Proxmox, you can have a largely identical experience with VRock on Linux plus LVM or you can run just LVM. So performance is very similar. Proxmox does not force a choice here, but it also expects you to be in the know about what you're doing. Oh, I should also mention Seph. Seph has native support in Proxmox. and it can offer an experience more like hypercon convergence with VSAN on VMware, but I really wouldn't recommend Seph for anything less than a virtualization cluster of six or seven nodes. Seph should be on your radar if you haven't heard of it. With a three node cluster, you know, each cluster would have a full copy of the data and you do get replication that's real time, but it's getting a little beyond what I can cover in this video. Suffice it to say, for that use case, I think ZFS is the more appropriate choice for most users. and most use cases. But if you've got a bigger cluster or you know special needs Proxmox plus Seth, we should engage in the community. But all this storage talk and the flexibility about the storage brings me to point number two. Step back from storage. All the storage awesomeness and all the options and nuance. It's not just storage in Proxmox. You're dealing with Linux. Proxmox is not the product exactly like VMware is. I mean under the hood, Proxmox is using Linux by design. And it doesn't try to cover every single use case. There are helpers in Proxmox to make dealing with storage and replication and ZFS and LVM and ice sky and point and click and all that as cool and as easy as possible, but they are fundamentally not technologies owned by Proxmox. Detractors will say that VMware is more mature than Proxmox for this reason. But those people I'm going to mock because of the backup situation in VMware in just a moment. Don't worry. How silly is it that aspects of this are basically the not invented here problem that Microsoft has. It's like, "Oh, Microsoft didn't make this. Let's try to actually make something useful." And then you end up with crazy things like WinFS taking 10 years and storage spaces basically being an unusable dumpster fire until what, like the last year or two. I mean, with Linux here and Linux under the hood, we got all this flexibility. And the flexibility can be a little bit of a trap, too. So, you know, if you want to customize the system and run some shell scripts, that's mostly okay. But sometimes you don't want to do things. And how do you know? And that comes with experience. A common pitfall for something you shouldn't do. It's like Docker. It's like, "Oh, let's put Docker on this. How do I run Docker on my Proxmox host?" It's like, "Oh, you don't want to run Docker on your Proxmox host." Most of the reason why is actually not to do with Docker or anything like that. It is the networking, the containerization networking system in Docker is not really fully compatible with the assumptions that Proxmox makes about networking. And so, I've been really deep down that rabbit hole. And I think the fix for that is actually best served with a set of patches against Docker and not Proxmox [laughter] which becomes kind of a political issue. Uh you're best off running Docker in a virtual machine. That works fine. One of the downsides is that uh I want to pass through the file system on the host, not a raw ZFS device to the VM. And that is now possible thanks to the Vert IO FS which is part of the underlying uh Linux file system virtualization stack. But that's a relatively recent development. There's also kind of a side conversation here with Proxmox because Proxmox supports containers as well as VMs. VMware did some stuff with Kubernetes and Kubernetes orchestration. But the only experience that I have at scale with VMware and Kubernetes ended up being Kubernetes on bare metal and using VMware for orchestration of the Kubernetes cluster. And that was really to do an end run around the licensing costs because the sockets that were on bare metal Kubernetes clusters didn't count as VMware sockets which is something that VMware did not expect. [snorts] But for LXC in the Proxmox context, it's just awesome. It's very lightweight virtualization and very robust. Lexe can make a lot more sense for access to hardware accelerators. Think like video encode and decode FPGAAS. uh maybe even Lexe containers that leverage CUDA resources because Lexe is awesome without really having a ton of virtualization overhead. There's not really a VMware equivalent. And that brings me to backups. Not really a VMware equivalent. So, backup or restore. How's that look? Enter Proxmox backup server. This runs best on bare metal, but you can run it as a VM. I've covered it more in depth in other videos on this channel. And it's basically [snorts] an appliance. You get a license for it when you got Proxmox enterprise server. You got your Proxmox enterprise license. But if you look at it earnestly, it blows my mind how good it is. The architects take backup and restore very very seriously, far more seriously than VMware and Microsoft has, at least Microsoft since like 2003 or 2004. I'm still really salty because early free ESXi had usable snapshot and backup capabilities. VMware deliberately removed them to upsell licenses years before Broadcom and you could enable SSH and kind of get some of the functionality back, but it was a really dumb experience and it was really dumb of VMR to do that. VMR philosophy on backup seemed to be that they were going to focus on HA replication and snapshots and that's what you should use. And because of that, products like VH evolved into the, you know, juggernaut that they are from a relatively cottage industry. Windows did the same thing. uh you know their attitude seem to be ah we make the OS backup is your problem. Windows backup like Windows had a reasonable built-in backup system that was scriptable but it's not being reliable around 2007 and even before then it was a little shaky. Think about that VMware and Windows both have a far far inferior backup and restore experience than you get with Proxmox backup server. It's very nearly as good as the oh and by the way the actually added support for Proxmox recently. So if you're already using veh you can just drop right in. PBS has incremental backups, ddup verified restore, no licensing games. It can also self-replicate, meaning that you have no excuse not to do a 321 backup, and the authentication system is hardened against ransom while style compromises of the box. So, this is really nice to see. This is also one of the clearest examples of the Proxmox project being run by sane people. That brings me to number four, Proxmox data center manager. This is the new entry. This is not vssenter, not yet, but you should be using it. It's relatively new. It's not exactly a single pane of glass for, you know, administration and those kinds of tasks that you may be thinking of in a VMware context or but it does well, it does let you manage multiple clusters and VMs across clusters and you can do migration of VMs across clusters. So, you can get it's not it's still there's one aspect that I wish Proxmach had more of and that's health monitoring. Don't get me wrong, Proxmox has built-in logging and graphs and and something, you know, it really should be more like Net Data. Net data is awesome and it's an add-on, but it has nothing to do with VMware, but you can install it because again, Linux, remember the whole Linux aspect. You can just install Net Data. It doesn't really have any negative side effects that I know of, and I don't think the Proxmox team doesn't recommend it. I think it's probably okay, but you do want to make sure that it's not listening on a public interface on the internet. um you know you want to firewall that off unlike VMware where you can configure some pretty slick automatic migration and load management stuff on the hosts that's left up to you. You can kind of see the load and you can see the load over time from Proxmox data center manager but we need more and most of the logging is actually on the node not in Proxmox data center manager. With net data you have more than you could possibly want. It's bananas. So you kind of you know should probably use net data. There's brings me to number five wanting more SDN and networking in general softwaredefined networking. Okay, so SDN is the hotness. But before I get to that basic proxmox networking cororosync is how the cluster manages sync and the heartbeat. Your core sync network doesn't need to be fast. It should be at least 1 Gbit, but you know bonded 1 Gbit would be better because it does need to be reliable. You should have a separate network for VM toVM communication and for high bandwidth activities like migration. That should not be the same network as your Cororosync network. You do not want to use the same network because a migration could interfere with the synchronization messages from Cororosync. There's not really good quality of service there. Uh you remember the old VMware clusters that had like four or six network interfaces? You can do that. Uh I think it's fine running two 25 gig interfaces and two 1 gig interfaces or two 10 gig interfaces and two 1 gig interfaces with chorusync basically being on the 1 gig network and you can do the VLANs on the dual 25 or dual 10 gig interface. Understand that chorus is leveraged heavily by Proxmox but also not really a Proxmox technology. Now Proxmox also offers softwaredefined networking that's open v switch and upgrades to what Linux does for bridging and bonding. And so Proxmox SDN is getting mature, but it's a relatively new feature and it's not something to sleep on. Like it's actually really good. SDN, software defined networking, zones, v-nets, options, IPAM, VNET, firewall, and fabrics. You can add every like it's the open v switch stuff. It's all here, but it's a little beyond what we can get into in this video, but it's there. And you can also set up fences and stuff for high availability. It also works well with SDN, but getting ahead of myself. So like if I could go off script here for a second and talk about the networking in VMware uh you know real truth Broncom they make silicon they make you know disc controllers they make why did they buy VMware and it has to do with the networking like if you look at the really insane stuff that Broadcom has done with their topof rack switches and the insane switching fabric stuff like terabits multi-terabit solution it is incredible hardware engineering and so from Broadcom's perspective the programmers at VMware came along and they basically put a not super sophisticated front end in front of the amazing silicon that Broadcom did. And VMware was able to charge more for uh some lipstick basically uh for their network interface than the Broadcom people were able to charge for their hard one silicon engineering. and they that didn't sit right because it's like look at this you know stuff that we're doing and in VMware you could configure your network your fabric your stuff in your cluster and VMware was smart enough to the drivers and everything else were smart enough to make changes to like your top of rack switch and you could do things like um offloading your uh TLS processing and you could offload that to the network card or you could offload that to your top of rack switch reduce jitter there's all sorts of cool like automatic VLAN configuration and the the line between hardware and software was really blurred. But most of the heavy lifting to actually accomplish that was in silicon and the hardware and VMware just sort of put a a you know a candy apple front end on it. And that really pissed off Broadcom and that's how we got into the thing where Broadcom's like, "Okay, if you're willing to pay this for the software part of it, we want these prices so that we cover the hardware part of it as well cuz we, you know, from Broadcom's perspective perspective, they killed themselves from an engineering standpoint." um to do this amazing silicon and VMware is getting all the glory when you know Broadcom was really far out ahead of other competitors and so that's why VMware costs as much as it does now IMHO and the SDN stuff in Proxmox doesn't do any of the hardware aspect of this but it does do the open v switch functionality stuff and maybe someday with open compute and we have some companies that are doing some stuff some creative stuff maybe they'll come in there uh ubiquity Microick, hello. Like, this is a vertical opportunity. Anybody got a few billion dollars? I got a great business idea. All we got to do is buy Microick and then actually roll this out and then like sky's the limit on the hardware software automation here. Like there the the silicon exists. There's just not the the candy apple front end for it. Anywho, before I get too far off track, the Proxmic stuff doesn't do the same type of top of rack off automatic offload, but it's very smart and open V switch and there's a lot of cool stuff going on in Linux. And speaking of cool stuff going on Linux, number six, uh there's a lot of other knobs and tunables. KSM kernel same page merging. Uh this one I just wanted to be able to talk about because it's interesting. The memory aspects of Proxbox. KSM doesn't activate by def default until you hit 80% of your memory utilization. The memory manager in Linux goes through and actually merges the memory pages that are the same. So there's only one reference to them. It's a ddup in memory. But you can actually pick when that happens. 50% of your memory utilization, 90%, 95% turn it off. It's a Linux thing. Turning it off can be beneficial in some workloads. Mostly no. There's also huge pages where you can allocate memory, you know, a gigabyte at a time. And for large VMs or VMs that are doing lots of huge memory IO's that can speed things up because it's less overhead translating from the VM to the host. This is really just sprinkled in here for the aspiring smart system administrator. This is something to look into for the workload that you're running and maybe put some tools together to test it. There are several other knobs and tunables like this which can help with some workloads, but the effect these knobs have is usually not as dramatic as uh completely messing up your storage config. So, this is just another opportunity for me to it's like fix your storage. Yeah. Uh and finally, number seven, our community. Big tip. community is a valuable resource especially when you're a refugee. You've got refugee status. Hello and welcome. Uh this is you know there's a thread that goes with this video on the forum and it has many more lessons learned from users like you and even insights from other big names in this space like Tom Lawrence. Tom Lawrence has been been there done that for a lot of this kind of stuff too. So you should check out his channel and as he's got lots of Proxmox content as well. Tom and I often trade stories about things that our users have stumbled over and so look there and you'll see even more context and content and nuance for lessons learned with Proxmox, especially for people that may be in refugee status. I would also hope that having made it through the video this far, you realize that both VMware and Windows HyperV have similar sorts of landmines that you probably have tripped over as an administrator in those contexts. You might have tripped over them without even realizing them, but hello and welcome because I'm telling you where the stuff is. It's nice to meet you. How's it going? That's about enough rambling for this one. If you have run into something else and you would like to share, then you should at the forum or below so that other people can learn from what you've learned. or if you're, you know, chasing a particular gremlin, sometimes the answer is more interesting than it would seem at a surface level. That's also really fun because that helps shape future content that we do. I'm level one. If you are excited by any of that, hit me up in the forum. Let's chat. And uh looking forward to the next video in this series. Woohoo. Check out the other videos that are related to this one because Prox fun times. All right, I'm signing out [music] and I'll see you in the forum. >> [music]

The times are changing, so don't get left behind! Check out the forum post here: https://forum.level1techs.com/t/proxmox-everything-you-wish-youd-known-sooner-especially-for-vmware-refugees/245088 OpenMetal.io forum post: https://forum.level1techs.com/t/proxmox-full-production-in-the-cloud-with-openmetal-io-dedicated-servers/244061 0:00 Intro 1:34 Storage 10:32 Flexibility With Linux 13:36 Backups 15:36 Data Center Manager 17:02 SDN & Networking 18:44 Tangent 21:36 Knobs and Tunables 22:45 Community and Conclusion You can find us... Twitter - https://twitter.com/level1techs Twitch - https://twitch.tv/teampgp Patreon - https://www.patreon.com/level1 For all our social links, websites, and more, check out our link tree! https://linktr.ee/level1techs Thank you for watching! ------------------------------------------------------------------------------------------------------------- *IMPORTANT* Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.