The ins and outs of product-testing a System76 computer

hi I'm Levi the quality assurance manager here at system 76 um I kind of wanted to just take a little bit of time to walk through what the uh initial QA process looks like for a brand new [Music] product when we first get them the very first thing that we do is pretty much what a lot of people do when they buy a piece of hardware and want to see if it runs Linux we just try to install pop or a buntu and see what breaks see what works uh write all of it down uh share with the engineers if something's not working quite right and uh if there's any weird little bugs or anything we uh write down the steps to reproduce them and share that with them um and then after that uh if it's an open firmware model which most of our Intel stuff has been lately it's really cool that we've been able to do that um the uh engineering team usually takes the model and starts developing firmware and embedded controller firmware we call that EC um the EC firmware is kind of the uh uh the control center you could say for the laptop it controls things like power uh the fans the keyboard backlight um the keyboard itself uh touchpad things like that um then there's also the BIOS firmware or I guess just the proper firmware um that's more uh talking to the CPU and making sure that the CPU knows what to do um and there's a lot of interplay between the EC firmware and the regular bios um and that is probably a little bit above my pay grade uh I just try to make sure it all works together nicely the engineers are the ones that really uh have the awesome knowledge of that and get really deep down into it I know most of our Engineers like to move into the machine that they're developing the firmware on um it's kind of a nice touch to actually make the the firmware from the machine that you're making the firmware for um they do a lot of uh flashing the machines externally um for that we use uh a bunch of different tools the main one would be the ch341a which is this little guy I have right here it's just a little bios flasher um although it is kind of slow so alternatively we also use a Raspberry Pi um and then we use these little clip things to interface with the uh bios chip itself we just clip directly onto the integrated circuit uh this is the old style for the s soic8 format chip and this is the new one for the Ws N8 chip these are little Pogo pins um this one's much harder to do because when you're actually flashing it you have to hold perfectly still for about 10 minutes while it's writing the BIOS file to the chip uh and if you move even just a little bit you break the contact and you have to start over because you have an incomplete flash um although the the Raspberry Pi is a lot faster so using that instead of this is definitely preferable um when we're flashing the embedded controller firmware um that actually happens through the keyboard connector so you remove the keyboard uh take out the FPC cable that connects the keyboard and we use this uh Arduino Mega with this little FPC breakout board and we connect this into the keyboard connector slot and that's able to uh program the embedded controller firmware right there through that connector when the engineering team has a pretty good uh first minimum viable product for the uh firmware and the EC firmware um that's when QA starts uh steps in uh we flash our own units uh usually the engineers have their own unit and we have a couple of units here in the engineering lab or the QA lab um so that's when we'll do the first flash which requires those external tools we'll take the back off we'll access the motherboard uh clip right onto the BIOS chip with those tools uh and do the very first flash after that um subsequent flashes it's much easier because you can do that just it can internally flash itself that's how we do firmware updates over the air um it's just that first one that we have to do with the the BIOS clip um and then once we are uh doing the the first round of quality assurance tests on the laptops we're kind of just making note of what works what doesn't work uh we have these uh checklists that only get longer and longer every single time we find something new that needs to be tested we add it to the list um things like uh do the hot keys on the keyboard work the volume up down touchpad lock uh keyboard brightness um there's a whole bunch of power behaviors in this too U making sure with USBC connected you can do things like a USBC charger connected you can uh you know stress the machine and the uh EC allows it to continue to charge while uh it's connected to to USBC and stressing um there's a lot of Hardware compatibility a whole bunch of charge level stuff so drain the laptop down to basically zero charge it back up all the way make sure it comes back on as expected once it has a little bit of charge in it and it runs and charges from 0% uh charge to 100% without interruptions uh there's a lot of Hardware compatibility things so we have a whole bunch of different types of nvmes a bunch of different types of uh SATA drives and uh RAM chips uh we basically we we mainly focus on the types of Hardware that we're going to actually sell the with of course cuz that's you know if you get a machine out of the box and doesn't work right with the drive that came with the machine that's pretty frustrating but we also do test a lot of other brands um because we know users are going to add drives they're going to add RAM uh so we test a whole bunch of stuff we want to make sure that that's going to be a painless process for the users another thing we pay a lot of attention to is power um so uh uclass machines like the Darter the uh gallago the Lemur the pangalan uh things that have you know typically integrated Graphics um and are usually below 100 Watts uh TDP um usually those are a little bit more complicated because they do support USBC charging that also means we have to make sure they work great with uh docking stations uh you know it charges over USBC it usually delivers uh a video signal out over USBC and then any USB devices that are connected to the dock those also have to work so that USBC Port is doing a lot of stuff right there we have to make sure all of that is working at once um if the laptop supports Thunderbolt that's even another thing that that USBC Port has to be uh tested with make sure that's behaving as expected um we have a we have a few monitors uh in the lab also that work the same way um you connect it with USBC it's got USB ports on it uh ethernet usually speakers um making sure all of that is working great while charging the machine is important um and then we also have to check the uh Barrel charger um obviously for the charger that comes with the laptop uh connecting the USBC charger and the barrel charger uh is something that the EC should be able to handle um if both of them try to charge at once it would probably I don't know what would happen but it probably be really bad but it's something we check just to make sure nothing bad happens uh uh making sure charges don't get overdrawn um making sure that the power limits that the laptop is supposed to have are um actually how the thing is behaving um you know you don't want to start compiling your your uh project and you get halfway through and then the CPU takes too much power from the from the charger and it shuts down the charger and sometimes also it'll overdraw the battery because the battery has a built-in circuit sort of like a circuit breaker you can only get so much out of a battery safely so there's safety mechanisms that turn the machine off when too much is drawn for the battery uh making sure we don't hit any of those limits with the embedded controller firmware is super important obviously you don't want your computer to turn off when you're compiling your code very frustrating one of the uh things we pay a lot of attention to uh especially for a machine like the gazelle here which is a little bit of a bigger machine draws a lot more power um performance is a big deal for something like this because it's got a discrete Nvidia graphics card you want to make sure that's actually putting out uh you know all of the performance that you're looking for people are going to play games on these things they're going to do machine learning stuff some GPU compute stuff um all of that needs to work right so one of the things we uh definitely test a lot of is uh benchmarking um making sure Under full power full draw um you know the battery the uh AC adapter is keeping up with that um as far as users that are going to use these things for gaming um one of the things we want to we always do to make sure that that's working correctly is sometimes we'll just fire up a game just play it for a little while uh one of the ones I like is uh jsx mankind divided uh it's a little bit older at this point but it's got a built-in benchmarking tool um which is super nice for me cuz I can just set that thing let it run for a few minutes while I'm working on a different machine or uh whatever else I might be working on um and it gives me empirical data it tells me what kind of frame rate it's putting out every time so I can see if that starts dipping then I probably need to look into you know is it thermal throttling uh what exactly is going on there other than that sometimes I just you know I'll take a machine home and just play some games with it uh I've been playing a lot of No Man's Sky lately that one's pretty fun uh trying to get into Elite dangerous that's got quite a learning curve but I'm figuring it out um but you know just actually using it the way it's going to be used by the people that want to play games is important um other than that other performance things are done usually through a series of fonics test Suite uh benchmarks uh that's just such a wonderful tool for Gathering all kinds of awesome data there's a lot of benchmarking Suites at openbenchmarking.org um that's usually where I go to look for a whole bunch of different ranges of tests that I run on these things um we usually run uh a lot of really CPU intensive benchmarks making sure that that's you know in line with expectations um there's some GPU ones that we run as well making sure that's all good then obviously you want to run them both at the same time sometimes making sure the thermal system is going to keep up when you're running the machine at full load like that another thing that becomes pretty important is the uh the uh acoustic performance of it we have uh over in the corner there we have a sound Cube uh it's painted up like a big Rubik's Cube it's pretty awesome um but it's like a you know like a drywall and uh Wood Construction with a a whole lot of insulation in the walls um there's another layer of uh sound deadening foam on the inside um cuz out here in the factory it gets pretty noisy um so getting a little place like that where we can just get away from that background noise and just listen to what the machine sounds like in a quiet environment is uh really useful for us um you know nobody wants a laptop that just whales like a harpy all the time sounds like a helicopter whatever you know all the all those things are very undesirable and we want to make sure we get rid of that so that is sort of a general overview of what the QA process looks like on a brand new product um obviously I was pretty vague about a lot of it because it's just a really high level overview of a uh pretty complicated process that sometimes takes weeks to months if you have any questions about it uh feel free to reach out to me my username in our uh popos uh mattermost chat is Levi Port lvi p o r t it's at chat. poop .org um and if you want to reach out with any questions I'd be happy to answer them uh and with that uh until next time take care

Levi, the Quality Assurance Lead at System76, gives us a general overview of the initial QA process for a brand-new product, from firmware flashing to hardware compatibility testing, acoustics, and more. System76 was born in Denver, CO with a passion for all things Linux and open source. Today, our band of merrymakers is on a mission to fundamentally define the future of Linux. We create powerful desktop computers and laptops driven by function & design and make discovery simple with Pop!_OS. Our work helps creators, makers, and builders engineer a better world—and have fun doing it. Chapters: 0:00 - Intro 0:15 - Installing and documenting 0:40 - Developing Firmware 1:55 - Flashing Firmware 3:20 - QA Testing 5:15 - Hardware Compatibility 5:47 - Ports and Power 7:23 - Power Limits 8:02 - Benchmarking 10:18 - Acoustics 11:10 - Outro Five facts about us: -Desktops and keyboards handcrafted in the US -Pop!_OS is free and collects zero user data -Laptops ship to over 60 countries -In-house support for the life of your hardware -Advocate for Right to Repair, Linux, and open source Check out what we make! Laptops: s76.co/WuEDOnoS Desktops: s76.co/Zn4NXTf9 Pop!_OS: s76.co/D_IWRvWD Share what you make with us! twitter.com/system76 facebook.com/system76 instagram.com/system76_com