RUSTFS - New AWS S3 Compatible Object Storage (from Minio)

Okay, so a cool review today of the Rust FS. So Rust FS is an S3 compatible blob storage which basically for me at least replaces Minio. So I was using MIO to mock S3 for my local developments and even my cloud development when I didn't want to set up S3 for production still use S3. But the problem is Mino won close source which makes sense. We wish them the best. Now we have RFS which is a new kid in the block and is actually amazing. So it's as simple as using minio. you have a docker things over there and you can just use a docker. So in my case I don't need to compile it and everything because I just want to use it as a mock of S3 because in fact my architecture when I do production SAS application is this kind of architecture which is basically I have my cloud platform my Kubernetes and then I have all of my pods. I use Valkyia as a message Q and then I have some management pods like the database management pod and so on and then the workers pods and then the web services that have the JSON RPC or rest API. Now when I am in the local development I'm going to use a dev DB which is posgress in a pod and when I'm doing my development as well I like to have a dev s3 which runs in a pod. So that was minio and now is rests which is amazing. And then when I go to production from that then this is where I use a pro blob storage and here I'm going to use S3 and then the pro DB that I'm going to use RDS. I could use RF fs perhaps in some cases but I like to use first the S3 and then if I need more I go more complex. But right now the rest FS is amazing for that. I'm going to show how it works. So if we go back over there we could take this one here that will work pretty well. But in fact, I have a little readmi. So I like to use podman right now and then I do a run make sure I uh remove everything and then I like to have it interactive such as I put it as a terminal and then when I stop it everything ends. So now if we go to my terminal I'm using alacrity. So this is actually zed and that is alacrity with t-max. So I'm a big fan of that. I've done quite a bit of JC task on zed and so on which is pretty cool. So I have a whole nice workflow. So now what we're going to do is we're going to take that guy and then we're going to go here and we're going to run it. So that will pull everything. I already pulled everything and that will start our server successfully. So now that we have that, we could go here to this URL and that will give us this web UI. So obviously the web UI is completely optional because everything run on the server but it gives us a web UI at 9,01 and then the password is user and password is this one. So now I'm going to do the login and then there's a lot of very cool features but in fact the only thing that we're going to use for example is this one where we're going to create I think a demo 01. I like to pad it like this and then we're going to create it like that. So this way now we can do that and we could upload files and stuff. So that is going to be our web UI. So now that we have that we are going to go to the run Xpaws3. I'm going to push it here and put the link below in the video. And then I like to have these two kind of things and then later I might have two but that is my practice. So I have the lib right now is empty in the cargo. We're making sure that unsafe forbid and then today I'm I'm allowing myself at the beginning to use unused and then I toggle on when like that we're going to be a sync. We're going to use these two guys and then the d of more is part of my best practices. So now I always use that. Now we're probably not going to use that too much here but then eventually we might. So now we can go back to my list over there. So the way that I like to start my examples is I like to use my box in over there. I done a video why I'm using that for development and test only, not for production code obviously. And then I have my okay over there and I do my print like this. This is a type that we're going to need. We're going to see that later. And now what I do is I go back there. So again is with Tmax and I'm going to do a CWE. So that is a little Kako watch. to install cargo watch. And if I do a functions CWE cuz it's been long time I haven't shown that. You can see it there. That's my function. So now I can do CWE C01 list. And so that should give me hello S3 experiment. So far so good. And so then after what I do is I go back there and the first thing that we're going to do is we are going to set up our credential. So that you can do a load from environment something like that. The thing is usually when I do production coding the environment keys for my S3 or whatever or whatever I want they are not the standard ones and I don't want them to be the standard ones. I want them to be prefix with my service underscore and the intent of the S3 because I might have different S3s and so typically what I do is I always get that from the environment and I have you know layers to do that and then I set the credential myself. I don't want the libs to be over smart over there. So that's why I've done this credential over there and I was making sure that I was learning the way here that I can do my own loading of these values. That is one thing that I'm doing. I will use that in production anyway except that it will be from environment obviously. And then we are going to build our configuration. So that is the way you do it now with AWS config. I mean from the world now and we're going to have our credential providers which is this guy. We are going to have US East one. So that is um to respect the spec I guess if you don't give a region even if region doesn't make sense here it will fail so you have to give one it's fine on this one the endpoint I've noticed that doing the 127001 works better than local host so I will recommend to use that and then obviously within the kubernetes but you will do give the host of the kubernetes that's the whole point yeah but that will come from your config map then we load and then we wait so everything is a sync so if we toggle the type here we have that credential tab looking good and then We create the client. So far so good. So now I could press a save here and that will run again. So we see that we didn't crash. That's good. And then we are going to exec the list bucket. And so that is a way here with a client that we do that we do a list bucket. We send it and then we wait for the response and we get the list bucket output that is API. And then we are going to count it which is basically we're going to take all of the buckets and then we're just going to have a count right now. And so because in our UI we created one guy over there. So press save. Go back to the terminal over there. And then we should have one. So so far so good. So that is a way that I like to code here because every time I press save and usually my terminal is lower. So I see everything at once but I can see the my progress live. And then the next step here that we're going to do is for each book we want to display the name. So far so good. Press save. Go back there. Then boom we see it. That means that now we use the official AWS library and it works like a charm. So that is great. So usually for my production development when I recreate a database and I recreate my blob storage, I will actually create that dynamically in my code such as I have a fresh environment every time I I restart my development. So that is awesome. Now we're going to stop that and we're going to run this guy here which is going to see how we're going to create some document and upload it. Let's click on that. So now we have this and so very similar over there and we're going to actually go there and cw2 create. And by the way after I'm going to show you how I create those files. The reason why I'm not writing by hand is because now I'm working a little bit differently. So I'm showing the final code and then we'll go how I'm learning new new technology these days. So we have this and this is very good. Again follow my best practices. And so now we're going to go back there. We are going to make our fixtures in a way. So that's not really a test, but we can do that. We have our bucket 01. We have our file01 txt and then hello 01, which will be the content. And then we're going to set up our client similar to the other one. And here I put kind of a little note. If you do a local host when we create files is kind of a little bit weirdo. I don't know really what the issue is, but the 127.0.1 it works very well. And then we're going to create our client. Same thing as before. So this one is not new. So on this guy, we create the bucket if it's not exist. So we're going to list the bucket. We try to find it. And we'll probably have some convenient APIs obviously for these kind of things. But that is a logic and then we're going to say if the bucket doesn't exist, we're going to create the bucket obviously always with question mark over there. And this way we don't have the unwrap. So that is something that I've said for years. I think even when you do example code do not put on unwrap or expect because if there's a copy paste and it goes the wrong place it's kind of a nightmare. So the question mark is good because if you copy paste into your production code you will get a compile error and so you will have to handle this one to make sure that you have the right phone. If you put an unwrap you don't have the compile error and then good luck. So that the reason why I always do that and I don't use spec and I don't use unwrap. So then I do the print here just to save the thing and then I'm going to create the file here. So here I'm saying creating the f this guy. So actually I'm going to press save over there. And if I go there we can see creating bucket. So we're creating the bucket 01 because we didn't have the bucket 01. I think the one that I had before was demo. Yeah. If we go back there was a demo. It was not bucket 01. So if I reload here I should have the two bucket which is perfect. That works beautifully. And then we're going to create a client here. So we're creating the file. That is the way it works. We put the object. So that will return kind of an object type. I don't know if uh yeah cool we put the object here and then we have this builder business. We build build and then we send over there. That is a future get their weight and we don't really care about the result at this point. And then we're going to say print successfully created the file key and the bucket name. Yeah. So we could have probably put something over there. We're going to see that later. We press save over there. We're going to remove that. And then we should have successfully created file one. And then we go to the web UI. Reload. I didn't need to reload but file new one click here we get that and we get a nice little preview over there that's it so that is pretty cool in fact one things we can do is we can say hey what does this guy return I like raise sometime and then I'm going to do um so I have a little snippets over there 10x debug print len so you can find my snippets in my JZ JCZ stuff unfortunately the extension in z doesn't support you doesn't support snippets yet. So hopefully it will come one day. But that's my rest. I put that this way. I search for those guys and I deleted them before my commit. And then if I go back there, we get the expression and all these kind of things which kind of makes sense in a way. Okay, that is pretty cool. So Rust FS is a gem for what I'm concerned. Big thanks to the team. It's awesome. I hope that their business is going to work very well. So now I want to show you how I'm learning these days. So first I'm going to do a g restore because I don't know what I've changed over there. So everything is good and you will have this branch under main into the rust aws3. Now let's clean this one and now we're going to go to g s which is switch main. So that will go back to main which is basically back to zero. So I'm going to close all these files. So going back there and here's the way I work. So I develop a pretty cool tool and you can do the same thing with cloud code or cursor and everything. feel free to add comment of how you work with uh cloud code and so so on but with AI pack it's a runtime you can install it on Linux windows and everything and it's an agentic runtime where you can build your own pack and on top of it I have a procoder the difference a little bit of procoder compared to cloud code and cursor and it's not better I'm not pushing my tools use the tool you want it doesn't really matter but the procoder is kind of a file base kind of things it's kind of control first magic second rather than magic first control as kind of things. Okay, so you install these kind of things. It's just a command line all done in Rust obviously and then once you have it, you can do your API check keys to make sure that you have the keys in the environment. Here I don't have the keys in the environment. I have a little local script that take it from the keychain. So I'm going to do that. And now if I do that again, all the keys are here. So just in the environment I'm not displaying them obviously. So now that we have that, you can do a AIP install pro at coder. So that is a procoder AI pack. Once you have the procoder AI pack, you can do AIP run procoder. So that will run the AI pack procoder. And again for me and I understand people that work with cloud code and cursor, it's awesome. I'm not pushing people to use my tool. I'm just did the tool because the way I work for me is I always work filebased. And so the trick here is I want everything to be kind of in a file and to have my own workflows the way I want. So here I can do raph, I can do anything I want. So if I go to the coder prompt. So that is a terminal UI done in rust and now I have these kind of things and it's what I call a parametric prompt. So this is my prompt. This is my parameters and then I'm going to put the prompt on top and then below that is going to be the response of the eye and the terminal UI will allow me to see all the runs, the cost and everything. Believe me, it's not for vibe coder because you will give that vibe coder he or she will have a heart attack. So the cool thing about this is you can install a pack also which is pro rust 10x and that is basically all of my best practices are there now and they are just markdown files. So you just put a globe like that and then everything is good that the knowledge globes basically root that is typically the way you want to go and then the context here that is where we're going to put the things we're going to do a cargo and we're going to give the readme even if we don't have a readme for now and that is the content of the file that I'm going to send to the AI. So again with the IP pack and procoder you have the exact control of what you send. The AI can ask you more but you have to copy paste back there and that is what you are going to send to the AI but just the file path and that is very useful because it will give a good context of the AIS of what the files you have. So for example here I could do actually a little something like um CML. So you have to put it like that a star. Yeah. Not to go down recursively but to have all of the the root files. Yeah, that doesn't hurt. And that is advanced staff. We are going to do true on the right mode. This is a new experimental things. It will become the default. And that is a flash. So flash three that is an alias for flash 3 is amazing. Yeah. But you can use opus for opus 4.5 and so on. So now what we're going to do and that is the way I work now because it's actually too powerful. So I have flash. I know exactly what I'm going to send. And right now this is justip pack. If I go to the g ignore, you will see that the AI pack and the pack because I can pack my agent if I want. They are ignore from the g by default. So once I'm here, I'm going to say looking at the rust 10x practices create a xps3 projects with a simple c1 list rs that just say hello. And so now this is usually what I do even at the beginning. I don't ask it to do too much. And then and then I come here and I press R or I click here. Same thing. In fact, with my JC task Z task, I can do a command R and that goes directly to the right T-x of the right equity and so on. So, it's pretty cool. So, boom. And now we're going to see here it created all these files, but it created my way. This is my thing. And so, because I put the best practices and you can find them, they on open source on the packs pro and so on. And so, you can take the restx and make your own. But now it's coding my way. It do the derive form and exactly that and my lips. And then I go back here and then I have these things. So, so far so good. And so now that I have that, I'm going going to go back here to the terminal. And so there's different way of doing it. I have something like that on my T-max that split all of these kind of things. So this one was 1 cents. And then I can do here a cargo build. On this one, a cargo build is useful because AWS library is actually pretty big. So doing a cargo build at the beginning helps a lot. And then we're going to do our C.WE. Okay. So that is good. And now we're going to do CWA C01 list and we should see hello world. So obviously it didn't really create the same thing that we had before because now I'm doing another prompt. But that is actually exactly what I wanted. So usually what I'm going to do is actually the cool thing here. This is a little terminal that is just more kind of a status kind of things. Everything I see here is mostly in my color prompt as well. Okay. just that when I say see that I put it in the terminal such as I can click on it to copy it and I go back here command V and boom I have it and now it's git the best practice here is every time you ask AI to do something you start from a clean commit yeah you don't want to rely on tools saying approve approve we have g for that we don't need to reinvent so we have this kind of things which is great and now I'm going to say on this guy I'm going to go back up that is where my prompt is I'm going to say in the C01 list add a way to list all of the buckets names and and print names clean code and use the uh coded credential rust fm uh rust fs admin I think and rest fs admin that is the two guys and now I do a command r because with my uh jc task and everything there it goes and find the right t-mox and send the r but you come here and you can press r and that will go and will create the thing so typically now what I do a lot is these kind of things. Now in the coder prompt here is actually pretty cool. I have some files everything is file based. Yeah. So if you put that in your knowledge and everything you can have a planbased development. If you put your dev chat here inside uh your context you can actually chat and the things will be added in the files. So again everything is a file not in a terminal kind of things or not in the chat somewhere. Everything is file based. Yeah. From the question to the response. So that is the way I like but again please add comment below if people do better things with cloud code and cursor it's awesome use the tools that work well for you. So now it does this and it created our list over there and we're getting pretty close of what we had before. Now the nice things about AI is because I'm going to and actually I forgot about something here. It didn't see the folder by the way. Yeah. So but it's fine. Yes. Because it didn't need to see it. Examples and we're going to do something like that to make sure. And I like to have struck. So like this the struct can see everything and then it can tells me hey you are missing these files. So that is a nice so that is with content that is without content. Now that we have the things here we can ask to fix it or whatever. Now I go to list and so the goal here is not that coding is not to code blind is to learn better and faster. But the most important is to learn better. Yeah. So now we do that and you will see that when I press save here the AI doesn't know how to format yet but now that it will have these files in the context when it's going to create the second file is going to be much better. The formatting is going to be much closer again AI has become amazing use it but don't use it to be lazy use it to be better. So we have the CR here which is what we do. We do the config. It's exactly what we had before the region over there that different a little bit code but that's good. And then he does a staff, it does the exec. So that kind of took my pattern for my test. I like this kind of things. So it it's a nice mix. And so again, this is is a mess is because I put rust 10x in uh in the context and that I get the the bucket. I get this found bucket very similar to what we had before. And then we had these guys actually have the same guy. This is actually this is a new prompt here that I'm doing. It's not the same one. So now let's see if it works. So we're going to go back there. and it's not working very well. So, we're going to do a control space D. So, that's my way to zoom here. I love T-max now. And we're going to see that there's an invalid access key, but I actually think that is my mistakes on the prompt, which is basically I didn't say the endpoint. Yeah. So, now I have two choice here because again, we're not v coding here. I could go back to the prompt and say, "Hey, can you change it?" Or I can change it manually. So, right now I will do something like this. I will say, hey, I forgot something. So, do they have a easy end and point URL? Okay, that's good. And then I'm going to do a HTTP and then I'm going to do uh 1027.01.01 and I think is a 9,000 API. Press save and then go back and then boom, we have our stuff. So again here that is not really vi coding. VIP coder will prompt to the end of day. Yeah, here we say hey config is over there. I mean I saw it before a little bit but even there I will test it and look at it or we'll print it and and save it. The important things is to make sure that you are committing before you do any other prompt. We're going to zoom out and I'm going to click here and paste here. Boom. Done. Well, actually we're going to do the same example. We're going to go back here in our UI. We are going to delete that guy. I think it's okay now. Then we're going to go to our browser here and we're going to delete that guy. You need to delete the content is the same thing as S3. So we have everything. Reload to make sure I only have demo one. And then I'm going to say list of bucket. And now that we have this and I make sure that I have everything committed here, which is the case. Now create now following the C01 list. Create a new C02 create file RS which create bucket01 if not exist and then create file file01.txt with content content01 regardless if it exist or not. I'm going to remove that and that's it. And then I'm going to do my command R which again will send the R to this guy. You can go back to your terminal and do R. So you can while it's running you can navigate your things and you can see the overview the task. You can have even multiple task. You see the prompt, the completion and stuff. And by the way, here I don't want to be too philosophic here, but prompt and the completion and the reasoning is the eye of the AI. So once you get used to that, you understand how the eye what is hard for it, what is not and all these kind of things. This is actually very good numbers to have all the time and you get this like spider sense over time. So we have this guy which is good and we're going to see that over there. It's kind of set up fixtures really like this which is fair enough. The credentials the same thing. The client that's good. The exec this bucket and stuff very similar to what we had before. That's perfect. Then it create uploading the files. That's good. And successfully. Oh, this one was the old one. So I'm go to here and I'm going to do a CWE2 create file. Boom. It successfully created and everything. Everything looks pretty good. So I really love T-max here because if I need to go there, I do like that. Otherwise I zoom out. I mean it's Stemox is very great and I can have different tabs over there. So now I'm actually extremely happy with that. So now we can check the UI. We're going to do a refresh. Boom. Bucket one. Go back there. This guy and that guy. That's pretty good. Now I go back here. I'm going to take this one. So I'm going to do that. I'm going to paste it here. Boom. We have this commit. And now for example I'm going to say hey you know perhaps you know all of these kind of things here all these create config and create and so on here where it would be nice to be in lib. So again if I were building the stuff I don't have time I would go away but now we save so much time that actually we can do that. So we're going to say now in the examples C RS um let's consolidate the uh create client into the uh lib RS so that all can reuse code. So I press save and then command R again. You can come here and press R and then we're going to see how that is going to work out. Okay. So now it's done. And now we can go back here and it changed the stuff with client. That is exactly the way I wanted to do new client. That's great. And then that's the error which is my error pattern of Rust 10x. So it kind of follows a pattern again. And again use your own pattern. Yes. And then this is my lib. And this is my little winny things here. Sometimes you kind of get the stuff wrong. It's no big deal. And then that's it. This is beautiful. And then if I go back here and CWE list. Boom. That's it. And I will do that. And then I'm going to push that into the URL you see below. And until next one, happy cutting.

RustFS is an S3-compatible object store for cloud applications, and it can be great for mocking an S3 server in local or remote dev environments. Here is a quick video on how to set it up locally, and how well it works with the standard AWS API. In our case, in production, we typically use standard AWS S3 as our blob storage. However, in development, we love having a robust, S3-compatible server that we can run in our local or remote dev pods. It really speeds up and simplifies the development cycle while minimizing differences from production. The last part of the video will be about how I embedded AI into my xp-... workflow (and how it is also embedded into my production coding). The trick with AI for production coding is Control first, magic second, and not magic first, and control last. Big thanks to the RustFS team for picking up the torch from MinIO as the most robust open-source, S3-compatible object store. Big thanks to MinIO as well for giving us robust, S3-compatible storage all those years. === Links ➜ Discord general-rust - https://discord.gg/W2besKCzjx ➜ Twitter - https://twitter.com/jeremychone Pro Coding Rust with AI ➜ Discord genai & aipack - https://discord.gg/6KkpQBKGFS ➜ AIPACK - https://aipack.ai/ - Built in Rust, 25MB CLI to run, build, share custom agents ➜ AIPACK Blog - https://news.aipack.ai/archive - Posts and videos about AIPACK ➜ pro@coder demo - https://news.aipack.ai/p/production-coding-example-with-procoder - Production coding with AI == JC Playlists: - Everything Rust Programming - Tutorials, Courses, Tips, Examples: https://youtube.com/playlist?list=PL7r-PXl6ZPcCIOFaL7nVHXZvBmHNhrh_Q - AI Production Coding - https://www.youtube.com/playlist?list=PL7r-PXl6ZPcB2zN0XHsYIDaD5yW8I40AE - AIPACK Playlist - https://www.youtube.com/playlist?list=PL7r-PXl6ZPcDBodiiTdUeCmUwsYFyDzGt - Rust AI Programming Series: https://www.youtube.com/playlist?list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj - Rust Web App Production Coding Series: https://www.youtube.com/playlist?list=PL7r-PXl6ZPcCTTxjmsb9bFZB9i01fAtI7 - Rust For Desktop App: https://youtube.com/playlist?list=PL7r-PXl6ZPcAQAi1lfVrtHCSbaJbwZ4O8 - Rust Programming for Web Development: https://youtube.com/playlist?list=PL7r-PXl6ZPcCLvwpdD2Vj1O4CyoFTiHKd - Rust for Java Developers: https://youtube.com/playlist?list=PL7r-PXl6ZPcD63DS2djSiz4SlXkaTfobc Rust10x - Coding Resources for Production Coding in Rust. ➜ https://rust10x.com Other notes: - Davinci Resolve and Fusion video editing. - ScreenBrush for the green lines. (Gromit seems to be the equivalent on Linux) - Sketchapp for some graphics. - VSCode with Google Material icon themes (with some customization)