This is my Endgame – Mother Vault Server Room Update

– Eeeeeeeeek! It’s all here! Look at this! It’s been almost a year since we discovered the major issue that effectively put our entire video archive server offline. That’s right! All the full quality footage, since I was still working at NCIX. (Linus sighs) Gone! Temporarily. After the painstaking process

Of recovering nearly all that data, love you Wendell, from Level1Techs, we finally have all the hardware in place, so today’s the day we get our vault back up and running. We’re talking, sprinting. We’re talking, better than before. We’re talking, an entire 3.6 petabyte array managed by 1 single Head Server,

Instead of the 5 servers that we have now. And I promise, it’s gonna be awesome like the segue to our sponsor. – iFixit! Looking to expand your selection of tools? iFixit, has you covered. Learn more about the Manta tool kit and it’s twelve, wait, a hundred and twelve steel bits. What?

With two premium drivers. Learn all about it at the end of this video. Manta. – This is the one computer that is going to run the Mother Vault, and the astute among you have probably already realized that we are not going to able to put 3.6 petabytes worth of storage inside it, unless it’s from Hogwarts. (magical instrumental music) – It’s magic. – Wingardium Leviosa in here.

– No, it’s Wingardium Leviosaaaa! – Oh, sorry. Excuse me. No, in fact, there are no hard drives in here. This is where the magic happens! The machine you saw just now, that’s only the brain. So instead of tryna put 3.6 petabytes of hard drives into there, they’re gonna go in separate enclosures

Like this one called JBODs, or Just a Bunch Of Disks. And you can think of an external JBOD enclosure kind of like an external hard drive dock, except instead of one hard drive, they hold up to a whopping 90 hard drives each. We’ll talk more about these a little bit later.

Just because the 2024US-TRT from SuperMicro is shirking it’s storage responsibilities, doesn’t mean that it’s not packin’. Inside here we find a dual socket AMD EPYC Milan system that honestly has a bit of an unusual configuration. Typically, a system with this much CPU horsepower, so those are two 24 core AMD EPYC 74F3s,

And the ability to run up to dual 64 Cores, would be decked out with a ton of NVMe slots, or some enterprise GPUs, or something like that. But this guy only has 4 NVMe slots, and these are an optional accessory. The rest of the storage up here

And in fact these 4 slots by default are, I kid you not, 6 gigabit SATA 3.5 inch slots, like that you would use for a hard drive. That’s because this system is actually designed with a different purpose in mind. Say for example, as a virtualization server, where you might need a ton CPU

That’s gonna be split up to different clients. Or if you’re like us, a ton of CPU that you’re gonna use to use to drive a boatload of connected storage that isn’t inside the chassis. So, instead of using up all those express lanes from those EPYC CPUs, which is, buddy,

A hundred and twenty eight lanes each with like NVMe drives in the front, we instead want them allocated to PCIe slots on the back of the server. Back there, we could install high-end networking adapters. What is this? Dual 100 gigabit? This guy right here? – Yep, Mellanox 6. – Or the other piece of our puzzle, host bus adapters. Just like with an external hard drive, our JBODs back there, which the keen-eyed among you will have noticed that there are 2, require some kind of connection to the actual server, with the actual CPU in it,

And the actual operating system running on it. In a modern JBOD, this is usually handled by Mini-SAS HD cables, like these ones from Infinite Cables. We’re gonna have these linked down below. So each of these ports on the back of our 3 cards can handle 4 Lanes of 12 gigabit SAS.

That’s the enterprise big brother of SATA, for a total of 48 gigabits per second per cable. Your average computer, and even server, probably doesn’t have any factory Mini-SAS HD ports, so we’re gonna be adding them to this machine with 3 of Broadcom’s 9500-16E Host Bus Adapters.

That means we’ll have a full dedicated card for each of our JBODs. Oh, yeah, there are 3 of these? Right! There’s one more JBOD that’s actually already in the server room. We will show you guys that one later. Now, each of our JBODs allows us to connect

Up to 3 cables for increased throughput. So that’s exactly what we’re going to do. On the subject of throughput, to push this thing to the absolute limit, we want to pack this machine with as much RAM as possible, since the file system that we’re using, called ZFS,

Can use RAM as a read caching layer to significantly improve the responsiveness of the array. ‘Cause mechanical drives on their own, yeah, even with this many of them, not that great in terms of access times. Luckily, the folks over at Micron just sent over 1 terabyte of 3200 mega transfer per second

Registered ECC memory, which, last time I checked, Smells like performance. But the performance improvements don’t stop there. We had some “leftover” Kioxia 30 terabyte NVMe drives from the Dream server build, sorry Dream. And we figured they would be the perfect choice to configure as a special metadata device in our ZFS pool. This is a game changer for performance,

Because it’ll allows metadata, like, where the files are physically stored on those hundreds of hard drives, the allocation tables, and optionally, even small complete files to be stored on a discreet higher performance device like these NVMe drives. This can make navigating through large data structures,

Like the thousands of video projects that we’re archiving, much, much faster, and that directly results in work flow improvements for our staff. I’d say, it’s about as much of a quality of life improvement as our Floatplane subscribers get, with no baked in ads, higher quality streaming, and exclusive behind the scenes content.

We’re gonna be setting these up in a mirrored stripe, leaving us with a total of 60 terabytes of capacity. Which is hilarious, given that it’s arrayed to handle just 4 drives. Which we think should be an ample amount of fast metadata storage for our 3.6 petabyte array.

Unfortunately, because we only have these 4 NVMe slots, we’re not gonna be able to expand that storage, unless higher capacity drives come along in the future and the performance, compared to some of our other all NVMe arrays, is not going to be that amazing. Another problem is that this doesn’t leave us

Any additional space for NVMe storage drives for a level 2 ARC read cache or an intent log, at least at the front. But there’s a solution to that. Ah, yes! This cute little half height PCIe card from Liquid, allows us to install four M. 2 drives in a single slot inside the server,

And this will be absolutely perfect for that role. What’s the total capacity of this bad boy? – I don’t actually know what SSDs are in there. – We don’t know what’s in it? – There is one we have that is four 4 terabyte SSDs. – Okay, so it might be this one?

– It could be this one. It could also not be this one. – Great! – It’s a problem for later Jake. – This is pretty cool Jake. Did you just design this 3.5 to 2.5 inch adapter thing? – No, we found it on Thingiverse. – Oh cool! – Dan Printed It.

– Shout out Thingiverse. Whoa! – What? – Well, that came out. Which slot did you want the Liquid card in? – You gotta change this guy, and then put it right here. – Ah, man! Can we find out if it’s the right one before we do this,

’cause then we’re gonna have to pull the whole thing apart. – What do you mean, what’s the right one? – The high capacity one. – Oh! No. – Oh, come on! There’s gotta be a test bench not far. – It’s a problem for later Jake.

– Come on, let’s go! – Oh, don’t. It’s like glue. No there’s none over there. – There’s none over there? – Yeah. – Jake says there is none. No test bench. – We’ll let’s just go to my test bench upstairs. – Logistics! – [Alex] Yes sir. I think it’s important for you guys

To know that Jake thinks you guys are utterly incompetent. That’s what he said. (Jake laughing) – Whoa! Whoa! – Those are his words, not mine. He said, “Those guys don’t even have a test bench over there at all. Not one.” – What is this? – Yeah, that’s what I thought!

Yeah! Yeah, Jake! – What is that? Mere minutes later. All right, hey, we’ve got a foreign dynamic disk here. These are, what, 3 terabytes each? – Wait go up, up. – What the hell is this? – Okay, right click on that one. Let’s see if it’s- – Toshiba 3 terabytes 84.

Yeah, perfect! These are the 4 terabyte ones. Thank you, fast and competent logistics team. I just want you guys to know that I defended you. I was the first to say, “No! I would never use those words about our logistics team.” Not like Jake. – [Alex] Thank you. (Linus laughs) – [Jake] Goodbye

– [Linus] Get out of here! (Linus laughs) Ah, here’s what we’ve got next. Now, I mentioned that this was 100 gigabit, but what I didn’t say, I don’t think, is that it has dual 100 gigabit ports. It can do a total of 200 gigabit. It’s PCIe Gen 4 by 16.

That is a flipping ton of bandwidth and it uses these sick QFSP28 connectors. While Jake orders his lunch, I’m gonna go ahead and get our other 2 HBAs installed here. – I’m getting Mucho Burrito. Editor, you better put this in the God damn cut. Okay, I’m just gonna put this in ’cause you’re taking too long. – What? What do you mean I’m taking too long? I’m not the one sitting here ordering lunch. – Touche. – This is amazing! So we’ve got our 3 HBAs, we’ve got our 100 gig network card, we’ve got our additional SSD,

Which is somewhere in the neighborhood of around 15 terabytes of storage for our SLOG and level 2 ARC. We’ll there’s 3.84 each – Oh, I see. – So like somewhere in there, 15 point something. Whew! How spicy is that? – You forgot about these. – What’d I forget? – The 120 terabytes

Of NVMes in the front. – Oh yeah, those. – Yeah. – Are we ready to turn it on? I’m turned on. – Okay. – I could be more turned on if I was shilling for lttstore.com Hey, check out the RGB diode shirt. It’s like, based on what the individual subpixels look like,

But then it’s also just a cool design. – So do you want the top? – Yeah! – Can we do a peel? – Very graceful Jake. You’re doing a- (plastic peeling) – That’s right, I’m a monster! – This thing is sick! Do we get to power it on now? Gotta laptop? – A laptop. – Uh, hi? Jeez! – Just wait, it’s actually really quiet. – Yeah! – Give it a little bit of time. – Are you gaslighting me sir? – No, I would never.

BMWs are reliable. – Sir this BMW’s very reliable. Damn it, you beat me to it! (Jake laughs) – Oh! – Oh, that is so much better, and can we turn off that alarm as well? – I have a solution. – I don’t think that’s going to work. – It does! – Pulling the power supply outright, makes the alarms for… Oh wow, it did work. – Huh, it worked.

– This is a 1600 watt power supply? What? I mean, to be clear, that’s only at 200 volt input. It’s a thousand watts if you’re from a 100 to 227 volts. – That’s what you wanted to do. – I wanna go fast! You said it was quiet, so I thought full fan speed must not be that bad, right? I had him log me separately into the IPMI, just so I could do that. – In other news, it’s booted!

We got our terabyte of RAM. – [Linus] Ooh! – Oh yeah, we got our Mellanox card in there, that’s good. Did you put the NVMes in the right slot? – I mean. – One of them is wrong. – [Linus] What do you mean wrong? – Look! – Oh, it’s wrong. – Try the bottom. Wait, whoa! One of those is a boot drive. – It might be good. There was activity flash. Yep, okay! – It’s cool, and look, because the boot device is a pool, it’s mint. (Linus clicks fingers) – Just immediately fine. – I mean, it’s probably mad.

Actually this is gonna be kind of complicated, ’cause some of the drives, we need to put in the right spot and not mix up, ’cause some of the arrays we’re keeping, and some of the drives are in another JBOD that Wendell’s working on right now, so we can’t really touch those ones.

– Uh-huh. – It’s gonna be fun, let’s go! – Yeah. – Start carrying. – How am I suppose to move this? This is a four man carry. It literally has a label on it that says four people need to carry that. – Does it actually? – Yeah.

– Dude, these are like the best earmuffs. Put them on. – Oh, wow! Now with 80 % less Jake. (Jake laughing) I can endorse these. – We don’t actually have to take all of these out. We could just take this one, and that one, and then just move this.

– So the head server will just go here, and then we’ll just go one, two, three. – Well, we don’t even have to move this one. So we just have to take the temp vault out. – Should I let the editor’s know Temp Vault’s going away?

– I already told them it was gonna happen today. – Okay. – You can tell them it’s happening now. – I’ll tell them it’s happening now. Hello hardworking and truly valued editing team. The vault, that you have come to know and love, Temp Vault, is now Dead Vault. Press F to pay respects.

They don’t care, at all. – Okay, see you later. – All right, bye. This kinda raises the question, do they need a vault at all then? Why don’t we just have no vault then? Forget it! (group chattering) If you guys aren’t even gonna miss it then-

– [Edzel] We copied the files before you pulled in. – What do you mean, you copied the files? You guys aren’t suppose to keep things on your local machines. – We copied it to the active server, new, new whonnock. – New new whonnock. Hopefully this clears everything up,

And is not confusing at all. – I think they’re more confused. Look, somebody is reading files from it right now. Like, Ed was lying. – What the hell? – Look. – Oh! – Somebody is reading from it or they’re writing to it. – What are they doing? – Oh my God! – At 350 megabytes a second. – Yeah. – This is an odd

And significant data transfer. – Look here, the chart’s like nothing, and then somebody’s like, whoo! You wanna shut it off now? – ‘Kay, who’s messing with the server? (Jake laughs) Are you messing with us Ed? – [Jake] Somebody is vaulting right now. – It’s not intentional. – Did you copy things

To the vault right now? – I didn’t, and hold on, hold on. – [Jake] Probationary employee number 17. You cut and pasted? – [Probationary employee #17] No, no, I used Tera. – TeraCopy. – TeraCopy yeah. – But it’s still going right now. – You can still cut and paste with TeraCopy.

– Dang it, guys! Okay we’re good, you’re good! – [Jake] God damn it! Ignore the disks with errors. – Uh, what? – We’ve lost a few. The original vault’s 10 terabyte drives are still “in production,” and they’re like 6 years old now. – And we’ve had problems too.

Our cooling in that room didn’t use to be super stable. There were times when you would walk in there, and it was like, (Linus shouts) wall of heat! – Yeah! – Because we had this air conditioning unit that would just leak refrigerant, – Well, what about when we didn’t have

An air conditioning unit. – and fail. – Yeah, at one point. – I remember when I started here, Linus was just like, “Nah, we don’t need it, we don’t need it.” And then all the UPSs exploded because it was too hot, and then he was like, “Okay fine.”

– How are we gonna move these? – Fortunately, these aren’t really rails. They’re actually like shelves? So all we really need to do is kinda put it on the ground, and then just, (Jake grunts) tilt one in, and then slide it in, and then tilt the other end.

That’s how we put the other one in there. – Installation by two persons is recommended. If these need two people, that server needs 25. (Jake laughs) I think I can handle it. – [Jake] I wonder when my burrito’s gonna arrive? – [Linus] All the drives are gonna have

To come out of this thing before we lift it out. – [Jake] No, no, no, it’s fine. – How are we supposed to lift this? – We can lift it. We’ve done it. – Okay, well how? – Stronk. – No, really though? – It’s not off the ground.

I just use my feet, like this usually. – Your feet? – Yeah, and then you just walk it out. – Walk it out? – Yeah! – Three, two, I’m holding the heavy side. – Yeah, yeah. (metal shuddering) – Don’t drop it, okay. – I didn’t drop it.

– Ow, you just put all the wait on my- – I didn’t know it was on your foot. – Cool! Well, that’s all the hard work. – Yeah, now that we’ve got Lab 2 over there, we may actually have a purpose for all these additional Storinators. Aw man!

I haven’t gone spelunking in the server rack in ages Jake! There’s so much treasure in here! – You’re giving everyone quite the view right now. – Whatever, they like it. – Usually, you gotta pay extra for that. – OnlyPlanes.com. How do you like this setup? I’m taking a little server cabinet nap here.

Got my little pad on the back of my head. Okay, bottom of the fifth, not bad! This was definitely the best way to do this. The front of this shelf is out? – Yeah, that’s what it appears. That’s why we screw them in. (Jake laughs)

– How full is this? – It’s got 60 drives in it. – Okay! What’s seems to have happened with this server, because we didn’t screw the shelf in from the front, is this little tooth here that holds it into place, seems to have slipped. So it’s sagging down like that.

Which means that a ton of this weight is being borne by a part that’s really not designed for that. Complicates our operation here a little bit, because we can’t just pop it back into place from here. We can’t really get at it. Looks like we need to shut this down, pull it off,

Put it back into place, screw it in, and then put it back in, which sucks! If I take the weight and you slide it forward or the other way around. – Here, I’ll use my leg. – Is it up? – There we go, it’s up. Just put the rail in!

– Don’t move. Wait, put what rail in? Well, I can’t see it. – Oh my God Linus. – Oh, I think we got it. I think we got it. Let go, let go. Yeah! – This actually went pretty okay. – Situations like this, are when I’m so glad we developed this screwdriver.

I got the nice strong magnet holding it in place. Almost no back force required. Just spinning the ratchet, like- – Whew! Is it in there? – Easy, yeah it’s in. – [Jake] Oh my God! – Wait, what? How are we suppose to get it from here, down to there?

– Well, we got it up there somehow, so. It’s just like a really heavy Storinator. Why don’t we just put the other one on top? This says this cart can take 600 pounds. – No, we’re not doing that. – Okay, I tried. What are the odds?

– That it just goes right in? – Yeah! – No, the angle is still too much. Okay, I’m getting under it boys. Push the right side in a bit more. Okay, good, good, good. – Oh my God, is it? – [Linus] Yeah, it’s on. – Okay. – Okay, go ahead,

You can push it in. Hi! Alright! – Now for the hard one! – Yeah! This is more of a Linus sized server. YOLO! – [Jake] Well, it’s easy to YOLO when somebody else already did all of the work. – Yeah! Good work Jake. There’s goes the brain.

And what’s really wild, is that if we want more capacity, more than 90, 180, 270 drives, with the cards we have in now, we could do another one, and I think there’s room for one more card, right? – I would say right now with no HPAs, we could probably add 2

And it would still be pretty good performance. – All Right, let’s put in the hard one. Ready? One, two, three, go! Yes, whew! What? Weak! (hands claps) – All right. – [Jake] Oh, still weak. – Now that they’re in there, working on these is glorious.

You just press the buttons, pull ’em out, boom! Funny story, when we first put this in, I was like, “Uh, it’s so heavy, it doesn’t really need to be bolted down does it?” I was told yes, in fact it does need to be bolted down and this is why.

Because if we had a handful of these hanging off the front of it and it wasn’t bolted into the concrete. We’d have a bit of a problem. – Bit of an update here, we shucked all those drives, put ’em in these servers. I plugged a bunch of stuff in, some power, and we have blinky lights now. – [Linus] Do I get to press the button? – [Jake] Yeah, it’s on the right side.

– Give ’em a sec. – Everybody in the next square mile’s gonna hear this. – They’re spinning up boys! – Wow, it’s not actually as bad in here. Come outside. Let’s shut the door. – Yeah, sure. – And you can hear how awful it is still, through the door. – Oh, yeah! We wouldn’t be able to shoot with those things going. – But they don’t go. – Okay. – That’s fine. Oh you just gotta add another layer of insulation. – There’s no space Jake! Well this is the backup plan, right?

‘Cause there’s a thermostat in here that will start up a fan that exhausts out the top, and will pull air passively through this rail in the event that the air conditioning completely fails. – [Jake] You remember the shroud for this thing? – Yeah. – I made that in metal shop,

When I was still in high school. – That’s hilarious. Okay, that was it. It’s up! So far, only 20 drives are showing up, but Jake tried a thing. Which is a what? A domain change? – We changed some cables, and I changed the JBODs from three zone to one zone.

– Are you gonna click it? – Yes! (Linus claps) – Okay! – Thank god! – Okay, one of these? – Ah! (Jake and Linus high five) 145 drives. God damn brother! – I mean, 45 drives is nice, but a 145 drives, now that’s walking around hard drives. – I’m just gonna try this. This is probably a bad idea. I’m just gonna do it, and try to import the pools. It should find both of them. There’s 2 discreet pools on those existing drives, – Yeah. – and then there’s a bunch of extra drives.

So let’s see what happens. – What are the odds this is just gonna be like, “Yeah, your ancient-ass pool,” and just- – They’re not that ancient. I think they’ve all been recreated now. – Okay. – But actually, the crazy thing is, I took like 2015 pools from a version of Zetafast

That was not recommended for production and had not been updated, and I put it in like 2022-ass TrueNAS. It just imported, it just upgraded. For some reason it was super slow, like scrubs would take a week. Which is really weird. – Yeah, but there could be kinda flaky drives in there.

– It wasn’t the drives, the drives were fine. I moved all of the data off, deleted the pool, made it, and then brought it back, and it was fine. It might just be like a fragmentation thing or… Oh, let’s see. – [Linus] It did something. – And- – Oh, look at that!

– [Jake] Gofast is the 12 drive setting. – Got it. – I wish I saved these I.Ds before I did this. Let’s see what happens! – Does this mean Vault is just gonna be like, “I’m back baby?” – Temp Vault will be back. The read only New Vault,

Because we’re still recovering a few things, that’s why I wanted the JBODs to still MIA, will be back. Just like that! (finger snaps) And I can just recreate new shares. I mean the IP will be different, but I can just – Wow. recreate the shares. – That’s awesome! – Zetafast is cool man!

– Don’t put that there. This is a really nice shirt though. – Wow, is that from lttstore.com? – It sure is. – It will be. – Oh, is it not out yet? – I don’t know. I don’t think so. Have you announced it on “WAN Show?” – I don’t think so.

30 hours later. – Wow, the pool’s finally imported. Holy crap! – Supermicro, massive shout out! Thank you for sending over the gear. It’s amazing! We did have a little bit of trouble with setting it up with 3 domains. – Yeah, for some reason, they’re only working in single zone mode.

That means all 90 drives are going over 1 cable. I don’t know why, because we had it set up on the bench like a couple weeks ago with 3 zones. – Anyway, it’s accessible. We’re ready to copy some files and hopefully that’s something they can sort out,

Either through helping us set up the configuration, or through firmware, or whatever else. – Yeah, it’s probably just, we’re missing something. This is one of those solutions, where it’s kinda like, you just know what you’re doing and if you’re doing it. – Yes, you’re an integrator who sets these thing up everyday,

Week after week, and that’s not us. But we’re there. – I copied a file. – Let’s do it. – Look, I’m copying a file from it to this, look, hundreds of megabytes a second. – It’s not the most impressive seed ever. – This is a SATA SSD,

And we’re screen recording to it now, so it’s like a. – But, hey! Remember guys! This is with ancient, degraded-ass, fragmented VDEVs. Once we strip all the data off of them, rearrange them, reshuffle them, set them up fresh and dump all the data back on to them, we should be able to be in the neighborhood of a gigabyte a second.

Which is all we’re gonna have. – More for sure. – Well yeah, but that’s all we’re gonna have. – Yeah! – The Mac interface. – It’s a hundred gig. – Oh that’s right, it is. – Well here, look at that! A 147 drives. – Never in actual production, that’s amazing.

(Jake gasps) – And then we’ll add 60 more, and it’ll be over 200. And then we’ll add 50 more, and it’ll be 250. – And then we’ll add this segue to our sponsor. – iFixit’s Manta Kit has 112 steel bits and comes complete with standard bits like Phillips and flathead,

As well as more niche options like pentalobe, hex and tri-point bits. It also includes two aluminum screwdriver handles, a weighty quarter inch drive for full size applications, and a smaller 4 millimeter for precision work. Both drivers feature magnetic bit sockets, knurled grips, and silky smooth spinning tops.

You can even use the tool kit’s lid as a built-in storage tray. Best of all, every iFixit tool is backed by a life time warranty. So if something breaks, they will replace it for free. So visit iFixit.com/ltt and pick up your Manta Kit today. Manta! – If you guys enjoyed this video,

Maybe you’ll enjoy the one where we tried to deploy Whonnock? – No, no, no, that was- – Awful. – Okay, sure, yeah. – Yeah, sure, go watch that. (Jake laughs) – Or the previous one where we just looked at JBODs, ’cause they’re cool.