Dominic Perez, Chief Technology Officer at Curtiss-Wright and Will Lester, Vice-President of Engineering with NextTech Solutions discuss PacStar IQ-Core and MANTLE - a practical guide to desired state configuration at the edge.
IQ-Core is a tool that PacStar has been working with it for about a decade and a half and it is really a single pane of glass interface that is meant for the soldier, the warfighter and their support infrastructure to be able to manage the complexity from day one and beyond. MANTLE is an incredible tool for getting your system up and running and then IQ-Core is a great tool for maintaining, monitoring and managing that system once it's already fielded.
IQ-Core NCM, our network communications manager, manages your routers and switches.
Beyond that we have added a new version called ROAM or remote access and management and ROAM takes that same power and moves it hierarchically so what that means is you can have the knock, have visibility down into subordinate systems and it's multi-level so it's not just one to several.
Transcript
George Seffers
Hello and welcome to our SIGNAL Media webinar series. I'm George Seffers with SIGNAL Magazine. The opinions expressed in this webinar are of the presenters and are not to be construed as official or reflecting the views of AFCEA International. This presentation is sponsored by Curtiss-Wright and NextTech Solutions or NTS. Today we will discuss how to configure tactical hardware in a matter of minutes rather than hours. This is intended to be a practical guide to desired state configuration at the edge. Our guests are Dominic Perez, Chief Technology Officer at Curtiss-Wright and Will Lester, Vice-President of Engineering with NextTech Solutions. Following the presentation we will host the Q&A session, so viewers are welcome to submit questions using the ask a question box on the webinar console. And with that it's time to get started. Dominic, would you like to kick it off?
Dominic Perez
Thanks, George. Let me start the presentation here, and here we go - Configuring Tactical Hardware in Minutes and not Hours. Just a little refresher, this Is us, the folks who will be presenting today. Myself in the middle - I'll be kicking this off with about the first third of the presentation, handing it over to Will and then back to myself. We're going to keep the cameras off while we're presenting the content just so that you can have the full screen available to see those slides. So some of us, some of you may know us better as PacStar but now that we've been part of Curtiss-Wright for a while we are starting to transition the branding. Going forward, PacStar will be known as Curtiss-Wright PacStar Communications Solutions. We're still based in Portland, Oregon, where we manufacture our hardware and where the vast majority of our engineering staff is located.
Our presentation today will be in three parts. The first I will cover: Automation Drivers and Edge Challenges before handing it over to Will for an Automated Edge Configuration Demo and a deep dive onto the NTS MANTLE product and then we'll pass it back to myself and we'll go over the hardware and software that make this work. So let's talk a little bit about the edge. When you reference edge, you see a Google search for edge, you end up with a lot of folks that think that a branch office or a remote bank is where the edge is and that's reflected by the gentleman in the top left.
To me that's still a really nice looking data center and it's not the edge. On the right you're getting closer to what we mean by the edge - some gentleman in a tent, they've obviously set stuff up ad hoc and then extending all the way to what we call the tactical edge where you're in, you know, inclement conditions, you're dealing with size weight and power as a primary driver. So, configuring equipment. Let's say today is new hardware day. If you're a geek like me, nothing makes your day a little brighter than seeing a new piece of hardware that you get to pull out of the box and start figuring out what you want to do with it and start that configuration. That's great. Until you realize that there may be racks and racks or pallets and pallets of equipment that goes along with this and it's going to be your responsibility for getting not just one of those devices configured but tens or potentially hundreds of devices and devices making up kits of even more devices. And that job is not getting any easier. Getting devices correctly configured is getting harder.
There is a ton of different interfaces you have to deal with in today's modern networks, whether they're at the data center or at the edge. System complexity is increasing, we're bringing in more security requirements which in turn is more devices, more interfaces, more complications. If there's one consistent theme that I've heard in talking to our end customers, the soldiers and the war fighters, is that training and resources are limited. We need to allow them to focus on the mission and not focus on the equipment and we believe that there's a better way. There's a lot of different titles that this better way can fall underneath, whether that's just general configuration management, and I don't like that because it's too ambiguous but, you know, what's the configuration, how are you managing it, it's a good start but it's not encompassing what we're trying to do.
Desired State configuration is the one that I chose to use in the title of today's presentation, although Microsoft has kind of co-opted that for a bit of their tooling around this infrastructure. It's not as generic as it used to be. But breaking that down, desired state configuration, what do I desire the state to be and what is the configuration that leads to that desired state? that's getting close to what we're trying to do.
Declarative infrastructure - my good buddy Will is a super nerd, he likes this. I thought that was a little too nerdy for what we're trying to do and then infrastructure as code - I think that really boils down to what we are doing but code can sound really scary and today in this next hour we'll show you that code doesn't have to be scary and code is just structured language and we'll break that down with some examples. If you've configured any type of network device it should look pretty natural to what you're used to. So why do we want to do this? Well one, we have more equipment than we can deal with one by one, hand by hand, jamming configurations, copy and pasting. Yes, that can work, but it's not standardized, it's not consistent, it's easy for people to make errors. People are imperfect. Computers or not, you give them the right information on the input, you get the same consistency on the output.
So we can maintain our security policies, our configuration consistently across the entire ecosystem of that we're deploying. It lets us use really cool tools. You may have heard of DevOps - that's kind of the intersection between writing software and the infrastructure that it runs on. And I don't think most of the people that are listening today are writing software but you're deploying software that someone else has wrote, whether those are tools that were written internally or tools that your organization has purchased from large providers like Microsoft or Cisco or Juniper.
So we can use those same practices when developing the configuration to run that software. It's much more time and cost efficient. Time is money and the way that MANTLE can work in parallel will save you tons of time. We're going to talk a little bit about the cloud. The cloud is not just a remote data center, it's not just something run by Microsoft or Amazon. Code is really, sorry, cloud is really a way of thinking about computing resources today and we want to extend the ability to think about computing resources the same way, whether that's in a hyperscale cloud provider, your data center or at the tactical edge.
And then finally, faster recovery and disaster resilience. If you have your code and your configuration encapsulated as code be blown back onto the system quickly, you don't have to worry so much about recovery. If it's just a few minutes away and a couple of button clicks to get back to a clean config it's easy and you can sleep better at night. If you've been around cyber security for any length of time you've probably encountered the CIA Triad - that's confidentiality, integrity and availability. Confidentiality is something that is pretty much taken for granted in security.
We want to keep the information confidential. But integrity and availability are just as key to protecting data. If the data doesn't have integrity then you may be preserving something that is actually worsening the problem and if it's not available then it's not useful. The most secure system in the world is probably locked in a closet behind locked doors but if no one can get to it, it's not really making that data very useful. So consistency in your configuration is key to maintaining all three of these core principles of cyber security. And configuration errors have real world consequences. It's not hard to open a newspaper or your favorite website pretty much any day of the week and hear about a new cyber security breach but I chose to highlight these particular ones because they're traceable back to simple configuration errors - a firewall that was misconfigured, a database that didn't have the correct permissions on it and these are things that could have been prevented if we had rigorous controls around our configuration.
And if you look at the bottom right, Korean Hydro Nuclear Power, that is critical infrastructure and potentially a very dangerous situation. So these configurations do continue to have real world consequences. So I talked about how we're going to get some cool tools to work with our configurations and once we've defined the infrastructure as code we get all of these cool tools that software developers like to use. We don't have to be a software developer to use them but version control is a really powerful concept once you get into it.
If you've ever tried to use Git and you've got a little confused it's understandable but when you're dealing with very simple text files and only a handful of people working on them it's really not that hard for the basics and it brings a ton of power when you don't have to worry about knowing what version 'hey did someone email me that configuration, was it the right configuration that they emailed me?'. No, you get this single source of truth. So it's transformative for developing your configuration. You can try new things, you can jump forward and you can jump back to versions. It's transformative in maintaining system updates and patching because we know that deploying is just the first step of a system life cycle and then always knowing what is being deployed is critical and you can ensure that every system out in the field is running the same thing. So these are not new concepts. This is something that has been around an industry for a number of years and there are plenty of commercial and open source options available for managing configurations in this way - things like Puppet Labs, Chef, Microsoft Desired State Configuration, Ansible and Terraform. These are all great and powerful tools that embody these concepts that we're talking about but they are primarily targeted at the hyperscale cloud.
They expect that network SSH access is available. They expect the hardware to be in a known state and what MANTLE is going to do is it is going to help get to the baseline with hardware that is not cloud-based and basically allow you to use these tools or extend MANTLE with them to manage your configurations. So you're dealing with edge hardware. It's not in a data center, it's not in a rack - I know, I'll go and build something myself. Trust me, I've done it and now I'm still on the hook for maintaining it a decade later. It does require significant expertise and it can be fragile. I set out to solve one particular problem and when different problems arise it's not flexible enough to adapt to those and frequently it's undocumented and unsupported. You know, in the DoD it's pretty much par for the course as soon as something gets working, the person that wrote it is going to get transferred out.
So that's why we have MANTLE. MANTLE is designed for standalone usage. It's designed for hardware in an unknown state. It's designed for disconnected environments that are common in the battlefield and it comes with a very low barrier entry and we'll show that in just a few minutes. So if you spend any time in IT, I hope you've seen the IT Crowd and they're famous for saying 'have you tried turning it off and turning it back on again?' Pretty much a universal troubleshooting step but I think that is a troubleshooting step from the last decade. The troubleshooting step today should probably be, well maybe after you've turned it off and turned it on again, is have you blown it away and restored it from the known configuration and when you can do that in minutes it saves hours of troubleshooting just problematic systems. If you know it was working and you can resend that configuration to it, boom, you're on your way.
So what are we trying to do with all of this? We're trying to move the data center to the edge and we do that with PacStar's Modular Data Center or the Modular Data Center NR that we show here that has NVMe and Remote Management and that is out at the edge collecting sensor and recon data, interacting with the warfighters and their EUDs, pushing and pulling data from them and synchronizing over whatever medium is available, whether that's commercial SATCOM, governments SATCOM, 4G, 5G, TRILOS RF, whatever, back to any type of cloud, whether that cloud is a hyperscaler or a private data center cloud that the government is running. So with that we're going to teach you how to make your configuration as code, plug it into MANTLE, and configure that system. With that, I will hand it over to Will.
Will Lester
Thank you, Dom. Let me get my screen share going here. I think you covered a lot of the material really well already but I'm excited to be able to share a little bit more about what this looks like with MANTLE. So I want to start, I think in IT we tend to be pretty bad at using words and not defining them, or especially acronyms where we use those and throw them around and never get to definitions. You did a great job of going over what desired state configuration is and we use that big word and scary word declarative and all that really means is that we're starting with the end in mind. We're declaring what we want the state of the world to be at the end of the process and we're not specifying how we get there, we don't have to do step by step which is something you might do in a lot of your scripts. Now to get to that we have to be able to express that desired state in some format and one of the right ways to do that is this idea of infrastructure is code.
Wow you might say, Will, I'm a network engineer, I'm a sys admin, I don't write code, I'm not a software developer but if you take a quick step back and you look at what code actually is, it's like Dom said, it's structured human readable language that the machine can understand. So I'll start out by picking on the network folks a little bit since they tend to get blamed for all of our outages anyways. And if you think what the job of a network engineer is, is to log into a router, a switch, or a firewall and issue commands that result in a good running configuration that produces a desired state. You're not logging in and telling the router how to route or the switch how to switch, you're issuing commands like, you know, enable, configure terminal, copy, run, start and that results in the config that you want. You also know intuitively that there's an order to this. You have to be able to stand up your Layer One before you can move on to Layer Two and Layer Three and so forth and if you get those out of order things go wrong. And this is where we typically stop. We look at our configuration, we know what order we need to apply it in and that's where we finish. But if you take another step back and you look at the configurations, what you'll notice is that there are parts of them that are more structural, that don't change, that are static between deployments and then other portions that change or are variable. And I use that word variable very intentionally, right?
So we're talking about extracting variables from configuration. We're talking about ordered operations and we're talking about human readable commands and that's coding - that's the same thing we do in software development. But there's just something magical that happens when you start to see your tools this way and extracting these out and you get access to all those tools that Dom was talking about to manage our configuration state better. Now we need something that can take these configurations as code and turn them into a desired state device and that's typically some sort of an automation tool. So automation is a word that we use to describe letting the computer do the work, letting the humans sit back and watch. And why do we like that? Primarily, we like that because it's fast. We like to brag about how much faster we can do things - it's even in the title of the webinar. And that's super cool because computers never take a break, they never have to go to sleep, never go on vacation, they're just always waiting for us to provide them instructions to take care of the things that we need to. And there's no delay between steps it just continues on.
In addition to that computers have way more cores and threads to be able to do things simultaneously than human beings ever could. If you've ever, you know, tried to multitask by watching a webinar and checking your email at the same time, you know that you're probably not doing either one of those things particularly well. And so that leads to mistakes. That leads us into the second point. We as human beings are prone to error. We just get distracted, we get tired, we've fat finger variables, we forget DNS servers and NTP and things break and as Don mentioned earlier, all of this new complexity that we get from modularity that's amazing to give us flexibility and deployment also creates all these new opportunities to make mistakes. And so what we see in dealing with our customers as a response to this increasing complexity is there's typically two paths they take. One path is to create very structured processes and silos within IT where the networking folks, for example, might start out with the kit to get the networking portion, then hand it off to server folks and application folks and then at the end cyber folks come in and they break everything because we didn't start the process with cyber in mind. And so they this process can take days, it can take weeks or even longer, and for a lot of units that's not acceptable.
So the other option is to bring the super nerd to the table - the person who knows how to do all the things, that's been there forever and can do it all themselves. Now as a leader I hate this idea because what we're saying is we're bringing our very smartest people in to do menial repetitive tasks that they were not, they would be better suited not doing. I want my smart people doing smart people stuff. I want the smart folks thinking about the future state of my systems, the risks that might be coming down the pipe and other things so that I can improve my baseline - not waiting on a progress bar or not clicking next until the system is configured. So I'm going to use the right folks and I had this problem at NTS years ago because, you know we do a lot of demos, we do a lot of trade shows back in the before Covid times and proof of concepts and things like that where we have, we do get that new hardware day that Dom was talking about quite often, where we get a loaner of a piece of gear and we got to get it up and running and so my response was put my very smartest nerds on it. Let them build it and that was a waste of their time.
And so we looked at the market and said there's got to be a better way to do this and we found a lot of really amazing tools for infrastructure that were focused on the cloud. That were focused on using things like Ansible and Terraform which all requires some level of SSH or API-level access into infrastructure that they presume already exists. And so we set out to build a tool that would help us in that earlier phase of the deployment process and that's what MANTLE became. So what is MANTLE? At its core, MANTLE is a purpose-built edge infrastructure as code tool. We take that same concept of declarative configuration that Dom and I talked about earlier, express that into human readable language, we feed that into MANTLE and out the other side you get a fully configured tactical kit. In this case it's a five node smart chassis from PacStar. This is one of those new hardware day things that Dom was able to loan us on and I promise I'll send it back eventually but I'm kind of enjoying playing with it at the moment.
So how does MANTLE actually connect into this device? Let's jump into the wiring diagram for the nerds among us. So on the right we have the five node chassis and you'll notice that the first three slots are full of servers. These are PacStar 451s, they are going to perform as our virtualized data center. So we'll install ESXi, vCenter and vSAN for storage and that will give us a fully hyper-converged edge data center. MANTLE takes care of that. And then next to that we have the PacStar 448 which is a great 10 gigabit switch with low latency and high performance. It's a great size and everything in this package. So that gives us Layer Two. So we have data center on the left, Layer Two and some Layer Three capabilities within the switch and then we have to start considering what are we going to do for Layer Three? A lot of times we reach for a router - something like a 447 from PacStar which is another great piece of equipment that Dom can talk about a little bit more in the coming slides.
But something that's becoming even more popular is to virtualize those Layer Three surfaces using things like Cisco's Catalyst 8000V or the CSR 1000V if you've used those in the past. Now that requires that we deploy a hypervisor and deploy a virtual machine and then configure the network and that really breaks all of those siloed processes that we were talking about earlier. So this five node kit - MANTLE can handle each of these three use cases - the data center, the physical networking devices and virtualized networking devices. On the left is actually where MANTLE lives. In this case I installed another 451 server and put MANTLE on it and that box kind of sits in between both my management network here in green and allows me to remotely log into it and send commands to it and have it execute those against the kit that we're going to build and then that purple network is the other connection in to complete the configuration on the device. In blue below you'll see all the USB serial connections. So MANTLE connects to every device in the kit over serial and that gives us very low level control so that when the hardware is in some unknown state we can very easily get into it and configure it and revert it to its factory default and start over. So that whole process, all the items in this kit, that virtualized data center, the Layer Two switching and the virtual routing can go from a completely out of the box state to fully configured in right about 90 minutes.
So a lot of time savings compared to doing this manually and a lot less brain power used by the personnel that are involved. You know, since we don't have 90 minutes today and I don't think you'd want to watch progress bars and paint dry on my side of the world I'll jump through and do a quick MANTLE demo of the UI and then show you some other videos and things that we've got to demonstrate. So here I am, logged into the MANTLE appliance. As I mentioned before it can run as a virtual machine either on top of ESXi or VMware Workstation if you want to run it that way and lastly you can also run it as bare metal so a lot of our customers do choose to deploy MANTLE as an appliance bare metal to the edge. So you log into that via the web UI and you have a couple of options here.
We want the UI to be very user friendly and give you the data that you need right away. Before you can dive into building anything though, you have to upload software. So we don't package any of the software you might need from VMware or Microsoft or Cisco. You have to bring that to the party yourself. So that's a one-time thing. You can come into MANTLE upload your ISOs, upload your OVAs, any licensing data that you want to use. You do that once and then MANTLE reuses those throughout the process. The more interesting one here is the templating. So this is where we start to do our network templates. You upload those as plain text files into MANTLE and then I had a couple of them here - one for the router one, for the switch.
I'll pop into the switch and you'll see MANTLE has scanned those templates. These are plain text documents, nothing special about them and all the variables get identified dynamically by MANTLE so those are able to be identified at the upload phase and then when you go to deploy in just a minute we'll show where you can actually input the variable data for that deployment. We also scan and find all the network interfaces so you know which interfaces are available on that particular config. If I pop here into the show contents button I'll get a syntax highlighted version of this that shows you a little bit more about that concept of separating structural components which are here in white, from the variable components that are here in the red.
And so the process of this is actually really easy. We've had customers go through this in, you know, five or ten minutes to understand what needs to be done and all you really have to do is identify the parts that change, surround those in these curly brackets, and then give it a name. As you can see there's a bunch of them that get reused throughout because there's a lot of repetitiveness in most of our configurations. You only have to specify those once and the automation rendering takes care of that. So it's very easy to separate those things out, structural from variable. So I'll start there. Let's jump back into the home page here. We'll start on the network. There's two sections of MANTLE on the home screen. You can build a network or you can build data center components. Typically we start with the network and that gives us a few options. So we can do physical devices, those routers, those switches or the virtual devices, those CSR 1000Vs or Catalyst 8000s as well. We also have the option to either build things from scratch if we've never done them before or use a previous build as the basis for our deployment this time around.
So that's what I'll do here. I'll click build from previous to move into the process of deploying this PacStar 448. You can see here's the one build that I've done on this particular MANTLE appliance. It took about four minutes to complete. If I click on that you'll see it automatically selects the template and the device port that we're going to use for this deployment and then pre-populates the values for all of the variables that you saw earlier in red. So you can reuse these, deploy exactly what you had to start with or come in here and change them. I do recommend that you come in here and change them if you're going into a production environment. Right, so you click the build button and in three or four minutes you come back to a fully configured Layer Two switch.
All right, so that gives us our Layer Two connectivity. It's always a good place to start. Next we might want to move up into Layer Three. So we can come in here, see where our virtual networking is, so we'll deploy something like that CSR, that Catalyst 8000. And this time we'll do it from new to show you some more of the options. So as I mentioned earlier, MANTLE automates at the very lowest level that we can do, which is the BIOS. This is done so that you can set things like the date and time on the server, set boot order, you know, simple things like that all the way up to configuring RAID controllers, and doing other more advanced things that need to be done at the low level within the BIOS. Once the BIOS is configured you can move on to ESXi.
In this case, as I mentioned, a virtualized router runs on top of a hypervisor so you have to set up those parameters to begin. Very familiar things if you've ever done that- IPs, host names, passwords, versions, licenses, that type of a thing. And then moving over to the last tab on here, we want to select out our network device. So this is where we bridge the networking into the physical or into the virtualization side. So we'll grab a template. We have the option of forwarding through any physical serial ports that might be on the hardware if you want to treat that server as a true router. That's possible. And then right here we can select our OVA. So we can say I want to do the Catalyst 8000v and then you can see the variables. Just the same as you would before on the switch configuration, fill those in and hit deploy. So that process there takes about 20 minutes, sometimes less, sometimes longer depending on the configuration. That's to do the full BIOS automation as well as the hypervisor installation, ESXi, deploying the virtual machine and configuring the router so about a 20 minute process all told. So now that you've got your network up and running the way that you want it, you can focus your attention on the data center. So same options - build from new, build from previous and then importing a config.
So this is common if you've had a configuration built from another MANTLE appliance and you want to rebuild that same kit on this appliance, you could import that config file or just do like we've done before and start from previous. So here's the one that I built for the webinar. It took about an hour, a little under an hour to build and I can click in, I see a lot of the same options - the BIOS, ESXi, in this case it's more nodes so we can configure more servers. Moving over into vCenter for all of those configurations, vSAN, advanced networking if you want to do things like distributed switching as an option and then a post install step to start deploying virtual machines. So this will deploy sort of the shell of the VM if you will so that your higher level Ansible and other scripts can take over and start doing the actual application configuration. So I can click the build button here. MANTLE will read in all that configuration data and set out a number of steps that need to be accomplished in order for the build to succeed. I can click the deploy button here. It'll let me know 'Will, you need to power on your devices'. So MANTLE starts all of these deployments with everything turned off so we know what state it's in. It's off. When we turn it on it powers on and then MANTLE takes over from there. It runs a series of pre-checks to make sure that all the variables are there, all the things that you need for the build are present and then it begins the automated deployment process. So if you're at this point in your build and everything's green up here you can go to lunch and come back in 45 minutes to an hour, sometimes less, and have a fully configured data center. I'll cancel this one out. We don't need to wait on that today.
One of the things that does happen at the end of the build process is that MANTLE goes in and it scans the kit that it just built. So it pulls things like hardware information for serial numbers and data about what's actually in the box, so CPUs and thread count cores and everything, as well as the software that was deployed so you know exactly which version of ESXi or vCenter that you deployed. And then all the disk complements and any other hardware that is inside the server. This can be really helpful if you're trying to track down a particular serial number of a disk without, you know, going through your infrastructure and pulling disks and trying to figure out what's going on. This is super helpful. Same thing with networking - you can find all that data here in the review tab within MANTLE. You can also view the logs so if you need to troubleshoot something or you need to save these away for some auditing type task, those are available. You can see the different tasks that were accomplished during the build right here and then you can download the configs so you can take this config and back it up to something like a git repository or you can transfer it over to another MANTLE appliance to build and rebuild the same kit. And that config kind of becomes the contract, if you will, between, you know, the person who built it and the person who's receiving it. If there's ever a question about how was it built you can always go back to here and say this is how it was built. If it's broken now, it was working when I had it. So it works on my machine.
So that's MANTLE. I'll pop back into the presentation a little bit and show you a few things in terms of what it looks like when the actual process runs. So I have a little video here. I connected my iPad to a capture card along with three of my PacStar servers. So you can see all the BIOS level automation occurs in parallel so every server has this BIOS automated. All the configuration steps that might be there are taken and the servers are rebooted in serial so you know which server booted at what time so that server one is server one and server two is server two so there's no ambiguity there. At this point it's all hands off. You've already gone to lunch waiting for this thing to configure. And when you come back, what you end up with is even through all of these different reboots of the server, is a fully STIG ESXi instance. So we start stigging as soon as we're able to start applying as many of the STIGs and security controls that we can early on in the process so there's no surprises later, right? So 20, 25 minutes, you can come back and have a fully stigged out ESXi installed and MANTLE takes over from there to do all the vSAN and vCenter stuff that need to happen and you can come back to a fully configured data center. So that whole process that we just went through for all three different components took about 90 minutes. sometimes it's more, sometimes it's less but we also know that when you do these configurations, as Dom mentioned, you may need to take them to the field and then maybe do a disaster recovery type situation and rebuild them offline.
And so MANTLE was built with offline in mind. We already package all of our documentation, all of our capabilities directly into the application so if you forget how to do something you can always get help without having internet connectivity or being able to log back into some cloud somewhere. All the docs are local to the machine. So we actually gave MANTLE as an OVA to some of Dom's engineers a few weeks ago and honestly, we got a little busy doing other things and forgot to train them on MANTLE and so I was really surprised last week when I got a phone call that said 'hey, we've been using it, deploying servers deploying network devices, we've got some great ideas for other ways we can use it', and that was with zero training. So that's either some testament to the usability of MANTLE or probably more than likely the caliber of Dom's engineering team. So in addition to the documentation, we also have a robust API that's documented and baked into MANTLE. That enables higher level Integrations with things like Ansible and Terraform to make MANTLE part of a larger automation strategy if you have some more advanced use cases. We don't see MANTLE as a silver bullet of automation. It is a great standalone use tool. It does help you get into this idea of treating your infrastructure as disposable or ephemeral but it does, it's not an end-all be-all. So we want to be a good citizen of the community and interoperate with a lot of the other tools that you already have. If you're using Ansible, MANTLE slots in right alongside Ansible and can even be called and executed from within Ansible, so we do that a lot in our environment. So now that we've gone through the process of building out that tactical edge data center, we have to start thinking about what's next, right?
The day zero - that hardware unboxing day is already come, we've already got it basically stood up but now we have to start monitoring it and start backing it up and doing other things that are normally part of the life cycle management process. And MANTLE is laser focused on doing the bare metal portion so that higher level tools can take over - your Ansibles, your Terraforms. One of the great tools in the space is actually made by our friends over at Curtiss-Wright and PacStar and that's called IQ-Core. And so I'm no IQ-Core expert. I did all of this without no training as well so Dom, if I did something wrong please correct me. But we love to work in tools like IQ-Core.
What I did was I deployed IQ-Core alongside the MANTLE appliance on that 451 as a virtual machine and then at the end of the process connected all the different pieces together. So you see I'm actually monitoring the MANTLE appliance, I'm monitoring the Catalyst 8000, the switch, the vCenter - all those different pieces are monitored by IQ-Core and connected to IQ-Core. And that lets us do cool things like see the interface status, read the configuration data, pull MAC addresses, see routing tables and in my example I actually configured one to connect to my home network over OSPF and set up a full Layer Three connection and more of a Garrison to edge type deployment that's common that we see. So IQ-Core definitely gives you the insight into it and then you have even direct access into your devices from within IQ-Core. So you can get direct terminal or console access via SSH to all these services now that they're up and running from MANTLE. You can even back up the configuration and store those and compare those to past baselines to make sure that you've got what you expect. You also get the ability to log into virtual machines in addition to your networking devices directly into the vCenter and pull snapshots and start and stop VMs, whatever you might need to do, run scripts, get access.
It's a very cool tool and very complementary to what we're doing with MANTLE. And so we see MANTLE and IQ-Core as better together. We want to be able to continue working with PacStar on this and we're excited to be able to share more about that in the coming days. And so with that I'll hand it back over to Dom who's actually an expert in this and we'll learn more about IQ-Core.
Dominic Perez
Great. Thanks Will and thanks for the kind words about my engineers. And let's face it, engineers were probably not going to read the manual anyway. That's fair. All right so Will mentioned IQ-Core. And if you're unfamiliar, IQ-Core is a tool that PacStar wrote. We've been working with it for about a decade and a half and it is really a single pane of glass interface that is meant for the soldier, the warfighter and their support infrastructure to be able to manage the complexity from day one and beyond. MANTLE is an incredible tool for getting your system up and running and then IQ-Core is a great tool for maintaining, monitoring and managing that system once it's already fielded. We have three primary versions of IQ-Core. The first is the core - it's IQ-Core NCM, our network communications manager and that has most of the items that Will showed you. It will manage your routers and switches.
It'll let you drop into the command line for those items. It will give you bandwidth charts and it will back up your configurations. Beyond that we have added a new version called ROAM or remote access and management and ROAM takes that same power and moves it hierarchically so what that means is you can have the knock, have visibility down into subordinate systems and it's multi-level so it's not just one to several. Each one of those sub nodes like you see in the middle could have sub nodes below it and each one of those nodes is a whole system and it very efficiently reports up system status to the next layer and then up again to the knock and up again to a higher level knock if that's what your system layout is. IQ-Core ROAM has really powerful tools for managing configurations without taking away the power of the folks that are at the edge because they have the most situational awareness. You don't want to force a software update onto the node but you want to make it available.
So IQ-Core can push those configurations whether they are for your Cisco devices, your VMware devices, even for a MANTLE that's out at the edge you can push a little package to the end user and it'll pop up a flag in IQ-Core and say 'hey, you've got an update waiting for you - when do you want to apply it?' And the third major version is IQ-Core CM or Crypto Manager and we work extensively in the CSfC ecosystem - that's Commercial Solutions for Classified. It's a program that's administered by the NSA for allowing access to secret and top secret networks using commercial equipment. And it involves a layered approach so you can't have two layers from the same vendor, so by definition you're already working with even more vendors than a normal system. IQ-Core can take the complexity out of all the certificates that are involved in setting up one of those systems, making sure your VPNs are configured in an NSA-approved manner. I don't have time to go into every detail about IQ-Core today. I think if you look in the SIGNAL archive there's probably some webinars that are a little more focused on IQ-Core or please just reach out to us and we can arrange for a one-on-one or even a demo for you guys to install on site. So I talked a little bit about IQ-Core CM and today we're talking about MANTLE.
So IQ-Core CM is meant for administering, maintaining, monitoring these CSfC systems in the field. We have a very large program with the US Army deploying these systems and I'll tell you the thing is a little bit complicated. There are three networks and seven servers involved and on those seven servers there are 14 different virtual machines. And when I talked about some automation that we wrote, actually I wrote it, years and years ago and it's become fragile. As all of these things have evolved over the last several years we've looked to MANTLE and found that as a potential solution for easing our internal ability to build this and for our customers ability to rebuild this in the field. So that was the system that one of my Engineers picked up MANTLE and just went and go on without picking up the phone to NTS. Very powerful tool. So what are the Tactical servers that we're using here?
Okay, we have the PacStar 451 - that encompasses all of our one slot modules like you see there. It's about a five by seven inch box, about two, two and a half pounds depending on the version and you can load that thing up with 16 cores, 128 gigs of RAM and three NVMe drives in our newest version. That newest version has two 10 gig interfaces which is really good if you want to do the data center clustering like Will showed with vSAN or in the five slot version, if you need to have lots of network interfaces, if you're doing those virtual network functions like the CSR 1000v or the Catalyst 8000v that Will was talking about from Cisco. And the new ones are available with IPMI for remote management so you no longer have to hook up a keyboard and monitor if you need to control those. We also have two and three slot modules that pack in powerful uh NVIDIA GPUs into the tactical edge and a three slot server where I can deploy eight disks and 122 Terabytes in about a six pound package.
One of the coolest things about being part of Curtiss-Wright is access to really, really rugged hardware. The PacStar 400 series - it's rugged, it's meant to go in communications vehicles, it's meant to go anywhere that a soldier is going to go. But Curtiss-Wright has a couple of model lines from Parvus and in our VPX line that are really rugged - you know, trench it down with a hose and run it over with a tank but at the core there's still x86 servers and they can benefit from these same MANTLE and IQ-Core tools for zero day and day one and beyond management. We also have, like Will mentioned, Cisco powered devices, routers and switches. He was focused on the 448 10 gig switch but we also have a 444 and a 446 that are more user access focused. So those are 10 and 24 user access ports respectively and each one of those has two gig, two 10 gig uplinks, so you can connect it into those servers or the 448. And then the 447 is that hardware-based router if you decide that that better fits your mission than a software-based virtualized solution. Similar to the servers, we have Parvus and Curtiss-Wright VPX versions of routers and switches as well. Where the power of this really comes together is with the PacStar smart chassis and our packaging solutions. With the smart chassis you can take four or five of our modules and build them out into battery backed robust units that are now flexible and interchangeable with our packaging solutions. The 4 slot plus the five slot happens to fit in a 4u rack mount and we have rack mount frames available for that - both ruggedized and for telco racks. and then something that we released last year because we love this cool super powerful super rugged VPX Hardware that Curtiss-Wright makes, is a chassis that is the same size as the PacStar four slot chassis but allows you to use those VPX cards.
VPX technology is probably more power efficient and doesn't have the CPU power that we have in the 400 series, but there are some technologies in VPX that probably are not going to come to the modular solutions like assured position navigation and timing, and software-defined radios. So with these two we kind of have the best of both worlds or a hybrid solution for getting the equipment that you need out into the field. And you can take that equipment that you need out into the field in a variety of packaging formats. If you're a small team and you just need a, you know, a router and a switch and maybe some batteries we've got Pelican case Solutions that'll do that. If you are trying to take that modular data center out or CSfC kit, those are frequently going into our carbon fiber Transit case that is airline carry-on sized. Once you get there you can pop it out and mount it in those racks whether they're in the telco rack or in a vehicle. Or a new one is the standardized vehicle envelope or SAVE. This is a form factor that the Army's defined. It's a little bit narrower than a 19 inch so in that case we can fit a four slot and a four slot on either side for eight modules or a four slot and that VPX chassis.
And that SAVE envelope is something that they've established in nearly every ground vehicle today with the expectation that it will be kind of carved out in other platforms going forward. With that we can drop it over to questions.
George Seffers
All right. Sorry about that guys, had a little trouble getting my mute turned off. Yes, thank you for a great presentation. It is time as you mentioned for our Q&A. So, first question. Is I think the wheel, let me get my glasses on so I can see this a little better. Actually, I think maybe I can combine the first two questions. What does MANTLE look like for a 50 ml network, not five nodes and what is the limit to hardware that MANTLE can be used to automate.
Will Lester
Yeah, really good question. So if you start thinking about that slide that Dom showed a little bit earlier of what we see as the Tactical Edge, 50 nodes doesn't always fit in that package. Maybe one day as we shrink those down nice and small but you're talking about more data center type deployments. I don't think of an example where you would want to, you know, bring down all 50 of those nodes at once and treat that as kind of a one piece deployment.
You might have individual deployments within the 50 or different enclaves and things like that and so you would deploy those using MANTLE. In a larger sense though that could sound more like a day two operations piece where you'd want to manage the processes within those nodes and that's really actually a good use case for IQ-Core to be able to log into those nodes and do more management activities on the day-to-day basis. So we do want to continue to increase our scope in terms of support. I think we've done as many as 32 nodes at one go which is more of an automated use case for a lab where you're doing training but we like to keep it, you know, in those smaller packages to what Dom was showing earlier.
Dominic Perez
Yeah, just to add to that, I think MANTLE is a really good tool I could conceive, you know, somewhere near 50 in a tactical deployment but that would be something like a five enclave solution for each enclave then had, you know, five to ten servers in each and you're doing some clustering and some comms and in that case you'd probably have a MANTLE config per enclave that you would run in each one of those in parallel.
So instead of 50 servers probably taking you five weeks to configure it might be something that you can get through in a day or two. In terms of IQ-Core, there's not an official upper limit. It really depends on the amount of hardware that you back behind it and what those devices are and how much you're doing with it. But it's definitely in the high hundreds to thousands depending on what those look like. So it's really ready for the knock and then as with ROAM those numbers go up even higher because each subinstance is a different IQ-Core managing those and then reporting up pretty efficiently.
George Seffers
Okay, great. I think you both might want to chime in on the next one as well. In demo how would you push new configurations to nodes already deployed to an ALR? You need to recover the nodes from the field and do this at a management depot?
Will Lester
Yeah, I think that actually does speak to this idea that we talked about earlier of using those same concepts that we would use in the cloud which is to treat those infrastructure components as ephemeral or temporary, disposable, depending on what word you want to use. So our approach is more on the cloud native side where we would come to those devices and do a start over, right? Rather than trying to troubleshoot and make small changes we would wipe them and start over.
That's more the cloud way of doing things but if you're trying to manage those on a day-to-day, once you already have access to them over SSH and other capabilities that's another use case for IQ-Core, to log in and manage the configuration baselines within that software.
Dominic Perez
Yeah, it really depends on what you mean by pushing a new configuration. Like I said, IQ-Core, especially in the ROAM Edition, has the ability to allow you to push configurations both down to the nodes and down to the devices. But it could just as easily push other information down for MANTLE. It could push new OVAs for MANTLE to do the redeploy so it really depends on what kind of style of configuration management you want to do after you do that initial deployment.
George Seffers
Okay, good. And how does MANTLE work if we have other automation tools in place.
Will Lester
No, that's a great question. So as I mentioned earlier we have this API that we expose. We don't expect that MANTLE is the end-all, be-all of your automation. It's not going to replace a fleet of Ansible scripts or Terraform or other capabilities that you have if you've already gone down that road. We want to interoperate with that. So you probably have a flow or an orchestrated process to go from, you know, that day zero process that MANTLE is really good at all the way through to full commissioning of applications and other things on top of that. So the way that we've seen most customers use it is as part of a larger workflow and I'm using an orchestration tool like Ansible Tower or even something like VRA or other tools from other vendors.
George Seffers
Okay, nice. I'm sorry - anything to add there?
Will Lester
I was just saying we play nice with all the different tools that are out there. We use all the other tools ourselves and so MANTLE fits in its place in that flow.
George Seffers
Okay, super. Now with automation there are some concerns around training. How would this affect our ability to train?
Will Lester
Yeah, that's another really good one. So a lot of people come up and say, hey I have folks that, you know, their job is to make sure these systems are configured. If they stop doing the work how are we going to, you know, to maintain our readiness if we don't have a tool like MANTLE. That's a really valid question and we've seen a couple of different responses. One of them is the production gear that you want to use a lot of times gets configured and then they say let's put it over somewhere else and nobody touch it, nobody breathe on it and it's kind of a holy sacred thing. And then you're not able to train on that gear because nobody wants to break something right before the exercise. With MANTLE what you're able to do is configure those kits and then bang on them and use them for different training purposes and then wipe them again and start fresh. So you don't have to treat it quite as preciously through the process. The other thing that's kind of interesting that we never expected customers to do is to go in and actually break stuff on purpose with MANTLE and then give those broken systems to folks and say - hey it's broken, fix it - and they can come up in lab type environments where they can rebuild those and wipe them and start over and even introduce errors on purpose for the purposes of training. So doesn't prevent your folks from getting the skills and keeping themselves up to speed.
Dominic Perez
Yeah, I think it opens up a whole new way of thinking about training when the systems are no longer precious and they can be, you know, redeployed. It's a lot like the revolution that we saw in virtualization a decade or two ago that, and I was a software test engineer at the time, so it was revolutionary. I didn't have to wait to re-image or ghost, if you guys remember that, you could just hit the revert button and run that test case over and over and over again. And today you can hit that revert button essentially on a piece of hardware and all the virtual machines or containers that are contained within that. So it's a different way to think about it. And I would argue that, you know, yes you have people whose job it is to do this today. They are still incredibly important. Someone has to figure out what that desired state is but with MANTLE you no longer have to have that person sit there and click next for three hours. And I've been that person. I've forgotten to put a DNS server up when I'm installing vCenter and waited two hours for it to tell me that nothing happened and I had to start all over again. So it's, again, it's freeing up your best people to do their best work.
George Seffers
Okay, thank you both. The next person is asking what additional training is required to run something such as MANTLE.
Will Lester
We've seen folks up and running with MANTLE in as little as 20 minutes. It's not super difficult once they start to understand and wrap their brains around this new concept of being able to model their desired state and that's really the process that takes the longest. Once they get comfortable with that it's just a matter of importing those and making tweaks and pushing big red buttons to get the outcome that you want. So it's not too much difficulty in terms of training. Like I said, we gave it to Dom's folks and they figured it out with without any, so not super difficult. People like me can use it. I think our fastest deployment of any data center was a four node deployment done by an army colonel in like 30 something minutes. So he had never used MANTLE, never configured anything before technical and was able to set our land speed record if you will.
Dominic Perez
Oh well, that gives me an idea for a contest. We can have a PacStar/NTS leaderboard about who can deploy a system faster.
Will Lester
Yes indeed. We've done it a bunch. If there was a Guinness world record for the most vSAN deployments I'm pretty sure we're close.
George Seffers
Okay, good. And what are the dependencies for MANTLE to work in a disconnected state.
Will Lester
So there are none. That's the great thing about the design that we put together for MANTLE. So we don't - when we built it, we built it with disconnected state in mind so there's no upstream dependency, there's no additional repositories or anything. All the installation materials, all the raw materials of the installs all live on MANTLE so there's no pulling those from other services or even Cloud. You know, we go so far as to make sure that all of the fonts for the user interface are there locally so you're not expecting they can reach out to a CDN and pull stuff from Google. So it's all meant to be disconnected so no dependencies there.
Dominic Perez
I think the one thing that might be obvious to Will but that might be worth pointing out is that you want MANTLE to be running adjacent to the system that you are configuring, whether that's on a separate 451 like he had shown, whether it's on a laptop but you don't want MANTLE running on the system that is being configured by MANTLE. As cool as that would be when you figure that out let me know.
Will Lester
Well, it's a little bit of inception if you do that.
Dominic Perez
Yes.
George Seffers
Okay, does MANTLE need IPMI?
Will Lester
It does not. So MANTLE operates at even lower levels than those tools, some of the out-of-band management stuff you might be comfortable with things like IPMI or iDRAC, something like that. We intentionally went to the lowest common denominator with some of the console and BIOS level automation so you don't need IPMI. We are getting some of the new fancy gear from Dom's team that has IPMI and are working on integrations to add that to the supported capabilities but today you don't need IPMI in order to MANTLE things.
George Seffers
Okay. Next question. Does the MANTLE only work with PacStar servers?
Will Lester
Are there any others?
Dominic Perez
Good answer. You get to keep the demo gear a little while longer.
Will Lester
Yeah, I get to keep it for another day. No, we support all the different vendors that you might be familiar with, both from a software and hardware perspective. So sorry Dom, it's not exclusive.
Dominic Perez
All right ,we'll just do it best.
George Seffers
Okay, good. What version of IQ-Core does it have to work with?
Dominic Perez
And that's an interesting question because there is not today a tight coupling between MANTLE and IQ-Core. So IQ-Core can work with any version of MANTLE and MANTLE can work with any version of IQ-Core and typically I think today where the deployment model would be like is, you set up your IQ-Core VM as an OVA and you put that into MANTLE and that could be any version of IQ-Core, whether that's the, you know, the NCM, you know, kind of base edition or the ROAM, you can have that set up and MANTLE will deploy that for you.
Will Lester
Yeah, no limits on our side.
George Seffers
Okay, and which source for software patches do you link to MANTLE, for example for WSUS, MSI etc, GitHub or FileShare?
Will Lester
So we don't have a tight coupling, we use a word that Don just used, with any of those services, so there's no connectivity upstream to any of the WSUS or GitHub, GitLab anything like that. That is an interesting option if you have a use case there, feel free to reach out or give us some additional color on that. But the patching and management of systems is not where we focus. We focus on treating them as disposable and ephemeral so there isn't really any patching on our side. We'll just wipe it and start over with the latest version which is typically how we approach that.
George Seffers
Okay, thanks Will and last question, perfect timing. How quickly can you deploy one of the MVCs with eight servers?
Will Lester
How quickly, always the fast question.
Dominic Perez
Going back to the leaderboard.
Will Lester
I know, right? I gotta have to ratio on this one. So one of the nice things about MANTLE I mentioned earlier is that parallelization, if that's even a word, being able to do multiple things at once and so all of our, you know, for example hypervisor installs are done in parallel so there really isn't any additional time from a three node to an eight node. There might be a little bit in there for things to switch over but there isn't really a big difference between the two. It might take a little bit longer for your vSAN cluster to coalesce but by and large the limiting factor is the time it takes to install the software, the hypervisor and the vCenter level and those are the same whether it's, you know, one server or twelve.
George Seffers
Okay, thank you very much. I thank both of you for a great presentation and thanks to Curtiss-Wright and NTS for sponsoring today's event. And thank you to our viewers for submitting questions. We certainly appreciate your time. Please note that there are some resources available to you under the resources tab. And to learn more about Curtiss-Wright and NTS you can of course visit their websites. I'd also like to point out that you can link to the archived version of this webinar and see previous SIGNAL webinars on AFCEA's SIGNAL Media website. That concludes our SIGNAL Media webinar for today. Again we thank all of you for being with us. Have a great day everybody.
Will Lester
Thanks everyone.
Dominic Perez
Thank you.