Data at Rest in Deployed Applications
This blog provides a brief overview of solid-state memory types from the perspective of data at rest (DAR) in deployed military applications. Deployed data at rest devices can take the form of direct attached storage (DAS) or network attached storage (NAS). For a comparison of direct attached storage and network attached storage, see the white paper: Choosing the Right Data Storage Solutions for Modern Military Missions.
An example network attached storage device is shown in Figure 1. The example network attached storage supports 10 gigabit Ethernet (10GbE) networks and has multiple terabytes of high-speed, removable memory.
This blog will focus on technical differences between solid-state memory types. It will not address cost directly, which will vary depending on the manufacturer, capacity, and form factor.
Solid-State Memory and Drives
To operate in harsh, rugged environments, deployed network attached storage devices (like the example shown) use solid-state memory instead of rotating drives. There are two types of solid-state memory devices: Serial Advanced Technology Attachment (SATA) and Non-Volatile Memory Express (NVMe). Both types of storage devices use flash memory for the actual storage.
The primary differences between the two options are the front-end interface and the underlying drivers. Both will be briefly discussed. The operating systems use a driver to communicate data with the storage devices.
Flash Memory
Flash memory is a non-volatile memory chip (solid-state). It can be electronically reprogrammed and erased. Flash memory is found in USB flash drives, MP3 players, digital cameras, and solid-state drives (SSD). Flash memory is much more rugged than rotating hard drives. So, it has been used with increasing frequency in rugged, deployed applications as the prices dropped.
A flash memory chip is composed of NOR or NAND gates, but most flash today is NAND flash. There are different write/erase capability levels ranging from 10,000 to 1,000,000 cycles.
The same NAND flash chips are used in both SATA and NVMe drives.
Speed: SATA Interface
Serial ATA (SATA) was conceived in 2000, with the first specification or revision introduced in 2003. It was developed to support storage peripherals with a smaller cable (when compared to Parallel ATA). The initial revision (Rev 1 or Rev 1.0 or Rev I) supported speeds of 150 megabytes per second (150MB/s) transfers and then quickly transitioned to 300 MB/s. The latest SATA revision III supports up to 600 MB/s. The serial nature of SATA limits speeds.
SATA uses 8b/10b encoding, which was developed originally by IBM decades before. Fibre Channel and other communication systems have used 8b/10b as well.
| SATA Revision | Speed (MB/s) | Introduced (Year) |
|---|---|---|
| SATA 1 | 150 | 2003 |
| SATA 2 | 300 | 2004 |
| SATA 3 | 600 | 2009 |
The SATA protocol is Advanced Host Controller Interface (AHCI). Intel originally defined it. It specifies the register-level interface of SATA host controllers. AHCI enables software to communicate with SATA drives. With only one command queue, AHCI is one of the limiting factors for SATA, more on that later.
NOTE: A faster version of SATA known as 3.2 was introduced but did not gain market share. So, it is not included in this brief analysis.
Speed: NVMe Interface
Non-volatile memory express (NVMe) is a newer storage access and transport protocol. It delivers very high throughput and response times. The NVMe protocol accesses flash storage via PCI Express (PCIe) bus, which is a parallel bus that is scalable.
Introduced by Intel, the PCIe bus has been around since the early 2000s. The throughput or transfer speed depends on the PCIe version and the number of lanes used. Version 7.0 is planned and will double the speed of version 6.0. PCIe ‘version’ can also be considered PCIe ‘gen’ or generation. Most people will say ‘PCIe Gen 3.0’.
As you can see in Table 2, the speed of PCIe depends on the number of lanes. Today, the ‘sweet spot’ for performance and value for NVMe drives seems to be ‘PCIe Gen 3.0 x 4’, which means PCIe generation 3 with four lanes. As you can see in the table, that combination provides transfer speeds of 3.94 GB/s. This number will be used for comparison purposes, but that sweet spot for value could move to newer generations with more lanes.
Command Queues and Commands: NVMe vs. AHCI
AHCI has only one command queue and supports only sending 32 commands per queue.
NVMe has 64K command queues, and it supports 64,000 commands per queue.
CPU Cycles: NVMe vs. AHCI
Beyond the number of queues, each AHCI command utilizes more CPU cycles than an NVMe command. While that is not an objective comparison, it does affect overall performance.
Latency: NVMe vs. AHCI
AHCI commands have a latency of around 6 microseconds (µsec). NVMe commands have a latency of 2.8 µsec. Some of this difference arises from the fact that NVMe communicates directly with the system CPU while AHCI must communicate with the SATA controller.
IOPS: NVMe vs. AHCI
There is quite a difference when looking at input/output operations per second (IOPS). AHCI has up to 100,000 IOPS, while NVMe has 1,000,000 IOPS.
Capacity: NVMe vs. SATA
With many different form factors for each option, capacity is difficult to pinpoint. The 2.5” form factor for SATA and the U.2 form factor for NVMe were chosen for simplicity purposes. Both have eight terabytes (8 TB) capacity available.
Another reason for no capacity difference is that both NVMe and SATA use the same NAND Flash chips.
In the future, NVMe will likely slowly pull ahead of SATA for capacity. This is simply because more industry effort and investment will likely be put into NVMe long term. With its higher speed and performance, it is understandable why this may prove true.
Comparison: NVMe vs. SATA
SATA SSDs have been used in deployed systems for nearly two decades. SSDs were perceived to be the holy grail for deployed storage. The rapid growth of SSD deployment began as the price per gigabyte of solid-state storage began to drop.
Storage prices dropped as the commercial markets widely used SSDs. Solid-state storage is now used in phones, televisions, automobiles, thermostats, USB sticks, toasters, refrigerators, microwave ovens, and almost any new device in your home. The laptop on which this blog was written has a SATA SSD inside.
The SATA drives also became available in extended temperature versions. This option allowed the possibility of deployment by the military. A temperature range of -40C to +85C was required.
NVMe drives have followed the same path. They burst onto the commercial scene a few years ago and promised higher performance for laptops and gaming systems. However, COTS manufacturers like Curtiss-Wright had to wait until the NVMe drive became available in extended temperature before using them, like in the example NAS in Figure 1.
The performance advantage is clearly in favor of NVMe drives. When NVMe SSDs use more PCIe lanes, the speed will only increase. As NVMe SSDs support PCI version 4 and above, they will gain more market share.
In addition to the speed difference, the underlying differences between the NVMe and AHCI drivers contribute to the overall superior performance of NVMe.
| NVMe | SATA | Difference | |
|---|---|---|---|
| Max Speed1 | 3,940 MB/s | 600 MB/s | 6.5 |
| Driver | NVMe | AHCI | - |
| Command Queues | 64,000 | 1 | 64,000 |
| Commands per Queue | 64,000 | 32 | 2,000 |
| Latency | 2.8 µsec | 6 µsec | -53% |
| IOPS | 1,000,000 | 100,000 | 10 |
| Capacity2 | 8 TB | 8 TB | - |
Conclusion
SATA drives have been successfully used for decades. With this deployed success, the military is unlikely to abandon SATA any time soon. Those military vehicles with SATA drives designed in and proven are unlikely to make changes for quite a while.
1 Speed compares NVMe PCIe Gen 3 x4 with SATA revision 3.0.
2 2.5” SATA SSDs and U.2 NVMe SSDs used for comparison.
If SATA becomes less popular, the manufacturing cost may increase, accelerating its demise. If manufacturers stop investing in SATA for larger capacities and higher speeds, that may also shorten the life span. Neither scenario is evident as of this writing.
The compound annual growth rate (CAGR) for the solid state drive market has been estimated at 15% from 2022 to 2029 . The market value is expected to grow from $53B to $143B during that period. With that type of market opportunity, you can see why Samsung, Intel, and other companies are investing heavily.
With its superior speed, the NVMe memory share of that overall solid-state market is expected to increase. The military markets will benefit from those new products and increased manufacturing capacity.
Curtiss-Wright has successfully used both SATA and NVMe drives and is busy working on new products to introduce in the future.
i PCI = Peripheral component Interconnect
ii Solid State Drives Market Forecast to 2029 - COVID-19 Impact and Global Analysis By Type; Technology; Storage; End-User and Geography (researchandmarkets.com)