Before embedded devices, file systems were designed to work in servers and desktops. Power loss was an infrequent occurrence, so little consideration was given to protecting the data. Frequent checks of the file system structures were important, and were often handled at system startup by a program such as chkdsk (for FAT) or fsck (for Linux file systems). In each case, the OS could also request a run of these utilities when an inconsistency is detected, or when the power was interrupted.

The method behind these tools is a check of the entire disk – reading each block to determine if it is allocated for use, then cross checking with an allocated list located elsewhere on the media. FAT file systems have little other protection, and can only flag sections of the media without matching metadata by creating a CHK file for later user analysis. Linux file systems add in a journal mechanism to detect which files are affected, and can often correct the damage without user intervention.

These utilities are necessary because these basic file systems are not atomic in nature – data and metadata are written separately. Datalight’s Reliance Nitro file system treats updates as a single operation, and thus the file system is never in a state where it would need to be corrected. Our Dynamic Transaction Point technology allows the user to customize just how atomic their design is, protecting not just a block of data and metadata but the whole file – half a JPG is pretty much useless, from a user perspective.

The repairs that fsck and chkdsk can perform are completely unnecessary with the Reliance Nitro file system. At the device design level, this results in quicker boot times for a system that is completely protected from power failure. A file system checker is of course provided, and is useful for detecting failures caused by media corruption.

Taking chkdsk and fsk to the next level of protection would be a tool to repair some media corruption. If a block of data on the media becomes only partially readable, this tool could read it multiple times (to try and collect the most data) and store the results in a newly allocated block, correcting the file system structures appropriately. User intervention would likely be required to understand if enough data was recovered to make this effort worthwhile. Stay tuned for more updates on this topic.

Learn more about Dynamic Transaction Point technology

Lately you may have noticed a lot of talk about eMMC on our blog and website. If so you may be thinking, “Why is Datalight so excited about eMMC?” Here are 5 top reasons we’re jumping on the eMMC bandwagon:

#5) Standards: Having a standard specification for hardware like eMMC enables our customers to spend less time integrating new hardware and more time focusing on their core competencies.

#4) ECC: As NAND flash lithography shrinks and density increases, error correction has become exponentially more complex and therefore difficult to do in software. Performing ECC in the hardware, like eMMC (or OnFI EZ-NAND parts like Micron’s ClearNAND), overcomes this problem and offers much simpler integration.

#3) Performance: In combining an accepted standard with on-board ECC, eMMC offers excellent performance and flexibility for customers.

#2) Advanced features like secure delete: With more critical data being stored, data security is becoming a much bigger issue. We’re happy to see a standards body like JEDEC focus on tackling this problem. And since customers will need a file system capable of enabling this feature, we’re delighted to be the first to offer support for eMMC 4.41 features like secure erase and secure trim.

#1) Customers: More and more of our customers are either evaluating or using eMMC. Whenever we see our customers using a new technology we ask ourselves how we can help enable them to take full advantage of it. Exploiting the features of eMMC at the file system level is a win for all of us.

Learn more about Datalight flash management solutions

It’s always gratifying when you run benchmarks and discover your product actually does outperform the competition. Months and months of development effort went in to making Reliance Nitro and FlashFX Tera run flawlessly in an open source environment. We were pretty sure our transactional architecture beat the pants off YAFFS2, JFFS2, and UBIFS, but until you run the final benchmarks, you really don’t know for certain. Recently we ran tests on two platforms, a ConnectCore Wi-i.MX51 (Cortex-A8) and an NVidia Tegra 2 (Cortex ARM9). The Flash part used for all tests was a Samsung 512 MB part. The specific test used was IOZone, with a specified file size sufficient to be larger than the Linux cache, in order to better reflect the raw throughput. The results speak for themselves:

Also see an article weighing the pros and cons of JFFS2

We’ve received a bit of feedback on our Bootstrapping Linux from NAND Flash with FlashFX Tera and Reliance Nitro whitepaper. Questions that have come up regarding this whitepaper include: “The sample project included in the whitepaper demonstrated booting from NOR rather than NAND, is it possible for the bootloader to reside in NAND?” and “Is booting from NAND reliable?”

I talked with Datalight Software Architect Bill Roman, the author of the whitepaper, to answer these questions…

Question: The sample project included in the whitepaper demonstrated booting from NOR rather than NAND, is it possible for the bootloader to reside in NAND?

Bill:

Yes, you can boot from NAND. The sample for the whitepaper was done on NOR partly so the example would be more simple and thus easier to understand, and because of the functionality of the board that we were working with. The main topic was how to read the Linux kernel from a Reliance Nitro root file system on flash managed by FlashFX Tera. How the bootloader initially gets loaded into RAM is an implementation detail. Some platforms make booting directly from NAND easy, while some can’t do this at all, and thus require some NOR flash.

To boot from NAND there are two stages required. The details of the first-stage loader are very platform-specific. Typically, the processor manufacturer would provide documentation and help to do this –this is why the focus for the whitepaper was on the second stage.

Question: Is booting from NAND reliable?

Bill:

Loading the initial bootloader from NAND can be made quite reliable. Most NAND manufacturers guarantee the first erase block to be entirely error-free for some limited number of erase cycles. This allows an unchanging first-stage loader to be stored there, which then typically finds and loads the second stage loader (the main topic of the white paper).

If you’d like to check out the whitepaper, download here:

Recently we developed a sample project to demonstrate how to boot Linux from flash on an embedded system.  Booting NAND with a single (root) file system can be a challenge, even when using a NOR chip to initialize as this project did. So how did we do it? In order to simplify the process, we loaded our Datalight bootloader into the NOR flash and programmed the monitor that comes with the board to jump directly to it. So if you use the sample project with a Freescale i.MX31, a single command starts the loader and allows it to load the kernel and the initial RAM disk from flash managed by FlashFX Tera with Reliance Nitro. It will then read them into memory, jump to the kernel, and the kernel will start running code from the initial RAM disk which loads the FlashFX Tera and Reliance Nitro drivers into the kernel. Next, it mounts the flash as the root file system and the system is powered up and running. Within about ten seconds your i.MX31 is up and running with FlashFX Tera running the flash and Reliance Nitro running as the root file system. If you’d like to try it for yourself, we’ve included sample code for this project with our latest product release. In addition, our sample loader could just as easily be loaded into memory other ways.

Bothell, WA, – May 5, 2010 – Today Datalight announced support for Linux kernel versions up to 2.6.33, the most recently released Linux versions available. FlashFX Tera, the file-system independent flash memory manager and Reliance Nitro, the highly-reliable, high-performance file system offer much faster mount times than UBIFS, YAFFS, or JFFS2. In addition, the Datalight products boost write speed over the standard file systems and provide out-of-the-box support for over 300 different flash memory parts from all the leading suppliers. Linux is finding its way into more devices such as smart phones, automotive infotainment, and industrial equipment which require both responsiveness and 100% data reliability.

“With the growth in adoption of Linux for data-intensive embedded devices, OEMs need a flash file system that better supports their reliability and performance requirements.” said Roy Sherrill, Datalight CEO. “By supporting the most recent kernel versions of Linux we’re filling that gap in the market with a robust, commercial-grade solution backed by our reputation for responsive, high-quality support.”

FlashFX Tera supports the full range of flash technologies including NAND, NOR, and MLC NAND flash in a single driver. Its patented wear-leveling and bad block management extend the useful life of devices using flash. While FlashFX Tera can be used with virtually any file system, pairing it with Reliance Nitro provides an optimized data storage software stack to simplify system development.

FlashFX Tera 1.2 and Reliance Nitro 1.2 are available immediately from Datalight and the Datalight worldwide network of channel partners. Please visit us at http://www.datalight.com/partners/worldwide-sales-partners to find a reseller near you.

The Reliance family of file systems and FlashFX family of flash media managers comprise the Datalight flash file system solution. Reliance was designed from the ground up for high reliability applications. Dynamic Transaction Point™ technology gives developers full control over performance and data protection characteristics, protecting users from file system corruption, even after unexpected system interruption. Embedded applications can benefit from faster boot times that remain consistent for the life of the product, regardless of disk size. FlashFX™ Tera features pre-written support for over 300 flash parts, works with virtually any NAND controller, and features wear leveling, bad block management, and background compaction for unrivaled performance.

For information, contact:
Kerri McConnell, Director of Marketing
425.686.1069
kerri.mcconnell@datalight.com

Bothell, WA, – March 18, 2010 – Datalight announces support for Linux kernel versions up to 2.6.29 with new versions of FlashFX Tera, the file-system independent flash memory driver and Reliance Nitro, the highly-reliable, high-performance file system. FlashFX Tera version 1.2 offers out-of-the-box support for over 300 different flash memory parts from all the leading suppliers, expanding the choice for OEMs using flash memory. Linux is finding its way into more devices such as smart phones, automotive infotainment, and industrial equipment which require both responsiveness and 100% data reliability, by adding Reliance Nitro 1.2, OEMs can ensure rock-solid reliability of their systems without paying a performance penalty.

“Datalight continues to raise the standard in delivering advanced file system features for MontaVista Linux,” said Dean Misenhimer, Director of Marketing at MontaVista Software. “Our partnership provides our customers the option of a pre-integrated professional flash file system solution to speed their development of flash-based devices. This ultimately leads to faster time to market with superior performance and reliability.”

FlashFX Tera supports the full range of flash technologies including NAND, NOR, and MLC NAND flash in a single driver. Its patented wear-leveling and bad block management extend the useful life of devices using flash. While FlashFX Tera can be used with virtually any file system, pairing it with Reliance Nitro provides an optimized data storage software stack to simplify system development.

Reliance Nitro version 1.2 also adds support for extended file attributes. Most commonly used to designate file permissions, extended attributes are important to customers using special Linux distributions like SMACK. As data quantity and complexity in devices grows, extended file attributes can also be used for storing additional metadata (data about the data), such as digital rights information, GPS location or any information that makes finding and using the data easier. The ability of Reliance Nitro to serve as the root file system for Linux can simplify the data storage stack for these devices by allowing them to operate with a single file system.

“With the growth in adoption of Linux for data-intensive embedded devices, OEMs need a flash file system that better supports their reliability and performance requirements.” said Roy Sherrill, Datalight CEO. “By migrating our flagship products to Linux we’re filling that gap in the market with a robust, commercial-grade solution backed by our reputation for responsive, high-quality support.”

FlashFX Tera 1.2 and Reliance Nitro 1.2 are available immediately from Datalight and the Datalight worldwide network of channel partners. Please visit us at datalight.com to find a reseller near you.

The Reliance family of file systems and FlashFX family of flash media managers comprise the Datalight flash file system solution. Reliance was designed from the ground up for high reliability applications. Dynamic Transaction Point™ technology gives developers full control over performance and data protection characteristics, protecting users from file system corruption, even after unexpected system interruption. Embedded applications can benefit from faster boot times that remain consistent for the life of the product, regardless of disk size. FlashFX™ Tera features pre-written support for over 300 flash parts, works with virtually any NAND controller, and features wear leveling, bad block management, and background compaction for unrivaled performance.

Bothell, Wash., – August 12, 2008 – Datalight announced today that it has released new versions of Reliance™ and FlashFX® Pro, with pre-configured support for  MontaVista® Linux®. The new install experience includes simplified integration within MontaVista DevRocket, an Eclipse-based IDE that streamlines common embedded  development tasks. Dropped into DevRocket, Datalight products build as kernel loadable modules that work with a project’s OS image. Sample bootstrap code is also  supplied for developers who need to integrate the products into a boot loader.

“Developers choose MontaVista Linux for faster time-to-market, integration, and stable, fully tested code. We are pleased that Datalight has extended these benefits at  the flash memory file system level, and to provide embedded Linux developers with added performance and reliability,” said Dan Cauchy, Senior Director of Market Development, MontaVista Software.

Other upgrades include a read-only version of Reliance inside the Datalight Loader. This small footprint version permits a bootloader to load an OS image directly from a reliance partition. Devices benefit from risk-free “in-place OS upgrades” enabled by the application-controlled transaction point feature of Reliance. The new versions also feature enhancements in reliability, as well as support for a wide range of new flash parts. FlashFX Pro now supports Spansion NS-P, Samsung FlexOneNAND, Micron 55nm flash parts, and all CFI-compliant NOR parts. The Datalight flash file system solution is comprised of the Reliance file system and FlashFX Pro intelligent flash media manager. Reliance was designed from the ground up for high reliability applications. Dynamic Transaction Point™ technology provides 100% immunity from file corruption, even after unexpected system interruption. Embedded applications can benefit from faster boot times that remain consistent for the life of the product, regardless of disk size.

FlashFX Pro features pre-written support for over 200 flash parts, works with virtually any NAND controller, and features wear leveling, bad block management, and garbage compaction for unrivaled performance. Datalight flash file system products are also available on other operating systems and integrated development environments.

Image via Wikipedia

Being in the flash memory management space for 15+ years, a very high number of our customers use our products on flash memory (NAND, NOR, NAND controllers, Fusion flash like Samsung OneNAND, etc). Now FlashFX Pro is designed only for flash memory but Datalight Reliance is a file system that works on all block devices. This includes hard drives, USB flash drives, removable cards like SD, CF, solid state drives (SSD), etc. The advantage Reliance brings to these devices is of reliability against data corruption, fast mount times and fast I/O throughput. It also mandates certain requirements on the physical media to ensure reliability against data corruption. We have had customers use Reliance on hard drives before and I want to share some requirements for Reliance to provide high reliability on rotating media. This post is specific to Linux but the general concepts should be applicable to all OSes.

Reliance is a transactional file system and at each transaction point it flushes all its internal caches and commits the data to disk in atomic operations. Primary requirement for Reliance to function on hard drives is that the hardware and the ATA driver must support the “FLUSH CACHE” command. The Linux IDE disk driver checks bits 12 and 13 of word 83 in the IDENTIFY DEVICE information to determine whether FLUSH CACHE is supported.  These bits are defined by the ATA-6 specification, and are not set in earlier drives.  The IDE disk driver will report whether it has detected this capability in a drive.  This is available in the system log.  A typical message will look like:

Jun  9 09:49:23 billr-qa kernel: [   18.621740] hda: cache flushes supported

Since there are a vast number of hard disks on the market and new ones are constantly being introduced (and old ones discontinued), it is a little difficult for Datalight to qualify all hard drives and recommend a specific one. Generally any disk that conforms to the ATA-6 specification and reports that it supports FLUSH CACHE should work correctly with Reliance.  Reliance reports whether it is able to use flush to ensure correct operation, the system log typically looks like this:

Jun  9 09:52:44 billr-qa kernel: [  240.283463] relfs: block device supports flush.

If this message appears in the log, Reliance should operate correctly when power is interrupted unexpectedly.

Datalight’s power interruption testing has been performed on a Western Digital AC29100D using kernel version 2.6.21.1

If you have any questions on the FLUSH CACHE on an OS other than Linux, please leave a comment.

Continuing the conversation started in Flash File Systems and JFFS2 blog posts, this post talks about a YAFFS, another Linux flash file system alternative. YAFFS (Yet Another Flash File System) was designed to solve some of the performance issues suffered by JFFS2 on NAND flash. Later, YAFFS was upgraded (to YAFFS2) to work with modern, high-density NAND flash. Like JFFS2, YAFFS2 is a log-structured flash file system. YAFFS2 is licensed under the GPL for use with Linux; it also can be ported to and licensed for non-GPL environments, if needed

Interesting facts about YAFFS

1. Reliability against data corruption – As a log-structured file system, YAFFS2 is intended to be power-fail safe, though there have been reports of data corruption during the garbage collection process and cases where YAFFS2 has lost directories.

2. Wear Leveling – YAFFS2 only implements dynamic wear leveling. Wear leveling is not performed for static data. This may cause a higher number of blocks to be rendered useless at a faster rate than if both static and dynamic wear-leveling scheme were available. [For more information on static and dynamic wear-leveling, see our whitepaper on the topic at www.Datalight.com/resources].

3. Performance: According to the YAFFS development team, YAFFS2 will perform best on disks that are greater than 64MB, while JFFS2 is still preferred for smaller disks.

For a detailed look at YAFFS, there is a great presentation on YAFFS by Wookey at Embedded Linux Conference 2007.

Checkout our recent Linux Whitepaper