Demonstrating how system software work in a visual manner is an interesting problem, especially in embedded space. There is no UI or visual effects to WOW the audience. To evaluate the value system software components bring to an embedded design, the customer usually needs to configure our software on his embedded development board. This provides for comprehensive evaluation but can be a significant effort. . We run into this very often at Datalight since our primary products (FlashFX Pro and Reliance) are system level embedded software. One way we have attempted to demonstrate our software is to make a full-functional version of our software run on user’s desktop PC. This allows the customer to run the software without any special hardware in matter of minutes and understand the core working of our software. Once they believe there is value, the can request source code access and try it on the actual embedded hardware.

The first demonstration that we have built is for Datalight Reliance. This demo is available to all users who are registered for MyDatalight account. The video below shows how you can use this software on any Windows PC and understand how Reliance unique transactional design allows for 100% reliability against data corruption and how Dynamic Transaction Point technology allows developers to tune their file system even while the device is running.

Image via Wikipedia

Being in the flash memory management space for 15+ years, a very high number of our customers use our products on flash memory (NAND, NOR, NAND controllers, Fusion flash like Samsung OneNAND, etc). Now FlashFX Pro is designed only for flash memory but Datalight Reliance is a file system that works on all block devices. This includes hard drives, USB flash drives, removable cards like SD, CF, solid state drives (SSD), etc. The advantage Reliance brings to these devices is of reliability against data corruption, fast mount times and fast I/O throughput. It also mandates certain requirements on the physical media to ensure reliability against data corruption. We have had customers use Reliance on hard drives before and I want to share some requirements for Reliance to provide high reliability on rotating media. This post is specific to Linux but the general concepts should be applicable to all OSes.

Reliance is a transactional file system and at each transaction point it flushes all its internal caches and commits the data to disk in atomic operations. Primary requirement for Reliance to function on hard drives is that the hardware and the ATA driver must support the “FLUSH CACHE” command. The Linux IDE disk driver checks bits 12 and 13 of word 83 in the IDENTIFY DEVICE information to determine whether FLUSH CACHE is supported.  These bits are defined by the ATA-6 specification, and are not set in earlier drives.  The IDE disk driver will report whether it has detected this capability in a drive.  This is available in the system log.  A typical message will look like:

Jun  9 09:49:23 billr-qa kernel: [   18.621740] hda: cache flushes supported

Since there are a vast number of hard disks on the market and new ones are constantly being introduced (and old ones discontinued), it is a little difficult for Datalight to qualify all hard drives and recommend a specific one. Generally any disk that conforms to the ATA-6 specification and reports that it supports FLUSH CACHE should work correctly with Reliance.  Reliance reports whether it is able to use flush to ensure correct operation, the system log typically looks like this:

Jun  9 09:52:44 billr-qa kernel: [  240.283463] relfs: block device supports flush.

If this message appears in the log, Reliance should operate correctly when power is interrupted unexpectedly.

Datalight’s power interruption testing has been performed on a Western Digital AC29100D using kernel version 2.6.21.1

If you have any questions on the FLUSH CACHE on an OS other than Linux, please leave a comment.

Flash memory has established itself as the technology of choice for device data storage on embedded devices. The advantages it brings in terms of storage capacity, I/O throughput, power consumption and board space savings are significant. In 2007, flash memory was a $7.7 billion industry. Analysts predict a 23% growth of the flash memory market between 2007 and 20111; surpassing the history-making growth of DRAM ten times over.

One of the barriers to flash memory adoption is its perceived complexity of integration into a product design. With the flash memory market branching to multiple product lines beyond traditional NAND and NOR devices, this perception, along with a concern about the reliability of flash, is becoming magnified. Basic flash management software can lessen the complexity of integration, and sophisticated flash software can ensure the optimum lifetime and reliability of a flash device.

The challenges of integrating flash memory are broad, including operations from the seemingly simple – like reading, writing, and overwriting data – to the exceedingly complex – such as bad block management and wear-leveling. When flash memory is not accompanied by an intelligent software manager, the system will suffer from slow reads and writes, data corruption, and a short usable life.

There has been a lot of interpretations for the term “Flash File System”. Some consider it as the combination of flash management software and a block file system. For some it is just the flash management piece. The following diagram shows the different layers involved in managing data on flash memory and the corresponding terminologies for software components

· Flash Driver: Basic software that provides rudimentary read/write access to flash; this software is often acquired from the chip provider, and is usually part-specific.

· Flash Manager: In addition to the functionality of a flash driver, a flash manager also intelligently determines which part is being used, and handles it accordingly – whether it is NAND, NOR, or a fusion of the two (i.e. Samsung OneNAND, or Spansion ORNAND). Bad block management, wear-leveling, garbage collection, and error detection and correction are features that a flash manager provides. A flash manager may also be designed to take advantage of unique performance or technical characteristics a specific part provides. Flash managers are sometimes referred to as FTL (flash translation layers).

· Flash File System: Contains the flash driver and the flash manager aspects, but also incorporates a file system that is designed for use with flash memory. In the way of performance optimizations, a flash file system includes a discard interface which ensures that erased blocks are immediately available for use by both the file system and the flash manager without additional queries to those blocks.

The following diagram shows the flash file system in perspective of an embedded device

Hope this post was useful in understanding the layers of flash management. In the next post in the series, we will look at various flash file systems for one of the most talked-about embedded OS – Linux.