Ever since we announced our high performance file system Reliance Nitro, we have been getting questions on how it compares to the original Reliance file system. Below is a quick-reference table noting some of the differences between the two. For a more detailed comparison (including performance benchmarks), please contact us.

Attributes	Reliance	Reliance Nitro	Recommendation
High performance on large number of files  (100+)		√	If your device stores a large number of files in a single directory, Nitro will perform much faster than Reliance.
High performance on large files		√	Nitro’s extent based design allows it to perform faster on larger files. For sake of this comparison, files can be considered large if they are 10+ times the block size of the device
Frequent transaction points		√	Nitro introduces a new structure called Delta transactions which speed up the time taken to conduct transaction points. Depending on how often you conduct transactions points, Nitro can provide significant advantage
Random I/O performance most critical	√	√	Reliance’s block based design provides an advantage on random I/O on small files. On large files both Reliance and Nitro perform equally well on this metric
Sequential I/O  performance most critical		√	Nitro outperforms Reliance on sequential I/O due to its extent based design
Support for Windows Mobile		√	FlashFX Pro 4.0 for Windows Mobile enables a new discard interface that allows Nitro to have much faster write speeds on flash memory
File-size limit	32-bit	64-bit	Nitro uses 64-bit variables for file size limits allowing for very large file sizes.
Read-only version	√		Reliance currently provides a read-only version called Reliance Reader. Nitro currently does not provide a reader application – this is scheduled for v2

With the release of our new file system this week, Reliance Nitro, we asked our Account Managers what they liked most about our new product. Their answers of course included reliability and high performance. Wes Johns and Phillip Allison were so excited they decided to make a video…  watch the youtube video

We’re totally psyched about Reliance Nitro, our newest file system (yes, we’re file geeks), and we’re always on the lookout for opportunities to show off the performance and reliability attributes it adds to Windows Mobile and Windows CE. When we discovered the relatively-new Beagle Board, it occurred to us that a small, low-cost platform might be just the thing to demonstrate Nitro’s amazing benefits. As you’ve probably heard, the Beagle is making waves with its low cost (around $150) and diminutive size. It uses an OMAP 3530 processor and 256MB of NAND. Though they are most commonly used with Linux, we lucked out in having a partner (MPC Data) who has already developed a Windows CE BSP for it. After a few phone calls, the wizards at MPC Data were able to develop a slick video playback demo app, and presto, the Reliance Nitro Beagle Demo was born! Amateur videographers that we are (ok, REALLY amateur), we recently videotaped John Burnham, who has been working on this project on the Datalight side (and who is a really good sport, btw) showing what happens when power is interrupted during a file write and the extra reliability factor of Reliance Nitro on Windows CE. Be sure to check it out here.

Datalight Reliance includes unique technology called Dynamic Transaction Point ™ which provides the flexibility and control device manufacturers need to tune the performance of their device. It enables multiple configurations that can run simultaneously to provide scenario-specific performance optimization. To highlight this technology, we have added a section to Datalight’s website to describe some common embedded devices and the corresponding file system tuning attributes for each of them

www.datalight.com/filesystemtuning

SanDisk recently announced that it is end-of-lifing several flash parts in the mDOC family and OEMs who were using these parts in their devices are now looking at viable alternatives.  In order to help these affected customers make informed decision, we have published a new whitepaper on options for migrating from Sandisk mDOC flash family.

The paper is available at http://www.datalight.com/mdocwhitepaper/

There are two possible configurations in how boot code might be stored on a device

1. Boot code is stored in raw flash (no file system) and directly accessed from bootloader
2. Boot code is stored on a Reliance formatted flash volume

Option 1: Raw flash
If the boot image is being stored in RAW flash outside the file system, then the only way to be able to ensure that you got an update without damaging the original would be to reserve extra RAW space such that you could simultaneously have two boot images. The bootloader now needs to be able to switch between them and/or locate both of them The process of updating the boot image to a new location would include erasing the old image after updating the new, and having some sort of checksum to ensure the image was intact in case both were still there.

In this case, there would be no really good way to protect the update of the file to that exact same location without compromising the boot image itself. Many customers still use this way to store their boot images, but of course this means that they can’t take advantage of disabling transactions, atomically updating the boot image, and then doing a single transaction to commit all (or none) of the changes.

Option 2: Reliance
In this case, customer would not have a bootloader that checked a physical location for a boot image – they would have a bootloader that opened a file in the Reliance file system at boot time instead, if they were using a file system. Datalight Reliance comes with an utility called “Datalight Loader” which includes a lightweight Reliance reader. This utility integrates seamlessly in your bootloader code and allows the bootloader to mount and read Reliance partitions. Since the bootloader is capable of “reading” a Reliance disk, it doesn’t care where in the file system Reliance stores the file – it just opens the file, and loads it.

In this mode, while updating the boot image, the update utility disables all transactions and initiates the boot image update. Reliance never overwrites live data and hence this new boot code is written to a free-area of the flash. Once the entire boot image code is written, the bootloader calls for a manual transaction event, in which we update the metaroots to point to the new boot code area as the committed area. Old boot code area is now marked as free and can be used for future operations.

If power loss occurs during this replacement process, the device still boots back using the previous boot image, which was never modified

Demonstrating how system software work in a visual manner is an interesting problem, especially in embedded space. There is no UI or visual effects to WOW the audience. To evaluate the value system software components bring to an embedded design, the customer usually needs to configure our software on his embedded development board. This provides for comprehensive evaluation but can be a significant effort. . We run into this very often at Datalight since our primary products (FlashFX Pro and Reliance) are system level embedded software. One way we have attempted to demonstrate our software is to make a full-functional version of our software run on user’s desktop PC. This allows the customer to run the software without any special hardware in matter of minutes and understand the core working of our software. Once they believe there is value, the can request source code access and try it on the actual embedded hardware.

The first demonstration that we have built is for Datalight Reliance. This demo is available to all users who are registered for MyDatalight account. The video below shows how you can use this software on any Windows PC and understand how Reliance unique transactional design allows for 100% reliability against data corruption and how Dynamic Transaction Point technology allows developers to tune their file system even while the device is running.

Being in the flash memory management space for 15+ years, a very high number of our customers use our products on flash memory (NAND, NOR, NAND controllers, Fusion flash like Samsung OneNAND, etc). Now FlashFX Pro is designed only for flash memory but Datalight Reliance is a file system that works on all block devices. This includes hard drives, USB flash drives, removable cards like SD, CF, solid state drives (SSD), etc. The advantage Reliance brings to these devices is of reliability against data corruption, fast mount times and fast I/O throughput. It also mandates certain requirements on the physical media to ensure reliability against data corruption. We have had customers use Reliance on hard drives before and I want to share some requirements for Reliance to provide high reliability on rotating media. This post is specific to Linux but the general concepts should be applicable to all OSes.

Reliance is a transactional file system and at each transaction point it flushes all its internal caches and commits the data to disk in atomic operations. Primary requirement for Reliance to function on hard drives is that the hardware and the ATA driver must support the “FLUSH CACHE” command. The Linux IDE disk driver checks bits 12 and 13 of word 83 in the IDENTIFY DEVICE information to determine whether FLUSH CACHE is supported.  These bits are defined by the ATA-6 specification, and are not set in earlier drives.  The IDE disk driver will report whether it has detected this capability in a drive.  This is available in the system log.  A typical message will look like:

Jun  9 09:49:23 billr-qa kernel: [   18.621740] hda: cache flushes supported

Since there are a vast number of hard disks on the market and new ones are constantly being introduced (and old ones discontinued), it is a little difficult for Datalight to qualify all hard drives and recommend a specific one. Generally any disk that conforms to the ATA-6 specification and reports that it supports FLUSH CACHE should work correctly with Reliance.  Reliance reports whether it is able to use flush to ensure correct operation, the system log typically looks like this:

Jun  9 09:52:44 billr-qa kernel: [  240.283463] relfs: block device supports flush.

If this message appears in the log, Reliance should operate correctly when power is interrupted unexpectedly.

Datalight’s power interruption testing has been performed on a Western Digital AC29100D using kernel version 2.6.21.1

If you have any questions on the FLUSH CACHE on an OS other than Linux, please leave a comment.

The read, write and erase timing characteristics of flash hardware specifications are useful for comparing different products, but don’t tell the whole story about what you will get from your real-world devices. When Flash memory is incorporated into a system, the performance of the system depends on a number of factors. One key factor that can reduce the effective performance of flash memory involves the shared bus topology of your system. Optimal flash performance depends on the speed and availability of the bus that connects the flash to the system. Also critical are the manner in which the operating system handles interrupts and whether the flash device is connected to the system’s interrupt architecture.

The published read, write, and erase timing characteristics of flash hardware specifications are useful for comparing different products, but don’t tell the whole story about what you will get from your real-world devices. When Flash memory is incorporated into a system, the performance of the system depends on a number of factors in addition to the capabilities of the flash hardware.

One key factor that can reduce the effective performance of flash memory involves the shared bus topology of your system. Optimal flash performance depends on the speed and availability of the bus that connects the flash to the system. For example, if your flash shares a bus with parts that operate at slower clock speeds, the timing of the accesses to the flash part may be extended to match. On the other hand, your flash part may be competing for bus availability with other demanding high-speed system components.

RAM memory, network interfaces, and LCD screens are demanding components that can compete with flash for bus and CPU bandwidth. The use of certain features of the processor and operating system, such as DMA and caching, can have a similar impact. As more components, peripherals, and device drivers are added to the system, more opportunities arise for the bus to be shared. The proliferation of high performance audio and video features, now common on mobile devices, can further tax a shared bus system on a general purpose chipset. For this reason special-purpose chipsets designed for a specific application, as well as tuning the characteristics of your flash management software to meet your specific needs, will generally enable higher levels of flash performance.

Well designed hardware bus topology can alleviate the issue of shared bus contention, yet other factors may still impact flash memory performance. Even if the flash part has full speed access to the processor’s external bus, the availability of the CPU to service that bus is still a question. Bus arbitration may take CPU cycles away from the flash bus in favor of other system busses or internal accesses. Operating system timer interrupts and other peripheral device driver interrupts can interfere with flash software operations, as can a CPU that is simply overloaded by running complex applications.

Also critical are the manner in which the operating system handles interrupts and whether the flash device is connected to the system’s interrupt architecture. Some flash is connected to processors in such a way that the signal generated by the flash is connected to a GPIO, or not connected at all. This may have little impact on flash performance, but it will limit the ability of the CPU to execute other flash-related software, such as garbage collection, or even unrelated tasks. Additionally, many systems have an explicit or implied interrupt priority that must be considered at the system level. Responsiveness requirements of all interrupt-driven components in the system must be carefully weighed against the desire to maximize flash performance.

An equally significant factor affecting flash performance that might be easily overlooked is the flash management software itself. There is a necessary amount of overhead inherent in running software to manage your flash memory, and there are some complex operations that the software needs to accomplish well in order to optimize flash performance. The software provided by your flash vendor may or may not provide satisfactory performance for your particular application.

While flash memory often appears to the end user like a virtual hard drive, the underlying technology is quite different and presents certain challenges. Flash management software can do more than bad block management and wear leveling, it can increase the effective performance of the flash part by addressing these challenges:

1. Flash performance can be impeded by the need for a slow erase operation before writing new data, but software that intelligently performs background garbage collection during idle time can solve that problem.
2. Fragmented data can degrade performance in applications such as streaming media from NAND memory, but compaction software that de-fragments the data can improve performance in these situations.
3. With some algorithms, throughput is maximized for performance until a percentage of the flash memory is used, at which point performance can degrade. The percentage of the flash that is used before performance suffers can be tuned in some implementations, by allowing the system designer to reserve a specified amount of ‘cushion’ of unused memory.
4. In some solutions, maintenance operations such as garbage collection can preempt high-priority read requests. Implementations that make careful use of multithreading operating systems’ capabilities to manage this issue can reduce read latency by orders of magnitude.

Several factors will affect the performance of flash memory in your real-world system, some of which may be beyond your control. Chipset hardware and system bus topology decisions may have been made already. No matter whether your hardware is specially designed for your application or you are using a general-purpose hardware design, though, the effective performance of your flash memory can be improved through software methods. Datalight FlashFX is a multithreading memory management software solution that enables garbage collection, data compaction, memory cushion, and high priority read interrupts to allow the highest real-world flash performance your hardware configuration can support.

Image via Wikipedia

Linux has been slowly but surely establishing itself as the predominant OS in the embedded industry. ABI research report suggested that 23% of Smartphones will be based on Linux by 2013. High-profile industry support from Android and the LiMo foundation has put the spotlight back on embedded Linux.

In a previous post, we talked about flash memory and the various layers of flash management. In this post, I will talk about JFFS2, the most popular flash file systems available on the Linux platform

JFFS2

The Journaling Flash File System version 2 (JFFS2) is a log-structured file system that was originally designed in 1999. The original JFFS was developed by Axis Communications (and later enhanced by Red Hat) to provide support for NOR flash devices. The current version has been updated to include support for NAND flash. JFFS2 is open-source software, distributed under the terms of GPL license.

 

JFFS2 Strengths

 

1. Portability to Development Environments: Included with the Linux kernel since version 2.4.10, JFFS2 has become a de facto standard flash file system for Linux developers. Today, it is included in most commercial Linux distributions (such as MontaVista and Wind River Linux). Because of this wide distribution and use, it has been integrated into many varying environments and is known to be relatively easy to build.

2. Reliability: As changes are made to the file system, a “log” is built; this log provides information about where a file and its associated metadata are located on the flash chip.^[1] As the log is consistently maintained, it will be read back in the event of an unexpected power loss to determine the location of a missing file. Although the log structure provides a level of data reliability, this is accomplished at a cost to performance

3. Support for disk-wide compression: The benefit, or cost, of using compression depends on each specific use case. Compression will be useful in making efficient use of disk space when several text-only or code data (OS, etc.) files are being stored. Media files are already compressed (in *.jpg, *.mpg, or other formats), so the time used to attempt data compression is wasted. It is even possible that media files may take up more space after an unnecessary compression than originally needed. Disk usage and performance for the type and number of files to be maintained must be considered by the device designer in order to determine whether compression will yield a benefit or not

JFFS2 Shortcomings

1. Resource usage: RAM usage by JFFS2 increases in linear proportion to the number of nodes. Hence on large flash volumes, the system resources required by JFFS2 can be very significant

2. System performance: For devices that primarily use the file system for read operations, JFFS2 performance may be acceptable. However, for multi-functional devices whose applications perform a continuous mix of read and write operations, it is likely that the performance of a system using JFFS2 will not pass a rigorous standard. In addition to slow writes, the flash disk mount times of JFFS2 are exceptionally slow and worsen as the amount of data stored increases. Upon start-up after an unexpected power failure, JFFS2 must reconstruct the file system structure from the log. This is a costly operation that requires several seconds – more for volumes that are large or near-capacity. The device will be halted during this check operation, as any data that is stored on the disk will not be ready for use until the start-up completes

Informative links on JFFS2

1. JFFS2 Technical paper [PDF]

2. JFFS2 RAM usage [ppt] – Presentation at Embedded Linux Conf 2007

In the next post in this series, we will look at another popular Linux flash file system – YAFFS.

---

[1] Details on JFFS2 log structure can be found here http://sourceware.org/jffs2/jffs2-slides-transformed.pdf