The USB Mass Storage class (also called UMS) is a protocol that allows a device connected through USB (Universal Serial Bus) to become accessible to a host computing device. This allows file transfers between the device and the desktop, as long as the file system used on the device is known to the desktop. One common example of this is the FAT file system. Other file systems can be supported through an external driver. An example of this is a device formatted with Reliance Nitro can be accessed through the Reliance Nitro Windows Driver (RNWD) on a Windows host.

To the host machine, the USB device appears similar to an external hard drive, enabling drag-and-drop file transfers. In fact the host has exclusive access to the media shared through this USB Mass Storage class. The connection happens below the file system layer – the physical media is revealed through this protocol. All transfers are done on a media block basis, and all metadata writes are separate from file data writes.

This can cause a few problems, especially when a transfer is interrupted. To truly prevent corruption, the host computer must release access to the device before it is disconnected. Often the software on the device cannot access the media while any transfer is in operation. One example of this is Android, which warns you that applications and data stored on the SD card will be inaccessible.

Another option which is gaining popularity is the Media Transfer Protocol, or MTP. This is an extension of the Picture Transfer Protocol which was originally developed for cameras. A host computer equipped with MTP will be able to communicate with the device at a layer above the file system with basic packet transfer commands. The unit of storage is a local file rather than a media block. The target device can use whatever file system is required for the device, and the device can access that file system simultaneous with the host computer with no fear of corruption.

In some ways this makes the Media Transfer Protocol rather like a transactional file system – either the entire file is written or nothing is. In most cases, the storage media is not affected by failed transfers, because the protocol on the device side handles this properly. This works especially well with the Reliance Nitro transactional file system. A customer implementing MTP on their device can use the Dynamic Transaction Points of Reliance Nitro to perform a manual transaction after each completed MTP transfer. Best of all, file data and metadata writes can be done at the same time, in an atomic fashion.

In summary, Datalight’s software works with both of these standard protocols. The improvements and advantages of Media Transfer Protocol make it the superior solution for embedded devices.

Learn more about Reliance Nitro

Embedded device designers have to come up with systems that handle variations in network traffic, resource constrained environments, and battery limitations. All that pales in comparison with challenges at system start and power down times.

In powering down a multi-threaded device, each application will want to get state information committed to the media. The operating system will have its own set of files or registries to update before the power goes off. At the low end, the MLC NAND flash or eMMC media requires power to be maintained while writing a block, in order to prevent corruption of that or other blocks on the media. Therefore a large group of block writes arrive at the media level simultaneously, all with the requirement to finish up before the power drops.

The only thing worse than having to perform several writes in a short time frame is when those writes display the worst performance characteristics of eMMC media. These block writes are often located in different locations, essentially defining Random I/O.

When restoring power, the same situation occurs in reverse, with each thread requesting state information. For some file systems, the interrupted writes will need to be validated, the file system checked, or the journal replayed. Here the time pressure is not with actual loss of power, but with frustrated users, who want the device to be on, now!

This is a very important use case to consider for designers of file systems and block device drivers. While shutdown consists of primarily writes, a multithreaded file system should not block reads. The same applies at startup time, where a write from one thread should not block the other read threads. Where possible, the block device driver should sequence the reads and writes to match the high performance characteristics of the media, allowing the most blocks to be accessed in the least amount of time.

We are currently working on an eMMC-specific driver that will manage the sequencing of writes. It promises to improve write performance by many times the rate of standard drivers. Check back with us for more on this topic next month!

If possible, applications should also be written to minimize the startup and shutdown impact. With more applications being written by outside developers, this is getting harder to control. The system designer must focus on what they can control – choice of hardware, device drivers and file system to mitigate these problems.

Learn more about eMMC

If you’ve noticed the numerous posts lately on the Datalight blog regarding JEDEC and eMMC, you might be wondering why we’re so excited about this particular standard. There are many features that this “smarter” memory will enable for OEMs; In this post I’ll focus on one of those features in the eMMC specification –secure delete.

Securely deleting information on flash memory is more complicated than it seems. For one thing, files are constantly being moved around to ensure even wear of the flash, resulting in multiple copies of file data on the media. Furthermore, when a file is marked for delete, it is typically not physically deleted, rather the space is only marked as available to be overwritten. Until that happens, the “deleted” data is still present and recoverable on the media. In fact, the University of California San Diego Non-volatile systems lab has produced an in depth study of file deletion on flash memory, where they found significant data still present on the media even after deleting the files. A copy of the report can be found at: http://cseweb.ucsd.edu/users/swanson/papers/Fast2011SecErase.pdf

In order to securely delete a file on raw flash, you must use a controller that will either track every block where the file has been stored, or will overwrite the space the file was stored in each time it is moved. The latter describes exactly the secure erase and secure trim features found in the eMMC 4.41 standard. This means that the hardware will finally be capable of securely deleting files –brilliant! There is just one problem: Who has software to support this functionality? As of this writing, there is no file system which supports the feature. While an application can make a call to the media to delete a file securely, the file system may have a backup copy stored somewhere. Fact is, the file system must support the secure delete capabilities of the hardware in order for these features to function correctly.

If an OEM wants to take advantage of the secure erase and secure trim features, their application will need to communicate with the eMMC driver, which may differ from part to part. As the only software company that is an active member of JEDEC, we are excited offer support for quite a few eMMC features. File system support for secure erase and secure trim will be coming later this summer!

Learn more about Datalight Embedded File Systems

To protect against unexpected power loss, so common in the embedded world, file writes need to be atomic.

Linux file systems ext3 and ext4 were designed for server or desktop environments. Google developer Tim Bray suggests that appropriate use of fsync() can mitigate the risk of data loss, but I am sure that’s not the best solution. The use of delayed allocation means that metadata is committed but the data is not. Alternatively, both can be committed to the journal at a performance penalty. Performance is crucial in both desktops and devices, but not at the expense of data corruption.

This problem is readily demonstrated when updating files, an action which usually happens “in place”. This is quite common in database and other important system files. When power is lost, data can be overwritten only partially, or else metadata can be altered to point to where the data will be updated but has not been. Another alternative to liberal use of fsync() is a rename strategy, that is, write only new data, then rename and replace the old file. Rename is atomic, at least.

The best solution, and one which does not require applications to change the way they do writes, is to perform all data writes atomically. In addition to that, the file system should never overwrite live data and always retain a “known good” state on the media. This way caching does not have to be removed – either user data changes get to the disk fully or not at all. No partial writes or incorrect metadata, and no mount-time journal rebuilds or disk checks either.

Instead of adapting a desktop or server file system for embedded use, it is far better to use a file system designed specifically for embedded use.

View whitepaper: Breakthrough Performance with Tree-based File Systems

Do you need defrag? It mostly depends on your hardware and your use case. While defragmenting a file system can make the computer run faster, it’s not the only answer.

Fragmentation is usually caused when modifying a file. Overwriting the file or making it larger means storing a fragment of the file in a new place, unless the file system creates a complete new copy of the file. Databases are particularly susceptible here – they are usually large files and often updated in the middle.

Another way fragmentation happens is when the file system initially stores the file in pieces. This could happen if the file system is not configured to keep file blocks together, or if the media is fairly full and there are no spaces of sufficient size for the new file.

What about the impact of fragmentation? In the days of rotating media, a fragmented file meant extra head movement and platter rotation to read the file. With flash media, the extra overhead is just additional block reads – a far smaller cost.

Avoiding fragmentation if you’re using Reliance Nitro can be as simple as customizing your transaction points. Instead of transacting on a timed basis, create a new transaction point only when the entire file is on the media, at “file close”. Similar settings may be available on other file systems.

If your use case causes fragmentation, a valid workaround might be to reformat the media after backing up the database files. A fresh file format is fairly quick on modern hardware, and can be coupled with a bad block test as well.

Read more about Reliance Nitro

Ever since we announced our high performance file system Reliance Nitro, we have been getting questions on how it compares to the original Reliance file system. Below is a quick-reference table noting some of the differences between the two. For a more detailed comparison (including performance benchmarks), please contact us.

There are two possible configurations in how boot code might be stored on a device

1. Boot code is stored in raw flash (no file system) and directly accessed from bootloader
2. Boot code is stored on a Reliance formatted flash volume

Option 1: Raw flash
If the boot image is being stored in RAW flash outside the file system, then the only way to be able to ensure that you got an update without damaging the original would be to reserve extra RAW space such that you could simultaneously have two boot images. The bootloader now needs to be able to switch between them and/or locate both of them The process of updating the boot image to a new location would include erasing the old image after updating the new, and having some sort of checksum to ensure the image was intact in case both were still there.

In this case, there would be no really good way to protect the update of the file to that exact same location without compromising the boot image itself. Many customers still use this way to store their boot images, but of course this means that they can’t take advantage of disabling transactions, atomically updating the boot image, and then doing a single transaction to commit all (or none) of the changes.

Option 2: Reliance
In this case, customer would not have a bootloader that checked a physical location for a boot image – they would have a bootloader that opened a file in the Reliance file system at boot time instead, if they were using a file system. Datalight Reliance comes with an utility called “Datalight Loader” which includes a lightweight Reliance reader. This utility integrates seamlessly in your bootloader code and allows the bootloader to mount and read Reliance partitions. Since the bootloader is capable of “reading” a Reliance disk, it doesn’t care where in the file system Reliance stores the file – it just opens the file, and loads it.

In this mode, while updating the boot image, the update utility disables all transactions and initiates the boot image update. Reliance never overwrites live data and hence this new boot code is written to a free-area of the flash. Once the entire boot image code is written, the bootloader calls for a manual transaction event, in which we update the metaroots to point to the new boot code area as the committed area. Old boot code area is now marked as free and can be used for future operations.

If power loss occurs during this replacement process, the device still boots back using the previous boot image, which was never modified

Image via Wikipedia

Linux has been slowly but surely establishing itself as the predominant OS in the embedded industry. ABI research report suggested that 23% of Smartphones will be based on Linux by 2013. High-profile industry support from Android and the LiMo foundation has put the spotlight back on embedded Linux.

In a previous post, we talked about flash memory and the various layers of flash management. In this post, I will talk about JFFS2, the most popular flash file systems available on the Linux platform

JFFS2

The Journaling Flash File System version 2 (JFFS2) is a log-structured file system that was originally designed in 1999. The original JFFS was developed by Axis Communications (and later enhanced by Red Hat) to provide support for NOR flash devices. The current version has been updated to include support for NAND flash. JFFS2 is open-source software, distributed under the terms of GPL license.

JFFS2 Strengths

1. Portability to Development Environments: Included with the Linux kernel since version 2.4.10, JFFS2 has become a de facto standard flash file system for Linux developers. Today, it is included in most commercial Linux distributions (such as MontaVista and Wind River Linux). Because of this wide distribution and use, it has been integrated into many varying environments and is known to be relatively easy to build.

2. Reliability: As changes are made to the file system, a “log” is built; this log provides information about where a file and its associated metadata are located on the flash chip.^[1] As the log is consistently maintained, it will be read back in the event of an unexpected power loss to determine the location of a missing file. Although the log structure provides a level of data reliability, this is accomplished at a cost to performance

3. Support for disk-wide compression: The benefit, or cost, of using compression depends on each specific use case. Compression will be useful in making efficient use of disk space when several text-only or code data (OS, etc.) files are being stored. Media files are already compressed (in *.jpg, *.mpg, or other formats), so the time used to attempt data compression is wasted. It is even possible that media files may take up more space after an unnecessary compression than originally needed. Disk usage and performance for the type and number of files to be maintained must be considered by the device designer in order to determine whether compression will yield a benefit or not

JFFS2 Shortcomings

1. Resource usage: RAM usage by JFFS2 increases in linear proportion to the number of nodes. Hence on large flash volumes, the system resources required by JFFS2 can be very significant

2. System performance: For devices that primarily use the file system for read operations, JFFS2 performance may be acceptable. However, for multi-functional devices whose applications perform a continuous mix of read and write operations, it is likely that the performance of a system using JFFS2 will not pass a rigorous standard. In addition to slow writes, the flash disk mount times of JFFS2 are exceptionally slow and worsen as the amount of data stored increases. Upon start-up after an unexpected power failure, JFFS2 must reconstruct the file system structure from the log. This is a costly operation that requires several seconds – more for volumes that are large or near-capacity. The device will be halted during this check operation, as any data that is stored on the disk will not be ready for use until the start-up completes

Informative links on JFFS2

1. JFFS2 Technical paper [PDF]

2. JFFS2 RAM usage [ppt] – Presentation at Embedded Linux Conf 2007

In the next post in this series, we will look at another popular Linux flash file system – YAFFS.

[1] Details on JFFS2 log structure can be found here http://sourceware.org/jffs2/jffs2-slides-transformed.pdf