There’s a lot of buzz on the MSDN blog site regarding their latest file system post. http://blogs.msdn.com/b/b8/archive/2012/01/16/building-the-next-generation-file-system-for-windows-refs.aspx - and plenty of insightful comments as well.

I for one am happy to see people talking about file system features, especially Data Integrity, knowledge of Flash Media, and faster access through B+ trees. Of course, Datalight’s own Reliance Nitro file system has had all this and more for some time now…

Microsoft has a new term for a thing we’ve seen often in the case of unexpected power loss – a “Torn Write”. They point this out as a specific problem for their journalling file system, NTFS, but updating any file system metadata in place can be problematic. It looks to me like this new file system, ReFS, handles this by bundling the metadata writes with other metadata writes or with the file data. If the former, this demonstrates the trade-off between Reliability and Performance that we are very familiar with at Datalight. Bundling smaller writes will help with spinning media and flash. In time we will see how much control the application developer has over this configuration – another important point for our customers.

One of the commenters posted that error correction belongs at the block device layer, and I tend to agree. Microsoft’s design goal “to detect and correct corruption” is a noble one, but how would they detect corruption for user data? Additional file checksums and ECC algorithms would be intrusive and potentially time consuming. Keeping a watch on vital file system structures is important, of course, and a good backup in case block level error detection fails.

I look forward to reading more from Microsoft’s file system team in the future, and especially hope to see a roadmap for when these important changes will make it down to the embedded space.

Learn more about what happens during a power interruption.

Do you need defrag? It mostly depends on your hardware and your use case. While defragmenting a file system can make the computer run faster, it’s not the only answer.

Fragmentation is usually caused when modifying a file. Overwriting the file or making it larger means storing a fragment of the file in a new place, unless the file system creates a complete new copy of the file. Databases are particularly susceptible here – they are usually large files and often updated in the middle.

Another way fragmentation happens is when the file system initially stores the file in pieces. This could happen if the file system is not configured to keep file blocks together, or if the media is fairly full and there are no spaces of sufficient size for the new file.

What about the impact of fragmentation? In the days of rotating media, a fragmented file meant extra head movement and platter rotation to read the file. With flash media, the extra overhead is just additional block reads – a far smaller cost.

Avoiding fragmentation if you’re using Reliance Nitro can be as simple as customizing your transaction points. Instead of transacting on a timed basis, create a new transaction point only when the entire file is on the media, at “file close”. Similar settings may be available on other file systems.

If your use case causes fragmentation, a valid workaround might be to reformat the media after backing up the database files. A fresh file format is fairly quick on modern hardware, and can be coupled with a bad block test as well.

Read more about Reliance Nitro

JEDEC, the Joint Electron Devices Engineering Council (see http://www.jedec.org), is a group of manufacturers and suppliers collaborating to create specifications for Flash memory access and parts. The current revision of their specification for Embedded MultiMedia Cards, eMMC, is 4.41, and is available on the website above.

I’m excited to be a part of Datalight and JEDEC, and am looking forward to the upcoming eMMC 4.5 and UFS 1.0 specifications. Datalight is not in the business of manufacturing hardware, of course, but our file system products like to work closely with the underlying driver. Until those products are fully eMMC 4.5 compliant, what can you expect?

The most fundamental thing a Reliance file system needs is a block device that writes data when it says it will. Any data left in a cache or not flushed upon command could be lost data in an unexpected power loss. The Enhanced Reliable Write feature means any eMMC 4.41 or 4.5 flash part will work perfectly with Reliance and Reliance Nitro.

The High Priority Interrupt feature of eMMC basically means that a block device write might pause, reporting back only a partial write. This is fully supported in the Reliance Nitro file system, which will then loop back and continue the write after the HPI is complete.

The Trim feature of eMMC 4.4.1 is being replaced by a Discard feature in eMMC 4.5. The latter fits in more closely with the way Reliance interacts with our own FlashFX product.

Basic functionality (Read, Write, and Erase) is of course supported, and full compliance with eMMC 4.5 is on Datalight’s roadmap, so keep an eye out here for more news soon.

Last week one of our customers sent the following evaluation report in an email to Datalight support staff:

“Right now we are in the process of testing the impact of Reliance Nitro in our application. Apparently, we noticed some boost in the performance:  faster write speed, significant speed increase of transaction point creation, faster read speed, and significantly faster directory read (we typically have 1000 files in the directory). So, in conclusion, the overall performance of the system is boosted quite significantly.”

Another real world example of how Reliance Nitro boosts performance in directories with a large number of files. To learn how Reliance Nitro does it, check out the whitepaper.

Palm just announced the 1.1 update to its popular WebOS that runs on the Palm Pre device. Apple released the 3.0 update to its Mac OSX for the iPhone and iPod touch. Microsoft is expected to launch Windows Mobile 6.5 soon and users are hoping that they will be able to update their 6.1 devices to 6.5. Google last month updated Android OS to 1.5. These events point to a recent and very fast growing phenomena: embedded devices are becoming more and more like PCs where users expect to be able to update their device long after it has been released. This was not always the case; OEMs refrained from updating embedded devices unless in cases of high severity bug fixes. There are several reasons for this:

1. Updating embedded devices is more difficult than updating PCs from a distribution standpoint
2. Because of #1, updating devices is also expensive
3. Potential of bricking devices (device does not boot anymore) due to user error is very high leading to high risk of warranty returns

Today’s blog post focuses on #3 because it has very real technical risks and solutions. Before we begin discussing risks of bricking device, let’s talk about 2 different types of updates

1. Update to application code – This is usually much simpler and does not typically involve changes to the bootloader or the boot image
2. Update to system code/OS image – most of the times when OEMs have to update devices, it is due to some severe error. In our experience it usually involves changing system files. If the entire OS is stored as a single image on disk/flash, then entire image has to be correctly replaced with the new one

If the update is of type 1, then the process has less likelihood of bricking the device. In most cases even if the update fails, support can help user start the device in “safe” mode and restore. Updates of type 2 are by nature riskier because any failure is likely to stop the device from booting up, negating any remote debugging options. Note that It is also possible that for some devices, the application and system code is stored in single boot image. In that case, the distinction of types made above are irrelevant for this discussion.

OS/System Updates

Here is how devices typically partition the data storage for boot and application data

Note: Some devices may not use the file system for the boot partition and instead directly talk to the block device. In that case, the remainder of this discussion is not applicable.

During an update process involving system code, the boot image has to be replaced with a  new one. Typically the update process will overwrite the existing image. The problem happens when the update process is interrupted due to erroneous circumstances such as

1. Device battery dies before the update process is completed
2. The user pulls out the USB cord connecting the device to host

In these cases, the OS image will get corrupted and the device may not be able to boot back up, leading to a bricked device.

One of the features of Reliance (and Reliance Nitro) file system is that it never overwrites live data. It will always use free space on disk or in case there is no space, it will give “disk full” error back to the application. Reliance also has a special transaction mode called “Application-controlled”. In this case, Reliance only conducts a transaction point when asked by the application. Here is how these 2 constructs help Reliance provide a fail-safe means of in-place updates

1. The OS image is stored on a Reliance partition
2. The update application calls Reliance API to disable all transaction modes. Reliance will now execute a transaction point only when specifically called by the update app
3. The update app starts “overwriting” the existing OS image. Because Reliance never overwrites live data, it will start copying the new image to free space on disk
4. In case power is interrupted, Reliance discards the new image and device can still boot back to the old OS image and restart the update process
5. Once the entire update process is completed, the update app calls Reliance to execute a transaction point. Reliance, in one atomic operation, updates its committed state to now use the new image. When the device boots back up, it now uses the new image. The old image is now marked as free space by the file system

Using Reliance for boot partition can thus help in providing a safe in-place update process. It also has the advantage of using Reliance extreme fast mount times, which can help in speeding device boot speeds.

Note that the obvious caveat of the above is that there has to enough free space for the new OS image. With disk storage being cheap (compared to device cost) and always increasing, this becomes less and less of an issue. OEMs should strongly consider going this alternative (whether they use Reliance or not) in order to ensure that the device update process will go smoothly for the end users.

Ever since we announced our high performance file system Reliance Nitro, we have been getting questions on how it compares to the original Reliance file system. Below is a quick-reference table noting some of the differences between the two. For a more detailed comparison (including performance benchmarks), please contact us.

With the release of our new file system this week, Reliance Nitro, we asked our Account Managers what they liked most about our new product. Their answers of course included reliability and high performance. Wes Johns and Phillip Allison were so excited they decided to make a video…  watch the youtube video