Watch Rob Hart demonstrating the Datalight flash file system on the Beagle Board with Windows CE 6.0

Palm just announced the 1.1 update to its popular WebOS that runs on the Palm Pre device. Apple released the 3.0 update to its Mac OSX for the iPhone and iPod touch. Microsoft is expected to launch Windows Mobile 6.5 soon and users are hoping that they will be able to update their 6.1 devices to 6.5. Google last month updated Android OS to 1.5. These events point to a recent and very fast growing phenomena: embedded devices are becoming more and more like PCs where users expect to be able to update their device long after it has been released. This was not always the case; OEMs refrained from updating embedded devices unless in cases of high severity bug fixes. There are several reasons for this:

1. Updating embedded devices is more difficult than updating PCs from a distribution standpoint
2. Because of #1, updating devices is also expensive
3. Potential of bricking devices (device does not boot anymore) due to user error is very high leading to high risk of warranty returns 

Today’s blog post focuses on #3 because it has very real technical risks and solutions. Before we begin discussing risks of bricking device, let’s talk about 2 different types of updates 

1. Update to application code – This is usually much simpler and does not typically involve changes to the bootloader or the boot image
2. Update to system code/OS image – most of the times when OEMs have to update devices, it is due to some severe error. In our experience it usually involves changing system files. If the entire OS is stored as a single image on disk/flash, then entire image has to be correctly replaced with the new one

If the update is of type 1, then the process has less likelihood of bricking the device. In most cases even if the update fails, support can help user start the device in “safe” mode and restore. Updates of type 2 are by nature riskier because any failure is likely to stop the device from booting up, negating any remote debugging options. Note that It is also possible that for some devices, the application and system code is stored in single boot image. In that case, the distinction of types made above are irrelevant for this discussion. 

OS/System Updates 

Here is how devices typically partition the data storage for boot and application data

Note: Some devices may not use the file system for the boot partition and instead directly talk to the block device. In that case, the remainder of this discussion is not applicable.

During an update process involving system code, the boot image has to be replaced with a  new one. Typically the update process will overwrite the existing image. The problem happens when the update process is interrupted due to erroneous circumstances such as 

1. Device battery dies before the update process is completed
2. The user pulls out the USB cord connecting the device to host

 In these cases, the OS image will get corrupted and the device may not be able to boot back up, leading to a bricked device.

 One of the features of Reliance (and Reliance Nitro) file system is that it never overwrites live data. It will always use free space on disk or in case there is no space, it will give “disk full” error back to the application. Reliance also has a special transaction mode called “Application-controlled”. In this case, Reliance only conducts a transaction point when asked by the application. Here is how these 2 constructs help Reliance provide a fail-safe means of in-place updates

1. The OS image is stored on a Reliance partition
2. The update application calls Reliance API to disable all transaction modes. Reliance will now execute a transaction point only when specifically called by the update app
3. The update app starts “overwriting” the existing OS image. Because Reliance never overwrites live data, it will start copying the new image to free space on disk
4. In case power is interrupted, Reliance discards the new image and device can still boot back to the old OS image and restart the update process
5. Once the entire update process is completed, the update app calls Reliance to execute a transaction point. Reliance, in one atomic operation, updates its committed state to now use the new image. When the device boots back up, it now uses the new image. The old image is now marked as free space by the file system

Using Reliance for boot partition can thus help in providing a safe in-place update process. It also has the advantage of using Reliance extreme fast mount times, which can help in speeding device boot speeds.

Note that the obvious caveat of the above is that there has to enough free space for the new OS image. With disk storage being cheap (compared to device cost) and always increasing, this becomes less and less of an issue. OEMs should strongly consider going this alternative (whether they use Reliance or not) in order to ensure that the device update process will go smoothly for the end users.

With the release of our new file system this week, Reliance Nitro, we asked our Account Managers what they liked most about our new product. Their answers of course included reliability and high performance. Wes Johns and Phillip Allison were so excited they decided to make a video…  watch the youtube video

We’re totally psyched about Reliance Nitro, our newest file system (yes, we’re file geeks), and we’re always on the lookout for opportunities to show off the performance and reliability attributes it adds to Windows Mobile and Windows CE. When we discovered the relatively-new Beagle Board, it occurred to us that a small, low-cost platform might be just the thing to demonstrate Nitro’s amazing benefits. As you’ve probably heard, the Beagle is making waves with its low cost (around $150) and diminutive size. It uses an OMAP 3530 processor and 256MB of NAND. Though they are most commonly used with Linux, we lucked out in having a partner (MPC Data) who has already developed a Windows CE BSP for it. After a few phone calls, the wizards at MPC Data were able to develop a slick video playback demo app, and presto, the Reliance Nitro Beagle Demo was born! Amateur videographers that we are (ok, REALLY amateur), we recently videotaped John Burnham, who has been working on this project on the Datalight side (and who is a really good sport, btw) showing what happens when power is interrupted during a file write and the extra reliability factor of Reliance Nitro on Windows CE. Be sure to check it out here.

Sea Change Hits Consumer Electronics as Customers Demand Long-term Value

For the first time in more than a decade, people are saving again. In 2007 and years prior, the savings rate hovered around zero as we maxed our credit cards and lines of credit, driving the savings rate into the red and giving the world’s manufacturing base an almost unbelievable boom. In January 2009 though, something unexpected happened; the US savings rate suddenly moved above 5%, the highest in decades. As news of our cloudy economic picture has emerged, consumer behavior is shifting away from status-seeking luxury purchases toward more value-based buying patterns, forcing manufacturers around the world to take notice. And after decades of excess, the shift to thrift is looking like a lasting trend.

But what does this mean for Embedded? As consumers focus on needs over wants, they will increasingly seek out products that are proven durable and reliable.

This will have broad implications for manufacturers of everything from cars to clothing, refrigerators to embedded devices. Today’s consumers are choosing efficiency, durability and value over gee-whiz gadgetry. Consumer mobile OEMs too must focus on delivering value and fewer, more targeted features. Rather than packing devices full of a laundry list of apps and expensive hardware, this means streamlined offerings and more segmented products, while making sure the consumer doesn’t feel like they’re missing out. Motorola’s new EM330 is a prime example of this kind of pared-down, demographic-specific approach. The phone, called the MOTOROKR STAR is marketed specifically toward music lovers, offering a basic clamshell with music recognition software and download-on-the-go at a price point in the sub-$200 range.

As OEMs scramble to add value and enhance their reputations for durability and reliability, Datalight responds with products that support those goals. The combination of flexible flash management that lowers bill of material costs, wear-leveling algorithms extend flash life by several times, and the rock-solid reliability of our file system become essential components of a strategy to provide value to customers.

Many have remarked that markets are driven by a combination of fear and greed. Though the pendulum has recently taken a dramatic –and we believe temporary– move in the direction of fear, ultimately we know a move away from excess is good for all of us and good for the world we live in. Here’s hoping the trend toward value and quality is a long-lasting one.

OEMs Position Themselves for the Economic Turnaround

As everyone’s mother used to say, “When life hands you lemons…” And at this particular time in embedded technology and elsewhere, it seems there is no shortage of them.  In just the last three weeks, we’ve heard about Nortel filing for bankruptcy, Motorola planning to cut 7,000 jobs, and Sony Ericsson’s dramatic profitability swing from $1.48 billion in 2007 to negative $96 million in 2008. And yet, one thing we know about recessions is that they inevitably end in expansion. The only question is when. With the fresh winds of optimism blowing in from Washington this week, now is as good a time as any to start planning your strategy for survival followed by world domination.

Indeed, if there is a silver lining to our current situation, it has to be the unprecedented opportunities for well-positioned, forward thinking companies to dominate their markets. Clearly, this takes grit, an eye on keeping costs down, and a truly differentiated product strategy. It’s more important now than ever before for OEMs to focus on their core competencies, keep bill of materials costs low and work with vendors who have experience weathering economic storms and possess the fortitude to survive tough times.

Datalight was born in 1983 during one of the worst recessions in the post-war era, and we remember well the challenges of the early 90’s and the aftermath of 9/11/01. While many of our competitors are pulling resources from flash file system development, we are moving ahead with a full schedule of product releases and innovation for 2009. While at CES this year, we discovered one of our competitors is pulling out of the flash file system business altogether. I won’t bore you with a long-winded sales pitch, because chances are if you’re reading this newsletter you already know Datalight can help you build better products with lower development costs and give you the ability to choose between the least expensive of over 300 NAND and NOR flash parts.

Like our new president, we are cautiously optimistic about the coming year. As President Obama said so well, this economy is going to take hard work and a little time to turn around. One thing we know for sure though, is that the turnaround is coming. Companies that design products their customers need and want, continue to invest in technology and communication, and prepare themselves to capitalize on a business environment with fewer competitors will do extremely well in the coming years.

5 Reasons for Ditching Managed NAND
Everyone knows that NAND has challenges: from factory bad blocks and spontaneous bit failures to endurance limits, etc. That’s why a few years ago managed NAND (NAND flash plus an integrated controller) seemed to be the answer, offering the density of raw NAND, while mitigating many of its inherent limitations. What many device manufacturers may not realize is that the management hardware comes with significant costs, both in terms of dollars per part as well as design limitations. In the world of tradeoffs in which every system designer lives, there are solid technical reasons to consider using raw NAND and leaving the management to software instead. While there are clear commercial advantages for Datalight (as a provider of vendor-neutral software-based flash management) to advocate this approach, we also believe that there are strong technical reasons that flash silicon vendors would do well to embed ECC capabilities into their NAND flash devices rather than relying on ‘total hardware’ solutions such as eMMC or other complex and costly controllers. Beyond the benefits outline below, this approach would allow the flash manufacturers the ability to continue to differentiate their products from others in the industry.

1.    Cost
The cost of managed NAND parts is coming down, but the stuff still sells at a premium over its raw NAND brethren. 

2.    Flash Optimization
There are many new features of NAND available to us today.  Performance features such as cached reads, multi-plane operations, concurrency, and others are becoming invaluable to keep performance at the ever-increasing demands of portable media. The Open NAND Flash Interface (ONFI) has defined a standard method to query the capabilities and characteristics of NAND flash which can be put to use by both software and hardware systems. A software media manager offers the flexibility to take advantage of the most current flash memory features and put them to use efficiently, or to avoid certain features that may be unproven or problematic. A software solution will allow a developer to take full advantage of the media’s characteristics and features unburdened by the indirection or inability for the hardware to expose them.

3.    Visibility/Flexibility
Software (in general) is easily inspected and validated. Features such as wear-leveling move data around the flash device to optimize its life expectancy. Without the ability to inspect source code, a managed NAND solution makes it difficult to validate wear-leveling operation and/or characterize its effect on performance and reliability.  Hardware implementations are often generalized to suit a majority of use cases, while a software solution is easily tailored to the specific use case during development.

4.    Performance
Speaking of use cases, there are many system features that are not available to hardware that may make a generalized hardware solution less advantageous to a specific use case.  For example, system idle time can be used to improve the media performance by scheduling background cache operations and compaction to occur then.  Coordination between the file system and flash media manager can further optimize operations by freeing space when it will no longer be needed and having the media manager code cache certain regions of the flash where meta data might be held. Migrating flash management features to hardware removes this ability to coordinate with other components of the software stack, such as file systems.

5.    Reliability
Lest you think we believe that everything is better left to software, consider error detection and correction (EDC). Error rates are increasing substantially as flash manufacturers push the limits of physics.  Errors can be introduced externally by heat or other radiation, during writes or reads of data, and even to data that was successfully written at a different time. Historically SLC NAND flash required only a single bit error detection and correction (a hamming code is usually sufficient), while MLC parts require minimally four bit EDC.  As the die sizes continue to shrink, error rates will continue to increase, even for SLC flash. 
Calculating the codes to detect and correct such errors is getting increasingly complex and solving such a solution in software for higher-bit EDCs (above 4-bit) is time consuming and often unacceptably slow.  Hardware ECC is a necessary requirement for systems with high EDC requirements and where performance is a concern.

Many of the processors on the market today are incorporating EDC in their NAND controllers. Choosing one of these processors (e.g. TI OMAP 35xx) in combination with raw NAND flash and software management can give you the high-performance EDC to handle next generation flash while maintaining the design flexibility that a software manager provides.

Flash manufacturers have much to gain by adding ECC code into their NAND flash parts.  They know better than anyone what kind of ECC is necessary for a specific part and by adding just that one piece of hardware to their offerings, rather than the jack of all trades, master of none approach of complete flash management, they will better serve the markets. 

In short, features should reside where they can be handled most efficiently; ECCs belong in hardware, other flash management functions belong in software.  While managed NAND certainly has its place and its appeal in the market, we believe the best combination of value, performance and flexibility lies in using a combination of raw NAND and hardware with built-in ECC capabilities.

The Challenge
Recently, LG Electronics, a well-known Korean-based manufacturer of consumer electronics, created a multimedia-enabled portable navigator for the North American market. The LN790 features a 4.3” LCD screen, Bluetooth hands-free functionality, and video-enabled playback. Ruggedness and fast access to data are important to consumers in this market, so the device was designed to boot directly from a NAND mass storage environment using Windows CE. Unfortunately, LG product engineers had a difficult time getting the device to boot fast enough using CE’s FAL/FMD flash drivers.  At just over two minutes, the startup time did not match LG’s reputation for high-performance consumer devices.

The Datalight Solution
As LG engineers went searching for solutions to the boot speed problem, they discovered that Datalight FlashFX® Pro uses a more efficient approach to managing bad blocks than CE’s standard FMD/FAL drivers, which can speed boot time significantly.  This difference is especially apparent when the device is using a large NAND disk, because boot time is somewhat proportional to the size of the flash.
Why is FlashFX Pro more efficient? Startup with FAL requires the driver to read more data as part of its mount sequence, a lengthy process particularly if the disk is large. In contrast, FlashFX Pro requires a much simpler check of the media to complete the initial mount.

The Customer Payoff
After implementation of FlashFX Pro, LG engineers were delighted to discover that the device’s boot time was cut by more than half. By using FlashFX Pro instead of the native Windows CE drivers, LG designers were able to achieve the performance their customers expect from a premium-quality personal navigator.  There was also an additional benefit they hadn’t counted on – FlashFX Pro support for over 200 flash parts means that the LN790 will be future-proof from flash parts going on allocation, unexpected price fluctuations, and end-of-life issues.

Image via Wikipedia

Being in the flash memory management space for 15+ years, a very high number of our customers use our products on flash memory (NAND, NOR, NAND controllers, Fusion flash like Samsung OneNAND, etc). Now FlashFX Pro is designed only for flash memory but Datalight Reliance is a file system that works on all block devices. This includes hard drives, USB flash drives, removable cards like SD, CF, solid state drives (SSD), etc. The advantage Reliance brings to these devices is of reliability against data corruption, fast mount times and fast I/O throughput. It also mandates certain requirements on the physical media to ensure reliability against data corruption. We have had customers use Reliance on hard drives before and I want to share some requirements for Reliance to provide high reliability on rotating media. This post is specific to Linux but the general concepts should be applicable to all OSes.

Reliance is a transactional file system and at each transaction point it flushes all its internal caches and commits the data to disk in atomic operations. Primary requirement for Reliance to function on hard drives is that the hardware and the ATA driver must support the “FLUSH CACHE” command. The Linux IDE disk driver checks bits 12 and 13 of word 83 in the IDENTIFY DEVICE information to determine whether FLUSH CACHE is supported.  These bits are defined by the ATA-6 specification, and are not set in earlier drives.  The IDE disk driver will report whether it has detected this capability in a drive.  This is available in the system log.  A typical message will look like:

Jun  9 09:49:23 billr-qa kernel: [   18.621740] hda: cache flushes supported

Since there are a vast number of hard disks on the market and new ones are constantly being introduced (and old ones discontinued), it is a little difficult for Datalight to qualify all hard drives and recommend a specific one. Generally any disk that conforms to the ATA-6 specification and reports that it supports FLUSH CACHE should work correctly with Reliance.  Reliance reports whether it is able to use flush to ensure correct operation, the system log typically looks like this:

Jun  9 09:52:44 billr-qa kernel: [  240.283463] relfs: block device supports flush.

If this message appears in the log, Reliance should operate correctly when power is interrupted unexpectedly.

Datalight’s power interruption testing has been performed on a Western Digital AC29100D using kernel version 2.6.21.1

If you have any questions on the FLUSH CACHE on an OS other than Linux, please leave a comment.