While the software that we develop integrates at system level in devices, we do keep an eye open for developments happening at the application level primarily because the heavy data storage use most modern applications make. Understanding what kind of applications are making headway in the market helps us determine the type of data profile to expect at the file system and flash management level. This helps is in creating new system level functionality that benefits all applications. One example of this is the work done in Reliance Nitro. We observed a lot of device applications and how they use files. We found that apart from some data logging / music playback apps, most apps do not indulge in sequential reads and writes that often. Most of the time, apps are primarily engaged in file operations (file open being the most common). Hence we designed the tree-based metadata architecture in Reliance Nitro to provide radical performance improvements in the file operation speeds.

When we talk about apps on devices, we cannot ignore the 800-pound gorilla – the Apple Appstore. The appstore has created a huge ecosystem for apps on the iPhone and the iPod touch by having a one-stop shop for discovering, reviewing and purchasing apps. Combine this usability with the popularity of the hardware, the app ecosystem has blossomed which has forced rivals to pay attention and start something similar of their own. Google’s Android marketplace is probably the next biggest player while Palm, Microsoft and Nokia are just getting started. While Apple does indeed have a head start of 12-14 months, what is interesting to note is the huge difference between it and its closest rival. This article by Fierce Developer highlights the significant difference in revenue that app developers are seeing on the Apple v/s the Android platform.

Source: AdMob’s July 2009 Mobile Metrics Report

What do these stats mean for device developers? In order to catch up or surpass Apple, innovation needs to happen on multiple fronts

1. Device hardware: Apple innovated with the capacitive touchscreen and no-keyboard design. OEMs need to find the next game changer on the hardware front

2. Device user experience: Hardware on its own does not win over users. There have been several iPhone competitors who have had better hardware of paper but without matching software capabilities, the devices have not met with spectacular success. Using an optimized data storage stack can help improve device responsiveness which users rank as one of their top requirements. Creating new simple user interface paradigms will also be critical.

3. Developer support: Use of standard tools and languages for writing apps along with a low cost of entry is critical to attract app developers to your platform

4. Discovery and purchase of apps: How do users of your devices find and purchase apps? How can you do it better and faster? What is the average price-point for the buyer profile of your device. All of these are critical questions to answer before creating an app store.

It will need strong execution on all of the above fronts for device manufacturers to catch up to the Apple success. We will be keeping an eye open to developments here; look out for further postings on this topic in the future.

Denali’s Lane Mason has a very good article on NAND flash pricing. Strongly recommended reading.

http://www.denali.com/wordpress/index.php/dmr/2009/07/16/nand-forward-prices-rate-of-decline-will

Palm just announced the 1.1 update to its popular WebOS that runs on the Palm Pre device. Apple released the 3.0 update to its Mac OSX for the iPhone and iPod touch. Microsoft is expected to launch Windows Mobile 6.5 soon and users are hoping that they will be able to update their 6.1 devices to 6.5. Google last month updated Android OS to 1.5. These events point to a recent and very fast growing phenomena: embedded devices are becoming more and more like PCs where users expect to be able to update their device long after it has been released. This was not always the case; OEMs refrained from updating embedded devices unless in cases of high severity bug fixes. There are several reasons for this:

1. Updating embedded devices is more difficult than updating PCs from a distribution standpoint
2. Because of #1, updating devices is also expensive
3. Potential of bricking devices (device does not boot anymore) due to user error is very high leading to high risk of warranty returns

Today’s blog post focuses on #3 because it has very real technical risks and solutions. Before we begin discussing risks of bricking device, let’s talk about 2 different types of updates

1. Update to application code – This is usually much simpler and does not typically involve changes to the bootloader or the boot image
2. Update to system code/OS image – most of the times when OEMs have to update devices, it is due to some severe error. In our experience it usually involves changing system files. If the entire OS is stored as a single image on disk/flash, then entire image has to be correctly replaced with the new one

If the update is of type 1, then the process has less likelihood of bricking the device. In most cases even if the update fails, support can help user start the device in “safe” mode and restore. Updates of type 2 are by nature riskier because any failure is likely to stop the device from booting up, negating any remote debugging options. Note that It is also possible that for some devices, the application and system code is stored in single boot image. In that case, the distinction of types made above are irrelevant for this discussion.

OS/System Updates

Here is how devices typically partition the data storage for boot and application data

Note: Some devices may not use the file system for the boot partition and instead directly talk to the block device. In that case, the remainder of this discussion is not applicable.

During an update process involving system code, the boot image has to be replaced with a  new one. Typically the update process will overwrite the existing image. The problem happens when the update process is interrupted due to erroneous circumstances such as

1. Device battery dies before the update process is completed
2. The user pulls out the USB cord connecting the device to host

In these cases, the OS image will get corrupted and the device may not be able to boot back up, leading to a bricked device.

One of the features of Reliance (and Reliance Nitro) file system is that it never overwrites live data. It will always use free space on disk or in case there is no space, it will give “disk full” error back to the application. Reliance also has a special transaction mode called “Application-controlled”. In this case, Reliance only conducts a transaction point when asked by the application. Here is how these 2 constructs help Reliance provide a fail-safe means of in-place updates

1. The OS image is stored on a Reliance partition
2. The update application calls Reliance API to disable all transaction modes. Reliance will now execute a transaction point only when specifically called by the update app
3. The update app starts “overwriting” the existing OS image. Because Reliance never overwrites live data, it will start copying the new image to free space on disk
4. In case power is interrupted, Reliance discards the new image and device can still boot back to the old OS image and restart the update process
5. Once the entire update process is completed, the update app calls Reliance to execute a transaction point. Reliance, in one atomic operation, updates its committed state to now use the new image. When the device boots back up, it now uses the new image. The old image is now marked as free space by the file system

Using Reliance for boot partition can thus help in providing a safe in-place update process. It also has the advantage of using Reliance extreme fast mount times, which can help in speeding device boot speeds.

Note that the obvious caveat of the above is that there has to enough free space for the new OS image. With disk storage being cheap (compared to device cost) and always increasing, this becomes less and less of an issue. OEMs should strongly consider going this alternative (whether they use Reliance or not) in order to ensure that the device update process will go smoothly for the end users.

It’s been over 20 years since NAND flash was supposed to take over the world, so why is the ‘ultra-portable’ laptop I’m writing this on still using HDD? And why, nearly 30 years after the invention of flash, are we still debating its applications vs. the rotating platter? Given the performance advantages of flash and the mechanical shortcomings of HDD – things like wear and susceptibility to shock and vibration, not to mention the latency associated with spinning up for reads – it’s no wonder that flash has long been the darling of memory technology futurists. So why after all this time has flash not replaced older technologies? I’ll give you a hint: It’s all about the money. But it’s not only about the money – check out our list of five top barriers to flash adoption:

1. Cost – Despite years of oversupply in the flash market, and the corresponding reductions in price, flash is still relatively expensive when compared to HDD, especially on a $/bit basis. To make matters worse, the current economic climate has taken its toll on the flash industry, spurring several rounds of consolidation and requiring flash vendors to curb manufacturing costs by shrinking portfolios and closing fabs. Predictably, these changes in the supply landscape are causing prices rise in many cases, making the cost factor an even bigger problem for flash.

2. Shrinking lithography = lower endurance – One way for flash manufacturers to remain competitive is to use smaller die size to reduce raw material costs. Just a couple years ago, the vast majority of NAND flash was manufactured with 90nm lithography. Most vendors are now planning to move to 30nm technology either this year or next. An unfortunate side-effect of smaller lithography is significantly decreased endurance. SLC NAND, which had 100K + erase cycles, is now predicted to be in the 50-70k range. The biggest impact is on MLC NAND where the endurance has gone from 10k erase cycles to around 3k (a 70% reduction!).

3. Increasing ECC – Another side-effect of shrinking lithography is an increase in error rates for flash, requiring stronger correction codes. Most SLC NAND flash today requires 1-bit correction. That number is predicted to increase to 4-bit on 30nm NOR parts. And the ECC outlook for MLC NAND is even worse, requiring ECCs greater than 12-bit (compared to 4-bit or 8-bit today). These increased ECC requirements mean the controller design for managing flash will become more complicated, and more difficult for OEMs to implement. Performance will also be impacted, especially if the ECC is done in software running on the host processor.

4. Vendor volatility – Churn or volatility in the flash market, the products of a difficult economic climate, are making it difficult for OEMs to find a reliable source of flash parts. Examples are everywhere; A major flash supplier is currently under Chapter 11. There are merger talks happening between SanDisk and Samsung. Asian vendors have been hit especially hard, particularly those also in the DRAM business. OEMs are rightfully concerned about interruptions to their production cycles in the midst of all this turmoil.

5. Lack of killer application – While NAND flash densities have continued to increase, the industry is still waiting for the killer application to gobble up these immense quantities of flash. For long SSDs have been viewed as that application but they have not taken off as fast as the flash industry would have liked.

In spite of the obstacles faced by the industry, flash remains a strong and growing choice for data storage and has put breakthrough devices like MP3 players and smart phones (iPhone!) into the hands of millions of consumers. Early adopters of SSD technology in laptop computers, netbooks and enterprise applications are making a solid case for mass market potential there, which should significantly drive flash adoption in the next few years. Visit the FlashFX Tera page to learn how Datalight is making flash easier and more competitive.

We have talked about managed NAND in a few blog posts before. Usually a combination of raw NAND flash (SLC or MLC) combined with a hardware controller that performs flash management features like bad block management, ECC and wear leveling is referred to as managed NAND. The term covers a huge spectrum of flash-based storage devices so in this post we will try and highlight some of the more prevalent types of managed NAND

The following is an enumeration of some of the popular managed NAND form factors. Please note that the list covers flash technologies used for resident storage and does not cover removable storage like USB flash, SD, etc.

•    eMMC
•    eSD
•    CompactFlash
•    Solid State Drives
•    BA NAND
•    Adaptable NAND
•    Specialized
–    Specially designed controller + raw flash

CompactFlash is included here because it is used both as resident and removable storage. CF comes with a Fixed-drive option which allows it to be used a resident managed NAND.

The above technologies differ from each other on several attributes

•    Form factor – managed NAND can come is several form factors. An SSD may sport a standard 2.5” drive enclosure whereas a CF card will take a 1.0” card form factor.
•    Plug-in interface: What interface does the managed NAND use to connect to the device platform
–    MMC
–    SD
–    ATA
–    Custom
•    Cost: Cost depends on several elements
–    Type of flash used: SLC is much more expensive than MLC
–    Type of controller used: consumer grade controllers (used for consumer grade CF for example) are much cheaper than specialized industrial grade controllers
•    Performance
–    Performance varies depending on the flash type, the controller attributes and the interface.

Some of the big players in the managed NAND business are

•    eMMC
–    Micron, Numonyx
•    eSD
–    SanDisk, Toshiba
•    BA NAND
–    Toshiba
•    Solid State Drives, CompactFlash
–    Too many players in these markets

This was a brief view of the managed NAND landscape. If there is interest, we will do a follow up going in details about the specific categories and interfaces

SanDisk recently announced that it is end-of-lifing several flash parts in the mDOC family and OEMs who were using these parts in their devices are now looking at viable alternatives.  In order to help these affected customers make informed decision, we have published a new whitepaper on options for migrating from Sandisk mDOC flash family.

The paper is available at http://www.datalight.com/mdocwhitepaper/

Just got the PR from SD Card Association (SDA) announcing availability of eSD – embedded flash technology standard (full release below). We applaud the development of standards that will make it easier for OEMs to meet the growing data storage demands in their devices. As a member of the SD Card Association, Datalight has long been a proponent of increasing the reliability of data storage. While the eSD standard will help with integration and hardware reliability challenges, it will also shift the spotlight to the shortcomings of traditional file systems. Building a fully robust and incorruptible product requires  reliability throughout the entire hardware platform and software stack.

Full Press release

SD CARD ASSOCIATION INTRODUCES EMBEDDED SD
FOR MOBILE PHONES, CONSUMER DEVICES

New Embedded SD standard offers field-proven SD functionality
and design ease to increase device storage options

SAN RAMON, Calif. – Sept. 11, 2008 – The SD Card Association (SDA) announced today it will enter the embedded storage market with the Embedded SD standard for storage devices in November. The Embedded SD specification extends the widely adopted, market-proven SD interface to embedded storage, doubling device storage potential and easing platform design in mobile handsets and consumer electronics devices.

As next-generation mobile handsets and CE devices demand high-capacity embedded storage and flash management solutions, Embedded SD simplifies system engineering by leveraging well-known SD standards. It allows all removable, embedded storage devices and input/output (I/O) peripherals to utilize the same common interface. Embedded SD separates flash management from the host and provides manufacturers and consumers with the flexibility of up to 32 GB scalable high-capacity on-board memory, plus 32 GB of removable memory with any SD High-Capacity memory card.

“The miniSD and microSD form factors are the leading interfaces for removable storage cards for mobile handsets, currently dominating the market, and are expected to constitute 90 percent of all card slots in mobile handsets in 2010, according to our analysis,” said Nam Hyung Kim, director and chief analyst for iSuppli Corp. “The new Embedded SD standard is built on that leading SD standard and takes aim at mobile handset storage needs like low power consumption, boot functionality, small form factor and integrated flash management.”

The Embedded SD standard regulates the specifications of SD interface-based embedded flash devices, easing the integration of storage devices and avoiding market fragmentation in the mobile and CE industries.

The Embedded SD specification defines the mechanical and electrical framework of the new embedded form factor and is also focused on enabling advanced functionality in embedded storage devices. Embedded SD devices offer a smart system solution with a smooth migration path from legacy designs, reduced design complexity and support for code, applications and user data storage.

“The SD Card Association plans to significantly reduce market fragmentation by delivering superior compatibility to the world of embedded storage, just as we have done in the storage card industry,” said James Taylor, president and chairman of the SD Card Association. “The association will continue to drive significant host market expansion while maintaining backwards compatibility with future interface development. The continuous collaboration and cooperation within the association’s ecosystem serves as a strong engine for innovation, consistency and growth.”

To speed adoption of Embedded SD, support of boot functionality requires only minor changes in the host ROM code of currently mass deployed baseband, application and multimedia processors with minimal additional changes to block device driver to support the additional functionality of embedded SD.

The Embedded SD Standard Defined:

•         Fully Compatible with SD standard – Embedded SD is fully compatible with the industry leading SDHC (SD 2.00) interface, ensuring a seamless migration from current SD to Embedded SD designs.
•         Both 3.3V and 1.8V Power Supplies – Supported for both flash and I/O power sources.
•         Boot from Embedded SD – Reliable and secure storage of boot code on an Embedded SD device reduces the number of memory components in a design, as well as the number of busses required, enabling more efficient designs.
•         Flexible Partition Mechanism – Allows multiple physical partitions accommodating varied data sources such as boot code, OS, applications and multimedia content with customizable levels of protection for each partition.  Original equipment manufacturers, mobile network operators and content providers gain optimal flexibility and control to configure the Embedded SD device for different usages and data types.
•         Protection Mechanism – Flexibility to separately configure each physical partition with different read and write/erase protection modes.
•         Data Robustness – Optional configuration of each physical partition of the Embedded SD device, including full immunity to power failure and protection of critical data (boot code, operator data, etc.).
•         Power-Saving Sleep Mode – Option to customize power needs and reduce power consumption and boost battery life.

SD Card Association
The SD Card Association is an open industry standards organization established in January 2000 by Matsushita Electric (Panasonic), SanDisk and Toshiba, and is supported by a consortium of more than 1,100 companies. The SDA’s mission is to set industry standards and promote SD product acceptance in a variety of applications. SD Memory Card standards are currently being built into a wide range of digital products such as cellular phones, audio players, automotive multimedia systems, handheld PCs and digital video and still cameras. For more information about SDA, please visit the association’s web site, www.sdcard.org. Parties interested in joining SDA are encouraged to visit the web site or contact helpdesk@sdcard.org.