Thursday, August 26, 2010
flash memory devices: Cache or not -Optimize for speed
Under Windows XP writing of lots of small files to a 'Removable' pen drive is extremely slow, while it's fast under Windows 2000. Writing large files there is no appreciable difference.
Obviously Windows XP doesn't enable a write cache for USB drives that appear as 'Removable'. The settings 'Optimize for quick removal' or 'Optimize for performance' doesn't seem to make any difference then, except that the latter enables the user to format 'Removable' USB drives with NTFS. But with NTFS Windows XP enables a write cache, writing small files becomes lightning fast.
But actually NTFS isn't suitable for flash medias because as journalling file system it has some overhead that wears out flash memory. But Windows XP optimizes and bundles write accesses to pen drives only when they are NTFS formatted, so I see NTFS as the right choice.
Furthermore NTFS stores small files together with the file information so they are written into the same flash block which is the best that can happen.
A drawback with NTFS and flash medias is that NTFS saves the access time when you read a file, so a simple read access causes flash wear out. In fact it is not as dramatically because Windows writes the data not immediately onto the media. It does it when it has to update something else there, when one hour is over on when a media is unmounted. Actually I've never seen XP writing a last access time to a USB drive.
The last access time is also updated on 'directory listing' (whatever this means). This can be disabled, unfortunately for all drives only:
Create a new DWord value under [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem] called NtfsDisableLastAccessUpdate and set it to 1. Or download the following REG file and doubleclick it:
To format a pen drive with NTFS there is no need to switch to 'optimize for speed'. Using the commandline tool CONVERT you can convert a FAT formatted drive to NTFS. Sample:
convert u: /fs:ntfs
But XP's CONVERT creates a NTFS with a cluster size of 512 Bytes when the FAT clusters are not 4K aligned.
If the drive is FAT formatted under XP, then the clusers are 4K aligned and XP's CONVERT works fine. W2K's CONVERT seems to creates 512 Bytes clusters in any case.
Using the Windows format dialog you can choose: 2048 or 4096 Bytes are ok. With a cluster size above 4096 Bytes file compression and encryption becomes unavailable.
Of course a NTFS formatted drive doesn't work under Windows 95/98/ME. Furthermore devices with a standalone functionality as MP3 players or image tanks will not work with NTFS.
If both, format and convert fail while formatting with FAT/FAT32 works, then the drive is a fake with a fraction of real memory only. The NTFS file system is written into the middle of the drive while the FAT is written to at the begin of the drive. So, if the drive is fake then the NTFS file system is written into non existent memory...
The effect of having a write cache or not depends on the USB drive: Hard drives have an 2 or 8 MB of cache RAM which prevents to much performance impact. Today's USB flash have no such cache, so their access times are very important for their real live speed.By means of HD Tune you can check the read access times. Here a screenshot of the bad example 'Corsair Flash Voyager' (2GB) - have a look at the yellow dots above the diagram...
It's from 2005, the current exemplars are better.
To test the file cache behaviour I've made FileCacheTest
It can read and write a test file using different flags for the API function CreateFile and using different block sizes. Remember the difference between writing a file for the first time and to an existing file: Writing the first time, the file size has to be adjusted after each block.
The most remarkable effects under XP are:
* no buffering of accesses to the file system on FAT formatted removable USB drives
* writing to FAT formatted removable USB drives using a block size below 4096 Bytes is cached, others are not (for NTFS drives the magic block size is about 512 KB, under Windows 7 32 MB!)
* using flag FILE_FLAG_RANDOM_ACCESS causes a blows up of the file cache's working set
* using flag FILE_FLAG_NO_BUFFERING is the most efficient way to read and write large files
* for 'removable' USB drives the removal policies 'Optimize for speed' and 'Optimize for quick removal' have absolutely no effect on the cache's behaviour