Local rsync of large files to a flash thumb drive

Flash drives have a limited number of writes. There is an interesting article with some tests on Josh's blog about this.

I am working on a utility which requires me to copy large files (2Gb+) from my local computer to the flash drive. These files are Virtualbox OS images, which always change but not enough to warrant a full copy to the flash drive. In order to extend the life of my flash drive I decided to use rsync and replace only the altered blocks in the file. Note: I am using my Windows machine for these tests.

Using rsync takes significantly longer then using a direct copy. This is due to the fact that rsync has to fully read both files: source and destination in order to compare the differences.

On windows I use cwRsync to accomplish this. After installing cwRsync don't forget to append it to your path.

Control Panel -> Advanced -> Environment Variables -> Edit Path

and append:

;C:\Program Files\cwRsync\bin\

to your Path variable. You may need a restart after that in order for the changes to take effect.

Here is the command I am using:

C:\>rsync -v --stats --inplace --no-whole-file /cygdrive/c/virtualbox/xubuntu.vdi /cygdrive/d/virtualbox/xubuntu.vdi

-inplace option forces rsync to overwrite the destination without staging the received file separately. Due to the nature of thumdrives the space may be limited.

--no-whole-file option is necessary because rsync disables this option when both file systems are local, without this option the file will just be copied in full, this is because local-to-local file transfers are much faster without using the rsync block algorythm. In my case the speed is not a concern, I am trying to prolong the life of the flash drive.

/cygdrive/c and /cygdrive/d are source drive and destination drive respectively. This is equivalent to c:\ or d:\ cwRsync uses this format due to the nature of cygwin emulation. rsync is a Unix program ported to windows using cygwin.

These are some of the results:

rsync to a destination with minor changes:

Number of files: 1
Number of files transferred: 1
Total file size: 2989523456 bytes
Total transferred file size: 2989523456 bytes
Literal data: 0 bytes
Matched data: 2989523456 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 218801
Total bytes received: 437487

sent 218801 bytes  received 437487 bytes  2751.73 bytes/sec
total size is 2989523456  speedup is 4555.20

rsync to a null size destination file:

Number of files: 1
Number of files transferred: 1
Total file size: 2989523456 bytes
Total transferred file size: 2989523456 bytes
Literal data: 2989523456 bytes
Matched data: 0 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2989888465
Total bytes received: 31

sent 2989888465 bytes  received 31 bytes  6235429.61 bytes/sec
total size is 2989523456  speedup is 1.00