A download manager is nothing new, they have been around for a long time. One of the reasons to use a download manager is that they can increase the speed at which you download large files.
But you may say: Isn't my download speed limited by my Internet provider?
The answer is not necessarily. Many download content providers throttle the bandwidth of your downloads and the download manager works around this limitation by a clever use of the HTTP protocol. A download manager will open multiple threads to the host and request the file in chunks which can greatly increase the speed at which you download.
One such download manager comes as an extension to our favorite internet browser: Firefox. The program is called DownThemAll! and it sports many other features like downloading all the links on a page, or resuming paused downloads.
Friday, November 28, 2008
Sunday, November 16, 2008
Local rsync of large files to a flash thumb drive
Flash drives have a limited number of writes. There is an interesting article with some tests on Josh's blog about this.
I am working on a utility which requires me to copy large files (2Gb+) from my local computer to the flash drive. These files are Virtualbox OS images, which always change but not enough to warrant a full copy to the flash drive. In order to extend the life of my flash drive I decided to use rsync and replace only the altered blocks in the file. Note: I am using my Windows machine for these tests.
Using rsync takes significantly longer then using a direct copy. This is due to the fact that rsync has to fully read both files: source and destination in order to compare the differences.
On windows I use cwRsync to accomplish this. After installing cwRsync don't forget to append it to your path.
Here is the command I am using:
--no-whole-file option is necessary because rsync disables this option when both file systems are local, without this option the file will just be copied in full, this is because local-to-local file transfers are much faster without using the rsync block algorythm. In my case the speed is not a concern, I am trying to prolong the life of the flash drive.
/cygdrive/c and /cygdrive/d are source drive and destination drive respectively. This is equivalent to c:\ or d:\ cwRsync uses this format due to the nature of cygwin emulation. rsync is a Unix program ported to windows using cygwin.
These are some of the results:
rsync to a destination with minor changes:
I am working on a utility which requires me to copy large files (2Gb+) from my local computer to the flash drive. These files are Virtualbox OS images, which always change but not enough to warrant a full copy to the flash drive. In order to extend the life of my flash drive I decided to use rsync and replace only the altered blocks in the file. Note: I am using my Windows machine for these tests.
Using rsync takes significantly longer then using a direct copy. This is due to the fact that rsync has to fully read both files: source and destination in order to compare the differences.
On windows I use cwRsync to accomplish this. After installing cwRsync don't forget to append it to your path.
Control Panel -> Advanced -> Environment Variables -> Edit Path
;C:\Program Files\cwRsync\bin\
Here is the command I am using:
C:\>rsync -v --stats --inplace --no-whole-file /cygdrive/c/virtualbox/xubuntu.vdi /cygdrive/d/virtualbox/xubuntu.vdi
--no-whole-file option is necessary because rsync disables this option when both file systems are local, without this option the file will just be copied in full, this is because local-to-local file transfers are much faster without using the rsync block algorythm. In my case the speed is not a concern, I am trying to prolong the life of the flash drive.
/cygdrive/c and /cygdrive/d are source drive and destination drive respectively. This is equivalent to c:\ or d:\ cwRsync uses this format due to the nature of cygwin emulation. rsync is a Unix program ported to windows using cygwin.
These are some of the results:
rsync to a destination with minor changes:
Number of files: 1
Number of files transferred: 1
Total file size: 2989523456 bytes
Total transferred file size: 2989523456 bytes
Literal data: 0 bytes
Matched data: 2989523456 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 218801
Total bytes received: 437487
sent 218801 bytes received 437487 bytes 2751.73 bytes/sec
total size is 2989523456 speedup is 4555.20
Number of files: 1
Number of files transferred: 1
Total file size: 2989523456 bytes
Total transferred file size: 2989523456 bytes
Literal data: 2989523456 bytes
Matched data: 0 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2989888465
Total bytes received: 31
sent 2989888465 bytes received 31 bytes 6235429.61 bytes/sec
total size is 2989523456 speedup is 1.00
Migrating Paltalk to a jailed Virtual Machine emulation with Xubuntu and Virtualbox
Every few months I would get complaints from my mom about how slow her computer ran. How it was freezing and dropping the internet connection. Every few months I would find a slew of Trojans and random spyware on the computer. Her Norton Anti Virus license expired so I installed AVG. Things were ok again.
Few months later she called me again. She said the computer was incredibly slow again. I found a "slick" piece of spyware this time by the name of Antivirus Pro 2009. This program would pretend it was a legitimate anti virus and firewall tool. It had hijacked her internet connection. It would notify the user of a detected virus it found and would demand $$$ to remove it. AVG didn't catch it and SpyBot had no luck with it either. This was the final straw for me. I was going to reformat her computer and install Ubuntu instead.
The only issue was that she is part of the Paltalk community. Palltalk is a chat client which she uses to talk to all her friends with. Palltalk provides moderated channels where people can play their music to their friends. It has a very large community. Palltalk is a windows only program though and I was in for a surprise of how challenging this was going to be. I dedicated a day to rebuild her computer and I was off.
Computer specs:
512 Mb RAM
3000 Sempron AMD processor
Onboard video card with shared ram
I stripped one of my old machines and bumped the computer to 1Gb of RAM and plugged in an ATI Radeon 9600 AGP video card I had laying around.
I used a 16Gb thumb drive to back up her documents and music. Burned a disk with the latest version of Xubuntu 8.10. Reason I chose Xubuntu over Ubuntu is it's small memory footprint, being that I needed it to run on an older computer.
Xubuntu install was flawless. Everything worked perfectly without any tweaking. Video, sound and the wireless card worked out of the box. This was definitely the easiest install of an operating system I've ever experienced!
I downloaded the latest version of Mozilla Firefox and installed Flash. Tried going to Paltalk Express, the sound notifications worked but there was an issue with the streaming audio. I discovered that this was due to missing Java support for Firefox. I got Java working in Mozilla but there was still no sound in Paltalk Express. Java sound worked in my browser when going to other Java applets with sound but no such luck in Paltalk Express. I experimented with aoss as well but had no luck. No matter what I tried I could not get Java sound streaming to work.
Paltalk express had a nasty bug as well. Every once in a while when I tried to load it, the Firefox would never finish loading but would memory leak till the computer started to crawl. In order to fix the issue I would have to kill firefox, go to Flash Settings and delete the locally stored data and try again. This is the point at which I abandoned the idea of using Paltalk Express.
Next thing to try was Wine. Wine is a an amazing piece of software that has served me well so many times before. I even managed to get Eve-Online going trough wine in the past which is amazing considering that Eve-Online client is way more advanced then a Paltalk client. Paltalk client under Wine would log in to the service but as soon as I attempted to join a room it would freeze. I found there was a .dll missing, and after a few hours of trying different things it was clear I wasn't going to have much luck with getting Paltalk working under wine. I almost think the folks at Paltalk purposely worked to prevent Paltalk from ever working on anything other then Windows. In retrospect I think it didn't work because Paltalk depends on Internet Explorer to display all it's annoying ads, as I was getting tired I decided to give up on Palltalk under Wine.
It was time to give virtual machines a go. I've used Vmware four years ago and had generally good luck with it but I needed to use Open Source alternatives. I've tried QEMU first but Windows XP install hung towards the end.
Virtualbox was next. Install worked perfectly. I had Windows XP running flawlessly inside of Xubuntu. Paltalk client was installed and working without any issues. Windows XP was working great as a guest system with 256Mb allocated to it in Virtualbox settings. And I instructed my mom not to use Windows for anything else other then Paltalk.
Windows XP performed great trough Virtualbox emulation, considering Virtualbox is a processor emulator and the Sempron 3000 does not support AMD-V technology.
On the Xubuntu side I installed Songbird (iTunes clone) as well as Catfish for fast finding of her music. And I shared her music folder with the Windows XP guest. I also installed Tkmixer as it allows her to easily switch between mic or line as a sound input source.
Installing Virtualbox guest additions was a bit problematic. In guest window Devices -> Install Guest Additions.. the download link was broken. I went to http://download.virtualbox.org/virtualbox/2.0.4/ and downloaded it manually. And then placed it in the right location for the "Install Guest Additions.." to work.
After I was done I made a backup of her Virtualbox windows xp image (.vdi file). If anything should happen with her Windows XP system due to a virus infection or similar it will be a matter of rewriting the active .vdi file from the backup copy. A fix which requires 5 minutes of my time and which can be accomplished remotely.
She has been very happy with Xubuntu. It took some getting used to but she quickly learned her way around it. The computer performed much better as well according to her. Due to both the hardware upgrades I did as well as the lack of unnecessary software running on the machine.
Few months later she called me again. She said the computer was incredibly slow again. I found a "slick" piece of spyware this time by the name of Antivirus Pro 2009. This program would pretend it was a legitimate anti virus and firewall tool. It had hijacked her internet connection. It would notify the user of a detected virus it found and would demand $$$ to remove it. AVG didn't catch it and SpyBot had no luck with it either. This was the final straw for me. I was going to reformat her computer and install Ubuntu instead.
The only issue was that she is part of the Paltalk community. Palltalk is a chat client which she uses to talk to all her friends with. Palltalk provides moderated channels where people can play their music to their friends. It has a very large community. Palltalk is a windows only program though and I was in for a surprise of how challenging this was going to be. I dedicated a day to rebuild her computer and I was off.
Computer specs:
512 Mb RAM
3000 Sempron AMD processor
Onboard video card with shared ram
I stripped one of my old machines and bumped the computer to 1Gb of RAM and plugged in an ATI Radeon 9600 AGP video card I had laying around.
I used a 16Gb thumb drive to back up her documents and music. Burned a disk with the latest version of Xubuntu 8.10. Reason I chose Xubuntu over Ubuntu is it's small memory footprint, being that I needed it to run on an older computer.
Xubuntu install was flawless. Everything worked perfectly without any tweaking. Video, sound and the wireless card worked out of the box. This was definitely the easiest install of an operating system I've ever experienced!
I downloaded the latest version of Mozilla Firefox and installed Flash. Tried going to Paltalk Express, the sound notifications worked but there was an issue with the streaming audio. I discovered that this was due to missing Java support for Firefox. I got Java working in Mozilla but there was still no sound in Paltalk Express. Java sound worked in my browser when going to other Java applets with sound but no such luck in Paltalk Express. I experimented with aoss as well but had no luck. No matter what I tried I could not get Java sound streaming to work.
Paltalk express had a nasty bug as well. Every once in a while when I tried to load it, the Firefox would never finish loading but would memory leak till the computer started to crawl. In order to fix the issue I would have to kill firefox, go to Flash Settings and delete the locally stored data and try again. This is the point at which I abandoned the idea of using Paltalk Express.
Next thing to try was Wine. Wine is a an amazing piece of software that has served me well so many times before. I even managed to get Eve-Online going trough wine in the past which is amazing considering that Eve-Online client is way more advanced then a Paltalk client. Paltalk client under Wine would log in to the service but as soon as I attempted to join a room it would freeze. I found there was a .dll missing, and after a few hours of trying different things it was clear I wasn't going to have much luck with getting Paltalk working under wine. I almost think the folks at Paltalk purposely worked to prevent Paltalk from ever working on anything other then Windows. In retrospect I think it didn't work because Paltalk depends on Internet Explorer to display all it's annoying ads, as I was getting tired I decided to give up on Palltalk under Wine.
It was time to give virtual machines a go. I've used Vmware four years ago and had generally good luck with it but I needed to use Open Source alternatives. I've tried QEMU first but Windows XP install hung towards the end.
Virtualbox was next. Install worked perfectly. I had Windows XP running flawlessly inside of Xubuntu. Paltalk client was installed and working without any issues. Windows XP was working great as a guest system with 256Mb allocated to it in Virtualbox settings. And I instructed my mom not to use Windows for anything else other then Paltalk.
Windows XP performed great trough Virtualbox emulation, considering Virtualbox is a processor emulator and the Sempron 3000 does not support AMD-V technology.
On the Xubuntu side I installed Songbird (iTunes clone) as well as Catfish for fast finding of her music. And I shared her music folder with the Windows XP guest. I also installed Tkmixer as it allows her to easily switch between mic or line as a sound input source.
Installing Virtualbox guest additions was a bit problematic. In guest window Devices -> Install Guest Additions.. the download link was broken. I went to http://download.virtualbox.org/virtualbox/2.0.4/ and downloaded it manually. And then placed it in the right location for the "Install Guest Additions.." to work.
After I was done I made a backup of her Virtualbox windows xp image (.vdi file). If anything should happen with her Windows XP system due to a virus infection or similar it will be a matter of rewriting the active .vdi file from the backup copy. A fix which requires 5 minutes of my time and which can be accomplished remotely.
She has been very happy with Xubuntu. It took some getting used to but she quickly learned her way around it. The computer performed much better as well according to her. Due to both the hardware upgrades I did as well as the lack of unnecessary software running on the machine.
Subscribe to:
Posts (Atom)
Add This
