Imaging a machine

...across the network
sourceforge.net

DD over Netcat for a Cheap Ghost Alternative

Recently I have seen several people ask “How can I clone my hardrive over Ethernet” on several forums that I read. There are several commercial options, such as Nortons Ghost, but I prefer the much cheaper solution of using the GNU DD utility (part of the fileutils package) and redirecting the results through Netcat (the GNU version can be found here). While it is not exactly the fastest solution out there, it is a decent enough of a solution, and I typically am never in a situation where I need to clone quickly through a network.

For the uninitiated, let me explain what these two tools do and how this whole trick plays out. First lets look at DD. From the DD Man page, DD will “copy and convert files”. Now you may be saying “So what”, but this description does not really do DD justice. DD is developed in such a way that is particularly suited to working with block devices, so it has a number of options for conversions. DD can also display the contents of those files to the standard output, or receive input via the standard input. Hard disk devices are usually accessible device files, usually under /dev, under *nix systems. So if I wanted to copy the contents of /dev/hda (usually the first IDE hard disk under Linux) to /tmp/harddisk_image, I could run the following command:

dd if=/dev/hda of=/tmp/harddisk_image

If I omit the “of=” option, it will stream onto the standard output device.  For more information about the options that DD supports, consult the man page. Alternatively, I found this article about using DD for forensic analysis.

Netcat is a tool that is called “the hackers swiss army knife”. That too is a description that I feel does not do this tool justice. The feature I will look at here is its ability to create a basic dummy connection. Basically we will create a simple listening socket on the target, and connect to it on the source system. This linux.com article describes this feature far better than I could, so I will not try to reinvent the wheel. Using the description provided by this article, if I wanted to copy a file from one system to another over a network, I would do something like this:

And on the target system first run
nc –l –p 7000 > file.txt

On the source system I would run:
cat file.txt > nc 192.168.1.120 7000 –q 10

This will do a basic copy of files from one system to another. Now, as I mentioned above, devices in *nix are files. So we are going to use this same principle to copy the contents of 1 hard drive one a machine to another, except instead of redirecting the standard inputs/outputs, we will pipe them to/from DD. To do so I would run the following commands:

On the target system I would run:
nc –l –p 7000 | dd of=/dev/hda

On the source system I would run:
dd if=/dev/hda | nc 192.168.1.120 7000 –q 10

Wait several hours and your good to go. You just saved yourself money on Ghost. There are other options here as well. If you’re worried about bandwidth, you can pipe the streams through gzip to compress the data transfer. So to do this, you would run the following commands:

On the target system:
nc –l –p 7000 | gzip –dfc | dd of=/dev/had

And on the source system:
dd if=/dev/hda | gzip  -cf | nc 192.168.1.120 7000 –q 10

In my opinion this has several advantages. If I have a central server, I can easily make disk images that I can use to remotely re-image a machine in the event of an issue. This works great in Standard Operating Environments, or environments where downtime needs to be cut down to a minimum and other fail over techniques have failed. In order to accomplish this, I would run something like this to create the disk image:

dd if=/dev/had | gzip –cf | dd of=source-192_168_1_120.img (or .zip, or whatever compressed format your comfortable with, such as tarball)

Now, when I need to re-image a machine, I can have the user either put in a special floppy disk or CD with a small image that simply boots and automatically runs the command to image to the target system. On the source system, since I already have a disk image ready to roll, I can run this command to send directly to the target system:

dd if= source-192_168_1_120.img | nc 192.168.1.120 7000 –q 10

I have been using this technique for several years now, and it works great. There are some caveats, of course. First, the target hard disk size must greater than or equal to the source disk size. If the disk is greater, you will have left over space on the drive, since this copy the partition tables exactly as they are. You can use a utility like Partition Magic, or I found this article explaining how to do this with NTFS using Knoppix. I am fairly certain that qtparted will also work on ext2/3 partitions as well. I will experiment with partition resizing and follow up with it. However, partition resizing is done at your own risk, and I would advise trying in a QA environment prior to using in any production environment. Also, if you create a disk image, you have to take into account system updates and patches. On a *nix system, you can mount the image and chroot to apply updates, however if you are using this on a Windows systems, this is not an option.
--------------
Netcat v1.10
Download windows version directly from:
http://www.loranbase.com/idx/13/034/article/Netcat-v110.html
--------------
If the new machine won't boot:
Modern Linux boxes need to have an new initrd made for the new machine.
Look here for more help
-------------
More recent versions of the netcat command (nc) will not allow the -p and -l options to be used at the same time, so instead of:
nc -l -p 9000 | dd of=/dev/sda
you would type:
nc -l 9000 | dd of=/dev/sda
If you are using nc with dd to transfer an image of a partition from one machine to the other, one of the problems is that dd and netcat won't show you a progress bar of the operation. One solution to this is to install pipe viewer http://www.ivarch.com/programs/pv.shtml by Andrew Wood. It then allows you to pipe the netcat command to the pipe viewer, allowing you to view the progress of the entire operation and for debugging.
target machine:
nc -l 9000 | pv | dd of=/dev/sda
source machine:
dd if=/dev/sda | nc 192.168.0.12 90000
-----------------
Use dd conv=noerror,sync
otherwise you'll have problems with bad sectors on your source disk.
------------------
You wrote your tutorial using the char "–" (hexa code 80 93) instead of "-" (hexa code 2D E2).
I've copied that directly from my browser to my CLI, and got this error:
–l: forward host lookup failed: Unknown host
Things work better with the good dash.
I don't know where the change came from, but thank it may be helpfull to notice it. 
============================
FOG (Free Opensource Ghost)
FOG is a free open-source cloning/imaging solution/rescue suite. A alt. solution used to image Windows XP, Vista PCs using PXE, PartImage, and a Web GUI to tie it together. Includes featues like memory and disk test, disk wipe, av scan & task scheduling.
fogproject.org
Installing FOG on Ubuntu 8.10
FOG Server Install

1 Install Ubuntu

I Wanted a GUI so I went with the desktop Ubuntu, you may want to go with Ubuntu server. So go to their website, download the workstation x86 ISO and burn it.
Video: http://www.youtube.com/watch?v=fvltHkAtW2A&fmt=18
1. English
2. Install Ubuntu
3. English
4. Select your Time Zone
5. Suggested Option
6. Use Entire Disk
7. Enter User Information and Computer information, I named mine 'foghat' with foguser
8. Install
9. Restart
10. Login

2 Install FOG Video: http://www.youtube.com/watch?v=fvltHkAtW2A&fmt=18 1. Open Firefox 2. Go to http://www.fogproject.org and download FOG. 3. Open Terminal Applications->Accessories->Terminal 4. cd Desktop (Remember Linux is case sensitive) 5. tar -xvzf fog* 6. cd fog* 7. cd bin 8. sudo ./installfog.sh 9. Select opt 2 Enter 10. N Enter 11. Default IP Enter 12. You dont need to set up a router IP but I will in case I ever use the server for DHCP. 13. Set up a DNS IP, just accept the default. 14. No do not change the default network interface.(you may not get this prompt if you have 1 nic) 15. I will not be using FOG for DHCP. (will require changing my current DHCP server.) 16. Note your IP settings and continue. 17. Enter to Acknowledge. 18. (I like to notify the FOG group, they have made a great product and deserve my feedback, choice is yours here) 19. gksu gedit /var/www/fog/commons/config.php and put your mysql password you typed during install in "MYSQL_PASSWORD", "" save and close. It has been noted that you also should change the MYSQL password here while you are at it /opt/fog/service/etc/config.php. 20. Browse to http://localhost/fog/management 21. Click install! 22. Click to log in. You can now reach this webpage from anywhere on the network where your server is installed by using it's ip address. eg http://192.168.0.100/fog/management I would reccomend putting an A record in your DNS called FOGSERVER this will make things easier to remember. default: fog/password 3 Set up DHCP (For Windows DHCP) 1. Log onto your Windows DHCP server 2. Open DHCP console. 3. Right Click Scope Options -> Configure Options 4. Scroll Down to number 66 check and give the hostname of FOGSERVER 5. Scroll down to number 67 check and give the value pxelinux.0 6. Apply and close. 4 Install FOG service on client (Host) Video: http://freeghost.sourceforge.net/videotutorials/FogServiceInstall.swf.html 1. Log onto the PC 2. Browse to http://FOGSERVER/fog/client 3. Click to download 4. Open 5. Run Setup.exe in the Fogservice folder 6. Accept defaults (FOG does not like to be installed in locations other than default) 7. Enter the ip or preferrably the hostname if you created one. FOGSERVER 8. Save Changes, Done, Close. 9. Open services Start->Run->services.msc 10. Scroll down and start "Fog Service" 11. This would also be a good time to make sure that PXE is your primary boot option in BIOS, you might need to enable it. 5 Create an image file. 1. Click on Image management. 2. New Image 3. Give it a name, description, default storage group, filename, and Image type (choose wisely). For dell computers with a utility partition you will HAVE to select multiple partition single disk. 4. Add. 6 Assign Image to Host Assign the Image to a host, this will be the image the host uses to restore and deploy from, You can assign this image to as many PCs as you want. Useful for group Deployment. 1. Click on Host management. 2. List all hosts. 3. Click Edit next to your machine. 4. In Host Image select the proper image that you created in the previous step from the dropdown. 7 Upload an image. Video: http://www.youtube.com/watch?v=jPPZr0abVfg&fmt=18 1. From another machine log onto http://FOGSERVER/fog/management login with fog/password 2. Click on Tasks (The Star) 3. List all Hosts 4. Click Upload next to the PC. 5. Upload image. 6. Click on active Tasks, and either wait for the PC to check in and autoreboot or Reboot it manually. 8 Test Restore/Deploy 1. Click on Tasks (The Star) 2. Click Deploy next to your PC. 3. Click image All computers. 4. Verify image by booting computer and making sure everything works.