Migrating to a new SSD without data loss

I have a Samsung N150 netbook that came with Windows 7 preinstalled on a 250GB hard disk.  I created a bunch of new ext4 partitions on this disk and was running Kubuntu on it.  I almost never used Windows, but I had kept it on the disk anyway.  Last week I thought of upgrading to an SSD, and bought a 240GB SSD.  This post is to document how I copied over the Windows installation and recovery partition to the SSD before swapping the disks.

I connected the SSD to my computer using an USB interface and ran sudo fdisk -l to see the partition tables of both disks.

% sudo fdisk -l

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd937cf59

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048    31459327    15728640   27  Hidden NTFS WinRE
/dev/sda2   *    31459328    31664127      102400    7  HPFS/NTFS/exFAT
/dev/sda3        31664128   232622079   100478976    7  HPFS/NTFS/exFAT
/dev/sda4       232624126   488394751   127885313    5  Extended
/dev/sda5       480397312   488394751     3998720   82  Linux swap / Solaris
/dev/sda6       232624128   291215359    29295616   83  Linux
/dev/sda7       291217408   480391167    94586880   83  Linux

Partition table entries are not in disk order

Disk /dev/sdb: 240.1 GB, 240057409536 bytes
255 heads, 63 sectors/track, 29185 cylinders, total 468862128 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00045231

   Device Boot      Start         End      Blocks   Id  System

/dev/sda is the original hard disk that came with the laptop, and /dev/sdb, which is empty currently, is the new SSD.  I have to clone the first 3 partitions (/dev/sda1, /dev/sda2, and /dev/sda3) bit-by-bit to retain all the preinstalled stuff — this includes Windows 7 installation and the recovery partition.  Replicating the Windows partitions is the tricky part, so this post will describe that in detail.  Copying data from Linux partitions can be done with a simple rsync.

The first step is to create partitions on the new disk that resemble the old disk.  I followed the fdisk guide of TLDP and created the first 3 partitions.  Now fdisk -l shows this configuration:

% sudo fdisk -l      

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd937cf59

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048    31459327    15728640   27  Hidden NTFS WinRE
/dev/sda2   *    31459328    31664127      102400    7  HPFS/NTFS/exFAT
/dev/sda3        31664128   232622079   100478976    7  HPFS/NTFS/exFAT
/dev/sda4       232624126   488394751   127885313    5  Extended
/dev/sda5       480397312   488394751     3998720   82  Linux swap / Solaris
/dev/sda6       232624128   291215359    29295616   83  Linux
/dev/sda7       291217408   480391167    94586880   83  Linux

Partition table entries are not in disk order

Disk /dev/sdb: 240.1 GB, 240057409536 bytes
255 heads, 63 sectors/track, 29185 cylinders, total 468862128 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00045231

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    31459327    15728640   27  Hidden NTFS WinRE
/dev/sdb2   *    31459328    31664127      102400    7  HPFS/NTFS/exFAT
/dev/sdb3        31664128   232622079   100478976    7  HPFS/NTFS/exFAT

The partitions in the new disk are of the same size and same type as in the old one.  /dev/sdb2 is bootable as is /dev/sda2.  (It won’t boot yet though, since the disk has no OS yet.)  Now to copy the data bits over.  I first unmounted all three partitions.  This is critical because changing data underneath when it’s being copied over is a darn good recipe for data corruption.

dd is the low-level data copying utility I used to clone the partitions.  Copying the data over was as simple as running these commands one by one.  (Swapping if and of can result in wiping out all data from the old partition.  dd cannot even know if you’re passing wrong arguments to it.)

% sudo dd if=/dev/sda1 of=/dev/sdb1 conv=notrunc
% sudo dd if=/dev/sda2 of=/dev/sdb2 conv=notrunc
% sudo dd if=/dev/sda3 of=/dev/sdb3 conv=notrunc

Copying can be painfully slow since we are moving hundreds of GBs around.  Blog O’ Matty has a post that shows how to find status of a running dd command.  Essentially you’d send SIGUSR1 signal to the dd process and it’d print the current status of the transfer.  One of the commenters suggests running sudo pkill -SIGUSR1 dd so that you don’t have to think about process IDs.

Once this was done, I installed Kubuntu on the SSD using the standard installation process, and everything went just fine.  Windows doesn’t boot probably because it thinks mine is a pirated copy.  (Shows an error saying some ‘important’ hardware has gone missing.)  But I can boot into the recovery partition, so I can restore factory settings to get Windows running again when I want it.

I restored all my installed software from the package selection list I had already generated.  That’s it... the computer is exactly like it was before with all my programs and configuration.

Tweaking stack size of Linux processes to reduce swapping

Since upgrading to Kubuntu 11.10, my laptop has been slow.  Slow because it’s been accessing the hard disk a lot.  I incidentally opened system monitor yesterday and found that more than 1GB of swapping space was in use although only about 1.1GB of the total 2GB RAM was in use.  That doesn’t sound right.  The computer shouldn’t swap when about half of the RAM is unused.

My friend Abhay had once told me about thread stack size configuration of Linux (Unix?) processes.  This configuration specifies how much RAM is given to each thread for its stack.  I ran the following command to see how much was the current stack size:

% ulimit -s
8192

That’s 8192KB allocated for each thread.  With some Googling around I figured this was a huge number.  Windows allocates only 1MB by default.  For a machine that’s low on RAM like mine, 8MB for stack is ludicrous.  I decided to make it 2MB instead.  Unsurprisingly, I wasn’t the first to try to do something like this; a thread on LinuxQuestions.org explained that I can edit /etc/security/limits.conf to set the default size.

I added the following lines to my /etc/security/limits.conf:

* soft stack 2048
* hard stack 2048

(Only root can modify this file; you’ll need to use sudo.)  To apply the configuration changes I restarted the machine.  After restarting, now my machine is using about 1.4GB of RAM and about 80MB of swap.  No need to mention, everything is fast as it used be.

Opening new browser tab when Caps Lock key is pressed

After using a Chromebook for a while, I realised how useful mapping Caps Lock key to opening a new browser tab can be.  Of course, it’s possible to set up key bindings to achieve this in Linux.

First, I set up .Xmodmap so that pressing Caps Lock is interpreted as the same as pressing Calculator key on my multimedia keyboard.  I chose Calculator key because it doesn’t currently do anything, and I don’t use it at all.  I added the following lines to my ~/.Xmodmap file.

remove Lock = Caps_Lock
keysym Caps_Lock = XF86Calculator

Now, pressing Caps Lock would be the same as pressing Calculator key.

We need to make pressing Calculator key send Ctrl+T keystrokes instead.  This can be done in KDE by defining a new global shortcut.  In KDE 4.7, this is done by navigating to System Settings > Shortcuts and Gestures > Custom Shortcuts.  Define a new Command/URL global shortcut.  Use Caps Lock as the trigger shortcut (it would show as Calculator in the UI).  Specify

/usr/bin/xte "keydown Control_L" "key t" "keyup Control_L"

as the command to run.  (You’d have to install xte if it isn’t already installed on your machine.)  That’s it; now pressing Caps Lock anywhere within KDE would send Ctrl+T keystrokes instead.

A few tips:

• You can use xmodmap -pk command to see the list of all available keys.
• Be sure to select a key that’s actually present on your keyboard; my laptop does not have a calculator key, so I am using the battery key instead.  Any key that's present in the keyboard but not currently in use would do.
• After you have modified your ~/.Xmodmap, you’ll have to log out and log back in for the mappings to apply.  Alternatively, you can apply the configuration to your current session from the command line, e.g. by running xmodmap -e "remove Lock = Caps_Lock".

zsh syntax highlighting

I like colours.  I have aliased all frequently used commands like ls, grep, etc. by adding flags to show colours in the output.  I have set my PS1 in such a way that the prompt is in a different colour.  It makes it easy for me to see where the prompt ends and the command starts.  When I heard about fish and that it does syntax highlighting, I was tempted to switch to it.  But I couldn’t because it didn’t do certain things I needed.  (I don’t remember the details now.)  Soon after that I found out that it’s very simple to add syntax highlighting to zsh!

All you have to do is download the code from zsh-syntax-highlighting project and “source” it in your .zshrc.  But I wasn’t happy with their defaults.  By default this script underlines path names, but I hate underlining because it makes text less readable.  I also didn’t like their choice of blue colour for globs.  On my black terminal, blue is hardly readable.  Customising the formatting was easy too; I only had to change the value of a variable.  This is what my .zshrc has now, and syntax highlighting works like a charm.

source ~/dload/src/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
ZSH_HIGHLIGHT_STYLES[globbing]='fg=yellow'
ZSH_HIGHLIGHT_STYLES[path]='bold'

You can find the list of different syntax highlighting options in this file.

Terminating scripts when any individual command fails

Let’s say you have a script that builds a project, runs all tests, and pushes the binary to a staging/production server.  If the build fails or a test fails you’d want the script to stop immediately.  Pushing a binary that failed some tests is obviously wrong.  You can check for a command’s return value using if and terminate your script.  But doing that for every command in the script would make your script less readable and more prone to bugs.

Shells provide a clean solution for this use case: you can set a script-level option to stop the script execution if any command you invoke from the script exits with a non-zero status.  You do that in bash using

set -e

and in zsh using

setopt err_exit

So your script would essentially look like this:

#!/bin/bash
set -e
make
make test
make push

zsh also has a err_return option that can be set to make a function return (as opposed to terminating the whole script) when a command invoked by a function fails.

Vim's text objects

Let's say I have this line in a file:

logging.info("some boring message")

and I want to change it to:

logging.info("request served")

For a long time I did it this way:
1. keep the cursor on 's' of 'some'
2. type ct" (which means change till (the first) " character)
3. type request served.

Recently I figured there's an easier/faster way:
1. keep the cursor anywhere inside the "some boring message" string
2. type ci" (which means change inside quotes)
3. type request served.

Like every Vim feature, this is just one among a dozen or so possible selections.  Check out text objects section in Vim manual.

Shortcut key to switch to any app

I use the command line quite a bit, so I always keep a Konsole window running.  Since I frequently switch to the Konsole window, I thought it'd be much faster if I can bind a global shortcut key.  Like with most things in Linux, the solution is only a Google search away :)

I had to install wmctrl first (sudo apt-get install wmctrl on Ubuntu).  And then I bound the shortcut key Ctrl+Shift+K from my KDE's settings dialog to run the command wmctrl -x -a konsole.Konsole.  -x says that I would be specifying windows using their WM_CLASS values; -a activates the window that follows it.  To get the list of currently open window with their WM_CLASS values, I used the command wmctrl -xl.

Finding information about commands you use

Q: How to find out where the binary of a command I'm running?
A: You can use type command (available on both zsh and bash):

% type ls
ls is an alias for ls -h --color=auto
% type cat
cat is /bin/cat
% type alias
alias is a shell builtin

Q: Sometimes I want to know if a command is a shell script or a compiled binary.  How do I do that?
A: If you use zsh, you can use =command to get to command's full path.

% file =backup
/home/manki/d/bin/backup: a /bin/rbash script text executable
% # To show you what =backup actually translates to
% echo =backup
/home/manki/d/bin/backup

If you use bash, you can use type command within backticks or the equivalent $(...).

$ file `type -p backup`
/home/manki/d/bin/backup: a /bin/rbash script text executable

When you have aliases, this can get tricky.  On bash I don't know how to do this, but zsh is smart enough to find the executable even when you have aliases set up.  For instance, I have aliased ls to ls -h --color=auto, but zsh gives me the right binary for =ls:

% file =ls    
/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped

Saving the input to `less' command

less is good for viewing long outputs from other programs.  There may be times when you pipe the output of a program to less and realise only later that you want the output in a file... maybe because you need to send the output to someone via email.

You can quit less and change the command line to output to a file.  But there are better/faster options.  less supports an s command to save its input to a file.  Just press s, enter a file path to save the text to, and you're done.

Watching Apple's Keynote Videos in Linux

Essentially, you’d have to dig through Apple site’s HTML/JavaScript and find the video URL for Windows.  And then pass that URL to a stand-alone media player like VLC.

Apparently Chrome for Linux has a QuickTime plugin!  (Or it’s bundled with Ubuntu; I am not sure.)  Watch Sep 2012 event (iPhone 5 launch) is available at goo.gl/Nn77F.

Watch Mar 7, 2012 iPad event video using this command:

vlc 'http://stream.qtv.apple.com/events/mar/123pibhargjknawdconwecown/12oihbqeorvfhbpiubqnfv3_650_ref.mov'

Watch Oct 4, 2011 iPhone event video using this command:

vlc 'http://stream.qtv.apple.com/events/oct/11piuhbvdlbkvoih10/11mnbzxcbnkhbvshdfeuygxvst10_350_ref.mov'

-----
If you have a Linux computer, you cannot watch Apple’s WWDC 2011 keynote video from Apple’s web site.  Because the site picks a video URL based on what your browser/OS is, but it fails miserably if you don’t run an Apple OS or Windows.  So, if you want to see the video, open http://stream.qtv.apple.com/events/jun/11piubpwiqubf06/11piunpaiubsdvadibvas06_350_ref.mov on VLC Media Player and you’ll be good.

Autosaving Vim sessions

Accidentally closing a Vim window with several open files can mean losing a lot of state.  Reopening all files again to continue editing can be daunting, if not in real at least while thinking about it.  Add the following line to your .vimrc:

autocmd VimLeave * mksession! ~/.vim/last-session.vim

You can resume an accidentally closed Vim session by using the command:

vim -S ~/.vim/last-session.vim

Map F1 to Esc in Vim

If you are like me, you press F1 key instead of Esc quite often.  It can be especially annoying when using Vim.  Add the following line to your .vimrc:

imap <F1> <Esc>

Now pressing F1 is treated as if you pressed Esc.  When you really need help you can use :help command instead.

pushd, popd, and dirs commands

If you spend a lot of time in the command line, you will have to cd to different directories frequently.  Instead of the cd you can use pushd command to switch to a different directory.  pushd command pushes the new directory into a "directory stack".  Later you can return to your original location using popd command.  Both pushd and popd print the latest contents of the directory stack.  You can also use dirs command see the directory stack.  Items in the directory stack are indexed from 0.  You can reference any directory in the stack by typing ~ followed by the index.  A sample session using the directory stack (user input is in bold font):

~% pushd /tmp
/tmp ~
/tmp% pushd /etc
/etc /tmp ~
/etc% pushd /var
/var /etc /tmp ~
/var% dirs
/var /etc /tmp ~
/var% popd
/etc /tmp ~
/etc% pwd
/etc
/etc% dirs
/etc /tmp ~
/etc% cd ~1
pingala /tmp % dirs
/tmp /etc ~

As the directory stack becomes larger, finding the index of a directory in the stack becomes harder.  I use -v flag to print the numeric index along with each item:

/tmp% dirs -v
0       /tmp
1       /etc
2       ~

If you are like me, you wouldn't remember to use pushd.  For users like us, the zsh has an auto_pushd option.  When it's set, all cd commands are treated like pushd commands.  With that option enabled, I use the shell normally, but zsh remembers the directories I have recently been to.  I can use dirs -v any time to get a list of recent directories and use the ~ notation to return to one those directories.

Controlling permissions of files on a drive

If you have mounted FAT or NTFS partitions on your Linux machine, you'd have noticed this: all files and directories on those partitions have their permissions set to 777, meaning anyone can do anything.  I get annoyed when files that are not commands have execute permissions.  In addition to being a security risk, I find them aesthetically displeasing.

There's an easy solution, of course.  We can make all files from that partition non-executable by setting an appropriate fmask value in its fstab entry.  Mine looks like this:

/dev/sda3 /mnt/win ntfs-3g defaults,fmask=111 0 0

I have set the fmask to 111, which is execute permission for owner, group, and everyone. Since masks specify what operations are not permitted, no file in that drive would be executable.

There's also a dmask that restricts directory permissions.

/tmp is your friend

Do you end up with a lot of unnecessary files in your home directory, e.g. in your ~/Downloads folder?  Make it a habit to use /tmp for files that you don't want to keep for a long time.  By default, Ubuntu empties  /tmp every time it boots up so you don't keep accumulating junk.  (If you'd rather keep your temporary files for a longer period, you can change TMPTIME variable in your rcS file.)

Counting files

I have organized my photos this way: there's a Photos directory, under which there's one subdirectory for each year.  In addition to the JPEG images, these directories also contain files like Picasa.ini.  So, to find out how many pictures I have I can use the find command:

~/d/Photos % find . -type f -iname \*.jpg | wc -l
5101

This command counts the files that have .jpg file extension.  Let's say I want to know the years in which I took more photos, and the years in which I took relatively lesser number of photos.  That can be done using the following command:

% for year in *
do
  echo -n "$year "
  find $year -type f -iname \*.jpg | wc -l
done
2006 98
2007 983
2008 1978
2009 1128
2010 749
2011 165

If you use zsh, you can modify the command to include only the last few years:

% for year in <2009-2011>                   
do
  echo -n "$year "
  find $year -type f -iname \*.jpg | wc -l
done
2009 1128
2010 749
2011 165

Entering command mode in shells with vi bindings

If you use zsh or bash with vi key bindings, you know that by default the shell is in insert mode.  Every time you need to run some editing command you would first press Esc to enter command mode.  I recently found that just pressing Alt along with the first key in your command will put you in command mode.

For example, if you want to go up in command history, press Alt+k.  That will put you in command mode and take you to the previously run command.  From there on, pressing k will scroll up the command history.  One very useful shortcut I use is Alt+I (which is Alt+Shift+i): this is equivalent to Ctrl+a in Emacs mode or pressing the Home key.

Quick access to frequently visited directories

Define hashes for directories you frequently cd to from the command line, and access them as ~hash_name.  For example, my .zshrc has:

hash -d fx=~/dload/app/comm/firefox
hash -d prog=~/d/prog
hash -d personal=~/d/personal

cd ~fx will take me to ~/dload/app/comm/firefox, cd ~prog will take me to ~/d/prog, and so on.

Changing Compiz's Alt+Tab animation

By default Compiz uses "Static Application Switcher" module for handling Alt+Tab.  At least that's how it's on Ubuntu.  When pressing Alt+Tab, the UI would show only the target window and hide everything else.  But this doesn't quite match my mental model of the windows I have on the screen.  (It could be because I don't maximize most of my windows and have them all overlapped.  So at any time I have several windows scattered on the desktop.)

I disabled Static Application Switcher and selected Application Switcher instead.  While this preserved the window layout, the animation was jarring.  Setting the value of "Zoom" parameter to 0.0 will disabled the zoom effect and everything is good now.

PS: You need to install compizconfig-settings-manager package.  After installation open System > Preferences > CompizConfig Settings Manager or run ccsm command.