Moving Virtual Machine Disk to Physical Partition

The following steps are tested on Ubuntu 14.04 host OS and Ubuntu 16.04 VirtualBox image. This process should work on any other systems with appropriate changes.

1. Convert VirtualBox drive to raw image. Find out the virtual disk filename from the settings of the virtual machine.

$ VBoxManage internalcommands converttoraw vmdisk.vdi disk.img

2. In most cases, the OS partition will have an offset due to MBR or EFI. Determine this offset and note it down. We will need that offset in next steps.

$ parted disk.img
WARNING: You are not superuser. Watch out for permissions.
GNU Parted 2.3
Using /home/ninja/Documents/vvv/disk.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit
Unit? [compact]? B
(parted) print

The output should look something like:

Model: (file)
Disk /home/ninja/Documents/vvv/disk.img: 32212254720B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1048576B 30064771071B 30063722496B primary ext4 boot
2 30064771072B 32212254719B 2147483648B extended
5 30065819648B 32212254719B 2146435072B logical linux-swap(v1)

(parted) q

3. In the above case, offset of the first partition (in which Ubuntu 16.04 is installed) is 1048576. I prefer renaming the file by suffixing the offset to the filename.

$ mv disk.img disk_1048576.img

4. Using GParted, create a new partition to store this image.
5. Mount the disk image

$ sudo mkdir /media/vmimage
$ sudo mount -o loop,offset=1048576 -t ext4 disk_1048576.img /media/vmimage

6. Mount the newly created partition. In the following command, /dev/sda4 is the new partition

$ sudo mkdir /media/iCub
$ sudo mount /dev/sda4 /media/iCub

7. Copy files from image to physical partition using either rsync of direct copy. Warnings related to special files while copying can be ignored.

$ sudo rsync -rtvpl /media/vmimage/ /media/iCub/

8. Mount /dev, /proc, and /sys

$ sudo mount --bind /dev /media/iCub/dev
$ sudo mount --bind /proc /media/iCub/proc
$ sudo mount --bind /sys /media/iCub/sys

9. Run bklid to find out the UUID of new partition (/dev/sda4 in this case). This is required in step 11.

$ sudo blkid
/dev/sda1: UUID="E1E2-36E0" TYPE="vfat"
/dev/sda2: UUID="ba551082-be43-426b-ac14-d32ef82f4648" TYPE="ext4"
/dev/sda3: UUID="e6a0fc82-0090-42e2-8494-82231d563f5b" TYPE="swap"
/dev/sda4: LABEL="iCub" UUID="b42fd8eb-dd33-455e-b400-8ef27932a046" TYPE="ext4"

10. chroot into the partition to fix fstab and grub

$ sudo chroot /media/iCub

11. As the disk was mounted on the virtual machine, its UUID was different. Now that we are mounting this disk on a physical partition, we should replace the UUID in /etc/fstab so that the system knows where to boot from

root:/# vi /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#

# MODIFIED NEXT LINE TO REFLECT UUID OF /dev/sda4
UUID=ba551082-be43-426b-ac14-d32ef82f4648 / ext4 errors=remount-ro 0 1

# MODIFIED NEXT LINE TO REFLECT UUID OF swap PARTITION
UUID=e6a0fc82-0090-42e2-8494-82231d563f5b none swap sw 0 0

# MODIFIED NEXT LINE TO REFLECT UUID OF EFI PARTITION
UUID=E1E2-36E0 /boot/efi vfat defaults 0 1

12. generate grub config and update grub.

root:/# grub-mkconfig
root:/# update-grub
root:/# update initramfs -u

13. Exit from chroot and unmount the partitions

root:/# exit
$ sudo umount /media/iCub/dev
$ sudo umount /media/iCub/proc
$ sudo umount /media/iCub/sys
$ sudo umount /media/iCub

14. run boot-repair and select recommended repair from the menu

$ sudo add-apt-repository ppa:yannubuntu/boot-repair
$ sudo apt-get update
$ sudo apt-get install -y boot-repair && boot-repair

15. Reboot your system and select the new OS from grub menu
16. Graphics might fail to load, drop to shell and run the following. In this case, icub is a user configured in the virtual machine, replace it with your user id from the virtual machine. If you have multiple users in VM, fix folder permissions for all the users.

root:/# chown -R icub:icub /home/icub
root:/# rm /home/icub/.Xauthority

root:/# rm -rf /var/cache
root:/# apt-get update
root:/# apt-get dist-upgrade
root:/# apt-get install --reinstall lightdm xserver-xorg-lts-xenial ubuntu-desktop apt apt-utils aptdaemon aptdaemon-data update-manager update-manager-core dbus
root:/# updatedb

17. Install Nvidia drivers, if applicable and reboot

root:/# apt-get install nvidia-384
root:/# reboot

That’s all folks!!!

Running Linux applications on Windows 10

Big thanks to Microsoft for introducing Linux sub-system on windows 10.  Given below are the steps to run ‘almost’ any Linux application on windows (should I say natively?). Details of individual steps are skipped on purpose, as the user is expected to understand what is going on. If it is not understandable, it is probably not safe to proceed.

  1. After enabling developer mode, install Linux sub-system. In PowerShell started with admin rights, enter the following command to install the sub-system:
    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
  2. Restart the computer.
  3. Run command prompt as administrator, type bash, and press enter. Follow the onscreen instructions to install ubuntu.
  4. Update the sources.list file on bash with the following repositories:
    #------------------------------------------------------------------------------#
    # OFFICIAL UBUNTU REPOS #
    #------------------------------------------------------------------------------#
    ###### Ubuntu Main Repos
    deb http://us.archive.ubuntu.com/ubuntu/ trusty main restricted universe
    deb-src http://us.archive.ubuntu.com/ubuntu/ trusty main
    ###### Ubuntu Update Repos
    deb http://us.archive.ubuntu.com/ubuntu/ trusty-security main restricted universe
    deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates main restricted universe
    deb http://us.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe
  5. Run apt-get update && apt-get install
  6. Run the following command in bash, which will allow seeing GUI on windows screen
    echo "export DISPLAY=:0" >> ~/.bashrc
    source ~/.bashrc
  7. Install Xming on windows. available here. Start the application. It starts minimized in the taskbar without any open window.
  8.  Run any application that needs display from bash, for example, “xeyes” to confirm that it is working fine.

Example: Running ARGoS simulator:
Once bash is working and you can see xeyes in a new window, follow instructions for installing ARGoS on ubuntu as given here. If you don’t see  simulator window, check if Xming server is up and running in windows.

argos

 

What doesn’t work:
Any application that needs access to hardware files on Linux. for example ifconfig command fails, so does lsmod, lspci, lsusb, etc. It also means that though ROS can be installed and configured, but roscore cannot start as the application cannot open up 11311 port 🙁  A loopback adapter “might” allow running ROS.

Ethercat for ROS indigo

Installation of ROS Indigo

Ros indigo is only supported on Ubuntu 14.04. Though Virtual machine can be used, it is recommended to have a complete installation on the machine.

Run the following commands to install ROS Indigo on ubuntu 14.04

sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'

sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net --recv-key 0xB01FA116

sudo apt-get update

sudo apt-get install ros-indigo-desktop-full

sudo rosdep init

rosdep update

echo "source /opt/ros/indigo/setup.bash" >> ~/.bashrc

source ~/.bashrc

Detailed information on these instructions is available at ROS website

Create a Workspace

Create a folder to store the codebase and initialize that folder as a catkin_workspace (catkin is a build system used by ROS)

mkdir -p ~/alligators_ws/src

cd ~/alligators_ws/src

catkin_init_workspace

catkin_create_pkg ethercat_example roscpp ethercat_hardware

cd ~/alligators_ws

catkin_make

echo "source ~/alligators_ws/devel/setup.bash" >> ~/.bashrc

source ~/.bashrc

Download sample code from here. Extract the downloaded file and replace ~/alligators_ws/src/example_ethercat with this package.

Ethercat Package

Install ethercat package

sudo apt-get update

sudo apt-get install ros-indigo-pr2-ethercat-drivers ros-indigo-pr2-ethercat

Find out the ethernet adapter name to which ethercat device is connected. This should be of format ‘ethX’ where X is an integer. ifconfig command can be used to query the available ethernet adapters in linux.

:~$ ifconfig -a
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fc:a7:19  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr a8:20:66:3e:2f:a4  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:16 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:5231 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:509642 (509.6 KB)  TX bytes:509642 (509.6 KB)

virbr0    Link encap:Ethernet  HWaddr 26:37:1a:bb:00:8d  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 5c:96:9a:7e:22:6d  
          inet addr:10.0.0.87  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::5e96:9dff:fe7e:b26d/64 Scope:Link
          inet6 addr: 2601:98a:4000:20f5:5e96:9dff:fe7e:b26d/64 Scope:Global
          inet6 addr: 2601:98a:4000:20f5:81e1:66ca:2709:69f0/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:108776 errors:0 dropped:0 overruns:0 frame:214316
          TX packets:69526 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:122922125 (122.9 MB)  TX bytes:8517033 (8.5 MB)
          Interrupt:17

Open a terminal window and run roscore command

~:$roscore
... logging to /home/ninja/.ros/log/d0267b0e-0a1a-11e6-8140-5c969d7eb26d/roslaunch-ninja-MacBookPro-12106.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is 

Open another terminal window and run the following

sudo su
<<Enter Password>>

source /opt/ros/indigo/setup.bash

source ~/alligators_ws/devel/setup.bash

rosrun pr2_ethercat pr2_ethercat -i eth1 -x ~/alligators_ws/src/ethercat_example/description/robot.xml

Compiling SCons project in Qt Creator

Quick note on how to configure SCons project in Qt Creator:

In Build settings under Projects, change the configuration to:

1. Build Directory : directory where SConstruct and SConscript are present
2. Build steps : Remove make and add a custom process as Command:= /usr/bin/scons, Arguments:= -j4, Working Directory:=%{buildDir}
3. Clean Steps : Remove existing step and add a custom process as Command:= /usr/bin/scons, Arguments:= -c, Working Directory:=%{buildDir}

Nvidia drivers on Ubuntu 14.04 kernel 3.19

Looks like Nvidia will keep bothering me as long as I use linux…
After installing nvidia drivers on 14.04 with kernel 3.19, the computer would not boot!

The problem is, xserver-xorg-lts-vivid and the solution is replace it with xserver-xorg-lts-trusty.
Boot in recovery, enable networking, drop onto shell, and enter the following lines to magically fix the issue

sudo apt-get update
sudo apt-get remove --purge nvidia-* xserver* libcheese7
sudo apt-get install xserver-xorg-lts-trusty
sudo apt-get install nvidia-346 nvidia-settings nvidia-prime ubuntu-desktop
sudo reboot 

This is tested on Nvidia GTX650

Nvidia drivers on Ubuntu 12.04

Nvidia drivers don’t readily work with ubuntu 12.04 on kernel version 13.03. If they do, multiple monitors stop working. This can be fixed by using xserver from saucy. Following steps would fix the problems in most cases.

sudo apt-get update
sudo apt-get remove --purge nvidia-* xserver*
sudo apt-get install xserver-xorg-lts-saucy
sudo apt-get install nvidia-331 nvidia-settings nvidia-prime ubuntu-desktop
sudo reboot 

This is tested on Thinkpad W540 (Nvidia Quadro K2100M)

Converting MBR partition table to GPT

Almost everyone who tried installing dual boot operating system on windows 8 machine have faced this issue. The issue is more severe when you upgrade your windows 7 machine to windows 8 and then try to install ubuntu 14.04. The installation aborts with an error message stating that the installation cannot continue without GPT (GUID Partition Table). This posts lists the possible reasons for the errors and a way to resolve this problem for good.

Prerequisites:
Windows 8 onwards it is recommended to have UEFI enabled instead of our old friend BIOS. More details about UEFI and BIOS can be found at this link . One thing to remember is, MBR (Master Boot Record) partition table works for BIOS and UEFI requires GPT, reasons are detailed in the link above. If you have a single partition of ~300MB at the start of the disk, it is mostly MBR and if there are 2 partitions of ~100/200MB then it is mostly GPT.

Identify the issue:
If you are trying to install Ubuntu 14.04 or any other OS and the error asks for GPT partition table, you need to convert your partition table (which in most cases would be MBR) to GPT. If you have installed Ubuntu(or any other linux) successfully but are unable to login into it, you just need a boot-recovery. In the latter case, just follow the instructions at this link and that should fix the boot related issues.

Convert MBR to GPT:
1. Enable UEFI in the BIOS settings (if your computer hardware does not support UEFI, you probably don’t need GPT)
2. Enable boot from USB device from the BIOS settings
3. Start Windows, and turn off BitLocker if it is enabled.
4. Ensure that a partition of ~300MB is present at the starting of the disk, if that is not available you might have to reformat or repartition your HDD (which is out of scope of this post)
5. If you dont have a system repair disk, create one.

***NO TURNING BACK AFTER THIS STEP***

6. Download gptgen from http://sourceforge.net/projects/gptgen/
7. Run cmd with administrative privileges. then run the following command to crash your windows, assuming that drive0 is your drive with MBR partition table that needs to be converted to gpt. If not sure, use “wmic diskdrive list” to identify the drive.

gptgen.exe -w \\.\physicaldrive0

8. Restart the computer using a system repair disk
9. after selecting Language and preferences, browse to Repair Your Computer -> Troubleshoot -> Advanced Options -> Command Prompt
10. Run the partitioning tool, list the disks and select the appropriate disk

diskpart
list disk

Results:
Disk ### Status Size Free Dyn Gpt

-------- ------------- ------- ------- --- ---

* Disk 0 Online 256 GB 0 B *

11. Select the required disk

select disk 0

12. Check the partitions

list partition


Results:

Partition ### Type Size Offset

------------- ---------------- ------- -------

Partition 1 Primary 350 MB 1024 KB

Partition 2 Primary 254 GB 350 MB


13. Delete the first partition that stores MBR table and create new partitions for GPT

select partition 1
delete partition

create partition EFI size=100 offset=1
format quick fs=fat32 label="System"
assign letter=S

create partition msr size=128 offset=103424

list partition

Results:

Partition ### Type Size Offset

------------- ---------------- ------- -------

Partition 1 System 100 MB 1024 KB

Partition 2 Reserved 128 MB 101 MB

Partition 3 Primary 254 GB 229 MB

14. List the volumes and assign C as the volume label for windows volume. Change the number 1 to whatever number you see for your volume

list volume
select volume 1
assign letter=C
exit

15. Generate the boot partition data

bcdboot c:\windows /s s: /f UEFI

16. Restart.

Congratulations, you just avoided troubleshooting that might have taken more than 2 days and several reinstallations of windows for no reasons 🙂