Using ClusterSSH to simultaneously perform tasks on several machines

Quite frequently I need to do the same configuration steps on several machines. This can get really cumbersome if you do one machine after the other. However there is a cool tool, which let's you execute the commands on several machines simulataneously - ClusterSSH.

On Debian/Ubuntu you can install it using

# apt-get install clusterssh

Now you can either create a cluster file, which helps to administrate the same machines on different occasions, or you just run

# cssh username@machine1 username@machine2

to work on two machines simultaneously.

Additional infos can be found here

Unattended Ubuntu install using preseed

In this post, I will describe the process of creating a desktop iso for unattended installation. I have tried this with 12.04 as well as with a daily build for 14.04. 32 or 64bit does not matter.

So first of all download a desktop image of Ubuntu.
Meanwhile install isomaster

# apt-get install isomaster

Start isomaster and open the downloaded desktop image. In the bottom window open the "isolinux" folder. Edit the "txt.cfg" file, that is in that folder. Insert or update the following lines:

default autoinstall
label autoinstall
  menu label ^Autoinstall1
  kernel /casper/vmlinuz.efi
  append  url=http://YOUR-URL-TO-PRESEEDFILE automatic-ubiquity \ locale=de_DE.UTF-8 boot=casper initrd=/casper/initrd.lz quiet \ splash noprompt auto=true --

The interesting part here is the url parameter. With a url you are more flexible, than adding the preseed file onto the iso image. In other words, you can change your pressed file, without the need of creating a new iso image.

Now save the txt.cfg file and then the iso file. By the way, if you do not want to create a new iso file, you could actually edit the menu entry at boot time by pressing tab.

The next step is to create the preseed file. Most of it is self-explanatory, like keymap, timezone,...

The password part is interesting, because you can also add an encrypted password using an md5 hash.

(echo -n "password" | md5sum)

The toughest part, was the partitioning section. My script creates a root partition (~30 GB), a swap partition (1 GB) and a data partition (~30 GB). However so far my experience is, that the last partition usually takes up the drive's remaining disk space. So my data partition could be much larger. Yes for my purposes, i do not need a home partition!

The three numbers on the expert_recipe string 28000 10 28000 mean:

28000: minimum size of the partition in megabytes

10: the priority that this partition gets it’s maximum size fulfilled (with lower numbers having a higher priority)

28000: maximum size of the partition in megabytes

Save the file and make it available on the intranet/web, so that your script can be found on boot time. It might help to use the IP-address instead of a domain name. 

Last interesting thing here is, the success_command string, which allows additional scripts to run after the main cd installation has finished.

By the way, to find out what can be preseeded you can use

# debconf-get-selections --installer > file
# debconf-get-selections >> file

Here is my file:

d-i debian-installer/locale string de_DE.UTF-8

d-i console-setup/ask_detect boolean false

d-i keyboard-configuration/modelcode string pc105

d-i keyboard-configuration/layoutcode string de

d-i keyboard-configuration/xkb-keymap select  de

d-i keyboard-configuration/layout string German

d-i keyboard-configuration/variant string German

d-i netcfg/choose_interface select auto

d-i clock-setup/utc boolean true

d-i time/zone string Europe/Berlin

d-i clock-setup/ntp boolean true

d-i clock-setup/ntp-server string 0.at.pool.ntp.org 

d-i partman-auto/method string regular

d-i partman-auto/expert_recipe string regularvnode :: 28000 10 28000 ext4 \ $primary{ } $bootable{ } method{ format } format{ } use_filesystem{ } \

filesystem{ ext4 } mountpoint{ / } .  1024 100 1024 linux-swap $primary{ } \ method{ swap } format{ } . 28000 50 28000 ext4 $primary{ } method{ format } \ format{ } use_filesystem{ } filesystem{ ext4 } mountpoint{ /data } .

d-i partman/confirm_write_new_label boolean trued-i partman/choose_partition select finish

d-i partman/confirm boolean true

d-i partman/confirm_nooverwrite boolean true

d-i passwd/root-login boolean true

d-i passwd/root-password-crypted password 5f4dcc3b5aa765d61d8327deb882cf99

d-i passwd/user-fullname string admin

d-i passwd/username string admin

d-i passwd/user-password-crypted password 5f4dcc3b5aa765d61d8327deb882cf99

d-i user-setup/encrypt-home boolean false

d-i apt-setup/restricted boolean true

d-i apt-setup/universe boolean true

tasksel tasksel/first multiselect ubuntu-desktop

d-i pkgsel/include string openssh-server

d-i pkgsel/language-packs multiselect de

d-i grub-installer/only_debian boolean true

d-i grub-installer/with_other_os boolean true

d-i finish-install/reboot_in_progress note

xserver-xorg xserver-xorg/autodetect_monitor boolean true

#d-i ubiquity/summary string empty

ubiquity    ubiquity/summary        note

ubiquity    ubiquity/reboot    boolean    true

# This command is run just before the install finishes, but when there is

# still a usable /target directory. You can chroot to /target and use it

# directly, or use the apt-install and in-target commands to easily install

# packages and run commands in the target system.

# ubiquity ubiquity/success_command string wget 

ubiquity ubiquity/success_command string wget http://URL/preseed/postinst.sh \

-O /root/postinst.sh; in-target /bin/bash /root/postinst.sh

Further reading:
http://cptyesterday.wordpress.com/2012/06/17/notes-on-using-expert_recipe-in-debianubuntu-preseed-files/
https://help.ubuntu.com/lts/installation-guide/i386/appendix-preseed.html
http://www.debian.org/releases/stable/i386/apb.html.en

Setting up an DHCP server - network boot part three

You can use any dhcp server to get pxe booting to run. We are using isc-dhcp-server, which is part of debian. Dnsmasq, which also is a dns server, would be an alternative.

So let's install the server

# apt-get install isc-dhcp-server

It's configuration file is  /etc/dhcp/dhcpd.conf.

default-lease-time 600;
max-lease-time 7200;
allow booting;

# in this example, we serve DHCP requests from 192.168.0.(5 to 253)
# and we have a router at 192.168.0.1
subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.5 192.168.0.253;
  option broadcast-address 192.168.0.255;
  option routers 192.168.0.1;             
  option domain-name-servers 192.168.0.1; 
  filename "pxelinux.0"; 
  next-server 192.168.0.3;  #make sure, that this is the ip address of 
                            # your tftp server
}

After you have configured your dhcp server, you need to restart it.

  # /etc/init.d/isc-dhcp-server restart

Now we are ready for booting any machine over the network. We set up a TFTP server, an NFS server and a DHCP server. That's all it takes.

By the way I usually use VirtualBox to try and test network booting.

Further reading:
http://www.debian-administration.org/articles/478
https://help.ubuntu.com/community/PXEInstallMultiDistro
https://help.ubuntu.com/community/DisklessUbuntuHowto

Setting up an NFS server - network boot part two

NFS is used to share files/directories among networked computers. First let's install NFS.

apt-get install nfs-kernel-server 

Next we will create a directory that holds our cd images. Of course NFS can be used for other things - e.g. holding user data centrally.
We will just provide some iso images and make them available for network boot.

mkdir ~/netbootimages

I put several different live cd images into that folder (e.g. ubuntu, clonezilla, and others)
In addition to where my net boot images are located, we need to create an nfs share.

mkdir -a /srv/boot/isoimages

Next we create a directory for each cd image on the tftp server share

mkdir -p "/var/lib/tftpboot/isoimage/MYISO"

Then we mount the iso into that share

mount -t iso9660 -o loop ~/netbootimages/MYISO.iso "/var/lib/tftpboot/isoimage/MYISO"

And we need to also make the image available on the nfs share. We use mount bind/rebind for that, so we have all the files there just once.

mount --rbind /var/lib/tftpboot/isoimages/MYISO /srv/boot/isoimages/MYISO


Next we need to export the files
exportfs -i -o async,no_root_squash,no_subtree_check,ro 0.0.0.0/0.0.0.0:/srv/boot/isoimages/MYISO

PXE boot also needs three boot files: the boot-strap (pxelinux.0), the menu (menu.c32), and the menu text configuration (pxelinux.cfg/default). So let's copy/create them:

cp /usr/lib/syslinux/pxelinux.0 "/var/lib/tftpboot/"
cp /usr/lib/syslinux/menu.c32 "/var/lib/tftpboot/"
mkdir -p "/var/lib/tftpboot/pxelinux.cfg"



In the folder pxelinux.cfg I created a default file, which looks like this


default menu.c32
prompt 0
timeout 300
ONTIMEOUT local

MENU TITLE Main Menu

LABEL local
        MENU LABEL Boot local hard drive
        LOCALBOOT 0

LABEL Ubuntu12.04.4  
    kernel isoimages/12.04.4-64/casper/vmlinuz.efi
    append boot=casper netboot=nfs nfsroot=NFS_SERVER_IP:/srv/boot/isoimages/12.04.4-64 initrd=isoimages/12.04.4-64/casper/initrd.lz 

Further reading:
https://wiki.ubuntu.com/LiveCDNetboot
http://tjworld.net/wiki/Linux/Ubuntu/NetbootPxeLiveCDMultipleReleases
http://www.howtogeek.com/61263/how-to-network-boot-pxe-the-ubuntu-livecd/


We are almost done. Last thing is, we need to configure a dhcp server, to serve the pxelinux.0 file. This will be part of one of the next blog entries.

Setting up a TFTP server - network boot part one

A TFTP server is usefull for PXE booting (network boot). It can be installed using

apt-get install tftpd-hpa 


After installation we need to configure it. Modify the file /etc/default/tftpd-hpa accordingly. Mine looks like this:

# /etc/default/tftpd-hpa
RUN_DAEMON="yes"
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/var/lib/tftpboot"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="--secure"


Last step is to restart the tftp service

sudo service tftpd-hpa restart


In a later blog entry we will put files to /var/lib/tftpboot so that we can boot over network. For that to work, additional software and configurations are necessary. To be conintued...

Creating a self healing /home/student envirenoment using overlayfs

In a school environment various students have to accomplish different tasks on the same computer every day. So one of the challanges is, to give each student using a computer the same tools, look and feel, and setup. There are several different approaches to that. Using overlayfs is the one we are using to achieve that.

First of all what is overlayfs?

Quoting from the documentation "An overlay-filesystem tries to present a filesystem which is the result over overlaying one filesystem on top of the other."

An overlayfs consists of a lowerdir and an upperdir.
The lowerdir is reflected in the upperdir, so when accessing the upperdir you see everything that is in the lowerdir as well. When you change a file in the upperdir it is not propagated down to the lowerdir, so the lowerdir always stays clean.

This picture illustrates our use case.

So we have a regular /home/student directory which is created and is set up accordingly. The /home/.student-rw folder is actually empty and just protects the default /home/student installation.

Whenever we boot our system, we need to make sure that the /home/.student-rw directory sits above the /home/student directory. It only takes an entry in /etc/fstab to do so

none /home/student overlayfs lowerdir=/home/student,upperdir=/home/.student-rw 0 0

The cool part about the line above is, that my mount point actually is /home/student.
So i take /home/student put my overlay above that and mount that into /home/student, which as of now is protected by stacking another filesystem above it.


Ok, this is part one. I protect /home/student. Part two makes sure that all the changes that a student made are wiped out on certain occasions. These occasions are
- reboot and logout

i created a script (in /usr/local/bin/cleanstudentdir.sh) that does that:
#!/bin/bash
cd /home/.student-rw && find . -maxdepth 1 -mindepth 1 -print0 |xargs -0 rm -rf
mount -o remount /home/student
#remount is necessary to re-read the filesystem


i call that script in the lightdm.conf on
session-setup-script and session-cleanup-script
so add these lines to /etc/lightdm/lightdm.conf
session-setup-scrupt=/usr/local/bin/cleanstudentdir.sh
session-cleanup-scrupt=/usr/local/bin/cleanstudentdir.sh


With the above setup all students get the exact same environment. And I do not care if they change any of their settings, because they will be gone after a reboot anyways.

Last thing that we need to explain. How can we make changes to /home/student that survive a reboot. Well, that's fairly easy.
Before logging in as a student I execute "sudo unmount /home/student" from the console, which just removes the overlay. Now I log in as a student and can do any changes that I want to persist. Logging out and adding the overlay using "sudo mount -t overlayfs overlayfs -olowerdir=/home/student,upperdir=/home/.student-rw /home/student" or a reboot is enough to protect /home/student again.


Further reading
https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/tree/Documentation/filesystems/overlayfs.txt?h=overlayfs.current

Package caching for debian based distros

Most of our computers run on linux, actually on debian based distros like ubuntu. To make installing and updating faster we have an apt proxy installed.

An apt proxy fetches files from remote repositories when needed, and caches them for local use. So it also saves bandwidth, which was very important before we had a fiber channel connection.
There are different apt proxies available, like apt-cacher, apt-cacher-ng, squid-deb-proxy or approx, which is the one we are using. It's easy to set up and works very well.

$ apt-get install approx

and the config file is

/etc/approx/approx.conf:

ubuntu    http://archive.ubuntu.com/ubuntu
ubuntu-extras    http://extras.ubuntu.com/ubuntu
ubuntu-partner    http://archive.canonical.com/ubuntu
ubuntu-security    http://security.ubuntu.com/ubuntu
debian    http://ftp.debian.org/debian
debian-security    http://security.debian.org/debian-security

Usually you only need to add your repositories. Of course there are other things that can be changed, like the port, or debugging. Just see $ man approx
Once it's running you can set up the client side.

So edit /etc/apt/sources appropriately. It could look like this:

deb http://proxy:9999/ubuntu precise main restricted multiverse universe
deb http://proxy:9999/ubuntu precise-security main restricted multiverse universe
deb http://proxy:9999/secure precise-security main restricted multiverse universe
deb http://proxy:9999/partner precise partner
Then run
$ apt-get update

That's mainly it. By the way, there are two apt related packages, that are very helpful.

apticron - it keeps you informed by email, if updates are available.
cron-apt - it downloads packages to your machine, and even installs them if you configure it that way.

Backing up and restoring a cubieboard nand installation

Last time, I showed you how to run chrome browser on cubieboard in kiosk mode.

As we have several TVs with a cubieboard, I did not want to go through all these steps again and again. There is a rather simple way of cloning a nand image.

You will need a micro-SD card and install any linux on that. I used http://cubian.org/downloads/ for that.

So download the appropriate image, unzip it and install it on your sd card using:

dd if=cubian.img of=/dev/YOUR_DEVICE bs=4096; sync

Once done, put the SD card into your cubieboard and boot. The default credentials are cubie/cubie.

To create an image run:
dd if=/dev/nand conv=sync,noerror bs=64K | gzip -c -9 > /nandimg.gz



Later you can restore that image using on a new board
gunzip nandimg.gz; dd if=/nand.img conv=sync,noerror bs=64K of=/dev/nand

Be aware that you need to change a few settings if you are reusing this image on several boards:
hostname - /etc/hostname and or /etc/hosts
hwaddress - /etc/network/interfaces

Using the cubieboard (A10) to display class schedules and updates using chromium-browser in kiosk mode

First of all download an A10 image

e.g. one of these: http://linux-sunxi.org/Install_Ubuntu_Linux_for_Cubieboard

or here: http://linux-sunxi.org/Bootable_OS_images

or here: http://cubieboard.org/download/

The next step is to download and install LiveSuit (http://linux-sunxi.org/LiveSuit), which is an application to flash the NAND.

Start LiveSuite, select the downloaded image and press the FEL key when powering up (the FEL key is the one under the USB OTG port), The CPU automatically enters FEL mode. Livesuit will start to write the image to Nand on board.

After that, you will be able to log into your cubieboard, with the appropriate credentials. In my case linaro/linaro.

Depending on the image you used, you need to resize some partitions

sudo resize2fs /dev/nandc

sudo resize2fs /dev/nandd

sudo resize2fs /dev/nande

sudo resize2fs /dev/nandf

Now we are ready make changes and install software. These are the changes I made

1.) Change default password and root password using passwd

2.) Edit /etc/network/interfaces and set a fixed hardware address

           auto eth0

iface eth0 inet dhcp
   hwaddress ether xx:xx:xx:xx:xx:xx

You could use a generated mac address, which ifconfig will display.

3.) To enable autologin edit /etc/lightdm/lightdm.conf and add

autologin-user=<YOUR USER>
autologin-user-timeout=0

In a terminal run

sudo groupadd autologin

sudo gpasswd -a linaro autologin

4.) Install/uninstall software

sudo apt-get update

sudo apt-get install unclutter xvkbd -y icedtea-6-plugin vino x11-xserver-utils vino openssh-server

sudo apt-get remove xscreensaver

5.) To change the locale settings, run the following commands in a terminal

sudo locale-gen de_DE.UTF-8

export LANG=de_DE.UTF-8

export LC_ALL=de_DE.UTF-8

sudo update-locale

sudo dpkg-reconfigure locales

6.) reconfigure the time zone

sudo dpkg-reconfigure tzdata

7.) Modify your autostart. vi /etc/xdg/lxsession/Lubuntu/autostart

@lxpanel --profile Lubuntu

@pcmanfm --desktop --profile lubuntu

@/usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1

@xset s off

@xset s 0 0

@xset -dpms

@xset s noblank

@ntpdate 0.de.pool.ntp.org

@/usr/lib/vino/vino-server &

@chromium-browser --kiosk --incognito --disable-translate --allow-outadated-plugins --disk-cache-dir=/tmpfs --always-authorize-plugins http://yourwebsite

The xset stuff is there to disable any screen saving mode.

ntpdate updates your time. As an alternative you coud install ntp daemon (apt-get install ntp)

vino-server is a vnc-server, so i can have a remote look at the screen.

An alternative to vino is x11vnc (apt-get install x11vnc), which did not work due to a bug. (It crashed on connect).

The last line starts the browser in kiosk mode and opens a certain website.

8.) If you are using vino, make sure you set appropriate preferences using

vino-preferences

9.) I also had some issues with screen resolution, so i created a xorg.conf in /etc/X11

Section "Monitor"

       Identifier      "Monitor0"

       ModelName    "Monitor Model"

       DisplaySize     240 320

       Option          "DPMS" "false"

       Option       "UseEDID" "false"

EndSection

Section "Device"

       #Option "ShadowFB"      "true"

       Identifier  "Card0"

       Driver      "fbdev"

       Option  "fbdev" "/dev/fb0"

EndSection

Section "Screen"

       Identifier      "Screen0"

       Device     "Card0"

       Monitor     "Monitor0"

    DefaultDepth    24

    SubSection "Display"

           Depth           24       

       Modes   "1280x1024"

    EndSubSection

EndSection

10.) Last thing I did, was create a crontab, which shuts down the cubieboard, a few minutes, before the TV-screen turns off. As the board is hooked up to the TV usb, the TV will turn on the cubieboard again in the morning. Yes, the TV has a timer function, so it turns itself on automatically.