During Quantal I started working on a Continuous Integration (CI) environment to run commit level testing of Gnome core components on the latest development release of Ubuntu.
This resulted in a Juju charm to make deployment easier which I used to setup a JHBuild enviromnent on Raring. The results of the builds are published on a jenkins instance .
Today was a major milestone, for the first time with the help of Martin Pitt, all the red dots are gone, leaving only blue/green (build successfully and make check pass) or yellow dots (module builds but make check failed) This is a great step forward \o/
The current system is fetching, building and checking of 160 packages on the latest development release of Ubuntu, here is how it works.
The testing environment is an LXC container with a minimal system to run Ubuntu Raring/AMD64. The host system is running on Ubuntu Server 12.10.
In the container, I installed JHBuild from git, system dependencies, a basic webserver to serve the results outside of the container (an alternative would have been to install a Jenkins slave inside the container, but I didn’t choose this option due to communication issues between the slave and the master) and a script run from cron to drive the builds and report the results.
Jenkins has the passive role of reporting the results.
The build/test workflow is the following:
- Every 15min, the list of modules to build meta-gnome-core is fetched with jhbuild
- The revision of the modules in the list is compared to the previous run. New packages are added automatically to Jenkins.
- If a package or one of it’s dependency has changed, it is addded to the rebuild list
- The rebuild list is then processed to first build the package with jhbuild, then re-run a jhbuild with option check to run a make check if the build succeeded. JHBuild is setup to run non-interactively of course, and to force a checkout and autogen on failure.
- Logs and results are captured and published via the webserver.
- Jenkins monitors the state files published by this webserver, updates the results and collects the log files.
The whole system could have been summarized by
while :; do jhbuild build; sleep 60; done
but I wanted a bit more granularity and control over what is being build and tested. Furthermore building the whole stack every single change in glib for example was not really efficient (WebKit tends to use a couple of CPU cycles to build 
)
So, overall it’s pretty simple, there are still reliability issues and some failures need manual interventions, but having kicked all the red dots is encouraging.
The next steps will be to add a notification system (and send messages the right people), detailed test reports (convert gtester results to jUnit XML), rely on jhbuild’s sysdeps instead of a yet another list of static dependencies, improve the performance of reverse dependencies checking, and …
… fix the yellow dots!
If you use a Yubikey with a French keyboard layout and set the Yubikey to OATH-HOTP mode (i.e the key generates 6–8 digits number) then you’ll probably have experienced this disappointing behavior: the key sends non-numerical values that correspond to their numerical values on a french keyboard layout.
This is because the key sends keycodes corresponding to the digits row on the keyboard but digits, on a French layout, are shifted.
The workaround is trivial, hold the <SHIFT> key before touching the YubiKey to get the digits.
If you own a key with firmware 2.3 , you can use the option ‘-o use-numeric-keypad’ and the key will generate a numerical sequence, even with a French keyboard.
But there is no such option with firmware 2.2 and earlier. If you don’t want to burn passwords because you forgot to press the <SHIFT> key then you can configure your system to recognize the key and set it up to use a US keyboard.
The old school way is to configure the device manually in /etc/X11/xorg.conf and add an entry like:
Section "InputClass"
Identifier "yubikey"
MatchVendor "Yubico"
MatchProduct "Yubico Yubikey II"
Option "XkbModel" "pc101"
Option "XkbLayout" "us"
Option "XkbVariant" "intl"
EndSection
But desktop environments like Unity/Gnome or KDE override these settings and there is no UI way (at least for gnome) to configure several keyboards with different layouts. A solution is to use setxkbmap and a udev rule to reconfigure the keymap of the yubikey.
The script below set a US keymap, you can paste it to /usr/local/bin/setyubikeymap
#!/bin/sh
. /lib/udev/hotplug.functions
logger "YubiKey inserted - Setting keymap to US"
eval $(xargs -n 1 -0 < /proc/$(pgrep -f ^gnome-session)/environ| awk '/^DISPLAY|^XAUTHORITY/ {print "export", $0}')
exec 1<&- 2<&- 5<&- 7<&-
(
wait_for_file /dev/input/by-id/usb-Yubico_Yubico_Yubikey_II-event-kbd
exec /usr/bin/setxkbmap -device $(/usr/bin/xinput list --id-only 'Yubico Yubico Yubikey II') us
) &
Then add the following rule to /etc/udev/rules.d/99-yubikey to call the script when the key is inserted
# Udev rules for setting keymap of the Yubikey USB to US
ACTION=="add|change", SUBSYSTEM=="usb", \
ATTRS{idVendor}=="1050", ATTRS{idProduct}=="0010", \
RUN+="/usr/local/bin/setyubikeymap"
The script can surely be improved and be more robust but you get the idea. Also the key sometimes loses its configuration on resume from suspend, which is probably fixable with a pm hook.
Hope this helps French keyboard users!
Today I was fixing the build-publisher plugin that is broken when a matrix configuration changes. So, I git cloned the plugin, installed maven (not maven2), and ran mvn package.
Compilation went fine then come the tests … FAIL!
Hm, it failed right at the start during test injection with the following trace:
------------------------------------------------------------------------------- Test set: InjectedTest ------------------------------------------------------------------------------- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218 sec <<< FAILURE! initializationError(InjectedTest) Time elapsed: 0.005 sec <<< ERROR! java.lang.UnsatisfiedLinkError: com.sun.jna.Native.open(Ljava/lang/String;)J at com.sun.jna.Native.open(Native Method) at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:236) at com.sun.jna.Library$Handler.(Library.java:140) at com.sun.jna.Native.loadLibrary(Native.java:366) at com.sun.jna.Native.loadLibrary(Native.java:351) [...]
Not good.
What is that, missing environment variable, broken path, jdk7 on quantal, wrong library ???
After digging a bit around which jna lib was installed on my system, I decided to uninstall libjna-java and all the jenkins stack since it is a dependency of jenkins-common.
Bingo! mvn test passed.
Now, that’s a problem because it’d mean I cannot run jenkins and build jenkins plugin on the same system :/ Hopefully jna is in the maven repository, and we can tell him to use it instead of the version installed on the system. Just add the following to your pom.xml file:
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>3.2.2</version>
</dependency>
If you want to use the latest version 3.4.0, run maven with the following option:
$ mvn -DargLine="-Djna.nosys=true" test
And that’s it, jenkins and the pluging development environment running hand in hand on Ubuntu Quantal! 
I filed launchpad bug 1065253 to track this issue.
I was working on automating commit level building and testing of the gnome stack, and one of the painful and boring task is to setup the development and testing environment. Especially when you know that developers will have to use the same environment to reproduce failures found with testing and the QA Lab has restricted access.
So I went the Juju way and the result is a JHBuild charm .
This charm can be set up for manual use of JHBuild or automated building and testing. It is currently deployed in the Ubuntu QA Lab to build and test the gnome-core moduleset (197 modules) on Quantal.
Deploying JHBuild is as easy as configuring the service in config.yaml to your needs, bootstrapping juju and deploying jhbuild. Depending on the number of modules you want to build, you’ll need a medium or bigger instance type. Which consists in the following instructions:
1. Bootstrap the environment and deploy the service
$ juju bootstrap --constraints "instance-type=m1.medium" $ juju deploy --config gnome.yaml \ --repository $HOME/ubuntu/qa/juju/charms \ local:jhbuild jhbuild
2. after installation, connect to the unit with:
$ juju ssh <unit_or_machine>
3. And run your first build
$ jhbuild build
4. You want to organize a hacking party
$ juju add-unit jhbuild -n 10
6. To setup the service for automated build and publish the results:
$ juju set jhbuild autobuild=True publish=True
Results will be accessible on `http://<unit_public_addr>/`
Please see the README for more details and deployment instructions.
I hope it’ll be useful for you as well.
I’m writting a Juju charm to automate the deployment of jhbuild. This system can use different configuration files with sometimes very little difference between them. So, I wanted a basic templating system to manage these files that could run on a minimal system.
The very limited features I needed were file inclusion and variable substitution.
Basically what I wanted is, if I have the following 3 files:
file1.txt
A
${VAR1}
#include file2.txt
B
file2.txt
AA
#include file3.txt
${VAR2}
BB
file3.txt
AAA BBB
The expected result with VAR1=’this is var1′ and VAR2=’this is var2′ is:
A this is var1 AA AAA BBB this is var2 BB B
I went for bash and here is the result:
#!/bin/bash
merge_template () {
# Merge template files and do variable substitution
#
# $1: Template file
#
# Supported directives:
# #include filename : Include filename and process it.
# ${variable} : substituted by the value of the variable.
#
[ -z "$1" ] && return
set -f
while IFS='' read -r line; do
if [[ "$line" =~ \#include\ (.*) ]]; then
$FUNCNAME ${BASH_REMATCH[1]}
else
while [[ "$line" =~ (\$\{[a-zA-Z_][a-zA-Z_0-9]*\}) ]] ; do
LHS="${BASH_REMATCH[1]}"
RHS="$(eval echo "\"$LHS\"")"
line="${line//$LHS/$RHS}"
done
printf "%s\n" "$line"
fi
done<$1
set +f
}
merge_template $1
Save it to merge_templates and run it with:
VAR1="This is var1" VAR2="this is var2" ./merge_templates file2.txt
Et voilà
VMs are widely used in QA to provision clean and controlled test environment. A traditional way to create new VMs automatically in Ubuntu is to download an ISO, prepare a preseed and install a system from this image. This method is nice but involves lot of technical parts and is more or less reliable when it comes to mass-provisioning VMs depending on the technologies you choose.
Cloud technologies bring us new ways to provision local images for testing. cloud-init is one of those. It helps you get a pristine environment in a couple of minutes without the trouble of preseeding, pxe booting, dnsmasq setup or libvirt machine definition.
The first thing you need is a base image. We are testing the development release so lets get a base server image of Ubuntu Quantal. Cloud images for the development release are built daily and available from
http://cloud-images.ubuntu.com/quantal/current/
mkdir /tmp/cloudinit && cd /tmp/cloudinit wget http://cloud-images.ubuntu.com/quantal/current/quantal-server-cloudimg-amd64-disk1.img
You can start this image directly with kvm but as it is, it is not really well suited for QA and we need to tune it a bit.
First, the size. By default the size of the image is 2GB
$ qemu-img info quantal-server-cloudimg-amd64-disk1.img image: quantal-server-cloudimg-amd64-disk1.img file format: qcow2 virtual size: 2.0G (2147483648 bytes) disk size: 221M cluster_size: 65536
This is not enough to build some packages, or install dependencies for desktop packages. Let resize it to 4GB. cloud-init will take care of expanding the existing filesystem to the maximum space available.
$ qemu-img resize quantal-server-cloudimg-amd64-disk1.img +2G Image resized.
Now we must create 2 files meta-data and user-data, the latter will contain the cloud-config settings.
{ echo instance-id: iid-local01; echo local-hostname: autopkgtest; } > meta-data
PUBKEY=$(cat ~/.ssh/id_rsa.pub )
cat > user-data << EOF
#cloud-config
locale: en_US.UTF-8
password: ubuntu
chpasswd: { expire: False }
ssh_pwauth: True
ssh_authorized_keys:
- $PUBKEY
apt_update: true
apt_upgrade: true
byobu_by_default: system
packages:
- eatmydata
- autopkgtest
- dpkg-dev
- pbuilder
EOF
meta-data and user-data will be loaded from a VFAT or an ISO9660 filesystem. We will create an ISO:
$ genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data I: -input-charset not specified, using utf-8 (detected in locale settings) Total translation table size: 0 Total rockridge attributes bytes: 331 Total directory bytes: 0 Path table size(bytes): 10 Max brk space used 0 183 extents written (0 MB)
All the pieces are here to provision our pristine test environment. It can be started it with kvm:
$ kvm -m 1024 -smp 2 -net nic -net user -redir tcp:54322::22 -drive file=quantal-server-cloudimg-amd64-disk1.img,if=virtio -drive file=seed.iso,if=virtio
This will start a Qemu window. After the boot sequence you should see and prompt with the hostname ‘autopkgtest’. You can login with the account with setup in user-data file: ubuntu/ubuntu
Alternatively you can ssh to the machine with:
$ ssh -i ~/.ssh/id_rsa -p 54322 -l ubuntu localhost
Once logged in run the following command to verify that the filesystem has been resized to 4G
$ df -h /dev/vda1
You can follow the progress of cloud-init in the log file /var/log/cloud-init.log (the example here takes some time to install due to the number of packages) You’ll know when the installation is finished when the file /var/lib/cloud/instance/boot-finished is created. It contains the time it took to setup the machine.
$ cat /var/lib/cloud/instance/boot-finished 283.51
Once the file is there you can verify that dpkg-dev has been installed for example:
$ apt-cache policy dpkg-dev dpkg-dev: Installed: 1.16.1.2ubuntu8 Candidate: 1.16.1.2ubuntu8
Finally, poweroff the VM
$ sudo poweroff
We have setup a pristine VM in less than 5 minutes. Now we can use it as our base VM to run autopkgtest for example.
Create a new qcow image to boot, backed by your original image
$ qemu-img create -f qcow2 -b quantal-server-cloudimg-amd64-disk1.img quantal-amd64-apport.imgFormatting 'quantal-amd64-apport.img', fmt=qcow2 size=4294967296 backing_file='quantal-server-cloudimg-amd64-disk1.img' encryption=off cluster_size=65536
Start it:
$ kvm -m 1024 -smp 2 -net nic -net user -redir tcp:54322::22 -drive file=quantal-amd64-apport.img,if=virtio
Wait until it boots (DataSourceEc2.py tries to reach http://169.254.169.254/2009-04-04/meta-data/instance-id for 2 minutes then gives up) then login. You are all set and ready to run autopkgtest.
Inside the VM run:
$ sudo -i # apt-get source -d apport [...] # adt-run apport_2.2.3*dsc --- adt-virt-null
For details about cloud-init and cloud-config you can read the documentation shipped with the source of cloud-init:
http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/
http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt
Precise Beta 2 is due next week (Mar. 29th) The first set of Beta 2 candidates is ready for smoke testing on the ISO tracker.
There are going to be more respins before we’ve got the final set but we’d appreciate any feedback we can get on these.
You can test on virtual machines, but at this stage, we’d need more tests on real hardware. If you can test on spare hardware or create a partition to install Precise Beta 2, it would be very appreciated.
As usual we’ll be asking everyone on the QA team to participate in the image testing to ensure we have good test coverage.
The procedures for testing ISO images and reporting results are explained on
https://wiki.ubuntu.com/Testing/ISO/Procedures
You can download Precise images directly from cdimages.ubuntu.com with zsync or use dl-ubuntu-test-iso from ubuntu-qa-tools available from launchpad with bazaar:
$ bzr branch lp:ubuntu-qa-tools
Test results are tracked on
http://iso.qa.ubuntu.com/
You’ll need a Launchpad account to login on the tracker. Please register if you are new to this.
Let us know if you have any questions.
We will coordinate testing in #ubuntu-testing on freenode. Please, go there often to see what others are testing or what need to be tested.
Thank you very much for your help and happy testing!
