TestingScript

Testing Script

In order to run the OE testing builds a small script has been created. It is felt that this might be useful for others so I am putting and documenting it here. The latest version of the script can also be found in the openembedded tree in contrib/testing/testscript.sh

The conf file

The local.conf used:

# fix next line if you want to use your own mirror, then remove the # for the next two lines
# You can start a trivial server with 'python -m SimpleHTTPServer'
#SOURCE_MIRROR_URL = "http://localhost:8000/directory"
#INHERIT += "own-mirrors"

DL_DIR = "${TOPDIR}/downloads"

# if you want to keep tmp dirs for different builds you might want to set TMPDIR to e.g. ${TOPDIR}/tmp_${MACHINE}_${DISTRO}
TMPDIR = "${TOPDIR}/tmp"
BBFILES = "${TOPDIR}/openembedded/recipes/*/*.bb"
ENABLE_BINARY_LOCALE_GENERATION = "0"

# Which files do we want to parse:
BBMASK = ""

# ccache always overfill $HOME....
CCACHE=""

# What kind of images do we want?
IMAGE_FSTYPES = "tar.gz "

# Make use of my SMP box
PARALLEL_MAKE="-j4"
BB_NUMBER_THREADS = "2"

OE_STRICT_CHECKSUMS = "1"

# if you are low on disk space you can remove the next #, disadvantage, nastier debugging in case of failures
#INHERIT += "rm_work"

# if you want to report build results (recommended) you need to edit OESTATS_BUILDER and add your name or nick in it, then uncomment the next 3 lines.
#INHERIT += "oestats-client"
#OESTATS_SERVER = "tinderbox.openembedded.net"
#OESTATS_BUILDER = "your nick goes here"

Note that in order to use the above conf you need to have:

BB_ENV_EXTRAWHITE="MACHINE DISTRO TOPDIR"
export BB_ENV_EXTRAWHITE MACHINE DISTRO TOPDIR

A full script

I have wrapped things together with a small shell script to checkout oe if needed, and to checkout the testing branch:

# test if we have an openembedded dir, clone it if it does not exist
if [ ! -d openembedded ]
then
    (git clone git://git.openembedded.org/openembedded)
    (cd openembedded; git checkout -b testing-next origin/testing-next)
fi

# switch to the testing branch
(cd openembedded; git checkout testing-next)

# test if bitbake exist; if not; fetch it and untar it
if [ ! -d bitbake-1.10.1 ]
then
    (wget http://download.berlios.de/bitbake/bitbake-1.10.1.tar.gz; tar xf bitbake-1.10.1.tar.gz; rm bitbake-1.10.0.tar.gz) 
fi

# TOPDIR is where we are now
TOPDIR=`pwd`

# add bitbake to the path
export PATH=${TOPDIR}/bitbake-1.10.1/bin:$PATH

# create a local.conf by using a here document
cat > ${TOPDIR}/openembedded/conf/local.conf << EOF
# fix next line if you want to use your own mirror, then remove the # for the next two lines
# You can start a trivial server with 'python -m SimpleHTTPServer'
#SOURCE_MIRROR_URL = "http://localhost:8000/directory"
#INHERIT += "own-mirrors"

DL_DIR = "${TOPDIR}/downloads"

# if you want to keep tmp dirs for different builds you might want to set TMPDIR to e.g. ${TOPDIR}/tmp_${MACHINE}_${DISTRO}
TMPDIR = "${TOPDIR}/tmp"
BBFILES = "${TOPDIR}/openembedded/recipes/*/*.bb"
ENABLE_BINARY_LOCALE_GENERATION = "0"

# Which files do we want to parse:
BBMASK = ""

# ccache always overfill $HOME....
CCACHE=""

# What kind of images do we want?
IMAGE_FSTYPES = "tar.gz "

# Make use of my SMP box
PARALLEL_MAKE="-j4"
BB_NUMBER_THREADS = "2"

OE_STRICT_CHECKSUMS = "1"

# if you are low on disk space you can remove the next #, disadvantage, nastier debugging in case of failures
#INHERIT += "rm_work"

# if you want to report build results (recommended) you need to edit OESTATS_BUILDER and add your name or nick in it, then uncomment the next 3 lines.
#INHERIT += "oestats-client"
#OESTATS_SERVER = "tinderbox.openembedded.net"
#OESTATS_BUILDER = "your nick goes here"

EOF

# smake sure BB_ENV_EXTRAWHIT is correct, and export the needed vars
BB_ENV_EXTRAWHITE="MACHINE DISTRO TOPDIR"
export BB_ENV_EXTRAWHITE MACHINE DISTRO TOPDIR
export BBPATH=${TOPDIR}/openembedded

# pull the current branch; in case a stale lock exists remove it
(cd openembedded; rm -f .git/index.lock;git pull)

# clean tmp; I want to start with a clean build; if you changed TMPDIR in the conf file better change it here too.
rm -rf ${TOPDIR}/tmp

# add an echo about the vars so we can see what has been done in a log file
echo $MACHINE $DISTRO $TARGET_RECIPE

# and do the actual work.
bitbake ${TARGET_RECIPE}

This script uses the external env variables MACHINE, DISTRO and TARGET_RECIPE.
You might of course add these to the beginning of the script. E.g.:

MACHINE="beagleboard"
DISTRO="minimal"
TARGET_RECIPE="console-image"

Getting things on Hudson autobuilder (1)

I use hudson (www.hudson.org) as autobuilder. If you also want to do that install hudson and create a small project.
The first option is to create a job with one build step and use the script above as build step. In order to get the env vars working I checked "This build is parametrized" and defined 3 enum parameters named MACHINE, DISTRO and TARGET_RECIPE. Make sure the parameters are of the enum type. The value you choose when running the build will be passed as environment variable.

Getting things on Hudson autobuilder (2)

A more advanced mechanism is to build a multiconfiguration with hudson.

This is done by selecting "Build multi-configuration project" when creating the job. Note that you cannot change a regular project in a multi-configuration project. After having done that you can select your configuration matrix. This can have multiple axes. When building hudson, by default it will compile all permutations. However mechanisms exist to limit the # of permutations.

I have been a little bit lazy here. I only created one axe (MACHINE) and as I only wanted to build one distro and image for each machine I prepended the script above with a little bit of selection code:

if [ $MACHINE = "neek" ]
then
    DISTRO="minimal"
    TARGET_RECIPE="console-image"
fi
if [ $MACHINE = "nslu2le" ]
then
    DISTRO="slugos"
    TARGET_RECIPE="slugos-image"
fi
if [ $MACHINE = "nslu2be" ]
then
    DISTRO="slugos"
    TARGET_RECIPE="slugos-image"
fi
if [ $MACHINE = "calamari" ]
then
    DISTRO="minimal"
    TARGET_RECIPE="console-image"
fi
if [ $MACHINE = "mpc8313e-rdb" ]
then
    DISTRO="minimal"
    TARGET_RECIPE="console-image"
fi
if [ $MACHINE = "sheevaplug" ]
then
    DISTRO="minimal"
    TARGET_RECIPE="console-image"
fi

You get the idea.

One more small remark: multi-configuration will build each config in its own dir. The disadvantage is that each dir has its own copy of oe, bitbake, downloads etc. I have avoided this by simply doing a

cd ../..

as the start of the script. This causes hudson to build everything two levels up.

And if you want to run things periodically create a build trigger for it. I use:

0 18 * * 5

Which starts the script every friday (5) at 18.00 local time, so hopefully there are results after the weekend.

Conclusion

This terminates my wrap up of my build script. Feel free to update it with improvements, better solutions etc.