Difference between revisions of "Adding a secondary toolchain"
(→classes/tc-secondary.bbclass) |
(→Implementation Example) |
||
| Line 119: | Line 119: | ||
meta-tc-example/classes/tc-secondary.bbclass | meta-tc-example/classes/tc-secondary.bbclass | ||
meta-tc-example/classes/tc-blacklist.bbclass | meta-tc-example/classes/tc-blacklist.bbclass | ||
| + | |||
| + | The following example is available at: [http://gate.crashing.org/~fray/yocto/meta-tc-example.tar.bz2 meta-tc-example.tar.bz2] | ||
=== conf/layer.conf === | === conf/layer.conf === | ||
Revision as of 22:32, 14 November 2012
Contents
Background
In OpenEmbedded-Core there in an included toolchain that is built up of GNU binutils, gcc, and eglibc. In addition related components such as gdb, cross-prelink, mklibs and others provider additional toolchain functionality. In this document we focus on explaining best practices for including an additional toolchain that is capable of assembling, compiling and linking code. This document does not cover replacing the existing toolchain or toolchain components with external elements.
Many companies have commercial / highly optimized toolchains for specific purposes, such as ICC from Intel, LLVM, ARM Ltd toolchains, etc.. In some cases, these highly optimized toolchains may result in better performance for some applications, however they are often not generally applicable as they can not compile all of the system software. So instead of replacing the included GNU toolchain, a mechanism for providing an alternative/secondary toolchain is desired.
Requirements
Secondary Toolchains should be implemented in a layer
In order to facilitate re-use and make it easier to completely disable the secondary toolchain, a layer must be used to implement the secondary toolchain.
Secondary Toolchain must be generally compatible with GNU toolchain
The secondary toolchain must be generally compatible with the idea of sysroots, linking to the defined libc, and various GNU conventions. This compatibility may be implemented directly by the secondary toolchain or via a wrapper of some type.
Per recipe selection of toolchain usage
A mechanism is needed to activate this alternative toolchain on a per-recipe basis. Using variables it should be possible to specify on a per-recipe basis which toolchain should be used, leaving the default up to the user. (It's assumed if the user does not select a default, the system GNU toolchain will be defaulted.)
Blacklist/Whitelist
Even with the requirement to be generally compatible, we can not expect all of the possible recipes to build properly with the secondary toolchain. As a consequence of this, it will be necessary to implement a blacklist (or whitelist) of specific recipes that are known to work. This way only supported components can be build by the user, avoid numerous complaints and unresolvable defects.
SDK
The SDK generated by OpenEmbedded-Core also includes the included toolchain components. It would be useful to also provide support for the secondary toolchain within the SDK environment.
Implementation
Bitbake variables
A set of common bitbake variables, defined in meta/conf/bitbake.conf, are used to define how to run the toolchain components and the proper arguments.
A number of key items may need to be changed, depending on the toolchain being provided. These items may include:
* TARGET_CPPFLAGS
* TARGET_CFLAGS
* TARGET_CXXFLAGS
* TARGET_LDFLAGS
* CC
* CXX
* F77
* CPP
* LD
* CCLD
* AR
* AS
* RANLIB
* STRIP
* OBJCOPY
* OBJDUMP
* NM
* SELECTED_OPTIMIZATION
* FULL_OPTIMIZATION
* DEBUG_OPTIMIZATION
Each of the above items has a default definition within the meta/conf/bitbake.conf file. In addition, many of the components are based on specific tune variables such as:
* TUNE_CCARGS * TUNE_LDARGS * TUNE_ASARGS
Deciding which items need to be overridden is up to the author of the layer. It is expected that only a subset of the above referenced variables will need secondary toolchain specific versions for most implementations.
It is suggested that the secondary toolchain values be defined in a single configuration file. Within the example this is "conf/tc-example.conf". The standard name for the file should be 'conf/tc-<toolchain>.conf'.
Override
In order to allow for the layer to selectively override the variables listed above, a standard override syntax is needed.
The standard override syntax will be 'toolchain-${TOOLCHAIN}'. This will allow us to specify multiple alternative and primary toolchain combinations as necessary.
Each of the toolchain related variables can then be overridden using the standard override syntax: _toolchain-<toolchain>, such as:
CC_toolchain-icc = "icc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
In order for an individual recipe to use the secondary toolchain, the user would define: TOOLCHAIN_pn-<recipe> = 'toolchain', such as:
TOOLCHAIN_pn-bash = "icc"
The OVERRIDE is implemented in the tc-secondary.bbclass in the example.
Blacklist/Whitelist
A standard blacklist/whitelist facility is to be implemented. This facility is independent of the existing blacklist class that is part of OpenEmbedded-Core.
Similar to the existing blacklist class, you will be able to define a blacklist, per toolchain type, such as:
TCBLACKLIST[pn] = "toolchain"
or alternatively create a whitelist facility using:
TCBLACKLIST = "toolchain" TCWHITELIST[pn] = "toolchain"
Using the second approach will allow you to blacklist all recipes, and only enable those known to work with the alternative toolchain.
The blacklist items should be defined within conf/tc-<toolchain>-blacklist.conf. The validation of the blacklist should use the standard tc-blacklist.bbclass implementation available within the example.
SDK
In order to make it easier to use the alternative toolchain within the SDK. An additional environment-* file should be created. The purpose of this file is to simply override the settings in the main environment file.
See the example section for an implementation that will create this file automatically.
Implementation Example
An example implementation has been created in the layer "meta-tc-example". The file format follows:
meta-tc-example/ meta-tc-example/bin meta-tc-example/bin/example-toolchain meta-tc-example/conf meta-tc-example/conf/layer.conf meta-tc-example/conf/tc-example.conf meta-tc-example/conf/tc-example-blacklist.conf meta-tc-example/classes meta-tc-example/classes/tc-secondary.bbclass meta-tc-example/classes/tc-blacklist.bbclass
The following example is available at: meta-tc-example.tar.bz2
conf/layer.conf
# We have a conf and classes directory, append to BBPATH
BBPATH .= ":${LAYERDIR}"
BBFILE_COLLECTIONS += "tc-example"
# If we have a recipes directory, add to BBFILES
#BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/*/recipes-*/*/*.bbappend"
#BBFILE_COLLECTIONS += "tc-example"
#BBFILE_PATTERN_tc-example := "^${LAYERDIR}/"
#BBFILE_PRIORITY_tc-example = "5"
Above is standard layer configuration. Note, the commented out sections should be re-enabled if the secondary toolchain includes recipes and/or bbappend files.
PATH := "${PATH}:${LAYERDIR}/bin"
Add to the path a standard system path, a layer specific path. This is the way you can add a secondary toolchain to the path.
# Define the alternative toolchain SECONDARYTC = 'example' # Enable the secondary toolchain INHERIT += 'tc-secondary tc-blacklist'
Define the alternative toolchain name in SECONDARYTC. The inherit causes the tc-secondary.bbclass and tc-blacklist.bbclass to be loaded later. Eventually this class used SECONDARYTC to load in the secondary toolchain configuration files.
conf/tc-example.conf
# Define default system toolchain (default to built-in) TOOLCHAIN = 'gnu'
Define a reasonable default such as gnu. The purpose of this default is to allow someone to specify the default or provide default override items for the built-in toolchain.
# Define a secondary toolchain called 'example' # the 'example' toolchain can be activated using: # # Per package (preferred): # TOOLCHAIN_pn-bash = 'example' # # Globally: # TOOLCHAIN = 'example' # TOOLCHAIN_pn-eglibc = 'gnu'
The above is an example of how to use the alternative toolchain.
# When the secondary toolchain is used, we add an automatic depend... DEPENDS_append_toolchain-example = " virtual/TOOLCHAIN-EXAMPLE" # ...the depend can be provided by a recipe or as an ASSUME_PROVIDED ASSUME_PROVIDED_append = " virtual/TOOLCHAIN-EXAMPLE"
Use the above as an example on how to add additional dependencies to specific packages. This can be used to trigger a recipe to build.
# Default values from bitbake.conf...
# Variables with a '*' are things likely to need to be overridden:
#
# FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
# DEBUG_OPTIMIZATION = "-O -fno-omit-frame-pointer ${DEBUG_FLAGS} -pipe"
# SELECTED_OPTIMIZATION = "${@d.getVar(['FULL_OPTIMIZATION', 'DEBUG_OPTIMIZATION'][d.getVar('DEBUG_BUILD', True) == '1'], True)}"
#
# * TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
#
# * TARGET_CPPFLAGS = ""
# * TARGET_CFLAGS = "${TARGET_CPPFLAGS} ${SELECTED_OPTIMIZATION}"
# * TARGET CXXFLAGS = "${TARGET_CFLAGS} -fpermissive"
# * TARGET_LDFLAGS = "-Wl,-O1 ${TARGET_LINK_HASH_STYLE}"
#
# TARGET_CC_ARCH = "${TUNE_CCARGS}"
# TARGET_LD_ARCH = "${TUNE_LDARGS}"
# TARGET_AS_ARCH = "${TUNE_ASARGS}"
#
# HOST_CC_ARCH = "${TARGET_CC_ARCH}"
# HOST_LD_ARCH = "${TARGET_LD_ARCH}"
# HOST_AS_ARCH = "${TARGET_AS_ARCH}"
#
# * CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
# * CXX = "${CCACHE}${HOST_PREFIX}g++ ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
# * F77 = "${CCACHE}${HOST_PREFIX}g77 ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
# * CPP = "${HOST_PREFIX}gcc -E${TOOLCHAIN_OPTIONS} ${HOST_CC_ARCH}"
# * LD = "${HOST_PREFIX}ld${TOOLCHAIN_OPTIONS} ${HOST_LD_ARCH}"
# * CCLD = "${CC}"
# AR = "${HOST_PREFIX}ar"
# AS = "${HOST_PREFIX}as ${HOST_AS_ARCH}"
# RANLIB = "${HOST_PREFIX}ranlib"
# STRIP = "${HOST_PREFIX}strip"
# OBJCOPY = "${HOST_PREFIX}objcopy"
# OBJDUMP = "${HOST_PREFIX}objdump"
# NM = "${HOST_PREFIX}nm"
# Override values:
CC_toolchain-example = "example-toolchain ${HOST_PREFIX}gcc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
CXX_toolchain-example = "example-toolchain ${HOST_PREFIX}g++ ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
CPP_toolchain-example = "example-toolchain ${HOST_PREFIX}gcc -E${TOOLCHAIN_OPTIONS} ${HOST_CC_ARCH}"
LD_toolchain-example = "example-toolchain ${HOST_PREFIX}ld${TOOLCHAIN_OPTIONS} ${HOST_LD_ARCH}"
The above implements the necessary overrides for the example toolchain.
conf/tc-example-blacklist.conf
Example configuration file that enables the blacklist and defines the defaults.
# Blacklist certain items... TCBLACKLIST[eglibc] = 'example' TCBLACKLIST[bash] = 'example' # # or # # Switch to a global blacklist and whitelist situation... # TCBLACKLIST = 'example' # TCWHITELIST[bash] = 'example'
classes/tc-secondary.bbclass
This is the standard secondary toolchain class. It should be the same in all secondary toolchain layers.
# In order to add overrides, we need to do it later then in the layers.conf
# the only standard way is using an inherit, which requires a bbclass
# Add the necessary override
TOOLCHAINOVERRIDES = ":toolchain-${TOOLCHAIN}"
TOOLCHAINOVERRIDES[vardepsexclude] = "TOOLCHAIN"
OVERRIDES .= "${TOOLCHAINOVERRIDES}"
OVERRIDES[vardepsexclude] += "TOOLCHAINOVERRIDES"
require conf/tc-${SECONDARYTC}.conf
The above code configures the basic overrides and loads the configuration file. There is one limitation to the above however. It will only load one secondary toolchain. A simple way around this is to rename the inherit and change to using something other then SECONDARYTC. This likely will not be a problem.
toolchain_create_sdk_env_script_alt () {
set -x
# Create environment setup script (secondary toolchain)
orig_script=${1:-${SDK_OUTPUT}/${SDKPATH}/environment-setup-${REAL_MULTIMACH_TARGET_SYS}}
script="$orig_script""-${SECONDARYTC}"
rm -f $script
touch $script
echo ". $orig_script" | sed -e "s:${SDK_OUTPUT}::" >> $script
if [ '${CC_toolchain-${SECONDARYTC}}' != '${''CC_toolchain-${SECONDARYTC}}' ]; then
echo 'export CC="${CC_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${CXX_toolchain-${SECONDARYTC}}' != '${''CXX_toolchain-${SECONDARYTC}}' ]; then
echo 'export CXX="${CXX_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${CPP_toolchain-${SECONDARYTC}}' != '${''CPP_toolchain-${SECONDARYTC}}' ]; then
echo 'export CPP="${CPP_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${AS_toolchain-${SECONDARYTC}}' != '${''AS_toolchain-${SECONDARYTC}}' ]; then
echo 'export AS="${AS_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${LD_toolchain-${SECONDARYTC}}' != '${''LD_toolchain-${SECONDARYTC}}' ]; then
echo 'export LD="${LD_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${RANLIB_toolchain-${SECONDARYTC}}' != '${''RANLIB_toolchain-${SECONDARYTC}}' ]; then
echo 'export RANLIB="${RANLIB_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${STRIP_toolchain-${SECONDARYTC}}' != '${''STRIP_toolchain-${SECONDARYTC}}' ]; then
echo 'export STRIP="${STRIP_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${OBJCOPY_toolchain-${SECONDARYTC}}' != '${''OBJCOPY_toolchain-${SECONDARYTC}}' ]; then
echo 'export OBJCOPY="${OBJCOPY_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${OBJDUMP_toolchain-${SECONDARYTC}}' != '${''OBJDUMP_toolchain-${SECONDARYTC}}' ]; then
echo 'export OBJDUMP="${OBJDUMP_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${NM_toolchain-${SECONDARYTC}}' != '${''NM_toolchain-${SECONDARYTC}}' ]; then
echo 'export NM="${NM_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${CFLAGS_toolchain-${SECONDARYTC}}' != '${''CFLAGS_toolchain-${SECONDARYTC}}' ]; then
echo 'export CFLAGS="${CFLAGS_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${CXXFLAGS_toolchain-${SECONDARYTC}}' != '${''CXXFLAGS_toolchain-${SECONDARYTC}}' ]; then
echo 'export CXXFLAGS="${CXXFLAGS_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${LDFLAGS_toolchain-${SECONDARYTC}}' != '${''LDFLAGS_toolchain-${SECONDARYTC}}' ]; then
echo 'export LDFLAGS="${LDFLAGS_toolchain-${SECONDARYTC}}"' >> $script
fi
if [ '${CPPFLAGS_toolchain-${SECONDARYTC}}' != '${''CPPFLAGS_toolchain-${SECONDARYTC}}' ]; then
echo 'export CPPFLAGS="${CPPFLAGS_toolchain-${SECONDARYTC}}"' >> $script
fi
# Fixup any sysroot references
sed -i -e "s:${STAGING_DIR_TARGET}:${SDKTARGETSYSROOT}:g" $script
# All environment scripts require OECORE_NATIVE_SYSROOT
echo 'export OECORE_NATIVE_SYSROOT="${SDKPATHNATIVE}"' >> $script
}
populate_sdk_image_append() {
toolchain_create_sdk_env_script_alt ${SDK_OUTPUT}/${SDKPATH}/environment-setup-${REAL_MULTIMACH_TARGET_SYS}
}
The above code will add an additional environment-setup-*-<toolchain> file. This can be used to enable the right environment to use the alternative toolchain. Note, it does not add anything to the path so the user will still need to configure the path to the alternative toolchain. (The layer creator can always provide the necessary 'nativesdk' or similar package that provides the alternative toolchain as part of the SDK as well.)
classes/tc-blacklist.bbclass
This is the standard secondary toolchain blacklist class. It should be the same in all secondary toolchain layers.
# Implement blacklist/whitelist functionality for alternative toolchains
# Based on the oe-core blacklist bclass
#
# To add a blacklisted package:
# TCBLACKLIST[pn] = 'toolchain'
#
# To blacklist everything, and whitelist a package:
# TCBLACKLIST = 'toolchain'
# TCWHITELIST[pn] = 'toolchain'
include conf/tc-${SECONDARYTC}-blacklist.conf
python () {
tc = d.getVar('TOOLCHAIN', True)
pn = d.getVar('PN', True)
blacklist = d.getVarFlag('TCBLACKLIST', pn, True)
if blacklist and blacklist == tc:
raise bb.parse.SkipPackage("Recipe '%s' is blacklisted for the '%s' toolchain" % (pn, tc))
blacklist = d.getVar('TCBLACKLIST', True)
whitelist = d.getVarFlag('TCWHITELIST', pn, True)
if blacklist and (not whitelist or whitelist != tc):
raise bb.parse.SkipPackage("Recipe '%s' is blacklisted for the '%s' toolchain" % (pn, tc))
}
