Migrating metadata to OE-Core

The classes and BitBake configuration used in OpenEmbedded-Core require a few changes for recipes brought over from OE-Classic (and for new recipes written by developers used to working with OE-Classic).

Recipes

The following section lists changes applicable to recipes (.bb files).

Mandatory changes

For recipes the following changes are required in order to build successfully:

• Add LIC_FILES_CHKSUM: this specifies one or more files or sections of files from the source that can be used to detect if the license of the software has been changed in a newer version. It is mandatory and checked at the end of do_configure. See Specifying LIC_FILES_CHKSUM for more details.
• Remove legacy staging: legacy staging (do_stage) is no longer supported. Items previously in do_stage should be moved to do_install as appropriate; however they should actually install the files to somewhere under ${D} and not copy them into the staging directory directly or things will break later on. See Legacy staging for more information.
• Ensure LICENSE is present: the LICENSE field, whilst usually included in most recipes in the past anyway, is now mandatory. It should be as specific and as correct as possible (e.g. "GPLv2", not just "GPL") and try to be consistent with other recipes.
• Ensure SRC_URI checksums are present: for entries in SRC_URI pointing to sources other than SCMs or local files, i.e. http, ftp, etc., at least one SRC_URI checksum (md5sum/sha256sum) must be provided. If these are not present for an entry where it is required you will get an error which gives you the appropriate values to be added to the recipe. You can use the "name=" parameter within SRC_URI to distinguish between multiple entries. (This used to be optional, now it is mandatory.)
• Remove comments in string values: recent versions of BitBake no longer allow comments in multi-line strings, so the tradition of being able to comment out lines within a multi-line SRC_URI or when setting configure flags etc. is no longer allowed. Remove the commented out lines or move them to beyond where the string terminates.
• Ensure values are quoted: current versions of BitBake require that values are quoted; this was optional in the past. Single or double quotes are allowed but all values must now be quoted. (This does not affect how numeric values are interpreted, these have always been treated as strings.)
• Use package name override with packaging variables: the runtime package specific variables (RDEPENDS, RRECOMMENDS, RSUGGESTS, RPROVIDES, RCONFLICTS, RREPLACES, FILES, ALLOW_EMPTY, and the pre/post install/uninstall script functions pkg_preinst, pkg_postinst, pkg_prerm, and pkg_postrm) should always have a package name override. For example, to set a runtime dependency for the main package use RDEPENDS_${PN}.
• Replace oe* logging commands: Any calls to oenote, oewarn, oefatal etc. in shell scripts need to be replaced with the "bb" equivalent, i.e. bbnote, bbwarn, bbfatal etc.
• Use FILESEXTRAPATHS to extend file search path: Modifications to FILESPATHPKG and/or FILESPATH (or FILESDIR which is deprecated) should be replaced by prepending to FILESEXTRAPATHS; in addition, you no longer need to define THISDIR (and should not do so). For example, to add a directory to be searched for patches and other files which is named with PN, do the following (note the immediate evaluation operator := and trailing colon, these are important):

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

Changes applicable to the "danny" release and later:

• Python functions should always use four spaces for indentation and not tabs - a warning will be produced if tabs are found. Previously four spaces was the convention but many python functions used tabs. This has been fixed in OE-Core and other layers, and since python uses indentation for statement blocks, is very important that you make the same changes to python functions in your recipes.
• Change proto= to protocol= in SRC_URI - all fetchers have been changed to be consistent. Previously, some fetchers used proto and some protocol.
• Change tasks to package groups - for custom task recipes, rename them from task-*.bb to packagegroup-*.bb, inherit packagegroup instead of inheriting task and remove anything that is now handled by meta/classes/packagegroup.bbclass. Also change any references to task-* to packagegroup-*.
• Change any assumptions that libexecdir is /usr/libexec: we now set libexecdir to ${libdir}/${BPN} to match with the Filesystem Hierarchy Standard (FHS).

Changes applicable to the "dylan" release and later:

• Fix continuation on last line of a comment: if a comment ends in a line continuation (\) then the next line must also be a comment; any instance where this is not the case will now trigger a warning. Either remove the continuation character or make the next line a comment as well.
• Remove use of BROKEN - use EXCLUDE_FROM_WORLD instead; ideally fix up the cause of the recipe being marked as BROKEN.

Changes applicable to the "dora" release and later:

• Rename functions/variables that currently contain "_remove_" or end with "_remove" - _remove is now a BitBake operator, so referencing it when you aren't intending to remove a value from a list variable will result in unexpected behaviour.

Changes applicable to the "daisy" release and later:

• Git SRCREV must match up with the branch: if you are specifying a SRCREV value then it must now be on the branch specified in the URL entry (with the branch= parameter, or the master branch if none is specified). Alternatively if you wish to fetch a revision which isn't on any branch, specify nobranch=1 in the URL.

Changes applicable to the "dizzy" release and later:

• Handle B != S in autotools recipes: if the software being built is capable of building in a separate directory to the source, you don't need to do anything. Otherwise, either patch it so that it does, or inherit autotools-brokensep instead of autotools.
• Handle the --foreign option in autotools recipes: some older autotooled software does not follow the GNU standards in distributing certain files such as ChangeLog, AUTHORS, etc. Most upstream software packages that need to already tell automake to enable foreign mode themselves, but some recipes will need patches for this change. You can easily make the change by patching configure.ac so that it passes "foreign" to AM_INIT_AUTOMAKE() - see this example.
• Use pkg-config instead of standalone configuration scripts: configuration scripts such as xml2-config, pcre-config, freetype-config etc. were being mangled in the past in order to have them return the correct paths within the sysroot, but this is no longer being done - instead of running these scripts, use pkg-config which has proper sysroot support. These scripts will now return an exit code and an invalid option for those calling scripts that don't pay attention to the exit code.
• Do not stage the same files in multiple recipes: overwriting files in the sysroot now triggers an error instead of a warning. Recipes should not be staging files to the sysroot (as a result of installing files within do_install) that are also staged by other recipes; allowing this in the past led to nasty situations that were often difficult to debug.

Changes applicable to the "fido" release and later:

• Set CLEANBROKEN if make clean doesn't work: if a Makefile exists, do_configure runs make clean when re-executing in order to clean out generated files. If this isn't expected to work for the software a recipe builds, set CLEANBROKEN = "1" in the recipe.
• Ensure dependencies on kernel headers are accounted for: recipes that rely on the kernel source code/headers that do not inherit the module classes might need to add explicit dependencies on the do_shared_workdir kernel task, for example: do_configure[depends] += "virtual/kernel:do_shared_workdir"

Changes applicable to the "krogoth" release and later:

• Explicitly expand variable references in python functions: BitBake no longer expands variable references (e.g. ${VARNAME}) in python functions - you should now explicitly expand these using d.expand() or simply by calling d.getVar() if you just want to get the value of a single variable.
• Ensure recipe names are all lower-case - BitBake now requires overrides to be all lower case, which in practice means that anything that can be an override (distro names, machine names and recipe names) must also be lower case if the corresponding overrides are to function.
• Ensure calls to d.getVar() and d.getVarFlag() specify the expansion parameter - d.getVar('SOMEVAR') and d.getVarFlag('SOMEVAR', 'flagname') are no longer allowed - the expansion parameter is now required. d.getVar('SOMEVAR', True) and d.getVarFlag('SOMEVAR', 'flagname', True) is usually what you want; in the rare cases that you explicitly do not want the value expanded, specify False instead of True (though it is worth pointing out that the default in earlier versions was False).
• Handle change of default EXTRA_OEMAKE: EXTRA_OEMAKE now defaults to "" instead of "-e MAKEFLAGS=". Setting EXTRA_OEMAKE to "-e MAKEFLAGS=" by default was a historical accident that has required many classes (e.g. autotools, module) and recipes to override this default in order to work with sensible build systems. When upgrading to the release, you must edit any recipe that relies upon this old default by either setting EXTRA_OEMAKE back to "-e MAKEFLAGS=" or by explicitly setting any required variable value overrides using EXTRA_OEMAKE, which is typically only needed when a Makefile sets a default value for a variable that is inappropriate for cross-compilation using the "=" operator rather than the "?=" operator.

Changes applicable to the morty release and later:

• Handle Python 3: BitBake now requires Python 3 and as such all python code in recipes and classes now needs to be compatible with Python 3.

Changes applicable to master:

• Fix for recipe specific sysroots: we now use a sysroot per recipe to resolve long-standing issues with config script auto-detection of undeclared dependencies. This means all build-time dependencies for your recipe must be explicitly declared, or they won't be populated into the sysroot for the recipe. Additionally we now sanitise PATH, symlinking only the binaries from the host mentioned in HOSTTOOLS and HOSTTOOLS_NONFATAL into our own directory added to PATH, thus any native binaries that you need to call should either be in HOSTTOOLS (or HOSTTOOLS_NONFATAL) at the configuration level, or alternatively provided by a -native recipe that is included in the recipe's DEPENDS value.
• Replace bb.data.getVar()/bb.data.setVar() with d.getVar()/d.setVar() - the old functions have now been removed.

Best practices

The following practices are required for patch submission to layers such as OE-Core and meta-oe, and recommended for recipes in your own layers:

• Machine and distro-specific overrides should be removed from recipes for generic layers (especially meta-oe and OE-Core). These can be moved to bbappends in the appropriate machine / distro layer respectively. (Note however that architecture-specific overrides are allowed if necessary.)
• Fix any QA issues reported during packaging.
• Avoid hardcoding system paths such as /etc, /usr/bin, /usr/share and so on. Instead the values of the appropriate variables should be used (e.g. ${libdir}, ${bindir}, ${datadir} ...). If there are scripts or other files provided by the application such as initscripts you can fix the paths in them within do_install using sed. See bitbake.conf for a list of standard path variables and their default values (which may be - and often are - overridden by the distro configuration).
• Initscripts should have the standard LSB header (especially if no systemd unit file is being provided, since systemd relies upon information in the header when handling an initscript.)
• In-tree patches to be applied to the source should have a proper header explaining what the patch does, a valid Signed-off-by, an Upstream-Status indicator.
• Reset PR to "r0" by removing it. PR is still used, but new recipes being migrated should start again from r0 (the default value if not specified), and if you need it to increment on recipe changes you should use the PR service.
• Use ${BPN} instead of ${PN} where you just want the recipe name without any prefixes/suffixes added by things like multilib and nativesdk (e.g. installation paths)
• Use the useradd class instead of running useradd/groupadd directly to add users/groups.
• NATIVE_INSTALL_WORKS for native recipes is no longer used and should be removed.
• PRIORITY is no longer used and should be removed.
• ENTERPRISE_DISTRO is no longer used. We now have LICENSE_FLAGS to support the uses that ENTERPRISE_DISTRO covered - see the Yocto Project Reference Manual.
• Use d.getVar()/d.setVar() instead of bb.data.getVar()/bb.data.setVar() in python functions. The latter will still work, but are deprecated.
• For class overrides (other than multilib) use class-xxx instead of virtclass-xxx. The latter will still work, but is deprecated.
• "Bashisms" in shell scripts (particularly those that will be executed on the host) should be avoided. This allows compatibility with alternative shells e.g. Ubuntu's default (dash).
• Pre/postinstall scripts (pkg_preinst / pkg_postinst) should ideally be written such that they can run on the host during do_rootfs instead of only on the target. In practice this means that you need to use ${D} before all paths, and anything that is architecture-specific will need to be run under qemu (although the latter is not very common).
• For readability, the variables/functions within a recipe should be ordered in the same order in which they are used - i.e. "meta" information above (DESCRIPTION, SUMMARY, DEPENDS, LICENSE etc.), followed by information needed for do_fetch (SRC_URI, etc.), then anything for do_patch, do_compile, do_install, and then do_package (e.g. PACKAGES, FILES, RDEPENDS, RRECOMMENDS...).
• Shell functions in OE-Core are using tabs for indentation, but other layers usually use consistent indentation with 4 spaces (in shell task, python tasks and for indentation of multi-line variables)

Machine configuration

The following section lists changes applicable to machine configuration files (conf/machine/*).

• Ensure you are using the correct tune files: the tune files in OE-Core have been reorganised and updated; see conf/machine/include/ and check that you are including the appropriate file(s).
• Remove variable settings already handled by the tune files: check if any values you are setting are already set by the tune files; if they are, you don't need to set them again. Of course if you wish to override one of the settings for your machine that is fine.
• Update to a newer Linux kernel: current versions of kernel-sensitive userspace components (e.g. udev) expect a modern kernel; older kernels may not allow the system to boot. If you're not able to do this you may need to provide downgraded versions of these components that are compatible with the kernel you are building. Note that all support for the 2.4 kernel has been removed.
• Check MACHINE_FEATURES value: check the list you are setting against the currently available features (see here, or you can do "git grep MACHINE_FEATURES" in the layers containing recipes you are using.) In particular, the "kernel26" feature can be removed as we no longer support the 2.4 kernel which necessitated having a flag to say the machine is using a 2.6 kernel instead.
• Remove unused variables: the following variables are no longer used:
  • KERNEL
  • PCMCIA_MANAGER
  • MACHINE_KERNEL_VERSION
  • ROOT_FLASH_SIZE
• Use =+ to set IMAGE_FSTYPES: this interacts best when adding/overriding from other configuration files.
• Remove MACHINE_KERNEL_PR: this is no longer needed when BB_SIGNATURE_HANDLER is OEBasicHash (now the default) and the PR service is being used (in the dylan branch and newer); any changes to underlying metadata or dependencies of the kernel will automatically rebuild the kernel so there is no need to force it by hand.

Distro configuration

• Build on top of defaultsetup.conf: OE-Core provides a very basic distro config in meta/conf/distro/defaultsetup.conf which is included automatically by bitbake.conf, thus you only need to include bits of this config file that you want to change.
• Replace references to glibc with eglibc: set TCMODE and TCLIBC to select the toolchain mode and libc variant if you are not using the defaults.
• Check DISTRO_FEATURES value: if you are setting DISTRO_FEATURES, check the list you have against the currently available features (look at the default value in conf/distro/include/default-distrovars.inc in OE-Core; you can also do "git grep DISTRO_FEATURES" in the layers containing recipes you are using.)
• Avoid setting machine configuration variables: machine configuration should almost always be handled in each machine configuration file and not in the distro configuration.
• Update blacklisting: Replace ANGSTROM_BLACKLIST_pn-somerecipe = "message" with BLACKLIST[somerecipe] = "message" (assuming the recipe still exists and the blacklisting is still needed)
• Replace references to ONLINE_PACKAGE_MANAGEMENT: this variable has been replaced by checking for package-management within IMAGE_FEATURES.