OEDEM 2016

NOTE: for the OpenEmbedded General Assembly 2016, held the same day at the same location, see the page Berlin, 2016

Location and Time

Friday, October 14 2016
(after ELC-E [Oct 11-13])

Maritim Hotel - Berlin, Germany Stauffenbergstraße 26, 10785 Berlin

Salon 16

Start at 0900

Attendees

• Florin Sarbu "fsarbu"
• Andrei Gherzan "agherzan"
• Khem Raj "khem"
• Armin Kuster "Armpit"
• Marco Cavallini "mckoan"
• Pierre-Jean Texier "pjtexier"
• Koen Kooi "koen"
• Anders Darander
• Sean Hudson "darknighte"
• Changhyeok Bae "chbae"
• Peter Kjellerstedt "Saur"
• Josef Holzmayr "LetoThe2nd"
• Vicentiu Neagoe
• Nicolas Dechesne "ndec"
• Denys Dmytriyenko "denix"
• Philip Balister "Crofton"
• Jan Lübbe "shoragan"
• Enrico Jörns
• Jeff Osier-Mixon "Jefro"
• Andrea Galbusera "gizero"
• Richard Purdie "RP"
• Changhyeok bae "chbae"
• Scott Murray "smurray"
• Behan Webster "behanw"
• Jan-Simon Moeller "dl9pf" 11:30 - 12:45 only
• Bruce Ashfield "zediii"
• Marcin Juszkiewicz "hrw"
• Mark Hatle "fray"
• Rich Persaud
• Kristian Amlie "kamender"
• Marek Vasut
• Eystein Stenberg
• Beth Flanagan

NOTE: if you add your name to this list, please let me know at jefro@jefro.net, otherwise watch this space for a link to attend by phone

Agenda Items

OpenEmbedded eV General Assembly Berlin,_2016

1. LTS (group discussion)
2. Make perl and python distro features? [Saur] (small group)
3. Support for meson build system? [Saur] (small group)
4. Can devtool be made standalone, or how to use latest devtool with older version of OE? [Saur]
   1. Are we going to make large changes in next release? (group discussion)
5. Show off Wind River 'setup' tool (response to discussions from last year and this spring) [Fray] (group discussion)
6. Suggestion for improvement of how to handle site and user configuration. See explanation below. [Saur] (group discussion)
7. BSPs and layer name recommendations (try to start around 3pm to accommodate remote folks from the US) (group discussion)
   1. https://github.com/dl9pf/BSP-LAYER_GUIDE
8. Layer index (group discussion)
9. Yocto Project v2 compatibility (group discussion)
10. Make Mender, the OTA update framework, part of Yocto?
    1. Clarification: General discussion on OTA (group discussion)
11. The use of ${COREBASE}/LICENSE in LIC_FILES_CHKSUM. [Saur] (small group)
    1. Many references to ${COREBASE}/LICENSE causes requirement for inclusion
12. Discuss merging duplicate classes and recipes, (metadata) (small group )
13. Discussion on mesa and splitting libgbm (small group)
14. CVE linking to patches
15. Upstream git URLs in recipes
16. Changes to deploy_image (RP)
17. Discussion of recipe maintainers for oe-core and beyond.

Site/user configuration

We would like to improve the possibilities for a developer to easily configure his/her environment to minimize the required changes that are needed after having sourced oe-init-build-env for a new build directory. Typically there are some options that must always be applied for all developers, e.g., proxy settings, and some that should be applied, e.g., locations of global download mirrors and sstate-caches. These typically belong in a site.conf file common to all developers. Then there are options that I as a developer always want as my defaults, e.g., how many bitbake threads and make processes to use in parallel, and to enable debug-tweaks and debug-tools by default. There is currently no suitable configuration file for these options.

To make it easier to configure these kind of static default options we suggest:

• That we add support for a new configuration file conf/user.conf that is included by bitbake.conf, e.g., after site.conf.
• Add a user specific directory to BBPATH by default, e.g., "${HOME}/.openembedded".
  • It is also suggested that the name of this directory is available as a variable set in bitbake.conf, e.g., "OE_USER_DIR", so that it can be used as a default location for, e.g., DL_DIR in one's user.conf file by setting DL_DIR = "${OE_USER_DIR}/downloads".
• Make it possible to inject directories into BBPATH via the environment, e.g., via a BBPATH_EXTRA variable. This can then be used to specify the common directory (e.g., NFS mount) where conf/site.conf can be found for all developers.

We have a similar solution in place today in our layers. But the drawback is that it is only supported when at least one of our own layers are included in the build, which means that if I, e.g., want to build with only Poky, it will not work out-of-the-box since it will no longer find the site.conf that I must use, and my user.conf that I expect for all my own tweaks are of course not used either.

Minutes

These are taken from https://docs.google.com/document/d/1fDu0lw_BRh0fNXBByqM6OD_iiHwlsoA9D185Xpe1cUg/edit?usp=sharing

Thanks to Jan Lübbe and others for taking notes!

OpenEmbedded GA & OEDEM, October 14 2016

GA:

Start time 9:05 Dissolving eV nonprofit in texas exists Board will send emails to everyone to verify interest in organization

Getting rid of need for proxies No more f2f requirement Can set up telephone bridge & have meetings

Treasurer report: 1700 euro

Meeting end 9:17

OEDEM:

1. LTS (group discussion)
2. Make perl and python distro features? [Saur] (small group)
3. Support for meson build system? [Saur] (small group)
4. Can devtool be made standalone, or how to use latest devtool with older version of OE? [Saur] Are we going to make large changes in next release? (group discussion)
5. Show off Wind River 'setup' tool (response to discussions from last year and this spring) [Fray] (group discussion)
6. Suggestion for improvement of how to handle site and user configuration. See explanation below. [Saur]
7. BSPs and layer name recommendations (try to start around 3pm to accommodate remote folks from the US) (group discussion)
8. Layer index (group discussion)
9. Yocto Project v2 compatibility (group discussion) (https://github.com/dl9pf/BSP-LAYER_GUIDE)
10. Make Mender, the OTA update framework, part of Yocto? Clarification: General discussion on OTA (group discussion)
11. The use of ${COREBASE}/LICENSE in LIC_FILES_CHKSUM. [Saur] (small group) Many references to ${COREBASE}/LICENSE causes requirement for inclusion
12. Discuss merging duplicate classes and recipes, (metadata) (small group )
13. Discussion on mesa and splitting libgbm (small group)

Special thanks to Jan Lubbe for taking notes!

LTS

Yocto-Compatibility requires pushing patches upstream, but Yocto doesn't handle QA for old releases A lot of work for the software vendors rp: have new/separate LTS tree for older releases crofton: how to coordinate between ppl who want this, collect interest in the wiki add maintainer information to the layer repos? rp: write a proposal? → enough interest to set this up

action: Armin will start conversation on the Architecture list

Devtool and other stuff (10:04)

Devtool backporting

Rp: devtool versions are tightly coupled to OE versions, possible way to split this: Classes have become more stable in the recent years? Can we create a stable API for recipes? This API would be a requirement for a standalone devtool. Who would be willing to work on this?

???: version the “API”? Can we report errors on mismatch?

Saur: would like to work on this. They took devtool and half of bitbake and added it to a separate repository. It’s not maintainable this way.

Crofton: do we push too much functionality into bitbake that should be separate? (to make this easier in the future)

Rp: we would need to move some classes from oe-core to bitbake to make bitbake more stand-alone. Currently we need to make only few changes to bitbake itself, because we have a good (low-level) API (without recipes). We have split devtool and recipetool. Which additional tools would we need? Communication problem, ppl don’t know about dev/recipe-tools. Have tutorials, blogs? Many ppl should document interesting/useful features. The interface will become more stable over time.

https://wiki.yoctoproject.org/wiki/Cookbook ?

(no action)

Sysroot-Contamination rp: solution “sysroot per recipe” Koen: want! Rp: previously dependencies blocked this task. Now requirements are all in place. Remaining problem: performance. Use hardlinks? Currently, we have do_populate_sysroot at the end of each recipe. Instead we need a new task at the start to populate the per-recipe sysroot.

General agreement that we want this

Koen: The angstrom autobuild clears the sysroot between each package

Richard: The BMW people want this. This would be a much bigger change than in the recent releases. Currently we can detect contamination, but this would fix it definitely.

Implementation approaches: Hard switch or runtime configuration. RP wants hard switch.

Koen: There has been interest for a long time, but rather low priority.

RP: The timing seems to be good right now. Who will work on it?

Mark: Rather have this than distributed builds in the next cycle.

RP: Hard switch will give us better testing.

Mark: Getting good performance build systems is a problem in many corporations.

RP: We chose multi-config for this cycle in preference to distributed builds.

Khem: ESDK might cause more rebuilding of packages than currently.

RP: ESDK is about locking down ppl to a specific configuration where they are not rebuilding much.

Conclusion: Highly desired, hard switch.

Windriver ‘setup’ Demo

Mark: How to start my own distribution builder? Many ppl. use repo, no one really happy.

Koen: Repo alternative: https://myrepos.branchable.com/

Mark: Wants to gauge community interest in this

Mark: Replace windriver install with something for the community. Have extended the layer index with a distro search. The layer index has a REST API: https://layers.openembedded.org/layerindex/api/ Tool should only setup the layers, everything else in local.conf ….

Exact Steps: mkdir new-project-dir cd new-project-dir git clone .../oe-setup oe-setup/setup.sh --help oe-setup/setup.sh --list-layers (Install build tools) meta-.... foo bar … oe-setup/setup.sh --list-machines Machine description Layer ….. …… ….. oe-setup/setup.sh --list-distro Distro description Layer ….. …… ….. vi oe-setup/bin/setup.py (support for local layer indexes) (search/replace for git urls for local repository mirrors) (default branch configuration based on the branch of the oe-setup checkout) (defaults for base layers, distro, machine, …) oe-setup/setup.sh --machine=beaglebone --distro=poky --layers=meta-oe create repo configuration (contains the discovered/required repositories) (has snippets to create repo configuration based on selected layers/repos) (default.xml contains config) Creates README file Creates bblayers Creates local.conf (containing all available machines and distros, which the default set as on the cmdline) repo manifest -r static manifest (contains git revisions instead of branch names) oe-setup/setup.sh --machine=beaglebone --distro=poky --layers=meta-oe,meta-python,meta-perl,meta-acer It updates the example files with new layers and commits git diff shows changes

Trevor: wants to have a --list-images

Koen: This seems a superset of the linaro scripts, want a dialog frontend.

Mark: Their customers don’t want dialog interfaces.

(side discussion: what’s the name of the default distro? “nodistro”)

Mark: Repo needs a patch to support bare repos. Cannot contribute to google because of lawyers. (fork as rOEpo). This functionality is not required for the OE community.

Khem: Use Hook-Support in repo?

Mark: Too limited.

(demo continued)

Mark: Will be released on github with wind river linux 9. Desire is to make it part of upstream OE

Behan: This is amazing. People were asking about this at the Yocto Project Developer Day

Mark: Some internal ppl want to be able to use a git-URL or local file for the --layers= option.

Mark: Running a local layerindex was hard, but is much easier now (locked to specific version of Django). oe-setup has import_index and export_index to “clone” the config from the public layerindex. We still need a better name!

Koen: Layer order in the index is not always correct. We need to be able to mark layers in the index as broken.

Mark: Layer index needs to grow some functionality to support ordering better

???: How do we handle duplicates?

Mark: Will stay duplicated in the --list-*, can use --machine=openembedded-core:qemuarm.

RP: At runtime bitbake will use bblayers order. Currently consistency is not ensured.

Mark: When multiple indexes are configured, it will use the first layer found in the index order.

Trevor: is it worth adding the capability to select SDKMACHINE, libc, and/or compiler (llvm vs gcc, linaro vs upstream)?

Mark: this seems to be feature creep, it should be a simple too. The index doesn’t index by these fields.

Koen: This can be done indirectly via the distro. How easy is it to make this distro specific? Wants to have config and tool in the same repository.

Mark: change setup config and add separate layer index. For customization change bin/settings, and config snippets.

Conclusion: We want it in OE instead of Yocto (unanimous) Mark: Once able to contribute, talk to halstead for new repo.

Suggestion for improvement of how to handle site and user configuration. (11:39)

Saur: Configure threads, debug tweaks, add strace, DLDIR, … per user and mirror per company. Suggestions: Add conf/user.conf Add ${HOME}/.config/openembedded to BBPATH Make it possible to add to BBPATH via the environment (e.g. BBPATH_EXTRA to company wide NFS).

RP: Because we don’t have this, site.conf is rather useless. The is an open bug about this.

Koen: Don’t want debug tweaks (no passwd) enabled from a magic file). It will leak out! Only if we can have protected variables?

Sean: They add .oe to BBPATH from bblayers.conf. Things get forgotten in there and modifies builds!

RP: We want global config. Of course you can shoot yourself in the foot. You can include a per-user.conf from site.conf.

Sean: Why pass it through the environment? Saur: Avoid manual modifications after cloning e.g. poky.

Khem: His uses will shoot themselves in the foot using .config/oe

Conclusion: Saur will implement the BBPATH_EXTRA and discuss it on the list.

BSPs and layer name recommendations (11:55)

https://github.com/dl9pf/BSP-LAYER_GUIDE

Jan-Simon: presentation of the guide

Sean: Implement checking tool, for example in the layer index.

RP: Need tool in OE-core to check on demand.

Multiple: Enthusiastic agreement that a validation script can run in oe, layer index, etc…

RP: where should this document go? Yocto compatible?

Jan-Simon: In the documentation?

Sean: Hosted in OE itself, base for the tool, feeds into the layer index.

RP: There is some overlap with Yocto-Compatible v2. It needs to be yes/no questions. Can we detect things like bbappending busybox with RFKILL? Yes: Compare ALL sstate checksums with and without the BSP layer (machine not selected). Could use the same approach for distro layers (“Does this layer change anything if the machine/distro is disabled?”)

Koen: How do we check for anti-social changes when enabling the machine?

RP: We can do this by … For Compatible v2 we want to raise the bar. Sanity check for things that keep breaking.

Koen: Additional proposal, separate base, multimedia, … into separate layers? See slide #20 at https://docs.google.com/presentation/d/1Svc0czZEFQoXAjqQyP5FmcUMWUxfftgY8gzwc46Q5qs/edit#slide=id.g16fb8e536d_0_0

Sean: incremental steps to improve the quality of the layers. Have something we can point them to.

???: How to have optional bbappend for things that are not in oe-core.

Jan-Simon: Minimizing the BSP to only Boot is preferred. Graphics, Multimedia, Connectivity is a grey area.

RP: Recommendation to Intel: Use a specific package architecture for strange graphics stuff. Avoids contamination of the generic stuff.

Sean: Get the agreed-on stuff into a script before continuing with more controversial stuff.

Conclusion: Jan-Simon will start a script check the basic stuff, RP should add the checksum checks.

OTA (12:20)

Overview from mender: Package-based Image-based (need bootloader integration) Container-base

Mender and RAUC use A/B images Resin uses containers

How to do the integration in the best way? Patches to u-boot do not apply to vendor forks. How to test?

Currently have a meta-mender layer, but may need to change. Only support MMC-Type storage because they know that works.

???: Split discussion between application updating and system updating.

RP: BSP using yocto-kernel are more easily configurable via fragments. This discussion should make pain points more visible, result maybe just documentation. The image approach is not as integrated into the core as packages.

???: AGL uses OSTree

JanL: keep changeable split from base system. Ostree requires read only root. User merge in core? Joshua (added later): we have some WIP changes for /usr merge, if there’s desire we can address early during 2.3

RP: clear linux have requirements like extended attributes

JanL: You really need a system wide split between volatile and non-volatile

???: we don’t have support in oe-core for building data partitions, the initial image contains rootfs and datafs

RP: WIP has improved

Khem: it’s tightly coupled with bootloader, geared towards SoC

JanL: “building blocks” for configuration

RP: need support for data/os split Bootloader communication Needs ppl who push these topics.

Sean: The next step needs to be a concrete proposal.

RP: This discussion needs to continue beyond OEDEM

Layer Quality (13:45)

BP: how to document layer quality? Stars on github? RP: Up-/Downvote on Layerindex, indicate yocto-compatible, script check Koen: Like debian popcon but for layers? RP: Error-Reporting Mechanism, report success? Mark: Surprisingly easy to change layer index, have to learn django. See in development branch. http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/ RP: we should get more ppl interested in the tools (all django based)

Conclusion: We need to get the word out and get suggestions.

Make perl and python distro features? [Saur] (13:58)

Saur: Optional Python tools in recipes end up in separate packages, but python is still a build-time dependency. Having python as a distro feature should solve that.

Mark: Have you checked how the depending packages use python? May be very invasive. Distro feature can select a python/perl main to allow or blacklist. Dependencies then trickle down. But unsure if this can even be done.

RP: X11 is a good example. Caused a lot of work for the ppl working on the core. Different configs need to be tested. Perl and Python are more difficult, i.e. used in ptest. Build time of perl and python is probably not that big of a problem (compared to the addition pain of making it optional).

<rburton> my opinion re py/perl distro features is that sounds like a vast

         amount of work for little gain.  

Mark: Will be a lot of work, but not totally against it. With the per-package sysroot it might be easier to get working.

RP: fine with individual package configs (like for gdb) or split packages

Conclusion: No change.

Support for meson build system? [Saur] (14:07)

http://mesonbuild.com/

Marcin: Does meson support cross-compile? (Yes.) So it should be easy to add.

RP: gst ppl talked to him at the booth.

Conclusion: Patches welcome, talk to Ross.

The use of ${COREBASE}/LICENSE in LIC_FILES_CHKSUM. [Saur] (14:10)

Many references to ${COREBASE}/LICENSE causes requirement for inclusion.

Koen: ${COREBASE}/LICENSE is not a license. COPYING.MIT is correct

Conclusion: RP: This needs to go to the ML

Discuss merging duplicate classes and recipes (metadata)

DTBs

Koen: DTB file naming is not good.

RP: Needs a proposal/bugs in the tracker.

Koen: The deploy dir should contain files which can be copied to a SD card and simply boot. People expect that stuff in oe-core is that way for a reason and don’t try to change.

Sean: This may be a result of us being older now.

RP: For architectural changes, we need to explain why we’re doing that. Otherwise ppl will complain that you broke their builds.

Duplicate Recipes (14:21) PB: Recipes with the same name in two layers

PB: The OpenStack layer is a large pain.

PB: Before adding a recipe, search layer index.

PB: Requiring specific versions may need duplication.

RP: One important example is the go stuff. Consider pulling it into core. We need to look at these cases. Talk to each other.

Khem: Can we have a duplicate recipe report? Locally and on the layer index

Sean: need a tool, good concrete suggestion

PBa: Report these on the ML

Koen: Are versioned RDEPENDS supposed to work? Bitbake works for one, but explodes when he has to many of them. (Django apps are an example)

RP: Bitbake doesn’t do any checking of these depends.

Multiple: File a bug.

Sean: We should try to get rid of duplicates where possible.

RP: are there other examples of duplication that need to be cleaned up?

Koen: We have many u-boot recipes. At least have namespaces to avoid conflicts?

Marcin: It the same with the kernel.

PB: Namespace the sh*t out of it.

Marek: It would be good to have u-boot updates more often.

Koen: Updating needs tests on real HW.

PB: We want to only have patches in BSP-layers not full recipes. This also makes it easier to pressure evil vendors.

???: People only beat u-boot until it works and never change it again.

Sean: We need to make it easy for ppl building products to base on the latest and greatest version (kernel and bootloader).

RP: The whole kernel-yocto area needs refactoring. Maybe split some of that into the liboe directory?

???: Duplicate machine names?

Koen: Look at it case by case. Meta-yocto should not duplicate machines from member companies.

Multiple: The bblayers order is a very brittle area.

RP: Would like to redo the internal bitbake layer implementation. Ppl. will complain because subtle build issues/failures. Very fragile

Mark: We now have layer recommends. Maybe just remove layer priorities and just use the BBPATH order.

Saur: The problem is that we have two priorities?

RP: We have more than two.

RP: He and Mark should look at the situation and suggest a way forward.

Conclusion: Duplicate recipes and machines need to be brought up with the maintainers.

Conclusion: Mark, RP, Chris and others should look to create a proposal about resolving the confusion over layer order/priority/etc.

Changes to deploy_image (14:48)

RP: Image generation has changed in 2.2. The system didn’t have any idea which goes in to images/SDKs. That’s the reason that workaround like rmoldimage exists. We now can track these in the sstate, also it produces the manifest. Sstate also now removes the old images, can get rid of rmoldimage.

Mark: have we solution for ppl using the old images

RP: No. Just copy/move out of the tmp/deploy before the next build.

Saur: Do we still need the timestamp.

RP: The timestamp is from the start of the build.

Koen: Ppl have been asking him that bitbake should print which images have been generated.

RP: If you want to keep old images, add an additional task which does the copy.

Conclusion: Document

Coffee Break (-15:22)

RP: Wants to talk about recipes which have markup of recipes for upstream git urls (after coffee)

Discussion on mesa and splitting libgbm

Conclusion: RP: Topic should be discussed with Ross on mailing list.

Upstream git URLs in recipes (RP) (15:22)

For some tools we have git urls and tar urls. For others only one of them. The can be of different quality. Maybe we can have a way to generate two instances from one file, so we can have git and tar with less divergence.

Sean: The have that as a separate tool. Use tarball by default and have it easy to switch to git.

Khem: This will also simplify upstreaming.

Sean: Devtool may help with switching back and forth. We can also do shallow clones for git.

Saur: bbclassextend implementations need to have code to handle different combinations (native/sdk/…)

RP: Has written bbclassextend to build and test in qemu 30 zephyr configs from a single file. You shouldn’t need to worry about native or not.

Saur: There may be more changes between tarball and git than the URL.

RP: It will work for many of them. Selection is done via preferred_version

ndec: how do we know the version?

Sean: The topic was just to see if there is interest.

Mark: some customers always rather want git repositories

Sean: some customers always want to have tarballs

RP: This support will make that work easier.

Conclusion: There is enough interest, we should investigate solution.

Discussion of recipe maintainers for oe-core and beyond (15:35)

BP: The advisory board wants to increase the number of maintainers. Maintainers: please watch the mailing list for package update emails (“[Recipe reporting system] Upgradable recipe name list”). That list is very Intel and WindRiver heavy.

RP: This list is a Yocto initiative. They try to get more involvement from the members. There is an incentive to get more ppl involved (recipe reporting system): Tool to check if packages are outdated Sends mails ptest support “Automatic upgrade helper” Automatically tries to update the recipe by itself, (runs ptest?) and then send a patch to the maintainer. Hoping others take this and apply it to other layers outside oe-core Delegated to Ross Burton Discussion is still ongoing on how to assign packages to people. Also document how to use this tool in other layers / environment.

Sean: How do we stay up to date?

Conclusion: RP: There is a status somewhere on the website… http://recipes.yoctoproject.org/ Weekly Bug Triage (https://wiki.yoctoproject.org/wiki/Bug_Triage email Stephen Jolley) & confcalls/irc meetings will also be held. https://www.yoctoproject.org/tools-resources/community/monthly-technical-call

RP: He get many questions on why we don’t have a MAINTAINERS file. The code structure doesn’t really lend itself to such a file. He doesn’t want to give the impression of giving a service level agreement, at least with the current amount of contributors.

Trevor: Use the MAINTAINERS file to autogenereate the CC list.

Mark: He’d like to be CC’ed on some files. If patchwork is working, it can send the mails to the correct people. Linking Commits to CVE numbers (15:46) PB: Do we want to send those to the security list?

Mark: Some ppl expect that they would see security changes on that list. If PB doesn’t find anyone to write the git hook, Mark will help.

Mark: if nobody else volunteers to help, contact me and I will find someone (at my employer) who will help do this.

Conclusion: PB will nag ppl, fallback: Remind Mark and he will find someone to help

Wrap-up

??? Last november/december there seemed to a low activity. Why?

RP: There is a milestone cycle “merge/stabilize”

Many: Get the patches in early after release.