[OE-core] RFC: BitBake's tinfoil API

[Paul Eggleton]

Hi there,

As some of you may know, BitBake has a module under lib/bb called "tinfoil" 
which you can use to create simple Python utilities that can call into BitBake 
code - for example, you can get variable values and parse recipes. Several 
utilities shipped with BitBake/OE make use of this (e.g. bitbake-layers, oe-
pkgdata-util, devtool, etc.). The OE layer index and Recipe Reporting Service 
update scripts also use it.

One of tinfoil's limitations up to now is that it bypasses BitBake's client-
server model - it only starts up the server, and it initialises the server 
only partially. This is fine for what it needs to do, but it does mean that 
scripts that use it cannot be run when BitBake is running in memory resident 
mode - there can be only one server instance per TMPDIR. Recently I've been 
trying to address that by moving tinfoil to the client side; my WIP branch is 
here:

  http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/tinfoil2

There are some other benefits to moving to this model:

* We can now connect an actual UI, thus you get things like the recipe parsing 
progress bar and parse error handling for free

* We can now handle events generated by the server - in the OE layer index for 
example, I'm pretty certain the lack of this is the source of a bug where we 
leak memory. (Note - this is still todo on my branch).

* We can potentially extend this to do more stuff, e.g. actually running build 
operations - that simply wasn't possible with tinfoil before. I haven't 
actually provided any specific access to this yet but it shouldn't be too hard.

The downside is that tinfoil no longer has direct access to cooker, the global 
datastore, and the cache. I have extended data_smart to provide remote access 
capability for the datastore, and created "fake" cooker and cooker.recipecache 
/ cooker.collection adapter objects in order to avoid breaking too many 
tinfoil-using scripts that might be out there (we've never officially documented 
tinfoil or bitbake's internal code, but we can still make accommodations where 
practical). I've at least gone far enough to support bitbake-layers with some 
changes, but I know there are scripts such as Chris's "bb" out there that do 
make other calls into BitBake code that I'm not providing access to through 
the adapters (yet?).

However, it struck me that we have an opportunity to create a proper API this 
time rather than simply exposing the internals of BitBake (which do 
occasionally change, and that has resulted in breakage of tinfoil-using 
scripts in the past). The datastore is probably fine to expose, but parts of 
cooker and the cache would probably be better represented as methods on 
tinfoil, object structures underneath it, or simply as client/server calls 
that can be accessed through runCommand() (with the current architecture, we 
need to add the latter anyway). Once we've set up the new API, we can then 
document it.

Does anyone have any thoughts on this? I'm particularly interested in hearing 
from people using tinfoil in their own scripts or who would be interested in 
writing such scripts in the future - what kind of API would you like to have?

Thanks,
Paul

-- 
Paul Eggleton
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47