[Openembedded-architecture] Bitbake server rework

[Martin Jansa]

On Fri, Jul 21, 2017 at 08:55:10AM +0100, Richard Purdie wrote:
> I just wanted to give people a heads up that a pretty significant
> change in bitbake's internal behaviour has merged.
> 
> In short, the server backend to bitbake has been reworked and bitbake
> now always forks off a "cooker" or server process (if needed) and then
> connects to it using a unix domain socket.
> 
> The xmlrpc and process server backends are replaced with one unified
> interface where there is always a process server which can optionally
> provide an xmlrpc interface for remote connections.
> 
> This means you get "memres" behaviour simply by setting
> BB_SERVER_TIMEOUT to some positive value and then the cooker will
> remain around for that length of time before unloading from memory.
> 
> Right now, the default is 0 giving the same behaviour as before, with
> the change that the cooker is always an independent process.
> 
> BB_SERVER_TIMEOUT being non-zero is known to break some oe-selftests
> right now, that is some we can start to address now we have the code
> baseline merged. It does seem to work well in normal use although the
> internal server resets for new connections may need optimisation too.
> 
> I do expect we may have some teething issues with this as its a major
> change to the way we've done things. I (and other bitbake developers)
> do believe this is the best way forward for the project though and now
> is the best time to merge this and get people using it. Not least, it
> replaces a load of horrible code nobody really understood with
> something simpler and more understandable and capable. I am delaying M2
> a little to ensure we get this in now and have time to fix issues
> before 2.4.

Are there any changes needed for PRserv (or it's configuration)?

In all my builds I'm now seeing warning about timeout unless I start
second build shortly after first one (or maybe after first one already
finished parsing or something like that).

NOTE: Starting bitbake server...
NOTE: Started PRServer with DBfile: /home/jenkins/oe/world/shr-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 35637, PID: 2856
WARNING: Timeout while attempting to communicate with bitbake server
Parsing recipes...done.
Parsing of 2442 .bb files complete (0 cached, 2442 parsed). 3262
targets, 136 skipped, 0 masked, 0 errors.


Different builder with the failure on first build (due to bad branch in
meta-raspberry being used):

OE raspberrypi3-64@ ~/build/oe-core $ bitbake zlib
NOTE: Starting bitbake server...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 36441, PID: 28241
WARNING: Timeout while attempting to communicate with bitbake server
WARNING: Timeout while attempting to communicate with bitbake server
ERROR: /OE/build/oe-core/meta-raspberrypi/recipes-core/images/rpi-test-image.bb: Deprecated variable(s) found: "IMAGE_DEPENDS_rpi-sdimg". Use do_image_<type>[depends] += "<recipe>:<task>" instead
ERROR: /OE/build/oe-core/meta-raspberrypi/recipes-core/images/rpi-hwup-image.bb: Deprecated variable(s) found: "IMAGE_DEPENDS_rpi-sdimg". Use do_image_<type>[depends] += "<recipe>:<task>" instead
ERROR: /OE/build/oe-core/meta-raspberrypi/recipes-core/images/rpi-basic-image.bb: Deprecated variable(s) found: "IMAGE_DEPENDS_rpi-sdimg". Use do_image_<type>[depends] += "<recipe>:<task>" instead
ERROR: Failed to parse recipe: /OE/build/oe-core/meta-raspberrypi/recipes-core/images/rpi-basic-image.bb

Summary: There were 2 WARNING messages shown.
Summary: There were 4 ERROR messages shown, returning a non-zero exit code.

... fix the branch ...

OE raspberrypi3-64@ ~/build/oe-core $ bitbake zlib
NOTE: Starting bitbake server...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 41013, PID: 28287
Parsing recipes: 100% |#############################################################################################| Time: 0:01:37
Parsing of 2306 .bb files complete (0 cached, 2306 parsed). 3080 targets, 176 skipped, 0 masked, 0 errors.
WARNING: No bb files matched BBFILE_PATTERN_ruby-layer '^/OE/build/oe-core/meta-openembedded/meta-ruby/'
NOTE: Resolving any missing task queue dependencies

Build Configuration:
...

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-architecture/attachments/20170721/d9f53788/attachment-0002.sig>