[OE-core] Automatic prebuilt management

[Loic Poulain]

HI Andre,

On Fri, 23 Aug 2019 at 20:11, Andre McCurdy <armccurdy at gmail.com> wrote:

> On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <loic.poulain at linaro.org>
> wrote:
> > Say a company works with an OE internal tree, with open-source and
> closed-source packages. The company wants to release the tree so that its
> customers can fully customize the images/distro. The closed sources being
> only available internally, the company has to generate and handle prebuilt
> binaries for proprietary packages. Ideally, with a unified public/internal
> OE tree (same recipes for internal/public build).
> >
> > This question is actually similar to an older thread:
> > https://marc.info/?l=openembedded-core&m=146779329804683
>
> That very old thread was also referenced recently in a follow up where
> I shared a later solution, see:
>
>
> http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html


Thanks, I missed that one.


>
> > I started to work on this and added a 'generate-prebuilt' class which
> generates a tarball of ${D} in deploy/prebuilts after do_install task.
> Symmetrically, It's also possible to create an 'install-prebuilt' class
> which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and
> instead uncompresses a previously generated prebuilt tarball into ${D} and
> continue normally. But I would like something smarter, e.g. first trying to
> check if the SRC_URI is available, if not switching on using the prebuilt
> package if available (e.g. in a DIR_PREBUILT)...
> >
> > Before going further is there already an existing solution for that? do
> you have any recommendation on the easier/best way to achieve this?
>
> I did initially try the approach of having a single recipe which can
> automatically support both building from source and extracting a
> prebuilts tar file, but that (for me anyway) turned out to be a dead
> end. Building from source requires build dependencies and config
> options but extracting a prebuilt tar file does not, so the two end up
> sharing very little... so 90% of the recipe ends up being conditional
> on which mode it's running in. The solution I ended up with (see link
> above) was for the class which creates the prebuilt tar file to also
> create a dedicated mini recipe to extract it.
>

> From my experience however an equally hard part of the problem is the
> distribution of prebuilts. It starts off easy (you share a tar file
> via email or an ftp site and the receiver manually copies to their
> downloads directory...) but that doesn't scale if you need to make
> regular updates.


Very good point. maybe having binaries also in a repo could help a bit
(e.g. using git LFS).


> My solution is discussed a little more in the thread
> linked to above.
>

Your solution is interesting and I'll probably jump on the thread with more
questions.
Having an upstream solution part of oe-core would be nice though.

Regards,
Loic

That's a very good point
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20190830/c1766783/attachment.html>