[OE-core] [PATCH] oeqa: enable testresults.json for testexport

[Stefan Kral]

Am 11.03.20 um 21:46 schrieb Alexander Kanavin:
> On Wed, 11 Mar 2020 at 21:38, Stefan Kral <sk at typedivision.de 
> <mailto:sk at typedivision.de>> wrote:
>
>     I just started to evaluate the use of testexport.
>
>     If you think about removing that feature, what is the preferred
>     way to
>     test a deployed target image?
>     The idea here is to have two separate pipelines:
>
>     1. build
>     - a target-image-test (same as target-image but additional test
>     facilities enabled, like ptest)
>     - the testexport output
>
>     2. test
>     - fetch build results and install the target-image-test to the
>     real hardware
>     - run the exported runtime tests (host and target based cases)
>     - check/save the generated testresults.json
>
>
> You can do all of these in a single pipeline using testimage:
>
> 1. built target-image-test
> 2. flash target-image-test to real hardware
> 3. bitbake -c testimage target-image-test (set TEST_TARGET to 
> 'simpleremote', and ip/port for ssh access in local.conf)
>
> Is there a reason you need two pipelines?
>
> Alex

Doing build and test in one pipeline turns out to be more complicated 
due to:

- build machine may be different from test machine (with access to hardware)
- build the image is a one time job, tests may have to be repeated 
reproducibly
- executing long running tests slows down image creation (for development)
- tests on dedicated hardware should be runable asynchronously when hw 
is available

Running tests by bitbake -c testimage could be an option but to provide 
a full repo/bitbake/cache setup to run test cases sounds not as easy as 
having just python installed. And for that, testexport does its job 
quite well.

Btw. have you ever thought about to integrate/support some external test 
framework (like robot framework)?

Stefan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20200311/88d2d7cc/attachment.html>