[OE-core] [RFC PATCH 0/5] revive runtime/cases/_ptest.py

[Robert Yang]

On 07/19/2017 04:16 PM, Robert Yang wrote:
> Hello,
>
> These patches can make ptest test case work, RP has suggested we write a tool to
> do the regression check on ptest result, I think that the use case is like:
>
> $ bitbake <image> -ctestiamge # Suppose we add ptest to default test cases in the future
> # Upgrade a recipe form V1.0 to V1.1
> $ bitbake <image> -ctestiamge # Run the test again
>
> Then the regression check tool can report what's different (passed, failed,
> skipped) between V1.0 and V1.1.
>
> Currently, I'm not sure about where to save the ptest results, I saved it to
> ${WORKDIR}/testimage/ptest_log atm, e.g.:
> tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/testimage/ptest_log

I still have to think about more about where to put the test result.

>
> But it will be removed after -cclean, then no regression check can be made, so
> I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no objections, and
> make -cclean not remove them. (Or only keep the latest 2 results).
>
> And I'm not sure where to add the regression check/tool, maybe one of:
> 1) Add a separate tool in oe-core/scripts, this can make it easy to do regression
>    check among different build directories, and runtime/cases/_ptest.py can invoke it.

I'm leaning to add a script called ptest-regression-check which is more
flexible, the cases/ptest.py or buildhistory (also the user) can run it
when needed.

// Robert

> 2) Add it to runtime/cases/_ptest.py directly
> 3) Add it to buildhistory
>
> I prefer the first one, please feel free to give your comments.
>
> // Robert
>
> The following changes since commit ef68005a8c527e9b1d05b7769f0ec8ebe9ec3f91:
>
>   webkitgtk: Upgrade to 2.16.5 (2017-07-17 13:49:04 +0100)
>
> are available in the git repository at:
>
>   git://git.openembedded.org/openembedded-core-contrib rbt/ptest
>   http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ptest
>
> Robert Yang (5):
>   oeqa/targetcontrol.py: simplify checking for qemu_use_kvm
>   runtime/cases/_ptest.py: revive it
>   oeqa/utils/logparser.py: add skip status
>   runtime/cases/_ptest.py: add skip status
>   runtime/cases/_ptest.py: rename it to ptest.py
>
>  meta/lib/oeqa/runtime/cases/_ptest.py | 103 ----------------------------------
>  meta/lib/oeqa/runtime/cases/ptest.py  |  77 +++++++++++++++++++++++++
>  meta/lib/oeqa/targetcontrol.py        |   5 +-
>  meta/lib/oeqa/utils/logparser.py      |   5 +-
>  4 files changed, 82 insertions(+), 108 deletions(-)
>  delete mode 100644 meta/lib/oeqa/runtime/cases/_ptest.py
>  create mode 100644 meta/lib/oeqa/runtime/cases/ptest.py
>