[OE-core] [RFC PATCH 0/5] revive runtime/cases/_ptest.py

[Robert Yang]

On 07/21/2017 08:32 PM, Richard Purdie wrote:
> On Thu, 2017-07-20 at 10:09 +0800, Robert Yang wrote:
>>
>> On 07/19/2017 04:16 PM, Robert Yang wrote:
>>>
>>> Hello,
>>>
>>> These patches can make ptest test case work, RP has suggested we
>>> write a tool to
>>> do the regression check on ptest result, I think that the use case
>>> is like:
>>>
>>> $ bitbake <image> -ctestiamge # Suppose we add ptest to default
>>> test cases in the future
>>> # Upgrade a recipe form V1.0 to V1.1
>>> $ bitbake <image> -ctestiamge # Run the test again
>>>
>>> Then the regression check tool can report what's different (passed,
>>> failed,
>>> skipped) between V1.0 and V1.1.
>>>
>>> Currently, I'm not sure about where to save the ptest results, I
>>> saved it to
>>> ${WORKDIR}/testimage/ptest_log atm, e.g.:
>>> tmp/work/qemux86-poky-linux/core-image-minimal/1.0-
>>> r0/testimage/ptest_log
>> I still have to think about more about where to put the test result.
>
> Would buildhistory make sense for these since we already have a
> mechanism to store results there between builds?

Thanks, I will add it to buildhistory.

>
>>> But it will be removed after -cclean, then no regression check can
>>> be made, so
>>> I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no
>>> objections, and
>>> make -cclean not remove them. (Or only keep the latest 2 results).
>>>
>>> And I'm not sure where to add the regression check/tool, maybe one
>>> of:
>>> 1) Add a separate tool in oe-core/scripts, this can make it easy to
>>> do regression
>>>    check among different build directories, and
>>> runtime/cases/_ptest.py can invoke it.
>> I'm leaning to add a script called ptest-regression-check which is
>> more flexible, the cases/ptest.py or buildhistory (also the user) can
>> run it when needed.
>
> I'd either add a script, or perhaps an option to the buildhistory-diff?
> Paul might have some adivce on that. I would like to understand if
> there is a good reason we can't use buildhistory to at least help with
> some of this?

Thanks, the script (I named it prc -- Ptest Result Compare  or Ptest Regression
Compile) already done, I will investigate buildhistory to see how to merge it
into buildhistory.

// Robert

>
> Cheers,
>
> Richard
>