[OE-core] [PATCH] oe-selftest: runqemu: add tests for qemu boot and shutdown

[Yeoh, Ee Peng]

Hi Anuj,

Thanks for your inputs on comment and code that was not used. I shall remove the comment and code that was not used. 

For the qemu.runner.stop_thread(), this was trigger to stop an instance of a LoggingThread class that was used primary to logging the qemu through serial console. Stopping this thread will not stop the qemu itself. In case that this logging thread object was not stop, when the qemu finally shutdown by the shutdown command, this logging thread will encounter an exception (Exception: Console connection closed unexpectedly). From the testing on multiple qemu architecture, this testcase was able to catch the shutdown hung error facing by existing qemuarm (https://bugzilla.yoctoproject.org/show_bug.cgi?id=12499), while it was as expected on qemux86, qemuppc, and other. I shall add comment to explain the important of qemu.runner.stop_thread() and the reason behind this. 

For the skipTest, I agreed with you, it shall be replace by assertion to highlight environment setup failure. 

Please let me know if you have any more input. Thank you very much!

Thanks,
Yeoh Ee Peng 


-----Original Message-----
From: Mittal, Anuj 
Sent: Tuesday, April 10, 2018 3:01 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH] oe-selftest: runqemu: add tests for qemu boot and shutdown

On 04/10/2018 01:43 AM, Yeoh Ee Peng wrote:
> QA team were testing qemu boot image and shutdown on each qemu 
> architecture manually. Add automated test to test qemu boot on
> ext4 and nfs, finally check that it can shutdown properly.
> 
> Original runqemu tests was dedicated for MACHINE=qemux86-64 and it was 
> testing various live image (iso and hddimg) will be able to boot while 
> live image was not supported on all qemu architecture.
> 
> The new tests were designed as a separate class as this tests focus on 
> testing qemu boot and shutdown on each qemu architecture.
> Furthermore, this tests focus on testing qemu could shutdown as 
> expected.
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh at intel.com>
> ---
>  meta/lib/oeqa/selftest/cases/runqemu.py | 61 
> +++++++++++++++++++++++++++++++--
>  1 file changed, 59 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/runqemu.py 
> b/meta/lib/oeqa/selftest/cases/runqemu.py
> index 47d41f5..7288ab2 100644
> --- a/meta/lib/oeqa/selftest/cases/runqemu.py
> +++ b/meta/lib/oeqa/selftest/cases/runqemu.py
> @@ -3,9 +3,10 @@
>  #
>  
>  import re
> -
> +import tempfile
> +import time
>  from oeqa.selftest.case import OESelftestTestCase -from 
> oeqa.utils.commands import bitbake, runqemu, get_bb_var
> +from oeqa.utils.commands import bitbake, runqemu, get_bb_var, runCmd
>  from oeqa.core.decorator.oeid import OETestID
>  
>  class RunqemuTests(OESelftestTestCase):
> @@ -136,3 +137,59 @@ SYSLINUX_TIMEOUT = "10"
>          cmd = "%s %s" % (self.cmd_common, rootfs)
>          with runqemu(self.recipe, ssh=False, launch_cmd=cmd) as qemu:
>              self.assertTrue(qemu.runner.logged, "Failed: %s" % cmd)
> +
> +class QemuTest(OESelftestTestCase):
> +
> +    @classmethod
> +    def setUpClass(cls):
> +        super(QemuTest, cls).setUpClass()
> +        cls.recipe = 'core-image-minimal'
> +        cls.machine =  get_bb_var('MACHINE')
> +        cls.deploy_dir_image =  get_bb_var('DEPLOY_DIR_IMAGE')
> +        cls.cmd_common = "runqemu nographic"
> +        cls.qemuboot_conf = "%s-%s.qemuboot.conf" % (cls.recipe, cls.machine)
> +        cls.qemuboot_conf = os.path.join(cls.deploy_dir_image, cls.qemuboot_conf)
> +        result = bitbake(cls.recipe)

This 'result' isn't used.

> +
> +    def _start_qemu_shutdown_check_if_shutdown_succeeded(self, qemu, timeout):
> +        status, output = qemu.run_serial("shutdown -h now")
> +        qemu.runner.stop_thread()

If you call stop_thread like this, this test will probably always pass?
Please check the value of output and then is_alive/check().

> +        print('DEBUG: shutdown and stop thread')
> +        time_track = 0
> +        while True:
> +            is_alive = qemu.check()
> +            if not is_alive:
> +                return True
> +            if time_track > timeout:
> +                return False
> +            time.sleep(1)
> +            time_track += 1
> +            print(time_track)

We shouldn't be printing these values.

> +
> +    def test_qemu_can_shutdown(self):
> +        if not os.path.exists(self.qemuboot_conf):
> +            self.skipTest("%s not found" % self.qemuboot_conf)
> +        cmd = "%s %s" % (self.cmd_common, self.qemuboot_conf)
> +        shutdown_timeout = 120
> +        with runqemu(self.recipe, ssh=False, launch_cmd=cmd) as qemu:
> +            qemu_shutdown_succeeded = self._start_qemu_shutdown_check_if_shutdown_succeeded(qemu, shutdown_timeout)
> +            self.assertTrue(qemu_shutdown_succeeded, 'Failed: qemu 
> + does not shutdown within timeout(%s)' % shutdown_timeout)
> +
> +    def test_qemu_can_boot_nfs_and_shutdown(self):
> +        bitbake('meta-ide-support')
> +        rootfs_tar = "%s-%s.tar.bz2" % (self.recipe, self.machine)
> +        rootfs_tar = os.path.join(self.deploy_dir_image, rootfs_tar)
> +        if not os.path.exists(rootfs_tar):
> +            self.skipTest("%s not found" % rootfs_tar)
> +        tmpdir = tempfile.mkdtemp(prefix='qemu_nfs')
> +        tmpdir_nfs = os.path.join(tmpdir, 'nfs')
> +        cmd_extract_nfs = 'runqemu-extract-sdk %s %s' % (rootfs_tar, tmpdir_nfs)
> +        runCmd(cmd_extract_nfs)
> +        if not os.path.exists(self.qemuboot_conf):
> +            self.skipTest("%s not found" % self.qemuboot_conf)

This and others should be marked failure I think if env wasn't setup properly after doing bitbake. Does skipTest result in a failure?

Thanks,
Anuj