[bitbake-devel] [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 8/19/19 4:34 PM, Robert Yang wrote:
> 
> On 8/16/19 7:03 AM, richard.purdie at linuxfoundation.org wrote:
>> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote:
>>>
>>> On 5/14/19 7:02 PM, Robert Yang wrote:
>>>>
>>>> On 5/12/19 4:28 PM, Richard Purdie wrote:
>>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>>>>> The bb.fatal() raises BBHandledException() which causes double
>>>>>> exceptions,
>>>>>> e.g.:
>>>>>>
>>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>>>>> $ bitbake -p
>>>>>> [snip]
>>>>>> During handling of the above exception, another exception
>>>>>> occurred:
>>>>>> [snip]
>>>>>> ERROR: The following required tools (as specified by HOSTTOOLS)
>>>>>> appear to be
>>>>>> unavailable in PATH, please install them in order to proceed:
>>>>>>     hello
>>>>>>
>>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix
>>>>>> the double
>>>>>> exceptions.
>>>>>>
>>>>>> [YOCTO #13267]
>>>>>>
>>>>>> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
>>>>>> ---
>>>>>>    bitbake/lib/bb/cookerdata.py | 4 +++-
>>>>>>    1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>>>>> b/bitbake/lib/bb/cookerdata.py
>>>>>> index f8ae410..585edc5 100644
>>>>>> --- a/bitbake/lib/bb/cookerdata.py
>>>>>> +++ b/bitbake/lib/bb/cookerdata.py
>>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>>>>                if multiconfig:
>>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>>>>> ), self.data)
>>>>>> -        except (SyntaxError, bb.BBHandledException):
>>>>>> +        except bb.BBHandledException:
>>>>>> +            raise
>>>>>> +        except SyntaxError:
>>>>>>                raise bb.BBHandledException
>>>>>>            except bb.data_smart.ExpansionError as e:
>>>>>>                logger.error(str(e))
>>>>>
>>>>> Hi Robert,
>>>>>
>>>>> This doesn't sound right, where is this exception being printed a
>>>>> second time? The point of "BBHandledException" is to say "don't
>>>>> print
>>>>> any further traces for this exception". If this fixes the bug, it
>>>>> means
>>>>> something somewhere is printing a trace for a second time when it
>>>>> should pass through BBHandledException?
>>>>
>>>> Hi RP,
>>>>
>>>> I found another serious problem when tried to raise
>>>> BBHandledException. There
>>>> is a deadlock when a recipe is failed to parse, e.g.:
>>>>
>>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>>> $ bitbake -p
>>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
>>>> 'helloworld'
>>>> [hangs]
>>>>
>>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub
>>>> processes
>>>> are still existed, and we need kill them manually, otherwise we
>>>> can't start
>>>> bitbake again.
>>>
>>> BTW, things becomes much better after the following two patches are
>>> merged:
>>> bitbake: knotty: Fix for the Second Keyboard Interrupt
>>> bitbake: cooker: Cleanup the queue before call process.join()
>>>
>>> Now we hardly can reproduce the problem:
>>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
>>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
>>> done
>>>
>>> It's not easy to hang any more, but still hangs sometimes, I tried to
>>> debug it,
>>> but didn't find the root cause, the ui/knotty.py can't get event from
>>> server,
>>> and goes into a dead loop.
>>>
>>>               event = eventHandler.waitEvent(0)
>>>               if event is None:
>>>                   if main.shutdown > 1:
>>>                       break
>>>                   termfilter.updateFooter()
>>>                   event = eventHandler.waitEvent(0.25)
>>>                   if event is None:
>>>                       continue
>>>
>>> The main.shutdown is always 0 when it hangs.
>>
>> In theory there are timeouts there so it should never hang waiting for
>> an event. Is it looping and not getting an event? or is the other end
>> disconnected?
>>
>> I guess the question is what we can do to detect a dead connection, or
>> if the server is still alive, why the server is hanging and not sending
>> any events?
> 
> After more investigations, it may hang at two places, they are very rarely to
> happen, but does happen, I can use the following command to reproduce it in
> 10 minutes:
> 
> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
> 
> * Hangs #1 in cooker.py:
> 
> 2065         # Cleanup the queue before call process.join(), otherwise there 
> might be
> 2066         # deadlocks.
> 2067         while True:
> 2068             try:
> 2069                self.result_queue.get(timeout=0.25)
> 2070             except queue.Empty:
> 2071                 break
> 
> It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
> here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
> but still have the problem, I think that it's a bug of python3's 
> multiprocessing, and we can call self.result_queue.cancel_join_thread()
> to fix the problem.
> 
> 
> * Hangs #2 in cooker.py:
> 2073         for process in self.processes:
> 2074             if force:
> 2075                 process.join(.1)
> 2076                 process.terminate()
> 2077             else:
> 2078                 process.join()
> 
> 
> It hangs at process.join(), I added debug code there, it is because the process
> is alive when join() it, I think that we can use a while loop to check whether
> the process is alive or not before join(), and force join() after many tries.
> 
> I will do more testing before send the patches, make sure it won't hang in hours.


After a lot tries, the only reliable way to avoid hanging is os.kill(), I've
sent the patch for it, and added reason in the commit message.

// Robert

> 
> // Robert
> 
>>
>> Thanks for the patches, those were tricky issues to track down and
>> solve!
>>
>> Cheers,
>>
>> Richard
>>
>>
>>