CPLWorkerThreadPool may deadlock at destruction

[Even Rouault]

Reported by Paul Thompson

I believe I may have found a bug in GDAL's CPLWorkerThreadPool involving
a deadlock on destruction.

I've attached a minimal reproducing test case, simply repeatedly opening
and destroying CPLWorkerThreadPool and relying on the destructor to wait
for the worker threads.
I've tested this against gdal release 2.1.1 and trunk (2.2.0dev)

I've encountered this behaviour, which I think is a bug, while using a
separate application which uses GDAL, which hangs with a stack trace as
follows:

#0 in pthread_join at pthread_join.c:92
#1 in CPLJoinThread at cpl_multiproc.cpp:1949
#2 in CPLWorkerThreadPool::~CPLWorkerThreadPool at
cpl_worker_thread_pool.cpp:75
#3 in GWKThreadsEnd at gdalwarpkernel.cpp:394
#4 in GDALWarpOperation::~GDALWarpOperation at gdalwarpoperation.cpp:156
#5 in VRTWarpedDataset::CloseDependentDatasets at vrtwarped.cpp:373
#6 in VRTWarpedDataset::~VRTWarpedDataset at vrtwarped.cpp:309

In my attached minimal test case
// The main thread waits at:
// CPLJoinThread cpl_multiproc.cpp:1949
// in CPLWorkerThreadPool::~CPLWorkerThreadPool at
cpl_worker_thread_pool.cpp:75

// The worker threads wait at:
// CPLWorkerThreadPool::GetNextJob at cpl_worker_thread_pool.cpp:458

#include "cpl_worker_thread_pool.h"

void initFunc(void* arg)
{
}

void workFunc(void* arg)
{
}

int main(void)
{
    for(unsigned int r=0; r < 10000; ++r)
    {
        CPLWorkerThreadPool pool;

        pool.Setup(10, &initFunc, NULL);

        for(unsigned int n=0; n < 20; ++n)
        {
            pool.SubmitJob( &workFunc, NULL );
        }

        // The main thread waits at:
        //  CPLJoinThread cpl_multiproc.cpp:1949
        //  in CPLWorkerThreadPool::~CPLWorkerThreadPool at cpl_worker_thread_pool.cpp:75

        // The worker threads wait at:
        // CPLWorkerThreadPool::GetNextJob at cpl_worker_thread_pool.cpp:458
    }

    return 0;
}

[Even Rouault]

In 35362:

Fix deadlock at CPLWorkerThreadPool destruction (fixes #6646)

[Even Rouault]

In 35363:

Fix deadlock at CPLWorkerThreadPool destruction (fixes #6646)