Fix rare divide by zero in JPEG

[gaige]

Under very rare circumstances, the quantification routines can divide by zero. Checking for zero resolves that.

jpeg/libjpeg/jquant2.c:

532,534c532,540
<   cinfo->colormap[0][icolor] = (JSAMPLE) ((c0total + (total>>1)) / total);
<   cinfo->colormap[1][icolor] = (JSAMPLE) ((c1total + (total>>1)) / total);
<   cinfo->colormap[2][icolor] = (JSAMPLE) ((c2total + (total>>1)) / total);
---
>   if (total>0) {
> 	  cinfo->colormap[0][icolor] = (JSAMPLE) ((c0total + (total>>1)) / total);
> 	  cinfo->colormap[1][icolor] = (JSAMPLE) ((c1total + (total>>1)) / total);
> 	  cinfo->colormap[2][icolor] = (JSAMPLE) ((c2total + (total>>1)) / total);
>   } else {
> 	  cinfo->colormap[0][icolor] = (JSAMPLE) 0;
> 	  cinfo->colormap[1][icolor] = (JSAMPLE) 0;
> 	  cinfo->colormap[2][icolor] = (JSAMPLE) 0;
>   }

[warmerdam]

Gaige,

Would you be willing to file this issue with the libjpeg project at sourceforge, and also provide an example of how to reproduce the problem that is being fixed? Once it is fixed upstream, I can pull down the fix into GDAL.

​http://sourceforge.net/tracker/?group_id=159521&atid=812162

[Jukka Rahkonen]

Can anybody tell if this is fixed in libjpeg?

[Even Rouault]

Replying to jratike80:

Can anybody tell if this is fixed in libjpeg?

No, unfixed in libjpeg. A way to reproduce the problem would be a first step

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.