Setup a Bot to run OSS FUZZ tests

[robe]

strk,

I've created a separate ticket to setup OSS Fuzz testing on our own bots. While we are at it, maybe we can figure out Even's issue with having the make run from main folder so that we can eventually test more than liblwgeom.

and address kcc's comment here:

​https://github.com/google/oss-fuzz/pull/709

 projects/postgis/build.sh
+make -j$(nproc) -s
+cd ..
+
+./fuzzers/build_google_oss_fuzzers.sh
@kcc
kcc 5 hours ago Contributor

This should work as is, but it's preferable to have the fuzz targets built as part of the regular developer build (make).
Otherwise the risk is that the fuzz targets will often bit rot and we will see failures only on oss-fuzz

I think you have more mojo to handle this than I do so I'm assigning this to you

[robe]

I was able to get this to run on Debbie with following command (I was running under root, I know a no no, but I couldn't access docker from postgis-git-sync account I had created earlier.

Basic steps

git clone --depth=1 https://github.com/google/oss-fuzz.git oss-fuzz
cd oss-fuzz
git pull
cd projects/postgis
docker build --no-cache -t postgis-fuzz .

docker run postgis-fuzz

I'm not sure how to see fuzz errors though. Didn't get that far in my learning.

Output shows:

libtool: link: clang -shared  -fPIC -DPIC  .libs/stringbuffer.o .libs/bytebuffer.o .libs/measures.o .libs/measures3d.o .libs/box2d.o .libs/ptarray.o .libs/lwgeom_api.o .libs/lwgeom.o .libs/lwpoint.o .libs/lwline.o .libs/lwpoly.o .libs/lwtriangle.o .libs/lwmpoint.o .libs/lwmline.o .libs/lwmpoly.o .libs/lwboundingcirc
arreferencing.o .libs/lwprint.o .libs/g_box.o .libs/g_serialized.o .libs/g_util.o .libs/lwgeodetic.o .libs/lwgeodetic_tree.o .libs/lwtree.o .libs/lwout_gml.o .libs/lwout_kml.o .libs/lwout_geojson.o .libs/lwout_svg.o .libs/lwout_x3d.o .libs/lwout_encoded_polyline.o .libs/lwgeom_debug.o .libs/lwgeom_geos.o .libs/lwgeo
libtool: link: (cd ".libs" && rm -f "liblwgeom-2.4.so.0" && ln -s "liblwgeom-2.4.so.0.0.0" "liblwgeom-2.4.so.0")
libtool: link: (cd ".libs" && rm -f "liblwgeom.so" && ln -s "liblwgeom-2.4.so.0.0.0" "liblwgeom.so")
libtool: link: ar cr .libs/liblwgeom.a  stringbuffer.o bytebuffer.o measures.o measures3d.o box2d.o ptarray.o lwgeom_api.o lwgeom.o lwpoint.o lwline.o lwpoly.o lwtriangle.o lwmpoint.o lwmline.o lwmpoly.o lwboundingcircle.o lwcollection.o lwcircstring.o lwcompound.o lwcurvepoly.o lwmcurve.o lwmsurface.o lwpsurface.o
luster.o lwgeom_geos_node.o lwgeom_geos_split.o lwgeom_topo.o lwgeom_transform.o lwgeom_wrapx.o lwunionfind.o effectivearea.o lwkmeans.o kmeans.o varint.o lwspheroid.o
libtool: link: ranlib .libs/liblwgeom.a
libtool: link: ( cd ".libs" && rm -f "liblwgeom.la" && ln -s "../liblwgeom.la" "liblwgeom.la" )
+ cd ..
+ bash ./fuzzers/build_google_oss_fuzzers.sh
Building fuzzer wkb_import_fuzzer
Building fuzzer wkt_import_fuzzer
+ bash ./fuzzers/build_seed_corpus.sh

I was hoping I could do this on my windows 7 docker install, but sadly I got stuck a little stuck:

I got as far as building the docker image, but when I go to run, the output looks like this: I had to dos2unix the files cause they came down with windows line breaks, but after that got stuck.

SECURITY WARNING: You are building a Docker image from Windows against a non-Win
dows Docker host. All files and directories added to build context will have '-r
wxr-xr-x' permissions. It is recommended to double check and reset permissions f
or sensitive files and directories.
dos2unix: converting file project.yaml to Unix format...
---------------------------------------------------------------
Compiling libFuzzer to /usr/lib/libFuzzingEngine.a ...ar: creating /usr/lib/libF
uzzingEngine.a
 done.
CC=clang
CXX=clang++
CFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAF
E_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitiz
e-coverage=trace-pc-guard,trace-cmp
CXXFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNS
AFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanit
ize-coverage=trace-pc-guard,trace-cmp -stdlib=libc++
---------------------------------------------------------------
+ ./autogen.sh
* Running /usr/bin/libtoolize (2.4.6)
   OPTIONS = --force --copy --install
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './config.guess'
libtoolize: copying file './config.sub'
libtoolize: copying file './install-sh'
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'macros'.
libtoolize: copying file 'macros/libtool.m4'
libtoolize: copying file 'macros/ltoptions.m4'
libtoolize: copying file 'macros/ltsugar.m4'
libtoolize: copying file 'macros/ltversion.m4'
libtoolize: copying file 'macros/lt~obsolete.m4'
libtoolize: Consider adding '-I macros' to ACLOCAL_AMFLAGS in Makefile.am.
* Running /usr/bin/aclocal (1.15)
* Running /usr/bin/autoconf (2.69)
======================================
Now you are ready to run './configure'
======================================
+ ./configure --enable-static --without-raster --without-protobuf
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to print strings... printf
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in `/src/postgis':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

[robe]

duh should have read the instructions. I got it running on my windows box by changing my command to:

docker run --cap-add SYS_PTRACE -ti postgis-fuzz

So now I get same output as debbie, but I'm lost with docker not sure how to see outputs.

[strk]

Outputs should be on stdout from docker run. Or I think you can add a --name good-name switch and then docker logs good-name to see the output.

Not passing --name assigns a random (but mnemonic) name, you can find out the names of runnign dockers via docker ps

[robe]

ah thanks strk I was wondering how to name it what I wanted.

Unfortunately docker logs just shows me what I already so in interactive mode.

[pramsey]

This going to happen? Give yourself a break, close the tasks you aren't going to get to, it's alright.

[robe]

Milestone renamed