sqlite + spatialite build failure on mac osx

[Kurt Schwehr]

This issue did not exist with gdal 1.9.2.

If I build using:

--with-sqlite3=%p \ --with-spatialite=%p \

I get this error:

-install_name /sw/lib/libgdal.1.dylib -compatibility_version 19 -current_version 19.0 -Wl,-single_module Undefined symbols for architecture x86_64:

"_sqlite3_column_table_name", referenced from:

OGRSQLiteSelectLayer::OGRSQLiteSelectLayer(OGRSQLiteDataSource*, CPLString, sqlite3_stmt*, int, int)in ogrsqliteselectlayer.o

ld: symbol(s) not found for architecture x86_64 collect2: ld returned 1 exit status make[1]: * [libgdal.la] Error 1

I fixed this in the fink package by doing a "SetLDFLAGS = -lsqlite3", which forces all linking to include linking against sqlite3. Overly aggressive, but it gets a working package.

See Also: ​http://trac.macports.org/ticket/39214

Seen on Mac OSX 10.8 with gdal 1.10.0 and svn checkout

complete configure line:

./configure --prefix=/tmp/gdal --mandir='${prefix}/share/man' --with-local=/sw --with-libz=/sw --with-liblzma=yes --with-cfitsio=/sw --with-netcdf=/sw --with-png=/sw --with-libtiff=/sw --with-geotiff=internal --with-jpeg=/sw --with-gif=/sw --with-ogdi=/sw --with-jasper=yes --with-bsb --with-ogr --with-xerces=yes --with-xerces-inc=/sw/include --with-xerces-lib='-L/sw/lib -lxerces-c -lpthread' --with-hdf5=/sw --with-geos=/sw/opt/libgeos3.3.6/bin/geos-config --with-static-proj4=/sw --without-pg --libdir=/sw/lib --includedir=/sw/include/gdal1 --with-spatialite=/sw --with-odbc=/sw --with-grass=no --with-pcraster=no --with-hdf4=no --with-oci=no --with-fme=no --with-ecw=no --with-kakadu=no --with-mrsid=no --with-dods-root=no --without-php --without-python --without-perl --without-ruby

With my hack to get it working, here are the shared libs for the key libraries:

otool -L /sw/lib/libgdal.dylib /sw/lib/libgdal.dylib:

/sw/lib/libgdal.1.dylib (compatibility version 19.0.0, current version 19.0.0) /sw/lib/libsqlite3.0.dylib (compatibility version 9.0.0, current version 9.6.0) /sw/lib/libproj.0.dylib (compatibility version 7.0.0, current version 7.6.0) /sw/opt/libgeos3.3.6/lib/libgeos_c.1.dylib (compatibility version 9.0.0, current version 9.6.0) /sw/lib/libodbc.1.dylib (compatibility version 2.0.0, current version 2.0.0) /sw/lib/libodbcinst.1.dylib (compatibility version 2.0.0, current version 2.0.0) /sw/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0) /sw/lib/libxerces-c-3.1.dylib (compatibility version 3.1.0, current version 3.1.0) /sw/lib/libjasper.1.dylib (compatibility version 2.0.0, current version 2.0.0) /sw/lib/libnetcdf.7.dylib (compatibility version 10.0.0, current version 10.0.0) /sw/lib/libhdf5.8.dylib (compatibility version 9.0.0, current version 9.0.0) /sw/lib/libogdi31.dylib (compatibility version 3.1.0, current version 3.1.5) /sw/lib/libgif.4.dylib (compatibility version 6.0.0, current version 6.6.0) /sw/lib/libjpeg.9.dylib (compatibility version 10.0.0, current version 10.0.0) /sw/lib/libtiff.5.dylib (compatibility version 8.0.0, current version 8.0.0) /sw/lib/libpng16.16.dylib (compatibility version 19.0.0, current version 19.0.0) /sw/lib/liblzma.5.dylib (compatibility version 6.0.0, current version 6.4.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) /sw/lib/libspatialite.1.1.3.dylib (compatibility version 3.0.0, current version 3.3.0) /sw/lib/libpcre1/libpcre.1.dylib (compatibility version 4.0.0, current version 4.1.0) /sw/lib/libcurl.4.dylib (compatibility version 8.0.0, current version 8.0.0) /sw/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.1.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5) /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

otool -L /sw/lib/libspatialite.5.dylib /sw/lib/libspatialite.5.dylib:

/sw/lib/libspatialite.5.dylib (compatibility version 7.0.0, current version 7.0.0) /sw/lib/libcharset.1.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0) /sw/lib/libfreexl.1.dylib (compatibility version 2.0.0, current version 2.0.0) /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0) /sw/lib/libproj.0.dylib (compatibility version 7.0.0, current version 7.6.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5) /sw/lib/libsqlite3.0.dylib (compatibility version 9.0.0, current version 9.6.0) /sw/opt/libgeos3.3.6/lib/libgeos_c.1.dylib (compatibility version 9.0.0, current version 9.6.0)

ran autogen.sh with autoconf (GNU Autoconf) 2.69 and still have the issue.

[dmacks]

Replying to goatbar:

This issue did not exist with gdal 1.9.2.

If I build using:

--with-sqlite3=%p \ --with-spatialite=%p \

I get this error:

Undefined symbols for architecture x86_64:

"_sqlite3_column_table_name", referenced from:

OGRSQLiteSelectLayer::OGRSQLiteSelectLayer(OGRSQLiteDataSource*, CPLString, sqlite3_stmt*, int, int)in ogrsqliteselectlayer.o

ld: symbol(s) not found for architecture x86_64 collect2: ld returned 1 exit status make[1]: * [libgdal.la] Error 1

This configure.in logic

dnl Check for SQLite (only if SpatiaLite is not detected) if test "${HAVE_SPATIALITE}" = "no" ; then

SQLITE3_REQ_VERSION="3.0.0" AX_LIB_SQLITE3($SQLITE3_REQ_VERSION)

if test "$HAVE_SQLITE3" = "yes"; then

LIBS="$SQLITE3_LDFLAGS $LIBS"

fi

fi

appears to be faulty. Activating spatialite does not disable various codeblocks that directly call sqlite. Therefore gdal needs to link against sqlite itself. Presumably on some platforms spatialite is linked against sqlite so linking against one inherits the other or all the symbols become available together in the final binary? But neither of those assumptions appear safe. Simply removing that if...then... wrapper on my OS X 10.8 machine allowed the build to succeed using the same fink packaging (and not needing to pass LDFLAGS=-lsqlite to the whole config/build process).

[Even Rouault]

Do you know if your libspatialite.dylib is an "amalgamation" build, that is to say if it embeds directly sqlite in it (instead of linking with it) ? (and sometimes when it does, it renames the symbols)

Could you create the following test file (call it test_sqlite3_column_table_name.c)

void sqlite3_column_table_name(void);

int main(int argc, char* argv[])
{
    sqlite3_column_table_name();
    return 0;
}

and then try compiling it with (you may need adding -L/path/to/libspatialite.dylib) :

gcc test_sqlite3_column_table_name.c -o test_sqlite3_column_table_name -lspatialite

And same test with test_sqlite3_open.c:

void sqlite3_open(void);

int main(int argc, char* argv[])
{
    sqlite3_open();
    return 0;
}

gcc test_sqlite3_open.c -o test_sqlite3_open -lspatialite

And if the nm utility exists on OS X, could you report the output of the following commands :

nm -D /path/to/libsqlite3.dylib  |grep open
nm -D /path/to/libspatialite.dylib  |grep open
nm -D /path/to/libsqlite3.dylib  |grep column_table_name
nm -D /path/to/libspatialite.dylib  |grep column_table_name

(if "nm -D" doesn't work, perhaps "objdump -T" will)

[dmacks]

I have spatialite v4.1.0. I used

#include <stdlib.h>
#include "spatialite.h"

and the appropriate -I and -L flags (according to spatialite.pc) for the two test programs. In each case, the build failed with a mile of errors related to spatialite.h (apparently it uses types defined in some other .h but doesn't #include it itself). For example:

In file included from /sw/include/spatialite/gaiageo.h:69,
                 from /sw/include/spatialite.h:73,
                 from test_sqlite3_open.c:2:
/sw/include/spatialite/gg_structs.h:296: error: expected specifier-qualifier-list before 'sqlite3_int64'

We do have nm, but its flags are different. I assume you want to see what symbols are in the public interface, so...

$ nm -gm /sw/lib/libsqlite3.dylib | grep open
                 (undefined) external _dlopen (from libSystem)
                 (undefined) external _open (from libSystem)
0000000000009d35 (__TEXT,__text) external _sqlite3_blob_open
000000000000b0bc (__TEXT,__text) external _sqlite3_blob_reopen
000000000000f28a (__TEXT,__text) external _sqlite3_open
000000000000fbbf (__TEXT,__text) external _sqlite3_open16
000000000000fbb5 (__TEXT,__text) external _sqlite3_open_v2

$ nm -gm /sw/lib/libspatialite.dylib | grep open
                 (undefined) external _fopen (from libSystem)
                 (undefined) external _freexl_open (from libfreexl)
                 (undefined) external _libiconv_open (from libiconv)

$ nm -gm /sw/lib/libsqlite3.dylib | grep column_table_name
00000000000094e6 (__TEXT,__text) external _sqlite3_column_table_name
00000000000094fc (__TEXT,__text) external _sqlite3_column_table_name16

$ nm -gm /sw/lib/libspatialite.dylib | grep column_table_name 
[no results]

I don't know what an "amalgamation" build is.

Looking at spatialite.pc, which is where it publishes what flags it thinks other things should use to compile/link against it, I see "-lsqlite3". Seems like a good first approach would be to just ask spatialite itself what to use before writing your own autoconf tests. You can then (as now) not explicitly test/activate sqlite3 if spatialite is requested, but you do get sqlite3 implicitly if spatialite thinks it's needed.

[Even Rouault]

What strikes me is why this error does appear only with sqlite3_column_table_name which is the new sqlite3 symbol used in GDAL 1.10, and not the other sqlite3 symbols that are also used by the OGR SQLITE driver in GDAL < 1.10.

[dmacks]

It's not the only one, just a part of the message (one example, and a truncated fragment of the compiler call). Attached file gdal-build-log.txt is the full linker call and result for me (build is driven by fink).

[Even Rouault]

ok, that's more logical. And just to be sure, could you try building GDAL 1.9.2 against the same spatialite version ? I'd expect that you get similar linking errors. I've diff'ed configure.in between GDAL 1.10 and GDAL 1.9.X and nothing related to linking to sqlite3 has changed between both, so I would expect failures too with 1.9.X. My hypothesis would be that the new spatialite version v4.1 has different linking info than the previously used version.

[Even Rouault]

Is it still an issue ?

[Even Rouault]

Closing assuming this is solved

[dmacks]

I just tried version 1.11.1 against what as best I can tell is the same major version (2.3.x) of spatialite that we were using ~20 months ago. Without LDFLAGS=-lsqlite3, final linking of libgdal.la fails:

Undefined symbols for architecture x86_64:
  "_sqlite3_column_table_name", referenced from:
      _OGRSQLITE_static_routines in ogrsqliteapiroutines.o
      OGRSQLiteSelectLayer::OGRSQLiteSelectLayer(OGRSQLiteDataSource*, CPLString, sqlite3_stmt*, int, int, int) in ogrsqliteselectlayer.o

Switching to a newer spatialite (4.1.1) compiles successfilly without needing that flag to be manually added. Because the new spatialite's spatialite.pc publishes that flag via its Libs: field whereas the old one's .pc does not.

So it's not "fixed" in that gdal still needs that flag itself, it just isn't visibly broken for me for now because some dependency happens to propagate it for now.

[Even Rouault]

If it works with recent spatialite, then I'd close it again as it seems more an issue with spatialite 2.3.1, which is clearly ancient.