postgis extension no-raster version

[robe]

UPDATE - I propose keeping the extension name: still postgis and not zeroing out raster, but encoding the lack of raster support in the version rather than the extension name. This approach takes a lot less changes and has the benefit of painless backward compatibility compared to other options proposed.

---

Was

talking on IRC with astrodog and it seems they are having a hard time

getting raster in the FreeBSD packages because of all the dependencies GDAL brings in which a user may not have.

The easiest option which he proposed which I think we can swing is to still have a PostGIS extension that zeros out raster if not available. Just returns a not supported note similar to what we do with ST_GeoJSON and several GEOS functions if your GEOS is not new enough.

Here is bit of IRC dialog for context

23:39] <astrodog> We build gdal as another package, with pretty 
broad support for formats.
[23:39] <astrodog> That way things just work for people.
[23:39] <robe2> and disable everything except some key ones like 
sqlite, odbc, curl
[23:40] <robe2> I even disable postgres
[23:40] <astrodog> But PostGIS brings in GDAL, which brings in a 
pile of other things.
[23:40] <astrodog> None of it is bad if you build from source... 
it's the binary packages that cause problems.
[23:41] <robe2> one more reason not to have a shared liblwgeom :)
[23:41] <robe2> astrodog: so you can't build an isolated gdal sits
 in postgres bin?
[23:42] <robe2> on windows always searches in postgres bin first
[23:42] <robe2> so I can override system dlls by replacing with my
 own version
[23:42] <robe2> not sure how it works on unix/linux
[23:43] <astrodog> robe: If you guys bundle it, I can. If you 
don't, I'm expected to use whatever the gdal port does, that way users 
can set whatever gdal options they want.
[23:43] <robe2> so if we included gdal source as part of our dist 
package?
[23:45] <robe2> pl/v8 (not sure if they still do have to check) 
included v8 source zip as part of theres
[23:45] <astrodog> Sure. Then it's just some thing you guys 
included... and any issues it creates would be sent up to PostGIS trac. 
*grin*
[23:45] <robe2> cause the v8 upstream was kinda flaky (I mean you 
hit a bad version and you are screwed)
[23:48] <robe2> astrodog: so if you have gdal already is the issue
 because depending on user installing, they might have a different gdal 
with not all the dependencies available
[23:49] <robe2> I got the impression from listening to 
devrimgunduz that was his pain
[23:49] <robe2> and why EL 5, CentOS 5 he was very apologetic for
[23:49] <astrodog> Yeah.
[23:49] <astrodog> For us... we install GDAL with a huge pile of 
things, so that the various bits that depend on it can all use one 
version.
[23:50] <astrodog> But it means requiring GDAL is a fairly large 
requirement.
[23:50] <robe2> I wonder how hard it would be to get GDAL to do 
run-time load like they do with proj already
[23:50] <robe2> that seems like the fundamental issue
[23:51] <robe2> astrodog: do you built spatialite and rasterlite
[23:51] <robe2> as I recall rasterlite relies on gdal I think
[23:51] <robe2> so would have similar issue
[23:52] <astrodog> Yeah, it does.
[23:52] <robe2> so you have same issue with it?
[23:53] <astrodog> Yeah, but with rasterlite, people know they're 
in for it.
[23:53] <robe2> one thing I always wanted was for people to be 
able to swap out the gdal I have with a beefier one
[23:53] <astrodog> The issue we run into with PostGIS is that the 
SQL method isn't as well documented... so we get PRs saying "Loading the
 extension doesn't work!"
[23:54] <robe2> well we could document that better I guess
[23:54] <robe2> I didn't like it cause it required too much 
explaining
[23:54] <astrodog> It's not so much you guys, as every tutorial on
 the planet skips over that option. :P
[23:54] <robe2> like find where your files are stored -- and I 
lost 90% of the audience at that point
[23:54] <astrodog> Be nice if the extension had a way to tell if 
it had raster support or not, and act accordingly.
[23:55] <robe2> and had to digress into if you are on this - its' 
here , if you are on that it's here etc
[23:55] <astrodog> *nod*
[23:56] <robe2> yah but then it wouldn't be the same extension
[23:56] <robe2> we could create a postgis_no_raster extension
[23:56] <astrodog> Could the extension stub out the raster bits if
 they aren't there?
[23:57] <astrodog> So the function exists, it just always returns 
not supported.
[23:57] <astrodog> :P
[23:57] <robe2> but the problem is the extension model doesn't 
have a way to say something like requires: postgis or postgis_no_raster;
[23:57] <robe2> So you'd have issues with like you can't install 
postgis_sfcgal, pgrouting etc.
[23:57] <robe2> astrodog: ah didn't think about that
[23:57] <robe2> yah it could
[23:57] <robe2> that's a great idea
[23:58] <robe2> we do that already for some functons like json and
 geos if you are running a lower geos
[23:58] <robe2> I'll ticket it

[strk]

I thought we had a ticket for this but could not find one. My idea was that we should have these extensions:

• postgis_core
• postgis_raster depends on postgis_core
• postgis_topology depends on postgis_core
• postgis_sfcgal depends on postgis_core
• (postgis depends on postgis_core and postgis_raster) for backward compatibility

How to get there is still to be found out, especially when it comes to upgrading an existing system.

[strk]

Robe: I shall note these are the kind of problems you have when insisting at every everything strictly required (aka statically linked) —- you associate a name with a full bunch of options, but then are stuck with all of them…

[robe]

It's too late to break those things out. It would screw downstream projects.

So you'd have to call it postgis (not postgis_core). But even that would be a mess, not to mention the retraining.

astrodog's solution is still the least invasive.

Funny thing is the only reason this is an issue is people prefer installing postgis the Extension route instead of using sql scripts. Go figure.

[strk]

Calling "postgis" an extension withouth raster would break packages expecting it to contain also raster.

If you see, in my proposal the "postgis" extension still has raster in it, as it's just a control file that mentions dependency on "postgis_core" and "postgis_raster", so for new installs it'd would not break anything but add the possibility to just use the "postgis_core" extension.

The difficult part would then be figuring out how to have an upgrade script go from the full-contained "postgis" to the shell "postgis" version as I suspect changing dependencies of an extension is not currently supported by the PostgreSQL extension model (should be checked).

[gdt]

It would be completely unreasoable to include gdal sources. This results in duplicated code and also means that postgis needs a security release immediately when there is an issue in gdal.

I think the real problem here is that gdal should be set up to build core and modules so that it can be separate packages, and postgis can depend only one the core part.

Also, how big does gdal end up being, compared to postgresql itself? The gdal-lib pkgsrc package on NetBSD 6 i386 is 94M including things it depends on. postgresql seems more like 25 MB. Still, that leaves postgis and everything it depends on at 164 MB.

Is the FreeBSD gdal package much bigger? (pkgsrc's does not yet have python bindings, which clearly should be split.)

[robe]

gdt I was expecting it would be . The issue is from what I understand from EvenR breaking up gdal as you describe would be a monumental undertaking. So back to astrodog's 0 out raster if it's a big deal to build with it.

strk I see no benefit to breaking out raster from postgis if we can do what astrodog proposes easily and since raster functions are in a separate file anyway, it doesn't seem that much hassle. I could probably do it myself.

[strk]

If astrodog wipes out raster from postgis extension, then "CREATE EXTENSION POSTGIS" would give you postgis and no raster, how would you then enable raster in a database ? There'd be no "CREATE EXTENSION" for you to use, unless astrodog goes on with doing more work, and those packages would still be incompatible with a manually-built postgis.

Implementing shells for the sake of printing UNSUPPORTED may be a solution, yes, as long as there's then a way to add raster if needed.

[robe]

ALTER EXTENSION UPDATE TO VERION "2.3.0next" would convert a non-raster postgis to a postgis one if the new one was compiled with raster. In fact it would be even easier if they started with no raster in version 2.3.0 and put raster in 2.3.1.

The issue I have with postgis_core/postgis_raster is

1) two steps for everybody (I know a lot of people don't want raster so not a big deal) 2) to do create extension postgis (until 9.6 comes along) would be 3 step process (yah I know for legacy) 3) We'd have to tell folks like pgrouting to change their required to postgis_core (and that isn't backward compatible for them) so that sucks. 4) Everyone is used to doing CREATE EXTENSION postgis; to get everything

So the only other least painful alternative would to have our postgis upgrade script drop raster functions from the extension.

Then they do

CREATE EXTENSION postgis_raster from unpackaged;

and we continue life from there telling people they need to install postgis_raster in future.

That does have the benefit that if a package DOES depend on postgis_raster, then they can state that and wouldn't be screwed by a postgis extension SANS raster.

[robe]

I just thought of a really stupid or super brilliant idea to resolve this issue.

The main reason we can't have 2 extensions called postgis with the same version number (1 having raster and 1 not) is because then it can't be easily upgraded.

However there is no rule against having two postgis extensions as long as they have different versions. Since versions don't have to be all numbers, we can encode the lack of raster right in the version.

Sooo.

version = 2.3.0 version = 2.3.0-no-raster

I propose if people compile with raster support, they get old behavior, without raster support they get this new wonder half-child 2.3.0-no-raster version.

There are 3 benefits of doing it this way.

1. Most extensions that rely on PostGIS just use the vector portion, so keeping the name the same they can continue on without having to change their requires.

2. We have the same set of instructions for everybody:

CREATE EXTENSION postgis;

3. With the version differentiator it's trivial to upgrade a version 2.3.0-no-raster to a full yummy 2.3.0 version. We just add upgrade scripts to go from 2.3.0-no-raster to 2.3.0 regular postgis

4. Similarly we can have a 2.3.0—2.3.0-no-raster that gets installed if you compile without raster support. Which will drop all the raster functions during the upgrade.

What do folks think about this? Am I nuts or is this a great idea?

[strk]

I'm concerned about the explosion of upgrade paths that would derive from such move, and would be wondering why then we should not have all configurations in versions.

CREATE EXTENSION postgis VERSION geos-3.8,nosfcgal,noraster,withmilk,inglass

[robe]

There wouldn't be much. Because

1) it would only apply for raster, since that is already part of postgis (sfcgal is already a separate extension and so is everything else), geos I propose we make sure whatever version has all the functions we need is the minimum (you guys always fight me on this point which is one of the reason I'm very against liblwgeom being split off, cause then the issue is even worse than GEOS) and I already do enough fighting every minor release for GEOS.

2) There will only be one set of postgis version extensions based on what you compile

e.g. you either get the postgis with raster or the version without raster

and I propose we only allow one version upgrade/downgrade path

so postgis with raster for version 2.3.0 will only have a postgis-2.3.0-no-raster—postgis-2.3.0.sql

not all the others that applied

I think this is fine because I think it's rare people switch from a no-raster to a raster and vice versa.

The more common case would be folks who had to install the scripts without support of extensions and those people would then just have to do

    -- upgrade the old way with scripts
    CREATE EXTENSION postgis FROM unpackaged VERSION "2.3.0-no-raster";

and then from then on its

ALTER EXTENSION postgis UPDATE;

like everyone else.

[robe]

astrodog,

Unfortunately I've gotten a lot of push back on this and none of the people who have complained in the past to me about this have voiced an opinion. So I'm afraid I'm just going to have to close this out unless someone speaks out about it.

​https://lists.osgeo.org/pipermail/postgis-devel/2015-October/025427.html

If you are interested in this, please post a comment on postgis-devel and then I'll consider reopening it for discussion.