[PATCH] ogr2ogr incorrectly populates fields for MapInfo Files

[zoltan]

It seems like ogr is incorrectly populating NULL fields for MapInfo files. xxx.mid and xxx.mif are provided. run: ogr2ogr -f "MapInfo File" -a_srs WGS84 yyy.tab xxx.mif then ogr2ogr -f "MapInfo File" -a_srs WGS84 zzz.mif yyy.tab

Now compare the two MID files

Floats: Where NULL is in the original MID, it now becomes a zero. I know that binary zero is NULL and also numeric zero, but is there not a way to keep the field empty rather than force it to zero? This is important because the datagoes back to Oracle via GeoMedia, and numeric zero is different to NULL (no value).

Integers As for float - can NULL not be kept as NULL instead of numeric zero?

Date fields. These are totally wrong. The first DATE field of NULL, correctly remains empty. Once a DATE field has been populated, notice that this value is now inserted into empty DATE fields further down the record.

Char I think I (oracle) can live with the empty quotes.....not sure - will have to wait for our clints to complain about that one.

[Even Rouault]

Attached a simple patch that fixes the issue. Please have a look at the "FIXME?" section in it.

[zoltan]

Sorry - I'm feeling really lame here - but I haven't applied patches before. Do I just download the latest src code (​http://download.osgeo.org/gdal/gdal-1.8.1RC2.tar.gz) and apply (amend the code) the patch & rebuild? I'm using Ubuntu 10.04 and gdal was installed from their repositories.

Just need some guidance that I am going about thi fix the correct way....

Thanks (especially for the quick turnaround time), Zoltan

[Even Rouault]

Actually, my comment was directed for the review of the MITAB mainteners I've CC'ed and who are in position to assess the correctness of the patch prior its inclusion to GDAL.

zoltan, this patch is against the SVN trunk version of GDAL, but I believe that MITAB hasn't changed much between 1.8.1 and latest trunk, so you can probably apply the patch to 1.8.1.

[zoltan]

Hi Even - I have added the changes to mitab_feature_mif.cpp and successfully recompiled. The results are now correct for DATE fields (they do not propagate to other DATE fields that were NULL), but the NULL text fields still have "" and the numeric NULLs still have 'zero'. The question now is, are these zeroes propagating the NULL fields in the TAB file or has the TAB file got the correct NULL values (due to your patch), and must we also apply this patch somewhere else in the code that handles TAB towards MIF formats? I'll be browsing the code, but if you have a quick answer, that would be great. It's 19h00 here and dinner is calling ..... :-) Thanks so far, Z

[Even Rouault]

Ah, I did read your initial message too fast. I only looked at mid reading/writing, not tab...

[zoltan]

Yep - me too, now that I've read the sourcecode more slowly :-). You are now reading and writing the MID files correctly and it's the TAB that needs looking at too. If you have the time to suggest some patches I will try them first thing tomorrow morning. Our clients are visiting us tomorrow and it would be sooo nice to rerun the Geomedia conversion with this patch (thereby replacing the dataset I delivered to them yesterday.) If you have other priorities, so be it - but mega-thanks so far. Regards, Z

[Even Rouault]

Hum, I tried a similar fix in the DAT writer ( if (IsFieldSet(iField)) continue in TABFeature::WriteRecordToDATFile(), but it results in corrupted file. Disclaimer : if have no knowledge of the spec of the .dat format and I'm not familiar with that code eithe, so it was just a blind guess, but my intuition is that due to the binary nature of the .dat format, which I suppose packs fields without separators, the concept of null values is likely inexistant in that format. So you should probably use another format if it is a critical need for you.

You could perhaps get better help by writing at ​http://groups.yahoo.com/group/mitab/

[Daniel Morissette]

Just a quick note to let you know that you are not being ignored, all your comments and progress make sense so far. I just don't have time to look into this now. If you can provide a sample TAB dataset generated by MapInfo that contains NULL field values for each type then we can have a look at the way they are stored and implement it in the MITAB driver.

[zoltan]

Hi, The TAB/MAP/ID/DAT files were converted with MapInfo 7.5 Table|Import command by Luca from the gdal-dev lists. Thank you, Luca. Regards, Zoltan

[lpercich]

Hi all,

the real problem is that MapInfo does not support null values (as of version 10.5):

• Date fields can be null just because they are basically stored as strings, "null dates" are zero-lenght date strings;
• Empty strings are undistinguishable from null strings
• There's no way to have null numbers, they are always converted to zeroes.

If you try to link an ORACLE or PostGIS table from MapInfo, you will get all the null numbers converted to zeroes, and the null strings to "". Quite annoying.

In my company we're (still) using MapInfo 7 in conjunction with PostGIS, without relying on the direct PG support (which is supported since version 9). We developed tools for automatically converting MapInfo special values ("_", -9999) into PostGIS nulls and back before/after the automated imnport/export procedures.

In the end I would suggest GDAL developers to implement the following MapInfo format-specific options, here provided with example values:

• NULL_FLOAT=-9999
• NULL_STRING="_"
• NULL_INTEGER=-9999

and let OGR apply this mapping both on reading and writing MapInfo layers.

All the best!

Sig

[zoltan]

Hi, OK - I've had two offers for creating a TAB file. Thanks to both of you. The one created by Autocad cicvil3d 2011 stripped all the attributes and then created a single fictitious one. The one by MapInfo 7.5 I have uploaded as test data, but I think the NULL fields have already been populated with zeroes, because when I convert that to MID/MIF, using my patched version (see uploaded patch file), the zeroes appear.

So, to Luca's comment above: I am only using MIF to TAB as a vehicle for converting data from my Linux based Genamap GIS package. The MID/MIF files I write are ASCII and have proper 'empty' fields - see xxx.mid. Unfortunately GeoMedia does NOT read MID/MIF files so I need a vehicle to convert (any, actually) ASCII to TAB (or some other form GeoMedia can open). So as MapInfo is nowhere near the production chain, it's current version is not relevant. Luca did use v7.5 (ie: below v10) to create the TAB file for us and yet the patched ogr2ogr still writes zero-filled NULLs, suggesting that Luca's TAB file already has the zeroes. Drat. To take this further, I would guess the next step would be to look inside the TAB file with a debugger or something. That's beyond me at this stage. Furthermore, the idea of using -9999 to signal NULLs will not work, as then GeoMedia (and other vendors) will have to conform to this standard too.

[Even Rouault]

Zoltan, you don't need to do a conversation to examine how OGR reads a file. Look at the output of "ogrinfo -al yourfile.tab" . I've tried this with the attached xxx.TAB and I can see 0's for integer/real values. So as suspected by Lucas, it is likely that the tab format doesn't support the null value concept. Or there would be values in the header that would indicate that some values should be considered as null ? hum, I just don't know.

I agree that using special values on OGR side will not improve interoperability with other software. I don't know GeoMedia, but doesn't it support shapefiles ? The .DBF format should have better support for null values, at least it works on OGR side.

[Even Rouault]

I meant "conversion" of course ;-)

[Jukka Rahkonen]

Heads-up needed. Is there progress going on on the mitab side?

[Even Rouault]

All those tickets have more than one year and nobody has acted on it, so the priority is not so high

[Even Rouault]

Removing blocker criticality

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.