mapcache windows build

[assefa]

mapcache windows build

[assefa]

.c files modification to build on windows.

[assefa]

Thomas,

Is it possible to apply the patch and tell me if things still work. I can then commit. The patch is only the first pass to bring the code to build on windows.

[tbonfort]

Assefa, I have applied the patch and all seems correct on my osx build.

I skimmed through the patch, can you confirm that it only moves the variable declarations to the beginning of their code block?

[assefa]

majority of the change is that. but I also added a bit of things using ifdefs (#ifndef _WIN32) for things that did not build properly on windows (ex buffer.c, ezxml.c, http.c, imageio_png.c) look for any_WIN32 to see them. I think it should be ok but can you give it a quick look at least for buffer.c

[tbonfort]

Assefa, seems good to me, thanks for fixing those, I'll try to be more careful in the future now you windows guys are poking around in the code :)

some of the _WIN32 ifdefs also be applied to the unix builds, but we can do that in a next step.

[assefa]

committed in r 12835. I will start committing the make files. It won't be complete though.

PS: I am an ubuntu guy since FOSS4G :)

[rouault]

I humbly suggest to turn (trunk/mapserver/mapcache/src/buffer.c) :

#ifndef _WIN32 
    memcpy(buffer->buf + buffer->size, data, len); 
#else 
    memcpy(((unsigned char*)buffer->buf) + buffer->size, data, len); 
#endif

into :

memcpy(((unsigned char*)buffer->buf) + buffer->size, data, len);

(or just (char*))

which should work on all platforms. void* pointer arithmetic compiled with -pedantic raises a warning.

[assefa]

I have done the change in buffer.c according to comment. r12838

windows make files committed:

• build the fcgi
• seed executable does not yet build: missing time related functions
• did not try yet the Apache module

Note for Jeff:

• I used : ​http://apache.sunsite.ualberta.ca//apr/binaries/win32/
• ​http://apr.apache.org/download.cgi to build the modules locally. I started with that but the 2nd module did not build out of the box. I did not make any efforts on building it.
• You can modify/fix ad you see fit

[tbonfort]

Assefa,

what are the time functions that are failing? All time handling is supposed to be done through apr which should take care of platform idiosyncrasies.

[assefa]

It is not that they are failing, It is just that they are not available on windows:

• strptime
• gettimeofday

We need to define them locally as it is done for MapServer-cgi.

The other link problem I had was with:

mapcache_seed.obj : error LNK2019: unresolved external symbol __imp__apr_time_an
si_put@12 referenced in function _main

I am not sure why it was not available in the pre-build binaries I used. Probably need to build a local copy of the apr libraries.

[jmckenna]

I have compiled from source (apr-1.4.5, apr-iconv-1.2.1, apr-util-1.3.12) and I now only get one error when I execute 'nmake /f makefile.vc' for mapcache, but I am not sure which library this function/symbol comes from:

fastcgi_mapcache.obj : error LNK2019: unresolved external symbol __imp__apr_date_parse_http@4 referenced in function _fcgi_write_response
mapcache.exe : fatal error LNK1120: 1 unresolved externals

[assefa]

It should be in apr-util (source code I see apr-util-1.3.12\misc\apr_date.c(145))

[jmckenna]

ok thanks, I think I am missing libaprutil-1.lib and it might be causing this.

[jmckenna]

I think I made it past that. 2 new missing symbols:

mapcache_seed.obj : error LNK2019: unresolved external symbol _strptime referenced in fun
tion _main
mapcache_seed.obj : error LNK2019: unresolved external symbol _gettimeofday referenced in
function _main
mapcache_seed.exe : fatal error LNK1120: 2 unresolved externals

[aboudreault]

I talked with Assefa, and I doubt he'll get time to finish this task. I'm going to try take it. Question: how did you get it to compile ... I'm getting a lot of illegal declaration errors. (due to a declaration not at the beginning of a scope. Didn't find the c99 flag for VC 2010 and VC 2008.

[aboudreault]

nvm about the declaration errors... I think my mapcache directory was not up-to-date.

[jmckenna]

I'm not sure if you are talking to me directly - but I left this since there were 2 missing functions in the mapcache code, not part of MSVC/SDK (see comment:13) I believe there are already workarounds for these in the MS base code, and I talked to Assefa about this also. I have been meaning to dive back in and port those 2 functions over, but, haven't yet :)

Short story: I never did get it to compile.

[assefa]

I have ported locally the code from MS to mapcahe for the 2 functions but did not commit yet. I have not done the module building yet and not sure when I will get to this.

[aboudreault]

Assefa, Please attach a patch of you changes not committed, I would like to try building/porting the mapcache module.

[jmckenna]

same comment here :)

[assefa]

It is committed ( r12915 and r12916)

[aboudreault]

I've just committed some changes to be able to build mapcache under Windows in r12980. Everything builds now. I have been able to make a mod_mapcache.dll for OSGeo4w and it loads properly. Unfortunately I'm getting this error when trying to access a tile (which do not exist yet):

failed to open file C:/OSGeo4W/tmp/test/g/01/000/000/000/000/000/001.png: Unknown error

Not sure what this error means. Nothing is created in my cache base path.

[jmckenna]

I am able to compile mod_mapcache.dll, mapcache.exe, and mapcache_seed.exe but have not tested yet.

[aboudreault]

Jeff, if you test the mod_mapcache.dll... and it works... let me know. I'm still working on this... looks like something is different under windows and it's tricky to debug.

[jmckenna]

• committed changes to makefile.vc (r12982) for embedding the manifest files in the exes and dll
• haven't tested other stuff yet

[jmckenna]

More feedback:

• after configuring httpd.conf, I received an error:

  D:\ms4w>apache-restart.bat

    Syntax error on line 375 of D:/ms4w/Apache/conf/httpd.conf:
    cache disk: host system does not support file symbolic linking

• workaround I used was to comment <symlink_blank/> in mapcache.xml

     <cache name="disk" type="disk">
       <base>D:/ms4w/tmp/ms_tmp/cache</base>
       <!--<symlink_blank/>-->
     </cache>

• I cannot load mapcache_module as a Windows service for some unknown reason:

  The Apache MS4W Web Server service is starting.
  The Apache MS4W Web Server service could not be started.

  A service specific error occurred: 1.

  More help is available by typing NET HELPMSG 3547.

• I am able to verify that the module can be loaded:

 httpd -t -D DUMP_MODULES

    ...
    negotiation_module (shared
    setenvif_module (shared)
    mapcache_module (shared)

• if I manually start httpd.exe through the command window I can in fact load the module (????)

• if I do that, I goto ​http://127.0.0.1/mapcache/demo/tms and the following is recorded in Apache's error.log

[Fri Jan 13 16:36:54 2012] [error] [client 127.0.0.1] received wms request with no service param
[Fri Jan 13 16:37:07 2012] [error] [client 127.0.0.1] failed to open file D:/ms4w/tmp/ms_tmp/cache/test/WGS84/01/000/000/000/000/000/000.png: Unknown error, referer: http://127.0.0.1/mapcache/demo/tms
[Fri Jan 13 16:37:07 2012] [error] [client 127.0.0.1] failed to open file D:/ms4w/tmp/ms_tmp/cache/test/WGS84/01/000/000/001/000/000/000.png: Unknown error, referer: http://127.0.0.1/mapcache/demo/tms
[Fri Jan 13 16:37:07 2012] [error] [client 127.0.0.1] failed to open file D:/ms4w/tmp/ms_tmp/cache/test/WGS84/01/000/000/002/000/000/000.png: Unknown error, referer: http://127.0.0.1/mapcache/demo/tms

• for the record, I am using the default mapcache.xml file, pasted here:

    <?xml version="1.0" encoding="UTF-8"?>

<!-- see the accompanying mapcache.xml.sample for a fully commented configuration file -->

<mapcache>
   <cache name="disk" type="disk">
      <base>D:/ms4w/tmp/ms_tmp/cache</base>
      <!--<symlink_blank/>-->
   </cache>

   <source name="vmap0" type="wms">
      <getmap>
         <params>
            <FORMAT>image/png</FORMAT>
            <LAYERS>basic</LAYERS>
         </params>
      </getmap>
      
      <http>
         <url>http://vmap0.tiles.osgeo.org/wms/vmap0</url>
      </http>
   </source>
   
   <tileset name="test">
      <source>vmap0</source>
      <cache>disk</cache>
      <grid>WGS84</grid>
      <grid>g</grid>
      <format>PNG</format>
      <metatile>5 5</metatile>
      <metabuffer>10</metabuffer>
      <expires>3600</expires>
   </tileset>


   <default_format>JPEG</default_format>

   <service type="wms" enabled="true">
      <full_wms>assemble</full_wms>
      <resample_mode>bilinear</resample_mode>
      <format>JPEG</format>
      <maxsize>4096</maxsize>
   </service>
   <service type="wmts" enabled="true"/>
   <service type="tms" enabled="true"/>
   <service type="kml" enabled="true"/>
   <service type="gmaps" enabled="true"/>
   <service type="ve" enabled="true"/>
   <service type="demo" enabled="true"/>

   <errors>report</errors>
   <lock_dir>/tmp</lock_dir>

</mapcache>

[aboudreault]

The commit r12987 fixes the *Unknown error* issue.... but I'm still getting a crash in _mapcache_source_wms_render_map(). Unfortunately... it looks like debugging on Windows is just a mess. Cannot get a clear debug info where the module really crashes. In Apache, you should get a connection reseet by peer. Do you have the same thing on your machine Jeff?

[aboudreault]

the crash is always in apr_snprintf.c, at line 1341 with an access violation. The backtrace I have is inconsistent and not very clear.

[jmckenna]

I fixed my service issue (now I can load mod-mapcache as a windows service). phew what a battle.

will keep testing.

[jmckenna]

Alan: I think I am getting your same httpd.exe crash. From the Windows Event logs:

  <System>
    <Provider Name="Application Error" /> 
    <EventID Qualifiers="0">1000</EventID> 
    <Level>2</Level> 
    <Task>100</Task> 
    <Keywords>0x80000000000000</Keywords> 
    <TimeCreated SystemTime="2012-01-17T17:48:26.000Z" /> 
    <EventRecordID>81022</EventRecordID> 
    <Channel>Application</Channel> 
    <Computer>kordic</Computer> 
    <Security /> 
  </System>
 <EventData>
   <Data>httpd.exe</Data> 
   <Data>2.2.21.0</Data> 
   <Data>4e6a3015</Data> 
   <Data>libapr-1.dll</Data> 
   <Data>1.4.5.0</Data> 
   <Data>4e6a2fa6</Data> 
   <Data>c0000005</Data> 
   <Data>0001101c</Data> 
   <Data>1044</Data> 
   <Data>01ccd5403687a124</Data> 
   </EventData>
 </Event>

[aboudreault]

yep. same issue. Still unable to debug something. I don't know if it's my virtual machine that is broken somehow but I'm getting upset :| Let's see if Assefa can debug anything in it's VM.

[jmckenna]

my crash always on line#971 of apr_snprintf.c:

  line#971 s = va_arg(ap, char *);

specifically that "ap" value seems to be the problem:

  CXX0030: Error: expression cannot be evaluated	

[aboudreault]

same function than me...

[jmckenna]

Is the related call stack helpful?

	libapr-1.dll!apr_vformatter(int (apr_vformatter_buff_t *)* flush_func=0x6eec8710, apr_vformatter_buff_t * vbuff=0x00fcfd34, const char * fmt=0x1008ed7d, char * ap=0x20000000)  Line 971	
 	libapr-1.dll!apr_pvsprintf(apr_pool_t * pool=0x030d2080, const char * fmt=0x1008ed68, char * ap=0x00000000)  Line 1142 + 0x16 bytes	
 	mod_mapcache.dll!apache_context_request_log(mapcache_context * c=0x030d3d58, mapcache_log_level level=MAPCACHE_DEBUG, char * message=0x1008ed68, ...)  Line 148 + 0x15 bytes	
 	mod_mapcache.dll!mapcache_tileset_tile_get(mapcache_context * ctx=0x030d3d58, mapcache_tile * tile=0x030e00b8)  Line 567 + 0x30 bytes	
 	mod_mapcache.dll!mapcache_prefetch_tiles(mapcache_context * ctx=0x030d3d58, mapcache_tile * * tiles=0x030d4050, int ntiles=1)  Line 93 + 0x13 bytes	
 	mod_mapcache.dll!mapcache_core_get_tile(mapcache_context * ctx=0x030d3d58, mapcache_request_get_tile * req_tile=0x030d4038)  Line 209 + 0x17 bytes	
 	mod_mapcache.dll!mod_mapcache_request_handler(request_rec * r=0x030d20c0)  Line 293 + 0xd bytes	
 	libhttpd.dll!ap_run_handler(request_rec * r=0x030d20c0)  Line 158 + 0x21 bytes	
 	libhttpd.dll!ap_invoke_handler(request_rec * r=0x1008e11c)  Line 376 + 0x6 bytes	
 	libhttpd.dll!ap_process_request(request_rec * r=0x030d20c0)  Line 286	
 	libhttpd.dll!ap_process_http_connection(conn_rec * c=0x009e89c0)  Line 190 + 0x6 bytes	
 	libhttpd.dll!ap_run_process_connection(conn_rec * c=0x009e89c0)  Line 43 + 0x21 bytes	
 	libhttpd.dll!ap_process_connection(conn_rec * c=0x009e89c0, void * csd=0x009e8868)  Line 190 + 0x6 bytes	
 	libhttpd.dll!worker_main(void * thread_num_val=0x009e89b8)  Line 784

[aboudreault]

Thanks to you jmckenna! I've been able to fix the windows crash with your nice call stack! Fixed the last issue in r12992. I haven't tested yet if the log reporting was working as expected yet but should do it early tomorrow.

[jmckenna]

Great. phew, that was hard work building an environment for that trace.

I verify that the fix removed the crash - I am watching my disk cache get generated through browser requests, on Windows!

[jmckenna]

sqlite cache seems very slow, pretty unusable I imagine. Works though.

[jmckenna]

tiff cache works (with Alan's r13003 changes).

[jmckenna]

I don't have good news with my fcgi attempts with mapcache:

[Thu Jan 19 15:47:06 2012] [warn] [client 127.0.0.1] mod_fcgid: stderr: missing a service
[Thu Jan 19 15:47:06 2012] [error] [client 127.0.0.1] Premature end of script headers: mapcache.exe

My feeling is that mapcache-windows can be released now with the Apache module working, and come back to this later.