Opened 9 years ago

Closed 3 years ago

#909 closed defect (wontfix)

FTBFS: r.mapcalc in trunk

Reported by: hamish Owned by: grass-dev@…
Priority: normal Milestone: 7.0.5
Component: Compiling Version: svn-trunk
Keywords: r.mapcalc Cc:
CPU: x86-64 Platform: Linux

Description

Hi,

latest trunk:

"make -j8" on debian/stable, 64bit quad-core

<stdout>:1555: warning: 'yyunput' defined but not used
<stdout>:1598: warning: 'input' defined but not used
gcc -L/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib -L/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib -Wl,--export-dynamic -Wl,-rpath-link,/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/lib   -o /usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/bin/r.mapcalc OBJ.x86_64-unknown-linux-gnu/check.o OBJ.x86_64-unknown-linux-gnu/column_shift.o OBJ.x86_64-unknown-linux-gnu/evaluate.o OBJ.x86_64-unknown-linux-gnu/expression.o OBJ.x86_64-unknown-linux-gnu/function.o OBJ.x86_64-unknown-linux-gnu/main.o OBJ.x86_64-unknown-linux-gnu/map.o OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o OBJ.x86_64-unknown-linux-gnu/xabs.o OBJ.x86_64-unknown-linux-gnu/xacos.o OBJ.x86_64-unknown-linux-gnu/xadd.o OBJ.x86_64-unknown-linux-gnu/xand.o OBJ.x86_64-unknown-linux-gnu/xand2.o OBJ.x86_64-unknown-linux-gnu/xasin.o OBJ.x86_64-unknown-linux-gnu/xatan.o OBJ.x86_64-unknown-linux-gnu/xbitand.o OBJ.x86_64-unknown-linux-gnu/xbitnot.o OBJ.x86_64-unknown-linux-gnu/xbitor.o OBJ.x86_64-unknown-linux-gnu/xbitxor.o OBJ.x86_64-unknown-linux-gnu/xcoor.o OBJ.x86_64-unknown-linux-gnu/xcos.o OBJ.x86_64-unknown-linux-gnu/xdiv.o OBJ.x86_64-unknown-linux-gnu/xdouble.o OBJ.x86_64-unknown-linux-gnu/xeq.o OBJ.x86_64-unknown-linux-gnu/xeval.o OBJ.x86_64-unknown-linux-gnu/xexp.o OBJ.x86_64-unknown-linux-gnu/xfloat.o OBJ.x86_64-unknown-linux-gnu/xge.o OBJ.x86_64-unknown-linux-gnu/xgraph.o OBJ.x86_64-unknown-linux-gnu/xgt.o OBJ.x86_64-unknown-linux-gnu/xif.o OBJ.x86_64-unknown-linux-gnu/xint.o OBJ.x86_64-unknown-linux-gnu/xisnull.o OBJ.x86_64-unknown-linux-gnu/xle.o OBJ.x86_64-unknown-linux-gnu/xlog.o OBJ.x86_64-unknown-linux-gnu/xlt.o OBJ.x86_64-unknown-linux-gnu/xmax.o OBJ.x86_64-unknown-linux-gnu/xmedian.o OBJ.x86_64-unknown-linux-gnu/xmin.o OBJ.x86_64-unknown-linux-gnu/xmod.o OBJ.x86_64-unknown-linux-gnu/xmode.o OBJ.x86_64-unknown-linux-gnu/xmul.o OBJ.x86_64-unknown-linux-gnu/xne.o OBJ.x86_64-unknown-linux-gnu/xneg.o OBJ.x86_64-unknown-linux-gnu/xnot.o OBJ.x86_64-unknown-linux-gnu/xnull.o OBJ.x86_64-unknown-linux-gnu/xor.o OBJ.x86_64-unknown-linux-gnu/xor2.o OBJ.x86_64-unknown-linux-gnu/xpow.o OBJ.x86_64-unknown-linux-gnu/xrand.o OBJ.x86_64-unknown-linux-gnu/xres.o OBJ.x86_64-unknown-linux-gnu/xround.o OBJ.x86_64-unknown-linux-gnu/xrowcol.o OBJ.x86_64-unknown-linux-gnu/xshiftl.o OBJ.x86_64-unknown-linux-gnu/xshiftr.o OBJ.x86_64-unknown-linux-gnu/xshiftru.o OBJ.x86_64-unknown-linux-gnu/xsin.o OBJ.x86_64-unknown-linux-gnu/xsqrt.o OBJ.x86_64-unknown-linux-gnu/xsub.o OBJ.x86_64-unknown-linux-gnu/xtan.o  -lgrass_gis -lgrass_raster -lgrass_btree  -lreadline  -lhistory   -lpthread   -lm

/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.tab.c:1442: undefined reference to `yylex'
OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o: In function `parse_string':
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:270: undefined reference to `initialize_scanner_string'
OBJ.x86_64-unknown-linux-gnu/mapcalc.tab.o: In function `parse_stream':
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:278: undefined reference to `initialize_scanner_stream'
collect2: ld returned 1 exit status
make[3]: *** [/usr/local/src/grass/svn/trunk/dist.x86_64-unknown-linux-gnu/bin/r.mapcalc] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/usr/local/src/grass/svn/trunk/raster/r.mapcalc'

Hamish

Change History (14)

comment:1 Changed 9 years ago by hamish

after cd r.mapcalc and make clean I can build it successfully. so a makefile dep problem I guess.

also, I noticed this whiz by:

mapcalc.l:216: warning, -s option given but default rule can be matched

Hamish

comment:2 in reply to:  description ; Changed 9 years ago by glynn

Replying to hamish:

latest trunk:

"make -j8" on debian/stable, 64bit quad-core

gcc ... OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o ...
...
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.tab.c:1442: undefined reference to `yylex'
...
/usr/local/src/grass/svn/trunk/raster/r.mapcalc/mapcalc.y:270: undefined reference to `initialize_scanner_string'
...

These should be in mapcalc.yy.o, compiled from mapcalc.yy.c, generated by lex from mapcalc.l.

mapcalc.yy.o is in the list of object files being linked, so make knows that it needs to build it and believes that it has done so, and gcc isn't complaining about its absence. Can you provide the make output corresponding to the generation and compilation of mapcalc.yy.c?

comment:3 in reply to:  1 Changed 9 years ago by glynn

Replying to hamish:

also, I noticed this whiz by:

mapcalc.l:216: warning, -s option given but default rule can be matched

Harmless. It indicates that not all possible inputs are valid, i.e. it's possible for the expression to have syntax errors. Using the -s flag causes syntax errors to generate an error rather than silently writing unmatched text to stdout.

comment:4 in reply to:  2 ; Changed 9 years ago by hamish

Replying to glynn:

These should be in mapcalc.yy.o, compiled from mapcalc.yy.c, generated by lex from mapcalc.l.

mapcalc.yy.o is in the list of object files being linked, so make knows that it needs to build it and believes that it has done so, and gcc isn't complaining about its absence. Can you provide the make output corresponding to the generation and compilation of mapcalc.yy.c?

sure. due to the -j8 the output is a bit jumbled up so I'll send you the entire build log off-list. I just tried again today and it built ok, but I guess who wins the race is just a whim of timing.

Hamish

comment:5 in reply to:  4 Changed 9 years ago by glynn

Replying to hamish:

build log with faulty r.mapcalc build attached.

Looking at that...

3175:	make[4]: Entering directory `/home/hbowman/dev/grass/svn/trunk/raster/r.colors'
3176:	make -C ../r.mapcalc
...
3178:	make[5]: Entering directory `/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc'
...
...
3569:	make -C r.mapcalc || echo /home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc >> /home/hbowman/dev/grass/svn/trunk/error.log
...
3577:	make[3]: Entering directory `/home/hbowman/dev/grass/svn/trunk/raster/r.mapcalc'
...
3589:	gcc ... -o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o -c mapcalc.yy.c
...
3796:	gcc ... -o OBJ.x86_64-unknown-linux-gnu/mapcalc.yy.o -c mapcalc.yy.c

IOW, it's compiling it twice.

Note: r.colors requires r.mapcalc for the thumbnails.

It appears that the raster -> r.mapcalc build starts while the raster -> r.colors -> r.mapcalc build is still ongoing. The first build compiles mapcalc.yy.c to mapcalc.yy.o, then later tries to link it while the second build is compiling it, and ends up using an incomplete file.

Unrelated parallel make tasks don't communicate with each other (if they did, the second task would note that the first task was already building mapcalc.yy.o, and just wait for it to finish).

When raster/Makefile "make"s r.colors and r.mapcalc concurrently, the tasks are considered unrelated, and that doesn't change when they bump into each other in r.mapcalc.

One option is to simply remove r.mapcalc from the SUBDIRS list in raster/Makefile, as r.colors is going to build it. At least, that was my first thought, but if r.colors.html is up-to-date, it won't happen; and, if r.mapcalc already exists, it won't be re-made.

Another option is to move the thumbnail generation into e.g. man/Makefile. But that won't re-build the thumbnails if r.colors is re-made (OTOH, the thumbnail images are really a derivative of lib/gis/colors/ rather than r.colors per se).

Another option is to change raster/Makefile thus:

-htmldir: parsubdirs 
+htmldir:
	$(MAKE) -C r.mapcalc
	$(MAKE) parsubdirs

BTW, given that there are 200 compilation and 40 linking commands occurring in the interval in question, I'm quite surprised that you managed to actually trigger this error.

comment:6 Changed 9 years ago by hamish

BTW, given that there are 200 compilation and 40 linking commands occurring in the interval in question, I'm quite surprised that you managed to actually trigger this error.

I'm not, I've always had this natural ability to cause computers to crash in strange and interesting ways.

... maybe the colorbar thumbnails python script is relatively slow to run versus compiling a dozen C modules? (for one thing it has to start python which can take a little while..) (the entire grass compile is done in less than 2 minutes on that machine if I give it -j8)

Hamish

comment:7 Changed 5 years ago by neteler

Untouched for 4 years, still an issue?

comment:8 Changed 5 years ago by hamish

Resolution: worksforme
Status: newclosed

In the past I was able to trigger this with regularity. For now it seems to be working on a similar machine but with an upgraded version of the linux distro. AFAIK it was never explicitly fixed, although it may have been incidentally fixed, so closing the ticket but will keep an eye out for it in case it comes back.

Hamish

comment:9 in reply to:  7 Changed 5 years ago by glynn

Resolution: worksforme
Status: closedreopened

Replying to neteler:

Untouched for 4 years, still an issue?

I believe so.

It's caused by a race condition, so it's a matter of chance whether it actually happens. The probability increases with the number of threads.

FWIW, the use of the "-s" flag was eliminated by r55504.

comment:10 Changed 3 years ago by neteler

Milestone: 7.0.07.0.3

What is the state of this ticket?

comment:11 Changed 3 years ago by neteler

Milestone: 7.0.3

Ticket retargeted after milestone closed

comment:12 Changed 3 years ago by neteler

Milestone: 7.0.4

Ticket retargeted after 7.0.3 milestone closed

comment:13 Changed 3 years ago by martinl

Milestone: 7.0.47.0.5

comment:14 Changed 3 years ago by martinl

Resolution: wontfix
Status: reopenedclosed

No activity, closing, feel free to reopen if needed.

Note: See TracTickets for help on using tickets.