Opened 4 years ago

Last modified 19 months ago

#3309 new enhancement

Avoid errors in parallel creation of tempfiles

Reported by: sbl Owned by: grass-dev@…
Priority: normal Milestone: 7.6.2
Component: LibGIS Version: svn-releasebranch72
Keywords: G_tempfile() startup init Cc:
CPU: Unspecified Platform: Unspecified

Description

When tempfiles are created in parallel, conflicts and thus fatal errors can arise when two processes try to create the .tmp/HOST dir at the same time: https://lists.osgeo.org/pipermail/grass-dev/2017-March/084471.html

The conflict seems to arise only from the directory.

So, maybe it could help if the directory is created at the start of a GRASS session and removed only when the GRASS session is closed (exit)?

Change History (13)

comment:1 in reply to:  description ; Changed 4 years ago by mlennert

Replying to sbl:

When tempfiles are created in parallel, conflicts and thus fatal errors can arise when two processes try to create the .tmp/HOST dir at the same time: https://lists.osgeo.org/pipermail/grass-dev/2017-March/084471.html

The conflict seems to arise only from the directory.

So, maybe it could help if the directory is created at the start of a GRASS session and removed only when the GRASS session is closed (exit)?

Now that you mention it, I can confirm that I've also been confronted to such errors when running commands in parallel that create temp files. So +1 to needing a solution.

I don't know though if the best is to create this directory once and for all at the startup, or whether the tempfile routines in lib/gis/tempfile.c could be modified to avoid such race conditions.

comment:2 in reply to:  1 Changed 4 years ago by mmetz

Replying to mlennert:

Replying to sbl:

When tempfiles are created in parallel, conflicts and thus fatal errors can arise when two processes try to create the .tmp/HOST dir at the same time: https://lists.osgeo.org/pipermail/grass-dev/2017-March/084471.html

The conflict seems to arise only from the directory.

So, maybe it could help if the directory is created at the start of a GRASS session and removed only when the GRASS session is closed (exit)?

Now that you mention it, I can confirm that I've also been confronted to such errors when running commands in parallel that create temp files. So +1 to needing a solution.

Strange. When starting GRASS in text or GUI mode, the .tmp/HOST folder is automatically created, i.e. the error

ERROR: Unable to make mapset element .tmp/HOST (/grassdata/ETRS_33N/timseries/.tmp): File exists

should not happen, at least not in trunk and relb72.

I don't know though if the best is to create this directory once and for all at the startup, or whether the tempfile routines in lib/gis/tempfile.c could be modified to avoid such race conditions.

You could use

rm -f "`g.tempfile pid=$$`"

before running xargs, just to make sure that .tmp/HOST has been created.

comment:3 Changed 4 years ago by sbl

I run the xargs command in a GRASS_BATCH_JOB in case that can be of relevance...

Will test if

rm -f "`g.tempfile pid=$$`"

helps and report back...

comment:4 Changed 4 years ago by sbl

For the record:

Running

rm -f "`g.tempfile pid=$$`"

in advance works around the issue.

Using GRASS 7.2.1svn (r70188) in BATCH mode I do get the error message above. But if I run g.tempfile and make sure that the tmp directory is created before I run r.in.gdal through xargs, everything works just fine!

comment:5 Changed 3 years ago by martinl

Milestone: 7.2.17.2.2

comment:6 Changed 3 years ago by martinl

Milestone: 7.2.27.4.0

All enhancement tickets should be assigned to 7.4 milestone.

comment:7 Changed 3 years ago by neteler

Milestone: 7.4.07.4.1

Ticket retargeted after milestone closed

comment:8 Changed 2 years ago by neteler

Milestone: 7.4.17.4.2

comment:9 Changed 2 years ago by neteler

Does this issue persist in GRASS GIS 7.6.svn?

comment:10 in reply to:  9 Changed 2 years ago by wenzeslaus

Keywords: startup init added

Replying to neteler:

Does this issue persist in GRASS GIS 7.6.svn?

Please, if you can, test trunk (7.7) too - a lot of changes to temporary directory handling (#613, #1286, #3635, ...). And if possible, please, test also GRASS_BATCH_JOB versus --exec.

comment:11 Changed 2 years ago by martinl

Milestone: 7.4.27.6.0

All enhancement tickets should be assigned to 7.6 milestone.

comment:12 Changed 21 months ago by martinl

Milestone: 7.6.07.6.1

Ticket retargeted after milestone closed

comment:13 Changed 19 months ago by martinl

Milestone: 7.6.17.6.2

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.