tac command missing in OSX

[kyngchaos]

v.in.garmin and v.in.mapgen use the 'tac' utility command. This is not included in Mac OS X, and possibly some other *nix flavors.

It looks like 'tail -r' achieves the same purpose. Included are patches for v.in.garmin and v.in.mapgen. It tested OK with v.in.mapgen on a mapgen file from NOAA. I have no matlab file to test it on, or garmin file.

Feedback and testing would be helpful.

Also, the BSD manpage I have for tail says the -r flag is an extension to the POSIX spec for tail, so it's possible it's not completely portable. Maybe a further test for the flag is needed.

[1gray]

Indeed, `tail -r' isn't portable, as per ​POSIX. Neither it's allowed by `tail' included with ​GNU Coreutils.

[hamish]

No "tail -r" in Debian's version of it.

It is a shame that OSX doesn't have tac, it is an elegant solution to the problem which I am hesitant to replace.

Can the Mac build provide a copy of tac somehow? (or just automatically apply the 'tail -r' patches with your build..)

Does tac exist in Cygwin / Msys?

I think the NOAA coastline extractor webpage gives you an option to download in "matlab" format. (??) anyway it is very simple, just 2 or 3 columns of numbers with arc-lines separated by 2 or 3 NaNs.

e.g. 2D:

NaN NaN
12.34 56.78
23.45 67.89
NaN NaN

For v.in.garmin I can provide gpstrans and gardump download text files if needed. (I randomly just noticed that GpsDrive provides a Garmin download tool called "garble" as well. hmph.)

Hamish

[kyngchaos]

Replying to hamish:

Can the Mac build provide a copy of tac somehow? (or just automatically apply the 'tail -r' patches with your build..)

I could provide a copy or patch it in my binary, but that's just a packager's solution, and those who build from source would still have to deal with it somehow. tac is annoying in that it's part of a GNU bundle, core-utilities, I couldn't find only tac source. And my first quick-n-dirty attempt to compile it failed.

How about testing for the -r flag, as I mentioned? A non-existent option on OSX returns tail: illegal option xxx plus a usage string with the available flags. What about other versions of tail? Or can you thing of a good way to test for the flag? If anything, I could test for uname=Darwin (tho that's awfully specific).

[1gray]

Replying to kyngchaos:

Replying to hamish:

Can the Mac build provide a copy of tac somehow? (or just automatically apply the 'tail -r' patches with your build..)

I could provide a copy or patch it in my binary, but that's just a packager's solution, and those who build from source would still have to deal with it somehow. tac is annoying in that it's part of a GNU bundle, core-utilities, I couldn't find only tac source. And my first quick-n-dirty attempt to compile it failed.

That's how most bundles work. One couldn't, e. g., easily compile r.what without compiling the rest of GRASS as well.

How about testing for the -r flag, as I mentioned? A non-existent option on OSX returns tail: illegal option xxx plus a usage string with the available flags. What about other versions of tail? Or can you thing of a good way to test for the flag? If anything, I could test for uname=Darwin (tho that's awfully specific).

I'd recommend to check for the existence of tac instead, like (untested):

search_for_executable () {
    echo "$PATH" | tr : \\n | while read _search_for_executable_dir ; do
        if [ -x "$_search_for_executable_dir/$1" ]; then
            echo "$_search_for_executable_dir/$1"
            break
        fi
    done
}

if [ -n "$TAC" ]; then
    TAC=tac
else
    TAC=$(search_for_executable "tac")
fi

if [ -z "$TAC" ]; then
    ## no `tac' found, fall back to `tail -r' instead
    TAC="tail -r"
fi

...

$TAC arguments...

[glynn]

How about testing for the -r flag, as I mentioned?

I'd recommend to check for the existence of tac instead,

Agreed.

like (untested):

I suggest either using "type", or just running e.g. "tac < /dev/null" and checking for an error. Scanning the path is problematic; it doesn't allow for shell built-ins or aliases, may not allow for extensions on Windows, etc.

That still doesn't solve the issue of what to do if the system has neither tac nor "tail -r", neither of which are mandated by POSIX/XPG/SUS.

Off the top of my head, the only thing that I can think of is to prefix each line with the line number using e.g. awk, use "sort -r", then strip the line numbers.

[kyngchaos]

Replying to 1gray:

I'd recommend to check for the existence of tac instead, like (untested):

My patch does this with which tac, a simple technique used used in many GRASS scripts. What I meant was, if tac isn't found, and tail is found, somehow check if tail has the -r flag.

But, this is really just an immediate fix to take care of the OSX case (and any other *nix without tac) where tail -r does exist. As Glynn points out, neither tac nor tail -r are in POSIX/SUS standards. A more portable fix would be nice in the long term. I wonder if some sort of requirement in GRASS 7 development would be a good idea to stick to POSIX/SUS commands?

[glynn]

Replying to kyngchaos:

I'd recommend to check for the existence of tac instead, like (untested):

My patch does this with which tac, a simple technique used used in many GRASS scripts.

FWIW, "which" isn't mandated by POSIX, while "type" is. Also, "which" doesn't work with built-ins or aliases (not that tac or tail are likely to be either).

[kyngchaos]

Ah. Looks like type is a shell builtin on OSX (and a script in /usr/bin/type to execute the builtin).

So, type -t tac and type -t tail to check for existence looks like the equivalent to which (ie no output if it's not found).

For checking for the -r flag for tail (for the immediate problem in OSX) -- maybe, similar to your tac example, tail -r < /dev/null? Tests OK here (missing tail and an illegal flag both return errors).

[hamish]

William:

tac is annoying in that it's part of a GNU bundle, core-utilities, I couldn't find only tac source. And my first quick-n-dirty attempt to compile it failed.

1gray:

That's how most bundles work. One couldn't, e. g., easily compile r.what without compiling the rest of GRASS as well.

it's not like tac has many dependencies:

$ ldd `which tac`
        linux-gate.so.1 =>  (0xffffe000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e0b000)
        /lib/ld-linux.so.2 (0xb7f5a000)

I assume it's a trivial C program. What's the complile error?

While not POSIX, I think tac will be widely available. I would still like to hear from Cygwin/MSys/other platforms.. to me the fact that it hasn't been moved out of Debian's hardcore minimalist coreutils is a good anecdotal endorsement for wide availability.

fwiw, on Debian/etch:

$ tail -r foo
tail: invalid option -- r
Try `tail --help' for more information.
$ echo $?
1

A not so nice for the user idea is to abort with an error if tac is not found.

Glynn:

Off the top of my head, the only thing that I can think of is to prefix each line with the line number using e.g. awk, use "sort -r", then strip the line numbers.

awk would be a great solution as this is fed directly into that. But I have no idea how to write that.

Here is where it is needed- create lines for 'v.in.ascii format=standard':

    # add vertex counts
    cat "${TMP}.gpst" | sed -e '1d' | tac | awk 'BEGIN { FS="\t" ; R=0 } \
        $1=="T" { printf(" %.7f %.7f\n", $4, $3) ; ++R } ; \
        $1=="" { printf("L %d 1\n", R) ; R=0 } END {;}' | tac > "${TMP}.base"

another thing for the future is to remove the vertex count requirement in the vector ascii file. I suppose it adds some level of error checking, but the computer can count.

William:

I wonder if some sort of requirement in GRASS 7 development would be a good idea to stick to POSIX/SUS commands?

that is the current goal, but I see little reason to go nuts trying to be very strict about that, e.g. for something like which that may not be POSIX but is, literally, everywhere. ie the overwhelming common use exemption.

for GRASS 7 the goal AFAIU is to replace shell scripts with more portable Python scripts.

Hamish

[glynn]

Replying to hamish:

While not POSIX, I think tac will be widely available. I would still like to hear from Cygwin/MSys/other platforms.

Cygwin has it. My MSys installation doesn't, but it's quite out of date (it has separate textutils and fileutils; AFAICT, newer versions have coreutils, which probably has it). I suspect that non-GNU systems (e.g. Solaris) probably won't have it.

Glynn:

Off the top of my head, the only thing that I can think of is to prefix each line with the line number using e.g. awk, use "sort -r", then strip the line numbers.

awk would be a great solution as this is fed directly into that. But I have no idea how to write that.

Replace:

... | tac | ...

with:

... | awk '{print NR,$0}' | sort -nr | sed 's/^[0-9]* //' | ...

I wonder if some sort of requirement in GRASS 7 development would be a good idea to stick to POSIX/SUS commands?

that is the current goal, but I see little reason to go nuts trying to be very strict about that, e.g. for something like which that may not be POSIX but is, literally, everywhere. ie the overwhelming common use exemption.

I don't see any reason to use "which" when "type" is mandated by POSIX.

[1gray]

Replying to glynn:

Replying to hamish:

![...]

awk would be a great solution as this is fed directly into that. But I have no idea how to write that.

Replace: ... | tac | ... with: ... | awk '{print NR,$0}' | sort -nr | sed 's/^[0-9]* //' | ...

I see no reason to use Awk to number lines, since there's ​nl. Neither I see a reason to use sort, since Awk has arrays:

awk_tac () {
    awk '1 { last = NR; line[last] = $0; } END { for (i = last; i > 0; i--) { print line[i]; } }'
}

if [ -n "$TAC" ] ; then
    ## do nothing
elif type tac ; then
    TAC=tac
else
    TAC=awk_tac
fi

... | $TAC | ...

[hamish]

Now we are getting somewhere. One little bug:

if [ -n "$TAC" ] ; then
    ## do nothing
elif type tac ; then
    TAC=tac
else
    TAC=awk_tac
fi

##do nothing with no command before it is a syntax error. you would need to run a noop like "true". but it is better to rewrite the logic of the test so that there is no empty case.

also output of type tac would need to be redirected to /dev/null.

Hamish

[1gray]

Replying to hamish:

Now we are getting somewhere. One little bug:

![...]

##do nothing with no command before it is a syntax error. you would need to run a noop like "true".

Indeed.

but it is better to rewrite the logic of the test so that there is no empty case.

also output of type tac would need to be redirected to /dev/null.

Yes. And -t should be given as well.

[1gray]

Replying to hamish:

![...]

That's how most bundles work. One couldn't, e. g., easily compile r.what without compiling the rest of GRASS as well.

it's not like tac has many dependencies:

> $ ldd `which tac`
>         linux-gate.so.1 =>  (0xffffe000)
>         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e0b000)
>         /lib/ld-linux.so.2 (0xb7f5a000)

I assume it's a trivial C program.

Unfortunately, the above doesn't show the static libraries linked in (and Coreutils probably comes with one.) Furthermore, IIRC, Coreutils uses ​Gnulib, which is not a library.

I'm not saying that making GRASS have a copy of tac is impossible. However, the amount of work needed to maintain it seems to be unreasonable.

What's the complile error?

![...]

[hamish]

I'd suggest adding a g.tac script using the above awk method in $(ETC), along the same lines as $(ETC)/echo, grocat, and run; but looking at the purpose of $(ETC)/grocat (grass65/lib/gtcltk/grocat.c), perhaps a -r flag could simply be added there for this?

 * MODULE:       grocat
 * AUTHOR(S):    Paul Kelly
 * PURPOSE:      Copies stdin to stdout in line-buffered mode until end
 *               of file is received.

(see also lib/init/echo.c, which "replaces the standard UNIX echo which varies from machine to machine")

I assume for grass 7 this is a non-issue, as v.in.mapgen (eg) methods have already been rewritten to do all this in python.

Hamish

[hamish]

As this is limited to a couple of shell scripts, I'll take the easy way out and use the awk replacement solution.

Hamish

[hamish]

modified patch applied in devbr6 r45886.

after testing on OSX recommend backporting to 6.4svn ASAP.

Hamish

[hamish]

backported fix to relbr64 in r45909. would still like to hear feedback from an OSX user.

Hamish