OSGeo mail server on DNS blocklist

[Jeff McKenna]

I noticed that on June 17th I stopped receiving all OSGeo mails from all OSGeo mailing lists (I must be on over 200 lists). Zero since June 17th.

(someone somewhere must be laughing at this though, as one of the only mails to still make it to me are from the geoserver-devel list, as it is of course through SourceForge, ha, someone must be laughing at how ironic that is for me)

I realize that this must be regarding my own email host/provider. I have checked the osgeo6 mail logs, and the logs are littered with entries similar to:

Jun 22 06:05:49 osgeo6 postfix/smtp[14162]: 887506143EA5: to=<jmckenna@gatewaygeomatics.com>, relay=mailroot3.namespro.ca[169.55.177.75]:25, delay=2.9, delays=0.1/1.8/0.43/0.58, dsn=5.0.0, status=bounced (host mailroot3.namespro.ca[169.55.177.75] said: 554 The IP address of the sender (140.211.15.3) was found in a DNS blacklist database and was therefore refused. (in reply to RCPT TO command))

I have contacted my email provider, and will file a request to remove the blocklist ban on the OSGeo6 mail server (140.211.15.3).

I guess this ticket has not much to do with OSGeo-SAC, but maybe someday someone else out there will face something similar, and this ticket's archive may help someone.

I'll update this ticket when the DNS blocklist entry is removed.

Short story: I'm now an expert on recent GeoServer activity :)

[Jeff McKenna]

Updates:

• osgeo6 mail server IP address is listed in the Barracuda blacklist: ​https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a140.211.15.3

• I have filed a request with Barracuda to remove the entry through ​http://www.barracudanetworks.com/reputation/?pr=1&ip=140.211.15.3 . The response is:

  Request Received
  Thank you for submitting your request. If this is your first request, your IP address will have its reputation increased to "normal" for 48 hours while we investigate. It may take up to 1 hour for the reputation increase to propagate to all Barracuda Spam Firewalls globally. We appreciate your patience and apologize for any inconvenience.
  
  Your confirmation number is XXXX.
  

/me Heart's 70's song lyrics "oooo barracuda" plays in my head now, unfortunately

[Jeff McKenna]

update: Barracuda has removed the blocklist entry, for 48 hours as they investigate. Will keep you posted here on the response (especially as this ban affects many users, as I see in the log entries on the server).

[Jeff McKenna]

update: reports from the QGIS community on others not receiving emails for a few days (likely the same June 17th day). This is obviously affecting thousands of OSGeo users on various mailing lists, although most won't realize.

Will keep you updated here on the status, I'm still waiting to hear back from the "Barracuda" blocklist people.

[rduivenvoorde]

I had a look into the admin interface for QGIS dev and users list, and conclude that problably a lot of users have this 'nomail[reason]' checkbox checked now.

Would it be a possibility to do an update query in the db (/me just guessing this is in a db....)? (AFTER we are assured that we/OSGEO are not on any spam list anymore...)

[neteler]

As it affects users across many OSGeo mailing lists:

Here a script candidate from ​https://www.msapiro.net/scripts/ which could be used as a basis for an automated user reactivation:

• set_nomail.py: Sets a list member's delivery to 'disabled by admin' or optionally sets to 'disabled by bounce' and starts the notification and deletion process.
  • ​https://www.msapiro.net/scripts/set_nomail.py

The script would require some changes as it has the opposite purpose (AFAIK).

Also interesting

• get_bounce_info.py: Prints the bounce information for all list members that have any.
  • ​https://www.msapiro.net/scripts/get_bounce_info.py

However, careful with subscribers who set "nomail" on purpose (unless that's different from a bounce based "nomail", I don't know enough about these mailman details.).

Importantly, be sure that there is a fresh backup in place before modifying any mailman subscriptions.

[neteler]

Ah I just see we addressed something similar 13 years ago :-)

​https://trac.osgeo.org/osgeo/ticket/175#comment:5

[Jeff McKenna]

Just now received this message:

140.211.15.3
Removed from BARRACUDA at 6/27/2020 7:25:47 AM (UTC-04:00) Atlantic Time (Canada)

This is good news.

I think this affected many users, including the Nabble forums.

[andreaerdna]

Replying to Jeff McKenna:

I think this affected many users, including the Nabble forums.

In the ​QGIS directory of Nabble forums there are some strange posts that shouldn't be there:

​Buy US/UK/German drivers license, ID card, passport online. Jul 12

​How I can earn money online? Jun 15

​Writing an article Jun 17

​Error adding point properties Jun 17

Could these be part of the issue?

[jsanz]

@andreaerdna that's not related. I removed those messages that were posted because the permissions at the "folder" level allowed to post there. Yet another target for spammers :-(

Thanks for reporting!

[andreaerdna]

Hi Jorge, thank you for removing the out of place posts and for setting the proper permissions to the QGIS forums directory.

However, it seems the "QGIS - Developer" forum is again read-only now.

[jsanz]

However, it seems the "QGIS - Developer" forum is again read-only now.

Fixed

[mbernasocchi]

Hi all, at opengis.ch we had at least 4 subscribers being bounced from qgis-developer and other lists. I've the feeling that this could be the case for many more users. Where any actions (like the ones suggested by @neteler taken on this?

[Jeff McKenna]

Personally I've tried to keep this ticket up-to-date with any actions related to this spam issue, and tried to point those reporting this on mailing lists to this ticket. I'm not aware of any other actions.

[darkblueb]

side note - I sent one email to pka @ sourcepole.ch from California on 13jul20, through carrierzone.com mail servers (not OSGeo mail servers). The email stalled for days and by Friday had bounced.

[neteler]

As per old experience (​https://trac.osgeo.org/osgeo/ticket/175#comment:7), I have now reset all bounce rates to avoid that more ppl get unsubscribed:

withlist -a -r reset_bounce --
Importing reset_bounce...
Running reset_bounce.reset_bounce()...
Loading list africa (unlocked)   
Loading list alberta (unlocked)  
Loading list announce (unlocked) 
Loading list argentina (unlocked)
Loading list atlanticcanada (unlocked)
Loading list board-ar (unlocked) 
...
Loading list oceania-board (unlocked)
Loading list geonode-psc (unlocked)
Loading list mapproxy-dev (unlocked)
Loading list mobilitydb-dev (unlocked)
Loading list mobilitydb-users (unlocked)
Finalizing

Apparently the affected users got an email

"Your membership in the mailing list XXXXX has been disabled due to excessive bounces..."

OK, I think I have an idea how to get the unsubscribed members back:

root@osgeo6: zgrep BYBOUNCE /var/log/mailman/subscribe* | wc -l
832

This list contains the emails and list names in this style:

/var/log/mailman/subscribe:Jul 13 09:00:02 2020 (30244) grass-user: xxxx@example.com auto-unsubscribed [reason: BYBOUNCE]

Using it we could re-enable all lost members.

[mkuhn]

Hi, I just re-subscribed manually.

Is this the recommended procedure and should a nice reminder be sent out to those having been auto-unsubscribed with a link to reactivate the account or should they better be re-enabled in an automated fashion as proposed by neteler in the last comment?

[neteler]

IMHO we need to find a volunteer who write to all unsubscribed folks (for the email list, see comment:15 above) or run the re-subscribe operation.

[mkuhn]

IMHO we should assess the lazy option "re-subscribe operation" first. Is there anything holding us back from running it? Technical risks, Scripts not up to date, Risks that we resubscribe users that have been unsubscribed for a good reason?

[mkuhn]

I currently have some time to devote into community projects. I am available to invest some time here if I know what, where and the priorities. Feel free to contact me here or directly by mail.

[neteler]

Replying to mkuhn:

I currently have some time to devote into community projects. I am available to invest some time here if I know what, where and the priorities. Feel free to contact me here or directly by mail.

Personally I'd appreciate that: @SAC, what do you think how to proceed here?

[neteler]

I recently activated the "bounce" notifications in mailman for the lists I am managing and got this notification:

550-Your message to <redacted_grass_subscriber> was classified as SPAM.  Please add
    550-more content, cut down on HTML links, use fewer naughty words etc.
    Also, 550-ask your IT dept to make sure your mailserver has REVERSEDNS,
    SPF, DKIM, 550 and is not on any black lists. Your score: 360 (in reply to
    end of DATA command)

Question: do we have all measures in place and properly set up?

• REVERSEDNS
  • unclear to me: ​https://www.debouncer.com/reverse-dns-check?ip=lists.osgeo.org
• DKIM
  • no idea: ​https://www.dmarcanalyzer.com/dkim/dkim-check/
• SPF
  • looks ok: ​https://www.dmarcanalyzer.com/spf/checker/?dmarcdns%5Btype%5D=spf&dmarcdns%5Bdomain%5D=lists.osgeo.org
• not backlisted
  • looks good: ​https://www.debouncer.com/blacklistlookup?t=lists.osgeo.org

[robe]

REVERSEDNS - looks like a pass to me. I don't think it needs to resolve to the domain name we are using for mail just needs to not look like an autogenerated.

DKIM - not familiar with this -- I don't see any DKIM notes when I look at messages that come thru my gmail mailbox from osgeo6, so I suspect not.

SPF - yap this is fine.

The error you have above though looks like it's coming from osgeo6. I know Sandro had implemented some of those kind of checks on postgis mailing lists and I had started to get those like my email has too much html and some such thing. Then our whole PSC PostGIS started throwing virtual stones at strk so he set things back.

Maybe you have something similarly configured on grass list?

[neteler]

Replying to robe:

Maybe you have something similarly configured on grass list?

I think I didn't touch the settings for 5+ years.

And the problems are affecting many lists (see above for the 800 lost subscribers).

[neteler]

New: #2506: mailman: unsolicited mass subscription attempts

[robe]

Flipping this to critical as people have been complaining to me about it from gdal-dev and libtiff.

As I mentioned to in email which is not on this ticket. Mailman provides these possible fixes

​https://wiki.list.org/DEV/DMARC

So sounds like we need to munge the FROM (I guess somehow leaving out the identity of the person who posted the message) so that their domain doesn't report back with "OSGEO can't send mail on behalf"

> >>> ARC-Authentication-Results: i=1; mx.google.com;
> >>>        dkim=neutral (body hash did not verify) header.i=@mapbox.com
> >>> header.s=google header.b=UDqlMKw4;
> >>>        spf=pass (google.com: domain of
> >>> gdal-dev-bounces@lists.osgeo.org designates 140.211.15.3 as
> >>> permitted
> >>> sender) smtp.mailfrom=gdal-dev-bounces@lists.osgeo.org;
> >>>        dmarc=fail (p=REJECT sp=REJECT dis=QUARANTINE)
> >>> header.from=mapbox.com ...

As far as I can tell, the issue arises if a domain we are sending on behalf for has dmarc check turned on. It may also depend on the receiving end to verify the dmarc check, not sure all providers do or not. It's a non-issue if dmarc check required is not enabled on the "on behalf of" domain.

[robe]

Reading about this more - we are running 2.1.34

That said -- this version does have settings for mail list admins to configure this to be DMARC friendly.

From the wiki:

The from_is_list feature from 2.1.16 is always available, but it is not recommended. Use dmarc_moderation_action instead.
There are new settings in Privacy options - Sender filters:

    dmarc_moderation_action is a five valued setting with values
        Accept - accept the post without rewriting From: or wrapping the message 
        Munge From - rewrite the From: and Reply-To: as in from_is_list
        Wrap Message - wrap the message as in from_is_list
        Reject - reject the post
        Discard - Discard the post 
    dmarc_moderation_notice is a custom reject message to replace the default Reject message. 

The above options other than Accept override the from_is_list setting for messages whose original From: domain publishes a DMARC policy of p=reject or p=quarantine. A per-list option is available to limit this to just p=reject or to apply it to either p=reject or p=quarantine. If the option is Accept, the from_is_list setting applies.

There is a site option to set the default for dmarc_moderation_action and list admins may not set the action to a setting which is above the site default in the above list. E.g., if the site default is Reject, list admins can only set Reject or Discard; if the site default is Munge From, list admins can select anything but Accept.

But I do think we should changed the default dmarc_moderation_action as this is becoming more of an issue.

[wildintellect]

Which setting solves this issue with the least change? Can we try it on a list or two to verify it behaves as expected?

Then we can re-raise if it should become the default.

[rouault]

For gdal-dev, I've just changed dmarc_moderation_action to "munge from"

[Jeff McKenna]

Thanks Regina, I've set dmarc_moderation_action to "munge from" for TIFF, PROJ, mapserver-users & dev

Let's see how that works...

[gdt]

(The DMARC stuff really should be a new ticket, as its a separate issue.)

The way DKIM works is that senders apply a signature and receivers can then say "was this email signed by the originating domain".

DMARC adds a policy "if it's From: this domain and DKIM validation fails, reject it".

Mailman, when configured to add things like "[tiff]" to the subject line, and things at the end, modifies the message in such a way that the dkim signature no longer validates. (gnupg signatures are also invalidated.)

So if you turn off subject tags, and turn off footers, then there is no need to work around dmarc because the message when it arrives will pass validation. Neither of these things is necessary because the information about which list is in the List-Id header and unsubscribe etc. is also in headers.

[gdt]

While "from munging" avoids DMARC failues, it's really icky. So it should be limited to domains that have DMARC reject/quarantine polices.

At least some mailman versions have a setting to only do munging for domains that have DMARC policies.

But, if subject tagging and footer spamming is turned off, I think no mitigation will be necessary.

[strk]

I'd be ok with removing subject tagging and footer spamming, but I'm not sure all subscribers would be happy with that

[gdt]

@strk: Maybe some would not be happy, but some are not happy with it happening and I think many are unhappy about getting messages with fake From: addresses. There is no total winning here, as I realize you understand...

[Jeff McKenna]

Tip for other list admins who want to set "dmarc_moderation_action" : it's tricky to find that setting in the Mailman interface, and I failed at this in my first attempt, so here is the proper way to set it:

• goto your main admin page for the list (example url: ​https://lists.osgeo.org/mailman/admin/xxxx-users)
• hint: don't just search for 'dmarc' on that main admin page, as it finds an incorrect setting
• instead, at the top of the main admin page, click on "Privacy options..."
• next click on "Sender filters"
• should bring you to a url such as ​https://lists.osgeo.org/mailman/admin/xxxx-users/privacy/sender
• now you can search for "dmarc_moderation_action" and set it to "Munge From"

Or you can edit the url for your own list name, and go directly to that setting page: ​https://lists.osgeo.org/mailman/admin/xxxx-users/?VARHELP=privacy/sender/dmarc_moderation_action (replace "xxxx-users" with your list name)

[gdt]

Could you explain how to make it so that only DMARC-impaired addresses get munged? The tiff list is currently in a bad state, munging addresses that don't need to be munged.

[Jeff McKenna]

@gdt what settings do you need set? EvenR set it properly for TIFF, and I had tried earlier but made a mistake in the TIFF settings. Hey, that's why they pay me the big bucks I guess, I tried. @gdt please let us know how to help you.

[gdt]

Every single message to tiff is being from-munged. On many mailinglists, I see messages only get from-munging if they are from yahoo/aol or someplace that has DMARC enforcement turned on. This includes messages from you - but gatewaygeomatics.com does not publish a DMARC record. A message three hours ago from codelibre.net was munged, and that domain does not publish a DMARC record either.

This is really about having "dmarc_mitigate_unconditionally" set to false.

(And separately, I think it would be far better subject tagging and footer spam to be disabled, and dmarc mitigation backed out as then the messages would no longer be modified.)

[Jeff McKenna]

@gdt I have been trying to explain that the TIFF mistakes were mine. Please re-try on the TIFF mailing list. This is why I listed the specific steps above for others, so they know which exact setting to change.

Enjoy your weekend.

[neteler]

As per comment:36 (thanks!) I have changed the following lists so that only DMARC-impaired addresses get munged:

• grass-user
• grass-dev

I'll carefully watch what's different now (or not).

[gdt]

Jeff: A message appeared on the TIFF list, and it was correctly not from-munged. So all is well. Thanks for your efforts to fix this.

[neteler]

Update: Using

​​https://lists.osgeo.org/mailman/admin/xxxx/?VARHELP=privacy/sender/dmarc_moderation_action (replace "xxxx" with the list name)

I have changed all GRASS GIS related lists to "dmarc_moderation_action" -> [x] "Munge From".

Works nicely.

[Jeff McKenna]

Closing this as the original reported issue is solved.