Opened 4 years ago

Closed 3 years ago

#2475 closed task (fixed)

OSGeo mail server on DNS blocklist

Reported by: Jeff McKenna Owned by: jsanz
Priority: critical Milestone: Unplanned
Component: SysAdmin/Mailman Keywords: mailman
Cc:

Description (last modified by Jeff McKenna)

I noticed that on June 17th I stopped receiving all OSGeo mails from all OSGeo mailing lists (I must be on over 200 lists). Zero since June 17th.

(someone somewhere must be laughing at this though, as one of the only mails to still make it to me are from the geoserver-devel list, as it is of course through SourceForge, ha, someone must be laughing at how ironic that is for me)

I realize that this must be regarding my own email host/provider. I have checked the osgeo6 mail logs, and the logs are littered with entries similar to:

Jun 22 06:05:49 osgeo6 postfix/smtp[14162]: 887506143EA5: to=<jmckenna@gatewaygeomatics.com>, relay=mailroot3.namespro.ca[169.55.177.75]:25, delay=2.9, delays=0.1/1.8/0.43/0.58, dsn=5.0.0, status=bounced (host mailroot3.namespro.ca[169.55.177.75] said: 554 The IP address of the sender (140.211.15.3) was found in a DNS blacklist database and was therefore refused. (in reply to RCPT TO command))

I have contacted my email provider, and will file a request to remove the blocklist ban on the OSGeo6 mail server (140.211.15.3).

I guess this ticket has not much to do with OSGeo-SAC, but maybe someday someone else out there will face something similar, and this ticket's archive may help someone.

I'll update this ticket when the DNS blocklist entry is removed.

Short story: I'm now an expert on recent GeoServer activity :)

Change History (44)

comment:1 by Jeff McKenna, 4 years ago

Updates:

  • I have filed a request with Barracuda to remove the entry through http://www.barracudanetworks.com/reputation/?pr=1&ip=140.211.15.3 . The response is:
    Request Received
    Thank you for submitting your request. If this is your first request, your IP address will have its reputation increased to "normal" for 48 hours while we investigate. It may take up to 1 hour for the reputation increase to propagate to all Barracuda Spam Firewalls globally. We appreciate your patience and apologize for any inconvenience.
    
    Your confirmation number is XXXX.
    

/me Heart's 70's song lyrics "oooo barracuda" plays in my head now, unfortunately

Last edited 4 years ago by Jeff McKenna (previous) (diff)

comment:2 by Jeff McKenna, 4 years ago

update: Barracuda has removed the blocklist entry, for 48 hours as they investigate. Will keep you posted here on the response (especially as this ban affects many users, as I see in the log entries on the server).

Last edited 4 years ago by Jeff McKenna (previous) (diff)

comment:3 by Jeff McKenna, 4 years ago

update: reports from the QGIS community on others not receiving emails for a few days (likely the same June 17th day). This is obviously affecting thousands of OSGeo users on various mailing lists, although most won't realize.

Will keep you updated here on the status, I'm still waiting to hear back from the "Barracuda" blocklist people.

Last edited 4 years ago by Jeff McKenna (previous) (diff)

comment:4 by rduivenvoorde, 4 years ago

I had a look into the admin interface for QGIS dev and users list, and conclude that problably a lot of users have this 'nomail[reason]' checkbox checked now.

Would it be a possibility to do an update query in the db (/me just guessing this is in a db....)? (AFTER we are assured that we/OSGEO are not on any spam list anymore...)

comment:5 by neteler, 4 years ago

Keywords: mailman added

As it affects users across many OSGeo mailing lists:

Here a script candidate from https://www.msapiro.net/scripts/ which could be used as a basis for an automated user reactivation:

The script would require some changes as it has the opposite purpose (AFAIK).

Also interesting

However, careful with subscribers who set "nomail" on purpose (unless that's different from a bounce based "nomail", I don't know enough about these mailman details.).

Importantly, be sure that there is a fresh backup in place before modifying any mailman subscriptions.

comment:6 by neteler, 4 years ago

Ah I just see we addressed something similar 13 years ago :-)

https://trac.osgeo.org/osgeo/ticket/175#comment:5

comment:7 by Jeff McKenna, 4 years ago

Just now received this message:

140.211.15.3
Removed from BARRACUDA at 6/27/2020 7:25:47 AM (UTC-04:00) Atlantic Time (Canada)

This is good news.

I think this affected many users, including the Nabble forums.

in reply to:  7 comment:8 by andreaerdna, 4 years ago

Replying to Jeff McKenna:

I think this affected many users, including the Nabble forums.

In the QGIS directory of Nabble forums there are some strange posts that shouldn't be there:

Buy US/UK/German drivers license, ID card, passport online. Jul 12

How I can earn money online? Jun 15

Writing an article Jun 17

Error adding point properties Jun 17

Could these be part of the issue?

Last edited 4 years ago by andreaerdna (previous) (diff)

comment:9 by jsanz, 4 years ago

@andreaerdna that's not related. I removed those messages that were posted because the permissions at the "folder" level allowed to post there. Yet another target for spammers :-(

Thanks for reporting!

in reply to:  9 comment:10 by andreaerdna, 4 years ago

Hi Jorge, thank you for removing the out of place posts and for setting the proper permissions to the QGIS forums directory.

However, it seems the "QGIS - Developer" forum is again read-only now.

comment:11 by jsanz, 4 years ago

However, it seems the "QGIS - Developer" forum is again read-only now.

Fixed

comment:12 by mbernasocchi, 4 years ago

Hi all, at opengis.ch we had at least 4 subscribers being bounced from qgis-developer and other lists. I've the feeling that this could be the case for man more users. Where any actions (like the ones suggested by @neteler taken on this?

Version 0, edited 4 years ago by mbernasocchi (next)

comment:13 by Jeff McKenna, 4 years ago

Personally I've tried to keep this ticket up-to-date with any actions related to this spam issue, and tried to point those reporting this on mailing lists to this ticket. I'm not aware of any other actions.

comment:14 by darkblueb, 4 years ago

side note - I sent one email to pka @ sourcepole.ch from California on 13jul20, through carrierzone.com mail servers (not OSGeo mail servers). The email stalled for days and by Friday had bounced.

comment:15 by neteler, 4 years ago

As per old experience (https://trac.osgeo.org/osgeo/ticket/175#comment:7), I have now reset all bounce rates to avoid that more ppl get unsubscribed:

withlist -a -r reset_bounce --
Importing reset_bounce...
Running reset_bounce.reset_bounce()...
Loading list africa (unlocked)   
Loading list alberta (unlocked)  
Loading list announce (unlocked) 
Loading list argentina (unlocked)
Loading list atlanticcanada (unlocked)
Loading list board-ar (unlocked) 
...
Loading list oceania-board (unlocked)
Loading list geonode-psc (unlocked)
Loading list mapproxy-dev (unlocked)
Loading list mobilitydb-dev (unlocked)
Loading list mobilitydb-users (unlocked)
Finalizing

Apparently the affected users got an email

"Your membership in the mailing list XXXXX has been disabled due to excessive bounces..."

OK, I think I have an idea how to get the unsubscribed members back:

root@osgeo6: zgrep BYBOUNCE /var/log/mailman/subscribe* | wc -l
832

This list contains the emails and list names in this style:

/var/log/mailman/subscribe:Jul 13 09:00:02 2020 (30244) grass-user: xxxx@example.com auto-unsubscribed [reason: BYBOUNCE]

Using it we could re-enable all lost members.

comment:16 by mkuhn, 4 years ago

Hi, I just re-subscribed manually.

Is this the recommended procedure and should a nice reminder be sent out to those having been auto-unsubscribed with a link to reactivate the account or should they better be re-enabled in an automated fashion as proposed by neteler in the last comment?

comment:17 by neteler, 4 years ago

IMHO we need to find a volunteer who write to all unsubscribed folks (for the email list, see comment:15 above) or run the re-subscribe operation.

comment:18 by mkuhn, 4 years ago

IMHO we should assess the lazy option "re-subscribe operation" first. Is there anything holding us back from running it? Technical risks, Scripts not up to date, Risks that we resubscribe users that have been unsubscribed for a good reason?

comment:19 by mkuhn, 4 years ago

I currently have some time to devote into community projects. I am available to invest some time here if I know what, where and the priorities. Feel free to contact me here or directly by mail.

in reply to:  19 comment:20 by neteler, 4 years ago

Replying to mkuhn:

I currently have some time to devote into community projects. I am available to invest some time here if I know what, where and the priorities. Feel free to contact me here or directly by mail.

Personally I'd appreciate that: @SAC, what do you think how to proceed here?

comment:21 by neteler, 4 years ago

I recently activated the "bounce" notifications in mailman for the lists I am managing and got this notification:

550-Your message to <redacted_grass_subscriber> was classified as SPAM.  Please add
    550-more content, cut down on HTML links, use fewer naughty words etc.
    Also, 550-ask your IT dept to make sure your mailserver has REVERSEDNS,
    SPF, DKIM, 550 and is not on any black lists. Your score: 360 (in reply to
    end of DATA command)

Question: do we have all measures in place and properly set up?

comment:22 by robe, 4 years ago

REVERSEDNS - looks like a pass to me. I don't think it needs to resolve to the domain name we are using for mail just needs to not look like an autogenerated.

DKIM - not familiar with this -- I don't see any DKIM notes when I look at messages that come thru my gmail mailbox from osgeo6, so I suspect not.

SPF - yap this is fine.

The error you have above though looks like it's coming from osgeo6. I know Sandro had implemented some of those kind of checks on postgis mailing lists and I had started to get those like my email has too much html and some such thing. Then our whole PSC PostGIS started throwing virtual stones at strk so he set things back.

Maybe you have something similarly configured on grass list?

in reply to:  22 comment:23 by neteler, 4 years ago

Replying to robe:

Maybe you have something similarly configured on grass list?

I think I didn't touch the settings for 5+ years.

And the problems are affecting many lists (see above for the 800 lost subscribers).

comment:24 by neteler, 4 years ago

New: #2506: mailman: unsolicited mass subscription attempts

comment:25 by Jeff McKenna, 4 years ago

Description: modified (diff)
Summary: OSGeo mail server on DNS BlacklistOSGeo mail server on DNS blocklist

comment:26 by Jeff McKenna, 4 years ago

Description: modified (diff)

comment:27 by robe, 3 years ago

Priority: normalcritical

Flipping this to critical as people have been complaining to me about it from gdal-dev and libtiff.

As I mentioned to in email which is not on this ticket. Mailman provides these possible fixes

https://wiki.list.org/DEV/DMARC

So sounds like we need to munge the FROM (I guess somehow leaving out the identity of the person who posted the message) so that their domain doesn't report back with "OSGEO can't send mail on behalf"

> >>> ARC-Authentication-Results: i=1; mx.google.com;
> >>>        dkim=neutral (body hash did not verify) header.i=@mapbox.com
> >>> header.s=google header.b=UDqlMKw4;
> >>>        spf=pass (google.com: domain of
> >>> gdal-dev-bounces@lists.osgeo.org designates 140.211.15.3 as
> >>> permitted
> >>> sender) smtp.mailfrom=gdal-dev-bounces@lists.osgeo.org;
> >>>        dmarc=fail (p=REJECT sp=REJECT dis=QUARANTINE)
> >>> header.from=mapbox.com ...

As far as I can tell, the issue arises if a domain we are sending on behalf for has dmarc check turned on. It may also depend on the receiving end to verify the dmarc check, not sure all providers do or not. It's a non-issue if dmarc check required is not enabled on the "on behalf of" domain.

comment:28 by robe, 3 years ago

Reading about this more - we are running 2.1.34

That said -- this version does have settings for mail list admins to configure this to be DMARC friendly.

From the wiki:

The from_is_list feature from 2.1.16 is always available, but it is not recommended. Use dmarc_moderation_action instead.
There are new settings in Privacy options - Sender filters:

    dmarc_moderation_action is a five valued setting with values
        Accept - accept the post without rewriting From: or wrapping the message 
        Munge From - rewrite the From: and Reply-To: as in from_is_list
        Wrap Message - wrap the message as in from_is_list
        Reject - reject the post
        Discard - Discard the post 
    dmarc_moderation_notice is a custom reject message to replace the default Reject message. 

The above options other than Accept override the from_is_list setting for messages whose original From: domain publishes a DMARC policy of p=reject or p=quarantine. A per-list option is available to limit this to just p=reject or to apply it to either p=reject or p=quarantine. If the option is Accept, the from_is_list setting applies.

There is a site option to set the default for dmarc_moderation_action and list admins may not set the action to a setting which is above the site default in the above list. E.g., if the site default is Reject, list admins can only set Reject or Discard; if the site default is Munge From, list admins can select anything but Accept.

But I do think we should changed the default dmarc_moderation_action as this is becoming more of an issue.

comment:29 by wildintellect, 3 years ago

Which setting solves this issue with the least change? Can we try it on a list or two to verify it behaves as expected?

Then we can re-raise if it should become the default.

comment:30 by rouault, 3 years ago

For gdal-dev, I've just changed dmarc_moderation_action to "munge from"

comment:31 by Jeff McKenna, 3 years ago

Thanks Regina, I've set dmarc_moderation_action to "munge from" for TIFF, PROJ, mapserver-users & dev

Let's see how that works...

comment:32 by gdt, 3 years ago

(The DMARC stuff really should be a new ticket, as its a separate issue.)

The way DKIM works is that senders apply a signature and receivers can then say "was this email signed by the originating domain".

DMARC adds a policy "if it's From: this domain and DKIM validation fails, reject it".

Mailman, when configured to add things like "[tiff]" to the subject line, and things at the end, modifies the message in such a way that the dkim signature no longer validates. (gnupg signatures are also invalidated.)

So if you turn off subject tags, and turn off footers, then there is no need to work around dmarc because the message when it arrives will pass validation. Neither of these things is necessary because the information about which list is in the List-Id header and unsubscribe etc. is also in headers.

comment:33 by gdt, 3 years ago

While "from munging" avoids DMARC failues, it's really icky. So it should be limited to domains that have DMARC reject/quarantine polices.

At least some mailman versions have a setting to only do munging for domains that have DMARC policies.

But, if subject tagging and footer spamming is turned off, I think no mitigation will be necessary.

comment:34 by strk, 3 years ago

I'd be ok with removing subject tagging and footer spamming, but I'm not sure all subscribers would be happy with that

comment:35 by gdt, 3 years ago

@strk: Maybe some would not be happy, but some are not happy with it happening and I think many are unhappy about getting messages with fake From: addresses. There is no total winning here, as I realize you understand...

comment:36 by Jeff McKenna, 3 years ago

Tip for other list admins who want to set "dmarc_moderation_action" : it's tricky to find that setting in the Mailman interface, and I failed at this in my first attempt, so here is the proper way to set it:

Or you can edit the url for your own list name, and go directly to that setting page: https://lists.osgeo.org/mailman/admin/xxxx-users/?VARHELP=privacy/sender/dmarc_moderation_action (replace "xxxx-users" with your list name)

Last edited 3 years ago by Jeff McKenna (previous) (diff)

comment:37 by gdt, 3 years ago

Could you explain how to make it so that only DMARC-impaired addresses get munged? The tiff list is currently in a bad state, munging addresses that don't need to be munged.

comment:38 by Jeff McKenna, 3 years ago

@gdt what settings do you need set? EvenR set it properly for TIFF, and I had tried earlier but made a mistake in the TIFF settings. Hey, that's why they pay me the big bucks I guess, I tried. @gdt please let us know how to help you.

comment:39 by gdt, 3 years ago

Every single message to tiff is being from-munged. On many mailinglists, I see messages only get from-munging if they are from yahoo/aol or someplace that has DMARC enforcement turned on. This includes messages from you - but gatewaygeomatics.com does not publish a DMARC record. A message three hours ago from codelibre.net was munged, and that domain does not publish a DMARC record either.

This is really about having "dmarc_mitigate_unconditionally" set to false.

(And separately, I think it would be far better subject tagging and footer spam to be disabled, and dmarc mitigation backed out as then the messages would no longer be modified.)

comment:40 by Jeff McKenna, 3 years ago

@gdt I have been trying to explain that the TIFF mistakes were mine. Please re-try on the TIFF mailing list. This is why I listed the specific steps above for others, so they know which exact setting to change.

Enjoy your weekend.

comment:41 by neteler, 3 years ago

As per comment:36 (thanks!) I have changed the following lists so that only DMARC-impaired addresses get munged:

  • grass-user
  • grass-dev

I'll carefully watch what's different now (or not).

comment:42 by gdt, 3 years ago

Jeff: A message appeared on the TIFF list, and it was correctly not from-munged. So all is well. Thanks for your efforts to fix this.

comment:43 by neteler, 3 years ago

Update: Using

https://lists.osgeo.org/mailman/admin/xxxx/?VARHELP=privacy/sender/dmarc_moderation_action (replace "xxxx" with the list name)

I have changed all GRASS GIS related lists to "dmarc_moderation_action" -> [x] "Munge From".

Works nicely.

comment:44 by Jeff McKenna, 3 years ago

Resolution: fixed
Status: newclosed

Closing this as the original reported issue is solved.

Note: See TracTickets for help on using tickets.