geoserver-users SF mailman to discourse?

[jive]

The GeoServer team has expressed interest in using discourse; initially for the user-list content: ​https://sourceforge.net/p/geoserver/mailman/

Is it possible to move the geoserver-users email list from source forge mailman to discourse?

It looks like an export is available for admins: ​https://sourceforge.net/p/forge/documentation/Mailing%20List%20Archives/

But it says:

mbox archives contain only obfuscated email addresses

I wonder if it will work?!

[cvvergara]

@jive

Please send me the mailbox, I will set it up in staging for you to evaluate.

[jive]

I was unable to download following this link:

• ​https://lists.sourceforge.net/mbox/geoserver-users

The error was:

The mailing-list mbox was not found.

I will ask another geoserver admin to try, and then we may have to open a source forge support ticket.

[jive]

Andrea has been able to obtained the mbox download from SF so this activity can now be worked. Please contact Andrea for a file (from a few weeks ago) that can be used for testing.

We can try it out, and then communicate the transition to our community.

[aaime]

Anyone with admin rights can now download the mbox from the link above, fresh of the day, when the migration is ready to go.

[cvvergara]

@aaime

I do not have admin rights there.

I need someone to send me the mbox via mail. vicky at erosion.dev

[cvvergara]

Mailbox received

[cvvergara]

I tried the import, basically getting errors on all mails

Short version:

...
Failed to index message in /shared/import/data/geoserver-users/geoserver-users.mbox at lines 33-1899
...
Failed to index message in /shared/import/data/geoserver-users/geoserver-users.mbox at lines 1900-1953
...
Failed to index message in /shared/import/data/geoserver-users/geoserver-users.mbox at lines 1954-2056
...
Failed to index message in /shared/import/data/geoserver-users/geoserver-users.mbox at lines 2057-2118
...
Failed to index message in /shared/import/data/geoserver-users/geoserver-users.mbox at lines 2119-2220
...

The error between each "Failed to index"

undefined method `sub' for :from:Symbol
["/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:91:in `block in index_emails'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:158:in `block (2 levels) in all_messages'", "/
var/www/discourse/script/import_scripts/mbox/support/indexer.rb:190:in `block in each_mail'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:210:in `block in each_line'", "/var/www/discourse/s
cript/import_scripts/mbox/support/indexer.rb:210:in `each_line'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:210:in `each_line'", "/var/www/discourse/script/import_scripts/mbox/support/ind
exer.rb:187:in `each_mail'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:150:in `block in all_messages'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:143:in `foreach'"
, "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:143:in `all_messages'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:67:in `index_emails'", "/var/www/discourse/script/imp
ort_scripts/mbox/support/indexer.rb:25:in `block in execute'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:22:in `each'", "/var/www/discourse/script/import_scripts/mbox/support/indexer.rb:2
2:in `execute'", "/var/www/discourse/script/import_scripts/mbox/importer.rb:43:in `index_messages'", "/var/www/discourse/script/import_scripts/mbox/importer.rb:27:in `execute'", "/var/www/discourse/script/import
_scripts/base.rb:47:in `perform'", "script/import_scripts/mbox.rb:13:in `<module:Mbox>'", "script/import_scripts/mbox.rb:11:in `<module:ImportScripts>'", "script/import_scripts/mbox.rb:10:in `<main>'"]

Example problematic header:

From aurelio@...3... Mon Jan 19 10:58:27 2004
Message-ID: <001f01c3dec6$d49f0cb0$6801a8c0@...4...>
From: =?iso-8859-1?Q?Aur=E9lio_Calegari?= <aurelio@...3...>
To: <geoserver-users@lists.sourceforge.net>
Date: Mon, 19 Jan 2004 17:00:04 -0300
MIME-Version: 1.0

An example of a header on an osgeo mailinglist

From sentto-3377054-1-991426944-postgis=postgis.refractions.net@returns.onelist.com  Fri Jun  1 13:22:16 2001
Return-Path: <sentto-3377054-1-991426944-postgis=postgis.refractions.net@returns.onelist.com>
Delivered-To: postgis@offsite.refractions.net
Received: from fj.egroups.com (fj.egroups.com [64.211.240.231])
	by offsite.refractions.net (Postfix) with SMTP id 4C17A43490
	for <postgis@postgis.refractions.net>; Fri,  1 Jun 2001 13:22:15 -0700 (PDT)
X-eGroups-Return: sentto-3377054-1-991426944-postgis=postgis.refractions.net@returns.onelist.com

Probably the aurelio@...3...

From aurelio@...3... Mon Jan 19 10:58:27 2004

Doing a grep '^From ' geoserver-users.mbox

From andrea.aime@...84... Fri Sep 27 19:40:34 2013
From mihail12@...84... Sun Sep 29 17:34:14 2013
From peter.vonminden@...2631... Mon Sep 30 02:45:30 2013
From andrea.aime@...84... Mon Sep 30 06:13:50 2013
From maurobartolomeoli@...84... Mon Sep 30 07:02:51 2013
From paulius@...4984... Mon Sep 30 07:36:57 2013
From alessio.fabiani@...84... Mon Sep 30 07:44:47 2013
From paulius@...4984... Mon Sep 30 08:01:16 2013
From andrea.aime@...84... Mon Sep 30 08:09:38 2013

[peterafrigis]

Hi Vicky

I have used sed to make the anonymised email addresses @...\d+... into a valid address, to see if this solves the import problem. Could you please try again with the file link that I emailed you?

Peter

[cvvergara]

The mailbox has been successfully created on staging for demonstration purposes

​https://discourse.staging.osgeo.org/c/geoserver-users/12

TODO to finish up the demo:

• create corresponding group
• add suitable description
• create the admins

Note: the demo does not include sending/receiving mails that feature is blocked on staging.

[jive]

In trying out staging I had some questions - ​https://discourse.staging.osgeo.org/t/about-the-geoserver-users-category/14078

I seem to be unable to edit the category description?

[robe]

Replying to jive:

In trying out staging I had some questions - ​https://discourse.staging.osgeo.org/t/about-the-geoserver-users-category/14078

I seem to be unable to edit the category description?

We were discussing that yesterday. It's a limitation of discourse that only discourse admins can edit category information metadata, though group admins can edit most metadata about a group.

Not sure if there is a workaround for that.

[jive]

The limitation is fine, we should document that the group description needs to be provided when requesting a new category.

Please use the text from the staging environment above (cut and paste for the links to work).

Welcome to the discussion category for GeoServer user community.

This category is administered by the GeoServer PSC.

* As an Open Source Geospatial Foundation project we require all participants follow our [code of conduct](https://github.com/geoserver/geoserver/blob/main/CODE_OF_CONDUCT.md). 
* This is a public channel and not to be used for the discussion of security vulnerability and exploits. See [security policy](https://github.com/geoserver/geoserver/security) for instructions on responsible disclosure.
* Being an open community requires public communication, allowing questions to be shared. Chances are someone else has the same question and we all benefit from public discussion. 
* The availability of a public forum means that sending email to an individual developer is discouraged. If you are unable to communicate in public [commercial support providers](https://geoserver.org/support/) are listed on our website.
* This forum is for use of the GeoServer software, development chatter is [directed elsewhere](https://geoserver.org/devel/). 

To use this category as a mailing list first register and then use geoserver-users@discourse.osgeo.org for communication.

The GeoServer ​proposal is now approved.

How do we obtain and provide you with a new processed SF mailman dump?

[robe]

@jive,

For what you have done above, I think that is generally done as a post on the category and then pinned so it doesn't fall off the top. That is something you can do yourself.

Kind of like what we have for pgrouting-dev category - ​https://discourse.osgeo.org/c/pgrouting/pgrouting-dev/17

The about post stays at the top.

The metadata I was talking about is like the logo and a very brief description, kind of like you see on QGIS-fr

​https://discourse.osgeo.org/c/qgis/qgis-fr-user/5

and that shows in the qgis category screen - ​https://discourse.osgeo.org/c/qgis/11

[jive]

Okay it is time to go, we have some instructions outlined, and once the discourse channel is ready we will take screen shots, update our website communication information, and share with our users.

• geoserver/user

Notes:

• You can let me know if geosever / geoserver-users is required for disambiguation

• I assume if the police changes to introduce a "projects" or "osgeo projects" top-level category that things will shuffle around a bit on us.

[robe]

Replying to jive:

Okay it is time to go, we have some instructions outlined, and once the discourse channel is ready we will take screen shots, update our website communication information, and share with our users.

• geoserver/user

Notes:

• You can let me know if geosever / geoserver-users is required for disambiguation

I think you can do geoserver/users if you want. I think the main impact I would be worried about is in the email subject line.

Right now we have that set to: %{optional_cat}%{topic_title}

I had thought that optional_cat was just the category name, but now I realize it includes the parent category so maybe just having it called users is better cause then the subject line would be shorter.

​https://meta.discourse.org/t/superseded-customize-the-subject-format-for-standard-emails/20801

• I assume if the police changes to introduce a "projects" or "osgeo projects" top-level category that things will shuffle around a bit on us.

I haven't looked at the email subject line of a category 3 levels deep - we do have ​https://discourse.osgeo.org/c/local-chapters/gfoss-it/33 which is 3 levels deep. As stated this is something discourse discourages and I had changed the config to allow it cause I figured we'd need it for what you are describing. Again I'd be concerned of the subject line.

Speaking of which all the mirrors I set up for sac, I put SAC on them just to replicate how the mailman mailing lists works, but probably a good test to take that off. So I'll do that now and see how the subject lines look.

[jive]

Thanks for experimenting with category names, advise what to do and I will update the geoserver and geonetwork planning accordingly.

• option top-level minimal: geoserver / user
• option top-level redundant: geoserver / geoserver-user
• option nested minimal: project / geoserver / user
• option nested redundant: osgeo project / geoserver / geoserver-user

I made an export of the mbox yesterday ...resulting in a 1.3G text file. What is the best way to provide that to you?

Notes:

• I tend to prefer singular (project,user) rather than plural (projects, users)
• We may of already lost any hope of consistency, can categories be renamed?

[robe]

Replying to jive:

Thanks for experimenting with category names, advise what to do and I will update the geoserver and geonetwork planning accordingly.

• option top-level minimal: geoserver / user
• option top-level redundant: geoserver / geoserver-user
• option nested minimal: project / geoserver / user
• option nested redundant: osgeo project / geoserver / geoserver-user

I'd start with the short version as you stated. I'm still feeling iffy about burying projects under project as that affects more than geoserver and QGIS main listings are planning to be coming on board in the next month or 2.

I feel like for now we should go with:

• option top-level minimal: geoserver / user

and move one level down after further discussions with all project stakeholders, which right now is just QGIS, geoserver, and pgrouting

One thing I did notice as another consequence of going 3 levels, is the description on the home page only shows for the uppest parent level even if subparents have descriptions. I haven't investigated if there is a fix for that.

Like if you look at SAC sub category, it has no description showing on homepage though it does have a description. So we might loose that for pgRouting and QGIS, their descriptions if they are buried one level down in a project folder.

I also haven't investigated other ways for organizing the home page like custom plugins etc. I think there might be many more we haven't considered. Right now we are using the categories page like a lazy homepage. It would be great if someone could take on that work to figure out our options, cause I'd rather focus on system administration than UI look and feel which I know I suck at.

I made an export of the mbox yesterday ...resulting in a 1.3G text file. What is the best way to provide that to you?

If it's the same format as you gave @cvvergara last time, it should work. I think she was able to make sense of the part obfuscated email addresses. I'll leave it up to her to respond.

Notes:

• I tend to prefer singular (project,user) rather than plural (projects, users)
• We may of already lost any hope of consistency, can categories be renamed?

Yes as mentioned categories can be renamed easily cause internally discourse just works on category id numbers. In fact I think Vicky has to in order to load the mailing lists, put them on the root and then rename them and move them down the category hierarchy where they belong cause the mbox importer uses the mbox to arrive at the name.

However if you start with one name and decide to change later, I think we might need to put redirects in (which I recall there being a page for that within discourse) so that if people were navigating via the UI, they will be redirected to the new location.

[cvvergara]

As I write I forwarded the mail that contains the way I got the information for the test on staging.discourse

The load happens at the root, then the category needs to be moved to the final destination.

Steps: Create the geoserver category

• logo (send me a link for the logo to use, I can use the logo on staging)
• description of the category
• security: everyone can: see (no Reply no create)

Create geoserver-user category

• Jody and Andrea as owners

The "user" category will be created on the root when importing

• Once its created to avoid mishaps
  • security: everyone can: See (no Reply no create)

Move "user" under geoserver category

• logo (send me a link for the logo to use if different from the above)
• description of the category
• security:
  • everyone can: See (no Reply no create) or can See, Reply (no Create)
  • geoserver-user can See, Reply, Create

About the 3 levels, its new! I could not do it the last time I tried. From other tickets, grouping categories like "Local Chapters", "Initiatives", "GeoInquietos" have been created.

We are still the "few" in discourse.

So, lets focus on this ticket about GeoServer here, and open a new ticket about reorganizing the categories. That one can take care of the consequences of the changes (like redirection).

PS. Personally I like to see pgRouting on the root

[jive]

branding is here: ​https://github.com/geoserver/geoserver/wiki/GeoServer-Branding

[robe]

Replying to cvvergara:

The "user" category will be created on the root when importing

• Once its created to avoid mishaps
  • security: everyone can: See (no Reply no create)

For clarification, assume you mean:

The "geoserver-user" category will be created on the root when importing

Move "user" under geoserver category

Move "geoserver-user" category under geoserver category and rename to "user"

About the 3 levels, its new! I could not do it the last time I tried.

We have some already You'll see that SAC and OSGeoLive categories are now 3 levels down:

PS. Personally I like to see pgRouting on the root

Yah I do too, and if I can ever convince PostGIS and GEOS PSC to move, I'd want to see them on the root too.

I also think that if QGIS moves down one level to projects it's going to crowd out the other projects, cause QGIS has developer, users, various language specific categories, various extension specific categories. Heck they might even want 3 levels from QGIS (which would make it 4 levels, too hard to manage).

Which means to be fair to the other projects, if QGIS is at the top other projects need to be allowed to be at the top too.

But I think having 3 levels down for our internal committees like SAC, Board, Marketing makes great sense.

[jive]

Can we move discussion of top-level category out of this ticket ​https://discourse.osgeo.org/t/how-would-you-like-to-add-new-categories-for-projects-or-committees/28330/18 (or just make a decision as SAC admins please).

For this activity:

1. geoserver and geoserver/user (and we can move it later if needed)

2. mbox export is Herę: ​https://www.dropbox.com/scl/fi/wvsl0hff7w4pe4dk44b4e/geoserver-users.txt.zip?rlkey=aagi38pzsplwznhc6fxogmtfu&st=qs3q455z&dl=0

3. peterafrigis if you can share your sed clean up recipe it would be useful for this and other migrations to discourse

4. branding is here: ​​https://github.com/geoserver/geoserver/wiki/GeoServer-Branding

Is anything else needed from me? I would like to make this transition in the next two weeks (before I go on vacation). This would give the community most of August to transition before shutting down email alongside the GeoServer 2.24.0 release (when more users will notice something is happening).

[peterafrigis]

cat geoserver-users.mbox | sed 's/@\.\.\.[0-9]\{1,6\}\.\.\./@anonymised.com/g' > geoserver-users-anon.mbox

This simply changes the anonymised email addresses @...\d+... ( e.g. andrea.aime@...84...) into a valid address, and it solved the import problem.