updated: 2024-01-31 Wed 00:00

Newsgroup moderation tool developers guide

Table of Contents


1. Overview of moderated newsgroup

User post to a moderated newsgroup
|
|
v
------> ISC ---> look up moderator(s) email ----> Mail Server
							|
							v
moderator verify post  <---------------------------------
|
v
---> add "Approved" header and remove unnecessary headers
							|
							v							   
USENET server (authorized the moderator to post with approved header)

2. Moderation without server

You may use the following script to download news post from your mail provider.

#!/usr/bin/perl

#    This is a program to download your mails to your local machine using
#    IMAP protocol.
#    Copyright (C) 2024 Salahuddin <salahuddin@member.fsf.org>
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <https://www.gnu.org/licenses/>.

use strict;
use warnings;
use Net::IMAP::Client;
use Data::Dump qw(dump);

my $message_dir = 'YOU_USENET_MESSAGE_DIR';
my $noreply_mail = 'no-reply@your_domain';
my $send_rec_ack = 1;

sub parseFromAndSubject {
    my $message       = shift;
    my $from          = '';
    my $subject       = '';
    my @message_array = split(/\n/, $message);

    foreach(@message_array) {
	# first empty line
	if(/^$/) {
	    last;	    
	}
	if( /^From: / ) {
	    $from = $_;
	    $from=~ s/^From: //i;
	} elsif( /^Subject: / ) {
	    $subject = $_;
	    $subject =~ s/^Subject: //i;
	}
    }
    # trim
    $from =~ s/^\s+|\s+$//g;
    $subject =~ s/^\s+|\s+$//g;
    return ($from, $subject);
}


sub sendReceiveAck {
    my ($from, $subject) = @_;
    # configure your smart host
    system('echo "Submission received." | mail -s "Re: ' .  $subject . '" -r ' . $noreply_mail . ' \'' . $from . '\'');
}

my $imap = Net::IMAP::Client->new( 
    server => 'YOUR_MAIL_PROVIDER_HOST',
    user   => 'YOUR_MAIL_LOGIN',
    pass   => 'YOUR_MAIL_PASS',
    ssl    => 1,                               # (use SSL? default no)
    ssl_verify_peer => 1,                      # (use ca to verify server, default yes)
    #ssl_ca_file => '/etc/ssl/certs/certa.pm', # (CA file used for verify server) or
    #ssl_ca_path => '/etc/ssl/certs/',         # (CA path used for SSL)
    port   => 993                              # (but defaults are sane) 
) or die "Could not connect to IMAP server";

$imap->login or die('Login failed: ' . $imap->last_error);

my $dirctory_exists = 0;
my @folders = $imap->folders;
foreach(@folders) {
    if ($_ eq 'INBOX.PROCESSED') {
	$dirctory_exists = 1;
    }
}
if ($dirctory_exists eq 0) {
    $imap->create_folder('INBOX.PROCESSED');
}

# select folder
$imap->select('INBOX');

my $inbox = $imap->status('INBOX');

if ($inbox->{UNSEEN} >= 0) {
    # fetch all message ids (as array reference)
    my $messages = $imap->search('ALL');    
    foreach(@{$messages}) {
	my $message_id = $_;
	my $data = $imap->get_rfc822_body($message_id);

	my $message_number = time . $$;
	my $filename = $message_dir . $message_number . "_" . $message_id;

	open(FH, '>', $filename) or die $!;
	print FH $$data;
	close(FH);

	$imap->add_flags($message_id, '\\Seen');
	$imap->copy($message_id, 'INBOX.PROCESSED');
	$imap->add_flags($message_id, '\\Deleted');

	my $incoming_message = $$data;

	if ($send_rec_ack eq 1) {
	    my ($from, $subject) = parseFromAndSubject($incoming_message);
	    sendReceiveAck($from, $subject);
	}
    } 
}

$imap->expunge;

2.1. Process messages

Please check the following sections for more information.

Message formatting
add custom headers
add Approved header(s)

2.2. Install tinews.pl

You may download tinews.pl from tin.org HTTP tin.org FTP

or in Debian based distribution:

# apt-get install tin
# gzip -d /usr/share/doc/tin/tools/tinews.pl.gz

Configure tinews.pl (provide - hostname, login, and password)

tinews.pl looks for a config in (first match counts)
	$XDG_CONFIG_HOME/tinewsrc
	$HOME/.config/tinewsrc
	$HOME/.tinewsrc
as "option=value" configuration pairs, last match counts and only
"value" is case sensitive. Available config options are those listed
via tinews.pl --help --verbose. Env.-vars and cmd-line options do override
the config settings. See "tinews.pl --help --verbose" and
"pod2man tinews.pl | man -l -"

Submit the message:

tinews.pl < post_to_send.txt

You may setup a USENET server for testing.

3. Moderation with server

3.1. local DNS server

Setup a local DNS server (bind9 or other), and add DNS entries of your local virtual servers.

usenet-server.home.lab - USENET server
usenet-client.home.lab - client will post news to USENET server via rnews (or inews)

note: Please make sure your virtual servers are using your DNS server.

3.2. Setup USENET server

usenet-server.home.lab

setup inn2 from source or in Debian based distribution.

# apt-get install inn2

edit /etc/news/inn.conf

Organization:                "Demo InterNetNews site" 
pathhost:                    usenet-server.home.lab
domain:                      usenet-server.home.lab

create a group

$ /usr/sbin/ctlinnd newgroup demo.comments

edit /var/lib/news/newsgroups

demo.comments   Demo comments.

edit /etc/news/readers.conf

auth "demo" {
    hosts: "*"
    default: "<demo>"
}

access "demo" {
    users: "<demo>"
    newsgroups: "demo.comments"
    access: RPA
}

3.2.1. enable incoming

edit /etc/news/incoming.conf

peer other {
  hostname:         "HOST_OR_IP_OF_YOUR_CLIENT_WITH_RNEWS"
}

Replace HOST_OR_IP_OF_YOUR_CLIENT_WITH_RNEWS with your usenet-client.home.lab name or IP.
It will allow usenet-client.home.lab to post news to usenet-server.home.lab via rnews.

restart

# systemctl stop inn2
# systemctl start inn2

3.2.2. Server logs

# journalctl -u inn2.service

3.3. Setup USENET client

usenet-client.home.lab

You may test your USENET server using a USENET client or telnet.

telnet usenet-server.home.lab 119
MODE READER
POST
//It will return you a Message-ID.
// add Message-ID in your manual post
Path: hi!not-for-mail
From: Home <demo@demo>
Newsgroups: demo.comments
Date: Fri, 5 Jan 2024 21:29:04 -0000 (UTC)
Subject: test
Message-ID: <unb5kg$17e$1@usernet-server.dynamic.lab>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

test
.
QUIT

Now, your message should appear in demo.comments news group.

3.3.1. Install rnews

Install rnews from source or in Debian based distribution.

# apt-get install inn2-inews

3.3.2. Test rnews

create test_message.txt

Path: hi!not-for-mail
From: home <demo@demo>
Newsgroups: demo.comments
Date: Fri, 5 Jan 2024 21:29:04 -0000 (UTC)
Subject: test2
Message-ID: <unb5kg$19e$1@usenet-client.dynamic.lab>

test2

post message

/usr/bin/rnews -h usenet-server.home.lab -P 119 -S usenet-server.home.lab < test_message.txt  

verify the message post status using your usenet reader.

3.4. Setup local Mail Server

usenet-client.home.lab

Setup local Mail Server for moderation (Sendmail, Exim, others).

warning: by default mail server disable email exchange for local IP series.
If you have local IP series 192.168.0.x, please make sure you allow this series for mail exchange.

Example: in Exim /etc/exim4/exim4.conf.template

dnslookup:
  debug_print = "R: dnslookup for $local_part@$domain"
  driver = dnslookup
  domains = ! +local_domains
  transport = remote_smtp
  same_domain_copy_routing = yes
  # ignore private rfc1918 and APIPA addresses
  ignore_target_hosts = 0.0.0.0 : 127.0.0.0/8 : 192.168.0.0/16 :\
			172.16.0.0/12 : 10.0.0.0/8 : 169.254.0.0/16 :\
			255.255.255.255
  no_more
.endif

remove 192.168.0.0/16 to allow local mail exchange.

3.4.1. Setup procmail

add $HOME/.procmailrc
------------------------
DEFAULT=$HOME/Maildir/
MAILDIR=$HOME/Maildir/
LOCKFILE=$HOME/.lockmail
LOGFILE=$MAILDIR/procmail.log
LOGABSTRACT=yes
VERBOSE=yes

:0c
| ../YOUR_INCOMING_MAIL_HANDLER

"YOUR_INCOMING_MAIL_HANDLER" will read mail via STDIN and save for further processing. You need to notify the sender an acknowledgment of the submission (automatically or manually).

3.4.2. Message formatting

If your comment or any header line is too long, it should be reformatted to next line with space(s).
Example:

HEADER: your long
comment for moderation

This will fail while posting in newsgroup.

You need to either make the comment shorter or add space(s) as prefix after header line

HEADER: your long
   comment for moderation

3.4.3. add custom headers

add your custom headers with the original submission for approval

X-SCRM-Policy: https://example.com/
X-SCRM-Info-1: Send submissions to             demo@example.com
X-SCRM-Info-2: Send technical complaints to    demo-admin@example.com
X-SCRM-Info-3: Send complaints about policy to demo-board@example.com
X-Comment: moderators do not necessarily agree or disagree with this article.
X-Robomod: YOUR_MODERATION, demo@example.com
X-Moderation-1: YOUR moderation tool
X-Moderation-2: See https://example.com/

3.4.4. add Approved header(s)

Remove unnecessary header(s) and add approved header before submission.

formail -f	  -I Path:                                    \
		  -I X-Moderate-For:                          \
		  -I Return-Path:                             \
		  -I X-Mailer:                                \
		  -I "Date:"                                  \
		  -I "X-400-Received:"                        \
		  -I Received:                                \
		  -I "From "                                  \
		  -a "Approved: YOUR_APPROVAL_EMAIL"          \
		  -I Lines:                                   \
		  -I Cc:                                      \
		  -I Status:

> post_to_send.txt

3.4.5. Test Post

Now you can post to your local newsgroup server via rnews.

/usr/bin/rnews -h usenet-server.home.lab -P 119 -S usenet-server.home.lab < post_to_send.txt

or you may use inews

NNTPSERVER=usenet-server.dynamic.lab inews -h -P -p 119 < post_to_send.txt

NNTPSERVER environment variable needed for inews to submit news to a remote server.

After successful posting, move the submitted mail submission in a different directory to avoid re-posting. You may automate the process by implementing scripts and web-interface.

If you plan to use it in a production/external USENET server, you need to add your login password in /etc/news/passwd.nntp

EXTERNAL_USENET_SERVER_HOST:YOUR_LOGIN:YOUR_PASSWORD

Use port with SSL/TLS (563) instead of insecure port 119.

3.5. Simulate newsgroup post

You may use usenet-client.home.lab for testing.
Send test message with usenet header (using -a).

You need to set -r –return-address=ADDRESS explicitly. Otherwise, it will send mail as SENDER_USER@localhost and exim will not deliver submission response to this account.

Destination "USER" is the account with procmail configured for moderation.

$ echo "Test message." | mail -s "Test" -r SENDER_USER@EMAILHOST 'USENETMOD <USER@YOUR_DOMAIN>' -a 'Newsgroups: YOUR_NEWSGROUP'

3.6. Develop incoming mail handler

Develop YOUR_INCOMING_MAIL_HANDLER to moderate newsgroup posts.

4. Credits

Thank you to Julien and Urs for their feedback in news.software.nntp