0 Members and 1 Guest are viewing this topic.

How I bypassed the SPAM folder
« on: March 12, 2018, 06:07:22 PM »
It is not impossible to keep sdateEvo messages out of the Spam folder. Currently if a mailtype is set to 'mail', it seems like every message gets sent to the SPAM folder. When the email is received there is a big 'unverified' question mark attached to each email (see 'email-message.jpg'). Gmail marked this as unverified/spam because there was a problem with the 'Return-Path' value in the email header and as a result the SPF value was marked as 'NEUTRAL' or some other non-passing value, which means it gets sent to the SPAM folder - see 'original-message.jpg')

There is actually a bug within osdateEvo, probably carried over from an earlier version. The problem is that 'Return-path' is never set. The function exists (/libs/sendMimeMail.php setReturnPath line 362) but it is never called by any other piece of code. In order for any mail system, like Gmail or Outlook, to categorize the email as not spam, the 'return-path' value must be the same as the 'from' email in the header - there is actually more to this but that is the quick explanation.

The SPF must be 'PASS' in order to be delivered to the Inbox (see email-original-after.jpg).

Here is the fix that I applied and was able to get all registration emails (and all other types) out of the spam folder and into the inbox.


/includes/internal/Functions.php (near line 539) in function sendMail()
Change
Code: [Select]
        $result = $mail->send(array( stripslashes($row['email'])), $mail_type ) ;

To
Code: [Select]
$result = $mail->send(array( stripslashes($row['email'])), stripslashes($row['from']), $mail_type ) ;


/includes/internal/Functions.php (near line 448) in function mailSender()
Change this:
Code: [Select]
$row = array();
$row['hdr_from'] =  strstr($hdr_from, '<') ? $hdr_from : sprintf("%s <%s>", $config['site_name'], $hdr_from);

Should be this:
Code: [Select]
$row = array();
$row['from'] = $hdr_from; 
$row['hdr_from'] =  strstr($hdr_from, '<') ? $hdr_from : sprintf("%s <%s>", $config['site_name'], $hdr_from);


/libs/sendMimeMail.php line 739 in function send()
Change
Code: [Select]
    public function send($recipients, $type='smtp')

To
Code: [Select]
    // need to pass the clean version of the from address i.e. no firstname lastname
    public function send($recipients, $from, $type='smtp')


/libs/sendMimeMail.php line 765 in function send()
Change
Code: [Select]
                if (!empty($this->return_path)) {
                    $result = @mail($to, $subject, $this->output, implode(CRLF, $headers), '-f' . $this->return_path);
                } else {
                    $result = @mail($to, $subject, $this->output, implode(CRLF, $headers));
                }

To
Code: [Select]
                // the -f  basically says to use the $from address as the return path
                $result = @mail($to, $subject, $this->output, implode(CRLF, $headers), '-f' . $from);

One word of caution,
The $from address to be used as the return-path needs to be in the form of 'example@domain.com' and not 'Firstname Lastname <example@domain.com>' or this will also fail. However, the from address to be used as the hdr_from can contain the name and the system does currently format this into the second form.  For the most part all the mailing functions are correctly formatting the 'from' address except for one place that I found.

/admin/sendletter.php (near line 253)
Change
Code: [Select]
     $from = '"' . $_SESSION['fromname'] . '" <'. $_SESSION['from'] .'>';

To
Code: [Select]
     $from = $_SESSION['from'];


Passing the SPF was what I needed in order to get the messages out of SPAM but I also added a DKIM authentication and well as DMARC. This prevents spoofing i.e. someone sending mail pretending to be from your domain by actually using your domain name. Here were some helpful resources:
https://support.google.com/a/answer/33786 (re: SPF)
https://glockapps.com/blog/email-authentication/
https://support.google.com/a/answer/174124?hl=en (DKIM)
https://support.google.com/a/answer/174126?hl=en (Note: my MX records are at Google so I generated my DKIM records there. You should generate them at whoever hosts your MX DNS)

Good luck.
Please backup your code before trying - and use at your own risk :)

« Last Edit: March 13, 2018, 02:53:31 AM by Pharg »
PHPStorm, MAMP Pro, Sequel Pro
Currently converting my 2.1.6 site to osDateEvo 1.2

*

Pharg

  • *****
  • 3,107
    • OsDateFourm
  • osDate Version: osDateEvo v1.3
Re: How I bypassed the SPAM folder
« Reply #1 on: March 12, 2018, 08:33:48 PM »
Hi arievaul,

Awesome work, thanks for the great share. nice work.  ;)
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.3 | PHP: 5.3.42 & PHP: 7.1.15 | MySQL: 5.5.35

*

Pharg

  • *****
  • 3,107
    • OsDateFourm
  • osDate Version: osDateEvo v1.3
Re: How I bypassed the SPAM folder
« Reply #2 on: March 13, 2018, 02:23:05 AM »
Hi all,

Myself and arievaul found a typo in the changes above and have tested and checked these changes and they
all work great now.

The typo is this:
Quote
/includes/internal/Functions.php (near line 448) in function mailSender()
Just after $row = array(); add the line below
Code: [Select]
$row = array();
$row['from'] = $hdr_from; //at this point $hdr_from is just the local-part@domain.com without firstname lastname

It Should be like this:
Code: [Select]
$row = array();
$row['from'] = $hdr_from;
$row['hdr_from'] =  strstr($hdr_from, '<') ? $hdr_from : sprintf("%s <%s>", $config['site_name'], $hdr_from);

I have also changed it in the first code so the mistake doesn't happen again :)
« Last Edit: March 13, 2018, 02:55:48 AM by Pharg »
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.3 | PHP: 5.3.42 & PHP: 7.1.15 | MySQL: 5.5.35

Re: How I bypassed the SPAM folder
« Reply #3 on: March 13, 2018, 03:00:59 AM »
Thanks Phill - looks like you have updated the original post too. I wanted to mention one more place that may need an update but this is untested as I don't use cron.

/cronjobs/send_osdatemails.php (line 55) we need to populate the $from value and we can get that from the hdr_from that was saved into the database record.

I placed the following code right before $ok = sendMail($mail);

Code: [Select]
                // create the $from address from hdr_from
$split = explode(' <', $mail['hdr_from']);
$name = $split[0];
$from = '<' . $split[1];
$mail['from'] = $from;

Just a warning though. I haven't had the chance to test this and PHP expert developers may want to re-write this with regex.
PHPStorm, MAMP Pro, Sequel Pro
Currently converting my 2.1.6 site to osDateEvo 1.2

*

Pharg

  • *****
  • 3,107
    • OsDateFourm
  • osDate Version: osDateEvo v1.3
Re: How I bypassed the SPAM folder
« Reply #4 on: March 13, 2018, 03:03:46 AM »
Hi arievaul,

Thanks for that, I also don't use cronjobs but I better put it in also.

Thanks heaps.
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.3 | PHP: 5.3.42 & PHP: 7.1.15 | MySQL: 5.5.35

Re: How I bypassed the SPAM folder
« Reply #5 on: March 13, 2018, 03:11:38 AM »
Just attaching image mentioned above (i.e. email-original-after.jpg). This is the email header of the new message that passed SPF, DKIM and DMARC.
PHPStorm, MAMP Pro, Sequel Pro
Currently converting my 2.1.6 site to osDateEvo 1.2