Can Solaris 11 Sendmail / SASL handle SMTP AUTH

I thought I would finally get SMTP AUTH working on my Solaris 11 servers, so that I could then setup my phone to send e-mail correctly.

So can you do it with default Solaris 11 software?
If you can't wait the answer is NO! but here is what I tried just in case someone can solve the problem.

• First Solaris 11 comes with Sendmail and SASL, but worst luck Sendmail has not been built with SASL support (you see the following when you edit sendmail.cf. see later).

# sudo /usr/lib/sendmail -d0 -bt < /dev/null
Version 8.14.5+Sun
Compiled with: DNSMAP LDAPMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8
MIME8TO7 NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB NIS
PIPELINING SCANF STARTTLS TCPWRAPPERS USERDB USE_LDAP_INIT
XDEBUG
Warning: Option: AuthMechanisms requires SASL support (-DSASL)
Warning: Option: AuthOptions requires SASL support (-DSASL)

• Lets try compiling sendmail to see if Solaris SASL can work!
• Install the packages which we requite and download sendmail  8.14.5

# sudo pkg install pkg:/developer/build/make system/header developer/gcc-3 text/gnu-grep database/berkeleydb-5

# echo $PATH
/usr/sbin:/usr/bin
# export PATH=$PATH:/usr/gnu/bin:/usr/sfw/bin

• Stop existing sendmail processes.

# sudo svcadm disable svc:/network/smtp:sendmail
# sudo svcadm disable svc:/network/sendmail-client:default

# gzip -dc sendmail.8.14.5.tar.gz | tar xf -
# cd sendmail-8.14.5

• Set up sendmail FEATURES (SSL, SASL, DB)

# cat > devtools/Site/site.config.m4
dnl ### Changes for STARTTLS support
APPENDDEF(`confENVDEF',`-DSTARTTLS')
APPENDDEF(`confLIBS', `-lssl -lcrypto')
APPENDDEF(`confLIBDIRS', `-L/usr/lib -R/usr/li')
APPENDDEF(`confINCDIRS', `-I/usr/include/openssl')

dnl ### SASL support
APPENDDEF(`confENVDEF', `-DSASL')
APPENDDEF(`conf_sendmail_LIBS', `-lsasl')
APPENDDEF(`confINCDIRS', `-I/usr/include/sasl')

dnl ### Berkley DB support
APPENDDEF(`confENVDEF', `-DNEWDB')
APPENDDEF(`confINCDIRS', `-I/usr/include')
APPENDDEF(`confLIBDIRS', `-L/usr/lib')
EOF

•  Build Sendmail

# ./Build -c
/* It should build but ignore the gtroff:error at the end */
# sudo mv /usr/lib/sendmail /usr/lib/sendmail_orig
# sudo cp obj.SunOS.5.11.i86pc/sendmail/sendmail /usr/lib/sendmail
# sudo sudo chgrp smmsp /usr/lib/sendmail
# sudo sudo chmod g+s,u-ws /usr/lib/sendmail

• Now a Vendor and SASL support to sendmail.cf file (I hope you know what you are doing here)
• Edit sendmil.mc add add the extra lines:

define(`VENDOR_NAME', `Berkeley')
define(`confAUTH_OPTIONS', `A')dnl
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl

# cd /etc/mail/cf/cf
# cat sendmail.mc
divert(0)dnl
VERSIONID(`sendmail.mc (Sun)')
define(`VENDOR_NAME', `Berkeley')
OSTYPE(`solaris11')dnl
DOMAIN(`solaris-generic')dnl
define(`confAUTH_OPTIONS', `A')dnl
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl define(`confFALLBACK_SMARTHOST', `mailhost$?m.$m$.')dnl
MAILER(`local')dnl
MAILER(`smtp')dnl

LOCAL_NET_CONFIG
R$* < @ $* .$m. > $* $#esmtp $@ $2.$m $: $1 < @ $2.$m. > $3
#
# sudo make sendmail.cf
# sudo cp sendmail.cf ../../sendmail.cf

# sudo /usr/lib/sendmail -bt -d0 < /dev/null
Version 8.14.5
Compiled with: DNSMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7
NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB PIPELINING
SASLv2 SCANF STARTTLS USERDB XDEBUG

• Now lets set up SASL plugin via a Sendmail.conf

# cat > /etc/sasl/Sendmail.conf
# pw_check: shadow

• I have tried the following in the above with no luck: shadow,pam,auxprop, but none seem to check password information.
• Testing, but first we need the encrypted username and password:

# perl -MMIME::Base64 -e 'print encode_base64("\000MyUser\000MyPassword")'
AE15VXNlcgBNeVBhc3N3b3Jk

• Test the sendmail part:

# sudo /usr/lib/sendmail -bv  -O LogLevel=14 -bs -Am
220 mailer5.dcs.bbk.ac.uk ESMTP Sendmail 8.14.5/8.14.5; Tue, 31 Jan 2012 09:52:24 GMT
EHLO localhost
250-mailer5.dcs.bbk.ac.uk Hello root@localhost, pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-EXPN
250-VERB
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5 PLAIN
250-DELIVERBY
250 HELP
AUTH PLAIN AE15VXNlcgBNeVBhc3N3b3Jk
535 5.7.0 authentication failed
Jan 31 09:52:35 mailer5 sendmail[5928]: Password verification failed
Jan 31 09:52:35 mailer5 sendmail[5928]: q0V9qOd1005928: AUTH failure (PLAIN): user not found (-20) SASL(-13): user not found: Password verification failed, relay=root@localhost
quit
221 2.0.0 mailer5.dcs.bbk.ac.uk closing connection

• So no luck, if you can spot anything I am doing wrong please let me know.
• I will post shortly how to get over the above.

refernces:
- Guía Instalación de Sendmail con SMTP-AUTH y (tema #2688)
- Sendmail-SMTP-AUTH-TLS-Howto
- OpenSolaris Sendmail Auth