UKOUG Technology Conference

UKOUG Technology Conference
UKOUG Technology Conference 2018 Call for Papers

Tuesday, October 10, 2017

Setting up Oracle Solaris iSCSI Targets for Multipathed

After wasting 2 weeks on getting this to work I thought I would post a step-by-step guide.

I have been using iscsi targets on Oracle Solaris for years, but only with the installation of a new storage system with two VLANS (multipathed) for failover I needed Solaris to recognise the different VLANs in case of a interface failure.

This wasn't easy, since the Oracle Solaris documentation  is very unclear about it. It has all the information there but finding it and understanding it is near impossible.

The following steps show you the procedure to connect and a single iscsi target accessible via two interfaces for failover.


Recap setup network

# dladm create-vlan -l trunk0 -v 2310 vlan2310
# ipadm create-ip vlan2310
# ipadm create-addr -T static -a 10.63.10.62/24 vlan2310/v4

# dladm create-vlan -l trunk0 -v 2311 vlan2311
# ipadm create-ip vlan2311
# ipadm create-addr -T static -a 10.63.11.62/24 vlan2311/v4
# ipadm
NAME              CLASS/TYPE STATE  UNDER      ADDR

..
vlan2310          ip         ok     --         --
   vlan2310/v4    static     ok     --         10.63.10.62/24
vlan2311          ip         ok     --         --
   vlan2311/v4    static     ok     --         10.63.11.62/24

Setup up MPxIO driver

Copy the system default value to /etc/driver/drv and edit:
(ref: http://docs.oracle.com/cd/E53394_01/html/E54792/iscsi-14.html#scrolltoc )
# cp -ip /kernel/drv/scsi_vhci.conf /etc/driver/drv/scsi_vhci.conf

# diff /etc/driver/drv/scsi_vhci.conf /kernel/drv//scsi_vhci.conf
77,79d76
< mpxio-disable="no";
< scsi-vhci-failover-override =
<       "SCST_BIO",     "f_sym";

  • mpxio-disable - turn MPxIO on (Oracle Solaris Storage Multipathing)
  • scsi-vhci-failover-override - specify third party disks which will be enabled for MPxIO. Be careful with format: Vendor (exactly 8 characters in length) & Product ID ( Max of 16 characters).
i.e.
#    "012345670123456789012345",    "failover-module-name" or "NONE"
#    "|-VID--||-----PID------|",
# scsi-vhci-failover-override =
#    "STK     FLEXLINE 400",        "f_asym_lsi",
#    "SUN     T4",            "f_tpgs",
#    "CME     XIRTEMMYS",        "NONE";

In our system I am setting it up for all disks so I can leave Product (PID) empty and the Vendor disk is called SCST_BIO. Obtain this from format
i.e.
format < /dev/null
  3. c5t2d0 <SCST_BIO-j8A1pLRSyVh3RHgf- 220-48.00GB>
    /iscsi/disk@0001iqn.2017-07.uk.ac.bbk.dcs%3Adss.orasrv0001,0
  4. c5t3d0 <SCST_BIO-j8A1pLRSyVh3RHgf- 220-48.00GB>
    /iscsi/disk@0000iqn.2017-07.uk.ac.bbk.dcs%3Adss.orasrv0001,0

  5. c5t10d0 <SCST_BIO-auhBVacjG7eI8Sr0- 220-180.00GB>
    /iscsi/disk@0000iqn.2017-10.uk.ac.bbk.dcs%3Adss.hermes0001,0

  x. c14t9d0 <|-VID--|-|-----PID------|- 220-500.00GB>
    /iscsi/disk@0000iqn.2017-02.uk.ac.bbk.dcs%3Adss.solari0001,0

Reboot System 

Add iSCSI targets

After system has been rebooted you are ready to add your iscsi targets, if you have already have them install then skip some of the following but it is always worth double checking.
# svcadm enable network/iscsi/initiator
# svcs iscsi/initiator
STATE          STIME    FMRI
online         14:55:38 svc:/network/iscsi/initiator:default

# iscsiadm modify discovery --static enable
# iscsiadm list discovery
Discovery:
        Static: enabled
        Send Targets: disabled
        iSNS: disabled
/* The following is to make sure you configure the number of session to one!
# iscsiadm modify initiator-node -c 1

# iscsiadm add static-config iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv,10.63.10.222:3260
# iscsiadm add static-config iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv,10.63.11.222:3260
# format < /dev/null
       3. c0t6A384131704C5253d0 <SCST_BIO-j8A1pLRSyVh3RHgf- 220-48.00GB>
          /scsi_vhci/ssd@g6a384131704c5253
You will see that the disk name has changed and the path to the disk has also changed to /scsi_vhci/
# cat /var/adm/messages | egrep "orasrv|6A384131704C5253"
... /scsi_vhci/ssd@g6a384131704c5253 (ssd0) multipath status: degraded: path 3 iscsi0/ssd@0001iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv0001,0 is online
... NOTICE: iscsi session(38) iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv online
...  /scsi_vhci/ssd@g6a384131704c5253 (ssd0) multipath status: optimal: path 4 iscsi0/ssd@0000iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv0001,0 is online: Load balancing: round-robin

If you get something like the above then you are ready to set up the multipathing failover. At this stage if one of your paths (VLANs) went down you would lose the disk.
# iscsiadm modify target-param -c 2 iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv
The above sets the number of sessions/paths this disk (iscsi target) has. VERY IMPORTANT you do this command after all the iscsi targets have been added

# iscsiadm list target -v -S iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv |grep Peer
      IP address (Peer): 10.63.11.222:3260   << See different IPS
      IP address (Peer): 10.63.10.222:3260   << failover


# iscsiadm list target -v -S iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv
Target: iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv
        Alias: -
        TPGT: 1
        ISID: 4000002a0001
        Connections: 1
                CID: 1
                  IP address (Local): 10.63.11.62:57708
                  IP address (Peer): 10.63.11.222:3260
                  Transport Type: socket
                  Discovery Method: Static
                  Login Parameters (Negotiated):
                        Data Sequence In Order: yes
                        Data PDU In Order: yes
                        Default Time To Retain: 0
                        Default Time To Wait: 0
                        Error Recovery Level: 0
                        First Burst Length: 65536
                        Immediate Data: yes
                        Initial Ready To Transfer (R2T): yes
                        Max Burst Length: 262144
                        Max Outstanding R2T: 1
                        Max Receive Data Segment Length: 1048576
                        Max Connections: 1
                        Header Digest: NONE
                        Data Digest: NONE

        LUN: 0
             Vendor:  SCST_BIO
             Product: j8A1pLRSyVh3RHgf
             OS Device Name: /dev/rdsk/c0t6A384131704C5253d0s2


Target: iqn.2017-07.uk.ac.bbk.dcs:dss.orasrv
        Alias: -
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1
                CID: 0
                  IP address (Local): 10.63.10.62:46877
                  IP address (Peer): 10.63.10.222:3260
                  Transport Type: socket
                  Discovery Method: Static
                  Login Parameters (Negotiated):
                        Data Sequence In Order: yes
                        Data PDU In Order: yes
                        Default Time To Retain: 0
                        Default Time To Wait: 0
                        Error Recovery Level: 0
                        First Burst Length: 65536
                        Immediate Data: yes
                        Initial Ready To Transfer (R2T): yes
                        Max Burst Length: 262144
                        Max Outstanding R2T: 1
                        Max Receive Data Segment Length: 1048576
                        Max Connections: 1
                        Header Digest: NONE
                        Data Digest: NONE

        LUN: 0
             Vendor:  SCST_BIO
             Product: j8A1pLRSyVh3RHgf
             OS Device Name: /dev/rdsk/c0t6A384131704C5253d0s2


you are now ready to uses these disks.
Good luck!

No comments: