mysql:version_80 Service in Maintenance State in Oracle Solaris

I recently tried upgradeing MySQL 5.7 to MySQL 8.0 on SPARC kernel zone (KZ) but it failed. Here is some information which may help people in the future. 

So, I run our webserver www.dcs.bbk.ac.uk in a Solaris Kernel Zone which using a small MySQL 5.7 database, so after the upgraded failed I moved it to a test system and here is the steps and output which came to the possible workarounds.

System: 

S7-2 - 2 x 600GB 10K SAS-3 disks (rpool mirror) & 128GB Memory 

        KZ (delphi) -> 32GB & 64 ncpus   => MySQL 8.0 failed 

        NGZ () -> 24GB & all ncpus          => MySQL 8.0 works 

 

S3-2 - 2 x 600GB 15K SAS-1 disks (rpool mirror) & 192GB Memory

        KZ (rhea) -> 16GB & 16 ncpus.       => MySQL 8.0 works 

 

On a new clean install of a KZ(delphi)

$ cat /var/mysql/8.0/data/delphi.err
2023-10-27T11:02:55.169515Z 0 [System] [MY-013169] [Server] /usr/mysql/8.0/bin/mysqld (mysqld 8.0.33) initializing of server in progress as process 689
2023-10-27T11:02:55.304606Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-10-27T11:03:07.199840Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-10-27T11:03:28.644784Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: f+-KsTq?J6mx
2023-10-27T11:03:52.926501Z mysqld_safe Logging to '/var/mysql/8.0/data/delphi.err'.
2023-10-27T11:03:53.003749Z mysqld_safe Starting mysqld daemon with databases from /var/mysql/8.0/data
2023-10-27T11:03:53.293398Z 0 [System] [MY-010116] [Server] /usr/mysql/8.0/bin/mysqld (mysqld 8.0.33) starting as process 819
2023-10-27T11:03:53.470902Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-10-27T11:03:53.524617Z 1 [ERROR] [MY-012960] [InnoDB] Cannot create redo log files because data files are corrupt or the database was not shut down cleanly after creating the data files.
2023-10-27T11:03:53.527959Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2023-10-27T11:03:53.528303Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-10-27T11:03:53.528380Z 0 [ERROR] [MY-010119] [Server] Aborting

So, ran truss on the initialize process only which worked and gave following figures: 

delphi$ truss -vall -dDE -feao /tmp/trussKZ.out /usr/mysql/8.0/bin/mysqld --initialize '--user=mysql' '--datadir=/var/mysql/8.0/data' '--log-error=/var/mysql/8.0/data/delphi.err' 

$ grep lwp_park trussKZ.out | wc -l
62530
$ grep sleeping trussKZ.out|wc -l
159
$ cat trussKZ.out | grep lwp_park | grep sleep|wc -l
132
$ cat trussKZ.out |grep lwp_park |grep ETIME | wc -l
50723
$ cat -n trussKZ.out | grep "pwrite" | nawk '{x=$4+x} END {print x}'
41.4928
$ tail -1 trussKZ.out
1984:   66.398589     0.006305     0.000000    _exit(0)

I did the same on the other platforms which some interesting numbers:

Zone type	lwp_park	sleeping	lwp_park and sleep	lwp_park and ETIMW	pwrite	time
S7-2 SPARC KZ on rpool	62530	159	132	50723	41.49	66.398
S7-2 SPARC NGZ on rpool	48299	67	56	36451	23.32	35.636
X3-2 x86 KZ on rpool	28832	138	110	20743	30.29	66.14
S7-2 SPARC KZ on ISCSI	23722	45	38	13669	6.49	13.83

As you can see running on a local disk is poor compared to a network device (we know that).

As you can see the SPARC KZ on rpool (local/system disk) is slower than every other platform zone.

To fix, in my case I still wanted to use a local KZ so I need to increase tineouts of the service.

$ svccfg -s svc:/application/database/mysql:version_80 setprop start/timeout_seconds = count: 120
$ svccfg -s svc:/application/database/mysql:version_80 setprop stop/timeout_seconds = count: 120
$ svcadm refresh svc:/application/database/mysql:version_80

$ svcprop svc:/application/database/mysql:version_80 | grep timeout_seconds
start/timeout_seconds count 120
stop/timeout_seconds count 120

$ svcadm enable mysql
$ svcs mysql
STATE STIME FMRI
online 2023-11-02T16:06:20 svc:/application/database/mysql:version_80

The conclustion is 

1. MySQL 8.0 out of the box is much worst performace that MySQL 5.7 and that is a subject that is covered a lot else where.
   
2. Always run zones on external storage and never on system disk. 
1. I did not test the zone on a free local disk (unused ZPOOL)
   
3. Even though X3-2 had a similar elapse time as the S7-2, mysql service did start without problems, but I guess that is down the lwp_park and pwrite being much lower