Nexenta 3.1.6-FP3/4.0.3 and vSphere 5.5

Systems check:
Nexenta 3.1.6-FP3
VMware ESXi 5.5.0
All LUNs are VMFS5 formatted

When attempting to upgrade to Nexenta 4.0.3 the other night I ran into an number of issues that ended with me carrying out a rollback to 3.1.6-FP3. A couple of notes from the problems I encountered;

1. As our log drive was a DDR card, we had to uninstall the device from the Pool as the driver had to be manually reinstalled once Nexenta had updated to 4.0.3.

2. After updating to 3.1.6-FP3, all of the mappings in SCSI Target Plus had been deleted so when I rescanned for storage it dropped all the connections.

3. 3.1.6-FP3 still caused me issues as none of our datastores (set up in 3.1.5) were visible despite the storage being mounted. I forced the datastores to be persistently mounted through the command line but obviously this was just a work-around rather than a solution. I did this by running the following commands;

esxcfg-volume -l (this generates a list of mounted storage on the host - I copied the UUID displayed for each LUN for the next step)
esxcfg-volume -M UUID (this persistantly mounted the datastore on the host)

These commands had to be carried out for each datastore on each host - not ideal!

4. After further investigation it appears that Nexenta have turned off Hardware Acceleration, not only in 4.0 but also in 3.1.6-FP3. I then connected to each host via the GUI and turned off the 3 settings related to this;

On the selected host go to the Configuration tab, then under Software go to Advanced Settings.
First go to DataMover and set both HardwareAcceleratedInit and HardwareAcceleratedMove to 0.
Then go to VMFS3 and set HardwareAcceleratedLocking to 0.

Changing these settings do not require a reboot.

Once this is done, you also need to turn off Hardware Acceleration on the LUNs themselves.

You only need to run this command on one one of the hosts that has the datastore attached. Running a rescan of datastores after making this change will update the datastore on all the hosts, allowing you to add the storage.

As per VMware's KB article HERE follow the steps below to disable ATSOnly on the LUN. Be aware that the command appears to be case sensitive - I typed ATSonly rather than ATSOnly and it failed to execute;

1. Connect to one of the hosts sharing the VMFS5 datastore with an SSH session. For more information, see
2. Run the following command:
   
   vmkfstools --configATSOnly 0 /vmfs/devices/disks/device-ID:Partition
   
   Where:
   
   device-ID is the NAA ID of the LUN on which the VMFS5 datastore was created.
   Partition is the partition number on which VMFS5 datastore was created. This is usually 1.
   
   For example:
   
   vmkfstools --configATSOnly 0 /vmfs/devices/disks/naa.6006016055711d00cef95e65664ee011:1
   
   Note: It is sufficient to run this command on one of the hosts sharing the VMFS5 datastore. Other hosts automatically recognize the change.
3. Run the following command to rescan for datastores:
   
   esxcli storage filesystem rescan
4. The VMFS5 datastore should now mount successfully.

Alternatively, rather than running the command at step 3, just rescan the datastores from within vSphere Client.

As a further note on disabling ATSOnly if you are using datastore heartbeating for HA, remember to turn this off for the datastore you are altering otherwise the disk will always show as being in use, even with it's VMs powered off.