Tuesday, November 22, 2016

Fix slow disk performance (vmw_ahci driver) in ESXi 6.5

Update (27.03.2017) - the problem still exists for vmw-ahci version:

vmw-ahci 1.0.0-34vmw.650.0.14.5146846 

Update (01.08.2017) - the latest ESXi 6.5 update 1 (build 5969303) seems to fix the performance problems. Read below on how to upgrade your hosts...



VMWare recently released it's VSphere line versioned as 6.5. I do not recommend to install it yet in a production environment - but it should be fine for your homelab.

Im reading several blog posts about disk performance problems with the new standard AHCI-driver (native vmw_ahci driver). 

I noticed the problems when I started to deploy a new VM from an existing OVA and also when I moved a VM from one datastore to another (Max. Disk-read/write performance 10 MB/s). Also chrome uploaded the OVA with a max. speed of 16 MBit/s.

Older ESXi hosts were using the legacy vmklinux drivers and in general the new native drivers should offer a better performance and lower overhead as those are build directly for ESXi.

You can check if your adapters are using the native VMWare driver in the GUI by going to STORAGE -> ADAPTERS:




If you are experiencing slow disk performance you can disable the new native driver module and reboot your hosts (enable SSH and log in):

>> esxcli system module set --enabled=false --module=vmw_ahci

ESXi should fall back to the legacy ahci drivers and disk performance should be a lot better again.

On my hosts the OVA now uploaded at 300 MBit/s and Disk-read/write performance is back to the usual values.

You can also check if the module is enabled or disabled by running the following cli-command:

>> esxcli system module list


The affected module version for the native AHCI driver reports as:

vmw-ahci  1.0.0-32vmw.650.0.0.4564106  VMW  VMwareCertified   2016-11-20

Update (27.03.2017) - the problem still exists for vmw-ahci version:
vmw-ahci 1.0.0-34vmw.650.0.14.5146846 


Update (01.08.2017) - the latest ESXi 6.5 update 1 (build 5969303) seems to fix the performance problems.
It includes vmw-ahci 1.0.0-39vmw.650.1.26.5969303 

You can upgrade your ESXi 6.5 host by cli using the following commands:
esxcli network firewall ruleset set -e true -r httpClient
esxcli software profile update -p ESXi-6.5.0-20170702001-standard \
-d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
esxcli network firewall ruleset set -e false -r httpClient

Or download and boot the ESXi 6.5 Update 1 ISO to upgrade your host.

I hope this guide will help you until VMWare fixed the AHCI drivers for ESXI 6.5.

39 comments:

  1. I struggled for a few days with the 'vmw-ahci' performance issues on my homelab--almost to the point of looking at dedicated SATA controllers w/cache. This resolved my issues in ESXi 6.5. Thanks!

    ReplyDelete
  2. Awesome! same issue here and you are a hero!

    ReplyDelete
  3. I had the same issue with a Z97 board with Intel 9 series AHCI. Thanks!!

    ReplyDelete
  4. Not every hero wears a cape. Thanks!

    ReplyDelete
  5. Thanks for the post. Dell PowerEdge T620... and was about to cry when I tried to transfer in a VMDK file.

    ReplyDelete
    Replies
    1. Actually this did not improve the situation for me, neither did downgrading to 6.0.0, so I must have a different problem.

      Delete
  6. Thanks for the post. I'm looking to move to 6.5 from 6 and will most likely encounter this issue.

    What network drivers are you using with your Avoton board? When I moved from 5.5 to 6.0 my network performance tanked. Was wondering if you had encountered that and solved it, or went with an additional network card?

    ReplyDelete
  7. This fix worked for me on one of my servers, but not the other. Well, it actually fixed both server for performance on that controller. The problem is that when I disable wmw_ahci, it fixes that problem but makes the entire system completely unstable and unusable. I guess the easy answer is just to move off of that controller since the other controllers are working fine, I just don't understand why this happens when I disable it.

    ReplyDelete
  8. OMG! My headaches are gone! I was planning to replace my motherboard! Thanks a lot!!! :)... (Using Esxi 6.5)

    ReplyDelete
  9. Do you mind if I copy this blog post, it was really helpful. If not I'll just take it down from mine.

    ReplyDelete
  10. Been struggling with this for most part of the day. vmw_ahci was very slow in 6.5 (5-10MB/s)

    tnx for the tip! This resolved the issue

    ReplyDelete
  11. Hi, just installed an SSD and was getting ~200kps on upload, where is I get ~22MB/s with a normal hard drive. I now get 14-22MB/s which is better, but I can get 118MB/s over the network to my NAS, so there is still a huge bottleneck / issue with the AHCI implementation.

    ReplyDelete
  12. Do it on my HP microserver gen 8 and get extremely slow esx. I wait about 30 minutes and my vm still boot up. ESXi web page stuck, and even don't display login fields.

    ReplyDelete
  13. Vmware has released a new version of vmw_ahci driver 1.0.0-34vmw.650.0.14.5146846, has anyone tried it?

    ReplyDelete
  14. I'm experienced low write performance on my SSD disk (50-120MB/s) under ESXI 6.5. First I disable vmw_ahci but transfer was more/less the same, Also I did revert ESXI to 6.0 u2 but again the same.
    What kind of performance I can achieve on hosted VM's? I'm using HP DC7900 USFF and Intel SSD 540s.
    I thought I will get write above 200MB/s without any problems. It's kind of supprise for me. I'm new to VMware so thanks for any hints

    Ps. I did upgrade to vmw_ahci driver 1.0.0-34vmw.650.0.14.5146846 but didn't observe any disk performance improvement

    ReplyDelete
  15. I have attempted to enable the vmw-ahci 1.0.0-34vmw.650.0.14.5146846 version and I see the original issue still persist.

    2017-03-27T02:40:46.044Z cpu3:65559)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "t10.ATA_____Crucial_CT750MX300SSD1__________________________162312EA9879" state in doubt; requested fast path state update...
    2017-03-27T02:40:51.041Z cpu0:65556)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "t10.ATA_____Crucial_CT750MX300SSD1__________________________162312EA9879" state in doubt; requested fast path state update...

    ReplyDelete
  16. Problem still exists for version:
    vmw-ahci 1.0.0-34vmw.650.0.14.5146846

    Just checked it out!

    ReplyDelete
  17. I got a gen5 NUC NUC5i5MYHE with a Crucial SSD (MX200 500GB) connected via M.2 and BIOS Mapping it to SATA. Booting ESXi from USB Stick.
    After Update to 6.5, I had the same Problems. Slow and buggy. When I tried your solution, it did not find the SSD at all. After a lot try-and-error I found out that the custom NUC Drivers/VIB need to be installed for it to fall back to old ahci.

    If you got the Problem, go to: https://vibsdepot.v-front.de/wiki/index.php/Sata-xahci
    get he latest drivers/mappings package and install it:
    esxcli software vib install -d /vmfs/volumes/... (zip-file)
    then disable vmw_ahci and reboot.

    ReplyDelete
    Replies
    1. I am glad I read the comments. You saved me from so much trouble.
      esxcli software acceptance set --level=CommunitySupported
      esxcli software vib install -d /vmfs/volumes/SSD/....zip
      Above is future reference for myself.

      Delete
  18. i did it different...
    only connect 1 ssd disk and 1 hdd(normal sata hdd) to the ESXi host

    i did it this way :

    [root@localhost:~] esxcfg-scsidevs -a
    vmhba0 vmw_ahci link-n/a sata.vmhba0 (0000:00:17.0) Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA AHCI Controller
    [root@localhost:~]

    then add a NORMAL hdd as datastore , just in the vmware client add extra datastore
    if they BOTH working delete the driver

    [root@localhost:~] esxcli software vib remove --vibname=vmw-ahci
    Removal Result
    Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
    Reboot Required: true
    VIBs Installed:
    VIBs Removed: VMW_bootbank_vmw-ahci_1.0.0-34vmw.650.0.14.5146846
    VIBs Skipped:

    after this REBOOT !!!!
    and your done !!!

    ReplyDelete
  19. Man, you rock! I had multiple hearbleed errors, I was thinking my HDD is dying... but you turned it all upside down! With the old driver the host is rock soldid.

    ReplyDelete
    Replies
    1. Hi there I still have the problem and I have a RAID controller with Cache enabled
      Network speeds start at 120MBP/s and then drop to a 10 or less and the VM eats all the ram help PLEASE :(

      Cheers

      Delete
  20. I tried this on two separate ESXi installations to fix my slow SSD performance and each time, I lose my datastore ... any ideas?

    ReplyDelete
  21. Worked like a charm. Thank you for sharing. Saved me a huge amount of time ... :-)

    ReplyDelete
  22. I set this command on my Microserver Gen8 and with the newest ESXi 6.5 Version, my server freezes after reboot in a few minutes.

    ReplyDelete
  23. Worked for me on an Intel H67 Motherboard. Write Performance has recovered and is now ok again...
    Really sad that VMware has not fixed this problem in almost 9 months!!! Performance without the fix was abysmal.

    ReplyDelete
  24. New HPE 6.5 ESXI custom ISO has been released on not sure if this fixes the longstanding issues?

    Name: VMware-ESXi-6.5.0-5310538-HPE-650.10.1.0.47-Jul2017.iso
    Release Date: 2017-07-11
    Build Number: 5310538

    I'm going to test it out and see how it performs.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. This comment has been removed by the author.

      Delete
  25. It seems that the new version 1.0.0-39vmw.650.1.26.5969303 has solved the problem.
    Tested on my HP Gen8 Server

    ReplyDelete
    Replies
    1. Thanks for your reply about update availability. I will give a try in my spare time and let you know the results. SHould I aware any other improvements/isues in the update?

      Delete
    2. It didn't helped for me. I did the 6.5U1 upgrade but write performance on my two guests are still around 40-70MB/s. I'm using HP Compaq dc7900 US Desktop with Intel 540 SSD

      Delete
  26. Esxi 6.5u1 was released. I was told by vmware that it solves this nasty problem (I didn't check yet)

    ReplyDelete
  27. A-m-a-z-i-n-g :-) Really helps me a lot.
    I've done it remotely and successfully.
    This is help what I supposed to be on Vmware site –>Shame on you Vmware :-(

    ReplyDelete
  28. Additional info:
    I have update to ESXi 6.5 update 1 (build 5969303). The good thing is that performance is grow from 2MB/s to 130MB/s, but if I disable vmw_ahci (esxcli system module set --enabled=false --module=vmw_ahci) on upgraded system, performance grow even to 230MB/s.

    So upgrade is good, but disable vmw_ahci is stillthe best option!

    ReplyDelete
  29. Hello! I had the same error with small files on the newest version of ESXi. After I reinstallad the older HP driver it works fine with smaller files.

    ReplyDelete
  30. 6.5.0 Update 1 (Build 5969303) comes with scsi-hpvsa-5.5.0-100. I tried to downgrade to scsi-hpvsa-5.5.0-88 (as described here: https://www.johandraaisma.nl/fix-vmware-esxi-6-slow-disk-performance-on-hp-b120i-controller) but didn't worked for me. Performance is still poor.

    ReplyDelete