r/sysadmin • u/notaramblingman • 16d ago
General Discussion Hyper-V Consultants
We moved from VMware to Hyper-V about a year ago and have an ongoing issue that we have not been able to resolve with MS support - both the typical support case avenue and working with their Hyper-V group. Our host servers lose connectivity with their CSV and VMs go offline. Support tells us it’s related to networking issues and not storage.
Has anyone found any good consultants who know Hyper-V well and are able to resolve difficult issues? We’ve tried a few bigger names but they haven’t been much help.
23
u/peraving 16d ago
One of my clusters developed the same behaviour a couple years ago. Had a long support case with MS premier support which was inconclusive after several weeks. Fed up and desperate to stop the random crashes I rebuilt the cluster hosts separately, one by one, the problem went away after all were rebuilt. I believe the cause was related to 3rd party software on the hosts, I suspect it was the EDR endpoint security agent, Qualys IIRC.
6
u/Academic-Storage3116 15d ago
MS support giving you the runaround while your cluster eats itself is a special kind of hell. I’d be halfway through a rebuild by now too.
→ More replies (1)11
u/TightBed8201 15d ago
You. Do. Not. Install. AV. On. Any. Hypervisor. Period.
That said, i work at pci dss enviroment and we do not have av or edr on hosts. And our auditor is fine with that.
→ More replies (4)5
→ More replies (1)6
u/PresidentScr00b 15d ago
Never use any sort of EDR on your hosts. Basic AV and then harden and lockdown other ways. EDR will destroy your cluster
6
u/the_lazy_sysadmin 15d ago
This is great and all, but in a DoD/CMMC environment its not possible :/ .... Boy oh boy the experience in Hyper-V I've gained just through SentinelOne support cases alone....
→ More replies (7)
14
u/JayFromIT 15d ago
Double check the MPIO very detailed level. Make sure each leg is mapped correctly and you explicitly define the starting IP and endpoint for each leg/interface. Don’t use the default/auto config initiator settings. Then make sure it’s persistent. Also I don’t care what everyone says don’t use anything except for NTFS, I just get to many weirdness with ReFS.
10
u/cantstandmyownfeed 16d ago
What have you done to isolate the problem and confirm that it is or isn't networking?
Because it's probably networking.
8
u/Renich 15d ago
When MS support says it’s a network issue rather than storage, they’re usually looking at CSV I/O redirection breaking down.
What typically happens is a minor latency spike or path failover on the SAN causes the volume to drop into redirected access mode. Instead of talking directly to storage, the host starts shoveling all VM disk I/O over the cluster network via SMB to the coordinator node. If that network doesn't have dedicated bandwidth, it saturates almost instantly. The cluster misses its heartbeat window, assumes a network partition, and drops the CSV to protect integrity. Support sees dropped packets and blames the switch, even though storage triggered the chain reaction.
Before bringing in an outside consultant, look closely at your MPIO timeout timers so Windows doesn't drop the LUN before the storage array finishes failing over, relax the cluster heartbeat sensitivity to give it breathing room during transient blips, and check whether hardware offloads like VMQ on your NICs are quietly dropping frames on the cluster adapters.
6
u/civbat 15d ago
Wow. Are you sure you didn't want to just post a "rant"? It seems to be what all the cool kids do.
You should post your cluster details.
We've had similar issues thrice over the past ~12 years. 1) One node would claim the witness went offline and seize it. All nodes would then barf and guests would go offline. Rebuilt the single problem node and the issue has not reappeared. 2) A specific piece of cluster/chassis hardware was going flaky. This part was a 40 Gb virtual connect. Networking internal to the chassis would go nuts, cluster would lose the CSVs, and all h-v nodes would barf and all guests would go offline. Note that the ESXi cluster would see the same network issue and just ignore it, so only the h-v blades were impacted. 3) Many years ago the fiber HBA drivers (2k8r2) would just stop passing traffic which would occasionally persist after the node rebooted. This would happen over night when the HBAs were saturated while backups ran. The OEM knew of the issue and told us to wait for the driver update that was just around the corner. Years later we moved the clusters to 2012r2 without seeing the driver update.
3
u/Ok_Figure7074 15d ago
This was very helpful post for me. We have a similar setup at one of our sites (virtual connect to be more specific) and one day all hosts had severe degradation of their network and storage performance, basically couldn’t do anything but move the cursor around in iLO for a few mins at a time. Then after several hours and no changes, all hosts at the same time came back up like nothing happened.
5
u/TechMonkey605 16d ago
How many hosts, and VMs per node. Let start there.
What is your storage, local, unity, Electra
What is your networking setup? What’s running over 1G and what’s on 10G/fiber?
4
u/gabgab33 15d ago
I had something similar a while ago for a client. In the end, it was a buggy lldp agent on the networking cards controller that would cause networking to cut.
Point is before looking at an hyper-v consultant you should look into the whole stack to make sure everything is in order, up-to-date and with no known issues.
3
u/blow_slogan 15d ago
Uninstall your antivirus from the host machine running Hyper-V. It can lock up and make make VMs crash or go offline. Haven’t had any issue switching to S1.
3
u/draconian_deference 15d ago
Had the same nightmare after moving to Hyper-V. Turned out to be VMQ on Broadcom NICs causing CSV blips. Disable VMQ/RSS on the host adapters and try again
8
u/thatfrostyguy 16d ago
Your best bet is to actually hire someone that knows hyper-v. Not just to consult
→ More replies (1)12
u/burnte 15d ago
No, you don't need to buy the cow when you just need a cup of milk.
→ More replies (13)
2
u/ludlology 15d ago edited 15d ago
This could be so many things from hyper-v problems to a bad NIC or cable, or switch misconfiguration, or SAN problem, or MPIO issue on the hosts, bad drivers, screwy mesh cabling, etc.
We definitely need more information like did the cluster work and then started having these problems after some change, did it never work, what have you tried, what error messages are you getting, and so on.
I’ll give you a few edge case gotchas I’ve run into that are tangential though:
-
If your only DCs run on these hosts, and the hosts are members of the domain, this kind of thing will happen even in single host environments. The VHDs become just like any other domain permission file and the hosts can’t access them without a DC to auth to
-
Similar issue where a time sync paradox comes up if the hosts sync time from the DCs which are VMs that sync time from the hosts
-
If your cluster witness is within the cluster, and especially if you only have two hosts instead of three, the hosts will never reach quorum and decide who should be primary. This results in VMs failing to boot
Shoot me a DM if you want to set up a time to talk. I do this kind of work from time to time. You need an overall best practices audit of the cluster environment (hosts, switches, storage) to look for low hanging fruit plus targeted troubleshooting of the symptoms.
2
u/ScreamingVoid14 15d ago
What format are your CSVs?
ReFS forces an IP layer write redirection to the host that owns the drive. One server becoming unhealthy causes a great deal of pain for the entire cluster. Doubly so if your management port is only gigabit, as now all sorts of disk IO is stuffed onto those links.
2
u/rspmkiv 15d ago
Not sure if you answered but Im going to jump on the EDR bandwagon, especially S1 blowing up the cluster database. Do you have all your Hyper-V recommended exclusions set and/or have you tried removing your AV completely to test?
Also try moving the quorum from disk witness to a file share as a temporary test.
MPIO was mentioned to review, and also check jumbo packets (ping -f -l 8000 to every piece of equipment in your storage system).
3
2
u/mangeek 15d ago edited 15d ago
I've seen this in an environment where the block storage was served to the hypervisor directly over FC rather than the recommended 'redundant SMB3 cluster holding VHDX files'.
A lot of people in the industry just don't like SMB/CIFS, and it's to their own detriment at this point. What used to be 'protocol overhead for no reason' is now 'a path to RDMA, multi-channel, redundancy, and secure transport'.
For a long time now, most VMware and Hyper-V hosting setups see practical benefits and minimal performance loss from following this pattern:
[core storage]--SMB3 or NFSv4 volume-> hypervisor mount.
Is there a place for direct Hypervisor->FC storage mapping? Yes, but the burden to justify a deviation from the recommendation above, which is more flexible and reliable, to chase a few percent of performance, should be on the person asking for it. I'd venture that it's in things like volumes that have high transactional loads, and all efforts to improve performance before going with direct mappings should happen first (e.g., jumbo frames, NIC settings, wrangling NUMA, avoiding oversubscription of vCPU resources on the perf-sensitive hosting cluster, software optimizations, caching layers, etc.)
The root cause of the problem is humans not wanting to admit that a lot of the expertise has moved from blinkenlights and fat pipes between $200K components to the software layers.
2
u/Nerd2259 15d ago
Move to Proxmox and get support direct from the developers or a gold partner if you need someone else to be hands-on. Both Proxmox themselves and their gold partners will spin up entire clusters and replicate your setup 1 for 1 to figure out the issue. Occasionally they'll even create custom software patches for you.
I've never had that experience from anything MS.
40
u/Aggressive_Common_48 16d ago
I’m not a consultant, but I’d like to suggest a few things.
Have you checked the cluster logs for specific events and the reasons behind the connection loss? It sounds like your MPIO configuration could be the cause.
Before creating the cluster, did you verify that the network interfaces were working correctly, particularly the failover functionality? For example, did you disable one interface to confirm that the connection to the storage solution remained active?
Also, I’m not sure what type of storage solution you’re using, but have you verified that redundant connections are properly configured between the Hyper-V cluster and the storage system? Microsoft’s Hyper-V guidance emphasizes the importance of having redundant paths to the storage solution to ensure continued connectivity in the event of a path failure.