The Wi-Fi outage that NPS never saw, part 1
At the end of July a customer, a manufacturing group with sites in several European countries, rolled out new server certificates on its domain controllers.
The domain controllers are also the RADIUS servers for the corporate WPA2-Enterprise network. Soon after, a subset of clients stopped joining the corporate SSID. They did not show an error. They ended up on the guest network or off the corporate SSID. Users noticed nothing except that the computer connected to the wrong network.
The puzzling part was the RADIUS side. NPS on two of the servers logged nothing at all for 14 days. No grants, no denies. Silence like that points you at the access points or the network.
For most of the clients, the cause was on the client side. The wireless profiles pushed by Group Policy pinned only the old root CA, the new server certificates chained to the new root, and the clients abandoned the EAP exchange after receiving the server certificate. This post is about how I got there, and about the output that misled me along the way.
What I saw
The setup, anonymised:
- Four domain controllers, DC01 to DC04, running NPS. The access points are managed by the network partner and send RADIUS to them.
- One wireless GPO with nine profiles, PEAP outside and EAP-TLS with machine certificates inside.
- A legacy PKI with the old root, and a new PKI whose root is valid for roughly another ten years. The new domain controller certificates were issued from the new issuing CA on two consecutive days and are valid for one year.
What users saw was the wrong network. What NPS showed on DC01 and DC02 was nothing. Not a single event in 14 days, successful or rejected.
Not everyone was affected, which made it look like a site or hardware problem. Some machines kept working throughout. Several of those turned out to have local profiles that pinned individual server certificates directly, created at some earlier connection by a user who clicked through the certificate warning. They kept working as long as that particular server answered. Every authentication I did observe later went to DC01, and how the network partner’s controller splits primary and secondary RADIUS was still an open question with the partner when this was written. So the silence is not the whole picture of the traffic, only of the failing clients.
For those failing clients the explanation is in the protocol. In PEAP, NPS receives the Access-Request and answers with an Access-Challenge carrying its server certificate. The client then validates that certificate against the roots its profile pins. When none of them match, the client abandons the EAP exchange after receiving the server certificate, so NPS never reaches a final accept or reject to audit. Silence in the NPS log does not mean RADIUS is innocent. It can just as well be the symptom.
Silence only means something once you know the server can log. During the incident I wrote a small tool that discovers the NPS servers from the RAS and IAS Servers group and runs a preflight on each one before it reads events 6272, 6273, 6274 and 6278. The audit check matters, because a disabled audit subcategory produces exactly the same empty log. The check, shortened from the tool so it runs on its own in an elevated PowerShell session on the NPS server itself (auditpol needs administrator rights):
# Locale independent GUID of the 'Network Policy Server' audit subcategory
$AuditGuid = '{0CCE9243-69AE-11D9-BED3-505054503030}'
$out = [ordered]@{ AuditPolicy = $null; Notes = @() }
$raw = & auditpol.exe /get /subcategory:"$AuditGuid" 2>&1 | Out-String
$m = [regex]::Match($raw,
'(?m)^\s*\S.*?\s{2,}(No Auditing|Success and Failure|Success|Failure)\s*$')
$out.AuditPolicy = if ($m.Success) { $m.Groups[1].Value } else { $raw.Trim() }
if ($out.AuditPolicy -match 'No Auditing') {
$out.Notes += 'NPS auditing is disabled. The Security log will stay empty.'
}
[pscustomobject]$out
The same preflight lists the RADIUS clients configured on each server. A client list that does not contain the access point subnet means requests are never audited at all. In my experience that explains silence more often than a broken audit policy does. My working explanation was that NPS never reached a decision to log.
Working it out
With nothing to read on the server side, the evidence had to come from the clients. This is where most of the time went.
netsh misreports where a profile came from. netsh wlan show profiles, run on the client, prints a heading Group policy profiles (read only). It said <None>. The GPO profiles were in fact on the machine, listed under User profiles. That output produced wrong conclusions before anyone noticed it was wrong. The reliable test is to try to delete the profile. A GPO-delivered profile refuses:
You do not have sufficient privileges or the profile "X" on interface "Wi-Fi"
is a group policy profile.
The authoritative client-side view is event 14001 in Microsoft-Windows-WLAN-AutoConfig/Operational, which lists exactly which profiles the wireless group policy applied and which it could not apply.
Wireless policy does not merge. This was the turning point. The corporate wireless GPO showed up among the applied GPOs on the failing machines, so it was easy to assume its profiles were in use. That is not how wireless policy works. The GPO that wins precedence delivers the whole profile list, and every other wireless GPO is ignored completely. gpresult /scope computer /r, run elevated, lists the applied GPOs in precedence order, highest first. It is the quickest way to see which wireless GPO actually wins. Being in the list does not mean its wireless policy is used.
One site had its own wireless GPO, linked closer to the computer objects and therefore with higher precedence. It delivered its own version of the profile, while the corporate GPO sat in the applied list doing nothing. The symptom on a client was that the profile carried two SSIDs where a reference machine had one, and a completely different list of trusted roots. Comparing against a known good machine is what made it visible.
The site profile pinned the old root plus a third root that did not exist in the clients’ Root or CA stores. So the only root that actually carried server validation on those machines was the old one, which explains why that site fell over at exactly the moment the domain controller certificates changed. The profile also had maxAuthFailures = 1. After that single failure the client stops trying the corporate profile and falls back to the guest network. The user noticed nothing except that the computer “connected to the wrong thing”.
The site GPO was unlinked in early August. While I was at it, an old WPA/TKIP wireless GPO, still carrying an XP policy block, turned out to be still linked in the OU tree. It was unlinked too.
Drift between clients. The client diagnostic script I built during the incident now reports this kind of drift directly. It exports every 802.1X profile from a set of computers and fingerprints them. Include one machine you know is good in the set. Two computers on the same SSID should present one fingerprint. The grouping, an excerpt from the script that does not run on its own:
# Excerpt. $results holds one object per computer from the collection phase,
# with Reachable, ComputerName and WlanProfiles (Name, Uses8021X, OuterEap,
# InnerEap, AuthMode, TrustedRootCA per profile).
$profileRows = foreach ($r in ($results | Where-Object { $_.Reachable })) {
foreach ($p in @($r.WlanProfiles | Where-Object { $_.Uses8021X })) {
[pscustomobject]@{
Ssid = $p.Name
Computer = $r.ComputerName
Fingerprint = '{0}|{1}|{2}|{3}' -f $p.OuterEap, $p.InnerEap,
$p.AuthMode, (@($p.TrustedRootCA) -join ',')
}
}
}
$drift = $profileRows | Group-Object Ssid | ForEach-Object {
[pscustomobject]@{
Ssid = $_.Name
Variants = @($_.Group | Group-Object Fingerprint).Count
}
}
More than one variant means the wireless configuration itself differs, and no amount of certificate work will fix it.
Every wireless GPO that lacks the new root. Once it was clear that more than one GPO could carry wireless profiles, I wanted all of them. This loop reads every GPO report, keeps the ones with WLAN profiles, and flags whether the new root’s thumbprint appears anywhere in them (thumbprint replaced with a fake one). It runs on any domain-joined machine with the GroupPolicy module from RSAT, and since it pulls a full XML report for every GPO it can take a while in a large domain:
Import-Module GroupPolicy
Get-GPO -All | ForEach-Object {
$r = [xml](Get-GPOReport -Guid $_.Id -ReportType Xml)
$wlan = $r.GetElementsByTagName('WLANProfile', '*')
if ($wlan.Count -gt 0) {
[pscustomobject]@{
Name = $_.DisplayName
Profiles = $wlan.Count
# first bytes of the new root's thumbprint (fake value here)
HasNewCa = ($r.OuterXml -match '3f\s*9a\s*0c\s*17')
Links = (@($r.GPO.LinksTo |
ForEach-Object { $_.SOMPath }) -join ' | ')
}
}
} | Sort-Object HasNewCa, Name | Format-Table -AutoSize
The cause
All nine profiles in the corporate wireless GPO pinned only the old root for server validation, so once the domain controllers’ certificates were renewed from the new issuing CA, the clients rejected the RADIUS server on their own side. Machines under the site-specific wireless GPO got there by a different route, since their extra root did not exist on the machines and the old root had been doing all the work.
The fix
The new root was added as an additional <TrustedRootCA> in all nine profiles, in both validation blocks.
The second half of that sentence is the one that bites. A PEAP profile has two independent root lists, one for the outer PEAP tunnel and one for the inner EAP-TLS method, and both must accept the server. In my experience, forgetting the inner block is the most common mistake, and it does not show in the outer dialog. Nine profiles therefore meant 18 ServerValidation blocks to change.
At the same time, in every profile and both blocks:
- The server names of all four domain controllers were set.
DisableUserPromptForServerValidationwas set totrue. In the GUI this isDon't ask user to authorize new servers or trusted CAsin the dropdown underProtected EAP Properties, and the checkbox inSmart Card or other Certificate Properties.
The old root was kept alongside the new one, so the change was non-breaking.
Order matters. Do not turn off the user prompt until the roots and server names are verified correct. The setting does not remove the fault, it removes the question. A misconfigured network then fails silently instead of showing a dialog you can troubleshoot against.
To verify, I wrote a script that evaluates per ServerValidation block, not per profile, so a root added to only the outer list is caught. The scope comes from the namespace of the enclosing EapType. An excerpt from the script, not meant to run on its own:
# Excerpt. $node is one WLANProfile element from the GPO report:
# $doc = [xml](Get-Content -LiteralPath C:\Temp\gpo-wlan.xml -Raw)
# foreach ($node in $doc.GetElementsByTagName('WLANProfile')) { ... }
foreach ($sv in $node.GetElementsByTagName('ServerValidation')) {
$scope = 'unknown'
$cur = $sv.ParentNode
while ($cur) {
if ($cur.LocalName -eq 'EapType') {
if ($cur.NamespaceURI -match 'MsPeap') { $scope = 'outer PEAP' }
elseif ($cur.NamespaceURI -match 'EapTls') { $scope = 'inner EAP-TLS' }
break
}
$cur = $cur.ParentNode
}
# ...then collect TrustedRootCA, ServerNames and
# DisableUserPromptForServerValidation for this block only
}
When WinRM to a domain controller is not possible, the script reads the certificate straight off the LDAPS listener on port 636. On a domain controller that is normally the same auto-enrolled certificate NPS presents inside the PEAP tunnel, and the script warns you to confirm it if the verdict is surprising. The verification run that closed the case, from an admin workstation with the GroupPolicy module (RSAT), in the folder that holds Test-RadiusServerCertificate.ps1. The Ldaps transport only needs TCP 636 to each domain controller:
$gpoName = 'Corporate Wireless' # example value, use your wireless GPO's name
New-Item -ItemType Directory -Path C:\Temp -Force | Out-Null
Get-GPOReport -Name $gpoName -ReportType Xml -Path C:\Temp\gpo-wlan.xml
.\Test-RadiusServerCertificate.ps1 `
-ComputerName DC01.corp.example, DC02.corp.example,
DC03.corp.example, DC04.corp.example `
-ProfileXml C:\Temp\gpo-wlan.xml -Transport Ldaps -Verbose
All 18 blocks have to come back with RootMatches=True and Verdict=ACCEPTED.
Two more findings
The group nobody could write to. Some machines had correct GPO profiles and still could not authenticate. This is a separate failure class: the server certificate is fine and it is the client that cannot prove who it is, so NPS gets far enough to reject it, and you should expect a 6273 rather than silence. Their certificate store held only the Intune and Entra certificates, and in some cases a certificate from the old CA with a GUID as subject instead of the machine FQDN, and no SID extension. None of those can be used for machine authentication. For EAP-TLS the certificate needs the Client Authentication EKU, a clean chain, a usable private key, the machine FQDN in subject or SAN, and the SID extension that strong certificate mapping requires after KB5014754. When that certificate is missing, the thing to troubleshoot is autoenrollment, not Wi-Fi.
Newly installed computers were missing from the group that controls autoenrollment of the client certificate. The Intune connector’s service account, call it svc-intune-connector, had GenericWrite on the two WLAN groups but no ACE at all on the certificate group. New computers landed in the WLAN groups and never in the certificate group.
The delegation was an explicit ACE per group object, not a delegation group and not inherited from the OU. That makes it easy to forget when a new group is created, and the failure does not show until someone tries to write. This lists explicit write rights on the member attribute for the two WLAN groups and the certificate group. It needs the ActiveDirectory module (RSAT), which also provides the AD: drive that Get-Acl reads from; the group names are examples:
Import-Module ActiveDirectory # loads the AD: drive used by Get-Acl
$memberGuid = 'bf9679c0-0de6-11d0-a285-00aa003049e2'
$allProps = '00000000-0000-0000-0000-000000000000'
$(foreach ($name in 'WLAN-Computers', 'WLAN-Office', 'Cert-ClientAuth') {
$dn = (Get-ADGroup $name).DistinguishedName
(Get-Acl -Path ("AD:\" + $dn)).Access | Where-Object {
([string]$_.ActiveDirectoryRights -match
'WriteProperty|GenericAll|GenericWrite') -and
(@($memberGuid, $allProps) -contains [string]$_.ObjectType) -and
($_.AccessControlType -eq 'Allow') -and (-not $_.IsInherited)
} | Select-Object @{ n = 'Group'; e = { $name } },
@{ n = 'Trustee'; e = { $_.IdentityReference.Value } }
}) | Format-Table -AutoSize
Without the -not $_.IsInherited filter, the interesting rows drown in inherited ACEs. A scheduled group sync now acts as a safety net, but the fix is the connector’s permission.
Three sites still failing. By early August the broad outage was over. The profiles were fixed, all 18 blocks verified, the competing site GPO gone. Three sites kept failing, one of them the site whose GPO I had just unlinked. The conclusion above, that pinned roots were the cause, holds for the broad outage but not for those three. Part 2 is about the RADIUS servers those sites actually talked to.
What I take with me
- Silence is not innocence: an empty NPS log can be the symptom, because a client that abandons the exchange leaves NPS with no decision to log.
- The tool shows a summary, not the stored state: netsh reported no Group Policy profiles while they were sitting on the machine.
- Wireless Group Policy does not merge, so the GPO that wins precedence is the only one that counts.
- A PEAP profile has two root lists, and a renewal that updates only the outer one is half a fix.
- Diff against a working reference machine; one extra SSID in a profile said more than any log.
- Before every certificate renewal, inventory everything that pins the old root.