EXOIT

10 min#finding#radius#pki#wifi#method

The Wi-Fi outage that NPS never saw, part 2: a binding, a wall and a proxy from 1996

In part 1 a PKI migration broke 802.1X Wi-Fi for a manufacturing group with sites in several European countries, and the first fix was in the WLAN profiles, which pinned the old root CA. That fix brought most sites back. Three sites kept failing.

Those three had something odd in common. The clients logged explicit EAP failures, yet the machines never showed up in the NPS logs. Not a reject, not a discard, nothing. All three reached the NPS servers, NPS01 and NPS02, through the hosting partner’s RADIUS proxy.

It turned out to be three faults in series, and each fix only revealed the next one: an NPS policy bound to a certificate that no longer existed, a CRL the servers could not fetch, and a proxy that talks to the internet in HTTP/1.0.

The symptom

From the client side it looked like any other certificate problem. EAP started, EAP failed. From the server side it looked like nothing at all. The NPS accounting logs and the Security log had no trace of the failing machines, so the obvious reading was that the traffic never arrived.

That reading was wrong. NPS logs authentication attempts. It grants or denies, and writes down which. If the EAP conversation dies before there is anything to grant or deny, there is nothing to log. Silence in the NPS log does not mean the request never came. It can also mean NPS could not even start the conversation.

Working it out

Fault one: a binding to a certificate that was gone. At the end of July, during the PKI migration, old server certificates were cleaned off both NPS servers. A PEAP policy points at its server certificate by SHA1 thumbprint, and that binding does not follow anything. Delete the certificate and the policy keeps pointing at a thumbprint that is not there.

A PEAP policy without its server certificate cannot initiate EAP. The client sees a broken handshake. NPS writes nothing, because there was never an authentication attempt.

The reason this was hard to see is the NPS console. Open Protected EAP Properties on a policy whose certificate has been deleted, and the dropdown shows the first suitable installed certificate: right name, right issuer, right expiry date. It looks completely healthy. It is a pre-selected default, not the stored value. Only when you pick the certificate and press OK does a working thumbprint get written back.

The only reliable way to see the real binding is to export the configuration and read the thumbprint out of the stored blob. The export runs on the NPS server itself, from an elevated PowerShell prompt; the nps netsh context exists only where the NPS role is installed, so on a workstation netsh answers that the command was not found:

# On the NPS server, elevated
$cfgFile = "$env:TEMP\nps-config.xml"
netsh nps export filename="$cfgFile" exportPSK=YES

exportPSK=YES is not optional. netsh refuses to export at all without it, so the export contains shared secrets and has to be handled accordingly.

In the export, the binding is not where you would look first. The settings for one policy are split across two subtrees joined by node name. Conditions, sequence and Policy_Enabled sit under NetworkPolicy, while msEAPConfiguration sits under RadiusProfiles. Read only the policy nodes and you find no certificate bindings at all.

Inside the msEAPConfiguration blob the server certificate hashes are stored as a counted list: a DWORD count of 1, a DWORD length of 0x14, then the twenty hash bytes, both DWORDs little endian. The core of the parser, shortened from my NPS configuration script, reads every msEAPConfiguration value under RadiusProfiles from the export above (an octet string written as hex), scans it, and checks each hash against the machine store. It runs in the same session on the NPS server and works in Windows PowerShell 5.1:

$xml   = [xml](Get-Content -LiteralPath $cfgFile -Raw)
$xpath = "//*[local-name()='RadiusProfiles']//*[local-name()='msEAPConfiguration']"
$found = [System.Collections.Generic.List[string]]::new()

foreach ($cfgNode in @($xml.SelectNodes($xpath))) {
    $hex  = ($cfgNode.InnerText -replace '^0[xX]', '') -replace '[^0-9A-Fa-f]', ''
    $Blob = [byte[]]::new($hex.Length / 2)
    for ($k = 0; $k -lt $Blob.Length; $k++) {
        $Blob[$k] = [Convert]::ToByte($hex.Substring($k * 2, 2), 16)
    }

    for ($i = 0; $i -le $Blob.Length - 28; $i++) {
        # DWORD count = 1, little endian
        if ($Blob[$i] -ne 1 -or $Blob[$i + 1] -ne 0 -or
            $Blob[$i + 2] -ne 0 -or $Blob[$i + 3] -ne 0) { continue }
        # DWORD length = 0x14 (20 bytes, a SHA1 hash)
        if ($Blob[$i + 4] -ne 0x14 -or $Blob[$i + 5] -ne 0 -or
            $Blob[$i + 6] -ne 0 -or $Blob[$i + 7] -ne 0) { continue }

        $sb = New-Object System.Text.StringBuilder
        for ($j = 0; $j -lt 20; $j++) {
            [void]$sb.AppendFormat('{0:X2}', $Blob[$i + 8 + $j])
        }
        $tp = $sb.ToString()
        if (-not $found.Contains($tp)) { $found.Add($tp) }
    }
}

foreach ($tp in $found) {
    $state = 'NOT INSTALLED'
    if (Test-Path -LiteralPath "Cert:\LocalMachine\My\$tp") { $state = 'installed' }
    '{0}  {1}' -f $tp, $state
}
Remove-Item -LiteralPath $cfgFile   # the export contains the shared secrets

The thumbprint that comes out is then compared against the store, and anything not present is flagged NOT INSTALLED. Export before and after the fix on NPS01:

before  3f9a0c17...   not present on the server
after   8d2e6b40...   valid, issued by the legacy issuing CA

Same certificate name in the dialog both times.

There was a second trap in the fix itself. The certificate has to be set in three places per policy. Under Constraints, Authentication Methods, there are Microsoft: Smart Card or other certificate and Microsoft: Protected EAP (PEAP) as separate EAP types, each with its own binding, and inside the PEAP dialog there is the inner method’s binding. Set only one and the others keep pointing at the deleted certificate, and which one wins depends on what the client negotiates. The target state from the script is a single row per policy with no NOT INSTALLED. Two rows mean one of the three places was missed.

The binding was re-selected on NPS01 first, all three places, and the export confirmed it. Then NPS02. One of the three sites confirmed working Wi-Fi after this step alone.

A side finding: NPS02 receives no RADIUS traffic at all, because the hosting partner’s proxy only sends to NPS01. That makes NPS02 useless as a canary for a certificate change, and it means the whole path hangs on one server.

Fault two: the revocation server was offline. After the binding fix the error changed. It now appeared in the log, as reason code 259:

The revocation function was unable to check revocation because the revocation
server was offline

This is progress, even if it does not feel like it. With the binding fixed, NPS was now getting far enough to validate the client certificate, and for EAP-TLS that includes a revocation check. The clients presented certificates from the new issuing CA, whose CDP is a CRL on Azure Storage, http://pki.example.com/cdp/IssuingCA1.crl. From the NPS servers that traffic went through the hosting partner’s tinyproxy pair, PROXY01 and PROXY02, which answered 403.

The host was added to the tinyproxy filter file on both proxies. 403 became 505.

Fault three: HTTP/1.0 in 2026. The new answer was:

505 Http Version Not Supported

tinyproxy 1.11.0 hard-codes HTTP/1.0 upstream, so there was nothing to configure. Azure Storage requires HTTP/1.1. That was verified from the proxy itself:

curl --http1.0 http://pki.example.com/cdp/IssuingCA1.crl   ->  505
curl --http1.1 http://pki.example.com/cdp/IssuingCA1.crl   ->  200

The fix was to take the proxy out of the path entirely for that one host.

The test that lied, and the one that did not

A firewall rule opened direct outbound TCP 80 and 443 from the NPS servers to the CDP addresses. Test-NetConnection -Port 80 against the CDP host now returned TcpTestSucceeded True. CryptoAPI still failed.

The port test proved that a TCP connection could be made. It said nothing about which path CryptoAPI takes, and CryptoAPI uses the WinHTTP machine proxy (for a service like NPS), which is configured separately from everything else. The CDP host was still being sent to tinyproxy. It had to go on the WinHTTP bypass list on both servers, checked on each NPS server (no elevation needed to read it) with:

netsh winhttp show proxy

There is a second trap here. Invoke-WebRequest does not respect the WinHTTP machine proxy, so a successful download with it proves nothing about what CryptoAPI will do. Both of the easy tests answer a question nobody asked.

The test that answers the right question is to build the chain with online revocation, for a certificate from the new issuing CA, and read the status. Run it on the NPS server, so that the fetch leaves from the same machine and network path as NPS:

# Example: any certificate issued by the new issuing CA. Adjust the filter, or load an
# exported client certificate instead:
# $cert = [Security.Cryptography.X509Certificates.X509Certificate2]::new('C:\Temp\c.cer')
$cert = Get-ChildItem Cert:\LocalMachine\My |
    Where-Object Issuer -like '*Issuing*' | Select-Object -First 1
$chain = [System.Security.Cryptography.X509Certificates.X509Chain]::new()
$chain.ChainPolicy.RevocationMode = 'Online'
$chain.ChainPolicy.RevocationFlag = 'ExcludeRoot'
$ok = $chain.Build($cert)
$ok
$chain.ChainStatus | ForEach-Object { $_.Status }

Then the URL cache was cleared and the NPS service (IAS) restarted, on each NPS server from an elevated prompt:

certutil -urlcache * delete
Restart-Service -Name IAS

After that the chain built clean on both servers, ChainValid True with an empty ChainStatus.

The binding fix, the proxy filter, the firewall rule, the bypass list and the cache flush were all done within one day. One site had confirmed working Wi-Fi after the binding fix. Another had not retested by the end of that day; its last rejection was a 259 logged before the CDP was opened. The third site’s result is not in my notes.

Keeping it fixed

A direct firewall rule to Azure is an IP-based rule against a name whose addresses rotate. It works until it silently does not, and the symptom then shows up as reason code 259 on a Wi-Fi network rather than as a firewall problem. So there is now a watcher. It resolves the CDP host against external resolvers only, deliberately skipping the internal zone, unions the answers over several queries per resolver because Azure front ends hand out different subsets, and compares against the last run. The query at its core, runnable on any Windows machine (the resolvers are examples):

$HostName = 'pki.example.com'
foreach ($s in '1.1.1.1', '8.8.8.8') {
    $answer = @(Resolve-DnsName -Name $HostName -Type A -Server $s `
                -DnsOnly -NoHostsFile -ErrorAction Stop |
        Where-Object {
            $_.PSObject.Properties.Name -contains 'IPAddress' -and $_.IPAddress
        })
    '{0}: {1}' -f $s, (($answer.IPAddress | Sort-Object -Unique) -join ', ')
}

It runs every four hours, mails immediately on a change, and otherwise sends one healthy report per day. The daily mail is the point: an absent report means the job itself has stopped, which a change-only watcher can never tell you.

The proxies got a clean-up script too, which normalises the tinyproxy configuration and filter file, runs dry by default and never adds anything to the allowlist.

At the time of the fix the re-selected certificate was still issued by the legacy PKI, and it had to be replaced before the old root’s CRL NextUpdate in late September. That date, not the CA certificate’s expiry, is the real deadline for a retiring CA: after it every revocation check against the old hierarchy fails, server and client certificates alike.

Enterprise CAs publish their CRLs into the configuration partition under CN=CDP, so the dates can be read without touching the CA. I parse the DER directly rather than the output of certutil -dump, because certutil localises its field names. With my own script, Get-PkiCrlStatus.ps1 (not published here), the list that matters during a decommission is:

# Reads every cRLDistributionPoint object in the configuration partition under
# CN=CDP,CN=Public Key Services,CN=Services, with Get-ADObject. It needs the
# ActiveDirectory module (RSAT) and only reads, it never contacts the CA.
.\Get-PkiCrlStatus.ps1 | Where-Object { $_.DaysRemaining -lt 60 } |
    Sort-Object DaysRemaining

Silent zeros

One bug in my own tooling surfaced along the way, and it was of the same kind as the outage: a zero that was read as a finding. The NPS configuration script read only the policy nodes, not RadiusProfiles, looked for msNPPolicyEnabled instead of Policy_Enabled, and so reported zero certificate bindings and every policy as enabled. Worse, it searched the blob for thumbprints already known to be installed.

That last part is the principled one. Searching for certificates you know exist can only ever confirm a match. It is structurally blind to the case that matters: a policy bound to a certificate that is missing.

What I take with me

  • The error you see is the last link in a chain, and fixing it may only promote the next one.
  • Silence is not innocence: an empty NPS log can mean NPS never got far enough to log.
  • The UI shows a default, not the stored value, so read the export, not the dialog.
  • Test from the right vantage point: a port test and Invoke-WebRequest do not travel the path CryptoAPI travels, but building the chain with online revocation does.
  • A script that searches for what it expects can only confirm, never discover.
  • The deadline for a retiring CA is its CRL NextUpdate, not its certificate expiry.