EXOIT

10 min#finding#azure#windows-update#method

The error code in the portal is the fourth link in the chain

Two Arc-enabled servers at one site of a customer, a manufacturing group with sites in several European countries, stopped reporting to Azure Update Manager in August. The second server had the same symptom. The file server, FS01, showed “assessment failed” with 813 accumulated failures and the HRESULT 0x80240007.

That code has a reputation. Search for it and almost every answer tells you the Windows Update datastore is corrupt and that you should clear out SoftwareDistribution. It is a reasonable reflex, and here it would have achieved nothing except a long rescan against a proxy that still did not answer, and an empty local update history.

The actual cause was a WinHTTP proxy left behind from a period when the servers sat in the wrong OU. The servers could not reach it, and the error changed its code three times on the way up to the portal. This post is about how to read past the code you are shown.

What I saw

In the portal: assessment failed, 0x80240007, which is WU_E_INVALID_INDEX. Azure Update Manager adds its own interpretation on top:

SearchForUpdateWasInterruptedDueToCancellationRequestedorTimeExpired

That is literally true. The scan ran for 174 plus 415 seconds before it gave up. But it points at a timeout as the cause, and that turned out to be wrong.

Arc itself looked healthy. The machines were Connected with a fresh heartbeat. On the server, the patch extension’s state.json showed ExtensionState: ENABLED with an empty ErrorMsg, and the *_stdout.txt file held nothing but the launcher line. Everything visible said the plumbing was fine and the scan simply failed.

The last successful assessment was some time before the middle of August.

Working it out

Read the ETL, not the summary. Everything of value sits in the extension’s windowsUpdateLog folder as binary ETL traces. They have to be decoded first, and Get-WindowsUpdateLog takes an -ETLPath, so you can point it at the extension’s own trace folder. Run this in an elevated PowerShell session on the Arc-enabled server itself:

$ext = 'C:\ProgramData\GuestConfig\extension_logs\' +
       'Microsoft.CPlat.Core.WindowsPatchExtension'
$log = 'C:\Temp\WindowsUpdate-decoded.log'
New-Item -ItemType Directory -Path (Split-Path $log) -Force | Out-Null

# Decode the extension's ETL trace, the only place the root error is visible.
# -ETLPath accepts a folder and decodes every .etl file in it.
Get-WindowsUpdateLog -ETLPath "$ext\windowsUpdateLog" -LogPath $log

# The whole block of the last federated search, not just the error lines.
# Filtering on FAILED misses the WebServices layer completely.
$end = (Select-String -Path $log -Pattern 'All federated searches have completed' |
        Select-Object -Last 1).LineNumber
if (-not $end) { throw "No completed federated search found in $log" }
Get-Content $log | Select-Object -Skip ([Math]::Max(0, $end - 170)) -First 180

The second half is the part that matters. Filtering on FAILED misses the WebServices layer completely. The lines that explain the failure are not marked as failures; they sit inside the same federated-search block.

Why two jobs died. Modern WUA fans a search out to every registered update service. On FS01 there were two in play:

Service ServiceID Endpoint
Microsoft Update 7971f918-a847-4430-9279-4a52d1efe18d tas02.cwsmu.update.microsoft.com
DCat Flighting Prod 8b24b027-1dee-babb-9a95-3517dfb9c552 fe3cr.delivery.mp.microsoft.com

The log said Jobs = 2, Succeeded = 0. Both child jobs had failed, which invites you to suspect the service registrations. They were correct the whole time. Four registered services (Microsoft Update, DCat Flighting, Windows Store, Windows Update) is normal and nothing to fix. Both jobs simply went through the same path, and the path was dead.

The line that gave it away sat inside the federated search block:

WebServices  Using default WinHTTP proxy settings: proxy name='PROXY01.corp.example:8888',
             bypass list='*.corp.example,<local>,*.example.com', auth scheme='0'
WebServices  WS error: The given proxy cannot be reached.
WebServices  MapToSusHResult mapped Nws error 0x803d0010 to 0x80240438

A proxy, on a server that was supposed to go direct. The bypass list did not exempt any Microsoft endpoints, so every Windows Update call went to it.

With that line in hand, the rest of the log reads as a chain. The error propagates up through four layers and changes its code at each one:

Layer Code What actually happens
WinHTTP / NWS 0x803D0010 The given proxy cannot be reached
WU ProtocolTalker 0x80240438 WU_E_PT_ENDPOINT_UNREACHABLE, SyncUpdates fails after 3 retries
ComApi FederatedSearchJob 0x8024000B WU_E_CALL_CANCELLED, both child jobs dead, parent cancelled
UpdateExceptionCollection 0x80240007 WU_E_INVALID_INDEX, empty exception list read, index out of range

0x80240007 is raised at line 44 of UpdateExceptionCollection.cpp, when Windows Update tries to read the errors out of a job that has already been cancelled. The list is empty. It is an artefact of the error handling, not a state of the machine. There was no sign of a datastore problem, and clearing it would not have helped.

Test-NetConnection gave false negatives. The obvious network test was misleading. To calibrate it, I ran it on DC01, a domain controller that demonstrably fetched updates without trouble. Test-NetConnection -Port 443 reported False against fe3cr.delivery.mp.microsoft.com, tas02.cwsmu.update.microsoft.com and sls.update.microsoft.com.

A working machine failing the test means the test is wrong, not the machine. The most likely reason: these endpoints sit behind Akamai and Azure Front Door, which rotate A records with short TTLs, and the firewall’s FQDN objects resolve their own list of IPs. A raw TCP connect to an IP outside that list fails. Without -Port it only pings, which is useless when ICMP is blocked, as it was here.

So I wrote a small replacement that does a real TCP connect followed by a TLS handshake with the correct SNI, and returns the certificate issuer. The issuer is a bonus: if something in the path does TLS inspection, it shows up there. The function runs in Windows PowerShell 5.1 or PowerShell 7 on any machine, without elevation:

function Test-TlsEndpoint {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory, ValueFromPipeline)][string]$HostName,
        [int]$Port = 443,
        [int]$TimeoutMs = 10000
    )
    process {
        $client = New-Object System.Net.Sockets.TcpClient
        $ssl    = $null
        try {
            $task = $client.ConnectAsync($HostName, $Port)
            if (-not $task.Wait($TimeoutMs)) {
                return [pscustomobject]@{
                    Endpoint = $HostName; Tcp = $false; Tls = $false; Issuer = $null
                    Detail   = "TCP connect timed out after ${TimeoutMs}ms" }
            }
            $ssl = New-Object System.Net.Security.SslStream(
                $client.GetStream(), $false, { param($s,$c,$ch,$e) $true })
            $ssl.AuthenticateAsClient($HostName)   # SNI is the host name
            $cert = [Security.Cryptography.X509Certificates.X509Certificate2](
                $ssl.RemoteCertificate)
            [pscustomobject]@{ Endpoint = $HostName; Tcp = $true; Tls = $true
                Issuer = $cert.Issuer; Detail = $cert.Subject }
        }
        catch {
            [pscustomobject]@{
                Endpoint = $HostName; Tcp = $client.Connected; Tls = $false
                Issuer = $null; Detail = $_.Exception.Message }
        }
        finally { if ($ssl) { $ssl.Dispose() }; $client.Dispose() }
    }
}

'fe3cr.delivery.mp.microsoft.com', 'sls.update.microsoft.com' | Test-TlsEndpoint

It is wrapped in a function, Test-TlsEndpoint, with HostName taking pipeline input, Port defaulting to 443 and TimeoutMs to 10000, so you can pipe a list of endpoints into it. Even so, the authoritative test is not a network probe at all. It is a real WUA COM scan, shown further down.

Diff against the machine that works. DC01 was the reference: Direct access, and updates arriving without complaint. FS01 used the proxy. That was the difference that mattered.

The next question was whether the proxy was down or just unreachable from where FS01 stood. I tested tcp/8888 from both subnets:

Source tcp/8888 to PROXY01
FS01 Fail against both proxy nodes
A reference machine in another subnet OK

The proxy was alive. FS01’s network segment simply had no path to it, and never should have needed one.

Why Arc was fine. How can a machine be Connected in Arc and unable to reach Microsoft at the same time? The Arc agent has its own proxy setting, proxy.url, and on these servers it was empty. The agent went straight out and worked. Windows Update used the machine’s WinHTTP proxy and did not.

Two separate network paths, two separate failure sources. A green Arc status with a fresh heartbeat says nothing about whether Windows Update works on the same machine.

The cause

The proxy is meant for branch offices and for one partner’s delivery. Servers at this site should never go through it. No GPO was setting it when I looked, so it was a leftover from a period when the servers had sat in the wrong OU.

The WinHTTP proxy lives as a binary blob in HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\WinHttpSettings. That is not a real policy key under \Policies\, so it is not cleaned up when the GPO stops applying. Once written, it stays until someone removes it on purpose.

The fix

Run this in an elevated PowerShell session on the affected server itself. The WUA COM search does not work over a remote session, so run it locally or over RDP:

netsh winhttp reset proxy
netsh winhttp show proxy   # should show Direct access (no proxy server)

Restart-Service wuauserv, bits, DoSvc -Force
# Defender sat behind the same dead proxy, see the side finding below
# May be blocked by Tamper Protection
Restart-Service WinDefend -Force -ErrorAction SilentlyContinue

# Authoritative test. This, not Test-NetConnection, decides whether it works.
$sw = [Diagnostics.Stopwatch]::StartNew()
try {
    $searcher = (New-Object -ComObject Microsoft.Update.Session).CreateUpdateSearcher()
    $res = $searcher.Search('IsInstalled=0 and IsHidden=0')
    "OK after $([int]$sw.Elapsed.TotalSeconds)s, $($res.Updates.Count) updates found"
}
catch {
    # PowerShell wraps the COMException; the WU error code is on the inner exception
    $e  = $_.Exception
    $hr = if ($e.InnerException) { $e.InnerException.HResult } else { $e.HResult }
    "FAILED after $([int]$sw.Elapsed.TotalSeconds)s: 0x{0:X8}" -f $hr
}

Then confirm that Windows Update itself has stopped using the old proxy string, rather than trusting netsh. Same elevated session on the server; C:\Temp was created in the first block:

Get-WindowsUpdateLog -LogPath C:\Temp\wu-after.log
Select-String -Path C:\Temp\wu-after.log -Pattern 'WinHTTP proxy settings' |
    Select-Object -Last 5

Does it need a reboot? Not strictly. The registry changes immediately. The problem is process caching: WinHTTP reads the configuration when a session is created with WinHttpOpen and WINHTTP_ACCESS_TYPE_DEFAULT_PROXY, so long-running services keep the old setting until they create a new session.

The argument for a reboot is that a service restart is not deterministic. wuauserv runs in a shared svchost under netsvcs, and UsoSvc consumes the same configuration without being on a typical restart list. A colleague who knows the environment well recommended a reboot. In this case restarting the services was enough. I treat a reboot as a safety margin, not a requirement, and always verify against the Windows Update log afterwards.

One server with a leftover proxy suggests others. I wrote an inventory to run across every server in the domain; running it was on the follow-up list. It reports which servers carry a WinHTTP proxy and whether they can reach it. Run it from a domain-joined admin machine with the RSAT ActiveDirectory module, as an account that is local admin on the servers, with WinRM enabled on them:

$servers = Get-ADComputer -Filter "OperatingSystem -like '*Server*'" `
    -Properties OperatingSystem | Select-Object -ExpandProperty Name

Invoke-Command -ComputerName $servers -ErrorAction SilentlyContinue -ScriptBlock {
    $proxy  = (netsh winhttp show proxy) -join ' '
    $match  = [regex]::Match($proxy, 'Proxy Server\(s\)\s*:\s*(\S+)')
    $server = if ($match.Success) { $match.Groups[1].Value } else { 'Direct' }

    $reachable = $null
    if ($server -ne 'Direct') {
        # Handles the simple host:port form only
        $parts = $server -split ':'
        $reachable = (Test-NetConnection -ComputerName $parts[0] -Port $parts[1] `
                        -WarningAction SilentlyContinue).TcpTestSucceeded
    }
    [pscustomobject]@{
        Computer       = $env:COMPUTERNAME
        WinHttpProxy   = $server
        ProxyReachable = $reachable
    }
} | Sort-Object ProxyReachable | Format-Table -AutoSize

Servers that refuse remoting simply vanish from the output, so compare the row count against the list of names.

The excerpt is trimmed: the full version also reports each server’s IPv4 address and sorts on ProxyReachable, then IPv4, which makes it easy to see whether the unreachable ones share a subnet.

The side finding that mattered more

Defender had been behind the same dead proxy for as long as Windows Update. One night in August, FS01’s signature version was well behind DC01’s: ten days without fresh definitions on a production file server. That is more serious than the missing patch reporting, and next time I see this pattern it is the part I escalate first. Updating signatures directly with Update-MpSignature -UpdateSource MicrosoftUpdateServer on both affected servers went on the follow-up list.

Nothing alerted on it. It just quietly fell behind.

What I take with me

  • The error code in the portal is the last link in a chain, so find the first one before clearing anything.
  • Read the whole block around a failure in a decoded log, because the cause may not be logged as an error at all.
  • When a test fails on a machine you know works, the test is wrong, so calibrate tools against a working reference first.
  • Diff the broken machine against a working one at the same site before forming theories.
  • A setting written by policy is not necessarily removed by policy, so check where it actually lives.
  • A green status in one agent says nothing about the network path another component uses on the same machine.