<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>EXOIT</title><description>Notes and findings from Marcus Mattsson on identity, Microsoft 365, infrastructure and the homelab.</description><link>https://exoit.se/</link><language>en</language><item><title>The Wi-Fi outage that NPS never saw, part 1</title><link>https://exoit.se/posts/wifi-outage-nps-never-saw-part-1/</link><guid isPermaLink="true">https://exoit.se/posts/wifi-outage-nps-never-saw-part-1/</guid><description>Clients dropped off corporate Wi-Fi after a certificate renewal, and NPS logged nothing for 14 days. For most clients the fault was in the wireless profiles.</description><pubDate>Sat, 03 Oct 2026 06:50:00 GMT</pubDate><content:encoded>&lt;p&gt;At the end of July a customer, a manufacturing group with sites in several European countries, rolled out new server certificates on its domain controllers.&lt;/p&gt;
&lt;p&gt;The domain controllers are also the RADIUS servers for the corporate WPA2-Enterprise network. Soon after, a subset of clients stopped joining the corporate SSID. They did not show an error. They ended up on the guest network or off the corporate SSID. Users noticed nothing except that the computer connected to the wrong network.&lt;/p&gt;
&lt;p&gt;The puzzling part was the RADIUS side. NPS on two of the servers logged nothing at all for 14 days. No grants, no denies. Silence like that points you at the access points or the network.&lt;/p&gt;
&lt;p&gt;For most of the clients, the cause was on the client side. The wireless profiles pushed by Group Policy pinned only the old root CA, the new server certificates chained to the new root, and the clients abandoned the EAP exchange after receiving the server certificate. This post is about how I got there, and about the output that misled me along the way.&lt;/p&gt;
&lt;h2 id=&quot;what-i-saw&quot;&gt;What I saw&lt;/h2&gt;
&lt;p&gt;The setup, anonymised:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Four domain controllers, DC01 to DC04, running NPS. The access points are managed by the network partner and send RADIUS to them.&lt;/li&gt;
&lt;li&gt;One wireless GPO with nine profiles, PEAP outside and EAP-TLS with machine certificates inside.&lt;/li&gt;
&lt;li&gt;A legacy PKI with the old root, and a new PKI whose root is valid for roughly another ten years. The new domain controller certificates were issued from the new issuing CA on two consecutive days and are valid for one year.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;What users saw was the wrong network. What NPS showed on DC01 and DC02 was nothing. Not a single event in 14 days, successful or rejected.&lt;/p&gt;
&lt;p&gt;Not everyone was affected, which made it look like a site or hardware problem. Some machines kept working throughout. Several of those turned out to have &lt;strong&gt;local&lt;/strong&gt; profiles that pinned individual server certificates directly, created at some earlier connection by a user who clicked through the certificate warning. They kept working as long as that particular server answered. Every authentication I did observe later went to DC01, and how the network partner’s controller splits primary and secondary RADIUS was still an open question with the partner when this was written. So the silence is not the whole picture of the traffic, only of the failing clients.&lt;/p&gt;
&lt;p&gt;For those failing clients the explanation is in the protocol. In PEAP, NPS receives the Access-Request and answers with an Access-Challenge carrying its server certificate. The client then validates that certificate against the roots its profile pins. When none of them match, the client abandons the EAP exchange after receiving the server certificate, so NPS never reaches a final accept or reject to audit. Silence in the NPS log does not mean RADIUS is innocent. It can just as well be the symptom.&lt;/p&gt;
&lt;p&gt;Silence only means something once you know the server can log. During the incident I wrote a small tool that discovers the NPS servers from the &lt;code&gt;RAS and IAS Servers&lt;/code&gt; group and runs a preflight on each one before it reads events 6272, 6273, 6274 and 6278. The audit check matters, because a disabled audit subcategory produces exactly the same empty log. The check, shortened from the tool so it runs on its own in an elevated PowerShell session on the NPS server itself (&lt;code&gt;auditpol&lt;/code&gt; needs administrator rights):&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Locale independent GUID of the &apos;Network Policy Server&apos; audit subcategory&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$AuditGuid &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;{0CCE9243-69AE-11D9-BED3-505054503030}&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$out &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;ordered&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; AuditPolicy &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;null&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Notes &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$raw &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; &amp;amp;&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; auditpol.exe&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; /&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;get &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;/&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;subcategory:&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$AuditGuid&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; 2&amp;gt;&amp;amp;1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Out-String&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$m &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;regex&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::Match&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$raw&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;    &apos;(?m)^\s*\S.*?\s{2,}(No Auditing|Success and Failure|Success|Failure)\s*$&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$out.AuditPolicy &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$m.Success&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $m.Groups&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Value &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; else&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $raw.Trim&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$out.AuditPolicy &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;No Auditing&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $out.Notes &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;NPS auditing is disabled. The Security log will stay empty.&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$out&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The same preflight lists the RADIUS clients configured on each server. A client list that does not contain the access point subnet means requests are never audited at all. In my experience that explains silence more often than a broken audit policy does. My working explanation was that NPS never reached a decision to log.&lt;/p&gt;
&lt;h2 id=&quot;working-it-out&quot;&gt;Working it out&lt;/h2&gt;
&lt;p&gt;With nothing to read on the server side, the evidence had to come from the clients. This is where most of the time went.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;netsh misreports where a profile came from.&lt;/strong&gt; &lt;code&gt;netsh wlan show profiles&lt;/code&gt;, run on the client, prints a heading &lt;code&gt;Group policy profiles (read only)&lt;/code&gt;. It said &lt;code&gt;&amp;lt;None&amp;gt;&lt;/code&gt;. The GPO profiles were in fact on the machine, listed under &lt;code&gt;User profiles&lt;/code&gt;. That output produced wrong conclusions before anyone noticed it was wrong. The reliable test is to try to delete the profile. A GPO-delivered profile refuses:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;You do not have sufficient privileges or the profile &quot;X&quot; on interface &quot;Wi-Fi&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;is a group policy profile.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The authoritative client-side view is event 14001 in &lt;code&gt;Microsoft-Windows-WLAN-AutoConfig/Operational&lt;/code&gt;, which lists exactly which profiles the wireless group policy applied and which it could not apply.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Wireless policy does not merge.&lt;/strong&gt; This was the turning point. The corporate wireless GPO showed up among the applied GPOs on the failing machines, so it was easy to assume its profiles were in use. That is not how wireless policy works. The GPO that wins precedence delivers the whole profile list, and every other wireless GPO is ignored completely. &lt;code&gt;gpresult /scope computer /r&lt;/code&gt;, run elevated, lists the applied GPOs in precedence order, highest first. It is the quickest way to see which wireless GPO actually wins. Being in the list does not mean its wireless policy is used.&lt;/p&gt;
&lt;p&gt;One site had its own wireless GPO, linked closer to the computer objects and therefore with higher precedence. It delivered its own version of the profile, while the corporate GPO sat in the applied list doing nothing. The symptom on a client was that the profile carried two SSIDs where a reference machine had one, and a completely different list of trusted roots. Comparing against a known good machine is what made it visible.&lt;/p&gt;
&lt;p&gt;The site profile pinned the old root plus a third root that did not exist in the clients’ &lt;code&gt;Root&lt;/code&gt; or &lt;code&gt;CA&lt;/code&gt; stores. So the only root that actually carried server validation on those machines was the old one, which explains why that site fell over at exactly the moment the domain controller certificates changed. The profile also had &lt;code&gt;maxAuthFailures = 1&lt;/code&gt;. After that single failure the client stops trying the corporate profile and falls back to the guest network. The user noticed nothing except that the computer “connected to the wrong thing”.&lt;/p&gt;
&lt;p&gt;The site GPO was unlinked in early August. While I was at it, an old WPA/TKIP wireless GPO, still carrying an XP policy block, turned out to be still linked in the OU tree. It was unlinked too.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Drift between clients.&lt;/strong&gt; The client diagnostic script I built during the incident now reports this kind of drift directly. It exports every 802.1X profile from a set of computers and fingerprints them. Include one machine you know is good in the set. Two computers on the same SSID should present one fingerprint. The grouping, an excerpt from the script that does not run on its own:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Excerpt. $results holds one object per computer from the collection phase,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# with Reachable, ComputerName and WlanProfiles (Name, Uses8021X, OuterEap,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# InnerEap, AuthMode, TrustedRootCA per profile).&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$profileRows &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$r &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$results &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Reachable &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$p &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$r.WlanProfiles &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Uses8021X &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Ssid        &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $p.Name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Computer    &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $r.ComputerName&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Fingerprint &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;{0}|{1}|{2}|{3}&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -f&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $p.OuterEap&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $p.InnerEap&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                          $p.AuthMode&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$p.TrustedRootCA&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -join&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;,&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$drift &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $profileRows &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Group-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Ssid &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; ForEach-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        Ssid     &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        Variants &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Group &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Group-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Fingerprint&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Count&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;More than one variant means the wireless configuration itself differs, and no amount of certificate work will fix it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Every wireless GPO that lacks the new root.&lt;/strong&gt; Once it was clear that more than one GPO could carry wireless profiles, I wanted all of them. This loop reads every GPO report, keeps the ones with WLAN profiles, and flags whether the new root’s thumbprint appears anywhere in them (thumbprint replaced with a fake one). It runs on any domain-joined machine with the GroupPolicy module from RSAT, and since it pulls a full XML report for every GPO it can take a while in a large domain:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Import-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; GroupPolicy&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-GPO&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;All &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; ForEach-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $r &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;xml&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;](&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-GPOReport&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Guid $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Id &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ReportType Xml&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $wlan &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $r.GetElementsByTagName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;WLANProfile&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;*&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$wlan.Count &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-gt&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Name     &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.DisplayName&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Profiles &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $wlan.Count&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;            # first bytes of the new root&apos;s thumbprint (fake value here)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            HasNewCa &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$r.OuterXml &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;3f\s*9a\s*0c\s*17&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Links    &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$r.GPO.LinksTo &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;                ForEach-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.SOMPath &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;})&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -join&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos; | &apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Sort-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; HasNewCa&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Name &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Format-Table&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;AutoSize&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;the-cause&quot;&gt;The cause&lt;/h2&gt;
&lt;p&gt;All nine profiles in the corporate wireless GPO pinned only the old root for server validation, so once the domain controllers’ certificates were renewed from the new issuing CA, the clients rejected the RADIUS server on their own side. Machines under the site-specific wireless GPO got there by a different route, since their extra root did not exist on the machines and the old root had been doing all the work.&lt;/p&gt;
&lt;h2 id=&quot;the-fix&quot;&gt;The fix&lt;/h2&gt;
&lt;p&gt;The new root was added as an additional &lt;code&gt;&amp;lt;TrustedRootCA&amp;gt;&lt;/code&gt; in all nine profiles, in &lt;strong&gt;both&lt;/strong&gt; validation blocks.&lt;/p&gt;
&lt;p&gt;The second half of that sentence is the one that bites. A PEAP profile has two independent root lists, one for the outer PEAP tunnel and one for the inner EAP-TLS method, and both must accept the server. In my experience, forgetting the inner block is the most common mistake, and it does not show in the outer dialog. Nine profiles therefore meant 18 &lt;code&gt;ServerValidation&lt;/code&gt; blocks to change.&lt;/p&gt;
&lt;p&gt;At the same time, in every profile and both blocks:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The server names of all four domain controllers were set.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DisableUserPromptForServerValidation&lt;/code&gt; was set to &lt;code&gt;true&lt;/code&gt;. In the GUI this is &lt;code&gt;Don&apos;t ask user to authorize new servers or trusted CAs&lt;/code&gt; in the dropdown under &lt;code&gt;Protected EAP Properties&lt;/code&gt;, and the checkbox in &lt;code&gt;Smart Card or other Certificate Properties&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The old root was kept alongside the new one, so the change was non-breaking.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Order matters.&lt;/strong&gt; Do not turn off the user prompt until the roots and server names are verified correct. The setting does not remove the fault, it removes the question. A misconfigured network then fails silently instead of showing a dialog you can troubleshoot against.&lt;/p&gt;
&lt;p&gt;To verify, I wrote a script that evaluates per &lt;code&gt;ServerValidation&lt;/code&gt; block, not per profile, so a root added to only the outer list is caught. The scope comes from the namespace of the enclosing &lt;code&gt;EapType&lt;/code&gt;. An excerpt from the script, not meant to run on its own:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Excerpt. $node is one WLANProfile element from the GPO report:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;#   $doc = [xml](Get-Content -LiteralPath C:\Temp\gpo-wlan.xml -Raw)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;#   foreach ($node in $doc.GetElementsByTagName(&apos;WLANProfile&apos;)) { ... }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sv &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $node.GetElementsByTagName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;ServerValidation&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $scope &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;unknown&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $cur &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $sv.ParentNode&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    while&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cur&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cur.LocalName &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-eq&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;EapType&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;     (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cur.NamespaceURI &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;MsPeap&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $scope &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;outer PEAP&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;            elseif&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cur.NamespaceURI &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;EapTls&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $scope &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;inner EAP-TLS&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;            break&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $cur &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $cur.ParentNode&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;    # ...then collect TrustedRootCA, ServerNames and&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;    # DisableUserPromptForServerValidation for this block only&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;When WinRM to a domain controller is not possible, the script reads the certificate straight off the LDAPS listener on port 636. On a domain controller that is normally the same auto-enrolled certificate NPS presents inside the PEAP tunnel, and the script warns you to confirm it if the verdict is surprising. The verification run that closed the case, from an admin workstation with the GroupPolicy module (RSAT), in the folder that holds &lt;code&gt;Test-RadiusServerCertificate.ps1&lt;/code&gt;. The Ldaps transport only needs TCP 636 to each domain controller:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$gpoName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Corporate Wireless&apos;&lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;   # example value, use your wireless GPO&apos;s name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;New-Item&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ItemType Directory &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path C:\Temp &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Force &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Out-Null&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-GPOReport&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Name $gpoName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ReportType Xml &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path C:\Temp\gpo&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;wlan.xml&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;\Test-RadiusServerCertificate.ps1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; `&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ComputerName DC01.corp.example&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; DC02.corp.example&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                  DC03.corp.example&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; DC04.corp.example &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;`&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ProfileXml C:\Temp\gpo&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;wlan.xml &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Transport Ldaps &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Verbose&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;All 18 blocks have to come back with &lt;code&gt;RootMatches=True&lt;/code&gt; and &lt;code&gt;Verdict=ACCEPTED&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&quot;two-more-findings&quot;&gt;Two more findings&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;The group nobody could write to.&lt;/strong&gt; Some machines had correct GPO profiles and still could not authenticate. This is a separate failure class: the server certificate is fine and it is the client that cannot prove who it is, so NPS gets far enough to reject it, and you should expect a 6273 rather than silence. Their certificate store held only the Intune and Entra certificates, and in some cases a certificate from the old CA with a GUID as subject instead of the machine FQDN, and no SID extension. None of those can be used for machine authentication. For EAP-TLS the certificate needs the Client Authentication EKU, a clean chain, a usable private key, the machine FQDN in subject or SAN, and the SID extension that strong certificate mapping requires after KB5014754. When that certificate is missing, the thing to troubleshoot is autoenrollment, not Wi-Fi.&lt;/p&gt;
&lt;p&gt;Newly installed computers were missing from the group that controls autoenrollment of the client certificate. The Intune connector’s service account, call it &lt;code&gt;svc-intune-connector&lt;/code&gt;, had &lt;code&gt;GenericWrite&lt;/code&gt; on the two WLAN groups but &lt;strong&gt;no ACE at all&lt;/strong&gt; on the certificate group. New computers landed in the WLAN groups and never in the certificate group.&lt;/p&gt;
&lt;p&gt;The delegation was an explicit ACE per group object, not a delegation group and not inherited from the OU. That makes it easy to forget when a new group is created, and the failure does not show until someone tries to write. This lists explicit write rights on the member attribute for the two WLAN groups and the certificate group. It needs the ActiveDirectory module (RSAT), which also provides the &lt;code&gt;AD:&lt;/code&gt; drive that &lt;code&gt;Get-Acl&lt;/code&gt; reads from; the group names are examples:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Import-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ActiveDirectory   &lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# loads the AD: drive used by Get-Acl&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$memberGuid &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;bf9679c0-0de6-11d0-a285-00aa003049e2&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$allProps   &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;00000000-0000-0000-0000-000000000000&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;$(&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$name &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;WLAN-Computers&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;WLAN-Office&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Cert-ClientAuth&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $dn &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-ADGroup&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $name&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.DistinguishedName&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-Acl&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;AD:\&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $dn&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Access &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        ([&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;string&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.ActiveDirectoryRights &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;            &apos;WriteProperty|GenericAll|GenericWrite&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -and&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$memberGuid&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $allProps&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -contains&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;string&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.ObjectType&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -and&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.AccessControlType &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-eq&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Allow&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -and&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (-not&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.IsInherited&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Select-Object&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; n &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Group&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; e &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $name &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; },&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;                      @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; n &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Trustee&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; e &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.IdentityReference.Value &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;})&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Format-Table&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;AutoSize&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Without the &lt;code&gt;-not $_.IsInherited&lt;/code&gt; filter, the interesting rows drown in inherited ACEs. A scheduled group sync now acts as a safety net, but the fix is the connector’s permission.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Three sites still failing.&lt;/strong&gt; By early August the broad outage was over. The profiles were fixed, all 18 blocks verified, the competing site GPO gone. Three sites kept failing, one of them the site whose GPO I had just unlinked. The conclusion above, that pinned roots were the cause, holds for the broad outage but not for those three. Part 2 is about the RADIUS servers those sites actually talked to.&lt;/p&gt;
&lt;h2 id=&quot;what-i-take-with-me&quot;&gt;What I take with me&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Silence is not innocence: an empty NPS log can be the symptom, because a client that abandons the exchange leaves NPS with no decision to log.&lt;/li&gt;
&lt;li&gt;The tool shows a summary, not the stored state: netsh reported no Group Policy profiles while they were sitting on the machine.&lt;/li&gt;
&lt;li&gt;Wireless Group Policy does not merge, so the GPO that wins precedence is the only one that counts.&lt;/li&gt;
&lt;li&gt;A PEAP profile has two root lists, and a renewal that updates only the outer one is half a fix.&lt;/li&gt;
&lt;li&gt;Diff against a working reference machine; one extra SSID in a profile said more than any log.&lt;/li&gt;
&lt;li&gt;Before every certificate renewal, inventory everything that pins the old root.&lt;/li&gt;
&lt;/ul&gt;
</content:encoded><category>finding</category><category>wifi</category><category>radius</category><category>pki</category></item><item><title>The Wi-Fi outage that NPS never saw, part 2: a binding, a wall and a proxy from 1996</title><link>https://exoit.se/posts/wifi-outage-nps-never-saw-part-2/</link><guid isPermaLink="true">https://exoit.se/posts/wifi-outage-nps-never-saw-part-2/</guid><description>Three faults in series behind one Wi-Fi outage: a dead NPS certificate binding, a blocked CRL and a proxy that only speaks HTTP/1.0.</description><pubDate>Sat, 03 Oct 2026 06:40:00 GMT</pubDate><content:encoded>&lt;p&gt;In &lt;a href=&quot;/posts/wifi-outage-nps-never-saw-part-1/&quot;&gt;part 1&lt;/a&gt; a PKI migration broke 802.1X Wi-Fi for
a manufacturing group with sites in several European countries, and the first fix was in the
WLAN profiles, which pinned the old root CA. That fix brought most sites back. Three sites
kept failing.&lt;/p&gt;
&lt;p&gt;Those three had something odd in common. The clients logged explicit EAP failures, yet the
machines never showed up in the NPS logs. Not a reject, not a discard, nothing. All three
reached the NPS servers, NPS01 and NPS02, through the hosting partner’s RADIUS proxy.&lt;/p&gt;
&lt;p&gt;It turned out to be three faults in series, and each fix only revealed the next one: an NPS
policy bound to a certificate that no longer existed, a CRL the servers could not fetch, and
a proxy that talks to the internet in HTTP/1.0.&lt;/p&gt;
&lt;h2 id=&quot;the-symptom&quot;&gt;The symptom&lt;/h2&gt;
&lt;p&gt;From the client side it looked like any other certificate problem. EAP started, EAP failed.
From the server side it looked like nothing at all. The NPS accounting logs and the Security
log had no trace of the failing machines, so the obvious reading was that the traffic never
arrived.&lt;/p&gt;
&lt;p&gt;That reading was wrong. NPS logs authentication attempts. It
grants or denies, and writes down which. If the EAP conversation dies before there is
anything to grant or deny, there is nothing to log. Silence in the NPS log does not mean the
request never came. It can also mean NPS could not even start the conversation.&lt;/p&gt;
&lt;h2 id=&quot;working-it-out&quot;&gt;Working it out&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Fault one: a binding to a certificate that was gone.&lt;/strong&gt; At the end of July, during the PKI
migration, old server certificates were cleaned off both NPS servers. A PEAP policy points
at its server certificate by SHA1 thumbprint, and that binding does not follow anything.
Delete the certificate and the policy keeps pointing at a thumbprint that is not there.&lt;/p&gt;
&lt;p&gt;A PEAP policy without its server certificate cannot initiate EAP. The client sees a broken
handshake. NPS writes nothing, because there was never an authentication attempt.&lt;/p&gt;
&lt;p&gt;The reason this was hard to see is the NPS console. Open
&lt;code&gt;Protected EAP Properties&lt;/code&gt; on a policy whose certificate has been deleted, and the dropdown
shows the first suitable installed certificate: right name, right issuer, right expiry date.
It looks completely healthy. It is a pre-selected default, not the stored value. Only when
you pick the certificate and press OK does a working thumbprint get written back.&lt;/p&gt;
&lt;p&gt;The only reliable way to see the real binding is to export the configuration and read the
thumbprint out of the stored blob. The export runs on the NPS server itself, from an
elevated PowerShell prompt; the &lt;code&gt;nps&lt;/code&gt; netsh context exists only where the NPS role is
installed, so on a workstation netsh answers that the command was not found:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# On the NPS server, elevated&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cfgFile &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &quot;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;env:&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;TEMP&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;\nps-config.xml&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;netsh nps export filename&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cfgFile&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; exportPSK&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;YES&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;exportPSK=YES&lt;/code&gt; is not optional. netsh refuses to export at all without it, so the export
contains shared secrets and has to be handled accordingly.&lt;/p&gt;
&lt;p&gt;In the export, the binding is not where you would look first. The settings for one policy are
split across two subtrees joined by node name. Conditions, sequence and &lt;code&gt;Policy_Enabled&lt;/code&gt; sit
under &lt;code&gt;NetworkPolicy&lt;/code&gt;, while &lt;code&gt;msEAPConfiguration&lt;/code&gt; sits under &lt;code&gt;RadiusProfiles&lt;/code&gt;. Read only the
policy nodes and you find no certificate bindings at all.&lt;/p&gt;
&lt;p&gt;Inside the &lt;code&gt;msEAPConfiguration&lt;/code&gt; blob the server certificate hashes are stored as a counted
list: a DWORD count of 1, a DWORD length of 0x14, then the twenty hash bytes, both DWORDs
little endian. The core of the parser, shortened from my NPS configuration script, reads
every &lt;code&gt;msEAPConfiguration&lt;/code&gt; value under &lt;code&gt;RadiusProfiles&lt;/code&gt; from the export above (an
octet string written as hex), scans it, and checks each hash against the machine store.
It runs in the same session on the NPS server and works in Windows PowerShell 5.1:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$xml   &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;xml&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;](&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-Content&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;LiteralPath $cfgFile &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Raw&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$xpath &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &quot;//*[local-name()=&apos;RadiusProfiles&apos;]//*[local-name()=&apos;msEAPConfiguration&apos;]&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$found &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;System.Collections.Generic.List&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;string&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::new&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cfgNode &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$xml.SelectNodes&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$xpath&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $hex  &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cfgNode.InnerText &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-replace&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;^0[xX]&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -replace&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;[^0-9A-Fa-f]&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $Blob &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;byte&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[]]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::new&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$hex.Length &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;/&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    for&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$k &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $k &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-lt&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob.Length&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $k&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;++)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$k&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Convert&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::ToByte&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$hex.Substring&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$k &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;),&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 16&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    for&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $i &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-le&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob.Length &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 28&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $i&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;++)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;        # DWORD count = 1, little endian&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -or&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -or&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -or&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; continue&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;        # DWORD length = 0x14 (20 bytes, a SHA1 hash)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 4&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0x14&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -or&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 5&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -or&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -or&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 7&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; -ne&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; continue&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $sb &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; New-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; System.Text.StringBuilder&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        for&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$j &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $j &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-lt&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 20&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $j&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;++)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;            [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;void&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sb.AppendFormat&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;{0:X2}&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Blob&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$i &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 8&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $j&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;])&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $tp &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $sb.ToString&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (-not&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $found.Contains&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$tp&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $found.Add&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$tp&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$tp &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $found&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $state &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;NOT INSTALLED&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Test-Path&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;LiteralPath &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;Cert:\LocalMachine\My\&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$tp&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $state &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;installed&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;    &apos;{0}  {1}&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -f&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $tp&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $state&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Remove-Item&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;LiteralPath $cfgFile   &lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# the export contains the shared secrets&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The thumbprint that comes out is then compared against the store, and anything not present
is flagged &lt;code&gt;NOT INSTALLED&lt;/code&gt;. Export before and after the fix on NPS01:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;before  3f9a0c17...   not present on the server&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;after   8d2e6b40...   valid, issued by the legacy issuing CA&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Same certificate name in the dialog both times.&lt;/p&gt;
&lt;p&gt;There was a second trap in the fix itself. The certificate has to be set in three places per
policy. Under &lt;code&gt;Constraints&lt;/code&gt;, &lt;code&gt;Authentication Methods&lt;/code&gt;, there are
&lt;code&gt;Microsoft: Smart Card or other certificate&lt;/code&gt; and &lt;code&gt;Microsoft: Protected EAP (PEAP)&lt;/code&gt; as separate
EAP types, each with its own binding, and inside the PEAP dialog there is the inner method’s
binding. Set only one and the others keep pointing at the deleted certificate, and which one
wins depends on what the client negotiates. The target state from the script is a single row
per policy with no &lt;code&gt;NOT INSTALLED&lt;/code&gt;. Two rows mean one of the three places was missed.&lt;/p&gt;
&lt;p&gt;The binding was re-selected on NPS01 first, all three places, and the export confirmed it.
Then NPS02. One of the three sites confirmed working Wi-Fi after this step alone.&lt;/p&gt;
&lt;p&gt;A side finding: NPS02 receives no RADIUS traffic at all, because the hosting partner’s proxy
only sends to NPS01. That makes NPS02 useless as a canary for a certificate change, and it
means the whole path hangs on one server.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Fault two: the revocation server was offline.&lt;/strong&gt; After the binding fix the error changed.
It now appeared in the log, as reason code 259:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;The revocation function was unable to check revocation because the revocation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;server was offline&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is progress, even if it does not feel like it. With the binding fixed, NPS was now
getting far enough to validate the client certificate, and for EAP-TLS that includes a
revocation check. The clients presented certificates from the new issuing CA, whose CDP is a
CRL on Azure Storage, &lt;code&gt;http://pki.example.com/cdp/IssuingCA1.crl&lt;/code&gt;. From the NPS servers that
traffic went through the hosting partner’s tinyproxy pair, PROXY01 and PROXY02, which
answered 403.&lt;/p&gt;
&lt;p&gt;The host was added to the tinyproxy filter file on both proxies. 403 became 505.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Fault three: HTTP/1.0 in 2026.&lt;/strong&gt; The new answer was:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;505 Http Version Not Supported&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;tinyproxy 1.11.0 hard-codes HTTP/1.0 upstream, so there was nothing to configure. Azure
Storage requires HTTP/1.1. That was verified from the proxy itself:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;curl --http1.0 http://pki.example.com/cdp/IssuingCA1.crl   -&amp;gt;  505&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;curl --http1.1 http://pki.example.com/cdp/IssuingCA1.crl   -&amp;gt;  200&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The fix was to take the proxy out of the path entirely for that one host.&lt;/p&gt;
&lt;h2 id=&quot;the-test-that-lied-and-the-one-that-did-not&quot;&gt;The test that lied, and the one that did not&lt;/h2&gt;
&lt;p&gt;A firewall rule opened direct outbound TCP 80 and 443 from the NPS servers to the CDP
addresses. &lt;code&gt;Test-NetConnection -Port 80&lt;/code&gt; against the CDP host now returned
&lt;code&gt;TcpTestSucceeded True&lt;/code&gt;. CryptoAPI still failed.&lt;/p&gt;
&lt;p&gt;The port test proved that a TCP connection could be made. It said nothing about which path
CryptoAPI takes, and CryptoAPI uses the WinHTTP machine proxy (for a service like NPS),
which is configured separately from everything else. The CDP host was still being sent to
tinyproxy. It had to go on the WinHTTP bypass list on both servers, checked on each NPS
server (no elevation needed to read it) with:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;netsh winhttp show proxy&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There is a second trap here. &lt;code&gt;Invoke-WebRequest&lt;/code&gt; does not respect the WinHTTP machine proxy,
so a successful download with it proves nothing about what CryptoAPI will do. Both of the
easy tests answer a question nobody asked.&lt;/p&gt;
&lt;p&gt;The test that answers the right question is to build the chain with online revocation, for
a certificate from the new issuing CA, and read the status. Run it on the NPS server, so
that the fetch leaves from the same machine and network path as NPS:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example: any certificate issued by the new issuing CA. Adjust the filter, or load an&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# exported client certificate instead:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# $cert = [Security.Cryptography.X509Certificates.X509Certificate2]::new(&apos;C:\Temp\c.cer&apos;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cert &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-ChildItem&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Cert:\LocalMachine\My &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Where-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Issuer &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-like&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;*Issuing*&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Select-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;First &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$chain &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;System.Security.Cryptography.X509Certificates.X509Chain&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::new&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$chain.ChainPolicy.RevocationMode &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Online&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$chain.ChainPolicy.RevocationFlag &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;ExcludeRoot&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$ok &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $chain.Build&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$cert&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$ok&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$chain.ChainStatus &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; ForEach-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Status &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then the URL cache was cleared and the NPS service (&lt;code&gt;IAS&lt;/code&gt;) restarted, on each NPS server
from an elevated prompt:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;certutil &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;urlcache &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; delete&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Restart-Service&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Name IAS&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;After that the chain built clean on both servers, &lt;code&gt;ChainValid True&lt;/code&gt; with an empty
&lt;code&gt;ChainStatus&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The binding fix, the proxy filter, the firewall rule, the bypass list and the cache flush
were all done within one day. One site had confirmed working Wi-Fi after the binding fix.
Another had not retested by the end of that day; its last rejection was a 259 logged before
the CDP was opened. The third site’s result is not in my notes.&lt;/p&gt;
&lt;h2 id=&quot;keeping-it-fixed&quot;&gt;Keeping it fixed&lt;/h2&gt;
&lt;p&gt;A direct firewall rule to Azure is an IP-based rule against a name whose addresses rotate.
It works until it silently does not, and the symptom then shows up as reason code 259 on a
Wi-Fi network rather than as a firewall problem. So there is now a watcher. It resolves the
CDP host against external resolvers only, deliberately skipping the internal zone, unions the
answers over several queries per resolver because Azure front ends hand out different
subsets, and compares against the last run. The query at its core, runnable on any
Windows machine (the resolvers are examples):&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$HostName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;pki.example.com&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$s &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;1.1.1.1&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;8.8.8.8&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $answer &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Resolve-DnsName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Name $HostName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Type A &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Server $s &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;`&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;                -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;DnsOnly &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;NoHostsFile &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ErrorAction Stop &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.PSObject.Properties.Name &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-contains&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;IPAddress&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -and&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.IPAddress&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        })&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;    &apos;{0}: {1}&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -f&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $s&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; ((&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$answer.IPAddress &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Sort-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Unique&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -join&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;, &apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It runs every four hours, mails immediately on a change, and otherwise sends one healthy
report per day. The daily mail is the point: an absent report means the job itself has
stopped, which a change-only watcher can never tell you.&lt;/p&gt;
&lt;p&gt;The proxies got a clean-up script too, which normalises the tinyproxy configuration and
filter file, runs dry by default and never adds anything to the allowlist.&lt;/p&gt;
&lt;p&gt;At the time of the fix the re-selected certificate was still issued by the legacy PKI, and
it had to be replaced before the old root’s CRL NextUpdate in late September. That date, not
the CA certificate’s expiry, is the real deadline for a retiring CA: after it every
revocation check against the old hierarchy fails, server and client certificates alike.&lt;/p&gt;
&lt;p&gt;Enterprise CAs publish their CRLs into the configuration partition under &lt;code&gt;CN=CDP&lt;/code&gt;, so the
dates can be read without touching the CA. I parse the DER directly rather than the output of
&lt;code&gt;certutil -dump&lt;/code&gt;, because certutil localises its field names. With my own script,
&lt;code&gt;Get-PkiCrlStatus.ps1&lt;/code&gt; (not published here), the list that matters during a decommission
is:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Reads every cRLDistributionPoint object in the configuration partition under&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# CN=CDP,CN=Public Key Services,CN=Services, with Get-ADObject. It needs the&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# ActiveDirectory module (RSAT) and only reads, it never contacts the CA.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;\Get-PkiCrlStatus.ps1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.DaysRemaining &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-lt&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 60&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Sort-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; DaysRemaining&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;silent-zeros&quot;&gt;Silent zeros&lt;/h2&gt;
&lt;p&gt;One bug in my own tooling surfaced along the way, and it was of the same kind as the
outage: a zero that was read as a finding. The NPS configuration script read only the policy
nodes, not &lt;code&gt;RadiusProfiles&lt;/code&gt;, looked for &lt;code&gt;msNPPolicyEnabled&lt;/code&gt; instead of &lt;code&gt;Policy_Enabled&lt;/code&gt;, and
so reported zero certificate bindings and every policy as enabled. Worse, it searched the
blob for thumbprints already known to be installed.&lt;/p&gt;
&lt;p&gt;That last part is the principled one. Searching for certificates you know exist can only ever
confirm a match. It is structurally blind to the case that matters: a policy bound to a
certificate that is missing.&lt;/p&gt;
&lt;h2 id=&quot;what-i-take-with-me&quot;&gt;What I take with me&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;The error you see is the last link in a chain, and fixing it may only promote the next one.&lt;/li&gt;
&lt;li&gt;Silence is not innocence: an empty NPS log can mean NPS never got far enough to log.&lt;/li&gt;
&lt;li&gt;The UI shows a default, not the stored value, so read the export, not the dialog.&lt;/li&gt;
&lt;li&gt;Test from the right vantage point: a port test and &lt;code&gt;Invoke-WebRequest&lt;/code&gt; do not travel the
path CryptoAPI travels, but building the chain with online revocation does.&lt;/li&gt;
&lt;li&gt;A script that searches for what it expects can only confirm, never discover.&lt;/li&gt;
&lt;li&gt;The deadline for a retiring CA is its CRL NextUpdate, not its certificate expiry.&lt;/li&gt;
&lt;/ul&gt;
</content:encoded><category>finding</category><category>radius</category><category>pki</category><category>wifi</category><category>method</category></item><item><title>The error code in the portal is the fourth link in the chain</title><link>https://exoit.se/posts/error-code-fourth-link-in-the-chain/</link><guid isPermaLink="true">https://exoit.se/posts/error-code-fourth-link-in-the-chain/</guid><description>Azure Update Manager said 0x80240007, a corrupt datastore. The real cause was a WinHTTP proxy nobody remembered, three layers further down.</description><pubDate>Sat, 03 Oct 2026 06:30:00 GMT</pubDate><content:encoded>&lt;p&gt;Two Arc-enabled servers at one site of a customer, a manufacturing group with sites in several European countries, stopped reporting to Azure Update Manager in August. The second server had the same symptom. The file server, FS01, showed “assessment failed” with 813 accumulated failures and the HRESULT &lt;code&gt;0x80240007&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;That code has a reputation. Search for it and almost every answer tells you the Windows Update datastore is corrupt and that you should clear out &lt;code&gt;SoftwareDistribution&lt;/code&gt;. It is a reasonable reflex, and here it would have achieved nothing except a long rescan against a proxy that still did not answer, and an empty local update history.&lt;/p&gt;
&lt;p&gt;The actual cause was a WinHTTP proxy left behind from a period when the servers sat in the wrong OU. The servers could not reach it, and the error changed its code three times on the way up to the portal. This post is about how to read past the code you are shown.&lt;/p&gt;
&lt;h2 id=&quot;what-i-saw&quot;&gt;What I saw&lt;/h2&gt;
&lt;p&gt;In the portal: assessment failed, &lt;code&gt;0x80240007&lt;/code&gt;, which is &lt;code&gt;WU_E_INVALID_INDEX&lt;/code&gt;. Azure Update Manager adds its own interpretation on top:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;SearchForUpdateWasInterruptedDueToCancellationRequestedorTimeExpired&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That is literally true. The scan ran for 174 plus 415 seconds before it gave up. But it points at a timeout as the cause, and that turned out to be wrong.&lt;/p&gt;
&lt;p&gt;Arc itself looked healthy. The machines were &lt;code&gt;Connected&lt;/code&gt; with a fresh heartbeat. On the server, the patch extension’s &lt;code&gt;state.json&lt;/code&gt; showed &lt;code&gt;ExtensionState: ENABLED&lt;/code&gt; with an empty &lt;code&gt;ErrorMsg&lt;/code&gt;, and the &lt;code&gt;*_stdout.txt&lt;/code&gt; file held nothing but the launcher line. Everything visible said the plumbing was fine and the scan simply failed.&lt;/p&gt;
&lt;p&gt;The last successful assessment was some time before the middle of August.&lt;/p&gt;
&lt;h2 id=&quot;working-it-out&quot;&gt;Working it out&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Read the ETL, not the summary.&lt;/strong&gt; Everything of value sits in the extension’s &lt;code&gt;windowsUpdateLog&lt;/code&gt; folder as binary ETL traces. They have to be decoded first, and &lt;code&gt;Get-WindowsUpdateLog&lt;/code&gt; takes an &lt;code&gt;-ETLPath&lt;/code&gt;, so you can point it at the extension’s own trace folder. Run this in an elevated PowerShell session on the Arc-enabled server itself:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$ext &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;C:\ProgramData\GuestConfig\extension_logs\&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; +&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;       &apos;Microsoft.CPlat.Core.WindowsPatchExtension&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$log &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;C:\Temp\WindowsUpdate-decoded.log&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;New-Item&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ItemType Directory &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Split-Path&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $log&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Force &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Out-Null&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Decode the extension&apos;s ETL trace, the only place the root error is visible.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# -ETLPath accepts a folder and decodes every .etl file in it.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-WindowsUpdateLog&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ETLPath &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$ext&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;\windowsUpdateLog&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;LogPath $log&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# The whole block of the last federated search, not just the error lines.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Filtering on FAILED misses the WebServices layer completely.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$end &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Select-String&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path $log &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Pattern &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;All federated searches have completed&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        Select-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Last &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.LineNumber&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (-not&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $end&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; throw&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &quot;No completed federated search found in &lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$log&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-Content&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $log &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Select-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Skip &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;([&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Math&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::Max&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $end &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 170&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;First &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;180&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The second half is the part that matters. Filtering on FAILED misses the WebServices layer completely. The lines that explain the failure are not marked as failures; they sit inside the same federated-search block.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why two jobs died.&lt;/strong&gt; Modern WUA fans a search out to every registered update service. On FS01 there were two in play:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Service&lt;/th&gt;
&lt;th&gt;ServiceID&lt;/th&gt;
&lt;th&gt;Endpoint&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Microsoft Update&lt;/td&gt;
&lt;td&gt;&lt;code&gt;7971f918-a847-4430-9279-4a52d1efe18d&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;tas02.cwsmu.update.microsoft.com&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DCat Flighting Prod&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8b24b027-1dee-babb-9a95-3517dfb9c552&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;fe3cr.delivery.mp.microsoft.com&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The log said &lt;code&gt;Jobs = 2, Succeeded = 0&lt;/code&gt;. Both child jobs had failed, which invites you to suspect the service registrations. They were correct the whole time. Four registered services (Microsoft Update, DCat Flighting, Windows Store, Windows Update) is normal and nothing to fix. Both jobs simply went through the same path, and the path was dead.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The line that gave it away&lt;/strong&gt; sat inside the federated search block:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;WebServices  Using default WinHTTP proxy settings: proxy name=&apos;PROXY01.corp.example:8888&apos;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;             bypass list=&apos;*.corp.example,&amp;lt;local&amp;gt;,*.example.com&apos;, auth scheme=&apos;0&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;WebServices  WS error: The given proxy cannot be reached.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;WebServices  MapToSusHResult mapped Nws error 0x803d0010 to 0x80240438&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A proxy, on a server that was supposed to go direct. The bypass list did not exempt any Microsoft endpoints, so every Windows Update call went to it.&lt;/p&gt;
&lt;p&gt;With that line in hand, the rest of the log reads as a chain. The error propagates up through four layers and changes its code at each one:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Code&lt;/th&gt;
&lt;th&gt;What actually happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;WinHTTP / NWS&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0x803D0010&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The given proxy cannot be reached&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WU ProtocolTalker&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0x80240438&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;WU_E_PT_ENDPOINT_UNREACHABLE&lt;/code&gt;, SyncUpdates fails after 3 retries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ComApi FederatedSearchJob&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0x8024000B&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;WU_E_CALL_CANCELLED&lt;/code&gt;, both child jobs dead, parent cancelled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;UpdateExceptionCollection&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0x80240007&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;WU_E_INVALID_INDEX&lt;/code&gt;, empty exception list read, index out of range&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;code&gt;0x80240007&lt;/code&gt; is raised at line 44 of &lt;code&gt;UpdateExceptionCollection.cpp&lt;/code&gt;, when Windows Update tries to read the errors out of a job that has already been cancelled. The list is empty. It is an artefact of the error handling, not a state of the machine. There was no sign of a datastore problem, and clearing it would not have helped.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Test-NetConnection gave false negatives.&lt;/strong&gt; The obvious network test was misleading. To calibrate it, I ran it on DC01, a domain controller that demonstrably fetched updates without trouble. &lt;code&gt;Test-NetConnection -Port 443&lt;/code&gt; reported &lt;code&gt;False&lt;/code&gt; against &lt;code&gt;fe3cr.delivery.mp.microsoft.com&lt;/code&gt;, &lt;code&gt;tas02.cwsmu.update.microsoft.com&lt;/code&gt; and &lt;code&gt;sls.update.microsoft.com&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;A working machine failing the test means the test is wrong, not the machine. The most likely reason: these endpoints sit behind Akamai and Azure Front Door, which rotate A records with short TTLs, and the firewall’s FQDN objects resolve their own list of IPs. A raw TCP connect to an IP outside that list fails. Without &lt;code&gt;-Port&lt;/code&gt; it only pings, which is useless when ICMP is blocked, as it was here.&lt;/p&gt;
&lt;p&gt;So I wrote a small replacement that does a real TCP connect followed by a TLS handshake with the correct SNI, and returns the certificate issuer. The issuer is a bonus: if something in the path does TLS inspection, it shows up there. The function runs in Windows PowerShell 5.1 or PowerShell 7 on any machine, without elevation:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;function&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Test-TlsEndpoint&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;CmdletBinding&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    param&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Parameter&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;Mandatory&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt; ValueFromPipeline&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)][&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;string&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$HostName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$Port &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 443&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$TimeoutMs &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 10000&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    )&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    process&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $client &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; New-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; System.Net.Sockets.TcpClient&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $ssl    &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;null&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        try&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $task &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $client.ConnectAsync&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$HostName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $Port&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (-not&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $task.Wait&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$TimeoutMs&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;                return&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                    Endpoint &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $HostName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Tcp &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;false&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Tls &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;false&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Issuer &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;null&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                    Detail   &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &quot;TCP connect timed out after &lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;TimeoutMs&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;ms&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;            }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $ssl &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; New-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; System.Net.Security.SslStream&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                $client.GetStream&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(),&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;false&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; param&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$s&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$c&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$ch&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$e&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; })&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $ssl.AuthenticateAsClient&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$HostName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;   # SNI is the host name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $cert &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Security.Cryptography.X509Certificates.X509Certificate2&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;](&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                $ssl.RemoteCertificate&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;            [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Endpoint &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $HostName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Tcp &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Tls &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                Issuer &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $cert.Issuer&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Detail &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $cert.Subject &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        catch&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;            [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                Endpoint &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $HostName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Tcp &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $client.Connected&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Tls &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;false&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;                Issuer &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;null&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Detail &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Exception.Message &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        finally&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$ssl&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $ssl.Dispose&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; };&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $client.Dispose&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;fe3cr.delivery.mp.microsoft.com&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;sls.update.microsoft.com&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Test-TlsEndpoint&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It is wrapped in a function, &lt;code&gt;Test-TlsEndpoint&lt;/code&gt;, with &lt;code&gt;HostName&lt;/code&gt; taking pipeline input, &lt;code&gt;Port&lt;/code&gt; defaulting to 443 and &lt;code&gt;TimeoutMs&lt;/code&gt; to 10000, so you can pipe a list of endpoints into it. Even so, the authoritative test is not a network probe at all. It is a real WUA COM scan, shown further down.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Diff against the machine that works.&lt;/strong&gt; DC01 was the reference: &lt;code&gt;Direct access&lt;/code&gt;, and updates arriving without complaint. FS01 used the proxy. That was the difference that mattered.&lt;/p&gt;
&lt;p&gt;The next question was whether the proxy was down or just unreachable from where FS01 stood. I tested tcp/8888 from both subnets:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;tcp/8888 to PROXY01&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;FS01&lt;/td&gt;
&lt;td&gt;Fail against both proxy nodes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A reference machine in another subnet&lt;/td&gt;
&lt;td&gt;OK&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The proxy was alive. FS01’s network segment simply had no path to it, and never should have needed one.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why Arc was fine.&lt;/strong&gt; How can a machine be &lt;code&gt;Connected&lt;/code&gt; in Arc and unable to reach Microsoft at the same time? The Arc agent has its own proxy setting, &lt;code&gt;proxy.url&lt;/code&gt;, and on these servers it was empty. The agent went straight out and worked. Windows Update used the machine’s WinHTTP proxy and did not.&lt;/p&gt;
&lt;p&gt;Two separate network paths, two separate failure sources. A green Arc status with a fresh heartbeat says nothing about whether Windows Update works on the same machine.&lt;/p&gt;
&lt;h2 id=&quot;the-cause&quot;&gt;The cause&lt;/h2&gt;
&lt;p&gt;The proxy is meant for branch offices and for one partner’s delivery. Servers at this site should never go through it. No GPO was setting it when I looked, so it was a leftover from a period when the servers had sat in the wrong OU.&lt;/p&gt;
&lt;p&gt;The WinHTTP proxy lives as a binary blob in &lt;code&gt;HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\WinHttpSettings&lt;/code&gt;. That is not a real policy key under &lt;code&gt;\Policies\&lt;/code&gt;, so it is not cleaned up when the GPO stops applying. Once written, it stays until someone removes it on purpose.&lt;/p&gt;
&lt;h2 id=&quot;the-fix&quot;&gt;The fix&lt;/h2&gt;
&lt;p&gt;Run this in an elevated PowerShell session on the affected server itself. The WUA COM search does not work over a remote session, so run it locally or over RDP:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;netsh winhttp reset proxy&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;netsh winhttp show proxy   &lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# should show Direct access (no proxy server)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Restart-Service&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; wuauserv&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; bits&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; DoSvc &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Force&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Defender sat behind the same dead proxy, see the side finding below&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# May be blocked by Tamper Protection&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Restart-Service&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; WinDefend &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Force &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ErrorAction SilentlyContinue&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Authoritative test. This, not Test-NetConnection, decides whether it works.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sw &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Diagnostics.Stopwatch&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::StartNew&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;try&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $searcher &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;New-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ComObject Microsoft.Update.Session&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.CreateUpdateSearcher&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $res &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $searcher.Search&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;IsInstalled=0 and IsHidden=0&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;    &quot;OK after &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;$([&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sw.Elapsed.TotalSeconds&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;s, &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;$(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$res.Updates.Count&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; updates found&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;catch&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;    # PowerShell wraps the COMException; the WU error code is on the inner exception&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $e  &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Exception&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $hr &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$e.InnerException&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $e.InnerException.HResult &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; else&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $e.HResult &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;    &quot;FAILED after &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;$([&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sw.Elapsed.TotalSeconds&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;s: 0x{0:X8}&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -f&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $hr&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then confirm that Windows Update itself has stopped using the old proxy string, rather than trusting &lt;code&gt;netsh&lt;/code&gt;. Same elevated session on the server; &lt;code&gt;C:\Temp&lt;/code&gt; was created in the first block:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-WindowsUpdateLog&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;LogPath C:\Temp\wu&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;after.log&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Select-String&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path C:\Temp\wu&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;after.log &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Pattern &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;WinHTTP proxy settings&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Select-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Last &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;5&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Does it need a reboot? Not strictly. The registry changes immediately. The problem is process caching: WinHTTP reads the configuration when a session is created with &lt;code&gt;WinHttpOpen&lt;/code&gt; and &lt;code&gt;WINHTTP_ACCESS_TYPE_DEFAULT_PROXY&lt;/code&gt;, so long-running services keep the old setting until they create a new session.&lt;/p&gt;
&lt;p&gt;The argument for a reboot is that a service restart is not deterministic. &lt;code&gt;wuauserv&lt;/code&gt; runs in a shared &lt;code&gt;svchost&lt;/code&gt; under netsvcs, and &lt;code&gt;UsoSvc&lt;/code&gt; consumes the same configuration without being on a typical restart list. A colleague who knows the environment well recommended a reboot. In this case restarting the services was enough. I treat a reboot as a safety margin, not a requirement, and always verify against the Windows Update log afterwards.&lt;/p&gt;
&lt;p&gt;One server with a leftover proxy suggests others. I wrote an inventory to run across every server in the domain; running it was on the follow-up list. It reports which servers carry a WinHTTP proxy and whether they can reach it. Run it from a domain-joined admin machine with the RSAT &lt;code&gt;ActiveDirectory&lt;/code&gt; module, as an account that is local admin on the servers, with WinRM enabled on them:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$servers &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-ADComputer&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Filter &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;OperatingSystem -like &apos;*Server*&apos;&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; `&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Properties OperatingSystem &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Select-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ExpandProperty Name&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Invoke-Command&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ComputerName $servers &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ErrorAction SilentlyContinue &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ScriptBlock &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $proxy  &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;netsh winhttp show proxy&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -join&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos; &apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $match  &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;regex&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;::Match&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$proxy&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Proxy Server\(s\)\s*:\s*(\S+)&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $server &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$match.Success&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $match.Groups&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Value &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; else&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Direct&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $reachable &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;null&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    if&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$server &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-ne&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Direct&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;        # Handles the simple host:port form only&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $parts &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $server &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-split&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;:&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $reachable &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Test-NetConnection&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ComputerName $parts&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Port $parts&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; `&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;                        -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;WarningAction SilentlyContinue&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.TcpTestSucceeded&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        Computer       &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;env:&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;COMPUTERNAME&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        WinHttpProxy   &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $server&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        ProxyReachable &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $reachable&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Sort-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ProxyReachable &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Format-Table&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;AutoSize&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Servers that refuse remoting simply vanish from the output, so compare the row count against the list of names.&lt;/p&gt;
&lt;p&gt;The excerpt is trimmed: the full version also reports each server’s IPv4 address and sorts on ProxyReachable, then IPv4, which makes it easy to see whether the unreachable ones share a subnet.&lt;/p&gt;
&lt;h2 id=&quot;the-side-finding-that-mattered-more&quot;&gt;The side finding that mattered more&lt;/h2&gt;
&lt;p&gt;Defender had been behind the same dead proxy for as long as Windows Update. One night in August, FS01’s signature version was well behind DC01’s: ten days without fresh definitions on a production file server. That is more serious than the missing patch reporting, and next time I see this pattern it is the part I escalate first. Updating signatures directly with &lt;code&gt;Update-MpSignature -UpdateSource MicrosoftUpdateServer&lt;/code&gt; on both affected servers went on the follow-up list.&lt;/p&gt;
&lt;p&gt;Nothing alerted on it. It just quietly fell behind.&lt;/p&gt;
&lt;h2 id=&quot;what-i-take-with-me&quot;&gt;What I take with me&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;The error code in the portal is the last link in a chain, so find the first one before clearing anything.&lt;/li&gt;
&lt;li&gt;Read the whole block around a failure in a decoded log, because the cause may not be logged as an error at all.&lt;/li&gt;
&lt;li&gt;When a test fails on a machine you know works, the test is wrong, so calibrate tools against a working reference first.&lt;/li&gt;
&lt;li&gt;Diff the broken machine against a working one at the same site before forming theories.&lt;/li&gt;
&lt;li&gt;A setting written by policy is not necessarily removed by policy, so check where it actually lives.&lt;/li&gt;
&lt;li&gt;A green status in one agent says nothing about the network path another component uses on the same machine.&lt;/li&gt;
&lt;/ul&gt;
</content:encoded><category>finding</category><category>azure</category><category>windows-update</category><category>method</category></item><item><title>The sensor that was stuck migrating had already arrived</title><link>https://exoit.se/posts/mdi-sensor-stuck-migrating/</link><guid isPermaLink="true">https://exoit.se/posts/mdi-sensor-stuck-migrating/</guid><description>A Defender for Identity sensor sat in Migrating for 48 hours. The v3 sensor was already healthy, hidden behind a case-insensitive portal view.</description><pubDate>Sat, 03 Oct 2026 06:20:00 GMT</pubDate><content:encoded>&lt;p&gt;A customer, a manufacturing group with sites in several European countries, had a domain
controller whose Defender for Identity sensor showed &lt;strong&gt;Disconnected&lt;/strong&gt; and &lt;strong&gt;Not healthy&lt;/strong&gt; in
the Defender XDR portal. The sensor services were stopped. That is a routine ticket: fix the
sensor or replace it.&lt;/p&gt;
&lt;p&gt;It stopped being routine when the replacement would not install, the portal refused to
delete the old sensor, and nothing moved for 48 hours. The portal said the sensor was
migrating. That entry was a leftover. The migration that mattered had already finished,
through a different channel, and the new sensor was already healthy. The portal simply showed me the wrong one of
two entries for the same host.&lt;/p&gt;
&lt;p&gt;The fix, once I could see that, took five minutes with two Graph calls, a list and a delete.&lt;/p&gt;
&lt;h2 id=&quot;what-i-saw&quot;&gt;What I saw&lt;/h2&gt;
&lt;p&gt;The first look was at DC01 itself. I used a small read-only triage script that collects the
things that usually explain a dead sensor: service state for both the legacy v2 services
(&lt;code&gt;AATPSensor&lt;/code&gt;, &lt;code&gt;AATPSensorUpdater&lt;/code&gt;) and the unified v3 ones (&lt;code&gt;MDISensor&lt;/code&gt;,
&lt;code&gt;MDISensorUpdater&lt;/code&gt;), recent warnings and errors from the two sensor event logs, the WinHTTP
proxy (&lt;code&gt;netsh winhttp show proxy&lt;/code&gt;), time sync (&lt;code&gt;w32tm /query /status&lt;/code&gt;), the five latest
hotfixes as regression candidates, and Service Control Manager events that mention the
sensor. This is the Service Control Manager part of it, run locally on DC01 in Windows
PowerShell 5.1 or PowerShell 7:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Service Control Manager events for the sensor services, last 14 days&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-WinEvent&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;FilterHashtable &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    LogName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;System&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ProviderName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Service Control Manager&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    StartTime &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-Date&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.AddDays&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;-14&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ErrorAction SilentlyContinue &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Message &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;AATP|Azure Advanced Threat|MDI Sensor&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; }&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Select-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; TimeCreated&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Id&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; LevelDisplayName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Message &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;First &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;20&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The Service Control Manager section answered the question on its own. Event 7031 for
&lt;code&gt;AATPSensorUpdater&lt;/code&gt;, “terminated unexpectedly”, with a counter of 206,727. The sensor had
been installed in 2023. It had in all likelihood never worked properly on that server.&lt;/p&gt;
&lt;p&gt;That changes the decision. A sensor that broke at the last cumulative update is worth
repairing. A sensor whose updater has crashed two hundred thousand times is not. The plan
became a clean uninstall of v2 and a fresh install.&lt;/p&gt;
&lt;p&gt;The uninstall did not complete cleanly. The &lt;code&gt;AATPSensor&lt;/code&gt; service was left in
&lt;code&gt;DELETE_PENDING&lt;/code&gt;, which a reboot clears, so the reboot went into the next night’s
window.&lt;/p&gt;
&lt;p&gt;The day before, the sensor had been marked for migration in the portal with &lt;strong&gt;Migrate&lt;/strong&gt;. That click matters later.&lt;/p&gt;
&lt;p&gt;After the reboot the host was clean, and the new install failed immediately:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Detected existing unified sensor installation, aborting installation&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The installer claimed a unified sensor already existed. As it turned out, it did. Running the
MSI directly does not get around this: it calls the same
&lt;code&gt;Microsoft.Tri.Sensor.Deployment.Package.Actions&lt;/code&gt; custom action as the bootstrapper.&lt;/p&gt;
&lt;p&gt;The portal showed the sensor as:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Migrating, Disconnected, 2.255.x, Not healthy&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and &lt;strong&gt;Delete&lt;/strong&gt; was refused:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Migration in progress, sensor removal is unavailable until the migration completes&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So the cloud side was waiting for a v2 sensor to convert into v3, the host refused a fresh
install because something unified already existed, and the portal would not let me remove the
stuck entry. Each piece blocked the next. No automatic timeout kicked in within 48 hours.&lt;/p&gt;
&lt;p&gt;A fresh cloud handshake from the host is not available: the Defender for Endpoint &lt;code&gt;Sense&lt;/code&gt;
service is tamper-protected, and re-onboarding the server is the only way to force one, which
is invasive on a domain controller.&lt;/p&gt;
&lt;h2 id=&quot;working-it-out&quot;&gt;Working it out&lt;/h2&gt;
&lt;p&gt;The useful question turned out to be a narrow one: what does the cloud actually think exists
for this host? The portal gives one answer. Graph gives the full list. The beta endpoint
&lt;code&gt;security/identities/sensors&lt;/code&gt; returns every sensor in the tenant with its id, version,
migration state and health. From any workstation with the &lt;code&gt;Microsoft.Graph.Authentication&lt;/code&gt;
module, signed in as a Security Administrator who can consent to the scope:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Invoke-MgGraphRequest only needs Microsoft.Graph.Authentication&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Install-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Microsoft.Graph.Authentication &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Scope CurrentUser&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Connect-MgGraph&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Scopes &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;SecurityIdentitiesSensors.ReadWrite.All&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$uri &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;https://graph.microsoft.com/beta/security/identities/sensors&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# PSObject output so Format-Table can read the properties&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sensors &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Invoke-MgGraphRequest&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Method GET &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Uri $uri &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;OutputType PSObject&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sensors.value &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Where-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; displayName &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;DC01&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Format-Table&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; displayName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; migrationState&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; healthStatus&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        deploymentStatus&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;-match&lt;/code&gt; is case-insensitive, which is why both rows came back. Two rows for the same host,
not one:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;displayName&lt;/th&gt;
&lt;th&gt;Version&lt;/th&gt;
&lt;th&gt;Graph showed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DC01&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;2.255.x&lt;/td&gt;
&lt;td&gt;&lt;code&gt;migrationState&lt;/code&gt; = &lt;code&gt;migrating&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;dc01&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;3.0.7&lt;/td&gt;
&lt;td&gt;&lt;code&gt;healthStatus&lt;/code&gt; = &lt;code&gt;healthy&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The two entries had different cloud-generated ids, different creation dates (the old one from
the original install, the new one from when v3 arrived) and different case in the display
name. The old entry used the uppercase form of the computer account. The new one used the
lowercase hostname.&lt;/p&gt;
&lt;p&gt;The portal groups sensors by name case-insensitively and shows one entry per host. It had
chosen the broken one. The healthy v3 sensor was there, reporting data normally, and
invisible in the UI.&lt;/p&gt;
&lt;p&gt;To confirm which entry was the real, active sensor, I checked the host’s own record. The
Graph id of the v3 entry matched the &lt;code&gt;senseId&lt;/code&gt; value under
&lt;code&gt;HKLM:\SOFTWARE\Microsoft\Windows Advanced Threat Protection&lt;/code&gt; on DC01, once the hyphens are
removed. That settled it: the lowercase entry was this server,
and the uppercase entry was an orphan.&lt;/p&gt;
&lt;h2 id=&quot;the-cause&quot;&gt;The cause&lt;/h2&gt;
&lt;p&gt;Two things happened at the same time and the portal hid one of them.&lt;/p&gt;
&lt;p&gt;First, &lt;strong&gt;Migrate&lt;/strong&gt; had been clicked on a broken v2 sensor. Migration is designed to morph a
healthy v2 sensor into v3. On a sensor that is disconnected, there is nothing to morph, and
the cloud side sits in &lt;code&gt;Migrating&lt;/code&gt; waiting for a v2 that will never report in.&lt;/p&gt;
&lt;p&gt;Second, the server was onboarded to Defender for Endpoint, and on servers onboarded to
Defender for Endpoint the unified v3 sensor can be activated through that channel without any
install, when the tenant has it enabled. That is what had happened, at the same time as the
stuck migration. It created a new,
parallel sensor entry that was healthy and up to date. It is also why the installer saw an
“existing unified sensor installation”: there was one.&lt;/p&gt;
&lt;p&gt;So the stuck migration was largely an illusion. The real migration had happened. Only the
old cloud entry was left behind, and the portal chose to show that one.&lt;/p&gt;
&lt;h2 id=&quot;the-fix&quot;&gt;The fix&lt;/h2&gt;
&lt;p&gt;The portal blocked Delete on the orphaned entry. Graph did not. In the same Graph session
as the listing above, with the id of the v2 entry in &lt;code&gt;migrating&lt;/code&gt; state from that output:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$uri &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;https://graph.microsoft.com/beta/security/identities/sensors&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example value: replace with the id of the v2 entry in migrating state&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$orphanV2Id &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;00000000-0000-0000-0000-000000000000&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Invoke-MgGraphRequest&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Method DELETE &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Uri &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$uri&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;/&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$orphanV2Id&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Verify&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sensors &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Invoke-MgGraphRequest&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Method GET &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Uri $uri &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;OutputType PSObject&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sensors.value &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Where-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; displayName &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-match&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;DC01&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Format-Table&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; id&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; displayName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; version&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; healthStatus&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The DELETE returned 204. The portal now showed the healthy v3 sensor for DC01. The
restriction that said “Migration in progress” lives in the UI, not in the backend. Total time
once I knew what to look for: five minutes, after 48 hours of waiting.&lt;/p&gt;
&lt;p&gt;The alternatives, for comparison:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Method&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Graph DELETE of the orphaned v2 entry&lt;/td&gt;
&lt;td&gt;Worked&lt;/td&gt;
&lt;td&gt;5 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Microsoft support ticket&lt;/td&gt;
&lt;td&gt;Force-abort on the cloud side, then Delete and fresh discovery&lt;/td&gt;
&lt;td&gt;1 to 3 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wait for an automatic timeout&lt;/td&gt;
&lt;td&gt;Did not happen within 48 hours&lt;/td&gt;
&lt;td&gt;Unknown&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Identify before you delete applies here twice. Check that a healthy v3 entry exists, and
match its id against &lt;code&gt;senseId&lt;/code&gt; on the host, before you remove anything.&lt;/p&gt;
&lt;p&gt;On repair versus migrate: a v2 sensor that shows “Ready for migration”, or whose updater has
crashed more than 1,000 times, should go to v3, which is quick on a server already onboarded
to Defender for Endpoint. Repairing v2 is only the faster path when the need is urgent and
Defender for Endpoint is missing. And the rule that would have saved this case: do not click &lt;strong&gt;Migrate&lt;/strong&gt; on a sensor that is
Disconnected or Not healthy. Fix the underlying problem first, or delete the sensor and
start again, or let the Defender for Endpoint channel activate v3 on its own.&lt;/p&gt;
&lt;h2 id=&quot;the-second-sensor-a-credentials-problem-that-looked-like-a-network-problem&quot;&gt;The second sensor: a credentials problem that looked like a network problem&lt;/h2&gt;
&lt;p&gt;The same week, a new sensor went onto a server that was not a domain controller, an AD CS
server I will call CA01. The install succeeded, and then &lt;code&gt;AATPSensor&lt;/code&gt; went into a crash loop:
more than 3,000 restarts, Service Control Manager event 7031 about every 20 seconds with
&lt;code&gt;WIN32_EXIT_CODE 1067&lt;/code&gt;. The sensor log said:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;GetGroupManagedServiceAccountTokenAsync ... IsSuccess=False&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;CreateLdapConnectionAsync failed to retrieve group managed service account password&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;Failed to communicate with configured domain controllers&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The last line is the one that gets read, and it points at the network. I tested the network
first anyway: TCP to the domain controllers the sensor uses on 389, 636, 3268, 3269 and 88, and the RPC
ports. All open from CA01’s VLAN. The network was fine.&lt;/p&gt;
&lt;p&gt;The first line was the real one. The sensor’s directory service account was a gMSA, and the
token request failed for all three configured domains. Domain controllers can retrieve the
gMSA password when the account allows the Domain Controllers group, but a sensor on any
other server needs that server’s computer account in
&lt;code&gt;PrincipalsAllowedToRetrieveManagedPassword&lt;/code&gt;. The list held individual computer accounts,
and CA01 was not one of them.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;Set-ADServiceAccount&lt;/code&gt; replaces the list rather than appending to it, so read the current
value first and add to it. Run this on a domain controller or a management host with RSAT
(the &lt;code&gt;ActiveDirectory&lt;/code&gt; module and &lt;code&gt;repadmin&lt;/code&gt;), as a Domain Admin:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Import-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ActiveDirectory&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$dsa &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;svc-mdi-gmsa&apos;&lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;   # example name of the sensor&apos;s gMSA&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$prop &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;PrincipalsAllowedToRetrieveManagedPassword&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$gmsa &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-ADServiceAccount&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $dsa &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Properties $prop&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$current &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$gmsa.$prop&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$toAdd &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;CA01&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; ForEach-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-ADComputer&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.DistinguishedName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$updated &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$current &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $toAdd&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Select-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Unique&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Set-ADServiceAccount&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $dsa &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;PrincipalsAllowedToRetrieveManagedPassword $updated&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Replicate to the DC the sensor talks to&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;repadmin &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;/&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;syncall DC02 &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;/&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;AdeP&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I added a newly built domain controller to the list at the same time, ahead of need. Then on
CA01, from an elevated prompt, purge the computer account’s Kerberos tickets so the new
permission takes effect, and start the sensor:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;klist &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;li &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;0x3e7&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; purge&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Start-Service&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; AATPSensor&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The log then said &lt;code&gt;CreateLdapConnectionAsync connected successfully&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The better long-term setup is a group in &lt;code&gt;PrincipalsAllowedToRetrieveManagedPassword&lt;/code&gt;
instead of individual computer accounts. A new sensor then needs only &lt;code&gt;Add-ADGroupMember&lt;/code&gt;,
with no change to the gMSA object itself. Changing the gMSA requires Domain Admin, while
group membership can be delegated.&lt;/p&gt;
&lt;p&gt;Before the install could even start, it had failed at &lt;code&gt;ValidateCreateSensorAsync&lt;/code&gt; because a
perimeter firewall closed the TLS session right after the ClientHello. The firewall partner
opened outbound TCP 443 to &lt;code&gt;*.atp.azure.com&lt;/code&gt; and &lt;code&gt;*.events.data.microsoft.com&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&quot;what-i-take-with-me&quot;&gt;What I take with me&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;The portal shows one entry per host by its own grouping rule, not what is stored; list the
objects through Graph before you believe a stuck state.&lt;/li&gt;
&lt;li&gt;A UI restriction is not necessarily a backend restriction, and Graph can do what the
portal refuses.&lt;/li&gt;
&lt;li&gt;Identify before you delete: match the cloud id against &lt;code&gt;senseId&lt;/code&gt; on the host.&lt;/li&gt;
&lt;li&gt;Never trigger a migration on a broken sensor; fix it first or let the Defender for
Endpoint channel do the work.&lt;/li&gt;
&lt;li&gt;The error you see is the last link in a chain: “Failed to communicate with configured
domain controllers” was a gMSA permission, not the network.&lt;/li&gt;
&lt;li&gt;A crash counter in the hundreds of thousands is a decision, not a repair task.&lt;/li&gt;
&lt;/ul&gt;
</content:encoded><category>finding</category><category>defender</category><category>mdi</category><category>identity</category><category>active-directory</category></item><item><title>Last logon is not usage: the SSO outage I caused</title><link>https://exoit.se/posts/last-logon-is-not-usage/</link><guid isPermaLink="true">https://exoit.se/posts/last-logon-is-not-usage/</guid><description>I disabled 19 inactive service accounts and broke SAP single sign-on. The accounts were in use; lastLogon just can&apos;t see it.</description><pubDate>Sat, 03 Oct 2026 06:10:00 GMT</pubDate><content:encoded>&lt;p&gt;One day this autumn I disabled 19 service accounts at a customer, a manufacturing group with sites in several European countries. None had logged on in the last 90 days, the SAP-related ones not since 2022 or 2023 and some never. Defender for Identity listed them as inactive, and every one of them had an SPN pointing at a host that did not exist as a computer object in AD. It was a routine Secure Score clean-up item.&lt;/p&gt;
&lt;p&gt;About thirty minutes later the SAP side raised an alarm: users signing in to SAP GUI were getting certificate and SNC errors. The alarm said certificate errors against SAP, nothing more.&lt;/p&gt;
&lt;p&gt;It was my change. Nine of the 19 accounts were the Kerberos SPN targets for SAP single sign-on. They never log on themselves, so &lt;code&gt;lastLogon&lt;/code&gt; had nothing to say about whether they were used. This post covers how that happened, how we got it back with two minutes of typing, and what I do differently now.&lt;/p&gt;
&lt;h2 id=&quot;the-change&quot;&gt;The change&lt;/h2&gt;
&lt;p&gt;The work was batch 2 of a Secure Score review, the on-premises identity part. It started with a read-only inventory of the domain’s service-style accounts, then the Defender for Identity export for “Remove stale service accounts”: 43 accounts, all with SPNs. Some were already disabled but never deleted, which still counts against the score. 19 were enabled.&lt;/p&gt;
&lt;p&gt;The SIDs were resolved in AD and every SPN host looked up. None of the hosts existed as computer objects. For the SAP accounts the SPNs pointed at SAP hosts that had never been AD computers, and that was read as “the hosts are gone”. The accounts had the description “Service account for SAP AD SSO”, had passwords set in 2022 that never expire, and no group memberships. The conclusion was that this was an abandoned attempt at Kerberos SSO for SAP from 2022, because SAP SSO today was believed to run through two other accounts with constrained delegation towards a separate forest. No owner was listed, so no owner was asked.&lt;/p&gt;
&lt;p&gt;That assessment was made by an AI assistant I had asked to work through the export, and it called the SAP accounts inactive and abandoned. I reviewed the recommendation and ran the change, so the decision was mine. I come back to what the assistant could and could not have known further down.&lt;/p&gt;
&lt;p&gt;The plan was sensible as far as it went. Two phases: delete the already-disabled accounts, disable the 19 enabled ones and tag them with a deletion date 30 days out, then delete in a second run. The script had guardrails: no account in a privileged group, no logon within 90 days. It wrote a before-file with SID, GUID, UAC, group memberships, SPNs and description for every account. The dry run came back clean. The real run on DC01 took two seconds. Each disabled account got a description prefix like this:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;[MDI inactive, disabled &amp;lt;date&amp;gt;, delete after &amp;lt;date + 30 days&amp;gt;]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The change log entry had the rollback written out before anything was touched. It runs per account on a domain controller or any machine with RSAT and the &lt;code&gt;ActiveDirectory&lt;/code&gt; module, as a Domain Admin; the restore from the Recycle Bin needs that to read Deleted Objects:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Import-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ActiveDirectory&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example values; the real ones come from the before-file&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sam &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;svc-sap-snc&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$originalDescription &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Service account for SAP AD SSO&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Deleted accounts: AD Recycle Bin, SID and groups come back with the object&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-ADObject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Filter &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&quot;sAMAccountName -eq &apos;&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sam&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos; -and isDeleted -eq &lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;`$&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;true&quot;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; `&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;IncludeDeletedObjects &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Restore-ADObject&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Disabled accounts: re-enable and put the original description back&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Enable-ADAccount&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $sam&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Set-ADUser&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $sam &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Description $originalDescription&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;the-symptom&quot;&gt;The symptom&lt;/h2&gt;
&lt;p&gt;Around half an hour after the run, the alarm came in from the SAP side: certificate errors against SAP. Users of SAP GUI with Secure Login got certificate or SNC errors at sign-in. Seven SAP systems were affected, from development and test to production. No data was affected and nothing else broke.&lt;/p&gt;
&lt;p&gt;A certificate error does not point at AD.&lt;/p&gt;
&lt;h2 id=&quot;working-it-out&quot;&gt;Working it out&lt;/h2&gt;
&lt;p&gt;The question was whether any of that day’s changes could affect this. The clean-up run that had disabled accounts described as SAP SSO accounts was the obvious candidate, and the link was made within five minutes of the alarm.&lt;/p&gt;
&lt;p&gt;The mechanism took a moment longer to put into words. With SNC, the SAP GUI client asks the KDC for a service ticket for &lt;code&gt;SAP/&amp;lt;SID&amp;gt;&lt;/code&gt;. The KDC finds the account that holds that SPN and issues the ticket. The SAP server, a Linux host outside AD, verifies it with a keytab generated from that account’s password. The account itself never authenticates to anything, so nothing in the flow updates its &lt;code&gt;lastLogon&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;When the account is disabled, the KDC stops issuing service tickets for its SPNs. The client gets no ticket, SNC cannot set up its secure context, and SAP GUI and Secure Login reported it as a certificate or SNC error, which is what the users saw. The error the user sees is several links away from the cause.&lt;/p&gt;
&lt;p&gt;The rollback was cheap and already written, and the hypothesis was specific, so I ran it.&lt;/p&gt;
&lt;h2 id=&quot;the-cause&quot;&gt;The cause&lt;/h2&gt;
&lt;p&gt;An account whose only job is to be a Kerberos SPN target never logs on. The 2022 &lt;code&gt;lastLogon&lt;/code&gt; on these accounts showed when the keytab was created, not whether the account was used. Three of the nine had never logged on at all, and all of them were in use.&lt;/p&gt;
&lt;p&gt;Three things reinforced each other:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Defender for Identity’s “inactive service account” check&lt;/strong&gt; is built on the logon activity the sensor sees. It does not catch accounts that are only used as ticket targets.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Our own “host has no computer object” check&lt;/strong&gt; was wrong for this case. It looked for the SPN hosts as computer objects in AD. The SAP hosts are Linux systems outside AD, so “the host does not exist” was the wrong conclusion. For a Windows service it would have been a reasonable signal, which is why it looked convincing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two separate SSO flows were mixed up.&lt;/strong&gt; The delegation accounts serve the SAP web interfaces in the separate forest. The Kerberos accounts serve SNC in SAP GUI. Seeing one in use made the other look replaced.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On the AI part, plainly: the assessment that called the accounts inactive came from an assistant working from &lt;code&gt;lastLogon&lt;/code&gt;, the MDI export and a computer object lookup. That data cannot show ticket issuance. Neither an assistant nor a person reading the same columns can tell from them that a Linux host outside AD holds a keytab and asks for tickets whenever someone signs in. The assistant reasoned consistently from what it was given; what it was given did not include the one signal that mattered. That signal is event 4769 on the domain controllers, and nobody looked there.&lt;/p&gt;
&lt;p&gt;I don’t take “don’t use an assistant for this” from it. I take that “is this account used?” has to be answered from evidence of use, and that the person running the change has to know which evidence that is and ask for it.&lt;/p&gt;
&lt;p&gt;The process failures were mine as well. There was no check with application owners before disabling, on the grounds that the accounts had no owner. A missing owner is a reason to ask more widely, not a reason to skip the question. And all 19 were disabled in one run. Waves, accounts without SPNs first and SPN accounts the following week, would have limited the damage.&lt;/p&gt;
&lt;h2 id=&quot;the-fix&quot;&gt;The fix&lt;/h2&gt;
&lt;p&gt;The nine accounts were re-enabled with the rollback command from the change log. Formulating it took two minutes and running it took one. The original description was restored on each. Two of the SSO accounts had had an empty description, so the tag was cleared instead. Per account, with the &lt;code&gt;ActiveDirectory&lt;/code&gt; module:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Import-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ActiveDirectory&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example values; the real ones come from the before-file&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sam &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;svc-sap-snc&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$originalDescription &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Service account for SAP AD SSO&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Enable-ADAccount&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $sam&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Set-ADUser&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $sam &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Description $originalDescription&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# For the two accounts that had no description before the change&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Set-ADUser&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $sam &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Clear Description&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The SAP side confirmed sign-in worked again seventeen minutes after the accounts were re-enabled. That was half an hour after the alarm and about an hour after the original change.&lt;/p&gt;
&lt;p&gt;The two-phase design mattered here. Because the accounts had been disabled and not deleted, the fix was &lt;code&gt;Enable-ADAccount&lt;/code&gt;. Had they been deleted, &lt;code&gt;Restore-ADObject&lt;/code&gt; from the Recycle Bin would have brought them back with the same SID, but the recovery would have taken longer.&lt;/p&gt;
&lt;p&gt;Then the accounts were marked so this cannot happen quietly again. Each got a description that says what it is and why it must stay, and &lt;code&gt;ProtectedFromAccidentalDeletion&lt;/code&gt;, so that deleting one takes a deliberate extra step. Again with the &lt;code&gt;ActiveDirectory&lt;/code&gt; module:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Import-Module&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ActiveDirectory&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example names; the real list was the nine SAP SNC accounts&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$sapSncAccounts &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;svc-sap-snc01&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;svc-sap-snc02&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;svc-sap-snc03&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$a &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $sapSncAccounts&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $u &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-ADUser&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $a&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Set-ADUser&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $u &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Description &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;        &apos;DO NOT DISABLE. SAP SNC Kerberos SSO, SPN target &apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; +&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;        &apos;(never logs on itself). Remove with SAP phase-out.&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;    # The parameter exists on Set-ADObject, not Set-ADUser&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Set-ADObject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $u &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ProtectedFromAccidentalDeletion $&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-ADUser&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Filter &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;Description -like &quot;DO NOT DISABLE*&quot;&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; `&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Properties Description&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ProtectedFromAccidentalDeletion &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Select-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; SamAccountName&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ProtectedFromAccidentalDeletion&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Nine of nine came back protected.&lt;/p&gt;
&lt;p&gt;The longer-term decision was to leave the accounts exactly as they are. The SAP team did not know the accounts existed and referred us to an external SAP consultant who could not be reached. Rotating the passwords would mean regenerating the keytabs on the SAP hosts, and nobody on the SAP side could do that. SAP is going to be phased out eventually. So the risk (passwords from 2022 that never expire, an obvious Kerberoasting target) is accepted and documented in the Secure Score row until then, and the accounts go with the SAP systems. We wrote down what we knew about the flow ourselves, account by account, for the next person.&lt;/p&gt;
&lt;p&gt;The other ten disabled accounts are still disabled, with their deletion date unchanged. Their SPNs point at Windows servers that no longer have computer objects. They will still get the event 4769 check on all domain controllers before the second phase deletes them. For accounts that are already disabled, 4769 shows failed ticket requests rather than successful ones, so that check has to include failures.&lt;/p&gt;
&lt;p&gt;This is a sketch of the check, run on each domain controller in an elevated session; the &lt;code&gt;ActiveDirectory&lt;/code&gt; module is not needed. It is a general example, not something that was run during the incident. It only finds anything if Kerberos service ticket auditing is enabled on the DCs, it covers a 14-day window, and it reads the &lt;code&gt;ServiceName&lt;/code&gt; and &lt;code&gt;Status&lt;/code&gt; fields from the event XML rather than the localised message text. Failed requests (Status other than &lt;code&gt;0x0&lt;/code&gt;) count as evidence of use too, and for an account that is already disabled they are the only evidence:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example values: the accounts you intend to disable&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$spnAccounts &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;svc-sap-snc&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;svc-sap-sso&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$filter &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    LogName   &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Security&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    Id        &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 4769&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    StartTime &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-Date&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.AddDays&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt;-14&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-WinEvent&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;FilterHashtable $filter &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ErrorAction SilentlyContinue &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    ForEach-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        $data &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; @&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;        foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$d &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; ([&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;xml&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.ToXml&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;())&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Event.EventData.Data&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            $data&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$d.Name&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $d.&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;#text&apos;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Account &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $data&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;ServiceName&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;            Status  &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $data&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;Status&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;   # 0x0 is success, anything else failed&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Where-Object&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $spnAccounts &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-contains&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color:#A9A7A0&quot;&gt;_&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Account &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Group-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Account&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Status &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Sort-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Count &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Descending &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Select-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Count&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Name&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A side note from the same run: the phase 2 filter used &lt;code&gt;-like&lt;/code&gt; with the square-bracketed tag, which &lt;code&gt;-like&lt;/code&gt; reads as a character class, and was fixed afterwards.&lt;/p&gt;
&lt;h2 id=&quot;what-i-take-with-me&quot;&gt;What I take with me&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Before an account with an SPN is disabled, check event 4769 (Kerberos service ticket issued) on all domain controllers for the account’s SPNs over the last 14 days. Hits mean the account is in use, whatever &lt;code&gt;lastLogon&lt;/code&gt; says. The event only exists if Kerberos service ticket auditing is enabled on the DCs, and a 14-day window misses rarely used accounts.&lt;/li&gt;
&lt;li&gt;Disable SPN accounts in their own waves, a handful per run at most, at a time when the application owners can test.&lt;/li&gt;
&lt;li&gt;Last logon is not usage. For an account that only exists to be a ticket target, the evidence of use is on the KDC, not on the account.&lt;/li&gt;
&lt;li&gt;The error you see is the last link in a chain. A certificate error in SAP GUI was a disabled account in AD.&lt;/li&gt;
&lt;li&gt;Identify before you delete, and disable before you delete: check that your test for “the host is gone” means the same thing for every kind of host, and keep the step that can be undone with a single &lt;code&gt;Enable-ADAccount&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Keep the rollback command next to the change, written before you run it. It made the recovery two minutes of typing.&lt;/li&gt;
&lt;/ul&gt;
</content:encoded><category>finding</category><category>incident</category><category>kerberos</category><category>active-directory</category><category>sap</category></item><item><title>The retention policy that ran successfully and did nothing</title><link>https://exoit.se/posts/retention-policy-did-nothing/</link><guid isPermaLink="true">https://exoit.se/posts/retention-policy-did-nothing/</guid><description>A move-to-archive policy ran without errors for a week and moved nothing, because the mailbox had no archive to move to.</description><pubDate>Sat, 03 Oct 2026 06:00:00 GMT</pubDate><content:encoded>&lt;p&gt;One user’s mailbox stood at 98.44 GB against a ProhibitSend quota of 99 GB. The send stop was days away. About a week earlier a retention policy had been assigned that moves anything older than five years to the online archive, and the mailbox held more than ten years of mail. Nothing had moved.&lt;/p&gt;
&lt;p&gt;The puzzling part was that nothing was failing: the Managed Folder Assistant had run, it had reported success, and there were no holds in the way. The answer, up front: here the MoveToArchive tag did nothing because the mailbox had no archive, and the assistant did not say so. It skipped the tag silently and logged a clean run with every counter at zero. The same gap turned out to apply to 2,033 of the 2,036 mailboxes on the tenant’s Default MRM Policy.&lt;/p&gt;
&lt;h2 id=&quot;what-i-saw&quot;&gt;What I saw&lt;/h2&gt;
&lt;p&gt;The starting facts for the mailbox:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Size&lt;/td&gt;
&lt;td&gt;98.44 GB, 407,198 items&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Largest folder&lt;/td&gt;
&lt;td&gt;Sent Items, 27.2 GB (89,348 items, going back to 2016)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quotas&lt;/td&gt;
&lt;td&gt;ProhibitSend 99 GB, ProhibitSendReceive 100 GB, RecoverableItems 30 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retention policy&lt;/td&gt;
&lt;td&gt;A retention policy with one tag: &lt;code&gt;Default 5 year move to archive&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tag settings&lt;/td&gt;
&lt;td&gt;Type All, 1825 days, MoveToArchive, WhenDelivered&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The expectation was that the assistant would pick up the policy, move the old mail out and the mailbox would shrink. It had not shrunk at all.&lt;/p&gt;
&lt;h2 id=&quot;working-it-out&quot;&gt;Working it out&lt;/h2&gt;
&lt;p&gt;The first question with any MRM complaint is whether the Managed Folder Assistant has processed the mailbox since the policy changed. The mailbox diagnostic log shows it. Every block in this post runs in a PowerShell session with the ExchangeOnlineManagement module, already connected with &lt;code&gt;Connect-ExchangeOnline&lt;/code&gt; as an Exchange administrator. This first block sets &lt;code&gt;$upn&lt;/code&gt; to the affected mailbox, and the later blocks reuse it:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$upn &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;user@contoso.com&apos;&lt;/span&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;   # example value: the affected mailbox&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$x &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Export-MailboxDiagnosticLogs&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $upn &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ExtendedProperties&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;([&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;xml&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$x.MailboxLog&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;.Properties.MailboxTable.Property &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;    Where-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Name &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-like&lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt; &apos;Elc*&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Format-Table&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Name&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Value&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The last successful run was the night before, at 03:34. So the assistant had run after the policy was assigned, and it had succeeded. The counters were all zero: nothing tagged with an archive action, nothing archived from the root.&lt;/p&gt;
&lt;p&gt;That ruled out waiting for the next scheduled run and made the problem more interesting. A successful run that touches nothing on a mailbox full of mail older than five years is not a timing issue.&lt;/p&gt;
&lt;p&gt;Next I checked everything that can stop MRM from acting on a mailbox:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;RetentionHoldEnabled&lt;/code&gt;: False&lt;/li&gt;
&lt;li&gt;&lt;code&gt;LitigationHoldEnabled&lt;/code&gt;: False&lt;/li&gt;
&lt;li&gt;&lt;code&gt;InPlaceHolds&lt;/code&gt;: empty&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ElcProcessingDisabled&lt;/code&gt;: False&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;No holds, and ELC processing was not disabled. The MRM log did contain two errors, both &lt;code&gt;ResourceUnhealthyException&lt;/code&gt;, but they were from years earlier and had nothing to do with this. Read the dates on old errors before you chase them.&lt;/p&gt;
&lt;p&gt;With the assistant running and nothing blocking it, the remaining question was the destination. The policy does not delete anything. Its only tag moves items to the online archive. So I looked at the archive:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;ArchiveStatus&lt;/code&gt;: None&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ArchiveGuid&lt;/code&gt;: empty&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The mailbox had no archive. The policy had nowhere to move anything, and the assistant had dealt with that by doing nothing and reporting success.&lt;/p&gt;
&lt;h2 id=&quot;the-cause&quot;&gt;The cause&lt;/h2&gt;
&lt;p&gt;A MoveToArchive retention tag is only acted on when the mailbox has an archive. In this case there was none, and the Managed Folder Assistant skipped the tag without an error. The run was recorded as successful and the counters showed zero, which reads exactly like “nothing was old enough to move”.&lt;/p&gt;
&lt;p&gt;Assigning the policy was half of the job. Enabling the archive was the other half, and the run gave no sign of the missing half.&lt;/p&gt;
&lt;h2 id=&quot;the-fix&quot;&gt;The fix&lt;/h2&gt;
&lt;p&gt;We agreed that the policy should stay a move policy, not a delete policy. Total storage keeps growing in the archive, but the primary mailbox shrinks below the quota. Because the mailbox held more than ten years of mail, much of it older than five years, the archive could outgrow a standard archive, so I enabled auto-expanding archiving as well. The organisation-level &lt;code&gt;AutoExpandingArchiveEnabled&lt;/code&gt; was False, so it went on per mailbox. These are two separate calls because &lt;code&gt;-Archive&lt;/code&gt; and &lt;code&gt;-AutoExpandingArchive&lt;/code&gt; belong to different parameter sets of &lt;code&gt;Enable-Mailbox&lt;/code&gt;, and the archive must exist before auto-expanding can be turned on (same session, &lt;code&gt;$upn&lt;/code&gt; from the first block):&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Enable-Mailbox&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $upn &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Archive&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Enable-Mailbox&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $upn &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;AutoExpandingArchive&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The archive came up with an ArchiveQuota of 110 GB and a warning at 100 GB.&lt;/p&gt;
&lt;p&gt;Rollback is not symmetric, so it is worth knowing before you run either line:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Disable-Mailbox -Identity $upn -Archive&lt;/code&gt; disconnects the archive. It stays disconnected for 30 days, and running &lt;code&gt;Enable-Mailbox -Identity $upn -Archive&lt;/code&gt; again within that window reconnects the same archive (&lt;code&gt;Connect-Mailbox -Archive&lt;/code&gt; is the on-premises equivalent and does not exist in Exchange Online). This is only safe before items have moved, because they go with the archive.&lt;/li&gt;
&lt;li&gt;Auto-expanding archiving cannot be turned off for a mailbox once it is on. There is no rollback for that line.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;With the archive in place I did not want to wait up to seven days for the next scheduled run, so a minute later I started the assistant by hand, in the same session:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Start-ManagedFolderAssistant&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $upn&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It failed:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;gRPC call for assistant &apos;ElcHighPriorityAssistant&apos; ... failed with ErrorCode=0x80040200&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;0x80040200 is -2147220992. It is a known error straight after an archive has been enabled: the identity is now ambiguous, and the fix is to pass the GUID of the primary mailbox instead. &lt;code&gt;Get-MailboxLocation&lt;/code&gt; gives you that, filtered to the location of type Primary (same session, &lt;code&gt;$upn&lt;/code&gt; from the first block):&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$primary &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-MailboxLocation&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;User $upn &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;MailboxLocationType Primary&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$primary &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Format-List&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; MailboxLocationType&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; MailboxGuid&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Start-ManagedFolderAssistant&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $primary.MailboxGuid&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;Get-MailboxLocation&lt;/code&gt; printed a warning that the object “has been corrupted … Id: You must provide a value”. That looks like a cosmetic validation bug in the cmdlet, not corruption. With the primary MailboxGuid, &lt;code&gt;Start-ManagedFolderAssistant&lt;/code&gt; succeeded, eight minutes after the first attempt.&lt;/p&gt;
&lt;p&gt;A first assistant pass over roughly 407,000 items can take hours to a couple of days, so I planned to check again the next day. Success means a newer &lt;code&gt;ELCLastSuccessTimestamp&lt;/code&gt; and &lt;code&gt;ElcLastRunArchivedFromRootItemCount&lt;/code&gt; above zero in the same diagnostic log, and &lt;code&gt;Get-MailboxStatistics&lt;/code&gt; showing the primary shrinking while the archive (&lt;code&gt;-Archive&lt;/code&gt;) grows.&lt;/p&gt;
&lt;p&gt;It did not take a day. An hour after the assistant run, the primary mailbox had dropped from 98.44 GB to 93.92 GB.&lt;/p&gt;
&lt;h2 id=&quot;the-tenant-wide-sweep&quot;&gt;The tenant-wide sweep&lt;/h2&gt;
&lt;p&gt;If one mailbox had a move-to-archive policy and no archive, the obvious question was how many others did. I ran a read-only inventory of all user and shared mailboxes in the tenant with &lt;code&gt;Get-EXOMailbox&lt;/code&gt; and &lt;code&gt;Get-EXOMailboxStatistics&lt;/code&gt;, covering size and retention, and exported it to CSV. It took about ten minutes. The block below is a shortened example of the method, not the exact script that ran; it needs the same connected Exchange Online session and only reads:&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#5F5E59;font-style:italic&quot;&gt;# Example of the method, not the exact script that ran&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$mailboxes &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-EXOMailbox&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;ResultSize Unlimited &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;`&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;RecipientTypeDetails UserMailbox&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; SharedMailbox &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;`&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Properties RetentionPolicy&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ArchiveStatus&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; ProhibitSendQuota&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$report &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; foreach&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$mbx &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $mailboxes&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;    $stats &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Get-EXOMailboxStatistics&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Identity $mbx.UserPrincipalName&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    [&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;pscustomobject&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        UserPrincipalName &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $mbx.UserPrincipalName&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        Type              &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $mbx.RecipientTypeDetails&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        RetentionPolicy   &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $mbx.RetentionPolicy&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        ArchiveStatus     &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $mbx.ArchiveStatus&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        ProhibitSendQuota &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $mbx.ProhibitSendQuota&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        TotalItemSize     &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $stats.TotalItemSize&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;        ItemCount         &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; $stats.ItemCount&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;$report &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt; Export-Csv&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;Path .\mailbox&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;retention&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;report.csv &lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;NoTypeInformation&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The results:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The same five-year policy without an archive&lt;/strong&gt;: one more mailbox, a shared mailbox near 90 GB, at 90 % of its quota. The other seven mailboxes on that policy had archives.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The Default MRM Policy&lt;/strong&gt;: it contains the default tag &lt;code&gt;Default 2 year move to archive&lt;/code&gt;. Of the 2,036 mailboxes on it, 2,033 had no archive. For those 2,033 the policy did nothing, while looking assigned and healthy.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No policy at all&lt;/strong&gt;: 832 mailboxes had an empty RetentionPolicy.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Sizes&lt;/strong&gt;: 5 mailboxes over 90 GB, 93 between 50 and 90 GB, 294 between 25 and 50 GB.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Close to the 99 GB send stop&lt;/strong&gt;: five more user mailboxes between 87.4 and 94.9 GB. The largest had an archive but no retention policy at all.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shared mailboxes against their 50 GB quota&lt;/strong&gt;: one at 91 GB, 184 % of its quota, probably converted from a user mailbox and no longer receiving mail. Seven more were between 92 % and 100 %.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So the archive is not enabled by default in this tenant, and the Default MRM Policy moves nothing for 2,033 of its 2,036 mailboxes, and a further 832 mailboxes have no retention policy at all. Anyone who looks at the policy assignments sees retention in place. Anyone who looks at the mailbox sizes sees that it is not working.&lt;/p&gt;
&lt;p&gt;What to do about that is a decision, not a fix. Enabling archives broadly changes where mail lives for a large part of the tenant, and archives over 50 GB on shared mailboxes need an Exchange Online Plan 2 or Exchange Online Archiving licence. So the sweep went back as a question for the business rather than as something to run inside a single ticket. The shared mailbox on the five-year policy without an archive is first on the list, with a licence check before enabling anything.&lt;/p&gt;
&lt;h2 id=&quot;what-i-take-with-me&quot;&gt;What I take with me&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Silence is not innocence: a successful run with zero counters is not “nothing to do”, so check that the action has somewhere to go.&lt;/li&gt;
&lt;li&gt;A policy shown as assigned tells you the intent; the mailbox sizes tell you the result.&lt;/li&gt;
&lt;li&gt;Read the dates on old errors before you chase them.&lt;/li&gt;
&lt;li&gt;When a cmdlet fails right after you change the object it targets, try the unambiguous identity, here the primary MailboxGuid.&lt;/li&gt;
&lt;li&gt;When one mailbox shows a configuration gap, sweep the tenant for it before you close the ticket.&lt;/li&gt;
&lt;/ul&gt;
</content:encoded><category>finding</category><category>exchange</category><category>method</category></item><item><title>The door is open</title><link>https://exoit.se/posts/the-door-is-open/</link><guid isPermaLink="true">https://exoit.se/posts/the-door-is-open/</guid><description>What this place is for, and why it starts with a grid of dots.</description><pubDate>Fri, 02 Oct 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;This is where I keep the things I would otherwise lose: findings from tenant work,
notes from the homelab, the occasional longer argument.&lt;/p&gt;
&lt;p&gt;The front page is a field of dots. Somewhere in it there is a shape. You do not have to
find it, a click anywhere opens the door, but it is there.&lt;/p&gt;
&lt;pre class=&quot;astro-code exoit&quot; style=&quot;background-color:#111113;color:#c9c7c0; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;powershell&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;Get-WinEvent&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt;LogName &lt;/span&gt;&lt;span style=&quot;color:#8FD6A4&quot;&gt;&apos;Microsoft-Windows-NTLM/Operational&apos;&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;  Where-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; Id &lt;/span&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;-eq&lt;/span&gt;&lt;span style=&quot;color:#ECEBE5&quot;&gt; 8004&lt;/span&gt;&lt;span style=&quot;color:#8A8880&quot;&gt; |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#3EF071&quot;&gt;  Group-Object&lt;/span&gt;&lt;span style=&quot;color:#C9C7C0&quot;&gt; WorkstationName&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Posts are Markdown files in a Git repository. Nothing here phones home.&lt;/p&gt;
</content:encoded><category>meta</category></item></channel></rss>