The sensor that was stuck migrating had already arrived
A customer, a manufacturing group with sites in several European countries, had a domain controller whose Defender for Identity sensor showed Disconnected and Not healthy in the Defender XDR portal. The sensor services were stopped. That is a routine ticket: fix the sensor or replace it.
It stopped being routine when the replacement would not install, the portal refused to delete the old sensor, and nothing moved for 48 hours. The portal said the sensor was migrating. That entry was a leftover. The migration that mattered had already finished, through a different channel, and the new sensor was already healthy. The portal simply showed me the wrong one of two entries for the same host.
The fix, once I could see that, took five minutes with two Graph calls, a list and a delete.
What I saw
The first look was at DC01 itself. I used a small read-only triage script that collects the
things that usually explain a dead sensor: service state for both the legacy v2 services
(AATPSensor, AATPSensorUpdater) and the unified v3 ones (MDISensor,
MDISensorUpdater), recent warnings and errors from the two sensor event logs, the WinHTTP
proxy (netsh winhttp show proxy), time sync (w32tm /query /status), the five latest
hotfixes as regression candidates, and Service Control Manager events that mention the
sensor. This is the Service Control Manager part of it, run locally on DC01 in Windows
PowerShell 5.1 or PowerShell 7:
# Service Control Manager events for the sensor services, last 14 days
Get-WinEvent -FilterHashtable @{
LogName = 'System'; ProviderName = 'Service Control Manager'
StartTime = (Get-Date).AddDays(-14)
} -ErrorAction SilentlyContinue |
Where-Object { $_.Message -match 'AATP|Azure Advanced Threat|MDI Sensor' } |
Select-Object TimeCreated, Id, LevelDisplayName, Message -First 20
The Service Control Manager section answered the question on its own. Event 7031 for
AATPSensorUpdater, “terminated unexpectedly”, with a counter of 206,727. The sensor had
been installed in 2023. It had in all likelihood never worked properly on that server.
That changes the decision. A sensor that broke at the last cumulative update is worth repairing. A sensor whose updater has crashed two hundred thousand times is not. The plan became a clean uninstall of v2 and a fresh install.
The uninstall did not complete cleanly. The AATPSensor service was left in
DELETE_PENDING, which a reboot clears, so the reboot went into the next night’s
window.
The day before, the sensor had been marked for migration in the portal with Migrate. That click matters later.
After the reboot the host was clean, and the new install failed immediately:
Detected existing unified sensor installation, aborting installation
The installer claimed a unified sensor already existed. As it turned out, it did. Running the
MSI directly does not get around this: it calls the same
Microsoft.Tri.Sensor.Deployment.Package.Actions custom action as the bootstrapper.
The portal showed the sensor as:
Migrating, Disconnected, 2.255.x, Not healthy
and Delete was refused:
Migration in progress, sensor removal is unavailable until the migration completes
So the cloud side was waiting for a v2 sensor to convert into v3, the host refused a fresh install because something unified already existed, and the portal would not let me remove the stuck entry. Each piece blocked the next. No automatic timeout kicked in within 48 hours.
A fresh cloud handshake from the host is not available: the Defender for Endpoint Sense
service is tamper-protected, and re-onboarding the server is the only way to force one, which
is invasive on a domain controller.
Working it out
The useful question turned out to be a narrow one: what does the cloud actually think exists
for this host? The portal gives one answer. Graph gives the full list. The beta endpoint
security/identities/sensors returns every sensor in the tenant with its id, version,
migration state and health. From any workstation with the Microsoft.Graph.Authentication
module, signed in as a Security Administrator who can consent to the scope:
# Invoke-MgGraphRequest only needs Microsoft.Graph.Authentication
Install-Module Microsoft.Graph.Authentication -Scope CurrentUser
Connect-MgGraph -Scopes 'SecurityIdentitiesSensors.ReadWrite.All'
$uri = 'https://graph.microsoft.com/beta/security/identities/sensors'
# PSObject output so Format-Table can read the properties
$sensors = Invoke-MgGraphRequest -Method GET -Uri $uri -OutputType PSObject
$sensors.value | Where-Object displayName -match 'DC01' |
Format-Table id, displayName, version, migrationState, healthStatus,
deploymentStatus
-match is case-insensitive, which is why both rows came back. Two rows for the same host,
not one:
| displayName | Version | Graph showed |
|---|---|---|
DC01 |
2.255.x | migrationState = migrating |
dc01 |
3.0.7 | healthStatus = healthy |
The two entries had different cloud-generated ids, different creation dates (the old one from the original install, the new one from when v3 arrived) and different case in the display name. The old entry used the uppercase form of the computer account. The new one used the lowercase hostname.
The portal groups sensors by name case-insensitively and shows one entry per host. It had chosen the broken one. The healthy v3 sensor was there, reporting data normally, and invisible in the UI.
To confirm which entry was the real, active sensor, I checked the host’s own record. The
Graph id of the v3 entry matched the senseId value under
HKLM:\SOFTWARE\Microsoft\Windows Advanced Threat Protection on DC01, once the hyphens are
removed. That settled it: the lowercase entry was this server,
and the uppercase entry was an orphan.
The cause
Two things happened at the same time and the portal hid one of them.
First, Migrate had been clicked on a broken v2 sensor. Migration is designed to morph a
healthy v2 sensor into v3. On a sensor that is disconnected, there is nothing to morph, and
the cloud side sits in Migrating waiting for a v2 that will never report in.
Second, the server was onboarded to Defender for Endpoint, and on servers onboarded to Defender for Endpoint the unified v3 sensor can be activated through that channel without any install, when the tenant has it enabled. That is what had happened, at the same time as the stuck migration. It created a new, parallel sensor entry that was healthy and up to date. It is also why the installer saw an “existing unified sensor installation”: there was one.
So the stuck migration was largely an illusion. The real migration had happened. Only the old cloud entry was left behind, and the portal chose to show that one.
The fix
The portal blocked Delete on the orphaned entry. Graph did not. In the same Graph session
as the listing above, with the id of the v2 entry in migrating state from that output:
$uri = 'https://graph.microsoft.com/beta/security/identities/sensors'
# Example value: replace with the id of the v2 entry in migrating state
$orphanV2Id = '00000000-0000-0000-0000-000000000000'
Invoke-MgGraphRequest -Method DELETE -Uri "$uri/$orphanV2Id"
# Verify
$sensors = Invoke-MgGraphRequest -Method GET -Uri $uri -OutputType PSObject
$sensors.value | Where-Object displayName -match 'DC01' |
Format-Table id, displayName, version, healthStatus
The DELETE returned 204. The portal now showed the healthy v3 sensor for DC01. The restriction that said “Migration in progress” lives in the UI, not in the backend. Total time once I knew what to look for: five minutes, after 48 hours of waiting.
The alternatives, for comparison:
| Method | Effect | Time |
|---|---|---|
| Graph DELETE of the orphaned v2 entry | Worked | 5 minutes |
| Microsoft support ticket | Force-abort on the cloud side, then Delete and fresh discovery | 1 to 3 days |
| Wait for an automatic timeout | Did not happen within 48 hours | Unknown |
Identify before you delete applies here twice. Check that a healthy v3 entry exists, and
match its id against senseId on the host, before you remove anything.
On repair versus migrate: a v2 sensor that shows “Ready for migration”, or whose updater has crashed more than 1,000 times, should go to v3, which is quick on a server already onboarded to Defender for Endpoint. Repairing v2 is only the faster path when the need is urgent and Defender for Endpoint is missing. And the rule that would have saved this case: do not click Migrate on a sensor that is Disconnected or Not healthy. Fix the underlying problem first, or delete the sensor and start again, or let the Defender for Endpoint channel activate v3 on its own.
The second sensor: a credentials problem that looked like a network problem
The same week, a new sensor went onto a server that was not a domain controller, an AD CS
server I will call CA01. The install succeeded, and then AATPSensor went into a crash loop:
more than 3,000 restarts, Service Control Manager event 7031 about every 20 seconds with
WIN32_EXIT_CODE 1067. The sensor log said:
GetGroupManagedServiceAccountTokenAsync ... IsSuccess=False
CreateLdapConnectionAsync failed to retrieve group managed service account password
Failed to communicate with configured domain controllers
The last line is the one that gets read, and it points at the network. I tested the network first anyway: TCP to the domain controllers the sensor uses on 389, 636, 3268, 3269 and 88, and the RPC ports. All open from CA01’s VLAN. The network was fine.
The first line was the real one. The sensor’s directory service account was a gMSA, and the
token request failed for all three configured domains. Domain controllers can retrieve the
gMSA password when the account allows the Domain Controllers group, but a sensor on any
other server needs that server’s computer account in
PrincipalsAllowedToRetrieveManagedPassword. The list held individual computer accounts,
and CA01 was not one of them.
Set-ADServiceAccount replaces the list rather than appending to it, so read the current
value first and add to it. Run this on a domain controller or a management host with RSAT
(the ActiveDirectory module and repadmin), as a Domain Admin:
Import-Module ActiveDirectory
$dsa = 'svc-mdi-gmsa' # example name of the sensor's gMSA
$prop = 'PrincipalsAllowedToRetrieveManagedPassword'
$gmsa = Get-ADServiceAccount $dsa -Properties $prop
$current = @($gmsa.$prop)
$toAdd = @('CA01') | ForEach-Object { (Get-ADComputer $_).DistinguishedName }
$updated = ($current + $toAdd) | Select-Object -Unique
Set-ADServiceAccount -Identity $dsa -PrincipalsAllowedToRetrieveManagedPassword $updated
# Replicate to the DC the sensor talks to
repadmin /syncall DC02 /AdeP
I added a newly built domain controller to the list at the same time, ahead of need. Then on CA01, from an elevated prompt, purge the computer account’s Kerberos tickets so the new permission takes effect, and start the sensor:
klist -li 0x3e7 purge
Start-Service AATPSensor
The log then said CreateLdapConnectionAsync connected successfully.
The better long-term setup is a group in PrincipalsAllowedToRetrieveManagedPassword
instead of individual computer accounts. A new sensor then needs only Add-ADGroupMember,
with no change to the gMSA object itself. Changing the gMSA requires Domain Admin, while
group membership can be delegated.
Before the install could even start, it had failed at ValidateCreateSensorAsync because a
perimeter firewall closed the TLS session right after the ClientHello. The firewall partner
opened outbound TCP 443 to *.atp.azure.com and *.events.data.microsoft.com.
What I take with me
- The portal shows one entry per host by its own grouping rule, not what is stored; list the objects through Graph before you believe a stuck state.
- A UI restriction is not necessarily a backend restriction, and Graph can do what the portal refuses.
- Identify before you delete: match the cloud id against
senseIdon the host. - Never trigger a migration on a broken sensor; fix it first or let the Defender for Endpoint channel do the work.
- The error you see is the last link in a chain: “Failed to communicate with configured domain controllers” was a gMSA permission, not the network.
- A crash counter in the hundreds of thousands is a decision, not a repair task.