EXOIT

10 min#finding#incident#kerberos#active-directory#sap

Last logon is not usage: the SSO outage I caused

One day this autumn I disabled 19 service accounts at a customer, a manufacturing group with sites in several European countries. None had logged on in the last 90 days, the SAP-related ones not since 2022 or 2023 and some never. Defender for Identity listed them as inactive, and every one of them had an SPN pointing at a host that did not exist as a computer object in AD. It was a routine Secure Score clean-up item.

About thirty minutes later the SAP side raised an alarm: users signing in to SAP GUI were getting certificate and SNC errors. The alarm said certificate errors against SAP, nothing more.

It was my change. Nine of the 19 accounts were the Kerberos SPN targets for SAP single sign-on. They never log on themselves, so lastLogon had nothing to say about whether they were used. This post covers how that happened, how we got it back with two minutes of typing, and what I do differently now.

The change

The work was batch 2 of a Secure Score review, the on-premises identity part. It started with a read-only inventory of the domain’s service-style accounts, then the Defender for Identity export for “Remove stale service accounts”: 43 accounts, all with SPNs. Some were already disabled but never deleted, which still counts against the score. 19 were enabled.

The SIDs were resolved in AD and every SPN host looked up. None of the hosts existed as computer objects. For the SAP accounts the SPNs pointed at SAP hosts that had never been AD computers, and that was read as “the hosts are gone”. The accounts had the description “Service account for SAP AD SSO”, had passwords set in 2022 that never expire, and no group memberships. The conclusion was that this was an abandoned attempt at Kerberos SSO for SAP from 2022, because SAP SSO today was believed to run through two other accounts with constrained delegation towards a separate forest. No owner was listed, so no owner was asked.

That assessment was made by an AI assistant I had asked to work through the export, and it called the SAP accounts inactive and abandoned. I reviewed the recommendation and ran the change, so the decision was mine. I come back to what the assistant could and could not have known further down.

The plan was sensible as far as it went. Two phases: delete the already-disabled accounts, disable the 19 enabled ones and tag them with a deletion date 30 days out, then delete in a second run. The script had guardrails: no account in a privileged group, no logon within 90 days. It wrote a before-file with SID, GUID, UAC, group memberships, SPNs and description for every account. The dry run came back clean. The real run on DC01 took two seconds. Each disabled account got a description prefix like this:

[MDI inactive, disabled <date>, delete after <date + 30 days>]

The change log entry had the rollback written out before anything was touched. It runs per account on a domain controller or any machine with RSAT and the ActiveDirectory module, as a Domain Admin; the restore from the Recycle Bin needs that to read Deleted Objects:

Import-Module ActiveDirectory
# Example values; the real ones come from the before-file
$sam = 'svc-sap-snc'
$originalDescription = 'Service account for SAP AD SSO'

# Deleted accounts: AD Recycle Bin, SID and groups come back with the object
Get-ADObject -Filter "sAMAccountName -eq '$sam' -and isDeleted -eq `$true" `
    -IncludeDeletedObjects | Restore-ADObject
# Disabled accounts: re-enable and put the original description back
Enable-ADAccount -Identity $sam
Set-ADUser -Identity $sam -Description $originalDescription

The symptom

Around half an hour after the run, the alarm came in from the SAP side: certificate errors against SAP. Users of SAP GUI with Secure Login got certificate or SNC errors at sign-in. Seven SAP systems were affected, from development and test to production. No data was affected and nothing else broke.

A certificate error does not point at AD.

Working it out

The question was whether any of that day’s changes could affect this. The clean-up run that had disabled accounts described as SAP SSO accounts was the obvious candidate, and the link was made within five minutes of the alarm.

The mechanism took a moment longer to put into words. With SNC, the SAP GUI client asks the KDC for a service ticket for SAP/<SID>. The KDC finds the account that holds that SPN and issues the ticket. The SAP server, a Linux host outside AD, verifies it with a keytab generated from that account’s password. The account itself never authenticates to anything, so nothing in the flow updates its lastLogon.

When the account is disabled, the KDC stops issuing service tickets for its SPNs. The client gets no ticket, SNC cannot set up its secure context, and SAP GUI and Secure Login reported it as a certificate or SNC error, which is what the users saw. The error the user sees is several links away from the cause.

The rollback was cheap and already written, and the hypothesis was specific, so I ran it.

The cause

An account whose only job is to be a Kerberos SPN target never logs on. The 2022 lastLogon on these accounts showed when the keytab was created, not whether the account was used. Three of the nine had never logged on at all, and all of them were in use.

Three things reinforced each other:

  • Defender for Identity’s “inactive service account” check is built on the logon activity the sensor sees. It does not catch accounts that are only used as ticket targets.
  • Our own “host has no computer object” check was wrong for this case. It looked for the SPN hosts as computer objects in AD. The SAP hosts are Linux systems outside AD, so “the host does not exist” was the wrong conclusion. For a Windows service it would have been a reasonable signal, which is why it looked convincing.
  • Two separate SSO flows were mixed up. The delegation accounts serve the SAP web interfaces in the separate forest. The Kerberos accounts serve SNC in SAP GUI. Seeing one in use made the other look replaced.

On the AI part, plainly: the assessment that called the accounts inactive came from an assistant working from lastLogon, the MDI export and a computer object lookup. That data cannot show ticket issuance. Neither an assistant nor a person reading the same columns can tell from them that a Linux host outside AD holds a keytab and asks for tickets whenever someone signs in. The assistant reasoned consistently from what it was given; what it was given did not include the one signal that mattered. That signal is event 4769 on the domain controllers, and nobody looked there.

I don’t take “don’t use an assistant for this” from it. I take that “is this account used?” has to be answered from evidence of use, and that the person running the change has to know which evidence that is and ask for it.

The process failures were mine as well. There was no check with application owners before disabling, on the grounds that the accounts had no owner. A missing owner is a reason to ask more widely, not a reason to skip the question. And all 19 were disabled in one run. Waves, accounts without SPNs first and SPN accounts the following week, would have limited the damage.

The fix

The nine accounts were re-enabled with the rollback command from the change log. Formulating it took two minutes and running it took one. The original description was restored on each. Two of the SSO accounts had had an empty description, so the tag was cleared instead. Per account, with the ActiveDirectory module:

Import-Module ActiveDirectory
# Example values; the real ones come from the before-file
$sam = 'svc-sap-snc'
$originalDescription = 'Service account for SAP AD SSO'

Enable-ADAccount -Identity $sam
Set-ADUser -Identity $sam -Description $originalDescription
# For the two accounts that had no description before the change
Set-ADUser -Identity $sam -Clear Description

The SAP side confirmed sign-in worked again seventeen minutes after the accounts were re-enabled. That was half an hour after the alarm and about an hour after the original change.

The two-phase design mattered here. Because the accounts had been disabled and not deleted, the fix was Enable-ADAccount. Had they been deleted, Restore-ADObject from the Recycle Bin would have brought them back with the same SID, but the recovery would have taken longer.

Then the accounts were marked so this cannot happen quietly again. Each got a description that says what it is and why it must stay, and ProtectedFromAccidentalDeletion, so that deleting one takes a deliberate extra step. Again with the ActiveDirectory module:

Import-Module ActiveDirectory
# Example names; the real list was the nine SAP SNC accounts
$sapSncAccounts = @('svc-sap-snc01', 'svc-sap-snc02', 'svc-sap-snc03')

foreach ($a in $sapSncAccounts) {
    $u = Get-ADUser -Identity $a
    Set-ADUser -Identity $u -Description (
        'DO NOT DISABLE. SAP SNC Kerberos SSO, SPN target ' +
        '(never logs on itself). Remove with SAP phase-out.')
    # The parameter exists on Set-ADObject, not Set-ADUser
    Set-ADObject -Identity $u -ProtectedFromAccidentalDeletion $true
}
Get-ADUser -Filter 'Description -like "DO NOT DISABLE*"' `
    -Properties Description, ProtectedFromAccidentalDeletion |
    Select-Object SamAccountName, ProtectedFromAccidentalDeletion

Nine of nine came back protected.

The longer-term decision was to leave the accounts exactly as they are. The SAP team did not know the accounts existed and referred us to an external SAP consultant who could not be reached. Rotating the passwords would mean regenerating the keytabs on the SAP hosts, and nobody on the SAP side could do that. SAP is going to be phased out eventually. So the risk (passwords from 2022 that never expire, an obvious Kerberoasting target) is accepted and documented in the Secure Score row until then, and the accounts go with the SAP systems. We wrote down what we knew about the flow ourselves, account by account, for the next person.

The other ten disabled accounts are still disabled, with their deletion date unchanged. Their SPNs point at Windows servers that no longer have computer objects. They will still get the event 4769 check on all domain controllers before the second phase deletes them. For accounts that are already disabled, 4769 shows failed ticket requests rather than successful ones, so that check has to include failures.

This is a sketch of the check, run on each domain controller in an elevated session; the ActiveDirectory module is not needed. It is a general example, not something that was run during the incident. It only finds anything if Kerberos service ticket auditing is enabled on the DCs, it covers a 14-day window, and it reads the ServiceName and Status fields from the event XML rather than the localised message text. Failed requests (Status other than 0x0) count as evidence of use too, and for an account that is already disabled they are the only evidence:

# Example values: the accounts you intend to disable
$spnAccounts = @('svc-sap-snc', 'svc-sap-sso')

$filter = @{
    LogName   = 'Security'
    Id        = 4769
    StartTime = (Get-Date).AddDays(-14)
}
Get-WinEvent -FilterHashtable $filter -ErrorAction SilentlyContinue |
    ForEach-Object {
        $data = @{}
        foreach ($d in ([xml]$_.ToXml()).Event.EventData.Data) {
            $data[$d.Name] = $d.'#text'
        }
        [pscustomobject]@{
            Account = $data['ServiceName']
            Status  = $data['Status']   # 0x0 is success, anything else failed
        }
    } |
    Where-Object { $spnAccounts -contains $_.Account } |
    Group-Object Account, Status |
    Sort-Object Count -Descending |
    Select-Object Count, Name

A side note from the same run: the phase 2 filter used -like with the square-bracketed tag, which -like reads as a character class, and was fixed afterwards.

What I take with me

  • Before an account with an SPN is disabled, check event 4769 (Kerberos service ticket issued) on all domain controllers for the account’s SPNs over the last 14 days. Hits mean the account is in use, whatever lastLogon says. The event only exists if Kerberos service ticket auditing is enabled on the DCs, and a 14-day window misses rarely used accounts.
  • Disable SPN accounts in their own waves, a handful per run at most, at a time when the application owners can test.
  • Last logon is not usage. For an account that only exists to be a ticket target, the evidence of use is on the KDC, not on the account.
  • The error you see is the last link in a chain. A certificate error in SAP GUI was a disabled account in AD.
  • Identify before you delete, and disable before you delete: check that your test for “the host is gone” means the same thing for every kind of host, and keep the step that can be undone with a single Enable-ADAccount.
  • Keep the rollback command next to the change, written before you run it. It made the recovery two minutes of typing.