Windows Event IDs 6, 13, 82, & 10028: Cleaning up after the removal of a server that was hosting the Active Directory Certificate Services (AD CS) role

This post assumes that only one server was hosting the AD CS role and no certificates issued by this server are currently in use. If you have certificates in use, then please see the links at the bottom of the page.

The following items appear in Event Viewer with the same (or very close) timestamp, and events #10028 & #13 reference a server that no longer exists (highlighted in green in the 2nd and 4th screenshots that follow):

Event ID #6

Event ID #13

Event ID #82

Event ID #10028

This can happen when a Domain Controller or other Windows server hosting the Active Directory Certificate Services (AD CS) role is removed from the domain improperly. Sometimes this is unavoidable, for example if a server crashes. The solution is to remove the nonexistent server via Active Directory Sites and Services.

Step #1: Open Active Directory Sites and Services and show the “Services Node”

Select the top-most node in the tree,  then click on 'View' and select 'Show Services Node'

Select the top-most node in the tree, then click on ‘View’ and select ‘Show Services Node’

Step #2: Delete the offending (non-existent) server from the AIA, Enrollment Services, Certification Authorities, and KRA nodes

Note: it is absolutely critical that you do not delete any entries for servers that currently exist on the network!

Remove the non-existent server from the AIA node

Delete the non-existent server from the Enrollment Services node

Delete the non-existent server from the Certificate Authorities node

Delete the non-existent server from the KRA node

Step #3: If no other AD CS servers exist on the network, then remove all Certificate Templates. If other AD CS servers exist, then skip this step.

Note: Seriously — do NOT do this part if any other AD CS servers exist on the network. You have been warned. If there are any other AD CS servers, they’ll show up when you remove the non-existent server in step #2.

Delete Certificate Templates

Step #4: Cleanup

Run the following commands from an elevated command prompt:

certutil -dcinfo deleteBad
gpupdate /force

Further Reading

How to Decommission a Windows Enterprise Certification Authority and How to Remove All Related Objects

How to decommission a Windows enterprise certification authority and how to remove all related objects from Windows Server 2003

MS16-072 Breaks Security Filtering on Group Policy Objects


This update makes permanent changes in the way Security Filtering on GPOs needs to be configured. First, read this. Pay close attention to this part:

Symptoms

All user Group Policy, including those that have been security filtered on user accounts or security groups, or both, may fail to apply on domain joined computers.

Cause

This issue may occur if the Group Policy Object is missing the Read permissions for the Authenticated Users group or if you are using security filtering and are missing Read permissions for the domain computers group.

Resolution

To resolve this issue, use the Group Policy Management Console (GPMC.MSC) and follow one of the following steps:

  • Add the Authenticated Users group with Read Permissions on the Group Policy Object (GPO).
  • If you are using security filtering, add the Domain Computers group with read permission.

A note on that last item under Resolution in the quote above: You are technically always using Security Filtering on GPOs. By default, the Authenticated Users object is applied to the Security Filtering on every newly created GPO. If you are looking at a GPO in GPMC, and you notice that Security Filtering is blank, then that policy isn’t being applied to any users or computers. Due to the changes made by the update, the computer account always requires Read permission on the GPO — this is by design:

MS16-072 changes the security context with which user group policies are retrieved. This by-design behavior change protects customers’ computers from a security vulnerability. Before MS16-072 is installed, user group policies were retrieved by using the user’s security context. After MS16-072 is installed, user group policies are retrieved by using the computer’s security context.

The bottom line is that you can grant Read permission to either the Domain Computers or Authenticated Users group under the delegation tab of the GPO, and that will fix the problem with the GPO not being applied. Neither of those groups need Apply permission unless they contain users or computers that are being targeted by the GPO, that aren’t already being targeted by Security Filtering.

When I first encountered this issue, I noticed quite a few recommendations to “just rollback the patch” — this is not the correct course of action! The trouble caused by MS16-072 was not the result of a bug or poorly tested code, it was intentional, so if you remove the patch, you risk making yourself vulnerable to whatever threat(s) that were severe enough to prompt Microsoft into making such a potentially disruptive change. Besides, the “fix” is easy.

How might this affect your processes? What does this mean from here on out?

For Security Filtering to work on GPOs, you can no longer simply remove the Authenticated Users group, and then add your targeted objects (Users, Computers, Groups). The Authenticated Users group now requires Read access to the policy 100% of the time, otherwise the policy will not apply.

Why would anyone remove Authenticated Users from Security Filtering to begin with? Because, that group includes any User or Computer account on the domain, so let’s say you wanted a GPO to a subset of Users or Computers in an OU. In the past you would remove Authenticated Users from Security Filtering, create a new Security Group, add your desired Users or Computers to that group, then add that group to Security Filtering and link that GPO to proper OU. This is a convenient way to exercise precise control over GPOs.

Note, this update does not permanently take away this precision, it just means that if you had policies where the Authenticated Users group was removed, you need to add it back with Read access via the Delegation tab in Group Policy Management:

Add authenticated users back to GPO

But my GPO only targets a tiny subset of Users — I don’t wanna grant Read permission to the Authenticated Users or Domain Computers groups!

Ugh. Fine. The following will work but I don’t recommend it. If you recall from earlier, after this update User GPOs are retrieved via the Computer security context, so the Computer account of the computer that the user is logging into needs Read access to the GPO. Warning: this might will be hard to maintain in larger environments with lots of GPOs:

  1. Create a Security Group containing the Computers that are being used by the Users who require the settings contained by your GPO. For example, if your GPO targets user1, user2, and user3; and those users use computerA, computerB, and computerC respectively, create a Security Group that contains those computer objects.
  2. Grant Read permission on the GPO to that Security Group via the Delegation tab. If you don’t want to use a Security Group, then grant read access to the computer (if “computers” then you *really* should be using a group), directly.
  3. Apply Security Filtering to the users as you normally would.

Further Reading:

Locating the original MSI for software deployed via Group Policy


Click here for a scenario where you might need to do this.

Step #1: We need to find the GUID of the original MSI.

We can find this information in the registry of any machine that has the software installed. The information we need will be in one of two places, depending on the architecture of the package in question. If the architecture was x86 (32 bit), we will need to start here:

HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall

If the architecture was AMD64 (x64, or 64 bit) then we need to start here:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall

Simply navigate to either of the two aforementioned keys, select the Uninstall key and hit CTRL+F. In the window that appears, type a fragment of what you’re searching for (hint: use any part of the package’s name — e.g., “Box Sync”, “Google Apps”, “Lastpass”, etc. that shows up when you view the package from within Programs and Features) and then hit Find Next:

Searching the registry

For example, if you were looking for the GUID of the x64 Box Sync MSI and you searched “Box Sync”, you might see something like this, found under the HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall key:

Success! We have the package GUID

Or, if you were looking for the GUID of the x86 Google Apps Sync MSI, and you searched for “Google Apps Sync” it will be found under the HKLM\SOFTWARE\Wow6432Node\Microsoft\CurrentVersion\Uninstall key:

Success! We have found the _other_ package

The GUID has been highlighted in each of the previous screenshots — in the Box Sync example, the GUID is: {4E456910-7D68-4AD5-8A4A-CBD4A4635D0E}; in the Google Apps Sync example, the GUID is {091C294E-F243-432C-93E1-DEC4C2B9635B}. Copy it to notepad or the clipboard because you’ll be needing it soon.

Step #2: Use the GUID from the previous step to locate the key that contains the path to the MSI.

Now that you’ve found the GUID of the package you’re looking for, navigate to:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer

… and use the search function to search for the GUID:

Searching the registry... again.

You might have to hit F3 a few times, but eventually, you’ll find something similar to the following two examples — the info we need can be in either of the following REG_SZ keys: LocalPackage or ManagedLocalPackage:

Finally found the LocalPackage value

Finally found the ManagedLocalPackage value

The path to the original MSI will be in either the LocalPackage or ManagedLocalPackage REG_SZ values. With any luck, the file will still exist in that location.

You might be wondering why I didn’t tell you to navigate to HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer first, and search for the text — “Box Sync”, “Google Apps Sync”, etc. — but instead, I told you to find the GUID of the package that appears in the Uninstall key, then navigate to the Installer key and search for the GUID. Even with the extra step, finding the key that contains the LocalPackage or the ManagedLocalPackage REG_SZ entry goes faster if you know the GUID.

Removing software that was originally deployed via Group Policy


Problem: Software that was installed via Group Policy needs to be removed or upgraded and the original policy responsible for deploying said software no longer exists. Furthermore, attempting to remove the software manually via Programs and Features while logged into an account with Administrative privileges results in error messages similar (or identical) to this one:

Even with an "admin" account, the software cannot be removed manually.

Even with an “admin” account, the software cannot be removed manually.

Solution: We need to create a brand new GPO for software removal. Before we do, here’s a quick outline of what needs to be done:

  1. Locate the original MSI package and copy it to an accessible network location.
  2. Create a new unlinked GPO
  3. Use the new GPO to re-deploy the package (or packages — it is possible to remove multiple packages with the same policy.) Make sure that “Uninstall the applications when they fall out of the scope of management.” is checked on either the package, or the entire policy (or both.)
  4. Use security filtering to target the objects that need to have the software uninstalled. Make sure you read this post first, it might save you a bunch of time and frustration.
  5. Allow enough time to pass for Group Policy to refresh. Alternatively, force a refresh via reboots and/or gpupdate /force
  6. Remove the machines targeted in step #3 from Security Filtering.
  7. Repeat step #5

Let’s get to it then…

Step #1: Locate the original MSI package — read this post — and copy it to an accessible network location.

Step #2: Create a new unlinked GPO.

Create a new, unlinked GPO so that we don't accidentally apply it to the wrong OU, Users, Machines, etc.

Create a new, unlinked GPO so that we don’t accidentally apply it to the wrong OU, Users, Machines, etc.

Make sure to give the new GPO an obvious name.

Make sure to give the new GPO a descriptive name.

Step #3: Use the new GPO to re-deploy the package or packages — it is possible to remove multiple packages with the same policy. Make sure that “Uninstall the applications when they fall out of the scope of management.” is checked on either the package, policy, or both.

Configure the new policy to deploy the package

Configure the new policy to deploy the package.

Browse to the original MSI that was located in step #1

Browse to the original MSI that was located in step #1.

If you are only using this policy to remove one package, select "Advanced" in this dialog. Otherwise, select "Assigned".

If you are only using this policy to remove one package, select “Advanced” in this dialog. Otherwise, select “Assigned” and repeat this step for each package that needs to be removed, then skip the next screenshot.

If you selected "Advanced" in the previous dialog, check the box titled "Uninstall this application when it falls out of the scope of management"

If you selected “Advanced” in the previous dialog, check the box titled “Uninstall this application when it falls out of the scope of management”.

If you are using this policy to remove multiple packages, there is no need to specify "Uninstall this application when it falls out of scope of management" for each package, instead, you can apply it to the entire policy. Right click on "Software installation" and select "Properties"

If you are using this policy to remove multiple packages, there is no need to specify “Uninstall this application when it falls out of scope of management” for each package, instead, you can apply it to the entire policy. Right click on “Software installation” and select “Properties”. Check the box “Uninstall the applications when they fall out of the scope of management” in the next screenshot.

Check the box "Uninstall the applications when they fall out of the scope of management".

Check the box “Uninstall the applications when they fall out of the scope of management”, and hit “OK”.

Close the Group Policy Management Editor when you are done configuring your policy

Step #4: Use security filtering to target the objects that need to have the software uninstalled. Make sure you read this post first, it might save you a bunch of time and frustration.

In the next few steps, I’m going to use Security Filtering to target only the machine that needs this policy. In the examples that follow, the machine name is W7PRO64-STAGING. You could just as easily Security Filtering to filter on a Security Group containing users, computers, or both; or you could target a specific user in the same way that I’m going to target the machine in my example. Just remember that the standard rules about OUs still apply: Regardless of whether you use a Security Group or whether you target the object (user or computer) directly, the targeted object needs to exist within the same OU as the policy is linked. Also remember that if your policy is supposed to target User objects, then any settings need to be configured under User Configuration; if it targets Computer objects, then your settings need to be configured under Computer Configuration.

Click the "Add..." button to add the targeted objects (Users, Computers, or Groups)

Click the “Add…” button to add the targeted objects (Users, Computers, or Groups)

We do not wish to target all Authenticated Users in the OU that this policy will eventually be linked , so we need to remove the Authenticated Users group from Security Filtering.

We do not wish to target all Authenticated Users in the OU that this policy will eventually be linked , so we need to remove the Authenticated Users group from Security Filtering.

... But, due to the previously mentioned post, the Authenticated Users group does require Read access to this policy. So click on the "Delegation" tab, click "Add", type "Authenticated Users" into the box titled "Enter the object name to select", click "Check Names", then click "OK".

… But, due to the previously mentioned post, the Authenticated Users group does require Read access to this policy. So click on the “Delegation” tab, click “Add”, type “Authenticated Users” into the box titled “Enter the object name to select”, click “Check Names”, then click “OK”.

Make sure that "Read" is selected in the "Permissions" dropdown box, and then click "OK".

Make sure that “Read” is selected in the “Permissions” dropdown box, and then click “OK”.

You should end up with something like this

You should end up with something like this

If you want to be certain your permissions are correct, click "Advanced", then click "Advanced" on the dialog that appears, and make sure that your targeted Users, Computers, or Group, have "Read" and "Apply" permissions and "Authenticated Users" only have "Read" permissions.

If you want to be certain your permissions are correct, click “Advanced”, then click “Advanced” on the dialog that appears, and make sure that your targeted Users, Computers, or Group, have “Read” and “Apply” permissions and “Authenticated Users” only have “Read” permissions.

Now all that's left to do is link the GPO to the OU

Now all that’s left to do is link the GPO to the OU

Select the GPO to be linked, and click "OK".

Select the GPO to be linked, and click “OK”.

Step #5: Allow enough time to pass for Group Policy to refresh. Alternatively, force a refresh via reboots (at least two reboots) and/or gpupdate /force

One way to tell if the MSI deployed by your “removal policy” was actually re-deployed is by checking the the install date (… found in the “Installed On” column) in Programs and Features — it should be updated to the date that the software was re-deployed.

Some things can go wrong here:

  • Group Policy might not refresh for a long time, or not at all. Try enabling the following two items on the policy (if it is being applied to a Computer). These settings need to be applied to the Computer, so you may need to create a new policy and link it to the OU containing your targeted Computer objects.
    1. Navigate to Computer Configuration\Policies\Administrative Templates\System\Logon and set Always wait for the network at computer startup and logon to Enabled
    2. Navigate to Computer Configuration\Policies\Administrative Templates\System\Group Policy and set Configure software installation policy processing to Enabled and check both Allow processing across a slow network connection and Process even if the Group Policy objects have not changed
  • The MSI you thought was the “original” turns out not to be, and you end up with something like this in Programs and Features:
    The software contained in the MSI deployed by your "removal" policy may have been the same version, but the MSI was not identical to the original. In this case, you need to try to retrieve the original MSI (... for the software that you were trying to remove in the first place) from a machine that still has the software installed.

    The software contained in the MSI deployed by your “removal” policy may have been the same version, but the MSI was not identical to the original. In this case, you need to try to retrieve the original MSI (… for the software that you were trying to remove in the first place) from a machine that still has the software installed.

    This happened to me once when I tried to use this technique to remove Google Apps Sync. Instead of retrieving the original MSI (as per my instructions, linked above in step #1) I downloaded an MSI from Google that was the same version of the software I was trying to remove, taking a gamble that the downloaded MSI might be identical to the originally installed one. You see the result. If this happens to you, don’t panic. Just proceed to step #6 as if it never happened. Once the duplicate copy has been removed, you can repeat step #1 (or actually do step #1 in case you skipped it the first time, which may be the reason you’re in this mess), skip step #2 (because you already have a policy that can be re-used) and then complete the rest of the instructions, starting from step #3.

Step #6: Remove the objects targeted in step #3 from Security Filtering and/or unlink the policy.

Do either (or both) of the following:

Navigate to the OU in which the policy is linked, right click on the policy, and select "Delete". When prompted with the following: "Do you want to delete this link? This will not delete the GPO itself." Click "OK".

Navigate to the OU in which the policy is linked, right click on the policy, and select “Delete”. When prompted with the following: “Do you want to delete this link? This will not delete the GPO itself.” Click “OK”.

Remove the targeted group from Security Filtering. When asked "Do you want to remove this delegation privilege?" Click "OK".

Remove the targeted group from Security Filtering. When asked “Do you want to remove this delegation privilege?” Click “OK”.

This will force an “out of scope” status, and the software should be uninstalled after Group Policy refreshes.

Configuring FreeNAS 9.10 to communicate with an APC Network Management Card

These instructions should work for just about any APC Network Management card (NMC), but the one I’m using is the AP9631.

Step #1: Navigate to the UPS configuration options

There are two ways to get into the UPS configuration:

  1. Navigate to Services -> UPS in the sidebar, or …
  2. Click on Services at the top of the page, then click the wrench icon next to UPS

It makes no difference which one you choose.

There are two ways to open the UPS configuration options. The end result is the same no matter which you choose.

There are two ways to open the UPS configuration options. The end result is the same no matter which you choose.

Step #2: Configure the UPS

As stated earlier, I’m using an AP9631 so I’ll select the driver closest to that model. In this case, it is the APC ups 3 (various) AP9630 monitoring card (snmp-ups privProtocol=AES) driver:

Select the most appropriate driver

Select the most appropriate driver

… And then continue with the configuration.

UPS configuration options

UPS configuration options

  • Identifier is how FreeNAS will refer to your UPS.
  • Port is the IP address of the network management card
  • Shutdown mode can be one of the following (you’ll have to decide which one of these is appropriate for your environment):
    1. UPS reaches low battery
    2. UPS goes on battery
  • Monitor User is the username of the user that exists on the NMC that will be used to issue shutdown commands. The AP9631 ships with a user called “device”, but you need to change the password for this user when you configure your NMC.
  • Monitor Password is the corresponding password for the NMC user.
  • Remote Monitor this option allows other servers to monitor the UPS via the NAS. You may or may not need this functionality.
  • Send Email Status Updates and To email these should be self explanatory.

Once you’ve saved your configuration, start the service:

Start the UPS service

Start the UPS service

Step #3: Verify the UPS service is communicating with the NMC

Open a terminal and ssh to your NAS and run the following command (where APCNMC is the Identifier you entered in your UPS configuration):

upsc APCNMC

You should see something like this:

ambient.1.humidity.alarm.high: 60.00
ambient.1.humidity.alarm.low: 30.00
ambient.1.temperature.alarm.high: 40.00
ambient.1.temperature.alarm.low: 10.00
ambient.humidity: 0.00
ambient.temperature: 34.0
battery.charge: 100.00
battery.date: 10/05/2014
battery.runtime: 2160.00
battery.runtime.low: 600
battery.voltage: 26.80
device.mfr: APC
device.model: Smart-UPS 1000
device.serial: AS1440225753
device.type: ups
driver.name: snmp-ups
driver.parameter.pollinterval: 2
driver.parameter.port: 10.0.3.6
driver.parameter.privProtocol: AES
driver.parameter.synchronous: no
driver.version: 2.7.3
driver.version.data: apcc MIB 1.2
driver.version.internal: 0.72
input.frequency: 59.90
input.sensitivity: high
input.transfer.high: 127
input.transfer.low: 106
input.transfer.reason: noTransfer
input.voltage: 120.10
input.voltage.maximum: 120.10
input.voltage.minimum: 120.10
output.current: 2.70
output.frequency: 59.90
output.voltage: 120.10
output.voltage.nominal: 120
ups.delay.shutdown: 120
ups.delay.start: 10
ups.firmware: UPS 09.2 (ID18) 
ups.id: APCUPS
ups.load: 43.50
ups.mfr: APC
ups.mfr.date: 10/05/2014
ups.model: Smart-UPS 1000
ups.serial: AS1440225753
ups.status: OL
ups.temperature: 34.10
ups.test.date: Unknown
ups.test.result: Ok