Fiction Writing and Other Oddities

Tuesday, May 29, 2007

Why doesn’t my Blackberry work? And other Techno-geek problems…

This one entry in my blog is a techno-geek blog and not a whiny writer's entry or even an entry about writing in general. Sorry. Next week. Or see my previous entry which sort of mashed my techno-geek and writing side together in a very uncomfortable mix.

So. You've been warned.

So. For all you desperate admins out there searching the Internet for help on why some of your users' Blackberry's aren't working…or why some of your users can't publish their certificates to the GAL…or why some of the permissions you've delegated in Active Directory don't seem to work all the time…or other anomalies like that…

This blog is for you.

Why Things Just Don't Work As Expected

Having problems with permissions in AD? Problems with Blackberries? Weird anomalies when trying to reset passwords?

I thought so.

Sometimes being an enterprise administrator for a very large organization is just…depressing. Since we've upgraded to Windows 2003, I've learned that Microsoft is serious about best practices such as not using your standard, mail-enabled/mailbox-owning user account for administrative tasks. And I've grown serious about it, too since I've begun to appreciate security. Now, I'm so serious that I'm thinking fondly of starting the Guido School of Admin Training (with sincere apologies to any existing Guidos or schools already named thusly).

Our Guido School of Admin Training is a very informal school held outside in the nice, fresh air in the alley between two office buildings. There are no formal registration procedures, however you do have to be nominated to attend. Admins just need to show up for class to begin.

What to expect: The instructor, Guido, will shake your hand and then gently haul you over to the nearest wall by your collar. Then it becomes really exciting and lots of fun. With a jaunty smile, Guido grabs you securely by the back of the neck and smacks your face against the wall while saying in a firm tone of voice:


And then, after a slight pause for refreshments:

mail-enabled/mail-box associated

Now, if you can repeat what you just learned, you receive a diploma and a short ride to the nearest hospital.

Our school guarantees success. For the rest of their life, students will remember what they've been taught, even if they imprinted on Windows NT and never learned any other operating system and refuse to use an administrative account because it's just too much trouble. Even if they repeatedly put a group containing all 8,000 users in their domain (within your AD tree, mind you) into Domain Admins because they thought that would be the simplest way to deploy something, say, a patch. And after they remove the group containing all the users from Domain Admins and they suddenly get a rash of calls about how all the managers' Blackberrys are all broken, they call you to fix it.

If this happens, it's time to send the admin to Guido's school, so just take the bull by the horns and nominate them.

Because now you have to deal with the admincount attribute.

You see, protected groups are special. They need to be special because in the past, fanatical admins have removed critical permissions from key groups instead of simply not putting unnecessary admin accounts into those groups to begin with. After removing critical permissions, these fanatics have lost control of their domain, or worse, their AD Tree or even Forest. Okay, maybe that's not the entire reason for protecting these key groups, but it is certainly one good reason. There are a lot of others and this blog isn't long enough for all of them.

So in Windows 2003, Microsoft helped admins avoid such idiocy by developing a mechanism to put back critical permissions on certain key groups called protected groups. Organizations which follow best practices for administration and security are most likely completely unaware of this secret mechanism and don't need to worry about it. If you follow best practices, it will have no impact on you and never will. You deserve to be worshiped as the deity you obviously are.

For the rest of us, we need to learn this rule: You don't put standard user accounts associated with a mailbox into any protected group; and you don't nest groups into protected groups (because you lose track of what you nested and could potentially nest groups containing standard user accounts into protected groups and elevate permissions that should not be elevated).

If you violate this rule, the admincount attribute will afflict you mightly. (And yes, I used "afflict" on purpose so don't leave me a lot of obnoxious comments about that. It's funny. Laugh, darn you.)

Protected Groups

What are the protected groups? With Windows 2003, they include:

Schema Admins

Enterprise Admins

Cert Publishers

Domain Admins

Account Operators

Print Operators

Administrators (domain local)

Server Operators

Backup Operators

How are they protected?

One piece of the mechanism protecting these special groups is an attribute called the admincount.

  • The protected groups have the admincount attribute set to 1.
  • Any group or user account nested into a protected group gets the admincount set to 1.
  • Any user nested into a group nested into a protected group gets the admincount set to 1.

The only exception is if there is a user account or group from another domain in your AD tree nested into a protected group, like the domain local Administrators group. The group/accounts from the other domain will not be affected. (But that does not mean you should be a knucklehead and use your standard, mailbox-associated user account for administrative purposes in other domains. Come on, grow up.)

Normally, the admincount is not set at all (it is null).

Every fifteen minutes or so, the operating system looks for the admincount attribute. If it finds it, it does some interesting things. It removes inheritance from the object so it will not inherit the permissions it might once have inherited from its parent OU or OU structure. This prevents unfortunate things from happening if you move one of the protected groups out of the Builtin container to a different OU or container where you might have diddled with the permissions it inherits might cause a breach of security or worse. Like a Help Desk person with full control over a user OU suddenly gaining full control over Domain Admins because some knuckle-dragging yahoo put Domain Admins into that OU. Stuff like that.

In addition, if an object like a user account has the admincount attribute set, the system strips certain key permissions granted to that account or group by the schema definition for that object type at the moment of creation. Specifically, most of the SELF permissions. These are the permissions that let a user account, for example, publish a certificate to AD (and hence, to the global address list or GAL if you are running a product like Exchange 2003).

So this could have the impact of preventing a user from publishing a cert.

Or it may break a user's Blackberry if the user happens to be put into a group like Domain Admins, because generally the Blackberry's service account is granted permissions through the OU hierarchy. These permissions for the BB service account are necessary for the service account to send/receive mail to/from the user's Blackberry to their Exchange mailbox. So if the user's account is affected by the admincount attribute, it will stop inheriting permissions from the OU, the BB service account will not get the permissions it needs, and this user will have a brick instead of a Blackberry.

Just a few examples. I've also seen it cause apparent permission anomalies where an admin grumbles that AD is broken or doesn't work well because sometimes they can't manage an object they supposedly have permissions to manage. When I get that type of call, the first thing I check is if the admincount attribute is set on either the admin's account or on the object they are trying to manage. Most of the time, one or the other (or both) has the attribute set and it's time to nominate the admin for my special school.

Because if a user object can't inherit permissions and has some self-permissions removed, how well do you think you're going to be able to manage an account with delegated permissions? How well will an account thus damaged work with inherited and/or delegated permissions? Not well, my friend. Not well at all.

The Kicker

Once a user account ( or group) is "contaminated" by the admincount attribute, it doesn't come clean just by removing the user account or group from the protected group. Oh, no. You have to fix it.

The nice thing about this is that you can always tell when some admin really needs to meet Guido out in the alley. You can just run ldifde and export a file containing all the users and groups with the admincount attribute set to 1. And of course the event logs on your domain controllers will show membership changes to protected groups so if you catch the problem soon after it occurs, you can see exactly who needs that training.

Here is an example of the ldifde command that will provide you with a text file called ac.txt of all users and groups with the admincount set in a domain (in the example, the domain is

Ldifde –f ac.txt –d dc=example,dc=com –l samaccountname –r "(admincount=1)"

Note: I like to include the samaccountname attribute in the output (it will by default include the distinguished name) because it helps me in other processes—but this is entirely optional.

You might think that the admincount is a bad thing. It is not. It is your friend because it shows when you are not following best practices and are, in fact, endangering the security of your enterprise. So I'm not in favor of turning off this functionality.

I'm in favor of training and implementing best practices that create a reasonable security model based on the use of a separate account for administration. This secondary account should not be mailbox enabled or associated with a mailbox.

Fixing It

Here is how to fix it if things weren't done exactly right in the past. I will warn you, however, that if you have a lot of user accounts affected (as I have had to fix multiple times for multiple domain admins) this can be a somewhat time consuming process, even if you script it up.

Fix Protected Group Membership

First, you must make sure you undo the cause, which is to say, make sure you don't have any groups nested into protected groups. If you do, remove them. There is no point in trying to cleanse things if you leave the source of contamination. Leaving groups nested into the protected groups means at some future date, you will be addressing this same problem again because people forget and put the wrong accounts into "innocent looking" groups.

So remove all nested groups and remove all standard user accounts that are associated with a mailbox. Just leave your agreed-upon administrative accounts which can and should have the admincount attribute set. Naturally, you won't be an idiot and remove all the accounts before you add the new administrative accounts, because you might find yourself suddenly unable to actually add the new admin accounts, particularly if you empty out Domain Admins without putting any new accounts into it, first. (Just an FYI. I know you know these things but I like to state the obvious. It's so satisfyingly…obvious.)

Clean the Groups and Users Removed

Once you have removed any nested groups and innocent user accounts, you can clean them. You have to clean both the user accounts and the nested groups, if any.

First, reset the admincount attribute to 0 (or null) on the nested groups and users. Null is best, but 0 will work and sometimes it is nice to set it to 0 because you then have a historical artifact you can search for later if you have other issues. A 0 will tell you that this user or group was afflicted by the admincount at one point. (Just as you are afflicted by whichever admin did this to the user or group.)

For one or two users and groups, you can simply edit them in adsiedit.msc, which will allow you to reset the admincount attribute. You can also script it up if you wish. (I use a script for bulk cleaning.)

If you are using adsiedit.msc, you should take the following steps:

  • Right click the user (or group) and select Properties.
  • On the Attribute Editor tab, find the admincount attribute. Select it and click the [Edit] button. Click on the [Clear] button (or set the value to 0 if you want the historical artifact). Click [Ok].
  • Select the Security tab
  • Click on the [Advanced] button. Click on the [Default] button. This will restore the removed permissions PLUS it will put a check mark next to the "Allow inheritable permissions…" box, which you want.
  • Click on [Ok] until you close out that user's properties.

Unfortunately, as you see, in addition to clearing the admincount, you have to reset (turn on) inheritance for that object (group or user). Finally, you must give it back the permissions that object normally gets when it is first created. These permissions are not inherited, they are defined in the schema for that object and are granted to the object when it is created. If you use certificates, you're going to want these permissions and that's why the [Default] button is so handy. It restores all those things for you.

DSACLS can also be used to restore inheritance and reset an object back to its default "state" by using the /P:N /S switches.

There is obviously a lot more to be said about this, including administrative practices, best practices, and security whys & wherefores, but I'll be here all day if I don't stop somewhere.

There is a relevant KB article, "Delegated permissions are not available and inheritance is automatically disabled" KB817433, but I don't recommend doing the workaround. In fact, I generally don't like referencing that KB article because it includes that workaround and some knuckleheads always want to do the workaround instead of just tackling the problem and fixing it properly.

I recommend doing things the right way so you don't have to deactivate something that is there to help you and prevent you from doing something really egregiously stupid that could cost you the control of your domain or AD forest.

So…enough already.

Good night and sweet dreams.


Icy said...

Outstanding post, and not something I'd run across before. This explains many things...

Anonymous said...

I am an onsite support tech for a company (a.k.a. 3rd party vendor). I deal with an 'admin' who refuses to listen to me about NOT putting standard users into the domain admin group. I can't wait to have him read this and then have a meeting!

Amy said...

I've had the same problem, where "helpful" admins add everyone to the Domain Admins group thinking it was a quick way to get things done. And I've subsequently spent hours cleaning up the mess, afterwards, to get things like Blackberries working again.

Sometimes, there is no helping admins.

I feel your pain.