Email Blunder

It's been awhile since I've posted, but I have a new blunder to share (not by me!). For the third time in my career, I've received a panicked phone call from a manager stating that he had accidentally fired off an email w/ sensitive salary information to a large audience. Most Exchange admins know how to delete a file using exmerge, but I thought I would put into writing some steps that can help address a situation like this as quickly as possible. Here are the steps:

• Have the user issue a recall from their system
• Immediately have your Anti-Virus engineer identify the file attachment as an “Unwanted Program” so that end users can not open it
  
• Run exmerge on all servers to remove the problematic email from end user mailboxes. Here's the link I always use when I need to do this task: http://www.petri.co.il/delete_messages_from_mailboxes_by_using_exmerge.htm

Good luck!

Back to the Future...

Two weeks ago I had one of the scariest “interruptions” that I’ve ever experienced in my 10 years of working as a professional geek. Let me take you back to two Saturdays ago…

Our building is on a pretty sketchy power grid and although we have a beefy UPS, we don’t have a generator backup. At about 2:00pm a nasty storm rolled and I was relieved that my son’s soccer game was cancelled. At 2:30ish, my phone rang; it was my boss telling me that the building had lost power. I’m the closest engineer to the building so I grabbed my laptop and headed into the office. We run a series of scripts that power down various systems to shed some load on UPS (this helps keep the core systems up longer). About 20 minutes later I get into the datacenter and try to access the server hosting the scripts via the Raritan console. As soon as I hit ctrl-alt-del, all power went out! No lights…no whirring sounds…no alarms…nothing. The UPS was cached. Literally a minute later, building power came back on and systems started coming back up. That’s when the adventure really began…

I went back to my desk so that I could use my workstation to monitor the systems as they came back online. One of my first tasks was to make sure mail was flowing again so I tried to log into one of our Exchange servers. My logon attempts failed and the error message indicated Kerberos problems. The event logs listed the following error:

I logged in locally to check the system time and everything looked ok.

I was able to successfully log into one of my domain controllers and immediately noticed the system time. It had changed to 8:45pm, February 28, 2002. Awhile ago I had reconfigured our NTP service to synchronize with one of our routers (at the request of our network manager)…I knew that something had to have gone wrong with that routers time. To correct the issue, I pointed my forest root PDC emulator to point to the US Naval Observatory’s ntp servers and forced a rediscover (w32tm /resync /rediscover) and the time corrected itself. I then sync’d the time on all of my other DCs.

Problem solved right? Wrong! I was able to get the Exchange stores to mount, but found another stomach turning problem. When I looked at the Directory Services logs, I saw nothing but red:

All of my domain controllers had been tombstoned due to the time changes! Fortunately the fix was clearly listed in the error message. I couldn’t demote and promote all of my DCs so I took step three listed below:

Event Type: Error

Event Source: NTDS Replication

Event Category: Replication

Event ID: 2042

Date: 5/31/2008

Time: 4:45:23 PM

User: NT AUTHORITY\ANONYMOUS LOGON

Computer: XXXXXXX

Description:

It has been too long since this machine last replicated with the named source machine. The time between replications with this source has exceeded the tombstone lifetime. Replication has been stopped with this source.

The reason that replication is not allowed to continue is that the two machine's views of deleted objects may now be different. The source machine may still have copies of objects that have been deleted (and garbage collected) on this machine. If they were allowed to replicate, the source machine might return objects which have already been deleted.

Time of last successful replication:

2002-02-28 20:11:01

Invocation ID of source:

0c86f6c8-f6b8-0c86-0100-000000000000

Name of source:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Tombstone lifetime (days):

60

The replication operation has failed.

User Action:

Determine which of the two machines was disconnected from the forest and is now out of date. You have three options:

1. Demote or reinstall the machine(s) that were disconnected.

2. Use the "repadmin /removelingeringobjects" tool to remove inconsistent deleted objects and then resume replication.

3. Resume replication. Inconsistent deleted objects may be introduced. You can continue replication by using the following registry key. Once the systems replicate once, it is recommended that you remove the key to reinstate the protection.

Registry Key:

HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and Corrupt Partner

Creating that registry key allowed replication to resume and later that evening I disabled the setting via my new best friend: GPO Preferences!

GPO Preferences - Rename Local Administrator

Our GPO Preferences deployment has been largely completed, with over 85% of our systems now running the client side extensions. One of our first uses of the new settings is to rename the Local Admin account on all workstations and servers. One thing I discovered during testing is that you can use system variables for naming the local admin account:



So you can name your local admin account %ComputerName%_Bob and each machine will have a unique local admin account that is easy to remember but unique enough to block a bot style worm even if your local admin password is compromised.

UPDATE on GPO Preferences.

So I manually kicked off a Windows Update session on one of my servers today and noticed that KB943729 was available as an optional download...I checked on our WSUS servers and sure enough, there it was! KB943729 is the Group Policy Preference Client Side Extensions which can now be automatically deployed to your enterprise via WSUS. Nice and simple!!!

GPO Preferences (Part 2)

OK...I thought I'd provide an update to my earlier post about implementing "Group Policy Preferences". I've created a virtual machine running Windows Vista as my GPO Management Console - since neither Windows XP or 2003 can manage the preferences policies. There were a few "Aha!" moments with the console configuration.

• GPMC is not available as a download for Windows Vista
• Here's a bit of a Catch 22:
• In order to manage the new GPO Preferences on Vista, you have to be running SP1
• The installation of SP1 removes GPMC from the OS!
• A little googling reveals that the Remote Server Administration Tools (RSAT) for Vista SP1 installs an updated GPMC.
• Microsoft instructs you to unistall all previous versions of administation tools before installing RSAT. After the RSAT installation, you have to do the following the view the toolset:
• Open Control Panel, click Programs, and then click Turn Windows features on or off under Programs and Features. If you are prompted to provide permission by User Account Control, click Continue.
  
• In the Windows Features dialog box, select the remote administration snap-ins and tools that you want to install, and then click OK.
• Configure the Start menu to display the Administration Tools shortcut.
• Right click Start, and then click Properties.
• On the Start Menu tab, click Customize.
• In the Customize Start Menu dialog box, scroll down to System Administrative Tools, and then select Display on the All Programs menu and the Start menu.
• Click OK. Shortcuts for snap-ins installed by RSAT are added to the Administrative Tools list on the Start menu.
  

Once you've taken those steps you can launch and see the familiar Group Policy Management Console. The only difference is the addition of a Preferences folder under the User and Computer configuration folders...

Auto-configure Outlook via GPO

I was sitting in our weekly change management meeting last week and our help desk manager was expressing a desire to have an automated way of setting up user profiles in Outlook. He wanted to know if Altiris could do the trick. I chimed in that an easier solution would be to use group policy.

This is one of those needs that just about every enterprise has, and the solution is surprisingly simple. This guide should have you auto-configuring Outlook in your enterprise in about 20 minutes.

Step 1 - Build/Configure an Outlook "prf" file. The "prf" file contains all of the settings needed for Outlook to connect your mail environment. The prf file is generated using the Office Custom Installation Wizard built for the version of office that you are using (we use 2003). The wizard is pretty handy, but unnecessary. You can download (and modify) a sample prf from Outlook-Tips.net. If you want to get into the nitty gritty of the contents/functions of the prf file, see this article: Understanding An Outlook Profile (PRF)

Now, for an Exchange environment, copy and paste the following into the sample prf (NOTE: section 5 is blank so delete everything in the sample up to section 6 and replace it with the following). Customize everything in bold/red for your environment.

;Automatically generated PRF file from the Microsoft Office Customization and Installation Wizard

; **************************************************************
; Section 1 - Profile Defaults
; **************************************************************

[General]
Custom=1
ProfileName=Exchange User
DefaultProfile=Yes
OverwriteProfile=No
ModifyDefaultProfileIfPresent=False
BackupProfile=No
DefaultStore=Service1


; **************************************************************
; Section 2 - Services in Profile
; **************************************************************

[Service List]
ServiceX=Microsoft Outlook Client
ServiceEGS=Exchange Global Section
Service1=Microsoft Exchange Server
ServiceEGS=Exchange Global Section
Service2=Outlook Address Book

;***************************************************************
; Section 3 - List of internet accounts
;***************************************************************

[Internet Account List]

;***************************************************************
; Section 4 - Default values for each service.
;***************************************************************

[ServiceX]
CachedExchangeMode=0x00000002
CachedExchangeSlowDetect=TRUE

[ServiceEGS]
CachedExchangeConfigFlags=0x00000100
MailboxName=%UserName%
HomeServer=YourExchangeServer

[Service1]
OverwriteExistingService=No
UniqueService=Yes
MailboxName=%UserName%
HomeServer=YourExchangeServer
OfflineAddressBookPath=%USERPROFILE%\local settings\application data\microsoft\outlook\
OfflineFolderPath=%USERPROFILE%\local settings\application data\microsoft\outlook\outlook.ost
AccountName=Microsoft Exchange Server

[Service2]

;[ServiceX]
;FormDirectoryPage=
;-- The URL of Exchange Web Services Form Directory page used to create Web forms.
;WebServicesLocation=
;-- The URL of Exchange Web Services page used to display unknown forms.
;ComposeWithWebServices=
;-- Set to TRUE to use Exchange Web Services to compose forms.
;PromptWhenUsingWebServices=
;-- Set to TRUE to use Exchange Web Services to display unknown forms.
;OpenWithWebServices=
;-- Set to TRUE to prompt user before opening unknown forms when using Exchange Web Services.


;***************************************************************
; Section 5 - Values for each internet account.
;***************************************************************

Step 2 - Save this prf file to your DFS root (i.e. \\YourDomain\netlogon)

Step 3 - Create a custom adm template. In a previous blog entry I mentioned Yizhar Hurwitz's invaluable Reg2Adm. I used it to create a custom adm template for customizing Outlook. To save you some time, just copy the following into a text file and save it with a .adm extension (again, modify the red text):

CLASS USER

CATEGORY "Software\Microsoft\Office\11.0\Outlook\Setup"
KEYNAME "Software\Microsoft\Office\11.0\Outlook\Setup"

POLICY "ImportPRF"
PART "ImportPRF"
EDITTEXT
DEFAULT "\\YourDomain\netlogon\YourPRF.prf"
VALUENAME "ImportPRF"
END PART
END POLICY

END CATEGORY




Step 4 - Create your GPO. Kick of gpmc.msc, expand your domain and then scroll down to Group Policy Objects.

• Right click on Group Policy Objects and select New
  
• Name your GPO
• Now, right click on your newly created GPO and select edit
• Since this is a "user" setting, we'll need to add our custom adm to the User Configuration\Administrative templates
• right click on Administrative Templates under User Configuration and select "Add/Remove Templates..."
• Browse to and select the template you created in step two, click OK and then close the "Add/Remove Templates" window
• IMPORTANT - by default, the GPMC only shows settings that can be fully managed. Custom .adms are "preferences" that can be set via GPO, but can be overwritten by the end user. You need to modify the view in order to view the settings of your custom adm.
• Right click on your Administrative Templates folder under User Configuration and select View\Filtering.
• In the next window, uncheck the last option entitled "Only show policy settings that can be fully managed"

Then click OK

• Now expand the Administrative Templates folder and find the folder called "Software\Microsoft\Office\11.0\Outlook\Setup" (Note that this is for Outlook 2003 - you'll need to make some changes in the .adm file for 2007 so that it points to 12.0)
• In the right panel you should now see a red icon with a setting called "ImportPRF"
• Double click on the setting and enable the policy. It should now look like this:

Now, click ok.

That's it! Now all you have to do is link the GPO to appropriate OU(s) in your Active Directory tree and the setting will go into effect.








So, what's the magic behind this working? The first time Outlook runs, it looks for the existence of the following two registry keys:

HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\Setup\First-Run

HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\Setup\FirstRun

If Outlook doesn't find them it will look for the ImportPRF key. If it exists, Outlook will use the file defined in the ImportPRF key's path to configure Outlook.

Time yourself and see if you can get it done in 20 minutes or less!

Active Directory Schema Version

My co-workers and I were discussing deploying a test Exchange 2007 server this morning in an effort to demonstrate the benefits of upgrading from our existing 2003 implementation. The topic of extending our AD schema to support Exchange 2007 came up. We'll need to submit a change request to proceed so I suggested bundling that change with the schema updates brought by Server 2003 R2. One of the engineers thought that the schema was already at R2 so I thought I'd check. Checking the schema version is not something I do every day, and the only thing I remember is that I can check it via adsiedit. The rest is up to Google. The problem is that it's hard to find articles explaining how to find the schema version. It usually takes me multiple google queries to find the info. So...here it is - nice and simple:

Open ADSiEdit

At the top of the tree, right click on ADSI Edit and select "Connect To..."

In the "Connection Point" section hit the drop down menu and select "Schema".

Right click on cn=schema,cn=configuration,dc= yourdomain and select properties.

Scroll down to the attribute "objectVersion" and look at its value.

Here’s the breakdown:

13=Microsoft Windows 2000

30=Original release version of Microsoft Windows Server 2003 and Microsoft Windows Server 2003 Service Pack 1 (SP1)

31=Microsoft Windows Server 2003 R2

As you can see from this screenshot, we're still at Server 2003 (Not R2)

GPO Preferences

GPO Preferences (Part 1)

For a couple of years now I've being using a tool created by Yizhar Hurwitz called "Reg2Adm" to create custom adm templates from registry keys for Group Policy Objects. The custom templates are deployed as preferences which are set once, but can be overwritten by the end user. Yizhar's tool automatically converts the .reg file to the appropriate syntax and format for .adm files, thereby making virtually any registry setting configurable via GPO.

Well, Microsoft has now built a lot of this functionality (and much more) into the next generation of group policy. MS acquired a company called Desktop Standard, makers of a product called PolicyMaker. PolicyMaker's GPO tools have been integrated into the Group Policy fabric of Windows Vista and Server 2008. The new functionality is being called "Group Policy Preferences"

So...how does that help Windows 2003/XP systems engineers? Well, Microsoft has made client side extensions available for Windows XP and Server 2003 systems, enabling us to take full advantage of a broad suite of configurable settings. Everything from registry keys, to drive mappings, to power settings can now be easily and natively controlled via Group Policy. The only caveat is that the settings can only be configured via a gpmc running on a Windows Vista or Server 2008 platform (we'll be using a Windows Vista VM to create/manage the GPOs).

Over the next few weeks I'll be documenting my experiences with Group Policy preferences as I begin to deploy the extensions throughout our enterprise. We'll be using Altiris for the rollout. One of the big selling points to my management team was the GPO Preferences' ability to CENTRALLY MANAGE LOCAL ADMNISTRATOR PASSWORDS!!!

I'll keep you posted on the developments.

Exchange & "Manage auditing and security log" - OUCH!!!

So last week one of our security engineers emailed me with a request to be able to read all of the security logs on workstations and servers in the enterprise. Seemed like a simple enough request but...

By default everyone can view the System and Application logs, but only administrators can view the Security logs. I did not want to grant our security guy domain admin rights (so that he could also view the logs on our domain controllers), but knew that he would need to tweak auditing settings. The "Manage auditing and security log" group policy setting looked like a nice easy solution to the problem.

The settings configure the following:

This security setting determines which users can specify object access auditing options for individual resources, such as files, Active Directory objects, and registry keys.

This security setting does not allow a user to enable file and object access auditing in general. For such auditing to be enabled, the Audit object access setting in Computer Configuration\Windows Settings\Security Settings\Local Policies\Audit Policies must be configured.

You can view audited events in the security log of the Event Viewer. A user with this privilege can also view and clear the security log.

After getting approval to implement the setting, I proceeded to make the following changes.

• Created a security group called "Global Event Log Auditors" and added our security engineer to that group.
• The "Manage auditing and security log" settings had already been defined in the Default Domain Controllers policy so I just added my security group to the accounts listed.
• I enabled the setting on the Default Domain Policy and added "Global Event Log Auditors" to the policy.

All set (or so I thought). This all took place at about 5:30pm when most of the users had already gone home. At about 5:45pm, my Systems Center Operations Manager began throwing out alerts that the Information Stores on one of our Exchange servers (2003 flavor) had gone offline. I reacted right away, but made absolutely no connection to the change that I had made earlier. I repeatedly to mount the stores, but struck out each time!

The following alert kept popping up after the failed mount:

The store could not be mounted because the Active Directory information was not replicated yet.

After several failed mounting attempts (blush), I googled the error. To my horror, the following KB came up:

http://support.microsoft.com/kb/896703

Issues that may occur when the "Manage auditing and security log" permission is removed from the Exchange Enterprise Servers group in Exchange 2000 Server

So...by defining the policy, I actually overwrote/removed "Exchange Enterprise Servers" group's auditing permissions defined locally on the Exchange servers. YIKES...I quickly added the "Enterprise Exchange Servers" group to the "Manage auditing and security logs" setting in the Default Domain Policy, forced a gpupdate on the Exchange box and was back in business.

Luckily the blunder had not taken affect on our other Exchange servers yet!

Welcome!

OK...so here's the first official posting of what will be an ongoing collection of the challenges and solutions that I face in my life as a Systems Engineer...I'll be regularly updating this site with tidbits I discover that will hopefully make other SE's lives easier!

Andy Gibson