McAfee Nightmare

I'm so thankful that I'm not a McAfee admin anymore! This dat fiasco is making techs around the globe burn the midnight oil. Here's a clip from a USA Today article:

Solera Networks, a supplier of network forensics technology, says it helped one large U.S. multi-national company quickly determine that the poisonous update from McAfee threw 50,000 of its PCs into a rebooting frenzy. McAfee advised the company that "remediation time is estimated to be 30 minutes per user, " says Solera CEO Shillingford.

http://content.usatoday.com/communities/technologylive/post/2010/04/massive-manual-pc-cleanup-triggered-by-mcafee-error/1

Adobe Patches via WSUS

This note just came in from a patching list that I subscribe to:

"In a follow-up to a live chat he was in, Adobe's Brad Arkin revealed yesterday that they are working with Microsoft to integrate their updates into SCCM (System Center Configuration Manager), SCUP (System Center Updates Publisher), and WSUS (Windows Software Update Services). The work should be done by the end of the year."

That would be huge news for those of us that don't currently have a robust configuration management system.

Email Blunder

It's been awhile since I've posted, but I have a new blunder to share (not by me!). For the third time in my career, I've received a panicked phone call from a manager stating that he had accidentally fired off an email w/ sensitive salary information to a large audience. Most Exchange admins know how to delete a file using exmerge, but I thought I would put into writing some steps that can help address a situation like this as quickly as possible. Here are the steps:

• Have the user issue a recall from their system
• Immediately have your Anti-Virus engineer identify the file attachment as an “Unwanted Program” so that end users can not open it
  
• Run exmerge on all servers to remove the problematic email from end user mailboxes. Here's the link I always use when I need to do this task: http://www.petri.co.il/delete_messages_from_mailboxes_by_using_exmerge.htm

Good luck!

Back to the Future...

Two weeks ago I had one of the scariest “interruptions” that I’ve ever experienced in my 10 years of working as a professional geek. Let me take you back to two Saturdays ago…

Our building is on a pretty sketchy power grid and although we have a beefy UPS, we don’t have a generator backup. At about 2:00pm a nasty storm rolled and I was relieved that my son’s soccer game was cancelled. At 2:30ish, my phone rang; it was my boss telling me that the building had lost power. I’m the closest engineer to the building so I grabbed my laptop and headed into the office. We run a series of scripts that power down various systems to shed some load on UPS (this helps keep the core systems up longer). About 20 minutes later I get into the datacenter and try to access the server hosting the scripts via the Raritan console. As soon as I hit ctrl-alt-del, all power went out! No lights…no whirring sounds…no alarms…nothing. The UPS was cached. Literally a minute later, building power came back on and systems started coming back up. That’s when the adventure really began…

I went back to my desk so that I could use my workstation to monitor the systems as they came back online. One of my first tasks was to make sure mail was flowing again so I tried to log into one of our Exchange servers. My logon attempts failed and the error message indicated Kerberos problems. The event logs listed the following error:

I logged in locally to check the system time and everything looked ok.

I was able to successfully log into one of my domain controllers and immediately noticed the system time. It had changed to 8:45pm, February 28, 2002. Awhile ago I had reconfigured our NTP service to synchronize with one of our routers (at the request of our network manager)…I knew that something had to have gone wrong with that routers time. To correct the issue, I pointed my forest root PDC emulator to point to the US Naval Observatory’s ntp servers and forced a rediscover (w32tm /resync /rediscover) and the time corrected itself. I then sync’d the time on all of my other DCs.

Problem solved right? Wrong! I was able to get the Exchange stores to mount, but found another stomach turning problem. When I looked at the Directory Services logs, I saw nothing but red:

All of my domain controllers had been tombstoned due to the time changes! Fortunately the fix was clearly listed in the error message. I couldn’t demote and promote all of my DCs so I took step three listed below:

Event Type: Error

Event Source: NTDS Replication

Event Category: Replication

Event ID: 2042

Date: 5/31/2008

Time: 4:45:23 PM

User: NT AUTHORITY\ANONYMOUS LOGON

Computer: XXXXXXX

Description:

It has been too long since this machine last replicated with the named source machine. The time between replications with this source has exceeded the tombstone lifetime. Replication has been stopped with this source.

The reason that replication is not allowed to continue is that the two machine's views of deleted objects may now be different. The source machine may still have copies of objects that have been deleted (and garbage collected) on this machine. If they were allowed to replicate, the source machine might return objects which have already been deleted.

Time of last successful replication:

2002-02-28 20:11:01

Invocation ID of source:

0c86f6c8-f6b8-0c86-0100-000000000000

Name of source:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Tombstone lifetime (days):

60

The replication operation has failed.

User Action:

Determine which of the two machines was disconnected from the forest and is now out of date. You have three options:

1. Demote or reinstall the machine(s) that were disconnected.

2. Use the "repadmin /removelingeringobjects" tool to remove inconsistent deleted objects and then resume replication.

3. Resume replication. Inconsistent deleted objects may be introduced. You can continue replication by using the following registry key. Once the systems replicate once, it is recommended that you remove the key to reinstate the protection.

Registry Key:

HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and Corrupt Partner

Creating that registry key allowed replication to resume and later that evening I disabled the setting via my new best friend: GPO Preferences!

GPO Preferences - Rename Local Administrator

Our GPO Preferences deployment has been largely completed, with over 85% of our systems now running the client side extensions. One of our first uses of the new settings is to rename the Local Admin account on all workstations and servers. One thing I discovered during testing is that you can use system variables for naming the local admin account:



So you can name your local admin account %ComputerName%_Bob and each machine will have a unique local admin account that is easy to remember but unique enough to block a bot style worm even if your local admin password is compromised.

UPDATE on GPO Preferences.

So I manually kicked off a Windows Update session on one of my servers today and noticed that KB943729 was available as an optional download...I checked on our WSUS servers and sure enough, there it was! KB943729 is the Group Policy Preference Client Side Extensions which can now be automatically deployed to your enterprise via WSUS. Nice and simple!!!

GPO Preferences (Part 2)

OK...I thought I'd provide an update to my earlier post about implementing "Group Policy Preferences". I've created a virtual machine running Windows Vista as my GPO Management Console - since neither Windows XP or 2003 can manage the preferences policies. There were a few "Aha!" moments with the console configuration.

• GPMC is not available as a download for Windows Vista
• Here's a bit of a Catch 22:
• In order to manage the new GPO Preferences on Vista, you have to be running SP1
• The installation of SP1 removes GPMC from the OS!
• A little googling reveals that the Remote Server Administration Tools (RSAT) for Vista SP1 installs an updated GPMC.
• Microsoft instructs you to unistall all previous versions of administation tools before installing RSAT. After the RSAT installation, you have to do the following the view the toolset:
• Open Control Panel, click Programs, and then click Turn Windows features on or off under Programs and Features. If you are prompted to provide permission by User Account Control, click Continue.
  
• In the Windows Features dialog box, select the remote administration snap-ins and tools that you want to install, and then click OK.
• Configure the Start menu to display the Administration Tools shortcut.
• Right click Start, and then click Properties.
• On the Start Menu tab, click Customize.
• In the Customize Start Menu dialog box, scroll down to System Administrative Tools, and then select Display on the All Programs menu and the Start menu.
• Click OK. Shortcuts for snap-ins installed by RSAT are added to the Administrative Tools list on the Start menu.
  

Once you've taken those steps you can launch and see the familiar Group Policy Management Console. The only difference is the addition of a Preferences folder under the User and Computer configuration folders...