Note: Sorry this wasn’t posted sooner, there was a bit of a shake-up internally as we tried to decide what all was appropriate to post.  I’ve had this post ready for a few days now and have just been waiting for definitive answers from my management.  This post represents nearly complete OCS deployment.  By the time it ends, we have Enterprise Voice complete.  The remaining things we will deploy are the archiving server, the QoE monitoring role, and edge servers.

1:07 PM : Creating UM Dial Plan

Note: there are three important things here.  The first is the dial plan name.  You’ll see that when I create the location profile in OCS that the name is slcutloc.extendhealth.com.  That must match.  Second is the URI type – it must be SipName for OCS integration.  The last thing is VoIP security, which should be Secured for OCS.  (Secured > SipSecured)

Have to add the dial plans to the UM servers – both mail1 and mail2.

1:20 PM : Running ExchUCUtil.ps1

Verified IP gateways.  If there were more, I’d have to disable them.

1:31 PM : Creating Location Profiles

I’m not going to comment on this much as there is a lot to say.  Screen caps should be sufficient to let you know what I’m doing.

2:07 PM : Running OcsUMUtil.exe

The last step is to integrate from the OCS side by running OcsUMUtil, which creates OCS objects for the auto assistant and subscriber access numbers in Exchange UM.  This facilitates access to these numbers from Communicator.

2:10 PM : Assigning a Default Location to the Pool

2:15 PM : Configuring Mediation Servers

2:22 PM : Configuring Policies and Phone Usages

8:08 AM : Loopback Fix

I’ve been here for a while, catching up on some of my non-blog communication, MBA coursework, etc.  About ten minutes ago, I started testing a probable fix for the validation error I had last night.  Just as a reminder, that validation error looked like this:

The fix is recorded in Appendix D of the Office Communications Server 2007 Enterprise Edition and Communicator 2007 Deployment Guide.  In a nutshell, you need to add a multi-string value to HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0.  The MSV should be named BackConnectionHostNames and should have a value of your pool’s FQDN.  What this does is allow IIS to validate certain FQDNs as being valid for loopback.  You’ll want to remove this value when you’re not validating, and more detail is available by reading the referenced guide.

When I followed the instructions for the fix, the validation wizard for the remaining steps executed properly.

8:16 AM : Validation Wizards

(Yes, that’s a different validation wizard.)

(Yes again.)

8:23 AM : Validation Results

So the current state of our deployment is that there are two validation warnings, neither of which I care about because I haven’t deployed Enterprise Voice or edge access yet.

From the Validate Front End Server Configuration wizard, we have:

From the Validate Web Components Server Functionality wizard, we have:

8:27 AM : Internal Deployment Complete

Aside from the above validation warnings, it seems that internal deployment is complete.  I do have one more warning in my Communicator client regarding Exchange Web Services, but the Exchange deployment on this domain isn’t complete yet, so it’s also expected.  The ramification at this point is that Communicator can’t automatically set my status to “In a Meeting” if I have a meeting scheduled in Outlook.

Next step is external user access, meaning I’ll be bringing up a scaled single-site edge topology.  I’ll try to explain that in more detail, but there will probably be some downtime here as I test Communicator internally and prep another couple of servers to be edge servers.  (I have to install Server 2003 at least.)

1:53 PM : Enterprise Voice

1:56 PM : Activating Mediation Server

2:00 PM : Assigning Certificates

3:16 PM : Enterprise Voice Prep

I’ve been reading (and will continue to read through) the Microsoft Office Communications Server 2007 Enterprise Voice Planning and Deployment Guide.  This will probably take the rest of the day and will ensure that I make minimal mistakes when deploying Enterprise Voice.  I have a good idea of what it is that I need to do, but I want to be certain.

All of these steps and screenshots were performed late last night.  I’ll fill in commentary now (morning of Day 8).

Back Story

I was crushingly disappointed when Microsoft told me that I’d have to reinstall my entire PKI because the hashing algorithms I used were for a Cryptography Next Generation (CNG) CSP, not a CryptoAPI Version 1 CSP.  Knowing what I know now, I can see some allusions to that on pp. 158-159 of Brian Komar’s book.  Before I left work yesterday, I e-mailed Brian and explained my situation and that I was on a support call with Microsoft.  I then updated him via e-mail of their response (“it’s not supported) and the fact that they were closing the support case.

He sent this response:

Mark,

There is a security update that will allow XP and 2003 clients to validate certificates that implement SHA-2 signatures.
The update is included in Windows XP service pack 3.
Per the release notes for service pack 3:

You cannot create these certs in 2k3, but you would be able to validate them.

Brian

Based upon that hope, I went out and did some strategic searching and came across this KB: http://support.microsoft.com/kb/938397.  After an hour of waiting on hold while some (nice enough) tech researched the history on my support case, I was finally given a link to download the hotfix.  Note that there is a link there to register for the hotfix also, which I did, but was told that it would take up to 24 hours.  It actually took about two hours. 

Hotfix in hand, I patched the server and all the certificates looked great!  There were still a couple of strange artifacts with how I had to request certificates, but I was able to do it without incident.

Now that the back story is complete, I’ll try to recreate the timeline as best I can based upon the timestamps in my screencaps.  Thanks, OneNote!

8:50 PM : Assigning the Certificate to IIS

This is where things went awry yesterday.  If you want to know what to do to get to this point, read that post.

8:52 PM : Starting Services

I’m deviating here from the norm of not including the wizard starts in the screen captures.  The final screen of a wizard generally has useful information (like success, hopefully), but the start of a wizard usually just says what it is you’re doing.  Since I generally label what it is that I’m doing already, I had been skipping the first screen for the wizards.  At this point, however, the wizards start to blur together, especially in the validation phases.  Therefore, I’m going to include some wizard start screens if I can to differentiate the wizards.  (That said, I think I noticed last night that all the validation wizards start with the same screen anyway.)

9:29 PM : Server/Pool Validation

[Delay reason: had to put my son to bed.]

Oops… in order to validate the server and pool functionality, I need a couple of user accounts to be enabled for Office Communications Server.  The trick to this is that you have to use Active Directory Users and Groups to enable the users, but you also have to have the OCS Administrative Tools installed on that computer.  Because my domain controller is Server 2008, I can’t install the OCS Administrative Tools there (and be supported).  In this case, I just opened an MMC on ocsfe1, added the Active Directory Users and Groups snapin, and connected to the extendhealth.com domain.  Right-clicking on users now exposes the following option:

Now that the users are enabled, I can see them if I open the Office Communications Server snapin (Start > All Programs > Administrative Tools > Office Communications Server 2007).

9:36 PM : Back to Validation

Note that I didn’t check test connectivity of federated users because I don’t have external access yet.

This was the only warning I had.  Since I haven’t deployed Enterprise Voice yet, I’m not concerned about this warning.

11:15 PM : More Validation

I think I took some time before this screenshot to correct some previous validation errors, but I can’t recall very clearly.  I do want to note that I ran into some validation errors last night, as the following screenshot shows:

I believe this particular screenshot is an artifact of a known issue with IIS loopback, so I’ll try to fix it this morning.  I didn’t think it was important last night since I recalled how to deal with it (although not the specific steps) and since the server and pool validated okay.

11:23 PM : The Payoff

Enough said.

8:33 AM : Picking Up Where We Left Off

As you may recall, I ran into an issue last night just before I left because I didn’t have the SQL client tools necessary (specifically the SQL 2005 Backwards Compatibility Pack and the SQL Native Client) installed on my front end server ocsfe1.  I did try installing the tools this morning to no avail – unfortunately I wasn’t even getting a good quality error message, just “Pool backend discovery failed” – the same message I posted yesterday.

I’m pursuing a workaround at this point for two reasons:

1. I need to keep the ball rolling.  I have to get the internal deployment completed today.
2. I’m planning to move the database to an official cluster anyway, per the directions in the Admin guide for moving the backend database for an Enterprise pool.

Primarily because of reason two, I don’t feel bad about installing SQL locally for a short time period (<1 month) until our cluster is ready to support the Enterprise pool.  As with other cautions I’ve offered, this isn’t recommended.  For me, it’s just real life.  To achieve the goal I want, I’ve created a CNAME (alias) in DNS to tell my computer that dbcluster1 is currently the same as ocsfe1.  I’ve also installed SQL Server 2005 Standard Edition SP2 32-bit locally.

8:39 AM : Creating the Enterprise Pool

Two notes here:

1. We specified a different internal web farm FQDN because we may eventually move to an expanded configuration, and having a different FQDN may facilitate that transition.
2. The planning documentation states that if you don’t specify an external web farm FQDN at this point, you’ll need to use the command line utility later.  Usefulness of command line utilities notwithstanding, I’d rather specify it now since I know what it is.

Another note: our database files will be going onto a SAN with the transition to the database cluster.  If you aren’t storing your database files on a SAN, you’ll want to make sure the database and log files are on different spindles (different physical volumes).  This is basic database optimization, not an OCS thing.

I didn’t enable meeting archiving yet as it probably requires the Archiving and CDR role, which doesn’t exist yet in my infrastructure.  I’m quite certain you can enable this later, so I’ll skip it for now.  I have put the path in, however, so that you can see what I would be using if I were to enable it right now.

Archiving is not enabled for the same reason listed above.

Ugh.  I made a mistake early on in the wizard – my pool is named ocspool.extendhealth.com, not pool.extendhealth.com.  I think I can probably fix this later, so I’ll keep going for now.  There were no other warnings in the log.

8:59 AM : Configuring Enterprise Pool

There’s the wrong pool name I mentioned above.

Note: Only one pool or server can authenticate automatic logon requests.

I’ll definitely be configuring external user access, but two things are stopping me from doing it right now:

1. I want the edge deployment to be distinct from the pool deployment for my own sanity and anyone’s sanity following along with this thread.
2. I think the only way you can configure your edge topology right now is if you’re migrating from LCS 2005 R2? and already have an edge topology deployed.  I’m not certain on that, I just think that’s what I recall.

9:10 AM : Adding Ocsfe1 to Pool

So far so good this morning – everything seems to be turning out okay aside from my dumb mistake with the pool name and the issues with the pool backend.  I’m now ready to add ocsfe1 to the pool as the first front-end server.

(Takes a while.  Lots of time for screen captures.)

Apparently Microsoft thinks it’s funny to continually remind me of my mistakes.

Yes, the password really is that long.  As a reminder (I think for the third time), I use WinGuides Password Generator to generate passwords for service accounts.

Same warnings as before:

Aside from that error being in the logs about 20 times, there were no other errors.  I think I’m still okay.

9:30 AM : Fixing the Pool FQDN

Before I proceed any further, I want to correct the pool FQDN.  I’ve been warned sufficiently.  As part of installing the Front End role, the administrative tools for OCS were installed.  I’m opening them from Start > All Programs > Administrative Tools > Office Communications Server 2007.

9:36 AM : ???

Wow … http://forums.microsoft.com/unifiedcommunications/ShowPost.aspx?PostID=2931495&SiteID=57

Apparently I’ll be removing the pool and creating it all over again.  Hope that goes okay.

Lesson learned: get the pool name right in the first place.

9:44 AM : Configuring Certificates

Well, at least it didn’t take too long to get back on track.  For this next step, please note that there are two distinct steps.  The Web Components role requires its certificate to be manually configured in IIS.  The rest of the Front End roles have a wizard.  I’ll deal with the wizard first, then IIS.

Because I have a PKI deployed, I can opt to send the request to an online certification authority (Active Directory will help me locate one).

In this case, we don’t care if the cert is exportable, but I left the box checked anyway.  We also don’t care about client EKU – the only place that matters is for the certificate assigned to the external interface for the Access Edge role.

I chose to include the local machine name in the SAN here.  If you’re configuring automatic client logon, the SAN must also contain sip.<domain>.  In my case, it was automatically populated because of the choices I made in earlier wizards to enable automatic client logon.

… and … I accidentally clicked through the next screen, so I think it succeeded but I’m not 100% certain.

Well, I got that far before realizing that the prior wizard had actually failed.  It has something to do with Server 2003 not recognizing the authenticity of the certificate chain.  My PKI is completely implemented with Server 2008, so I guess it’s time to go research what to do.

3:22 PM : Square 1

As if there weren’t enough blocks already…

I just got off the phone with Microsoft support.  The certificate issue is “by design”.  In this case, I interpret “by design” to mean, “We knew about the problem but haven’t taken the initiative to fix it.”  The specific issue is that Server 2003 and Windows XP don’t support certificate chains with algorithms > SHA1.  Since my root CA had a SHA512 thumbprint, and my other CAs had a SHA256 thumbprint (per NIST guidelines), Server 2003 barfed.

Generally speaking I’m very happy with Microsoft.  Today, I’m not.  Off to rebuild the PKI from scratch…

I spent the entire day yesterday dealing with administrative and management issues.  As such, there was nothing to report.

5:35 AM : Amber Alert (Ex post facto)

This morning, I arrived at our data center to finish up some final issues remaining from the previous day.  Installing all of this new equipment has caused heartburn, to say the least.  The IP KVM we have (by Avocent) is not particularly incredible and has been on the fritz since Sunday, meaning that I couldn’t remote control any computers to install them from the office.  That said, the plan this morning was to bypass the IP KVM, install a couple of servers with Windows Server 2003, and head back to the office to actually start on the OCS deployment steps past planning complete.  Upon arrival, however, I immediately noticed that I didn’t get an IP address from our DHCP server there.  The second thing I noticed was that all of our slave switches in the enclosures appeared dead.  The third thing I noticed is that the consoles on the front of the blade enclosures were amber.  In case you’re not a network admin (which I’m not any more, but experience has taught me), amber = bad.

It turned out that overnight, our data center had a significant A/C failure and had caused lots of problems.  This isn’t a small data center, it’s enterprise class.  A failure like this hasn’t happened in the entire history of the facility.  Of course it would have to happen while I’m trying to deploy OCS: administrator’s law.

12:00 PM : Amber Remediated (Ex post facto)

By noon, we had the issues straightened out at the data center.  I should note here that Dell wasn’t particularly well trained on our equipment, which is brand new (in the sense of recently released to manufacturing).  It turned out that our Cisco switches had overheated and shut themselves down as a protective measure.  Reseating the switches finally resolved most of our problems there.  On the plus side, the work with fixing the amber alerts also somehow fixed the IP KVM.

Back at the office, I was finally able to deploy Windows Server 2008 (for an Exchange deployment) and Windows Server 2003 to servers.  The current deployment toolset is using Microsoft Deployment as I was never able to get Configuration Manager 2007 running properly.

2:28 PM : Windows Server 2003 R2 with SP2 Deployment Complete

After working through several minor driver issues, I was just able to finish deploying Windows Server 2003 R2 (with SP2) via Microsoft Deployment.  There were actually two different Broadcom drivers necessary, and I had to be sneaky about where I put one of them.  If you happen to run into issues with a similar situation and need help, you can submit a comment here, but I don’t feel the need to detail what I did – it’s time to get into OCS, finally!

2:40 PM : Planning Recap

Since there were some final adjustments to several IPs internally, I’ll repost the planning table I posted last week with the updated IPs.  If you can’t see it all, just copy and paste it into Excel.

Edit: Removed planning table

2:50 PM : Created A Records

I just created the A records for ocspool, ocsmeetings, and ocsmeetingsext.  Note that certain parts of the planning documentation are pretty picky about whether these are A or CNAME records.  I was also under the impression that I needed to create a sip.extendhealth.com A record, but can’t find mention of it in the planning docs for now, so I’ll skip it until it becomes a problem.

2:54 PM : Crashed MMC 3.0

It might be just me, but the MMC 3.0 seems particularly unstable.  I just tried to add the SRV record for automatic configuration (_sipinternaltls._tcp.extendhealth.com) and the MMC crashed.

2:57 PM : Created SRV Record for Client Automatic Configuration

Note: this record gets created in the Forward Lookup Zones/<domain>/_tcp node.

2:59 PM : Finishing Updates

The ocsfe1 server will be the first server to come up (be added to the pool).  It’s currently finishing some updates, which is why I’ve been picking away at DNS requirements.  I should also note (if you didn’t read the posts from last week) that I have a PKI infrastructure in place to deal with the certificate requirements.

The one other critical thing I should highlight while I wait is that we expect some load balancers within two weeks.  The VIPs referenced above would normally be assigned to the load balancer.  For now, since we’re still missing this hardware, I plan to proceed with deployment as if they already existed.  In order to (hopefully) fool OCS, I plan to assign the IP address that will be assigned to the VIP to ocsfe1 (temporarily).  That means that ocsfe1 will currently have the following three IPs: 10.10.3.1, 10.10.3.51, 10.10.3.53.  Please note that this is almost certainly not the recommended course of action, and I’m only ignoring my own advice out of necessity.  When the load balancer comes in, I’ll assign the VIP IP to it, remove it from the server, and rerun the validation wizard and the best practices analyzer.

3:08 PM : Creating File Shares

Another thing you need to do before deploying OCS is set up some file shares that will store (mostly) Live Meeting related files.  I have set up four shared folders on my file server: OCS\AddressBook, OCS\MeetingArchive*, OCS\MeetingContent, and OCS\MeetingMetadata.

* Optional, will only need this if archiving and CDR archives meetings.

3:20 PM : Installed IIS

Since I will be deploying an OCS Enterprise Pool, Consolidated Configuration, I installed IIS from the Add Role wizard.  I didn’t enable ASP.NET as I don’t think OCS uses ASP.NET.  (The planning documentation says you need ASP, however.)

3:30 PM : Opening the Setup Wizard

I think I’ve completed all the prerequisite steps for OCS installation and am opening the setup wizard for the first time.  I’ll try to take as many screenshots as are relevant through the installation process.

3:32 PM : Preparing Active Directory

(Snipped for some semblance of brevity.)

(This wizard happened too fast to even grab a screen cap of the process.)

3:45 PM : Active Directory Prepared

Everything went flawlessly (or at least apparently so) in the Active Directory preparation phase.  I’m now ready to create the Enterprise Pool.  The one thing I think I might need here is user accounts that I haven’t created yet.  I create my passwords from the WinGuides Password Generator for security’s sake.

3:47 PM : Creating Enterprise Pool

As with above, relevant screenshots.

Curses!  The first error.  I just forgot to install the SQL client tools.

4:14 PM : SQL Client Install

4:30 PM : EOD

Unfortunately, that’s where it’s going to have to sit for tonight.  Hopefully will be able to finish off the pool by mid-morning tomorrow, barring the type of disasters that happened today.

8:08 AM : Sufficient Information

I arrived at about 6:30 AM and began gathering the data I would need to facilitate deployment of OCS.  I have put together a spreadsheet that has most of the information I’ll need in it.  Several IP addresses are missing from the edge servers (not that I would want to post that on a public Web site anyway) and I haven’t looked into certificate requirements for the Enterprise Voice servers.  That said, I have enough to start creating entries in DNS for client autoconfiguration and I have enough information to install my first front-end server.

I should note that after I got my family situated last night, I did some more looking into Configuration Manager’s deployment things and I found some other resources that may or may not come in handy.  I’ll list them here for future reference or for others’ perusal.

1. The guy who did the whirlwind tour of configuring Configuration Manager: http://blogs.technet.com/deploymentguys/default.aspx
2. Also related to them: http://www.deploymentforum.com/
3. Microsoft Deployment blog: http://blogs.technet.com/msdeployment/
4. Desktop Deployment tech center: http://technet.microsoft.com/en-us/desktopdeployment/default.aspx

Most of those links came from a Web cast from a couple of days back which I watched last night: http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032373731&CountryCode=US

The bad news is that I at least have to get Microsoft Deployment running in order to deploy some bare-metal servers.  The good news is that I know how to work with Microsoft Deployment.  It’s Configuration Manager that’s giving me grief.

9:21 AM : Review Complete

Just finished reviewing IP addresses with my boss and have completed filling out my spreadsheet.  I would recommend filling out a similar spreadsheet if you are working on deploying OCS.  A couple of notes: first, I left our public IPs off the spreadsheet.  Second, I still don’t have the certificate details for mediation server or speech server completed.  I’ll work on those in more detail when I’m deploying enterprise voice.  Here’s the list:

Edit: Removed planning sheet

12:34 PM : Configuring Microsoft Deployment

KMS is now running on the new domain, which facilitates deployment by allowing volume license operating systems to activate against a local server rather than MAK, which authenticates against Microsoft’s servers.  I’m also picking away at producing requirements for my team(s) so that they stay busy and getting the vanilla Microsoft Deployment solution accelerator running.  Until I have at least a bit done on OCS, I can’t dedicate any more time to Configuration Manager.  Microsoft Deployment will at least allow me to push operating systems without actually physically touching the box, since we have an IP KVM.  I’m also very used to setting up these types of deployment (I used BDD 2007 quite a bit).

12:53 PM : Added Windows Server 2003 R2 (32-bit) and Windows Server 2008 (64-bit)

I have two operating systems set up now in the Deployment Workbench.  I also configured Windows Deployment Services with pretty much the default values (but set it to respond to all client requests, known and unknown).  I imported the same Broadcom drivers I originally got from their RIS download, and I’m ready to set up a lab deployment point and create boot images.

3:15 PM : F Lock

That’s really not intended to be a derivative of a curse word, although I almost wish it were:  I just spent the last hour of my life feeling even more frustrated because I knew I had Microsoft Deployment configured properly, but I couldn’t get PXE to actually pull down the boot image.  It turned out that my F Lock key was on.  (Some Microsoft keyboards have an F Lock key that open up some keyboard shortcuts.)  The F12 command was actually being sent as Print.  At least I didn’t print 1200 copies of a boot screen.

4:42 PM : WinPE2.1 & Broadcom

Apparently WinPE2 changed the way it enumerates hardware and accordingly has trouble installing/recognizing Broadcom network devices, at least in a 64-bit environment.  Thankfully Jeff Huston has a solution that I’m trying right now.  At least it didn’t have to do with my F Lock key.

5:47 PM : No Luck

Still no luck.  It seems, however, that this was probably the problem with Configuration Manager in the first place.  I’m quite certain I had the right drivers imported, but this post gives me hope that it’s just a network driver issue.  Maybe if I can find the right driver, I’ll be able to get Configuration Manager running.  For now, I need to run and help a friend drywall.

7:43 AM : Firehose

Wow.  I just watched a screencast with the most information I’ve ever seen packed into 32 minutes and 14 seconds.  The screencast was actually tremendously informative and would be beneficial for anyone who is working on deploying operating systems.  Here’s the link: http://edge.technet.com/Media/System-Center-Configuration-Manager-2007-and-Microsoft-Deployment-Toolkit-Screencast/

I also found another interesting link last night that I forgot to mention.  I didn’t use it, but it might be useful to someone else out there: http://myitforum.com/cs2/blogs/cstauffer/archive/2008/02/13/notes-on-getting-pxefilter-vbs-working.aspx

I’m going to try to apply some of what I saw in the screencast at this point.

7:52 AM : Fixed up Windows Deployment Services

I removed the boot images I’d added to WDS and set the PXE Response Policy to not respond.  Apparently this is important so that Configuration Manager will respond instead.

7:55 AM : Configuring Client Agent Properties

I am now working on configuring the client agent properties appropriately to reflect customization in titles and subtitles.  I entered a network access account on the General tab and customized the text on the Customization tab.  For reference, I used:

8:15 AM : Configuring SAN

Trying to get some SAN volumes straightened out so our deployment files go onto the SAN rather than the local drive.

8:30 AM : Deleting All Packages, Advertisements, Etc

I’m taking Configuration Manager back down to square one as far as what I did last night (creating an OS, task sequence, etc).  That means that the steps I record next would be as if Configuration Manager was brand new.  The one thing I’m not going to do is recreate the collection I created last night, called PXE Registered Systems.  When a new machine attempts to PXE boot, WDS will reply to that PXE boot and (through the integration via the PXE filter) redirect that PXE boot to Configuration Manager.  However, Configuration Manager can’t do anything with the machine until it’s registered in the system.  The PXE filter helps to register the machine with Configuration Manager so that Configuration Manager can then take over and complete its work.

10:09 AM : Adding Boot Images

I just added both of the default (WinPE2) boot images back into Configuration Manager and am assigning them to both distribution points (our primary distribution point plus the distribution point created by adding the PXE site role).

10:18 AM : Adding Windows Server 2008 64-bit

I set up a nice folder structure for Windows Server and added the operating system image to Configuration Manager.  I also assigned the operating system to just the primary distribution point (not the PXE distribution point).

10:28 AM : Added Configuration Manager Client Software Package

Added a software package (from definition) for Configuration Manager Client Upgrade.  The “upgrade” should also work for the base installation per the screencast referenced above.  To add the software package, I chose Add Software Package From Definition, then chose Configuration Manager Client Upgrade (I later renamed this to Configuration Manager Client Installation).  I copied the source files from the Configuration Manager 2007 installation CD into a shared folder structure and pointed the source to that folder.  I then renamed the software package and published it to the primary distribution point only.

10:47 AM : Added Custom Backgrounds to Boot Images

I added a couple of nice background images to each of the boot images for aesthetic value.

10:51 AM : Deleting Operating System Image

Apparently I should have been doing this with the screencast – it turns out I was supposed to copy the entire folder for operating system source.  I’m wiping out the operating system image, copying the whole source DVD, and recreating the operating system image to the same specs.

10:54 AM : Creating Operating System Install Package

I’m now creating an operating system install package, which is where I need the full source of the operating system.  I also had to deploy the install package to the primary distribution point.

11:06 AM : Adding Drivers

I need to add the Broadcom NetXtreme II drivers since I frequently use these for deploying.  I commented on this last night, so I won’t belabor the point further.  I simultaneously created a driver package and added the drivers to the boot images, and then pushed new versions of the boot images to the distribution points.

5:29 PM : Heading Home

So I dealt with trying to get PXE running successfully for the rest of the day and am ready to give up for a while.  I can’t forestall the actual Office Communications Server deployment any longer.  I am pretty frustrated.  OCS tomorrow, just so I have a break from this.

12:35 PM : Reinstall Complete

Yes, you read that right.  In addition to everything else I had to deal with this morning (meetings, requirements delivery for some members of my team), I did a full reinstall of Configuration Manager 2007, this time on Windows Server 2003 R2.  I had some nagging errors in the logs that just wouldn’t clear up so I attributed it to Windows Server 2008 and started over.  The good news is that I was able to get the entire Site Status tree to show up green this time, meaning there are no problems.  The weird part was what I had to do to resolve the errors.  I think (am not sure) the resolution was to go look at and delete the error messages (which ended two hours previously) and refresh the component.  I don’t like that as a resolution, but on another component I went to do the same thing and clearing the error messages and refreshing the component didn’t fix the problem: I actually had to go resolve a problem.  The funny part is that it was back to SPNs again.  During the reinstall, I just installed SQL Server 2005 SP2 to the same box to ease my pain, and I didn’t update the SPNs.

Whatever happened, it’s entirely green now.  I’m hoping to get through a few more Webcasts on operating system deployment this afternoon, but I have 2.5 hours worth of meetings and only 2.5 hours left to go in the day.  Might be another late night.

6:58 PM : Catch Up

Just finished grilling.  For anyone who cares, it’s not a good idea to grill a still-partially-frozen New York Strip steak.  It takes forever, burns the outside 1/4 inch, and goes from rare to well-done in about two minutes, which I missed.  I like my steak medium-rare.  Not a pleasant meal.

Today was pretty chaotic overall, which doesn’t lend itself well to blogging my way through the deployment experience.  It largely failed today because I only had a few minutes here and there to work on deployment between meetings.  So, to catch you up to where I am now, let’s run through what I did in those few minute sessions.  I mostly toyed with operating system deployment.  I watched all of Keith Comb’s’ Webcast entitled “Deploying Operating Systems with System Center Configuration Manager (Part 1 of 2)”.  It was absolutely fantastic as far as detail is concerned, and should be watched even before deploying Configuration Manager – there are a couple of key things to note that he says (permissions on Active Directory’s System node and schema extension).

I also used his Webcast to start setting up the structure for deploying operating systems.  One of the first things I did was to add the Broadcom NetXtreme II drivers to Configuration Manager – they are used by some of our desktops and servers, and WinPE2 doesn’t have the driver embedded, meaning that attempts to do a light-touch or zero-touch install on those computers fails.  The driver was fairly easy to import and I won’t belabor the point by stepping through what I did.  I will note, however, that you have to download the RIS (remote installation services, the name for Microsoft’s deployment solution from years ago) version of the drivers or you’ll just bang your head against the wall trying to figure out how to extract the drivers.  Once the drivers are added, I made sure to add them to the appropriate boot images and update the distribution point.  I’ll also note at this point that I love the ability to easily embed the drivers here.  In the Microsoft Deployment Toolkit I know how to get around, but this approach was much more intuitive.

After adding the NetXtreme drivers, I pulled the file called “install.wim” off of the Windows Server 2008 x64 Datacenter, Enterprise, and Standard volume licensing DVD.  A WIM file is a compressed image; Microsoft started using WIM files for deployment with Windows Vista.  A WIM file can also store multiple operating system installs.  In this case, the install.wim was over 2GB but contained images for both the full and core installs of all three predominant flavors of Windows Server 2008 x64.  (Web Server 2008 ships on a different DVD.)  When I first added the WIM to Configuration Manager, I added it to the node called Operating System Images.  I also created a nice folder structure to make separation of images easier to understand.  When I added the WIM, I couldn’t tell whether or not I was going to be able to access all of the images in the WIM.  I’ll save you the heart-stopping anxiety by informing you that you’ll be able to access the images once you get to the task sequence portion of deploying operating systems.

That was actually the next thing I tackled: creating a simple task sequence to deploy the operating system.  Let me stress that it was extremely simple: I entered the options to join the machine to the domain and selected the version of Windows Server I wanted to install from the WIM, and that’s about it.  My thought at that point was, “I can’t believe how easy this is!”

7:14 PM : Not So Fast

The last thing I did before I left work (this is still technically part of the Catch Up, but warranted a logical division in the flow of the post) was to attempt to install the operating system.  We have an IP KVM and 12 blade servers sitting bare metal in our data center, so I used the IP KVM to try a PXE boot off of one of those servers.  No dice.  The first thing I found out was that I needed to install the PXE service point role.  After muddling through the interface to find where to add the role (root/Site Database/Site Management/<site>/Site Settings/Site Systems/<server>), I was able to add the role and set the options pretty much to their defaults.  I did set a PXE password which I removed later when I found out that it would cripple zero-touch installs.  I tried the PXE boot.  No dice.

After a bit more reading, I found out that I also needed the Microsoft Deployment Toolkit 2008, so I downloaded and installed that.  The article I was reading (http://technet.microsoft.com/en-us/library/bb978399.aspx) also said that I needed to configure the WDS PXE filter, which I tried to do from the Start menu, but there is some error with the tool.  I’ve tried a couple more PXE boots after altering a few settings, but still: no dice.

7:46 PM : Validation Exists for a Reason!

Twenty minutes of the last half hour was spent responding to an e-mail, the other ten spent trying to figure out the problem with the PXE filter.  I should have thought of this sooner, but mousing over the red exclamation point revealed the validation error: no Windows Deployment Services.  I had installed the Microsoft Deployment Toolkit, which is also necessary, but hadn’t installed WDS (something the Microsoft Deployment Toolkit relies on).  It is now installing; I had to add it through the Add/Remove Windows Components wizard because it wasn’t in the Add Role wizard.

7:54 PM : Reboot

Rebooted the server after install and am waiting…

8:06 PM : PXEFilter.vbs

Configured PXEFilter.vbs with the following settings:

sProviderServer = “MGR1″
sSiteCode = “DC1″
sNamespace = “root\sms\site_” & sSiteCode
sUsername = “”
sPassword = “”
sCollection = “DC10000D” ‘ Corresponds to PXE Registered Systems Collection (a custom collection I created)

8:11 PM : No Dice

8:28 PM : Break Time

I think part of my problem is related to Windows Deployment Services, but haven’t triangulated what the exact problem is yet.  Have to take a break to get my son to bed.

9:43 PM : Back Again

10:07 PM : Breakthrough

Some success!  I have a successful PXE boot.  I made a couple of changes to get the PXE boot: I told WDS that it shouldn’t authorize itself with DCHP, recycled the service, and reauthorized it and recycled the service again.  I don’t think that had anything to do with it, though.  I think the change that had an effect was explicitly advertising the task sequence for installing Windows Server 2008 to the PXE Registered Systems collection, where the PXEFilter.vbs told Configuration Manager to create a machine account.

10:08 PM : Configuration Manger Logo

I see a big background that says “Microsoft System Center Configuration Manager 2007″.  I’m guessing that’s a good sign.

10:12 PM : Success in IP KVM

Same thing via the IP KVM with a bare-metal server.

10:18 PM : Done for the Night

So after reboot, both servers I was working with (one virtual server, one real bare-metal server) still don’t have an operating system.  I’m guessing that’s a problem with the task sequence, but since I didn’t put much effort at all into the task sequence, I’m not to worried about it.  I think it will come up easily from here in the morning.  I’m quitting for the evening.

Edit (21 May 2008):

Apparently encryption algorithms > SHA 1 will prevent any Server 2003 or less, or Windows XP or less, machine from obtaining a certificate.  I implemented my root CA with a SHA512 hash algorithm and my subordinate CAs with a SHA256 hash algorithm.  I now get to redeploy the entire PKI.

6:41 AM : Back to Work

I actually got here over an hour ago, but have been catching up on my e-mail and such.  It’s almost 7am, and I’m ready to tackle Configuration Manager again.  There will probably be fewer updates today as I think I need to spend some time watching a few Webcasts this morning.  I checked briefly into the error with the Management Point and it seems like I recall something I needed to do in Active Directory with permissions.  I’ll try to find that and fix that problem first.

7:31 AM : Coffee Break

Taking a breather from the Webcast I’m watching (on System Center Configuration Manager 2007 SP1 and R2 upcoming releases) to grab a small cup of coffee.

7:37 AM : PKI

Another tangent, but I received the number I was waiting for to deploy our PKI.  I’m going to deviate from Configuration Manager long enough to get the PKI going, and then when I get around to it I can switch Configuration Manager over to Native Mode instead of Mixed Mode.  Again, I’m using Brian Komar’s Windows Server 2008 PKI and Certificate Security to make sure I follow updated best practices for deploying the PKI.

8:02 AM : Root CA capolicy.inf

I’m using the following configuration to initialize my enterprise root CA:

[Version]
Signature = “$Windows NT$”

[BasicConstraintsExtension]
PathLength = 3
Critical=true
[Certsrv_Server]
RenewalKeyLength = 4096
RenewalValidityPeriodUnits = 20
RenewalValidityPeriod = years
CRLPeriod = days
CRLPeriodUnits = 7
CRLDeltaPeriod = hours
CRLDeltaPeriodUnits = 4
DiscreteSignatureAlgorithm = 1

8:20 AM : Root CA Installed

I’m using the following script after installation to guarantee settings:

::Declare Configuration NC
certutil -setreg CA\DSConfigDN CN=Configuration,DC=extendhealth,DC=com

::Define CRL Publication Intervals
certutil -setreg CA\CRLPeriodUnits 52
certutil -setreg CA\CRLPeriod “Weeks”
certutil -setreg CA\CRLDeltaPeriodUnits 0
certutil -setreg CA\CRLDeltaPeriod “Days”
certutil -setreg CA\CRLOverlapPeriod “Weeks”
certutil -setreg CA\CRLOverlapUnits 2

::Apply the required CDP Extension URLs
certutil -setreg CA\CRLPublicationURLs “1:%windir%\system32\CertSrv\CertEnroll\%%3%%8%%9.crl\n10:ldap:///CN=%%7%%8,CN=%%2,CN=CDP,CN=Public Key Services,CN=Services,%%6%%10″

::Apply the required AIA Extension URLs
certutil -setreg CA\CACertPublicationURLs  “1:%windir%\system32\CertSrv\CertEnroll\%%1_%%3%%4.crt\n2:ldap:///CN=%%7,CN=AIA,CN=Public Key Services,CN=Services,%%6%%11″

::Enable all auditing events for the Extend Health Root CA
certutil -setreg CA\AuditFilter 127

::Set Validity Period for Issued Certificates
certutil -setreg CA\ValidityPeriodUnits 10
certutil -setreg CA\ValidityPeriod “Years”

:: Enable discrete signatures in subordinate CA certificates
Certutil -setreg CA\csp\DiscreteSignatureAlgorithm 1

::Restart Certificate Services
net stop certsvc & net start certsvc

certutil –crl

8:26 AM : Root CA Configuration Complete

Everything seems good on the root CA, moving on to the policy CA.

10:21 AM : Back on Task

I was distracted for a couple of hours talking to Microsoft and taking care of some tasks around the office, but am back on task.  I just imported the certificate revocation lists onto the policy CA.  I wasn’t able to make Brian’s command line (page 125) work, so I just right-clicked the certificate and allowed them to import the way they wanted to.  I’m a bit concerned since that adds them to the user account’s stores, but we’ll see if it causes a problem.

11:02 AM : Policy CA capolicy.inf

I’m using the following configuration to initialize the policy CA:

[Version]
Signature = “$Windows NT$”

[PolicyStatementExtension]
Policies = ExtendHealthCPS

[ExtendHealthCPS]
OID = 1.3.6.1.4.1.31088.1.1
Notice = “By enrolling a certificate from this certificate server, you agree to the posted legal notice.”
URL = “http://capolicies.extendhealth.com/defaultCps.aspx”

[Certsrv_Server]
RenewalKeyLength = 2048
RenewalValidityPeriodUnits = 10
RenewalValidityPeriod = years
CRLPeriod = days
CRLPeriodUnits = 7
CRLDeltaPeriod = hours
CRLDeltaPeriodUnits = 4
DiscreteSignatureAlgorithm = 1

I also just realized that I was supposed to save capolicy.inf to the %WINDIR% (usually C:\Windows) folder, not the system32 folder.  Maybe that’s why it didn’t work last time.

11:12 AM : Policy CA Installed

I’m using the following script after installation to guarantee settings:

::Declare Configuration NC
certutil -setreg CA\DSConfigDN CN=Configuration,DC=extendhealth,DC=com

::Define CRL Publication Intervals
certutil -setreg CA\CRLPeriodUnits 52
certutil -setreg CA\CRLPeriod “Weeks”
certutil -setreg CA\CRLDeltaPeriodUnits 0
certutil -setreg CA\CRLDeltaPeriod “Days”
certutil -setreg CA\CRLOverlapPeriod “Weeks”
certutil -setreg CA\CRLOverlapUnits 2

::Apply the required CDP Extension URLs
certutil -setreg CA\CRLPublicationURLs “1:%windir%\system32\CertSrv\CertEnroll\%%3%%8%%9.crl\n10:ldap:///CN=%%7%%8,CN=%%2,CN=CDP,CN=Public Key Services,CN=Services,%%6%%10″

::Apply the required AIA Extension URLs
certutil -setreg CA\CACertPublicationURLs  “1:%windir%\system32\CertSrv\CertEnroll\%%1_%%3%%4.crt\n2:ldap:///CN=%%7,CN=AIA,CN=Public Key Services,CN=Services,%%6%%11″

::Enable all auditing events for the Extend Health Root CA
certutil -setreg CA\AuditFilter 127

::Set Validity Period for Issued Certificates
certutil -setreg CA\ValidityPeriodUnits 5
certutil -setreg CA\ValidityPeriod “Years”

:: Enable discrete signatures in subordinate CA certificates
Certutil -setreg CA\csp\DiscreteSignatureAlgorithm 1

::Restart Certificate Services
net stop certsvc & net start certsvc

certutil –crl

11:32 AM : Publish to Active Directory Complete

I just finished publishing all CRLs and relevant certificates to Active Directory so that they are still available when I take the root and policy CAs offline.  I’m taking the root CA down and beginning installation of an issuing CA.

11:50 AM : Issuing CA capolicy.inf

I’m using the following configuration to initialize the policy CA:

[Version]
Signature = “$Windows NT$”

[Certsrv_Server]
RenewalKeyLength = 2048
RenewalValidityPeriodUnits = 5
RenewalValidityPeriod = years
CRLPeriod = days
CRLPeriodUnits = 3
CRLOverlapPeriod = hours
CRLOverlapPeriodUnits = 4
CRLDeltaPeriod = hours
CRLDeltaPeriodUnits = 12
DiscreteSignatureAlgorithm = 1

11:59 AM : Issuing CA Installed

I’m using the following script after installation to guarantee settings:

::Declare Configuration NC
certutil -setreg CA\DSConfigDN CN=Configuration,DC=extendhealth,DC=com

::Define CRL Publication Intervals
certutil -setreg CA\CRLPeriodUnits 3
certutil -setreg CA\CRLPeriod “Days”
certutil -setreg CA\CRLDeltaPeriodUnits 12
certutil -setreg CA\CRLDeltaPeriod “Hours”
certutil -setreg CA\CRLOverlapPeriod “Hours”
certutil -setreg CA\CRLOverlapUnits 4

::Apply the required CDP Extension URLs
certutil -setreg CA\CRLPublicationURLs “1:%windir%\system32\CertSrv\CertEnroll\%%3%%8%%9.crl\n10:ldap:///CN=%%7%%8,CN=%%2,CN=CDP,CN=Public Key Services,CN=Services,%%6%%10″

::Apply the required AIA Extension URLs
certutil -setreg CA\CACertPublicationURLs  “1:%windir%\system32\CertSrv\CertEnroll\%%1_%%3%%4.crt\n2:ldap:///CN=%%7,CN=AIA,CN=Public Key Services,CN=Services,%%6%%11″

::Enable all auditing events for the Extend Health Root CA
certutil -setreg CA\AuditFilter 127

::Set Validity Period for Issued Certificates
certutil -setreg CA\ValidityPeriodUnits 2
certutil -setreg CA\ValidityPeriod “Years”

:: Enable discrete signatures in subordinate CA certificates
Certutil -setreg CA\csp\DiscreteSignatureAlgorithm 1

::Restart Certificate Services
net stop certsvc & net start certsvc

certutil –crl

12:10 PM : PKI Complete

For all intents and purposes, I believe the PKI deployment to be complete.  Going to lunch and then back to Webcasts.

1:50 PM : Finished Webcasts

Just finished watching a Webcast on Configuration Manager 2007 SP1 and R2, and a two-part series on using Configuration Manager to deploy operating systems.

4:09 PM : End of Day

Watched several more Webcasts and tried to fix some errors in Configuration Manager’s status view.  No luck tonight.  Will start again tomorrow.

And so it begins.  As promised, I plan to chronicle in detail my journey through deploying Office Communications Server 2007.  The first several days will be filled with deploying supporting infrastructure.  We have made the decision to cut over from a separate internal domain name to a domain name that aligns with our external e-mail address domain and what will be our SIP domain (extendhealth.com).  All times are in MST.

11:50 AM : Current State

I should probably start by detailing my starting environment.  As I said above, we are cutting over to a new domain.  As such, we have a completely clean domain to work with in a brand new forest.  The new domain is called extendhealth.com.  I haven’t done anything to the domain outside of creating a user account for installation.  The user account is a domain, enterprise, and schema admin in anticipation of the OCS deployment (where I’ll need rights from all three groups).  In general, that’s not the recommended action.  Various tasks should be delegated to different personnel, and the permissions should be locked down much more tightly than they currently are in this domain.

The domain controller is running Windows Server 2008 Standard x64 (RTM).  It is up-to-date with all patches.  The domain controller is also virtualizing four virtual machines (VM) at this point, three of which will go offline shortly.  Two of the machines are an enterprise root certification authority (CA) and a policy server for the public key infrastructure (PKI).  Both of these machines will be taken offline and only be brought online when the issuing CAs need to have their certificates renewed.  [Side note: I knew quite a bit about setting up a PKI before starting this process, but am referencing Brian Komar's Windows Server 2008 PKI and Certificate Security for any questions I have.]  The PKI is currently pending a private OID request from the IANA.  This will allow us to use certificates in a manner that caters to external publishing of those certificates.  The certificates will not be used for public certificate chains – we have a wildcard certificate for that – but the official OID makes it easier to publish certificate policies and have them accepted by other parties.

The other machine that will go offline shortly is a temporary database server running SQL Server 2008 x64 February CTP.  When the next CTP comes out, we will install a database cluster and move our databases to the cluster.  In case I need to reference it, the name of this server is currently dbcluster1 (even though it’s not clustered).  The SQL Server install is complete and has all features installed.  I used a service account with a 64-character password generated by an online password generator.  Because these passwords are so random, we actually store them in a database that is particularly locked down.  Only one or two people in our entire organization have access to this database.

12:07 PM : Configuration Manager Intro

I’m currently looking at the pre-validation screen for System Center Configuration Manager 2007.  Configuration Manager is the fourth virtual machine (the only one that won’t go offline) on the domain controller.  I’ve allocated four processors and eight gigs of RAM for this server, which is currently named mgr1.  The Configuration Manager installation isn’t related to OCS; it’s more of a general infrastructure setup that I want to get out of the way.  We will be using Configuration Manager to deploy operating systems and updates.  I picked up a book on Configuration Manager the other day at the Microsoft Company Store.  (I was up in Redmond for the mid-sized market CIO summit.)  The plan is to set up very simple deployment of Configuration Manager before deploying OCS.  I have to deploy operating systems all along the way, so there’s no telling how much time it will actually take.  Configuration Manager should facilitate the deployment of those operating systems and their updates.  My naive estimate would be that it will take today and tomorrow to install and play with Configuration Manager, and then deployment of OCS will start Wednesday.  I feel very well prepared on my OCS deployment.  Configuration Manger scares me, however.  The deployment doesn’t seem especially streamlined, meaning I may make a significant mistake and have to redeploy.  We’ll see how things actually turn out.

12:13 PM : Configuration Manager Pre-validation Run 1

I haven’t done anything at all to this server (other than updates, joining to the domain, and enabling Remote Desktop).  It’s listing two warnings (I haven’t run the schema extensions and I don’t have the WSUS SDK) and several errors related to IIS, BITS, and WebDAV not being installed/running.  The only error that actually surprises me is the SQL Server sysadmin rights error.  Aside from that, I’m going to fix the other errors before looking into that one.

12:22 PM : Fixing Validation Errors

I installed a default installation of the IIS and Application Server Roles, and am downloading/installing Windows Software Update Services (WSUS) 3.0 SP1 from http://www.microsoft.com/downloads/details.aspx?FamilyId=F87B4C5E-4161-48AF-9FF8-A96993C688DF&displaylang=en.  I’m also downloading and installing the 64-bit version of WebDAV for IIS7.  Last but not least, I’m deciding whether or not I need to extend the Active Directory schema by reading this article.

12:34 PM : Schema Extensions and WSUS

I’ve decided to extend the schema per Microsoft’s recommendation and am adding some functionality to IIS per WSUS installation requirements in the ReadMe.  This is what I need to verify is enabled/installed in IIS:

I also noticed the BITS Server Extensions wasn’t enabled when I went into the Features part of Windows Server 2008, so I enabled them.  After installing those pieces, WSUS still alerts me that I don’t have the Microsoft Report Viewer 2005 Redistributable installed, but I don’t care about that until I need it.

12:55 PM : Installing and Updating WSUS

Still working on installing WSUS.  I had to provision a drive for updates on our SAN, which took a bit, and I’ve worked through all the other issues that I know of.  The installer is currently running.

1:26 PM : Lunch

Frustrated.  WSUS installed successfully and BITS and WebDAV certainly seem to be installed, but the Prerequisite Checker doesn’t seem to see them.  Rebooting and breaking for lunch.

2:02 PM : Back to Work

No change on reboot.

2:19 PM : Success!

Extended the Active Directory schema using ExtADSchema.exe in SMSSETUP/I386.  Installed a couple of additional IIS components (WMI compatibility, console) that cleared up the errors regarding BITS and WebDAV.  All systems are go at this point, but I’m a bit leery of what will be installed on dbcluster1 (my temporary SQL Server).  I had to turn off the firewall to get all checks to pass.  I’ll re-enable it after the install is complete, but having to turn it off to get the Prerequisite Checker to work doesn’t seem like a good sign.

2:22 PM : Configuration Manager Installation

Step by step:

1. Selected “Install a Configuration Manager site server”
2. Agreed to license terms
3. Selected “Custom settings” (largely because the book recommends it)
4. Selected “Primary site” since this is my first (and only) site
5. Agreed to Customer Experience Improvement Program – I want Microsoft to improve installation environment awareness
6. Product key was read-only
7. Left default path (C:\Program Files (x86)\Microsoft Configuration Manager)
8. Entered site code (DC1) and name
9. Chose “Configuration Manager Mixed Mode”*
10. Added NAP to selected client agents
11. Specified SQL Server (dbcluster1) and database (sccm2007_dc1)
12. Left default location (mgr1) for SMS provider – since the database will eventually be on a cluster, I can’t install the SMS provider there
13. Left defaults for management point (install a management point on mgr1)
14. Left defaults for port settings (HTTP/80 since I selected Mixed Mode)
15. Allowed checking for updated prerequisite components
16. Specified a download path for prerequisite components

2:33 PM : Settings Complete

After downloading a number of unnecessary prerequisites (multiple languages for Windows XP and Server 2003, neither of which are running), settings are complete and installation is ready to begin.  Installer, however, complains that the machine account for mgr1 does not have admin privileges on the SQL Server.

2:36 PM : Settings Complete, Take 2

Added computer account for mgr1 to the Administrators group on dbcluster1.  Prerequisite check has passed.  Install began at 2:37 PM.

2:40 PM : Fatal Error

Fatal errors during database initialization.  Not sure what that means since it created the database and tables.  Some tables are also populated.  (I looked at dbo.Agents.)  Great.  I have a message that says: Setup has detected an incomplete primary site installation on this computer.  You must uninstall the incomplete installation before continuing.  Here we go.

2:48 PM : Fatal Error, Take 2

Again with the fatal error.  Log (C:\ConfigMgrSetup.log) says: <05-12-2008 14:38:58> ***SqlError: [42000][650][Microsoft][ODBC SQL Server Driver][SQL Server]You can only specify the READPAST lock in the READ COMMITTED (if not based on row versioning) or REPEATABLE READ isolation levels. : sp_SetupSDMPackage

Googling it.

2:52 PM : Not Good

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=329707

Starting over with a SQL Server 2005 SP2 database.  Back in a couple of hours.

5:24 PM : A New Error

After installing SQL Server 2005 as the SQL2005 instance and trying to bind it to the standard SQL port (1433), I couldn’t get the Configuration Manager installer to see the instance, so I uninstalled both SQL 2008 and SQL 2005, and then reinstalled SQL 2005 and SP2 for the second time today.  That means the bulk of my time today has been spent installing and uninstalling SQL Server.  I’m now on to a new error: the error message says “Setup failed to install SMS Provider.”  Logs give me the following errors:

<05-12-2008 17:24:14> CompileMOFFile: Failed to compile MOF C:\Program Files (x86)\Microsoft Configuration Manager\bin\i386\smsRprt.mof, error -1
<05-12-2008 17:24:14> Setup cannot compile MOF file C:\Program Files (x86)\Microsoft Configuration Manager\bin\i386\smsRprt.mof.  Do you want to continue?
<05-12-2008 17:24:14> Setup failed to install SMS Provider.  For more information about this error, see Microsoft Knowledge Base at http://microsoft.com or contact Microsoft Technical Support for further assistance.

Other .mof files apparently compiles successfully before this one.  Back to Google.

6:28 PM : Finished?

I’ve finally made it through the wizard (it only took most of the day).  I have some pretty serious complaints.  The first would be that things like extending the schema should be part of the wizard.  The second was the problem I just spent an hour on: Kerberos issues.  I did eventually find my answer at http://myitforum.com/cs2/blogs/rcrumbaker/archive/2007/10/12/system-center-configuration-management-with-remote-sql-installations.aspx.  That happens to be the clearest explanation of a couple of really complex issues – SPNs and delegation.  We’ve had a ticket open with Microsoft for over 18 months regarding a particular Kerberos issue and have had many, many people unsuccessfully try to fix the issue.  Anyway, I had to set up two SPNs, one each for the NETBIOS and FQDNs of dbcluster1.  The commands I ran (from the domain controller) were:

1. setspn -A MSSQLSvc/dbcluster1.extendhealth.com:1433 extendhealth\sqlservice
2. setspn -A MSSQLSvc/dbcluster1:1433 extendhealth\sqlservice
3. setspn -l extendhealth\sqlservice

Two notes: first, the last command runs setspn in “list” mode, so that you don’t have to run adsiedit.msc.  Don’t get me wrong, I actually think adsiedit.msc is much better (and faster) at editing SPNs – but I thought I didn’t have it available, which brings me to my second note.  Setspn is available from the command line on Windows Server 2008 domain controllers (more accurately, computers with the AD DS role installed).  Adsiedit is also apparently available there, but doesn’t bind to your directory root by default.

It seems to me that the Prerequisite Checker should have caught the problem if the SPNs weren’t configured properly.  Whining aside, I did make it the rest of the way through the wizard and only one thing had a red X by it: the management point.  After reviewing the log, it seems that just the monitoring of the management point failed, and when I open the console everything seems to be functional.  I think I’ll leave it at this point (when I can be optimistic) and pick it up again tomorrow.

Just a quick post to let readers know that I’m headed underground for the next couple of weeks as I thoroughly read the available documentation for OCS.  I plan to journal in near-real-time my OCS deployment experience, which will likely start on approximately May 1.  Stay tuned.

I was writing a blog post the other day and used the word, “notwitholding.”  The spell checker immediately underlined the word as misspelled, so I checked what the suggestions were and the word I was looking for didn’t appear in the list.  A cursory check of dictionary.com and m-w.com didn’t turn up any results, but a Google search showed the word being used in multiple places, some of which seem to be pretty credible.  I don’t have an Oxford Unabridged handy, so the question hangs in my mind: is it, or is it not a word?

Language is a funny thing.  The English language, in particular, has a number of really strange artifacts.  Although we have a language largely descended from the same family as German and heavily influenced by Greek, we’ve lost the case endings that allow Greeks to move words around in the sentence to stress importance.  In English, the structure of the sentence largely dictates the case of the word.  Nominative case usually comes near the beginning of the sentence, dative frequently follows a preposition.  There are all sorts of complex rules that I remember having difficulty learning in grade school; I can’t imagine what it’s like for a foreigner.  Granted, we don’t have to learn any declensions, but I think we lose something at the same time.

Add to that our verb system.  In many languages (Romance languages, German, Greek, at least), verbs have different conjugations for different tenses.  In English, we largely use the same conjugation (or a very few conjugations) and fill in the gaps with auxiliaries (helping verbs).  Verbs are already one of the most difficult parts of language – most learners of a foreign language don’t struggle with vocabulary as much as they do proper verb conjugations.

The thing that really gets my goat (we’re leaving idioms out of this rant), though, are words like inflammable.  Dictionary.com has a good note on usage, but at first blush the average person aware of prefixes in the English language would think that inflammable and flammable are opposites.  They are, in fact, synonyms – both mean combustible.  Other prefix problems like inhabiting a habitation have always befuddled me.

I suppose I’m just getting a little bit of this off my chest, but there are things that worry me: as I was growing up, my mother was very meticulous about grammar.  I read quite a bit and became comparatively articulate.  However, I am finding it increasingly difficult to communicate.  In many instances, I present a statement in what I believe to be very clear terms, and the statement is misunderstood.  Is it my presentation, or their understanding, or both that are at fault?  Or does language have some part to play in that miscommunication?  I’ve always been thankful that English doesn’t have a board of people admitting and dismissing words from the official language, but sometimes I wonder where we’ll be in 10, 20, 100 years.

Back to Day 1
Back to Day 2 

Thursday

Birds of a Feather

The third and final day of Interact 2008 kicked off with a Birds of a Feather session rather than a keynote.  This morning, I sat at the Blogging table for a while before moving to the Voice Infrastructure table.  At the blogging table, I met Scott Schnoll and Nino Bilic.  Scott writes technical documentation for Exchange Server and struck me as a very friendly person.  Nino manages the Exchange Team blog in addition to his day job, which apparently has to do with supportability for Exchange.  The discussion started with several people, and one of the questions we kicked around was whether anyone’s job description actually included blogging.  Although no one sitting at the table at that time had blogging in their job description, a Mindsharp representative showed up shortly thereafter to fulfill my prediction that even if it wasn’t happening today, it would happen.  We also talked about where to find inspiration for blog posts and other forms of collaboration before I moved on to the Voice Infrastructure table.

At the Voice Infrastructure table, there were representatives of some larger firm talking about call center uses for OCS, which I obviously found interesting.  Although I missed their introduction, I gathered from the conversation that they were using it in a very different way than we were: they wanted a telephony solution that would serve as a telephony solution; we want a telephony solution that integrates deeply into our existing software to bring together telephony and computing such that it is difficult to distinguish between them.  Wajih Yahyaoui was managing the table and asked a number of questions about what we (customers) would like to see in the voice infrastructure.  The one other noteworthy event was that I ran into a Steven White, who came out of Cisco and offered to answer some of my questions about the Cisco AS5400XM.

Topologies and Routing for Microsoft Exchange Server 2007 (Todd Luttinen)

I don’t have much to say about this particular session.  I thought that it would be more about single-site topologies (meaning configuration of server roles) and the routing through those different roles.  Rather, this session was about enterprise-level, multi-site routing and its intricacies.  Aside from finding the discussion interesting, it wasn’t particularly relevant to me.

High Availability in Microsoft Exchange Server 2007 SP1 Part 1 – CCR vs other HA solutions (Ayla Kol; Alex Wetmore)

I had a great chance to speak with Alex before this session, and was pleasantly surprised to find that he was a very down-to-earth guy.  Alex was the lead developer for cluster continuous routing, one of the high availability features in Microsoft Exchange.  After listening to my description of our needs, he suggested that standby continuous replication would probably be sufficient for our situation if our service agreement could tolerate 40 minutes worth of downtime.  Of course, that all revolves around the Service Level Agreement that we should have in place (but don’t).  What I really appreciated about this session was the atmosphere in the room.  There were several Exchange Services team members in the room, and there was actually a fair amount of friendly heckling that went on.  Ayla saw a couple of them come in and immediately told them (in a joking manner) that they weren’t allowed to ask any questions.  The community atmosphere contributed significantly to the discussion, and I found that there were a number of people who were able to speak to various situations using real-life stories that made it to top-level Exchange support.

High Availability in Microsoft Exchange Server 2007 SP1 Part 2 – Disaster Recovery and SCR Deep Dive (Scott Schnoll)

The last session I went to, this was a great way to close out the conference.  The room had the same atmosphere (which isn’t surprising considering it had the same people) for this session.  Scott had a ton of material to work through, but he made it through efficiently and delivered all of the knowledge necessary to recover from a disaster in the event that one should occur.  As I recall (and this is almost a week ago now, so it’s not word-for-word, but it does have the concept), Scott said that losing a datacenter was a good thing.  There was one other notable quote that fired up the hecklers, but what I appreciated most about this situation was the anecdotal experience Microsoft had to share in testing their replication scenario between Singapore and Puget Sound.  Apparently many more things went wrong than they wanted to go wrong, but they were still able to recover within three hours.

Conclusion

When all is said and done, I’d go back to the comments I made when starting this summary: this has been one of the best-spent weeks of my life as far as careers are concerned.  I was able to network, dig deep, and find answers and vision for the solutions I need to design this year.  I sincerely hope to make it back to this conference next year, and highly recommend that anyone interested in Microsoft OCS do the same.

Back to Day 1

Wednesday

Keynote (Terry Myerson)

In my opinion, this keynote was better than Tuesday’s keynote, which is an odd thing to say considering that Gurdeep overall did a better job of presenting.  I think what I liked about this presentation were the number of times that I felt that Microsoft made themselves truly vulnerable.  No big company is perfect, and when you ship the amount of code that Microsoft ships, you probably ship significantly more bugs than the average company.  Terry was straightforward about a number of problems that have happened in the past year (they even showed the code that shipped that caused the leap year bug), but he also said that they were lessons learned for the Exchange team.  The overall feeling that I had walking out of the keynote was that the Exchange team treated me very much like an old friend: they were affable, and comfortable talking frankly about very sensitive matters.

Microsoft’s Quality of Experience: Defending, Deploying, and Succeeding (Neil Deason; Sam Chon)

Neil Deason has been involved in a number of the sessions I’ve attended, and I’ve been very impressed with him every time he speaks.  In contrast to my previous comment about an argument relating to ports on firewalls, Neil’s presentation today was very solid (the same argument re: ports notwitholding).  The emphasis of this session was not conveyed well by the title; the real thing we learned here was how to ensure that we have a consistent and good QoE.  There was at least one interesting discussion about why audio conferences use the Siren codec rather than the typical RTAudio codec.  The response was that computing power isn’t yet at the point where RTAudio streams can be decoded and encoded in scale amounts.  When computing power reaches that point, Microsoft will definitely look at using the RTAudio codec in favor of Siren.  Until then, those of us who use audio conferencing and have a discerning ear will notice a drop in call quality when shifting to using Siren.

Probably the thing that I found most interesting was something anecdotal.  There was a conversation about ambient noise on the voice conversation, and Mu Han related that they had received so much feedback from customers stating that the ambient noise level was so low that they couldn’t tell whether or not the person was still on the other end of the line.  His statement, and I’m not at all qualified to rate the degree to which he was being facetious, was that they were seriously thinking about reintroducing a higher level of ambient noise in future versions of OCS.  I’ve never heard of a product purposefully introducing more noise – that speaks pretty highly for call quality and their existing noise filtering mechanisms.

Planning and Deploying Voice Routes in Microsoft Office Communications Server 2007 using Enterprise Voice Route Helper (Byron Spurlock)

This was another excellent session with Byron, this time about using the Enterprise Voice Route Helper.  The Route Helper tool is designed to help configure dial plans and other facets of outbound routing.  My development background has involved a fairly significant amount of experience with regular expressions, which was what Byron emphasized several times as one of the most important features of this tool.  If you have regular expressions experience (even if it’s only a little experience), you might find other facets of the tool more helpful.  I especially liked the ability to set up and save test cases, which may help to validate that changes to the outbound routing achieve the expected result.  Another tidbit from this session is the best practice of setting up your phone usages to align with actual phone usages.  Microsoft’s recommended best practice is to create a local, national, and international usage at a minimum.  Routes, on the other hand, should align with their “break-out point”, the physical location where a media gateway connects to the PSTN.  There should be at least one route for each distinct break-out point.

Diagnose and Solve Voice Quality Issues with Microsoft OCS 2007 Quality of Experience Monitoring Server (Wajih Yahyaoui; Jisun Park)

I sat and had lunch with Ji Sun before this session.  We talked about what he had done before coming to Microsoft (he had just finished his PhD in Texas) and what he was doing now.  My impression overall, which was only reinforced during the session, is that Ji Sun is profoundly intelligent but very quiet.  This session was primarily a further exposition on Microsoft’s philosophy of QoE, and an introduction to one of the tools available to diagnose QoE: the Monitoring Server.  The QoE Monitoring Server shipped after OCS was released to manufacturing.  At the most fundamental level, the QoE Monitoring Server is a database which collects metrics sent in by OCS endpoints.  These metrics primarily revolve around MOS scores.  In most cases, MOS scores are subjective opinions (hence the name) of call quality.  Microsoft made an attempt at a more objective score, but maintained the legacy name “MOS”.  Furthermore, Microsoft breaks MOS down into distinct categories of sending, receiving, network, et cetera.  Some basic analytics are run on the metrics after they are collected, but for the most part the OCS administrator should expect to spend some time deciphering MOS scores, dropped packets, and jitter by looking at lots of numbers.  The top layer for the QoE Monitoring Server consists of a report pack for SQL Server Reporting Services that give a nicer visual indicator of overall QoE health.  As a personal anecdote, I’ll add that developing reports in SSRS is a headache, but once they are developed, the reports support e-mail based subscriptions so that a user may receive a report in his/her inbox every hour, day, etc.

Advanced Troubleshooting for Voice in Microsoft Office Communications Server 2007 (Byron Spurlock; Roy Kuntz)

This session was really a wrap-up of Byron’s other two sessions re: tools for managing and troubleshooting OCS problems.  We reviewed and went into more depth on Snooper, we looked more at the Enterprise Voice Route Helper, and we talked about the Office Communications Server 2007 VoIP Troubleshooting Guide.  The VoIP Troubleshooting Guide was authored in large part by Roy Kuntz and is an excellent document for identifying available tools for troubleshooting VoIP issues.

Birds of a Feather

Birds of a feather was designed to be a small roundtable discussion that ran from six to nine.  David and I spent most of our time at the QoE vs. QoS discussion.  Most of the people at the table noted that the session was pretty poorly named.  Microsoft as an entity does not see QoE as an alternative to QoS, but they do embrace the philosophy that you should only implement QoS if necessary.  In an ideal world, QoE and proactive monitoring will preempt the need for QoS.  The discussion at the table was largely driven by us – there were four or five Microsoft personnel, a Psytechnics representative, and an occasional conference attendee.  We were able to go through some of our concerns in detail and have a large amount of time for Microsoft to specifically address our concerns.

Overall, a great second day, but there was so much information and so little time.

I am now wrapping up the third and final day of the Interact 2008 (not to be confused with Interact 2008) conference.  I don’t know what I can say about it other than to say that is has been time extraordinarily well spent.  There are many things that you do in life that are worthwhile days; as far as careers go, this ranks among the most useful days I’ve ever had.  I don’t intend that statement to be either hyperbole or summarily discardable.  There has been absolutely fantastic face time with Microsoft employees and a wonderful opportunity to interact (no pun intended) with key vendors and other attendees.  Although I’m completely saturated and exhausted, I’ll try to give a rundown of the sessions and events.  First up, the sessions that I attended.

Tuesday

Keynote (Gurdeep Singh Pall)

Overall a great keynote, but very similar in content to the keynote delivered at VoiceCon 2008.  I can’t hold that too much against him since I’m sure that writing new keynote speeches for every event he speaks at probably isn’t and shouldn’t be his priority.  That said, I recommend you watch the actual keynote from VoiceCon rather than reading my poor summary of what Gurdeep said.

Panel on Planning Voice Architecture and Deployment in Microsoft Office Communications Server 2007 (Mahendra Sekaran; Sean Olson; John Kenerson; Francois Doremieux; Russell Bennett; Jens Trier Rasmussen; Ken Ewert)

I tremendously enjoyed this session; it was an open forum for anyone with a question to address some of the brightest minds on the OCS team.  The people that stood out to me in particular were Mahendra Sekaran, who answered my question about topologies with 100-person outsource shops (and whether we needed to deploy a pool/enterprise voice equipment at that location); Sean Olson, who answered most of the questions about general vision; Francois Doremieux, who handled many questions about actual deployments; and Russell Bennett, who contributed intelligent comments to several questions.  I particularly enjoyed the range of knowledge available in the room, it seemed that there were answers for every question asked.

Microsoft Office Communications Server 2007 Edge Drill Down (Wajih Yahyaoui)

This was a very good session on details for the various edge servers.  Wajih has a very noticeable accent, but was obviously passionate about the subject matter, so it was a real pleasure to listen to him.  He was able to handle most of the questions in the room, but it was helpful that several other knowledgeable Microsoft personnel were there also.  The conversation had one major interruption when a guy who I considered to be acting very belligerently.  The contention was based around whether OCS’s requirement to open port ranges in external firewalls unnecessarily creates security vulnerabilities.  Neil Deason responded that a firewall is only ultimately secure if all ports are closed: it makes no difference whether there are 10,000 ports or one port open, your network is vulnerable if you have ports open in your firewall.  While I don’t think that response was a good response overall, the point of the response should be considered sufficient.  The point of what Neil was saying is that firewalls are only one element of a properly hardened network’s defenses.  Hardening a network involves hardening multiple elements, not just the firewall.  The firewall-only approach is typically described like an M&M: crunchy on the outside, chewy on the inside.  If you have a network defended only by a firewall, your network is vulnerable to internal attacks.  Although research shows that most attacks originate from outside the network, the same research shows that an alarming percentage of breaches originate from inside the network.  See http://answers.google.com/answers/threadview?id=15439 for links to credible sources.

Advanced Validation and Troubleshooting for OCS 2007 (Byron Spurlock, Tom Laciano)

Byron and Tom handled this session on how to ascertain the source of an OCS 2007 problem.  Both presenters were enjoyably humorous, considering the amount of time we’d been sitting that day.  We got a chance to see Byron use a number of tools such as the Snooper tool, validation wizards and more.  As I sat there, I realized how much I would have benefited from knowing that those tools existed a few weeks ago, when I spent a significant amount of time using Wireshark to diagnose what was wrong with OCS.  Afterwards, I went to one of the Coffee Chats with Tom and sat for a while as he explained in detail what subject names and subject alternative names are necessary for certificates in various scenarios.

Evening Event: Surfing @ Wave House

In the evening, we relaxed on the beach at the Wave House.  It was a great time breathing the (very) cool salt air, throwing back a few drinks, and doing some surfing!  I stunk (figuratively), but here’s a shot, courtesy of my colleague David DeWinter who went with me:

In part 1 of this post, we examined the core pool roles for Microsoft Office Communications Server 2007.  Specifically, we covered front-end servers, directors, the three variants of conferencing servers, and the archiving and CDR server.  There are still several key roles to be covered to understand the full breadth of the OCS offering.  These roles fit into three key areas: edge servers, telephony servers, and “other”.  Before we get into the specifics of the roles, please take a brief moment to review the vocabulary from part 1:

• Office Communications Server: A Microsoft product designed to facilitate communications both inside and outside the office.
• Presence: A metric that takes into account both your availability (available, idle, away) and your willingness (available, busy, on a call) to communicate.
• Endpoint: Any device (SIP phone) or software package that registers itself with Office Communications Server as belonging to a user, meaning that the user can be contacted through the device or software package.
• Enterprise Voice: Probably the most noteworthy addition to the product since 2005; allows calls to enter and exit Office Communications Server.  This means that from any endpoint, users can make or receive calls to traditional phone numbers.
• Public Switched Telephone Network (PSTN): The traditional telephone network that delivers telephone service over dedicated copper cables.

Edge Servers

Access Edge Server

The access edge server provides three very key services: authenticating and enables connectivity for remote users, negotiating federated communications, and connecting to public IM services such as MSN, AOL, and Yahoo.  Authentication and communications with remote users is unequivocally the most common usage of access edge server.  This server is critical whenever an employee needs to use Communicator but is outside of the corporate LAN.  Traveling sales representatives with Communicator Mobile, home-based employees and other situations are supported when using Access Edge Server.  Federation is the term used to refer to two Active Directory domains that have set up a federated relationship.  Note that a federated relationship is not the same thing as a domain trust, but is similar.  Generally federation happens along corporate boundaries.  Two companies in a strategic alliance or other partnership will federate to allow key contacts greater visibility and easier access to communications.  Microsoft OCS can also allow connectivity with public IM services, enabling communications from Communicator to MSN Messenger, AIM, or Yahoo! Messenger.

Personal Side-note: Access edge server is one of the most amazing roles in my opinion.  I have been witness to quite literally taking an OCS endpoint, moving it outside of the network, and having it seamlessly connect back up to OCS without any additional configuration.  Imagine being able to grab your desk phone and go home for the day!  We currently have a Cisco UCCX system in place.  In order to take my phone home, I have to take a hardware VPN home, hook it up directly to my cable modem (in the basement) and then hook my phone straight to that.  With OCS, I was able to take my laptop home, turn on my wireless and connect immediately.  If I can say one thing that would be the most important thing for a Cisco customer to hear, it’s this:

Our Cisco system is technically capable of achieving everything we need it to achieve, but our experience with OCS has blown us away.  Actually getting your hands on to a sample OCS setup is the best thing that you can do for yourself.

To summarize, the access edge server:

• Authenticates and enables connectivity for remote users
• Allows two entities to federate, which in turn allows greater visibility for communications
• Allows connectivity to public IM networks

A/V Edge Server

The A/V edge server enables audio and/or video conferences to happen with users outside of the corporate LAN.  It is important to note that telephony conferences are considered distinct from this scenario and are covered by the telephony conferencing server (see part 1).  The A/V edge server allows remote users authenticated by access edge server to establish internal audio or video calls, or VoIP calls for enterprise telephony scenarios.

Web Conferencing Edge Server

Similar to the A/V edge server, the web conferencing edge server enables Live Meeting 2007 sessions to include users outside of the corporate LAN.  Many companies will use this role slightly differently than they will the other edge server roles.  Where access edge server and A/V edge server are deployed to allow external known users to connect and conference, Web conferencing edge server may arguably be used to conference in more anonymous users (who are still actually authenticated by digest authentication) than known users.  This allows companies an internally controlled, paid-for mechanism similar to WebEx that allows public sharing of desktops and other information.

Requirements** (for all edge servers):

• Dual processor, dual core 3.0GHz+ processor
• 2 x 18GB HDD
• 4GB+ RAM
• 2 x Gigabit NIC
• Windows Server 2003 SP1+*

* I was not able get the OCS primary installer to run successfully on Windows Server 2008 RTM.  It may be that the individual installers would run successfully, but I have not confirmed this.  The only role I have successfully installed on Windows Server 2008 is Speech Server 2007.

** The work of mixing audio channels is intense; A/V servers will benefit from more robust hardware.

Communicator Web Access

Communicator Web Access (CWA) is to Office Communications Server what Outlook Web Access is to Exchange Server 2007.  It provides an attractive, AJAX (slick update without refresh) based interface for internal or external users to use.  CWA functions much like the director role in that it proxies connections, but differs in that it also proxies internal connections.  Also, CWA is restricted to communicating via instant messaging.  There is no support for audio/video conferences, Live Meeting, or enterprise voice.

Requirements:

• Dual processor, 3.2GHz+ processor
• 1 x 36GB HDD
• 4GB+ RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

* I have not yet attempted to install this role on Windows Server 2008.

Web Components Server

This role has probably the least visible functionality of all server roles: it’s primary responsibilities are to allow users to join Web conferences by clicking a URL, allow download of Address Book data, and expand membership in distribution groups (in ways, simply an expansion of the Address Book functionality).

Requirements:

• Dual processor, dual core 2.6GHz+
• 2 x 18GB HDD
• 2GB+ RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

* I have not yet attempted to install this role on Windows Server 2008.

Mediation Server

Contrary to the Web components server, the mediation server role has profound visibility and is arguably as important as a front-end, back-end, or edge server.  The mediation server is what makes enterprise voice possible.  When Microsoft implemented enterprise voice, they elected to use proprietary codecs (RTAudio and RTVideo) in order to overcome some significant hurdles such as inconsistent bandwidth.  However, their choice to use these proprietary codecs meant that right from the beginning, Microsoft wasn’t able to play nicely with many pieces of PSTN hardware.  In their defense, the enterprise voice market is very confused right now.  There are many competing standards such as ICE, SIP, and others that still aren’t fully or consistently supported.  Microsoft saw this and decided that it would be easier to simply draw a strong line between external and internal voice traffic.  That line is drawn right through mediation server.

Microsoft states that there are three ways to connect Office Communications Server to the PSTN.  The first is through a basic media gateway.  A basic media gateway is simply a piece of hardware that terminates PSTN lines (whether in FXS/FXO or T/E/DS form).  The media gateway’s responsibility is to accept incoming calls on the PSTN lines and hold the line open until the call is complete.  To know when the call is completed, the basic media gateway talks to the mediation server, generally via G.711.  The mediation server does the job of decoding G.711 voice traffic and encoding into RTAudio (and vice versa, for outbound voice traffic).

A basic hybrid gateway does essentially the same thing except that it merges the mediation server role directly onto the media gateway.  The benefit of a basic hybrid gateway over a basic media gateway fundamentally boils down to TCO: it’s cheaper and easier to manage one box than it is to manage two.

The final means of connecting Office Communications Server to the PSTN is for the media gateway itself to directly support the native OCS protocols (like RTAudio and ICE).  Microsoft calls this an advanced media gateway.  Please note that the difference between the advanced media gateway and the basic hybrid gateway is that in the basic hybrid scenario there are two functions coexisting on one box – they are still distinguishable functions.  With advance media gateways, the functions are no longer distinguishable.  The media gateway natively speaks OCS’ language.

In the next post in this series, we’ll consider a final smattering of server roles that don’t always require a full server, consider coexistence scenarios and some final “gotchas” that I wish I’d known about when I started deploying OCS.

In part 1 of this article, I raised some issues with traditional ACD algorithms.  The issues raised are best summarized by generalizing* ACD algorithms as FIFO queues with the only variances being skill levels and the actual agent allocation algorithm.  Agent allocation algorithms generally break down into something fairly simple such as which agent has been off the phone the longest, taken the fewest calls, or spoken to the person before.  There is a lack of intelligence when it comes to considering a many-to-many call/agent match.

* I really do understand that there are likely algorithms out there that do achieve 90% of what it is that we need to achieve.  The question is not, “How much does solution x achieve for company y out of the box?”, the question is, “To what level does solution x allow all companies to customize the algorithm to achieve 100% of their needs?”

A Proposed Solution

Before I propose a solution, I should make two things clear.  First, I am by no means an expert in contact center theory.  There is a world of things I don’t know – contact center theory is one of them.  Second, I am not fully educated on the existing solutions out there.  Due to the proprietary nature of algorithms and the obvious interests of the companies in protecting them, the best I can do is find descriptions of how algorithms currently work.  I can also see the flaws in our current UCCX system.

So how do we determine which is the best agent to assign to a call?  Skills-based routing excels at limiting the pool of agents to available agents who are capable of taking the call.  We want to break past that barrier to achieve the following:

A desirable algorithm for contact centers will create an optimal call-agent match, adjusted for time considerations.

Note that the statement above does not de facto consider availability.  Availability certainly should be part of the equation, but only part.  For contact centers who have frequent repeat callers, agent consistency may be a desirable trait.  If agent consistency is desirable, the algorithm may state that if the caller’s assigned agent will be available in less than a minute, the caller will be placed on hold until the agent is available.

Base Match Score

Skills-based routing has achieved the first part of the equation: making sure that someone capable of handling the call is assigned to the call.  Because there are only the few dimensions involved (call disposition, available agents, and agent skills), call match may not be entirely optimal.  The first step to creating an optimal call-agent match is to increase the number of dimensions that factor into the final assignment.  As was stated in the introduction to this post, we have designed an algorithm that takes a number of factors into consideration.  A sample list of considerations may include the following factors:

** Note that unless an alternative receptor has been configured for calls with a zero match score for all agents, all multipliers should be greater than zero.

The above chart over-simplifies in one significant respect: it treats ranges as a flat match or non-match score.  In our actual algorithm, we support ranged multipliers, but this example needs to be easy to understand.  I also recommend this type of a chart for gathering requirements from the business; it’s easier to understand and easier to supply rows rapidly.  That said, this may be the match score matrix for a fictional company.  The first row indicates that, as with many companies, skill match is critical to agent assignment.  The non-match multiplier minimizes the chance of a agent without a skill match being assigned to this call.  Unless voice mail or an alternate “queue” is configured to handle calls with a zero match score, I highly recommend having all multipliers be greater than zero.  The second and third rows assign a significant priority to agents who have spoken with this person before.  The third to fifth rows flatten a ranged multiplier.  In true implementation, I recommend attempting to use an actual mini-algorithm to deal with these types of issues as the result is a more accurate, less “bucketed” match score.  In this case, we are attempting to de-prioritize any agent that is not currently available.  The longer it will take the agent to become available, the less likely it is that they will be assigned to the call.

It is critical to note at this point that we feel this is one of the key differentiators of this algorithm.  Most algorithms only consider the available agent pool and rescan every few seconds if a match cannot be found.  In this case, a match can be made even before an agent finishes a call if the reason (previous contact, for instance) for assigning the call to that agent is compelling enough.  The IVR could even theoretically ask the person if they wanted to speak to their assigned agent and alter the match/non-match multipliers for assigned agent based upon their answer.

The final row of the table assigns a value to variable cost.  If our company handles many of the calls in-house, those calls probably have a very low variable cost (most of the costs would be considered fixed or sunk costs).  If additional calls can be routed to an outsourcer for $20/call, it is important to know the value of routing that call.  In general, the algorithm should prefer routing to agents with the lowest variable cost.

Once the match score criteria have been determined, an example matrix can be set up.  In this case, we are attempting to assign four incoming calls to three agents.  We consider each criterion for the match score and evaluate a base match score for all agents for all calls.  [We will continue to build on this same matrix when we introduce the timeline modifier.]

We calculated the match score by multiplying one times each of the values for the match/non-match values as appropriate for each match factor.

Timeline Modifier

Now that we have a concept of a base match score, we need to introduce a timeline modifier.  The timeline modifier ensures that calls with poor match scores across the board eventually get picked up.  How compressed the timeline modifier is depends upon your business model.  If you wish to have a good match and have a high call volume, a longer timeline may make sense.  If you don’t care about match and just want to ensure that calls are picked up, a more compressed timeline may make sense.  You could even replicate a FIFO queue by using an extremely compressed timeline modifier.  We currently use this equation to calculate the timeline modifier: if the call is not yet ready to be transferred, we divide 1.25 by the number of minutes until transfer.  If the call is ready to be transferred, we add three to the number of minutes it has been ready for transfer.  If we treat negative numbers as calls that are ready to be transferred and positive numbers as calls that are not ready to be transferred, extending our example above yields the following timeline modifiers for our calls.

One of the most interesting things about the timeline modifier is that it really boils down to just another match factor, but we treat it differently for two reasons: first, it’s more critical that we have a smooth range rather than buckets when referring to the amount of time a call has been on hold.  Second, business people understand timelines to be distinct from other match criteria.

Modified Match Score

We then use the timeline modifier to calculate the modified match score.  Multiplying the base match score by the timeline modifier yields the modified match score, as shown:

Call Assignment

Finally, we assign calls by maximizing the sum of agent-call matches.  In this case, our sum is maximized at 55.8125 by assigning Call 4 to Agent 1, Call 2 to Agent 2, and Call 3 to Agent 3.

Implications and Conclusion

There are a number of implications to how we have assigned calls.  Note that calls 2 and 3, which are not ready to be transferred, are already assigned to an agent by the algorithm.  We distinguish between optimal match and actual assignment.  For actual assignment, we prevent calls more than two minutes from transfer from being assigned.  We could achieve the same thing by tweaking our timeline modifier equation to yield a different timeline modifier.  This brings up perhaps the most important differentiator of this algorithm, however.  Because we consider calls that aren’t yet ready to be transferred, we have some level of predictive ability that allows a better call-agent match.  If we go back to part 1 of this post, it is easy to see why predictive ability is important.  Rather than just looking at the here-and-now, we look at the soon-to-be and are able to reserve agents whose skills match with calls that will soon be ready to transfer.  The key to getting this information is to have the IVR send the ACD periodic notifications of calls en route, their current disposition and probable end state.  Finally, it is important that we consider not only the best agent for the call, but also the best call for the agent.  Looking at the call assignment grid above, you’ll note that we assigned Call 3 to Agent 3.  However, Call 3 had a higher match score with Agent 1.  The reason we assign Call 3 to Agent 3 is because Agent 1 has a better match with a different call, meaning that Agent 3 gets Call 3 by process of elimination.

The result of our experimentation with OCS has been this: last year, we struggled for months to hook into UCCX’s ACD in order to direct calls to the right destination based solely on one piece of information.  In our pilot with OCS, we were able to achieve multi-factor routing in a matter of days for a small fraction of the cost we incurred last year.  It is entirely accurate to say that Office Communications Server does not ship with an ACD.  The sleeper, however, is that they do ship a platform that is simple to hook into and allows development of a very complex and highly customized ACD that fits your business model.  For us, unless we find a blocking problem with OCS the choice is simple.

In future posts, we will start to lay out a high-level architectural diagram of how the various pieces work, where the messaging links are, and any gotchas that we find.

First off, I need to say that I sincerely hope that no one takes offense to the title of this post.  The title is meant to be a play on the phrase, “gone to the birds,” not an ethnocentric slur.  The alert reader will also pick up on the double entendre that in Britain, queue is a much more common word.

I have stated in earlier posts that I lead a team of top-notch developers.  My primary responsibility is early research and architecture; my team are the ones to transform the vision into reality.  We recently completed our pilot project, which I plan to blog about at some length.  One of the requirements that we met was the ability to design an advanced ACD that takes into account any number of variables and implements a “best-match” algorithm.  We believe this to be a significant advance over traditional algorithms of the skills-based routing.  In fact, a multiple-hour search yielded an awfully slim amount of relevant data.

The Problem with Skills-Based Routing

The primary problem with queues is implicit in the name: most queues operate as first-in-first-out mechanisms.  Before we can address why FIFO is a bad idea for contact centers, we need to have some context.

There is an even bigger problem that is a fundamental part of the queue problem.  This problem is in the existing call distributor algorithms.  A simple example will illustrate:

Assume we have three available agents.  Agent 1 is licensed to sell insurance in Michigan and Indiana.  Agent 2 is licensed to sell insurance in Indiana and Ohio.  Agent 3 is licensed to sell insurance in Indiana only.  Assume also that 80% of our inbound calls are from Michigan residents, 10% from Indiana residents, and 10% from Ohio residents.  If a call comes in from a resident of Indiana, any of the three may be able to sell an insurance policy to the caller.  However, only one is best suited in this context to sell the insurance policy to the caller: Agent 3.  There is a 90% chance that the next call will be from either Michigan or Ohio, meaning that Agent 3 would not be able to handle the call.  That means that we want to reserve those agents whose skills are most in demand for the calls that demand those skills.

Many contact and call center software providers trumpet their algorithms for call distribution, but most of the algorithms are some variant of, “How do I spread the load of calls evenly over my agent pool?”  The question that should be asked is, “How do I find the best possible agent to handle this call?”  Some software providers have started to deal with this question and have introduced a second factor into skills-based routing.  By assigning a rating to an agent’s skill, there is indeed additional intelligence available to the call distributor.  Consider the following example:

Assume we have three available agents, with skills rated as shown in the matrix.

Agent	Windows skill level	Office skill level	Internet skill level
Agent 1	70	40	10
Agent 2	10	70	40
Agent 3	40	10	70

 

Upon receipt of an inbound call, the skill matrix is analyzed and the agent with the highest rating in the skill necessary is assigned to the call.  Therefore, if we receive a Windows call, the call is assigned to Agent 1.  If we receive another Windows call while Agent 1 is still occupied, Agent 3 is assigned.  If all agents become available again and an Internet call is received, the call is assigned to Agent 3.

In many senses, the second example is still vulnerable to the problem noted in the first example.  Assume our call load is 90% Windows, 5% Office and 5% Internet.  If a call comes in for Windows, it should be rightly assigned to Agent 1.  However, if a subsequent call comes in for Internet support, it might be more appropriate to assign Agent 2 to the Internet call and reserve Agent 3 for the next Windows call.

The fundamental problem with traditional skills-based routing is that it takes so few factors into account.  In part 2, we will consider a potential solution to this problem.

This morning, I received an invitation to Microsoft’s Interact 2008 conference that is taking place from April 8-10 in San Diego, California.  Those of you who have been monitoring this blog for the past month know that I’ve been on the very-fast track to learning about Microsoft Unified Communications.  Although we primarily are developing against Speech Server and Office Communications Server at this point, we envision writing some code against Exchange Web Services to bring e-mails into our CRM also, truly unifying communications.  What gets me fired up is the fact that we truly have the ability to do whatever it is that we want to do.  I’ve said it before and I’ll say it again: what I like most about Microsoft is that they give you a solid foundation, a blank slate, and then they step back and  get out of the way.  Our biggest problem is convincing the business people in our company that even though the sky is the limit, we need to contain things at a more manageable level.

If you’re headed to Interact 2008 and you work in the contact center space or are interested in call distributors, I’d love to take some time to meet up with you and chat one-on-one.  If there is enough interest, we may even be able to form a small community of our own to address contact center concerns and OCS.  It seems that we could all benefit by coming together and building a solid community.  I will continue to blog about our experiences in developing a call distributor, blended predictive dialer, and other software, but we need to get more voices out there.  Feel free to ping me on my blog through comments or by e-mailing me at definitejunkmail (yes, definitejunkmail) on GMail.  I’ll reply back with my work e-mail so that we can stay in touch.

Looking forward to meeting some of you!

One of the first things I really struggled with when ramping up on Microsoft Office Communications Server 2007 was understanding roles, their responsibilities, and how roles overlap.  What roles do I need?  How many of them do I need?  Can an edge role be installed on a mediation server?  What’s more, Microsoft tosses out diagrams like the following with very little explanation:

The information that is available out there about OCS server roles is scattered across many different documents, Web sites, and blog posts.  Hopefully, this post will distill information on the roles available in OCS, their coexistence constraints and hardware/software requirements.  That said, let’s get a little bit of vocabulary out of the way first.

• Office Communications Server: A Microsoft product designed to facilitate communications both inside and outside the office.
• Presence: A metric that takes into account both your availability (available, idle, away) and your willingness (available, busy, on a call) to communicate.
• Endpoint: Any device (SIP phone) or software package that registers itself with Office Communications Server as belonging to a user, meaning that the user can be contacted through the device or software package.
• Enterprise Voice: Probably the most noteworthy addition to the product since 2005; allows calls to enter and exit Office Communications Server.  This means that from any endpoint, users can make or receive calls to traditional phone numbers.
• Public Switched Telephone Network (PSTN): The traditional telephone network that delivers telephone service over dedicated copper cables.

Pool Server Roles

In enterprise-class deployments, multiple servers are typically assigned the traditional front-end server roles.  Although this is sometimes referred to as a cluster, the term that has been assigned in this case is “pool”.  As an aside, I would recommend that any business who is seriously considering OCS for Enterprise Voice use an enterprise-class deployment even if scalability is not a concern.  Resiliency is also something you gain when you deploy multiple servers playing the same role.

Front-End Servers

The front-end server handles a lot of the plumbing for OCS.  It authenticates users against Active Directory, routes invitations to appropriate endpoints, and generally coordinates all other features of OCS.  It is important to note that once a front-end server has assisted in routing the invitation to appropriate endpoints, it gracefully steps back and allows the endpoints to communicate directly.  This means that a front-end server will not be overwhelmed by handling thousands of simultaneous video chats.  It merely directs the invitations and coordinates the set-up of communications, then allows the endpoints to communicate however they may choose.  (The exception to this rule is instant messaging sessions, which always travel through the front-end server for archiving purposes.)  It is also important to note that a front-end server may handle merely these responsibilities or more if the deployment team decides to consolidate multiple roles onto the front-end server.

Specifically, the front-end server:

• Authenticates users against Active Directory
• Aggregates and disseminates presence, contact, and other similar information
• Routes invitations to appropriate endpoints (or gateways) and cancels other invitations once an endpoint has accepted
• Tracks status of sessions (even though the session communicates from endpoint to endpoint) and delivers status update messages like typing, ringing, accepted, etc.

Requirements:

• Dual processor, dual core 2.6GHz+ processor
• 2 x 18GB HDD
• 2GB RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

Directors

Directors have two primary responsibilities: user authentication and direction.  User authentication comes in especially handy when dealing with remote users.  Because remote users cannot be redirected to their home server/pool, the director role proxies their connection through to the correct server/pool.  However, user authentication by a director is also important when an enterprise hosts multiple standard servers or enterprise pools.  Users are “homed” to a server/pool, meaning that that server/pool is the one that stores the users account information.  A director will get the user to the right pool on the first try rather than repeatedly polling every server/pool to see if the user’s account is stored there.  In summary, directors are always recommended when you have remote users and/or multiple standard servers or enterprise pools.

Conferencing Servers

Conferencing servers, in contrast to front-end servers, are designed to maintain a central hub for communications when more than two parties are involved.  This minimizes bandwidth requirements for the conference by maintaining one open channel per endpoint.  Conferencing servers also register and update “focus” on front-end servers.  “Focus” means security and state management for conferences.  Conferencing servers come in two major flavors: A/V conferencing servers, which facilitate audio and/or video conferences, and Web conferencing servers, which facilitate Live Meeting 2007 conferences.

The IM conferencing server:

• Controls focus for three or more IM participants
• Focus includes:
  • Participant list tracking
  • Leader determination
  • Security and management controls
• Is installed by default on all front-end servers and cannot be installed separately

The A/V conferencing server:

• Enables three or more parties to have audio and/or video chats (two parties use a point-to-point connection)
• Reduces bandwidth requirements by mixing audio from other participants into one channel instead of n channels

The Web conferencing server:

• Allows application and file sharing
• May leverage an A/V conferencing server to distribute audio/video in a Live Meeting

The Telephony conferencing server:

• Provides functionality for exposing an audio conference to the PSTN
• Allows the organizer to
  • Mute everyone except the presenter
  • Mute themselves
  • Remove parties
• Is installed by default on all front-end servers and cannot be installed separately

Requirements**:

• Dual processor, dual core 3.0GHz+ processor
• 2 x 18GB HDD
• 4GB+ RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

Archiving and CDR Server

The archiving and CDR server is designed to provide a first-shot at archiving instant messaging sessions and call detail records for compliance purposes.  Although this is Microsoft’s stated intent for the role, it’s hard for a layman to imagine how an out-of-the-box solution could really meet the compliance needs of any reasonable organization.  I have not been able to locate any customizable fields, which means that we have a great track record of who called whom at what time, but that’s pretty much it.

Requirements:

• Dual processor, dual core 2.6GHz+ processor
• 2 x 18GB HDD
• 4GB+ (16GB+ if archiving is enabled) RAM
• Gigabit NIC
• Windows Server 2003 SP1+*

* I was not able get the OCS primary installer to run successfully on Windows Server 2008 RTM.  It may be that the individual installers would run successfully, but I have not confirmed this.  The only role I have successfully installed on Windows Server 2008 is Speech Server 2007.

** The work of mixing audio channels is intense; A/V conferencing servers will benefit from more robust hardware.

My recent experience with discovering, evaluating, and evangelizing Office Communications Server at Extend Health has been great in many senses.  Many projects fail or are axed in their infancy stages.  Some projects probably deserve to be axed at that stage – one of the primary principles of brainstorming is that you entertain all ideas, even the bad ones.  But what about the projects that shouldn’t fail at that stage?  What about the project that are simply struggling to get off the ground and deserve a credible pilot?

I would contest that many projects fail in the idea/skunk works phase because of management failures.  I’m sure many managers of skunk works projects have the best intentions and are qualified managers, but managing a skunk works project is different than managing a sanctioned project.  I scoured the Web (after the project had moved into pilot phase) to see whether there were any documented best practices for managing this type of project.  I didn’t find any distilled, coherent best practices repositories, but I did find some good material that should be referenced here.

Martin Tate had the most to contribute as far as I’m concerned.  He posted a presentation called Implementing Best Practice – A Different Approach, Using ‘Skunkworks’.  In this presentation, he details what skunk works projects are and some of the classic examples (including 3M’s Post-it Notes, Audi’s Quattro, DuPont’s Kevlar, Ford’s Mustang, IBM’s first PC, and the original, Lockheed Martin’s SR-71).  [Note: there weren't any citations, so I'd encourage you to do your own research on those examples.]  He also included several bullet points that just happened to be true of the project I just completed, which was probably dumb luck on my part.  He says that skunk works projects should:

• Like the customer
• Start small and build up
• Maintain low visibility
• Find senior sponsor
• Manage the approvals process
• Create team from the right people

Judy Artunian also had some excellent input in her article for ComputerWorld entitled Skunk Works: The Sweet Smell of Success.  She adds these recommendations:

• Handful of employees with knack for taking a fresh look
• Keep the operation under wraps
• Demonstrate alignment with business objectives
• Keep tabs, but don’t impose a schedule on the project

Finally, Ken Smith blogged about skunk works projects and said:

• Don’t allow skunk works projects to delay normal projects
• Check with product & project/product management first
• Be ready for disappointment

I think the two things that really went right for this project were the fact that I had the CTO as my corporate sponsor, and the fact that I released information at a controlled pace.  I firmly believe that if I had disseminated the information as rapidly as I gathered it, there would have been an overwhelming and sometimes inaccurate flow of information out of the project.  Therefore, if I were to list my own bullet point best practices for skunk works project management (including rehashed versions of points above), this would be it in order of decreasing priority*:

1. Think like the customer (even if the customer is internal)
2. Demonstrate alignment with business objectives
3. Don’t volunteer to manage a project you don’t believe in
4. Find a corporate sponsor who believes in the project (or at least understands its importance and believes in you)
5. Define a series of informal milestones to serve as gating factors for going forward
6. Maintain low visibility
7. Disseminate information at a controlled pace
8. Keep in mind that only a very small percentage of skunk works projects succeed; balance cost-benefit ratio constantly

* If your project doesn’t take into account the first four objectives, don’t bother.  Scrap it and move onto something with more business value.

Overall Rating: 2/5
Response Time: 3.5/5
Overall Knowledge: 1.5/5
Personalities: 4.5/5
Levels to SME*: ?
Familiarity with “Greenfield“: No
Primary Contact: Erwin Gunnells

*Number of contacts (people) I talked to before reaching a subject matter expert.

Shortly before we ruled out Nortel as a potential partner for implementing and deploying Office Communications Server, we contacted Dell to talk about providing the same services.  Where our conversation with Nortel left me uncomfortable, the conversation with Dell ended within two days.  As with Nortel, the people that we spoke with were very pleasant, but I just didn’t feel that they were ready to execute an enterprise deployment of a complicated product.

I might have been able to predict the result of this conversation simply by looking at Dell’s UC site.  There really isn’t very much there at all.  They don’t talk about OCS (at least that I could find), their material seems to be marketing alone, and there aren’t any real case studies.  Aside from the fact that Dell uses OCS internally, there was nothing to make us turn our heads.

The one really positive experience we had with Dell was talking about software concerns.  Casey Spear and a number of Microsoft representatives (Janet Gresco, Steve Todd) have been fantastic in addressing our many questions and even suggesting appropriate alternatives that we didn’t know existed.  If you’re in the Pac/West region, I highly recommend that you contact Casey if you have any questions about Microsoft licensing.

Overall Rating: 3/5
Response Time: 2.5/5
Overall Knowledge: 3/5
Personalities: 4/5
Levels to SME*: 2
Familiarity with “Greenfield“: No
Primary Contact: Matthew Christopher (Western Region ICA Overlay Team)

*Number of contacts (people) I talked to before reaching a subject matter expert.

Nortel was the first partner I contacted, based upon their strategic alliance with Microsoft.  My initial account contact was handled by Efrem Anderson, who did a great job of finding someone for me to talk to.  I was pleased to find that Efrem at least knew the term Office Communications Server, but beyond that there didn’t appear to be a solid understanding of what I needed.  This is completely acceptable at the account manager level; knowing whom to talk to is sufficient.

Efrem put me in contact with Matthew Christopher, who truly seemed to be aware of a lot of the high-level aspects of OCS.  I don’t think I’d have felt comfortable passing all of my needs to him, but as a primary point of contact, Matthew was awesome.  I should also mention that Matthew was very receptive to my questions and seemed to have good knowledge of connectivity to the PSTN, which is one of the areas where I am weakest.  Matthew initially suggested that we pick up a small Nortel CS1000 PBX to help run the system, but when I pressed him for why we needed it and explained our situation, he agreed and even offered that we might want to connect directly to the PSTN through media gateways.  He never offered the word “Greenfield” as the name that typically references this type of deployment, so I got the feeling that he wasn’t completely familiar with how it worked.

In spite of that, if all Nortel employees had been as responsive and knowledgeable as Matthew, we might have selected them as a partner.  Unfortunately, the team Matthew was working with dropped the ball and failed to get me a scope of work.  That, in combination with the fact that I didn’t feel particularly comfortable with the other SMEs on the team, helped us to decide that Nortel wasn’t the partner for us to work with.

I should note that if your business needs a PBX (to hang non-OCS phones, faxes, etc off of), Nortel might make an excellent partner.  My primary problems with them were responsiveness and lack of understanding of “Greenfield” deployments.

As I head into more technically slanted posts for a while, I feel that I should probably make clear what my qualifications and history are.  This will help readers to understand what a certain level of experience will yield when ramping up on various technologies.  It should also be noted that this is a very verbose curriculum vitae, but I believe it to be relevant.

I have a long history of network administration and general computing interest.  I have been dabbling in computers since the late 1980’s, where I started on my parent’s IBM PS/2.  Through the years we upgraded (most notably to Windows ‘95 – that was a memorable August for me) until in college I started writing Web sites in HTML and JavaScript (LiveScript).  I left college after a year and changed jobs rapidly for a while.  I swapped out hard drives and imaged computers for Underwriters Laboratories.  I worked as a PC/network tech for May & Speh (now Acxiom).  I installed computers, ISDN and other phones, and did more networking at The Northern Trust.  I did Y2K research for the Chicago Board of Trade.

While I was at The Northern Trust, I started studying for my MCSE.  I took the tests in rapid succession (mostly due to the fact that I knew the material inside and out) and achieved full MCSE status in the late 90’s.  After I completed my MCSE, I took a job teaching MCSE, CIW, and A+ curriculums while I went back to college.  This is really where a lot of my foundation was ingrained.  I knew the material when I was studying for my MCSE; I had it memorized by the time I completed my second year of teaching.

I taught for what was MicroTech Training Center in suburban Chicago.  Although I loved that job and they trusted me when I had no experience as a technical trainer, I left them due to some questionable business practices.  I’ll just say that if they had ever been audited by Microsoft, Adobe, or any other large software companies, they’d have been put out of business.  After MicroTech, I taught a class or two for random companies and then settled in for a while with I/Tech.  Through that job, I started teaching at Wilbur Wright Community College, where I stayed for the duration of my college education.

After graduating with my first bachelor’s degree, I went back to southwest Michigan to help my dad with some major construction on our family house.  I was tired of teaching, so I took a job at a local school district administering about 700 computers.  I stayed up on the latest technology and made it most of the way through my upgrade to MCSE 2000.

Two years later, I found myself writing software on and off for the school district, which I really loved.  I left that job and moved to Salt Lake City, Utah, where I completed a second bachelor’s (in computer science) and most of an MBA at Neumont University.  Through the school, I took on my current job at Extend Health, Inc.  My current job is leading a team of truly amazing software developers; most of them are smarter than I am.  I make architectural decisions when I’m not writing code, which is how I came to be involved in making decisions that bridge the IT-business divide.

All that to draw this conclusion: I have a solid 10 years worth of network administration, mostly focused in Microsoft technologies with a reasonable amount of Novell and Linux (stick with Gentoo, not Ubuntu!) mixed in.  I have a reasonable amount of experience with software development.  My strongest abilities are problems solving and narrowing potential culprits to determine the true cause of a problem.  This helps me to rapidly assess new technologies, perform pilot deployments, and draw conclusions.

One of the things I value most about being in IT is that there are so many brilliant people around me.  As I mentioned above, most of my team is better than I am at software development.  There are many, many people who are better at raw network troubleshooting than I am.  There are plenty of people who are more intelligent than I am.  What I bring to the table is a well-rounded and well-seasoned background.  Please bear that in mind as you read the more technical posts.

In a previous post, I highlighted the discovery of Microsoft Office Communications Server 2007 as a unified communications solution.  I also speculated somewhat about the viability of the solution for a company whose core business revolves around enterprise voice.  Enterprise voice comes part and parcel with a series of challenges.  Routing inbound phone calls to appropriate personnel based on what the caller is seeking (skills-based routing), using presence indicators, and integration with CRM software are only a few of the problems contact centers face.

Typically, inbound call routing is handled by a software package known as an automatic call distributor (ACD).  ACDs can range in intelligence from the most basic level, where the ACD simply hands the call to the next available person, to advanced, where the ACD makes more involved decisions.  ACDs are critical to companies like Extend Health, Inc.  We have employees in different roles, licensure and HIPAA compliance are significant concerns, and we need to control costs across our various internal and outsourced employees.  I was more than a little concerned that we would have trouble achieving this level of functionality in OCS as there were no visible references to how to handle that.  (Not to mention, our Cisco guys are putting pressure on us by saying that it isn’t possible.)

This concern was almost completely resolved today when I stumbled across a blog post by a guy named Michael Dunn.  Michael worked for Magenic at one point in time, but has recently jumped ship to Microsoft.  Michael’s post (aptly called “Creating a WCF ACD (Automatic Call Distributor) Server Part I“) covers in great detail how to build an ACD, from scratch, using .NET 3.0’s Windows Communication Foundation.  After a frantic flurry of communications from me, Michael politely answered the burning question.  He told me that Speech Server 2007, which is part of OCS, ships with the ability to program against it using Windows Workflow Foundation.

Microsoft has essentially handed us a blank slate and told us to go at it.  It has been my experience in the past that this is where Microsoft shines the most: providing that foundation to build your own building on.  According to what I learned today, I should be able to write an ACD whose intelligence is perfectly matched to what we need.

I took the results from my conversation with Michael and went with it.  The results were nothing less than amazing.  In literally one hour, I went from having zero knowledge of programming Speech Server 2007 to having a fully developed application that examined the incoming number, routed it to a licensed agent (determined by the area code), and if the number had called previously, routed it to the same agent unless they were busy, in which case it selected a new agent that was licensed.  For sixty minutes worth of work and no experience, that’s substantial!

This changes my view of OCS from being a potential solution to being a probable solution.  The price delta mentioned earlier, in combination with the ability to program an ACD in an hour, means that if continue exploring OCS and can program our other features as easily as the ACD, we will likely select OCS as our contact center communications infrastructure.

The next step is to detail what we (Extend Health, Inc.)will be looking at to decide for certain whether OCS is an acceptable solution.  I’ll lay out some initial parameters here, so that readers know what we’re examining, but will cover each of these line items in much more detail as we get to it.

Automatic Call Distributor
The ACD I wrote in an hour is impressive, but we need a much more sophisticated ACD.  The ACD we will write will involve five levels of decision making:

1. What type of employee does the person need to speak to?  Enroller?  Agent?  CSR?  Narrow the list to people in that role.
2. Has the person spoken with someone in that role before?  If so, narrow our list to those employees the person has spoken with.
3. If licensure is an issue, narrow the list to licensed employees.
4. Narrow the list to available employees.
5. Narrow the list to the employees with the lowest variable cost.  If an agent that we’re paying per hour is available, lets route the call there before we route it to an agent that gets paid per call.

We don’t even know if we can write an ACD of that level of sophistication in our current Cisco solution.  If we can, we’d have to outsource it and it would likely take several weeks and a lot of money to complete.  If we want to tweak the ACD, we run into more time and money issues.  My guess is that we can write this ACD in a couple of hours internally and tweak in a matter of minutes.

Predictive Dialer
This is something we haven’t tried yet, but we need to be able to take a list of one thousand people we need more information from, assign a pool of ten agents, and keep those ten agents busy by having the system dial every time an agent completes a call.  The system should also screen answering machines since we need to talk to a person.  Speech Server 2007 is capable of making outbound calls as well as handling inbound calls, so this shouldn’t be an issue.

Offline Queuing
This is also something we haven’t tried yet.  There are some Web sites that allow you to click a button and have someone call you when they are available.  We want to combine this with the ability to give the option to those callers who will be on hold for more than five minutes to be called back.  In short, we want to develop a call queue that is agnostic about call directionality: it doesn’t care whether the call is inbound or outbound.

Voice Signature
In the insurance world, signatures are critical.  Voice signatures (like digital signatures) have recently been deemed legally binding.  Last year, we paid a lot of money and spent a lot of time having a voice signature process developed for our Cisco UCCX solution.  Unfortunately, it never worked.  We believe we can develop a solid voice signature solution for a fraction of the cost, in a fraction of the time.

In summary, we are looking to build out a lot of features that are specific to a contact center.  I’m excited about this and will expose as much information as my NDA allows me to on this blog, so stay tuned!

The company I work for, Extend Health, is currently experiencing the best type of problem: rapid growth.  As many are aware, unchecked or overly rapid growth can sound a death knell for companies.  One potential reason for this correlation is the speed at which decision makers must make decisions.  More decisions are part and parcel of a larger company, but the additional staff necessary to make additional decisions generally lags behind the decisions themselves.  This leaves additional decisions to be made to existing decision-makers, which ultimately leads to a shorter decision lifecycle.

This past year, our company grew 10x in year-over-year numbers.  We did more business in a few days at the start of Medicare season than we did in the entire year previous.  We hope to grow another 4x this year.  This type of growth is putting a significant strain on our existing system, a Cisco UCCX with a few integration pieces installed.  For the past month, we have been evaluating two alternative solution: a Cisco UCCE with more integration servers, and an Avaya solution.  Both solutions are flagship contact center solutions, but they carry a hefty price tag: a low estimate puts the solution around $2.5 million, with a final figure more likely to come in above $3 million.

Part of my job is to review solutions and report on their overall ability to integrate into our environment.  Part of my nature is to add commentary on how viable I feel the solution to be.  As I was researching the Cisco solution, I stumbled across a newer offering: Microsoft Office Communications Server (OCS) 2007.

Microsoft Office Communications Server 2007 was launched not six months ago on 16 October 2007.  The launch was traditional Microsoft strategy: conquest through partnership.  Microsoft’s current claim is that it isn’t there to replace the PBX – yet.  As such, it has strategically allied with a number of hardware vendors such as Nortel, Polycom, Ericsson, Mitel, Dialogic, Audiocodes, and more.  The full list of sanctioned hardware vendors can be found at http://www.microsoft.com/uc/partners_hardware.mspx.  In short, Microsoft partners with hardware vendors for one of two reasons:

1. Connections to the public switched telephone network (PSTN).  No one in their right mind would claim that Microsoft is a telephony expert, especially when it comes to the PSTN or PBX spaces.  Microsoft has aligned itself with hardware vendors that handle nailing down calls and releasing calls when finished.  Microsoft has also worked diligently to leverage the feature set on PBX systems to supplement a weak feature set of an initial offering.  While there are plans to implement features like music on hold, call park, etc, Microsoft simply integrates with existing PBX systems at this point to achieve features it lacks.
2. Handsets/headsets.  The telephony paradigm has not changed significantly in 130 years.  People have used hardware to hear, speak, and initiate/terminate calls.  While the paradigm is changing with the advent of softphones, some people prefer to have the comfort of a traditional telephone set.  Microsoft has partnered with a few vendors (Polycom, Nortel, Jabra) to deliver a telephone experience more in line with traditional telephony.  Mind you, these aren’t your average telephone sets: they have fingerprint recognition, colorful presence indicators, and more!

Microsoft’s forte is providing a solid, well-integrated foundational software layer.  Establishing partnerships for hardware is a must at this point in time.  With the partnerships, Microsoft is able to offer OCS as a truly profound communications platform.

What Microsoft already had a strong background in was collaboration, e-mail and instant messaging, presence, and pervasive integration.  Microsoft Word, for instance, has tight connections with Sharepoint, which can be consumed as an RSS feed in Outlook.  Microsoft, with the delivery of OCS, integrates all the same familiar desktop software with the OCS system, essentially exposing presence and the ability for instant communication from Word, Outlook, or Sharepoint.  Exchange gets the ability to store voicemail and archived IM sessions, meaning you have your entire communications history in one place.  You can initiate a call to the author of a Word document, from the document.  You can start a video chat with someone in a discussion thread on Sharepoint.  The depth and thoughtfulness of the integration alone are sufficient to make a solutions-oriented person take pause.

What Microsoft didn’t have was enterprise voice.  In this context, enterprise voice means that we have lots of inbound and outbound calls.  There was voice chat and even video chat in earlier versions of Microsoft products, but those sessions were limited to internal conversations.  OCS leverages the hardware partnerships mentioned above to add the ability to deal with inbound and outbound calls as well.

I’m not certain that OCS is viable for us.  Our business is critically dependant on enterprise voice: we expect to hit a peak of 10,000 calls in an hour this fall.  However, I have taken significant note of where OCS is at and will continue to investigate the solution as rapidly as possible.  One thing is certain: with an estimated implementation-complete price tag of $500,000 for a jaw-dropping solution that we can build on top of, we’re taking this very seriously.

Game theory is a branch of applied mathematics employed in making high-level strategic decisions.  There are a number of well-known applications of game theory, one of which is the zero-sum game.  A zero-sum game is one in which the sum of all participants gains and losses is zero: that is, each participants gains or losses are exactly offset by other participants’ gains or losses.  I have had the privilege of working for a number of noteworthy companies, each of which had their own unique perspective on IT-business alignment.  It’s unfortunate that many of these companies treated alignment as if it were a zero-sum game.  At the other end of the spectrum are the “synergy” companies.  These positive thinkers portray the IT-business relationship as centered in a radiant beam of beautiful energy shining down directly from heaven.  Doves flutter, angels sing, and IT and business just work harmoniously.

The truth is that aligning business and IT is a lot of work.  Coming back to game theory, there is a better paradigm for this relationship: the prisoner’s dilemma.  The prisoner’s dilemma is classically stated as follows: Two suspects, A and B, are arrested by the police. The police have insufficient evidence for a conviction, and, having separated both prisoners, visit each of them to offer the same deal: if one testifies for the prosecution against the other and the other remains silent, the betrayer goes free and the silent accomplice receives the full 10-year sentence. If both remain silent, both prisoners are sentenced to only six months in jail for a minor charge. If each betrays the other, each receives a five-year sentence. Each prisoner must make the choice of whether to betray the other or to remain silent. However, neither prisoner knows for sure what choice the other prisoner will make. So this dilemma poses the question: How should the prisoners act?

Prisoner A realizes that the best case scenario is for them to both stay silent and spend only six months in prison.  The thought comes to him, however, that prisoner B may betray him.  If the B betrays him and he stays silent, he’ll spend 10 years in jail.  If he betrays prisoner B, he’ll either go free or only spend five years in jail.  This results in a Nash equilibrium, which is a fancy way of saying that more often than not, both prisoners will betray the other.

It frequently seems as if business and IT find themselves in the same scenario.  As an IT guy, I have harbored more than my share of grudges at what sales people make (both promises and salary).  The thought surfaces, “They wouldn’t have anything to sell if it wasn’t for me!”  I’m sure the equivalent thought surfaces on the business side.  But despite the analogy to the prisoner’s dilemma, there is a fundamental difference.  There can be communication between business and IT.  Although it is arguably as difficult as facilitating communication between prisoners in different cells, communication between business and IT can be facilitated and can create a synergistic relationship.

The intent of this blog is to chronicle one IT guy’s attempt to understand and communicate across the divide to the business side.  As such, there should be very few technical posts here that lack a statement of a business case.  And maybe, just maybe, someone reading this blog might someday hear angels singing.