Office Communications Server Deployment, Day 5

8:08 AM : Sufficient Information

I arrived at about 6:30 AM and began gathering the data I would need to facilitate deployment of OCS.  I have put together a spreadsheet that has most of the information I’ll need in it.  Several IP addresses are missing from the edge servers (not that I would want to post that on a public Web site anyway) and I haven’t looked into certificate requirements for the Enterprise Voice servers.  That said, I have enough to start creating entries in DNS for client autoconfiguration and I have enough information to install my first front-end server.

I should note that after I got my family situated last night, I did some more looking into Configuration Manager’s deployment things and I found some other resources that may or may not come in handy.  I’ll list them here for future reference or for others’ perusal.

1. The guy who did the whirlwind tour of configuring Configuration Manager: http://blogs.technet.com/deploymentguys/default.aspx
2. Also related to them: http://www.deploymentforum.com/
3. Microsoft Deployment blog: http://blogs.technet.com/msdeployment/
4. Desktop Deployment tech center: http://technet.microsoft.com/en-us/desktopdeployment/default.aspx

Most of those links came from a Web cast from a couple of days back which I watched last night: http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032373731&CountryCode=US

The bad news is that I at least have to get Microsoft Deployment running in order to deploy some bare-metal servers.  The good news is that I know how to work with Microsoft Deployment.  It’s Configuration Manager that’s giving me grief.

9:21 AM : Review Complete

Just finished reviewing IP addresses with my boss and have completed filling out my spreadsheet.  I would recommend filling out a similar spreadsheet if you are working on deploying OCS.  A couple of notes: first, I left our public IPs off the spreadsheet.  Second, I still don’t have the certificate details for mediation server or speech server completed.  I’ll work on those in more detail when I’m deploying enterprise voice.  Here’s the list:

Edit: Removed planning sheet

12:34 PM : Configuring Microsoft Deployment

KMS is now running on the new domain, which facilitates deployment by allowing volume license operating systems to activate against a local server rather than MAK, which authenticates against Microsoft’s servers.  I’m also picking away at producing requirements for my team(s) so that they stay busy and getting the vanilla Microsoft Deployment solution accelerator running.  Until I have at least a bit done on OCS, I can’t dedicate any more time to Configuration Manager.  Microsoft Deployment will at least allow me to push operating systems without actually physically touching the box, since we have an IP KVM.  I’m also very used to setting up these types of deployment (I used BDD 2007 quite a bit).

12:53 PM : Added Windows Server 2003 R2 (32-bit) and Windows Server 2008 (64-bit)

I have two operating systems set up now in the Deployment Workbench.  I also configured Windows Deployment Services with pretty much the default values (but set it to respond to all client requests, known and unknown).  I imported the same Broadcom drivers I originally got from their RIS download, and I’m ready to set up a lab deployment point and create boot images.

3:15 PM : F Lock

That’s really not intended to be a derivative of a curse word, although I almost wish it were:  I just spent the last hour of my life feeling even more frustrated because I knew I had Microsoft Deployment configured properly, but I couldn’t get PXE to actually pull down the boot image.  It turned out that my F Lock key was on.  (Some Microsoft keyboards have an F Lock key that open up some keyboard shortcuts.)  The F12 command was actually being sent as Print.  At least I didn’t print 1200 copies of a boot screen.

4:42 PM : WinPE2.1 & Broadcom

Apparently WinPE2 changed the way it enumerates hardware and accordingly has trouble installing/recognizing Broadcom network devices, at least in a 64-bit environment.  Thankfully Jeff Huston has a solution that I’m trying right now.  At least it didn’t have to do with my F Lock key.

5:47 PM : No Luck

Still no luck.  It seems, however, that this was probably the problem with Configuration Manager in the first place.  I’m quite certain I had the right drivers imported, but this post gives me hope that it’s just a network driver issue.  Maybe if I can find the right driver, I’ll be able to get Configuration Manager running.  For now, I need to run and help a friend drywall.

Office Communications Server Deployment, Day 4

7:43 AM : Firehose

Wow.  I just watched a screencast with the most information I’ve ever seen packed into 32 minutes and 14 seconds.  The screencast was actually tremendously informative and would be beneficial for anyone who is working on deploying operating systems.  Here’s the link: http://edge.technet.com/Media/System-Center-Configuration-Manager-2007-and-Microsoft-Deployment-Toolkit-Screencast/

I also found another interesting link last night that I forgot to mention.  I didn’t use it, but it might be useful to someone else out there: http://myitforum.com/cs2/blogs/cstauffer/archive/2008/02/13/notes-on-getting-pxefilter-vbs-working.aspx

I’m going to try to apply some of what I saw in the screencast at this point.

7:52 AM : Fixed up Windows Deployment Services

I removed the boot images I’d added to WDS and set the PXE Response Policy to not respond.  Apparently this is important so that Configuration Manager will respond instead.

7:55 AM : Configuring Client Agent Properties

I am now working on configuring the client agent properties appropriately to reflect customization in titles and subtitles.  I entered a network access account on the General tab and customized the text on the Customization tab.  For reference, I used:

Field	Value
Organization name	Extend Health, Inc.
Software updates	Installing approved software updates
Software distribution	Installing new applications
Operating system deployments	Installing operating system

8:15 AM : Configuring SAN

Trying to get some SAN volumes straightened out so our deployment files go onto the SAN rather than the local drive.

8:30 AM : Deleting All Packages, Advertisements, Etc

I’m taking Configuration Manager back down to square one as far as what I did last night (creating an OS, task sequence, etc).  That means that the steps I record next would be as if Configuration Manager was brand new.  The one thing I’m not going to do is recreate the collection I created last night, called PXE Registered Systems.  When a new machine attempts to PXE boot, WDS will reply to that PXE boot and (through the integration via the PXE filter) redirect that PXE boot to Configuration Manager.  However, Configuration Manager can’t do anything with the machine until it’s registered in the system.  The PXE filter helps to register the machine with Configuration Manager so that Configuration Manager can then take over and complete its work.

10:09 AM : Adding Boot Images

I just added both of the default (WinPE2) boot images back into Configuration Manager and am assigning them to both distribution points (our primary distribution point plus the distribution point created by adding the PXE site role).

10:18 AM : Adding Windows Server 2008 64-bit

I set up a nice folder structure for Windows Server and added the operating system image to Configuration Manager.  I also assigned the operating system to just the primary distribution point (not the PXE distribution point).

10:28 AM : Added Configuration Manager Client Software Package

Added a software package (from definition) for Configuration Manager Client Upgrade.  The “upgrade” should also work for the base installation per the screencast referenced above.  To add the software package, I chose Add Software Package From Definition, then chose Configuration Manager Client Upgrade (I later renamed this to Configuration Manager Client Installation).  I copied the source files from the Configuration Manager 2007 installation CD into a shared folder structure and pointed the source to that folder.  I then renamed the software package and published it to the primary distribution point only.

10:47 AM : Added Custom Backgrounds to Boot Images

I added a couple of nice background images to each of the boot images for aesthetic value.

10:51 AM : Deleting Operating System Image

Apparently I should have been doing this with the screencast - it turns out I was supposed to copy the entire folder for operating system source.  I’m wiping out the operating system image, copying the whole source DVD, and recreating the operating system image to the same specs.

10:54 AM : Creating Operating System Install Package

I’m now creating an operating system install package, which is where I need the full source of the operating system.  I also had to deploy the install package to the primary distribution point.

11:06 AM : Adding Drivers

I need to add the Broadcom NetXtreme II drivers since I frequently use these for deploying.  I commented on this last night, so I won’t belabor the point further.  I simultaneously created a driver package and added the drivers to the boot images, and then pushed new versions of the boot images to the distribution points.

5:29 PM : Heading Home

So I dealt with trying to get PXE running successfully for the rest of the day and am ready to give up for a while.  I can’t forestall the actual Office Communications Server deployment any longer.  I am pretty frustrated.  OCS tomorrow, just so I have a break from this.

Office Communications Server Deployment, Day 3

12:35 PM : Reinstall Complete

Yes, you read that right.  In addition to everything else I had to deal with this morning (meetings, requirements delivery for some members of my team), I did a full reinstall of Configuration Manager 2007, this time on Windows Server 2003 R2.  I had some nagging errors in the logs that just wouldn’t clear up so I attributed it to Windows Server 2008 and started over.  The good news is that I was able to get the entire Site Status tree to show up green this time, meaning there are no problems.  The weird part was what I had to do to resolve the errors.  I think (am not sure) the resolution was to go look at and delete the error messages (which ended two hours previously) and refresh the component.  I don’t like that as a resolution, but on another component I went to do the same thing and clearing the error messages and refreshing the component didn’t fix the problem: I actually had to go resolve a problem.  The funny part is that it was back to SPNs again.  During the reinstall, I just installed SQL Server 2005 SP2 to the same box to ease my pain, and I didn’t update the SPNs.

Whatever happened, it’s entirely green now.  I’m hoping to get through a few more Webcasts on operating system deployment this afternoon, but I have 2.5 hours worth of meetings and only 2.5 hours left to go in the day.  Might be another late night.

6:58 PM : Catch Up

Just finished grilling.  For anyone who cares, it’s not a good idea to grill a still-partially-frozen New York Strip steak.  It takes forever, burns the outside 1/4 inch, and goes from rare to well-done in about two minutes, which I missed.  I like my steak medium-rare.  Not a pleasant meal.

Today was pretty chaotic overall, which doesn’t lend itself well to blogging my way through the deployment experience.  It largely failed today because I only had a few minutes here and there to work on deployment between meetings.  So, to catch you up to where I am now, let’s run through what I did in those few minute sessions.  I mostly toyed with operating system deployment.  I watched all of Keith Comb’s’ Webcast entitled “Deploying Operating Systems with System Center Configuration Manager (Part 1 of 2)”.  It was absolutely fantastic as far as detail is concerned, and should be watched even before deploying Configuration Manager - there are a couple of key things to note that he says (permissions on Active Directory’s System node and schema extension).

I also used his Webcast to start setting up the structure for deploying operating systems.  One of the first things I did was to add the Broadcom NetXtreme II drivers to Configuration Manager - they are used by some of our desktops and servers, and WinPE2 doesn’t have the driver embedded, meaning that attempts to do a light-touch or zero-touch install on those computers fails.  The driver was fairly easy to import and I won’t belabor the point by stepping through what I did.  I will note, however, that you have to download the RIS (remote installation services, the name for Microsoft’s deployment solution from years ago) version of the drivers or you’ll just bang your head against the wall trying to figure out how to extract the drivers.  Once the drivers are added, I made sure to add them to the appropriate boot images and update the distribution point.  I’ll also note at this point that I love the ability to easily embed the drivers here.  In the Microsoft Deployment Toolkit I know how to get around, but this approach was much more intuitive.

After adding the NetXtreme drivers, I pulled the file called “install.wim” off of the Windows Server 2008 x64 Datacenter, Enterprise, and Standard volume licensing DVD.  A WIM file is a compressed image; Microsoft started using WIM files for deployment with Windows Vista.  A WIM file can also store multiple operating system installs.  In this case, the install.wim was over 2GB but contained images for both the full and core installs of all three predominant flavors of Windows Server 2008 x64.  (Web Server 2008 ships on a different DVD.)  When I first added the WIM to Configuration Manager, I added it to the node called Operating System Images.  I also created a nice folder structure to make separation of images easier to understand.  When I added the WIM, I couldn’t tell whether or not I was going to be able to access all of the images in the WIM.  I’ll save you the heart-stopping anxiety by informing you that you’ll be able to access the images once you get to the task sequence portion of deploying operating systems.

That was actually the next thing I tackled: creating a simple task sequence to deploy the operating system.  Let me stress that it was extremely simple: I entered the options to join the machine to the domain and selected the version of Windows Server I wanted to install from the WIM, and that’s about it.  My thought at that point was, “I can’t believe how easy this is!”

7:14 PM : Not So Fast

The last thing I did before I left work (this is still technically part of the Catch Up, but warranted a logical division in the flow of the post) was to attempt to install the operating system.  We have an IP KVM and 12 blade servers sitting bare metal in our data center, so I used the IP KVM to try a PXE boot off of one of those servers.  No dice.  The first thing I found out was that I needed to install the PXE service point role.  After muddling through the interface to find where to add the role (root/Site Database/Site Management/<site>/Site Settings/Site Systems/<server>), I was able to add the role and set the options pretty much to their defaults.  I did set a PXE password which I removed later when I found out that it would cripple zero-touch installs.  I tried the PXE boot.  No dice.

After a bit more reading, I found out that I also needed the Microsoft Deployment Toolkit 2008, so I downloaded and installed that.  The article I was reading (http://technet.microsoft.com/en-us/library/bb978399.aspx) also said that I needed to configure the WDS PXE filter, which I tried to do from the Start menu, but there is some error with the tool.  I’ve tried a couple more PXE boots after altering a few settings, but still: no dice.

7:46 PM : Validation Exists for a Reason!

Twenty minutes of the last half hour was spent responding to an e-mail, the other ten spent trying to figure out the problem with the PXE filter.  I should have thought of this sooner, but mousing over the red exclamation point revealed the validation error: no Windows Deployment Services.  I had installed the Microsoft Deployment Toolkit, which is also necessary, but hadn’t installed WDS (something the Microsoft Deployment Toolkit relies on).  It is now installing; I had to add it through the Add/Remove Windows Components wizard because it wasn’t in the Add Role wizard.

7:54 PM : Reboot

Rebooted the server after install and am waiting…

8:06 PM : PXEFilter.vbs

Configured PXEFilter.vbs with the following settings:

sProviderServer = “MGR1″
sSiteCode = “DC1″
sNamespace = “root\sms\site_” & sSiteCode
sUsername = “”
sPassword = “”
sCollection = “DC10000D” ‘ Corresponds to PXE Registered Systems Collection (a custom collection I created)

8:11 PM : No Dice

8:28 PM : Break Time

I think part of my problem is related to Windows Deployment Services, but haven’t triangulated what the exact problem is yet.  Have to take a break to get my son to bed.

9:43 PM : Back Again

10:07 PM : Breakthrough

Some success!  I have a successful PXE boot.  I made a couple of changes to get the PXE boot: I told WDS that it shouldn’t authorize itself with DCHP, recycled the service, and reauthorized it and recycled the service again.  I don’t think that had anything to do with it, though.  I think the change that had an effect was explicitly advertising the task sequence for installing Windows Server 2008 to the PXE Registered Systems collection, where the PXEFilter.vbs told Configuration Manager to create a machine account.

10:08 PM : Configuration Manger Logo

I see a big background that says “Microsoft System Center Configuration Manager 2007″.  I’m guessing that’s a good sign.

10:12 PM : Success in IP KVM

Same thing via the IP KVM with a bare-metal server.

10:18 PM : Done for the Night

So after reboot, both servers I was working with (one virtual server, one real bare-metal server) still don’t have an operating system.  I’m guessing that’s a problem with the task sequence, but since I didn’t put much effort at all into the task sequence, I’m not to worried about it.  I think it will come up easily from here in the morning.  I’m quitting for the evening.

Interact 2008 Summary, Day 3

Back to Day 1
Back to Day 2 

Thursday

Birds of a Feather

The third and final day of Interact 2008 kicked off with a Birds of a Feather session rather than a keynote.  This morning, I sat at the Blogging table for a while before moving to the Voice Infrastructure table.  At the blogging table, I met Scott Schnoll and Nino Bilic.  Scott writes technical documentation for Exchange Server and struck me as a very friendly person.  Nino manages the Exchange Team blog in addition to his day job, which apparently has to do with supportability for Exchange.  The discussion started with several people, and one of the questions we kicked around was whether anyone’s job description actually included blogging.  Although no one sitting at the table at that time had blogging in their job description, a Mindsharp representative showed up shortly thereafter to fulfill my prediction that even if it wasn’t happening today, it would happen.  We also talked about where to find inspiration for blog posts and other forms of collaboration before I moved on to the Voice Infrastructure table.

At the Voice Infrastructure table, there were representatives of some larger firm talking about call center uses for OCS, which I obviously found interesting.  Although I missed their introduction, I gathered from the conversation that they were using it in a very different way than we were: they wanted a telephony solution that would serve as a telephony solution; we want a telephony solution that integrates deeply into our existing software to bring together telephony and computing such that it is difficult to distinguish between them.  Wajih Yahyaoui was managing the table and asked a number of questions about what we (customers) would like to see in the voice infrastructure.  The one other noteworthy event was that I ran into a Steven White, who came out of Cisco and offered to answer some of my questions about the Cisco AS5400XM.

Topologies and Routing for Microsoft Exchange Server 2007 (Todd Luttinen)

I don’t have much to say about this particular session.  I thought that it would be more about single-site topologies (meaning configuration of server roles) and the routing through those different roles.  Rather, this session was about enterprise-level, multi-site routing and its intricacies.  Aside from finding the discussion interesting, it wasn’t particularly relevant to me.

High Availability in Microsoft Exchange Server 2007 SP1 Part 1 - CCR vs other HA solutions (Ayla Kol; Alex Wetmore)

I had a great chance to speak with Alex before this session, and was pleasantly surprised to find that he was a very down-to-earth guy.  Alex was the lead developer for cluster continuous routing, one of the high availability features in Microsoft Exchange.  After listening to my description of our needs, he suggested that standby continuous replication would probably be sufficient for our situation if our service agreement could tolerate 40 minutes worth of downtime.  Of course, that all revolves around the Service Level Agreement that we should have in place (but don’t).  What I really appreciated about this session was the atmosphere in the room.  There were several Exchange Services team members in the room, and there was actually a fair amount of friendly heckling that went on.  Ayla saw a couple of them come in and immediately told them (in a joking manner) that they weren’t allowed to ask any questions.  The community atmosphere contributed significantly to the discussion, and I found that there were a number of people who were able to speak to various situations using real-life stories that made it to top-level Exchange support.

High Availability in Microsoft Exchange Server 2007 SP1 Part 2 - Disaster Recovery and SCR Deep Dive (Scott Schnoll)

The last session I went to, this was a great way to close out the conference.  The room had the same atmosphere (which isn’t surprising considering it had the same people) for this session.  Scott had a ton of material to work through, but he made it through efficiently and delivered all of the knowledge necessary to recover from a disaster in the event that one should occur.  As I recall (and this is almost a week ago now, so it’s not word-for-word, but it does have the concept), Scott said that losing a datacenter was a good thing.  There was one other notable quote that fired up the hecklers, but what I appreciated most about this situation was the anecdotal experience Microsoft had to share in testing their replication scenario between Singapore and Puget Sound.  Apparently many more things went wrong than they wanted to go wrong, but they were still able to recover within three hours.

Conclusion

When all is said and done, I’d go back to the comments I made when starting this summary: this has been one of the best-spent weeks of my life as far as careers are concerned.  I was able to network, dig deep, and find answers and vision for the solutions I need to design this year.  I sincerely hope to make it back to this conference next year, and highly recommend that anyone interested in Microsoft OCS do the same.

Interact 2008 Summary, Day 2

Back to Day 1

del.icio.us Tags: interact 2008,microsoft,office communications server

Wednesday

Keynote (Terry Myerson)

In my opinion, this keynote was better than Tuesday’s keynote, which is an odd thing to say considering that Gurdeep overall did a better job of presenting.  I think what I liked about this presentation were the number of times that I felt that Microsoft made themselves truly vulnerable.  No big company is perfect, and when you ship the amount of code that Microsoft ships, you probably ship significantly more bugs than the average company.  Terry was straightforward about a number of problems that have happened in the past year (they even showed the code that shipped that caused the leap year bug), but he also said that they were lessons learned for the Exchange team.  The overall feeling that I had walking out of the keynote was that the Exchange team treated me very much like an old friend: they were affable, and comfortable talking frankly about very sensitive matters.

Microsoft’s Quality of Experience: Defending, Deploying, and Succeeding (Neil Deason; Sam Chon)

Neil Deason has been involved in a number of the sessions I’ve attended, and I’ve been very impressed with him every time he speaks.  In contrast to my previous comment about an argument relating to ports on firewalls, Neil’s presentation today was very solid (the same argument re: ports notwitholding).  The emphasis of this session was not conveyed well by the title; the real thing we learned here was how to ensure that we have a consistent and good QoE.  There was at least one interesting discussion about why audio conferences use the Siren codec rather than the typical RTAudio codec.  The response was that computing power isn’t yet at the point where RTAudio streams can be decoded and encoded in scale amounts.  When computing power reaches that point, Microsoft will definitely look at using the RTAudio codec in favor of Siren.  Until then, those of us who use audio conferencing and have a discerning ear will notice a drop in call quality when shifting to using Siren.

Probably the thing that I found most interesting was something anecdotal.  There was a conversation about ambient noise on the voice conversation, and Mu Han related that they had received so much feedback from customers stating that the ambient noise level was so low that they couldn’t tell whether or not the person was still on the other end of the line.  His statement, and I’m not at all qualified to rate the degree to which he was being facetious, was that they were seriously thinking about reintroducing a higher level of ambient noise in future versions of OCS.  I’ve never heard of a product purposefully introducing more noise - that speaks pretty highly for call quality and their existing noise filtering mechanisms.

Planning and Deploying Voice Routes in Microsoft Office Communications Server 2007 using Enterprise Voice Route Helper (Byron Spurlock)

This was another excellent session with Byron, this time about using the Enterprise Voice Route Helper.  The Route Helper tool is designed to help configure dial plans and other facets of outbound routing.  My development background has involved a fairly significant amount of experience with regular expressions, which was what Byron emphasized several times as one of the most important features of this tool.  If you have regular expressions experience (even if it’s only a little experience), you might find other facets of the tool more helpful.  I especially liked the ability to set up and save test cases, which may help to validate that changes to the outbound routing achieve the expected result.  Another tidbit from this session is the best practice of setting up your phone usages to align with actual phone usages.  Microsoft’s recommended best practice is to create a local, national, and international usage at a minimum.  Routes, on the other hand, should align with their “break-out point”, the physical location where a media gateway connects to the PSTN.  There should be at least one route for each distinct break-out point.

Diagnose and Solve Voice Quality Issues with Microsoft OCS 2007 Quality of Experience Monitoring Server (Wajih Yahyaoui; Jisun Park)

I sat and had lunch with Ji Sun before this session.  We talked about what he had done before coming to Microsoft (he had just finished his PhD in Texas) and what he was doing now.  My impression overall, which was only reinforced during the session, is that Ji Sun is profoundly intelligent but very quiet.  This session was primarily a further exposition on Microsoft’s philosophy of QoE, and an introduction to one of the tools available to diagnose QoE: the Monitoring Server.  The QoE Monitoring Server shipped after OCS was released to manufacturing.  At the most fundamental level, the QoE Monitoring Server is a database which collects metrics sent in by OCS endpoints.  These metrics primarily revolve around MOS scores.  In most cases, MOS scores are subjective opinions (hence the name) of call quality.  Microsoft made an attempt at a more objective score, but maintained the legacy name “MOS”.  Furthermore, Microsoft breaks MOS down into distinct categories of sending, receiving, network, et cetera.  Some basic analytics are run on the metrics after they are collected, but for the most part the OCS administrator should expect to spend some time deciphering MOS scores, dropped packets, and jitter by looking at lots of numbers.  The top layer for the QoE Monitoring Server consists of a report pack for SQL Server Reporting Services that give a nicer visual indicator of overall QoE health.  As a personal anecdote, I’ll add that developing reports in SSRS is a headache, but once they are developed, the reports support e-mail based subscriptions so that a user may receive a report in his/her inbox every hour, day, etc.

Advanced Troubleshooting for Voice in Microsoft Office Communications Server 2007 (Byron Spurlock; Roy Kuntz)

This session was really a wrap-up of Byron’s other two sessions re: tools for managing and troubleshooting OCS problems.  We reviewed and went into more depth on Snooper, we looked more at the Enterprise Voice Route Helper, and we talked about the Office Communications Server 2007 VoIP Troubleshooting Guide.  The VoIP Troubleshooting Guide was authored in large part by Roy Kuntz and is an excellent document for identifying available tools for troubleshooting VoIP issues.

Birds of a Feather

Birds of a feather was designed to be a small roundtable discussion that ran from six to nine.  David and I spent most of our time at the QoE vs. QoS discussion.  Most of the people at the table noted that the session was pretty poorly named.  Microsoft as an entity does not see QoE as an alternative to QoS, but they do embrace the philosophy that you should only implement QoS if necessary.  In an ideal world, QoE and proactive monitoring will preempt the need for QoS.  The discussion at the table was largely driven by us - there were four or five Microsoft personnel, a Psytechnics representative, and an occasional conference attendee.  We were able to go through some of our concerns in detail and have a large amount of time for Microsoft to specifically address our concerns.

Overall, a great second day, but there was so much information and so little time.