As part of implementing some improvements to a client’s MOSS Enterprise site that I’d recently recommended they undertake, I had to change the accounts that were being used in their environments to ones that were independent from each environment. I’ve been caught too many times by a developer doing some development who manages to miss-key the AD User’s password into a config file / registry key. The app being developed is then fired up and promptly locks the account out. At the same time, another unrelated system in another part of the building stops working. Why? Because the same AD Account was used in critical aspects of both systems.
Never Ever Use The Same Accounts to Run Your Development And Production Environments!!!
Anyway, they got me back to make this change because it can be tricky. Sure enough, it was. The biggest challenge when making wholesale changes to the service / app pool accounts is that if the farm account update fails to work properly, you then struggle to perform the rest of the changes because the security decryption keys MOSS uses to keep a track of passwords is corrupted. Oh man!
So I fired off the first change and waited… and waited… the WWW Publishing service failed to shut down properly, so after 20 minutes I figured I’d reboot the server and try it again (looking back, I may have saved some time by using pskill to terminate the process instead of rebooting, but it would have resulted in the same issue). Then in the event log, I started to see a proliferation of these error messages – "Error during encryption or decryption. System error code 997" and "An unhandled exception occurred in the user interface.Exception Information: Unable to connect to the remote server"
Once in this precarious position, Microsoft’s recommended solution – for the closest example of something that comes close to the error message – is to rebuild the config database (http://support.microsoft.com/kb/927156) – but it’s not much help onsite at a client’s place… luckily there are some alternatives. Once the Farm account is half-changed, you cannot successfully change the rest of the accounts through the UI… but stsadm is the answer. Joel has the information on this page – http://blogs.msdn.com/joelo/archive/2006/08/22/712945.aspx
Just like using a sledgehammer to swat a fly, stsadm is not encumbered by a user interface or any of those nice-to-have things – it seems to be built around the premise "If it doesn’t work, force it. If it breaks, it needed reinstalling anyway!" 🙂
So with stsadm, I went through the following steps:
First, to fix the central admin account’s decryption key used to drive the app pools, complete the following on the server running the Admin site –
- From the bin directory, run Stsadm –o updatefarmcredentials –userlogin <domain\name> -password <password>
- Then run iisreset /noforce.
On each other server in the farm, you will need to perform the following step –
- As each server stores an encrypted version of the admin account password, you will also need to execute the following command for the account used to run the admin app pool –
- stsadm –o updatefarmcredentials –userlogin <domain\name> -password <newpassword> -local
- Then run iisreset /noforce.
- You will have to remove the "Administration Application Pool Credential Deployment" job that gets created (if it still exists) using the timer job definitions page (otherwise it will prevent you from progressing through the next steps). You can delete it using
"%commonprogramfiles%\microsoft shared\web server extensions\12\bin\stsadm" -o deleteconfigurationobject -id "d36cd1d0-4f27-490a-842a-80e587110411"
- Then to update the other moss site app pools –
- Stsadm –o updateaccountpassword –userlogin <domain\name> -password <password>
- If you happen to use the same account to drive the admin site and the web site(s) (naughty) then you will need the noadmin switch eg
- Stsadm –o updateaccountpassword –userlogin <domain\name> -password <password> [-noadmin]
- The Web site app pools for the non-admin sites should take care of themselves once you do the Farm Admin account, but if not then just use the "UpdateAccountPassword" function on the server to resolve the issue.
Then you will need to fix the SSP’s…
- From the server running the SSP you need to run the following command for each SSP that uses credentials to operate (like ECS and FS), except for the search services (they’re next) –
- stsadm -o editssp -title <ssp name from the farm’s Shared Services page> -ssplogin <domain\name> -ssppassword <password>
- You then run the following commands for the search services
- stsadm -o osearch -farmserviceaccount <OSS searchserviceaccount> -farmservicepassword <OSS password>
- stsadm -o spsearch -farmserviceaccount <WSS searchserviceaccount> -farmservicepassword <WSS password>
- You now need to go into the search service section of the UI and change the indexing and crawling accounts if required.
- Lastly, the SSO Service has to be changed using the Services Applet in the Administration Control section of the server it runs on.
At this point, an IISReset is probably a good idea (a reboot is an even better idea) – once this is done, attempt to access each affected area of the farm and verify that they are all now functioning correctly. If you still see some issues, use the relevant part of this guide to try and reset the credential information. Eventually (after 4 attempts) I moved past the first set of steps to change the Admin site app pool account – the rest was plain sailing from there.
Hope this is of help to someone one day 🙂