<Aug 3, 2009 12:38:48 PM EDT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Aug 3, 2009 12:38:48 PM EDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>
<Aug 3, 2009 12:38:48 PM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:941)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1029)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:854)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
>
<Aug 3, 2009 12:38:48 PM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Aug 3, 2009 12:38:48 PM EDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Aug 3, 2009 12:38:48 PM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

From the error message it looked like the boot.properties on the Managed Server does not have the right username and password, we made sure it was updated. Tried to re-start the server, same issue. We tried to delete the entire LDAP directory on the Managed Server side so that it syncs the latest from Admin Server which should not have any issues, but that didn’t work either.

Well, now we ran out of options and contacted Oracle Support, they looked at it and asked us to do a couple of other things which did not work either. Finally, before we escalated further Oracle Support Engineer asked us to try one more last thing that is to enable MasterFirst on EmbeddedLDAP. Honestly I never knew such attribute existed and the help for that attribute looks something like this.

MasterFirst-Specifies whether a Managed Server should always connect to the embedded LDAP server on the Administration Server, instead of connecting to the local replicated LDAP server.

Miraculously enabling this attribute did seem to work and the Managed Server started up happily without any issues. Now the question is, Why do we need to specifically enable this attribute when we do not have issues in other environments? The answer is we don’t know “yet”. We suspect its Network, but not entirely sure and we will find out very soon. But whatever it is “MasterFirst” did save my day and probably days!