Ghattu's Technology Blog » Uncategorized

TIBCO acquires DataSynapse

sghattu — Tue, 25 Aug 2009 14:41:47 +0000

TIBCO joining the Cloud Band wagon by acquiring DataSynapse.

http://www.tibco.com/company/news/releases/2009/press985.jsp

VMWare to acquire Spring Source

sghattu — Tue, 11 Aug 2009 15:48:31 +0000

This is a pretty interesting move by VMWare. Does this mean VMWare is trying to get into Java App Server space and become the new Under-dog? Only time will tell.

http://blogs.zdnet.com/BTL/?p=22526

WebLogic EmbeddedLDAP's MasterFirst Attribute saved my day!

sghattu — Mon, 10 Aug 2009 16:43:47 +0000

While provisioning WebLogic environments to a new infrastructure, recently we started experiencing a problem where the Managed Server had problems starting up. It consistently failed with the error shown below.

weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:941)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1029)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:854)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
>

From the error message it looked like the boot.properties on the Managed Server does not have the right username and password, we made sure it was updated. Tried to re-start the server, same issue. We tried to delete the entire LDAP directory on the Managed Server side so that it syncs the latest from Admin Server which should not have any issues, but that didn’t work either.

Well, now we ran out of options and contacted Oracle Support, they looked at it and asked us to do a couple of other things which did not work either. Finally, before we escalated further Oracle Support Engineer asked us to try one more last thing that is to enable MasterFirst on EmbeddedLDAP. Honestly I never knew such attribute existed and the help for that attribute looks something like this.

MasterFirst-Specifies whether a Managed Server should always connect to the embedded LDAP server on the Administration Server, instead of connecting to the local replicated LDAP server.

Miraculously enabling this attribute did seem to work and the Managed Server started up happily without any issues. Now the question is, Why do we need to specifically enable this attribute when we do not have issues in other environments? The answer is we don’t know “yet”. We suspect its Network, but not entirely sure and we will find out very soon. But whatever it is “MasterFirst” did save my day and probably days!

JRockit – Error Message: Illegal memory access. [54]

sghattu — Thu, 02 Oct 2008 21:16:00 +0000

Recently we upgraded all our WebLogic domains to use JRockit from Sun JDK 1.4. After we restarted our servers we started seeing a JRockit dump in few of our servers. After some research and advice from Oracle Support team we were able to start our servers without the crash only if we add the JVM flag “-Xverify:all”. In case you run into this issue you can get over this by adding this flag. The crash dump looks as shown below:

===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 00:02:32 on Wed Oct 1 09:25:47 2008

Additional information is available in:
/opt/satya/fooDomain/jrockit.30205.dump
/opt/satya/fooDomain/core or core.30205
If you see this dump, please open a support case with BEA and
supply as much information as you can on your system setup and
the program you were running. You can also search for solutions
to your problem at http://forums.bea.com in
the forum jrockit.developer.interest.general.

Error Message: Illegal memory access. [54]
Signal info : si_signo=11, si_code=2 si_addr=(nil)
Version : BEA JRockit(R) R27.2.0-131-78843-1.4.2_13-20070320-1511-linux-ia32
GC Mode : Garbage collection optimized for throughput
GC Strategy : Generational Parallel Mark & Sweep
: Current OC phase is: not running. YC is not running.
: GC strategy for GC 7 was genparpar
: GC strategy for GC 8 was genparpar
: GC strategy for GC 9 was singleparpar
: GC strategy for GC 10 was genparpar
: GC strategy for GC 11 was singleparpar

Rotating your Standard Out/Err log files

sghattu — Thu, 10 Apr 2008 01:50:00 +0000

If you are a WebLogic Developer or an Administrator there is 90 – 100% chance you have had situations where the WebLogic Server log files are filling up the disk space. Typically, when a WebLogic Server is started you will notice in addition to other files it creates an .out and .log files in your logs directory. The .log file will contain all the weblogic log messages and the .out file will contain the Standard out, i.e if you are doing a System.out.println in your Application code that is running inside WebLogic, this .out will contain that print line.
WebLogic Server provides a way to configure rotate policies on your .log file depending on the size or a specific time interval. But unfortunately you cannot configure the same on the out files for obvious reasons. This could be a challenge if you are managing a Web Farm that contains a large number of WebLogic domains spitting out messages that are going to standard out. One can argue, why would you be in this situation if applications use Log4J or WebLogic’s Non Catalog Logger to make sure none of the application code is doing a System.out.println. Yes, that can be possible but unfortunately we do not live in a perfect world. With the plethora of Open source software that is being bundled into application deployments it is very hard to make sure all these are doing the right thing. A simple e.printStackTrace() that is caught in a recursive loop can bring down your system.
We tried to solve this issue and the first and obvious solution we came up was to replace the System.out and System.err streams with custom Logging stream’s that will do the Rotation. This has been explained very well here. We configured a StartupClass that did the configuration of custom logging stream while the server is starting up. This solution did work well, but we quickly realized that the rolling logs do not capture the Thread Dump. The second solution (if you are using unix OS’s) was to use logrotate, you can run this utility as a cron job at regular intervals. We have added a StartupClass that registered a Timer during Server start that ran every hour (this is configurable via a System property) and rotated the out files if the size is greater than 2MB. The logrotate configuration we use is shown below.
logrotate.conf
“logs/*.out” {
copytruncate
compress
rotate 1
size=2M
sharedscripts
olddir ../old-logs/
postrotate
time=`date +%s`
echo $time
mkdir -p /export/weblogic/user_projects/mydomain/servers/myserver/old-logs/$time
mv old-logs/*.out.1.gz /export/weblogic/user_projects/mydomain/servers/myserver/old-logs/$time

endscript
}
The logrotate utility uses the configuration above to rotate all .out files in the logs directory if they are above 2MB and after they are rotated they are archived into an old-logs directory. You can also add more custom logic to the postrotate script.
Once you have configured this, as a WebLogic or any Middleware product Administrator you do not have to worry about log files filling your disk space and spinning the CPU’s.