If you are a WebLogic Developer or an Administrator there is 90 – 100% chance you have had situations where the WebLogic Server log files are filling up the disk space. Typically, when a WebLogic Server is started you will notice in addition to other files it creates an .out and .log files in your logs directory. The .log file will contain all the weblogic log messages and the .out file will contain the Standard out, i.e if you are doing a System.out.println in your Application code that is running inside WebLogic, this .out will contain that print line.
WebLogic Server provides a way to configure rotate policies on your .log file depending on the size or a specific time interval. But unfortunately you cannot configure the same on the out files for obvious reasons. This could be a challenge if you are managing a Web Farm that contains a large number of WebLogic domains spitting out messages that are going to standard out. One can argue, why would you be in this situation if applications use Log4J or WebLogic’s Non Catalog Logger to make sure none of the application code is doing a System.out.println. Yes, that can be possible but unfortunately we do not live in a perfect world. With the plethora of Open source software that is being bundled into application deployments it is very hard to make sure all these are doing the right thing. A simple e.printStackTrace() that is caught in a recursive loop can bring down your system.
We tried to solve this issue and the first and obvious solution we came up was to replace the System.out and System.err streams with custom Logging stream’s that will do the Rotation. This has been explained very well here. We configured a StartupClass that did the configuration of custom logging stream while the server is starting up. This solution did work well, but we quickly realized that the rolling logs do not capture the Thread Dump. The second solution (if you are using unix OS’s) was to use logrotate, you can run this utility as a cron job at regular intervals. We have added a StartupClass that registered a Timer during Server start that ran every hour (this is configurable via a System property) and rotated the out files if the size is greater than 2MB. The logrotate configuration we use is shown below.
logrotate.conf
“logs/*.out” {
copytruncate
compress
rotate 1
size=2M
sharedscripts
olddir ../old-logs/
postrotate
time=`date +%s`
echo $time
mkdir -p /export/weblogic/user_projects/mydomain/servers/myserver/old-logs/$time
mv old-logs/*.out.1.gz /export/weblogic/user_projects/mydomain/servers/myserver/old-logs/$time
endscript
}
The logrotate utility uses the configuration above to rotate all .out files in the logs directory if they are above 2MB and after they are rotated they are archived into an old-logs directory. You can also add more custom logic to the postrotate script.
Once you have configured this, as a WebLogic or any Middleware product Administrator you do not have to worry about log files filling your disk space and spinning the CPU’s.
Powered by FireStats