High loads and server dies question ?

[bear]

for the past week i have seen out of the ordinary high loads, i have noticed certain
process use high CPU up to 90%
checking the process the ones causing the high loads or high CPU have this at the bottom of the output

httpd 5335 root 65u REG 8,3 0 16466323 /tmp/ZCUD0RfHdY (deleted)
httpd 5335 root 66w REG 8,3 0 23724041 /var/log/httpd/ssl_mutex.5334 (deleted)

Any ideas to what would be causing this or how to resolve the problem, when theres a lot of these process the load goes so high it shuts the server down.

Thank you

[Justec]

That /tmp/ one kinda looks like a PHP based file upload although it could be anything, just something that pop'd into my head.

[bear]

Further checks seems to be apache on port 443 is causing the problems for some reason.

[root@hostname ~]# netstat -lnp | grep 443
tcp 0 0 :::2443 :::* LISTEN 5596/iworx-web
tcp 0 0 :::443 :::* LISTEN 5335/httpd

httpd 5335 root 65u REG 8,3 0 16466323 /tmp/ZCUD0RfHdY (deleted)
httpd 5335 root 66w REG 8,3 0 23724041 /var/log/httpd/ssl_mutex.5334 (deleted)

[tsl]

Probably you have been exploited by a rootkit of some sort.
I bet that if you try to restart apache (httpd) it will complain that something else is using port 443.
If thats the case you have to try to kill that process, clear out /tmp and/or /var/tmp from scripts and investigate which binaries that might have been replaced.
You can use rpm for that actually, i.e:
rpm -V binutils
rpm -V findutilsDo a "man rpm" and look for the "Verify" part to get the different formats of outputs and what they mean.

I might be misstaken about your problem but it's very similar to what have happened to servers we have hosted before running phpBB, post-nuke and similar...

-tsl-

[bear]

Thats what i though but no nothing, restarts no problems, looked through logs and they seem ok.
checked whats running on 443 and that clear.
The only thing i have noticed is the logs did stop working on /usr/local/interworx/var/log
still feel theres a problem on the ssl side .
Thanks for your reply


should there actually be a ssl_mutex file in

SSLMutex file:/home/interworx/var/run/ssl_mutex

i have several missing from there will check my backup

[bear]

Well from tests on the server i have found its using HTTPS thats causing the problem.
using interworx cp for various tasks soon raised the load, just browsing or simple settings changes raised the load from 0.60 to 3.5
noticed also once logged out of interworx the problem continues unless the process are killed,
the interworx process seem to carry on for a long time and continue to cause higher than normal loads
now got to figure out why this is now a problem and may be auto kill those process off if they continue for longer than a certain length of time.

[Justec]

Take a look at this thread, maybe it will help??

http://interworx.com/forums/showthread.php?t=2427

[bear]

Take a look at this thread, maybe it will help??

http://interworx.com/forums/showthread.php?t=2427

Thank you for the link
i tried tweeking httpd-custom.conf but will think about auto killing the proccess.

for example my server is very busy tonight but only showing
load average: 0.22, 0.42, 0.46 that because no ones in the CP
Login to the CP and the load will rise, the more you use it the higher it goes and thats just one person.

Thanks for your reply

[bear]

Well good job this forums here a source of information for us not so experienced peasants :p

Any way this really made a difference, in fact because the interworx CP is loading
so much quicker the CPU does not stay high for long and keeps the load at bay.

Thank you pascal theres some great tutorials around here

http://www.interworx.com/forums/showthread.php?p=3523#post3523
I forgot to tell you that I also done this for iworx 2443 port.

Indeed I had a lot of users whom complained for iworx was very very slow. (see one of my other post about this)

And since I've done this also in Interworx everybody tell us it is much more quick

I've change the file


and I've just replaced exiting


by


exactly as we done for our default vhost.

We should ask for iworx to upgrade the SSL config for virtualhost both for our vhosts and their SSL config file

Pascal
Pascal