Critical: Centos 4.x Vulnerability

[IWorx-Paul]

There is a critical vulnerability affecting CentOS 4.x, and other distributions not supported by InterWorx-CP. CentOS 3.x is not vulnerable.

Details can be found: http://isc.sans.org/diary.php?storyid=1482 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-2451

This vulnerability enables an attacker to get elevated privileges on a local machine. There have been several exploits released and we can confirm that they work.

This means if you aren't running the very latest kernel released by CentOS, or your server has not been rebooted since the release, and therefore has not loaded the newest kernel yet, your server is vulnerable.

It also means if you're running the CentOS kernel provided by us (which includes the spin_lock kernel panic fixing patch), your server is probably also vulnerable.

A quick way to workaround the vulnerablity is to log into your server as root and issue the following command:

echo /root/core > /proc/sys/kernel/core_pattern

This will protect your servers from the vulnerability until we get the newest kernel built and tested that includes the spinlock kernel panic fixing patch.

Paul

[freegrafton@mac]

Paul, is a reboot required once this patch is run or is it immediate?

[IWorx-Paul]

A reboot is not required after running:

"echo /root/core > /proc/sys/kernel/core_pattern"

Pau

[WebXtrA]

Well... We were the lucky ones today who experienced what this can do with our server.....

Basicly all main*.* index*.* home*.* files were injected with a text from a hacker...

Thanks to the support of the interworx team (and offsite backups) we were able to fix the software on our server.

[WebXtrA]

Hi Paul,

When you guys are going to build a new kernel, could this be included:
http://community.novacaster.com/showarticle.pl?id=4980

The old kernel driver which is standard included with Centos 4.3 does not work with new 3ware RAID controllers.

Thanks,

Rámon

[Justec]

Thanks for posting this information guys!!!

[IWorx-Paul]

The new kernel RPMS are now built and ready for use.

These RPMS are only neccessary if you're currently using a kernel that we have provided to you directly, to fix the spin_lock kernel panic error.

They are located here:

32-bit servers:
http://updates.interworx.com/iworx/RPMS/cos4x/experimental/i386/

64-bit servers:
http://updates.interworx.com/iworx/RPMS/cos4x/experimental/x86_64/

If you feel comfortable doing the install yourself, all you have to do is, as root on your server, do:

rpm -ivh http://url-to-the-new-kernel.rpm

and then reboot your server for the new kernel to load.

If you aren't comfortable doing this yourself, or you're not sure which RPM is the correct one for your server, feel free to open a support ticket and we will assist you.

Paul

[MeddlePAL]

I saw that you mentioned this pertains to CentOS 4.x, since CentOS 4.x is built off RHEL 4.x, is RHEL 4 also at risk and should we update?

[IWorx-Paul]

You would indeed need to update if you are currently running the previous version of InterWorx provided kernel, which fixed the spin_lock kernel panic bug. I wasn't aware of any RHEL 4.x folks that were running that kernel, but if you are, then you should indeed update.

Paul

[nbright]

Could you provide any details about the spin_lock issue? I'm having a few problems with that on some other systems, and trying to figure out how to resolve it. Thanks.

[IWorx-Socheat]

The spin_lock issue (which is independent of the kernel vulnerability mentioned in this thread) causes a kernel panic, and you will a message on the serial console similar, if not identical, to this:

Kernel panic - not syncing: fs/block_dev.c:396: spin_lock(fs/block_dev.c:c035fc80) already locked by fs/block_dev.c/396".

If you can confirm that this is the cause of your problems, then I would install the patched kernel we created, as it solves the problem. If, however, it is not the cause or you aren't sure, I would *not* install our custom kernel, since it's generally better for you to use the official, stock, CentOS kernel.

Socheat

[WebXtrA]

Are there plans to include the new 3ware RAID controllers driver which is included with the new RHEL 4.4 kernel 2.6.9-42.EL?

I know it wasn't the plan before because you guys want to keep the new kernels as standard as possible, but since there is a new RHEL kernel with this now driver I was wondering if it would be included now.

[IWorx-Socheat]

WebXtra,

You may be in luck, I just checked out the srpm for the 2.6.9-42 kernel, and it appears they have included one of the two kernel panic/spinlock patches, if not both. We're testing it on one of our boxes right now. If it stays up for a few days to a week, I think we can all have a little party. :)

Socheat

[WebXtrA]

That sounds very good!

I hope we can have the party :D

- Rámon

[Justec]

Is the party sponsered by InterWorx (hey tax right off :D ), if so get some grey goose!!!

[nbright]

The spin_lock issue (which is independent of the kernel vulnerability mentioned in this thread) causes a kernel panic, and you will a message on the serial console similar, if not identical, to this:

Kernel panic - not syncing: fs/block_dev.c:396: spin_lock(fs/block_dev.c:c035fc80) already locked by fs/block_dev.c/396".

If you can confirm that this is the cause of your problems, then I would install the patched kernel we created, as it solves the problem. If, however, it is not the cause or you aren't sure, I would *not* install our custom kernel, since it's generally better for you to use the official, stock, CentOS kernel.

Socheat

That would be exactly the issue I was seeing. Any kernel newer than 2.6.9-11 would spin_lock on almost every boot up on one of my machines. A rather important one, too. I'm a little nervious to try the latest one, as the machine is remote.

[jcws649]

I just got the control panel installed.. is this still vulnerable?

Im using CentOS 4.4...

[IWorx-Tim]

As Socheat said in another thread somewhere the newest CentOS 4x kernel appears to have used the same patch we were so you should not have any problems using the default kernel.

[jcws649]

Thank you.

[IWorx-Socheat]

After installing the 2.6.9-42 kernel from the official CentOS repos, two of our boxes have been up for almost two weeks now:

[socheat@test1 ~]$ uname -a
Linux tes1 2.6.9-42.0.2.ELsmp #1 SMP Wed Aug 23 00:17:26 CDT 2006 i686 i686 i386 GNU/Linux
[socheat@test1 ~]$ uptime
11:22:28 up 13 days, 10:29, 1 user, load average: 0.00, 0.03, 0.00

At this point, we feel confident that CentOS has incorporated the patch, and we highly recommend that everyone using our custom kernel return to using the official CentOS kernel. This is all you need to do is grab the appropriate kernel for your machine from:

http://updates.interworx.com/centos/4.4/updates/i386/RPMS/
http://updates.interworx.com/centos/4.4/updates/x86_64/RPMS/

or any other CentOS mirror you prefer. Then, install the kernel using rpm -ivh --force and reboot. After reboot confirm you are running the -42 kernel by running "uname -a".

At this point, you may choose to remove our custom kernel, but it's not necessary to do so.

So who's bringing the chips and dip? :cool:

[IWorx-Socheat]

I should rephrase what I said above: We confirmed that CentOS incorporated the patch by looking in the source RPM file, and after running the -42 kernel for almost two weeks now without any problems, we are confident this problem is fixed in the official CentOS kernel.

[Justec]

Do all the RPMs need to be installed?

[ ] kernel-2.6.9-42.0.2.EL.x86_64.rpm 23-Aug-2006 14:07 12M
[ ] kernel-devel-2.6.9-42.0.2.EL.x86_64.rpm 23-Aug-2006 14:08 3.6M
[ ] kernel-doc-2.6.9-42.0.2.EL.noarch.rpm 23-Aug-2006 05:03 2.1M
[ ] kernel-largesmp-2.6.9-42.0.2.EL.x86_64.rpm 23-Aug-2006 14:12 11M
[ ] kernel-largesmp-devel-2.6.9-42.0.2.EL.x86_64.rpm 23-Aug-2006 14:14 3.7M
[ ] kernel-smp-2.6.9-42.0.2.EL.x86_64.rpm 23-Aug-2006 14:18 11M
[ ] kernel-smp-devel-2.6.9-42.0.2.EL.x86_64.rpm 23-Aug-2006 14:19 3.7M


Also, will yum auto update this or will it not see it as an update?

For my system i had the following:

[main]
# WARNING! The kernel is excluded from the update list because this system
# contains the nvnet driver. If you wish to update your kernel to a new
# version, you MUST rebuild the nvnet driver against the new kernel BEFORE
# rebooting or you will lose access to your system!
exclude=kernel-*


I have no changed the exclude line to `exclude=` because the new CentOS has the right nvnet driver.
I did a yum update form SSH and it didn't find anything. Am I doing something wrong or do I just have to update this *manually* (RPM) ?

[IWorx-Socheat]

Hi Justec,

No, you definitely do not want to install all those kernels. Type uname -a on your machine, and look for the one that matches. You'll most likely want either:

kernel-2.6.9-42.0.2.EL.x86_64.rpm -OR-
kernel-smp-2.6.9-42.0.2.EL.x86_64.rpm

I talked to the guys over at Steadfast, and he says he answered your nvnet question via the support ticket you have with them.

Hope that helps,
Socheat

[IWorx-Socheat]

Also, for those who are interested in the nitty-gritty details:

Those of you who are using our patched kernel will not be auto-updated to the official CentOS -42 kernel. This is because when we built our custom kernel, we made the revision number much higher (example, the latest kernel we built is -200). "yum update" will see -200 is higher/"newer" than -42, and thus skip over the -42 kernel.

This is why you must "rpm -ivh --force" install the -42 kernel.

[Justec]

Thanks Socheat.

This is because when we built our custom kernel, we made the revision number much higher (example, the latest kernel we built is -200). Yeah, Im slow, I didn't even think about that. You patch your system with the iworx kernel then CentOS release another broken kernel and it auto updates over your Iworx one... doh...

Steadfast told me the nvnet issue isn't a problem anymore. So I am going to just do this myself, but they are there to back me up if anything happens (Really good guys over there at Steadfast if anyone is looking for a new DC).

So I will leave the yum.conf to no longer exclude the kernel and I should get updates in the future after manually force'n this one.

Thanks again! :D

[bear]

thats good news i will have a look at that later today.
thanks you

updated to the new kernel no problems,
will see how this one goes as my system used to go down every 5 days before swtching to the custom iworx kernel, thanks for testing socheat and keeping us informed.
cheers

[nbright]

Just a quick note to confirm that the -42 is working great on the machine that I was having severe spinlock problems on. I've updated my whole fleet to that kernel and it's working great.

[IWorx-Chris]

Awesome nbright! We've updated 90% of our internal machines as well and haven't seen any new issues.

Chris

[WebXtrA]

Same here, works great!

[bear]

still running great no problems here to :D :D