store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Thu Oct 02, 2014 12:22 am

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 34 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: 1and1 ASL Kernel Problem
Unread postPosted: Tue Jun 21, 2011 10:05 am 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
I have a problem with a new 1and1 server. Whenever I run an ASL kernel, it will randomly lock up and I have to reboot it from the 1and1 control panel. I'm not so sure what happens to it. There is nothing in the logs, etc., it just stops responding. I can run the stock CentOS kernel and it doesn't do this. The 2 ASL kernels I have run are 2.6.32.41-4.art.x86_64 and 2.6.32.41-1.art.x86_64. They both do the same thing. The CentOS kernel I am running that doesn't have this problem is 2.6.18-238.9.1.el5. It is also random when it locks up. This last time I was running for 4 days, but I have had it run for 6 days, 1 day, 2 days, etc. Here is /var/log/messages from when this happened this morning.

Code:
Jun 21 09:12:47 myhost xinetd[2971]: EXIT: smtp status=111 pid=5186 duration=0(sec)
Jun 21 09:12:49 myhost xinetd[2971]: START: smtp pid=5193 from=x.x.x.x
Jun 21 09:12:50 myhost xinetd[2971]: EXIT: smtp status=0 pid=5172 duration=5(sec)
Jun 21 09:12:50 myhost xinetd[2971]: EXIT: smtp status=0 pid=5193 duration=1(sec)
Jun 21 09:28:52 myhost syslogd 1.4.1: restart.
Jun 21 09:28:52 myhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 21 09:28:52 myhost kernel: Initializing cgroup subsys cpuset
Jun 21 09:28:52 myhost kernel: Initializing cgroup subsys cpu
Jun 21 09:28:52 myhost kernel: Linux version 2.6.32.41-4.art.x86_64 (mockbuild@archelon.atomicorp.com) (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP Wed Jun 8 11:54:24 EDT 2011

As you can see, it just seems to stop. There is nothing else until I send the restart signal from the 1and1 control panel. Is there any way to narrow down what is going on here?


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Tue Jun 21, 2011 2:23 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3638
Location: Chantilly, VA
First I would check to see if its actually locked up by looking at your last and uptime data to see if it crashed and your performance stats to see if the system was just under heavy load and was nonresponsive. The main syslog isnt the best tool to tell you if the system stops responding (nothing may be logged), but if you want to use your syslogs as a timer for uptime then turn on MARK mode so its always marking the logs even if it hasnt nothing to log.

Also, check to make sure you were not shunned, that can make the system seem like its nonresponsive when really you just cant get to it.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Tue Jun 21, 2011 3:03 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
Yeah, I just used the system log to show that nothing was going on for that period. From what I can tell, it is a hang. I don't see anything logged anywhere once it happens. I do know that I am not shunned and I have my IP address in the whitelist. I can't connect to any service from anywhere. From reviewing everything I can see, it seems to be not responding. Is there anything I can try at this point?


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Tue Jun 21, 2011 5:31 pm 
Offline
Forum Regular
Forum Regular

Joined: Sat Mar 28, 2009 6:58 pm
Posts: 853
Location: Germany
have you tried connecting via serial console to have a look on the server meanwhile it seems stuck?


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Tue Jun 21, 2011 7:16 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3638
Location: Chantilly, VA
Yeah, /var/log/messages is actually pretty quiet so its hard to find the exact time things occurs from that allow. So lets see if we do that, and also can rule some things out at the same time to find the root cause. First, get sysstat setup on the box so you can see if this is or is not a load or performance driven issue, if I had to guess Id say something load related is happening, like a large directory that syscheck has been configured to look at, and a lot of i/o as a result. Just a hunch though, so lets see if we can rule that out. As root run this command:

yum install sysstat

Which will install the systems built in performance logging capabilities. From there we will have data we can look at to start ruling root causes out. I'd also enable marking in syslog. If you have a standard setup, thats usually as simply as adding -M to your syslog startup which should be in this file:

/etc/sysconfig/syslog

Look for a line like this:

SYSLOGD_OPTIONS="-m 0"

And set it to -m 1:

SYSLOGD_OPTIONS="-m 1"

Then restart syslog:

/etc/init.d/syslog restart

then syslog will mark your syslog file every minute, even if there is no event. You'll see a log entry like this:

Jun 21 19:14:38 hostname -- MARK --

This will help to narrow down the real time this occurs, which will further help to isolate the root cause.

Once you have a time frame that this occurs, then look at your sysstat data with the command "sar" (run man sar, as its got a lot of options to look at different performance data, and you want to look at everything going on, i/o, cpu usage, memory, swapping, etc.)

From there we can start to isolate whats going on, and find a solution together.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Wed Jun 22, 2011 1:20 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
Thanks, Mike. I will get this setup over the weekend and let you know what I find out.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Fri Aug 12, 2011 3:13 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
Well, I decided to give the asl kernel another go after 1and1 got some other issues resolved with rebooting my server and it ran great for almost a week. It just did the same thing again though and I logged into the serial console and saw this. Does this tell you anything?
Code:
PAX: execution attempt in: <anonymous mapping>, 6c7b4f274000-6c7b4f278000 6c7b4f274000
PAX: terminating task: /usr/libexec/paxtest/anonmap(anonmap):8268, uid/euid: 0/0, PC: 00006c7b4f274000, SP: 00007fff64296ba8
PAX: bytes at PC: c3 00 00 00 00 00 00 00


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Fri Aug 12, 2011 3:21 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3638
Location: Chantilly, VA
Quote:
Well, I decided to give the asl kernel another go after 1and1 got some other issues resolved with rebooting my server and it ran great for almost a week. It just did the same thing again though and I logged into the serial console and saw this. Does this tell you anything?
Code:
PAX: execution attempt in: <anonymous mapping>, 6c7b4f274000-6c7b4f278000 6c7b4f274000
PAX: terminating task: /usr/libexec/paxtest/anonmap(anonmap):8268, uid/euid: 0/0, PC: 00006c7b4f274000, SP: 00007fff64296ba8
PAX: bytes at PC: c3 00 00 00 00 00 00 00


That message is completely harmless and normal, please see this FAQ:

https://www.atomicorp.com/wiki/index.ph ... st_mean.3F

Now as I recall you said it "locked up", can you explain what you mean by that? Are you not able to log in? Are you sure you havent been shunned by the system?

And are you running any performance tracking softwae on the system that can tell you what was happening when it "locked up", such as was the load high, did you have high I/O, cpu usage, memory usage, etc?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Fri Aug 12, 2011 6:35 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
I am sure I am not shunned. I have tried accessing the server from multiple locations. Nothing works from any location. No websites, no email, nothing. I log into the serial console and can't do anything until I manually reboot the server from the 1and1 admin panel. I am pretty sure it isn't a load issue either because I was on the server right before it did this today. I actually look at the load and didn't see any problems. I am going ahead with your suggestions from the earlier post to setup some tests to see if I can't tell the load right before it happens. The time of day is random too. Today it was at around 3, but it has happened in the middle of the night and at around 11:30am too. The day is also random.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Fri Aug 12, 2011 6:48 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3638
Location: Chantilly, VA
Please let us know. Generally if a Linux kernel has a problem, it will log it, and since thats not happening here that sounds like the server may be simply unresponsive due to load, memory, I/O etc. If you can setup some performance monitoring on your system, that would tell us thats whats happening or not.

Right now, without any log messages, and no other data to work from its hard to say its the kernel either. It could be any manner of things, so lets see if we can eliminate load, cpu, memory and i/o issues first.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Sat Aug 13, 2011 5:22 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
I got the monitoring setup. I have set it up to put in a log entry every minute. I'll let you know what I find out. I did make the changes you suggested to have it mark every minute, but I don't see the mark in the logs. Any idea why that wouldn't work? I have tried all kinds of different settings, but it just won't put the mark in the log.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Sat Aug 20, 2011 8:51 am 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
OK, this happened again last night. Here is a general report from sar. I have it reporting every minute and it doesn't look like anything stange was going on before it happened and there were no more entries once it happened. What else would you need?
Code:
02:02:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
02:03:01 AM       all      2.01      0.00      0.42      0.62      0.00     96.95
02:04:01 AM       all      2.05      0.00      0.47      0.72      0.00     96.75
02:05:01 AM       all      2.00      0.00      0.50      0.63      0.00     96.87
02:06:01 AM       all      2.31      0.00      0.59      0.85      0.00     96.25
02:07:02 AM       all      1.92      0.00      0.47      0.20      0.00     97.41
Average:          all      2.03      0.00      0.51      0.39      0.00     97.07

08:43:23 AM       LINUX RESTART

08:45:02 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
08:46:01 AM       all      8.70      0.00      1.06      4.49      0.00     85.75
Average:          all      8.70      0.00      1.06      4.49      0.00     85.75


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Sat Aug 20, 2011 12:23 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3638
Location: Chantilly, VA
Did you ever get marking configured in your logs? It looks like may have a general time when your system is experiencing this issue, possibly down to a few minutes window, syslog marking would tell us all for sure (sar usually runs every 5 minutes).

If so, what does syslog record as the last mark?

And do you have any scheduled events that run at that time, backup jobs, cronjobs, raid sync cron jobs, etc.

Does this happen at the same time every night, or only on specific days, etc?

If it happens at different times, does it happen during a similar window, such as in the first quarter of the hour, or is it demonstrably random? And if so, what are the timestamps for when this occurs (there may be a pattern to the times thats non-obvious on first glance, software raid syncs for example happen on a pattern).

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Mon Aug 22, 2011 4:53 am 
Offline
Forum User
Forum User

Joined: Tue Apr 27, 2010 2:21 am
Posts: 30
Hi,

I think I may be having a similar problem with my 1&1 dedicated machine. I can connect via the serial console and rebooted... after reboot I still connected reach via network only serial, until I did a "service network restart". The output was:

service network restart
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
grsec: denied modification of grsecurity sysctl value : harden_ptrace by /sbin/sysctl[sysctl:4050] uid/euid:0/0 gid/egid:0/0, parent /etc/rc.d/init.d/network[network:4041] uid/euid:0/0 gid/egid:0/0
Bringing up loopback interface: [ OK ]
Bringing up interface eth0:
Determining IP information for eth0... done.
[ OK ]
grsec: denied modification of grsecurity sysctl value : harden_ptrace by /sbin/sysctl[sysctl:4270] uid/euid:0/0 gid/egid:0/0, parent /etc/rc.d/init.d/network[network:4041] uid/euid:0/0 gid/egid:0/0

Don't know if the above helps?


Top
 Profile  
 
 Post subject: Re: 1and1 ASL Kernel Problem
Unread postPosted: Mon Aug 22, 2011 5:11 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue May 10, 2005 1:24 pm
Posts: 193
I never got syslog mark to work, but I did setup sar to run every minute instead of every 5. As you can see it put in an entry at 2:07AM and then didn't put another until I rebooted the server from the 1and1 admin console at 8:43AM. Do you need me to get the syslog mark working? The day and time of day are random. I'm not sure about what part of the hour though. I will need to keep an eye on that to tell.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 34 posts ]  Go to page 1, 2, 3  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot] and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group