Here is an error message that I remember facing without ever finding a solution.
But BlackmoreOps did – slow disk speed.

The Error

Aug 22 15:38:07 servercore kernel: Call Trace:
Aug 22 15:38:11 servercore kernel: INFO: task httpd:30482 blocked for more than 120 seconds.
Aug 22 15:38:11 servercore kernel: Not tainted 2.6.32-431.11.2.el6.x86_64 #1
Aug 22 15:38:11 servercore kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

The Reveal

Linux uses up to 40% of the available memory for file system caching.
After this mark has been reached,  the file system flushes all outstanding data to disk. It does this at kernel-level priority, blocking other tasks.
By default, there is a time limit of 120 seconds to flush the data to disk. In this case, the I/O subsystem is not fast enough to flush the data within 120 seconds.

The Resolve

Start flushing sooner. Update /etc/sysctl.conf

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
Advertisements