I recently had a problem with my Amazon EC2 server that caused me to have to fire up another fresh instance and transfer the files over. My original server was a “small” instance (which was the smallest available when I first set it up). I mistakenly chose a “micro” instance for the new server thinking that it was just another name for “small”.
3 days of misery ensued.
The server kept on maxing out with high server loads. (The server load average was hovering between 4 and 20 — ideal is around 0.3). However there were no processes that were using high CPU or memory resources according to the top command. With a server load of over 10, the most intensive process was using 0.1% of the CPU – which puzzled me. There was only one reading that appeared to be topping out (other than the load average), and that was %st (Stealth Time) – it was sitting at over 98%st.
Output of Top Command on Micro Instance
After some research I discovered that “Stealth Time” is percent of operations that are made to “wait” for the virtual machine to allow it to continue. This means that there was some throttling going on under the covers. This led me to this excellent review of Amazon EC2’s micro instances where the author describes the exact same experiences with his simple blog.
After converting the instance back to a small instance, all is well again.
Output of Top Command on Small Instance