During the May Day holiday, the service team upgraded the server environment of the Yitaobang website, from PHP5.3 to PHP5.6, and after the successful upgrade, the server CPU was blown up to more than 90%, which has remained high. Check the server and find that multiple PHP-FPM processes cause the CPU load to be too high, resulting in the website being unable to be accessed normally. Reinstalling the server environment and PHP version multiple times, and even changing PHP to HHVM cannot solve the problem of high CPU load.
E-Taobang server operation process (service architecture diagram): User client → Domain name resolution → Baidu Cloud Acceleration Node (CDN/caching/anti-DDOS/CC attacks) → Alibaba Cloud Shield node (anti-CC/DDOS/WAF attacks) → ECS source server (CSS, JS, and images for CDN diversion), and Alibaba Cloud Cloud Shield does not have any attack information, so it can be completely ruled out as an attack.
Looking at the nginx logs, php-fpm logs, and slow logs, there are no abnormal factors, and I see multiple TIME_WAIT information from the netstat -n command, which is caused by the IP segment of 100.97.x.x (the last IP segment is the IP address of Alibaba Cloud SLB listening service).
Without further ado, let's talk about the specific solution, which was finally handled by the professional operation and maintenance team of the Yitao Gang (V Station Power), and the final result was that the CPU load caused by the SLB's service monitoring settings was too high.
The original SLB service listens to HTTP protocol port 80, and the http protocol health check will regularly send http requests from multiple heads, resulting in continuous access to read HTTP, resulting in php-fpm causing the CPU load to reach more than 90% for a long time, and the specific CPU load usage quota depends on the configuration of ECS. After changing the health check service, the CPU usage goes offline.
If this situation also occurs, change the health check configuration of SLB service listening from HTTP protocol 80 to TCP protocol 80 configuration as follows:
|
Posted on 5/8/2015 8:22:09 PM
|
|
|
|
