ひさびさにZABBIXからメールが来たー!!
Trigger: Processor load is too high on vincentina Trigger status: PROBLEM Trigger severity: Warning Trigger URL: Item values: 1. Processor load (1 min average per core) (vincentina:system.cpu.load[percpu,avg1]): 5.17 19:35 (5時間前)
Trigger: Processor load is too high on vincentina Trigger status: OK Trigger severity: Warning Trigger URL: Item values: 1. Processor load (1 min average per core) (vincentina:system.cpu.load[percpu,avg1]): 2.865 19:44 (5時間前)
大したことねーなーっと思ってたらお手製のLA高検知スクリプトからのメールも来たー!
Load average 8 19:36 (5時間前) Load average 16 19:37 (5時間前) Load average 17 19:38 (5時間前) Load average 16 19:39 (5時間前) Load average 18 19:40 (5時間前) Load average 19 19:41 (5時間前) Load average 19 19:42 (5時間前) Load average 20 19:43 (5時間前) Load average 18 19:44 (5時間前) Load average 7 19:45 (5時間前) Load average 2 19:45 (5時間前)
Zabbixからのメールはちょうど山の登り口と降り口のところらしい(笑)
ログを見たら、なかなかいい感じにDoSられていたみたいなんだが、mod_dosdetectorがうまく動いてなかったようだね。
198.1.153.12 - - [29/Jan/2014:19:37:33 +0900] "GET /register.php HTTP/1.1" 404 612 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:33 +0900] "GET / HTTP/1.1" 503 585 "-" "Mozilla/5.0 (Windows NT 5.1; rv:25.0) Gecko/20100101 Firefox/25.0" 198.1.153.12 - - [29/Jan/2014:19:37:33 +0900] "GET / HTTP/1.1" 503 585 "-" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:33 +0900] "GET /register HTTP/1.1" 503 585 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:29 +0900] "GET / HTTP/1.1" 200 13710 "-" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:34 +0900] "GET /tools/quicklogin.one HTTP/1.1" 503 586 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:34 +0900] "GET /index.php?register HTTP/1.1" 503 586 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:34 +0900] "GET /login.php HTTP/1.1" 503 586 "-" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:34 +0900] "GET /login.php HTTP/1.1" 503 586 "-" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:37:36 +0900] "GET /member.php?mod=logging&action=login HTTP/1.1" 404 612 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" -- 198.1.153.12 - - [29/Jan/2014:19:37:54 +0900] "GET /index.php HTTP/1.1" 301 20 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:38:00 +0900] "GET /logging.php?action=login HTTP/1.1" 503 584 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:38:00 +0900] "GET /register.php HTTP/1.1" 503 584 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:38:00 +0900] "GET /tools/quicklogin.one HTTP/1.1" 503 584 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:38:00 +0900] "GET /index.php?register HTTP/1.1" 503 584 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0" 198.1.153.12 - - [29/Jan/2014:19:38:01 +0900] "GET /signup/ HTTP/1.1" 404 612 "http://www.vincentina.net/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"
mod_dosdectorの閾値の見直しとかをしてapache再起動しておいた。
ほぼデフォ設定なので、5秒置きに監視してrewriteでとばすという設定なのだけど、5秒置きのログをとってみると、上手い具合に回避されてたのかもしれん。
ってことで設定変更だー。
この程度ではDoSとは言わないのかもしれんな。
この2年間でも単一のIPアドレスのアクセス過多はあったけど、DDoSと言われるような攻撃はいまだに一度もないし。
[root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:35:$a | wc -l ; a=$(($a+5)) ; done 3 4 1 3 2 0 4 4 0 2 0 [root@takechan takeken]# [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:36:$a | wc -l ; a=$(($a+5)) ; done 4 3 4 0 6 2 5 4 5 3 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:37:$a | wc -l ; a=$(($a+5)) ; done 6 8 3 4 1 3 7 2 10 4 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:38:$a | wc -l ; a=$(($a+5)) ; done 5 5 2 8 5 7 7 0 0 1 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:39:$a | wc -l ; a=$(($a+5)) ; done 3 0 2 3 3 2 1 2 3 0 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:40:$a | wc -l ; a=$(($a+5)) ; done 2 1 2 5 3 0 2 1 2 4 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:41:$a | wc -l ; a=$(($a+5)) ; done 3 2 4 6 0 5 2 2 0 4 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:42:$a | wc -l ; a=$(($a+5)) ; done 4 4 3 2 2 1 2 1 6 2 0 [root@takechan takeken]# a=10 ; while [ $a -le "60" ] ; do grep 198.1.153.12 /var/log/httpd/access_log | grep 2014:19:43:$a | wc -l ; a=$(($a+5)) ; done 0 0 0 0 0 0 0 0 0 0 0
閾値の設定ってのは、なかなか難しいものなのだなあ。