Centos7 build python3.8.5+scrapy+gerapy
Centos7 build python3.8.5+scrapy+gerapy
环境:CentOS Linux release 7.6.1810 (Core) mini
更新Centos7 yum源
yum -y install epel-release
安装编译环境
yum groupinstall -y "Development tools"
更新bash和openssl漏洞以及基本软件
yum install -y bash openssl* ntp vim wget telnet nscd
更新服务器时间
ntpdate ntp1.aliyun.com
安装python3可能使用的依赖
yum install -y openssl-devel bzip2-devel expat-devel gdbm-devel readline-devel sqlite-devel zlib zlib-devel libffi-devel
yum install -y readline-devel.x86_64 (一定要安装这个包，不然升级完成后会发现删除退格键用不了)
下载最新版本python
cd /opt/
wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz
编译安装python3.8.5
tar zxvf Python-3.8.5.tgz -C /usr/src/
cd /usr/src/Python-3.8.5
#注意：必须指定--with-ssl，否则使用pip3安装包时，访问ssl链接时，报TLS/SSL错误！
./configure --with-ssl --prefix=/usr/local/python38
make && make install
出现以下信息，表示成功安装
Installing collected packages: setuptools, pip
WARNING: The script easy_install-3.8 is installed in '/usr/local/python38/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip3 and pip3.8 are installed in '/usr/local/python38/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-20.1.1 setuptools-47.1.0
将python38添加到系统环境变量中
vim /etc/profile
最后一行添加
export PATH=$PATH:/usr/local/python38/bin
加载环境变量
source /etc/profile
添加软链接
ln -s /usr/local/python38/bin/python3.8 /usr/bin/python38
ln -s /usr/local/python38/bin/pip3.8 /usr/bin/pip38
升级pip
/usr/local/python38/bin/python3.8 -m pip install --upgrade pip
查看python版本
python38 -V
输出以下信息：
Python 3.8.5
安装gerapy
pip38 install gerapy
出现以下信息,表示安装成功
Successfully installed Automat-20.2.0 MarkupSafe-1.1.1 PyDispatcher-2.0.5 PyHamcrest-2.0.2 Twisted-20.3.0 appdirs-1.4.4 apscheduler-3.5.1 attrs-20.1.0 beautifulsoup4-4.9.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 constantly-15.1.0 cryptography-3.1 cssselect-1.1.0 django-1.11.29 django-apscheduler-0.3.0 django-cors-headers-3.2.0 djangorestframework-3.9.2 furl-2.1.0 gerapy-0.9.6 gevent-20.6.2 greenlet-0.4.16 hyperlink-20.0.1 idna-2.10 incremental-17.5.0 jinja2-2.10.1 lxml-4.5.2 orderedmultidict-1.0.1 parsel-1.6.0 protego-0.1.16 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyee-7.0.2 pymongo-3.11.0 pymysql-0.10.0 pyppeteer-0.2.2 pyquery-1.4.1 python-scrapyd-api-2.1.2 pytz-2020.1 queuelib-1.5.0 redis-3.5.3 requests-2.24.0 scrapy-1.8.0 scrapy-redis-0.6.8 scrapy-splash-0.7.2 service-identity-18.1.0 six-1.15.0 soupsieve-2.0.1 tqdm-4.48.2 tzlocal-2.1 urllib3-1.25.10 w3lib-1.22.0 websocket-0.2.1 websockets-8.1 zope.event-4.4 zope.interface-5.1.0
创建工作目录
mkdir /data/gerapy_scrapy
gerapy初始化目录
[root@localhost data]# gerapy init /data/gerapy_scrapy/
Initialized workspace /data/gerapy_scrapy/
gerapy初始化数据库
cd /data/gerapy_scrapy
[root@localhost gerapy_scrapy]# gerapy migrate
Operations to perform:
Apply all migrations: admin, auth, authtoken, contenttypes, core, django_apscheduler, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying authtoken.0001_initial... OK
Applying authtoken.0002_auto_20160226_1747... OK
Applying core.0001_initial... OK
Applying core.0002_auto_20180119_1210... OK
Applying core.0003_auto_20180123_2304... OK
Applying core.0004_auto_20180124_0032... OK
Applying core.0005_auto_20180131_1210... OK
Applying core.0006_auto_20180131_1235... OK
Applying core.0007_task_trigger... OK
Applying core.0008_auto_20180703_2305... OK
Applying core.0009_auto_20180711_2332... OK
Applying core.0010_auto_20191027_2040... OK
Applying django_apscheduler.0001_initial... OK
Applying django_apscheduler.0002_auto_20180412_0758... OK
Applying sessions.0001_initial... OK
创建管理员用户,密码需要符合复杂性
[root@localhost gerapy_scrapy]# gerapy createsuperuser
Username (leave blank to use 'root'): admin
Email address: admin@qq.com
Password:
Password (again):
Superuser created successfully.
查看selinux状态
[root@localhost gerapy_scrapy]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31
关闭selinux 重启生效
vim /etc/selinux/config
SELINUX=disabled
临时生效
setenforce 0
为了方便直接清空iptables策略
iptables -F
运行gerapy
gerapy runserver 0.0.0.0:8000
访问gerapy:ip地址+端口
http://192.168.18.131:8000/#/login
django后台登录url(可以更改账号密码)
http://192.168.18.131:8000/admin/login/
