在 Mac 上设计流程,成功爬取数据后,尝试在 Linux 服务器上使用 Docker 部署执行(采用多任务的方式)。

作者: JasonChen-ecnu创建于 2024年8月23日更新于 2026年3月6日

The task is with user information, and the data retrieved is written to MySQL. Here is my operation:

  1. Download EasySpider_0.6.2_Linux_x64_with_docker_support.tar.xz, extract it on the server, and get EasySpider_Linux_x64.
  2. Upload the config.json, mysql_config.json, and user_data folders on the Mac to the EasySpider_Linux_x64 folder on the server.
  3. Modify the content of the config.json file on the server: "absolute_user_data_folder": "/export/EasySpider_Linux_x64/user_data" "sys_arch": "amd64"
  4. Upload the file 1.json in the tasks folder on the Mac to the /export/EasySpider_Linux_x64/tasks directory on the server.
  5. Upload the file 55.json in the execution_instances folder on the Mac to the /export/EasySpider_Linux_x64/execution_instances directory on the server.
  6. Run the following commands on the server: sudo Docker network create grid sudo Docker run -d -p 4442-4444:4442-4444 --net grid --name selenium-hub selenium/hub sudo Docker run -d --net grid -e SE_EVENT_BUS_HOST=selenium-hub
    --shm-size="4g"
    -e SE_EVENT_BUS_PUBLISH_PORT=4442
    -e SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    selenium/node-chrome Screenshot 2024-08-23 16 10 19
  7. In the EasySpider_Linux_x64 directory on the server, run the following command: ./EasySpider/resources/app/chrome_linux64/easyspider_executestage --ids [55] --docker_driver http://localhost:4444/wd/hub --user_data 1 --server_address http://localhost:8074 --config_folder "/export/EasySpider_Linux_x64" --headless 1 --read_type local --config_file_name config.json --saved_file_name The error message is as follows: id: 55 local Task Name: 巨量引擎工作台直播列表 Task Name: Giant Engine Workbench Live List File Download path: /export/EasySpider_Linux_x64/Data/Task_55/files Using remote driver Headless mode Traceback (most recent call last): File "easyspider_executestage.py", line 2458, in File "myChrome.py", line 30, in init File "selenium/webdriver/remote/webdriver.py", line 209, in init File "selenium/webdriver/remote/webdriver.py", line 293, in start_session File "selenium/webdriver/remote/webdriver.py", line 348, in execute File "selenium/webdriver/remote/errorhandler.py", line 229, in check_response selenium.common.exceptions.SessionNotCreatedException: Message: Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: session not created from unknown error: cannot create default profile directory Host info: host: '19fb995e6e76', ip: '172.18.0.3' Build info: version: '4.23.1', revision: '656257d8e9' System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-147.el8.x86_64', java.version: '17.0.12' Driver info: driver.version: unknown Build info: version: '4.23.1', revision: '656257d8e9' System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-147.el8.x86_64', java.version: '17.0.12' Driver info: driver.version: unknown Build info: version: '4.23.1', revision: '656257d8e9' System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-147.el8.x86_64', java.version: '17.0.12' Driver info: driver.version: unknown Stacktrace: at org.openqa.selenium.grid.node.remote.RemoteNode.newSession …

内容来源: NaiboWang/EasySpider