[BUG] 贴吧指定贴吧ID列表,然后爬取一级评论和二级评论时,获取不到评论。
Author: QlgbhCreated Oct 20, 2025Updated Sep 12, 2026
Labelsbug
问题检查清单
- 我已经仔细阅读了项目使用过程中的常见问题汇总
- 我已经搜索并查看了已关闭的issues
- 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题
问题描述
贴吧指定贴吧ID列表,然后爬取一级评论和二级评论时,获取不到评论的数据。
复现步骤
运行环境
- 操作系统: conda
- Python版本: 3.11
- 是否使用IP代理: 否
- 是否使用VPN翻墙软件:否
- 目标平台(抖音/小红书/微博等): 贴吧
错误日志
(base) D:\PyCharm\MediaCrawler-main>uv run main.py --platform tieba --lt qrcode --type detail
2025-10-20 16:50:04 MediaCrawler INFO (core.py:75) - [BaiduTieBaCrawler] 使用CDP模式启动浏览器
2025-10-20 16:50:09 MediaCrawler INFO (cdp_browser.py:94) - [CDPBrowserManager] 检测到浏览器: Google Chrome (Unknown Version)
2025-10-20 16:50:09 MediaCrawler INFO (cdp_browser.py:97) - [CDPBrowserManager] 浏览器路径: C:\Program Files\Google\Chrome\Application\chrome.exe
2025-10-20 16:50:09 MediaCrawler INFO (cdp_browser.py:137) - [CDPBrowserManager] 用户数据目录: D:\PyCharm\MediaCrawler-main\browser_data\cdp_tieba_user_data_dir
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:154) - [BrowserLauncher] 启动浏览器: C:\Program Files\Google\Chrome\Application\chrome.exe
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:155) - [BrowserLauncher] 调试端口: 9222
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:156) - [BrowserLauncher] 无头模式: False
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:186) - [BrowserLauncher] 等待浏览器在端口 9222 上准备就绪...
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:195) - [BrowserLauncher] 浏览器已在端口 9222 上准备就绪
2025-10-20 16:50:10 MediaCrawler INFO (cdp_browser.py:111) - [CDPBrowserManager] CDP端口 9222 可访问
2025-10-20 16:50:10 httpx INFO (_client.py:1740) - HTTP Request: GET http://localhost:9222/json/version "HTTP/1.1 200 OK"
2025-10-20 16:50:10 MediaCrawler INFO (cdp_browser.py:175) - [CDPBrowserManager] 获取到浏览器WebSocket URL: ws://localhost:9222/devtools/browser/696748be-b579-4df2-a438-41fc9c794aef
2025-10-20 16:50:10 MediaCrawler INFO (cdp_browser.py:194) - [CDPBrowserManager] 正在通过CDP连接到浏览器: ws://localhost:9222/devtools/browser/696748be-b579-4df2-a438-41fc9c794aef
2025-10-20 16:50:11 MediaCrawler INFO (cdp_browser.py:200) - [CDPBrowserManager] 成功连接到浏览器
2025-10-20 16:50:11 MediaCrawler INFO (cdp_browser.py:201) - [CDPBrowserManager] 浏览器上下文数量: 1
2025-10-20 16:50:11 MediaCrawler INFO (cdp_browser.py:226) - [CDPBrowserManager] 使用现有的浏览器上下文
2025-10-20 16:50:11 MediaCrawler INFO (core.py:647) - [TieBaCrawler] CDP浏览器信息: {'version': '116.0.5845.180', 'contexts_count': 1, 'debug_port': 9222, 'is_connected': True}
2025-10-20 16:50:11 MediaCrawler INFO (core.py:485) - [TieBaCrawler] Injecting anti-detection scripts...
2025-10-20 16:50:11 MediaCrawler INFO (core.py:534) - [TieBaCrawler] Anti-detection scripts injected
2025-10-20 16:50:11 MediaCrawler INFO (core.py:401) - [TieBaCrawler] 模拟真实用户访问路径...
2025-10-20 16:50:11 MediaCrawler INFO (core.py:405) - [TieBaCrawler] Step 1: 访问百度首页 https://www.baidu.com/
2025-10-20 16:50:11 MediaCrawler INFO (core.py:409) - [TieBaCrawler] Step 2: 等待 2秒 模拟用户浏览...
2025-10-20 16:50:13 MediaCrawler INFO (core.py:413) - [TieBaCrawler] Step 3: 查找并点击'贴吧'链接...
2025-10-20 16:50:13 MediaCrawler INFO (core.py:428) - [TieBaCrawler] 找到贴吧链接 (selector: a[href="http://tieba.baidu.com/"])
2025-10-20 16:50:13 MediaCrawler INFO (core.py:439) - [TieBaCrawler] Step 4: 点击贴吧链接...
2025-10-20 16:50:13 MediaCrawler INFO (core.py:443) - [TieBaCrawler] 链接target属性: _blank
2025-10-20 16:50:13 MediaCrawler INFO (core.py:447) - [TieBaCrawler] 链接会在新标签页打开,等待新页面...
2025-10-20 16:50:15 MediaCrawler INFO (core.py:461) - [TieBaCrawler] ✅ 已切换到新标签页 (贴吧页面)
2025-10-20 16:50:15 MediaCrawler INFO (core.py:469) - [TieBaCrawler] Step 5: 页面加载完成,等待 2秒...
2025-10-20 16:50:17 MediaCrawler INFO (core.py:473) - [TieBaCrawler] ✅ 成功通过百度首页进入贴吧! 当前URL: https://tieba.baidu.com/
2025-10-20 16:50:17 MediaCrawler INFO (core.py:548) - [TieBaCrawler.create_tieba_client] Begin create tieba API client...
2025-10-20 16:50:17 MediaCrawler INFO (core.py:552) - [TieBaCrawler.create_tieba_client] Extracted User-Agent from browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
2025-10-20 16:50:17 MediaCrawler INFO (client.py:174) - [BaiduTieBaClient.pong] Begin to check tieba login state by cookies...
2025-10-20 16:50:17 MediaCrawler INFO (client.py:190) - [BaiduTieBaClient.pong] Login state verified by cookies (STOKEN: True, PTOKEN: True, BDUSS: True)
2025-10-20 16:50:17 MediaCrawler INFO (core.py:272) - [BaiduTieBaCrawler.get_note_detail] Begin get note detail, note_id: 10125670400
2025-10-20 16:50:17 MediaCrawler INFO (client.py:287) - [BaiduTieBaClient.get_note_by_id] 访问帖子详情页面: https://tieba.baidu.com/p/10125670400
2025-10-20 16:50:20 MediaCrawler INFO (client.py:298) - [BaiduTieBaClient.get_note_by_id] 成功获取帖子详情HTML,长度: 443217
2025-10-20 16:50:22 MediaCrawler INFO (core.py:279) - [TieBaCrawler.get_note_detail_async_task] Sleeping for 2 seconds after fetching note details 10125670400
2025-10-20 16:50:22 MediaCrawler INFO (core.py:272) - [BaiduTieBaCrawler.get_note_detail] Begin get note detail, note_id: 9922347749
2025-10-20 16:50:22 MediaCrawler INFO (client.py:287) - [BaiduTieBaClient.get_note_by_id] 访问帖子详情页面: https://tieba.baidu.com/p/9922347749
2025-10-20 16:50:27 MediaCrawler INFO (client.py:298) - [BaiduTieBaClient.get_note_by_id] 成功获取帖子详情HTML,长度: 311936
2025-10-20 16:50:29 MediaCrawler INFO (core.py:279) - [TieBaCrawler.get_note_detail_async_task] Sleeping for 2 seconds after fetching note details 9922347749
2025-10-20 16:50:29 MediaCrawler INFO (__init__.py:65) - [store.tieba.update_tieba_note] tieba note: {'note_id': '10125670400', 'title': '武汉大学回应杨某某论文,15岁初中生怎么看?_武汉大学吧_百度贴吧', 'desc': '武汉大学回应杨某某论..我 觉得我们可以注意到一个细节,就是武汉大学在回应杨某某论文的时候,用了“百余处错误”这个词儿,你想如果武汉大学单纯是为了保她,说“多处错误”不就行了吗?为什么一定要用“百余处错误”(敲黑板)这个词儿呢', 'note_url': 'https://tieba.baidu.com/p/10125670400', 'publish_time': '', 'user_link': 'https://tieba.baidu.com/home/main?id=tb.1.50941e3f.I-cgiqszehk0GUQwl5Kmbg&fr=pb&ie=utf-8', 'user_nickname': '峰dH', 'user_avatar': '//himg.bdimg.com/sys/portrait/item/tb.1.50941e3f.I-cgiqszehk0GUQwl5Kmbg', 'tieba_name': '武汉大学吧', 'tieba_link': 'https://tieba.baidu.com/f?kw=%E6%AD%A6%E6%B1%89%E5%A4%A7%E5%AD%A6&ie=utf-8', 'total_replay_num': 131, 'total_replay_page': 1, 'ip_location': '', 'source_keyword': '', 'last_modify_ts': 1760950229319}
2025-10-20 16:50:29 MediaCrawler INFO (__init__.py:65) - [store.tieba.update_tieba_note] tieba note: {'note_id': '9922347749', 'title': '武大学生都干了吗', 'desc': '武大学生都干了吗..首先明确的是杨就是个cs,学术不端品行恶劣,败诉了还要对男方穷追猛打的cs,也希望学校尽快拿出一个态度出来,本人也是在直通车上反映了这件事,但是本科生人微言轻能做的实在有限。鼠鼠只是一个最最最普通的学', 'note_url': 'https://tieba.baidu.com/p/9922347749', 'publish_time': '', 'user_link': 'https://tieba.baidu.com/home/main?id=tb.1.1b45acdf.VKojzXhcYk2N_c-Cp6opyw&fr=pb&ie=utf-8', 'user_nickname': 'szczs', 'user_avatar': '//himg.bdimg.com/sys/portrait/item/tb.1.1b45acdf.VKojzXhcYk2N_c-Cp6opyw', 'tieba_name': '武汉大学吧', 'tieba_link': 'https://tieba.baidu.com/f?kw=%E6%AD%A6%E6%B1%89%E5%A4%A7%E5%AD%A6&ie=utf-8', 'total_replay_num': 19, 'total_replay_page': 1, 'ip_location': '', 'source_keyword': '', 'last_modify_ts': 1760950229324}
2025-10-20 16:50:29 MediaCrawler INFO (core.py:333) - [BaiduTieBaCrawler.get_comments] Begin get note id comments 10125670400
2025-10-20 16:50:31 MediaCrawler INFO (core.py:339) - [TieBaCrawler.get_comments_async_task] Sleeping for 2 seconds before fetching comments for note 10125670400
2025-10-20 16:50:31 MediaCrawler INFO (client.py:335) - [BaiduTieBaClient.get_note_all_comments] 访问评论页面: https://tieba.baidu.com/p/10125670400?pn=1
2025-10-20 16:50:35 MediaCrawler INFO (client.py:353) - [BaiduTieBaClient.get_note_all_comments] 第1页没有评论,停止爬取
2025-10-20 16:50:35 MediaCrawler INFO (client.py:377) - [BaiduTieBaClient.get_note_all_comments] 共获取 0 条一级评论
2025-10-20 16:50:35 MediaCrawler INFO (core.py:333) - [BaiduTieBaCrawler.get_comments] Begin get note id comments 9922347749
2025-10-20 16:50:37 MediaCrawler INFO (core.py:339) - [TieBaCrawler.get_comments_async_task] Sleeping for 2 seconds before fetching comments for note 9922347749
2025-10-20 16:50:37 MediaCrawler INFO (client.py:335) - [BaiduTieBaClient.get_note_all_comments] 访问评论页面: https://tieba.baidu.com/p/9922347749?pn=1
2025-10-20 16:50:40 MediaCrawler INFO (client.py:353) - [BaiduTieBaClient.get_note_all_comments] 第1页没有评论,停止爬取
2025-10-20 16:50:40 MediaCrawler INFO (client.py:377) - [BaiduTieBaClient.get_note_all_comments] 共获取 0 条一级评论
2025-10-20 16:50:40 MediaCrawler INFO (core.py:133) - [BaiduTieBaCrawler.start] Tieba Crawler finished ...
在此粘贴错误日志
错误截图
Source: NanmiCoder/MediaCrawler