fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-09 07:58:11 +08:00
Code Issues Packages Projects Releases Wiki Activity
763 Commits 11 Branches 0 Tags
f997befce9a55838c478d7d4f4f005ea438e09c0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江(Relakkes) f328ee35b5 fix: restore Tieba crawling after PC page rewrite 
Tieba search, detail, comments, creator, and forum-list pages now rely on the current signed PC JSON APIs instead of brittle HTML selectors. The CLI also maps Tieba detail and creator arguments into the platform-specific config so command-line runs exercise the intended mode.

Constraint: Tieba PC pages no longer expose stable HTML structures for search, creator, and forum-list extraction
Constraint: Current PC APIs require browser cookies, tbs, and the web client signing convention
Rejected: Keep expanding HTML selectors | search and creator pages returned large documents with empty parsed results after the redesign
Confidence: high
Scope-risk: moderate
Directive: Do not replace these API paths with page HTML parsing without re-verifying the current Tieba network requests
Tested: uv run pytest tests/test_tieba_client_pagination.py tests/test_cmd_arg_tieba.py tests/test_tieba_extractor.py -q
Tested: uv run python -m py_compile cmd_arg/arg.py media_platform/tieba/help.py media_platform/tieba/client.py media_platform/tieba/core.py tests/test_cmd_arg_tieba.py tests/test_tieba_client_pagination.py tests/test_tieba_extractor.py
Tested: uv run main.py --platform tieba --type search --keywords 编程兼职 --get_comment false
Tested: uv run main.py --platform tieba --type detail --specified_id 9835114923 --get_comment true --max_comments_count_singlenotes 3
Tested: uv run main.py --platform tieba --type creator --creator_id https://tieba.baidu.com/home/main?id=tb.1.6ad0cd4a.7ZcjVYWa7UpHttCld2OppA --get_comment false
Not-tested: Second-level Tieba comment API migration; this path still uses the existing /p/comment HTML parser
Not-tested: Full pytest suite has one pre-existing unrelated XHS Excel store assertion failure
 2026-04-30 18:20:46 +08:00
程序员阿江(Relakkes) 1572b64334 docs: 展示 LegionProxy 赞助商 
在中英文 README 的现有赞助商区域补充 LegionProxy，使用压缩后的本地 banner，避免 README 依赖外部图片热链。

Constraint: 赞助商图片来自本地提供的 Canva banner，需要控制仓库体积和 README 展示尺寸
Rejected: 直接提交原始 PNG | 2.6MB 体积过大，不适合作为 README 资源
Confidence: high
Scope-risk: narrow
Tested: 检查 README 引用路径和压缩后图片尺寸
Not-tested: 未进行 README 页面渲染截图验证
 2026-04-25 22:00:43 +08:00
程序员阿江(Relakkes) 0c5f281212 fix: 避免复用浏览器时跨域 Cookie 过长导致请求失败 
连接已有 Chrome 会把整个浏览器上下文的 cookie 带入平台 client。
除 xhs 外，多数平台仍直接读取全量 cookies，导致请求头过长并放大跨域污染。
本次将各平台的 cookie 读取统一收口到平台域名，并补上基础回归测试。

Constraint: 必须继续复用用户真实浏览器里的平台登录态
Rejected: 仅修复 xhs | 其他平台在连接已有浏览器时仍会携带超长 Cookie
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: 后续新增平台或调整 update_cookies 和 create client 流程时，只按平台域名读取 cookies
Tested: uv run pytest test/test_utils.py; python3 -m compileall tools/crawler_util.py media_platform/douyin/core.py media_platform/douyin/client.py media_platform/kuaishou/core.py media_platform/kuaishou/client.py media_platform/bilibili/core.py media_platform/bilibili/client.py media_platform/zhihu/core.py media_platform/zhihu/client.py media_platform/tieba/core.py media_platform/tieba/client.py media_platform/xhs/core.py media_platform/xhs/client.py media_platform/weibo/core.py media_platform/weibo/client.py test/test_utils.py
Not-tested: 各平台在真实 CDP 浏览器连接下的端到端抓取流程
 2026-04-21 13:49:37 +08:00
程序员阿江(Relakkes) 15a20a7983 docs: README 新增打赏作者区块 2026-04-16 14:01:41 +08:00
程序员阿江(Relakkes) 5294b6d9b7 feat: 支持连接用户已有的 Chrome 浏览器进行爬取 
新增 CDP_CONNECT_EXISTING 配置项，默认开启，通过 Chrome 远程调试功能
(chrome://inspect/#remote-debugging) 直接连接用户正在使用的浏览器，
复用真实的 Cookie、扩展和浏览历史，大幅降低平台风控检测风险。

主要变更:
- 新增 _connect_existing_browser 方法，通过 ws:// 直接连接已有浏览器
- 支持等待用户在浏览器端确认连接对话框（60秒超时）
- cleanup 时不关闭用户的浏览器进程
- 修复小红书在真实浏览器下 cookie 过多导致签名失败的问题
- 更新 README、CDP使用指南和常见问题文档
 2026-04-15 10:54:29 +08:00
程序员阿江-Relakkes e5ec29d4ff Merge pull request #867 from wanteatfruit/feature/xhs-international-support 
feat: 添加海外版小红书(rednote.com)支持
 2026-04-10 17:50:03 +08:00
程序员阿江(Relakkes) 16e8965035 fix: add xhshow dependency 2026-04-07 21:20:44 +08:00
程序员阿江(Relakkes) 699a90f830 fix: xhs creator error 2026-04-07 12:54:39 +08:00
程序员阿江-Relakkes 21b3f90c7d Add GitHub Sponsors FUNDING.yml 2026-04-03 16:07:19 +08:00
程序员阿江(Relakkes) e8b18683a0 update docs 2026-03-24 09:52:30 +08:00
程序员阿江-Relakkes 2b049d05a3 Merge pull request #847 from w21180239/fix/ssl-verify-proxy 
fix: disable SSL verification for proxy/VPN environments
 2026-03-19 00:24:47 +08:00
程序员阿江(Relakkes) 6742cd598b docs: update README.md 2026-03-17 16:49:02 +08:00
程序员阿江(Relakkes) 71168a46f6 fix: 修正 OpenClaw 链接地址为 openclaw.ai 2026-03-10 02:45:38 +08:00
程序员阿江(Relakkes) 6f45b570a7 docs: Pro 功能列表新增 AI Agent Skill 支持 2026-03-10 02:42:31 +08:00
程序员阿江(Relakkes) 0282e626c9 feat: 新增 JSONL 存储格式支持，默认存储格式改为 jsonl 
JSONL（JSON Lines）每行一个 JSON 对象，采用 append 模式写入，
无需读取已有数据，大数据量下性能远优于 JSON 格式。

- 新增 AsyncFileWriter.write_to_jsonl() 核心方法
- 7 个平台新增 JsonlStoreImplement 类并注册到工厂
- 配置默认值从 json 改为 jsonl，CLI/API 枚举同步更新
- db_session.py 守卫条件加入 jsonl，避免误触 ValueError
- 词云生成支持读取 JSONL 文件，优先 jsonl 回退 json
- 原有 json 选项完全保留，向后兼容
- 更新相关文档和测试
 2026-03-03 23:31:07 +08:00
程序员阿江-Relakkes 4331b91fe1 Merge pull request #838 from jznrhnn/main 
feat(msql_model):添加msql表字段注释
 2026-03-01 01:59:46 +08:00
程序员阿江-Relakkes 19d974ccfc Merge pull request #839 from ravenling/fix-zhihu-comment 
fix: 修复zhihu评论爬取分页问题
 2026-03-01 01:53:46 +08:00
程序员阿江-Relakkes 13b6140f22 Merge pull request #831 from ouzhuowei/fix_redis_and_proxy 
适配没有redisKeys和快代理没有账号密码的情况
 2026-02-13 21:18:26 +08:00
程序员阿江(Relakkes) d614ccf247 docs: translate comments and metadata to English 
Update Chinese comments, variable descriptions, and metadata across
multiple configuration and core files to English. This improves
codebase accessibility for international developers. Additionally,
removed the sponsorship section from README files.
 2026-02-12 05:30:11 +08:00
程序员阿江-Relakkes 257743b016 Merge pull request #828 from ouzhuowei/add_save_data_path 
补充代理配置的arp
 2026-02-12 04:47:25 +08:00
程序员阿江-Relakkes dcaa11eeb9 Merge pull request #829 from ouzhuowei/update_sub_comment_error 
处理子评论获取失败导致整个流程中断问题
 2026-02-12 04:46:34 +08:00
程序员阿江-Relakkes 4ad065ce9a Merge pull request #825 from ouzhuowei/add_save_data_path 
新增数据保存路径,默认不指定则保存到data文件夹下
 2026-02-04 18:03:22 +08:00
程序员阿江(Relakkes) c309871485 refactor(xhs): improve login state check logic 2026-02-03 20:49:46 +08:00
程序员阿江(Relakkes) 6625663bde feat: #823 2026-02-03 20:40:15 +08:00
程序员阿江(Relakkes) fb42ab5b60 fix: #826 2026-02-03 20:35:33 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 413b5d9034 docs: fix README heading levels, sync Pro section across languages 
- Fix h3→h2 for standalone sections (交流群组, 赞助商展示, 成为赞助者, 其他) in README.md
- Remove WebUI standalone heading (kept as collapsible only)
- Remove WandouHTTP sponsor from EN/ES versions
- Expand Pro section (remove <details> collapse) in EN/ES to match CN
- Add Content Deconstruction Agent to Pro feature list in EN/ES

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-03 00:40:27 +08:00
程序员阿江(Relakkes) dbbc2c7439 docs: update README.md 2026-02-02 20:25:51 +08:00
程序员阿江-Relakkes 51a7d94de8 Merge pull request #821 from wanzirong/feature/max-concurrency-param 
feat: 添加并发爬虫数量控制参数 --max_concurrency_num
 2026-01-31 00:31:15 +08:00
程序员阿江-Relakkes 94553fd818 Merge pull request #817 from wanzirong/dev 
feat: 添加命令行参数控制评论爬取数量
 2026-01-21 16:49:13 +08:00
程序员阿江(Relakkes) be5b786a74 docs: update docs 2026-01-19 12:23:04 +08:00
程序员阿江-Relakkes 04fb716a44 Merge pull request #815 from 2470370075g-ux/fix-typo 
修复拼写错误
 2026-01-18 22:24:57 +08:00
程序员阿江-Relakkes 00a9e19139 Merge pull request #809 from orbisai0security/fix-cve-2023-50447-requirements.txt 
[Security] Fix CRITICAL vulnerability: CVE-2023-50447
 2026-01-13 14:40:23 +08:00
程序员阿江(Relakkes) 4de2a325a9 feat: ks comment api upgrade to v2 2026-01-09 21:09:39 +08:00
程序员阿江-Relakkes 2517e51ed4 Merge pull request #805 from MissMyDearBear/feature-bear 
fix the login status error after scan the QR code
 2026-01-09 14:18:16 +08:00
程序员阿江-Relakkes 7c240747b6 Merge pull request #807 from DoiiarX/main 
feat(database): add PostgreSQL support and fix Windows subprocess encoding
 2026-01-09 10:53:57 +08:00
程序员阿江(Relakkes) 57b688fea4 feat: webui support light theme 2026-01-06 11:16:48 +08:00
程序员阿江(Relakkes) ee4539c8fa chore: stop tracking .DS_Store 2026-01-06 11:11:49 +08:00
程序员阿江(Relakkes) c895f53e22 fix: #803 2026-01-05 22:29:34 +08:00
程序员阿江(Relakkes) 99db95c499 fix: 'utf-8' codec can't decode error 2026-01-04 10:48:15 +08:00
程序员阿江-Relakkes 483c5ec8c6 Merge pull request #802 from Cae1anSou/fix/douyin-concurrent-comments 
fix: fetch Douyin comments concurrently after each page instead of waiting for all pages
 2026-01-03 22:38:26 +08:00
程序员阿江(Relakkes) a47c119303 docs: update 2025-12-30 17:10:13 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 157ddfb21b i18n: translate all Chinese comments, docstrings, and logger messages to English 
Comprehensive translation of Chinese text to English across the entire codebase:

- api/: FastAPI server documentation and logger messages
- cache/: Cache abstraction layer comments and docstrings
- database/: Database models and MongoDB store documentation
- media_platform/: All platform crawlers (Bilibili, Douyin, Kuaishou, Tieba, Weibo, Xiaohongshu, Zhihu)
- model/: Data model documentation
- proxy/: Proxy pool and provider documentation
- store/: Data storage layer comments
- tools/: Utility functions and browser automation
- test/: Test file documentation

Preserved: Chinese disclaimer header (lines 10-18) for legal compliance

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-26 23:27:19 +08:00
程序员阿江(Relakkes) 1544d13dd5 docs: update README.md 2025-12-26 22:41:32 +08:00
程序员阿江(Relakkes) 55d8c7783f feat: webo full context support 2025-12-26 19:22:24 +08:00
程序员阿江(Relakkes) ff1b681311 fix: weibo get note image fixed 2025-12-26 00:47:20 +08:00
程序员阿江(Relakkes) 11500ef57a fix: #799 2025-12-24 11:45:07 +08:00
程序员阿江(Relakkes) b9663c6a6d fix: #798 2025-12-22 17:44:35 +08:00
程序员阿江(Relakkes) 1a38ae12bd docs: update README.md 2025-12-19 00:23:55 +08:00
程序员阿江(Relakkes) 4ceb94f9c8 docs: webui 支持文档 2025-12-19 00:15:53 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 508675a251 feat(api): add WebUI API server with built frontend 
- Add FastAPI server with WebSocket support for real-time logs
- Add crawler management API endpoints (start/stop/status)
- Add data browsing API endpoints (list files, preview, download)
- Include pre-built WebUI assets for serving frontend

API endpoints:
- POST /api/crawler/start - Start crawler task
- POST /api/crawler/stop - Stop crawler task
- GET /api/crawler/status - Get crawler status
- WS /api/ws/logs - Real-time log streaming
- GET /api/data/files - List data files
- GET /api/data/stats - Get data statistics

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-19 00:02:08 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 eb66e57f60 feat(cmd): add --headless, --specified_id, --creator_id CLI options 
- Add --headless option to control headless mode for Playwright and CDP
- Add --specified_id option for detail mode video/post IDs (comma-separated)
- Add --creator_id option for creator mode IDs (comma-separated)
- Auto-configure platform-specific ID lists (XHS, Bilibili, Douyin, Weibo, Kuaishou)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 23:59:14 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 a8930555ac style: increase aside width and ad image size 
- 增加右侧 aside 宽度从 256px 到 300px
- 增加广告图片宽度从 200px 到 280px

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:29:36 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 fb66ef016d docs: add vitepress-plugin-mermaid for Mermaid diagram rendering 
- 添加 vitepress-plugin-mermaid 和 mermaid 依赖
- 更新 VitePress 配置以支持 Mermaid 图表渲染
- 在 sidebar 中添加项目架构文档链接

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:25:21 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 26c511e35f docs: add project architecture documentation with Mermaid diagrams 
添加项目架构文档，包含：
- 系统架构总览图
- 数据流向图
- 爬虫基类体系和生命周期图
- 存储层架构图
- 代理、登录、缓存系统图
- 模块依赖关系图

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:16:32 +08:00
程序员阿江(Relakkes) 08fcf68b98 docs: update README.md 2025-12-17 12:12:29 +08:00
程序员阿江(Relakkes) 2426095123 docs: update README.md 2025-12-17 11:04:26 +08:00
程序员阿江(Relakkes) 3c75d4f1d0 docs: update docs style 2025-12-16 14:49:14 +08:00
程序员阿江(Relakkes) 332a07ce62 docs: update docs 2025-12-16 14:41:28 +08:00
程序员阿江(Relakkes) 8a0fd49b96 refactor: 抽离应用 runner 并优化退出清理 
- 新增 tools/app_runner.py 统一信号/取消/清理超时逻辑
- main.py 精简为业务入口与资源清理实现
- CDPBrowserManager 不再覆盖已有 SIGINT/SIGTERM 处理器
 2025-12-15 18:06:57 +08:00
程序员阿江(Relakkes) 9ade3b3eef chore: use playwright sign xhs and update dependency 2025-12-09 14:47:48 +08:00
程序员阿江-Relakkes 2600c48359 fix: xhs sub comment sign error 
fix: params参数以及路径问题
 2025-12-03 11:02:52 +08:00
程序员阿江-Relakkes 630d4c1614 Merge pull request #789 from NanmiCoder/feature/test_new_rule_01 
docs: update data store
 2025-11-28 22:35:45 +08:00
程序员阿江(Relakkes) f14242c239 docs: update data store 2025-11-28 22:21:20 +08:00
程序员阿江(Relakkes) 29832ded91 chore: add coderowner rules 2025-11-28 22:17:40 +08:00
程序员阿江(Relakkes) 11f2802624 docs: update README.md 2025-11-28 18:16:04 +08:00
程序员阿江-Relakkes ab19494883 Merge pull request #785 from hsparks-codes/feat/update_readme 
docs: Move data storage section to separate guide
 2025-11-28 18:07:56 +08:00
程序员阿江-Relakkes ba64c8ff9c Merge pull request #784 from NanmiCoder/feature/excel-export-and-tests 
feat: excel store with other platform
 2025-11-28 15:15:31 +08:00
程序员阿江-Relakkes ebbf86d67b Merge pull request #783 from hsparks-codes/feature/excel-export-and-tests 
feat: Add Excel export functionality and unit tests
 2025-11-28 15:14:25 +08:00
程序员阿江(Relakkes) 6e858c1a00 feat: excel store with other platform 2025-11-28 15:12:36 +08:00
程序员阿江-Relakkes 31a092c653 Merge pull request #782 from NanmiCoder/fix/xhs-sign-20251127 
feat: xhs sign playwright version
 2025-11-27 11:05:24 +08:00
程序员阿江(Relakkes) f989ce0788 feat: xhs sign playwright version 2025-11-27 10:53:08 +08:00
程序员阿江-Relakkes 15b98fa511 ip proxy expired logic switch 
Fix/proxy 20251125
 2025-11-26 16:05:01 +08:00
程序员阿江(Relakkes) f1e7124654 fix: proxy extract error 2025-11-26 16:01:54 +08:00
程序员阿江(Relakkes) 6eef02d08c feat: ip proxy expired check 2025-11-25 12:39:10 +08:00
程序员阿江(Relakkes) 1da347cbf8 docs: update index.md 2025-11-22 09:12:25 +08:00
程序员阿江(Relakkes) 422cc92dd1 docs: update README 2025-11-22 08:20:09 +08:00
程序员阿江(Relakkes) 13d2302c9c docs: update README 2025-11-18 17:56:55 +08:00
程序员阿江(Relakkes) ff8c92daad chore: add copyright to every file 2025-11-18 12:24:02 +08:00
程序员阿江(Relakkes) 5288bddb42 refactor: weibo search #771 2025-11-17 17:24:47 +08:00
程序员阿江(Relakkes) 6dcfd7e0a5 refactor: weibo login 2025-11-17 17:11:35 +08:00
程序员阿江(Relakkes) e89a6d5781 feat: cdp browser cleanup after crawler done 2025-11-17 12:21:53 +08:00
程序员阿江(Relakkes) a1c5e07df8 fix: xhs sub comment bugfix #769 2025-11-17 11:47:33 +08:00
程序员阿江(Relakkes) b6caa7a85e refactor: add xhs creator params 2025-11-10 21:10:03 +08:00
程序员阿江(Relakkes) 1e3637f238 refactor: update xhs note detail 2025-11-10 18:13:51 +08:00
程序员阿江(Relakkes)andClaude b5dab6d1e8 refactor: 使用 xhshow 替代 playwright 签名方案 
感谢 @Cloxl/xhshow 开源项目

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-10 18:12:45 +08:00
程序员阿江-Relakkes 54f23b8d1c Merge pull request #768 from yangtao210/main 
优化mongodb配置获取逻辑，移动存储基类位置。集成测试
 2025-11-07 05:44:07 -05:00
程序员阿江(Relakkes) 60cbb3e37d fix: weibo container error #568 2025-11-06 19:43:09 +08:00
程序员阿江-Relakkes 05a1782746 Merge pull request #764 from yangtao210/main 
新增存储到mongoDB
 2025-11-06 06:10:49 -05:00
程序员阿江(Relakkes) 45ec4b433a docs: update 2025-11-06 00:08:03 +08:00
程序员阿江(Relakkes) 0074e975dd fix: dy search 2025-11-04 00:14:16 +08:00
程序员阿江(Relakkes) 889fa01466 fix: bili词云图修复 2025-11-02 13:25:31 +08:00
程序员阿江(Relakkes) 3f5925e326 feat: update xhs sign 2025-10-27 19:06:07 +08:00
程序员阿江(Relakkes) ed6e0bfb5f refactor: tieba 改为浏览器获取数据 2025-10-19 17:09:55 +08:00
程序员阿江(Relakkes) 26a261bc09 Merge branch 'feature/config-refactor-20251018' 2025-10-19 15:32:42 +08:00
程序员阿江(Relakkes) 03e384bbe2 refactor: cdp模式下移除stealth注入 2025-10-19 15:32:03 +08:00
程序员阿江-Relakkes 56bf5d226f The configuration file supports URL crawling 
Feature/config refactor 20251018
 2025-10-18 07:42:14 +08:00
程序员阿江(Relakkes) ae7955787c feat: kuaishou support url link 2025-10-18 07:40:10 +08:00
程序员阿江(Relakkes) a9dd08680f feat: xhs support creator url link 2025-10-18 07:20:09 +08:00
程序员阿江(Relakkes) cae707cb2a feat: douyin support url link 2025-10-18 07:00:21 +08:00
程序员阿江(Relakkes) 906c259cc7 feat: bilibili support url link 2025-10-18 06:30:20 +08:00