fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-10 06:40:56 +08:00
Code Issues Packages Projects Releases Wiki Activity
755 Commits 11 Branches 0 Tags
e5ec29d4ff3f09caeda6492959848e9323f7255c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江-Relakkes e5ec29d4ff Merge pull request #867 from wanteatfruit/feature/xhs-international-support 
feat: 添加海外版小红书(rednote.com)支持
 2026-04-10 17:50:03 +08:00
程序员阿江(Relakkes) 16e8965035 fix: add xhshow dependency 2026-04-07 21:20:44 +08:00
程序员阿江(Relakkes) 699a90f830 fix: xhs creator error 2026-04-07 12:54:39 +08:00
程序员阿江-Relakkes 21b3f90c7d Add GitHub Sponsors FUNDING.yml 2026-04-03 16:07:19 +08:00
程序员阿江(Relakkes) e8b18683a0 update docs 2026-03-24 09:52:30 +08:00
程序员阿江-Relakkes 2b049d05a3 Merge pull request #847 from w21180239/fix/ssl-verify-proxy 
fix: disable SSL verification for proxy/VPN environments
 2026-03-19 00:24:47 +08:00
程序员阿江(Relakkes) 6742cd598b docs: update README.md 2026-03-17 16:49:02 +08:00
程序员阿江(Relakkes) 71168a46f6 fix: 修正 OpenClaw 链接地址为 openclaw.ai 2026-03-10 02:45:38 +08:00
程序员阿江(Relakkes) 6f45b570a7 docs: Pro 功能列表新增 AI Agent Skill 支持 2026-03-10 02:42:31 +08:00
程序员阿江(Relakkes) 0282e626c9 feat: 新增 JSONL 存储格式支持，默认存储格式改为 jsonl 
JSONL（JSON Lines）每行一个 JSON 对象，采用 append 模式写入，
无需读取已有数据，大数据量下性能远优于 JSON 格式。

- 新增 AsyncFileWriter.write_to_jsonl() 核心方法
- 7 个平台新增 JsonlStoreImplement 类并注册到工厂
- 配置默认值从 json 改为 jsonl，CLI/API 枚举同步更新
- db_session.py 守卫条件加入 jsonl，避免误触 ValueError
- 词云生成支持读取 JSONL 文件，优先 jsonl 回退 json
- 原有 json 选项完全保留，向后兼容
- 更新相关文档和测试
 2026-03-03 23:31:07 +08:00
程序员阿江-Relakkes 4331b91fe1 Merge pull request #838 from jznrhnn/main 
feat(msql_model):添加msql表字段注释
 2026-03-01 01:59:46 +08:00
程序员阿江-Relakkes 19d974ccfc Merge pull request #839 from ravenling/fix-zhihu-comment 
fix: 修复zhihu评论爬取分页问题
 2026-03-01 01:53:46 +08:00
程序员阿江-Relakkes 13b6140f22 Merge pull request #831 from ouzhuowei/fix_redis_and_proxy 
适配没有redisKeys和快代理没有账号密码的情况
 2026-02-13 21:18:26 +08:00
程序员阿江(Relakkes) d614ccf247 docs: translate comments and metadata to English 
Update Chinese comments, variable descriptions, and metadata across
multiple configuration and core files to English. This improves
codebase accessibility for international developers. Additionally,
removed the sponsorship section from README files.
 2026-02-12 05:30:11 +08:00
程序员阿江-Relakkes 257743b016 Merge pull request #828 from ouzhuowei/add_save_data_path 
补充代理配置的arp
 2026-02-12 04:47:25 +08:00
程序员阿江-Relakkes dcaa11eeb9 Merge pull request #829 from ouzhuowei/update_sub_comment_error 
处理子评论获取失败导致整个流程中断问题
 2026-02-12 04:46:34 +08:00
程序员阿江-Relakkes 4ad065ce9a Merge pull request #825 from ouzhuowei/add_save_data_path 
新增数据保存路径,默认不指定则保存到data文件夹下
 2026-02-04 18:03:22 +08:00
程序员阿江(Relakkes) c309871485 refactor(xhs): improve login state check logic 2026-02-03 20:49:46 +08:00
程序员阿江(Relakkes) 6625663bde feat: #823 2026-02-03 20:40:15 +08:00
程序员阿江(Relakkes) fb42ab5b60 fix: #826 2026-02-03 20:35:33 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 413b5d9034 docs: fix README heading levels, sync Pro section across languages 
- Fix h3→h2 for standalone sections (交流群组, 赞助商展示, 成为赞助者, 其他) in README.md
- Remove WebUI standalone heading (kept as collapsible only)
- Remove WandouHTTP sponsor from EN/ES versions
- Expand Pro section (remove <details> collapse) in EN/ES to match CN
- Add Content Deconstruction Agent to Pro feature list in EN/ES

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-02-03 00:40:27 +08:00
程序员阿江(Relakkes) dbbc2c7439 docs: update README.md 2026-02-02 20:25:51 +08:00
程序员阿江-Relakkes 51a7d94de8 Merge pull request #821 from wanzirong/feature/max-concurrency-param 
feat: 添加并发爬虫数量控制参数 --max_concurrency_num
 2026-01-31 00:31:15 +08:00
程序员阿江-Relakkes 94553fd818 Merge pull request #817 from wanzirong/dev 
feat: 添加命令行参数控制评论爬取数量
 2026-01-21 16:49:13 +08:00
程序员阿江(Relakkes) be5b786a74 docs: update docs 2026-01-19 12:23:04 +08:00
程序员阿江-Relakkes 04fb716a44 Merge pull request #815 from 2470370075g-ux/fix-typo 
修复拼写错误
 2026-01-18 22:24:57 +08:00
程序员阿江-Relakkes 00a9e19139 Merge pull request #809 from orbisai0security/fix-cve-2023-50447-requirements.txt 
[Security] Fix CRITICAL vulnerability: CVE-2023-50447
 2026-01-13 14:40:23 +08:00
程序员阿江(Relakkes) 4de2a325a9 feat: ks comment api upgrade to v2 2026-01-09 21:09:39 +08:00
程序员阿江-Relakkes 2517e51ed4 Merge pull request #805 from MissMyDearBear/feature-bear 
fix the login status error after scan the QR code
 2026-01-09 14:18:16 +08:00
程序员阿江-Relakkes 7c240747b6 Merge pull request #807 from DoiiarX/main 
feat(database): add PostgreSQL support and fix Windows subprocess encoding
 2026-01-09 10:53:57 +08:00
程序员阿江(Relakkes) 57b688fea4 feat: webui support light theme 2026-01-06 11:16:48 +08:00
程序员阿江(Relakkes) ee4539c8fa chore: stop tracking .DS_Store 2026-01-06 11:11:49 +08:00
程序员阿江(Relakkes) c895f53e22 fix: #803 2026-01-05 22:29:34 +08:00
程序员阿江(Relakkes) 99db95c499 fix: 'utf-8' codec can't decode error 2026-01-04 10:48:15 +08:00
程序员阿江-Relakkes 483c5ec8c6 Merge pull request #802 from Cae1anSou/fix/douyin-concurrent-comments 
fix: fetch Douyin comments concurrently after each page instead of waiting for all pages
 2026-01-03 22:38:26 +08:00
程序员阿江(Relakkes) a47c119303 docs: update 2025-12-30 17:10:13 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 157ddfb21b i18n: translate all Chinese comments, docstrings, and logger messages to English 
Comprehensive translation of Chinese text to English across the entire codebase:

- api/: FastAPI server documentation and logger messages
- cache/: Cache abstraction layer comments and docstrings
- database/: Database models and MongoDB store documentation
- media_platform/: All platform crawlers (Bilibili, Douyin, Kuaishou, Tieba, Weibo, Xiaohongshu, Zhihu)
- model/: Data model documentation
- proxy/: Proxy pool and provider documentation
- store/: Data storage layer comments
- tools/: Utility functions and browser automation
- test/: Test file documentation

Preserved: Chinese disclaimer header (lines 10-18) for legal compliance

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-26 23:27:19 +08:00
程序员阿江(Relakkes) 1544d13dd5 docs: update README.md 2025-12-26 22:41:32 +08:00
程序员阿江(Relakkes) 55d8c7783f feat: webo full context support 2025-12-26 19:22:24 +08:00
程序员阿江(Relakkes) ff1b681311 fix: weibo get note image fixed 2025-12-26 00:47:20 +08:00
程序员阿江(Relakkes) 11500ef57a fix: #799 2025-12-24 11:45:07 +08:00
程序员阿江(Relakkes) b9663c6a6d fix: #798 2025-12-22 17:44:35 +08:00
程序员阿江(Relakkes) 1a38ae12bd docs: update README.md 2025-12-19 00:23:55 +08:00
程序员阿江(Relakkes) 4ceb94f9c8 docs: webui 支持文档 2025-12-19 00:15:53 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 508675a251 feat(api): add WebUI API server with built frontend 
- Add FastAPI server with WebSocket support for real-time logs
- Add crawler management API endpoints (start/stop/status)
- Add data browsing API endpoints (list files, preview, download)
- Include pre-built WebUI assets for serving frontend

API endpoints:
- POST /api/crawler/start - Start crawler task
- POST /api/crawler/stop - Stop crawler task
- GET /api/crawler/status - Get crawler status
- WS /api/ws/logs - Real-time log streaming
- GET /api/data/files - List data files
- GET /api/data/stats - Get data statistics

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-19 00:02:08 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 eb66e57f60 feat(cmd): add --headless, --specified_id, --creator_id CLI options 
- Add --headless option to control headless mode for Playwright and CDP
- Add --specified_id option for detail mode video/post IDs (comma-separated)
- Add --creator_id option for creator mode IDs (comma-separated)
- Auto-configure platform-specific ID lists (XHS, Bilibili, Douyin, Weibo, Kuaishou)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 23:59:14 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 a8930555ac style: increase aside width and ad image size 
- 增加右侧 aside 宽度从 256px 到 300px
- 增加广告图片宽度从 200px 到 280px

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:29:36 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 fb66ef016d docs: add vitepress-plugin-mermaid for Mermaid diagram rendering 
- 添加 vitepress-plugin-mermaid 和 mermaid 依赖
- 更新 VitePress 配置以支持 Mermaid 图表渲染
- 在 sidebar 中添加项目架构文档链接

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:25:21 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 26c511e35f docs: add project architecture documentation with Mermaid diagrams 
添加项目架构文档，包含：
- 系统架构总览图
- 数据流向图
- 爬虫基类体系和生命周期图
- 存储层架构图
- 代理、登录、缓存系统图
- 模块依赖关系图

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:16:32 +08:00
程序员阿江(Relakkes) 08fcf68b98 docs: update README.md 2025-12-17 12:12:29 +08:00
程序员阿江(Relakkes) 2426095123 docs: update README.md 2025-12-17 11:04:26 +08:00
程序员阿江(Relakkes) 3c75d4f1d0 docs: update docs style 2025-12-16 14:49:14 +08:00
程序员阿江(Relakkes) 332a07ce62 docs: update docs 2025-12-16 14:41:28 +08:00
程序员阿江(Relakkes) 8a0fd49b96 refactor: 抽离应用 runner 并优化退出清理 
- 新增 tools/app_runner.py 统一信号/取消/清理超时逻辑
- main.py 精简为业务入口与资源清理实现
- CDPBrowserManager 不再覆盖已有 SIGINT/SIGTERM 处理器
 2025-12-15 18:06:57 +08:00
程序员阿江(Relakkes) 9ade3b3eef chore: use playwright sign xhs and update dependency 2025-12-09 14:47:48 +08:00
程序员阿江-Relakkes 2600c48359 fix: xhs sub comment sign error 
fix: params参数以及路径问题
 2025-12-03 11:02:52 +08:00
程序员阿江-Relakkes 630d4c1614 Merge pull request #789 from NanmiCoder/feature/test_new_rule_01 
docs: update data store
 2025-11-28 22:35:45 +08:00
程序员阿江(Relakkes) f14242c239 docs: update data store 2025-11-28 22:21:20 +08:00
程序员阿江(Relakkes) 29832ded91 chore: add coderowner rules 2025-11-28 22:17:40 +08:00
程序员阿江(Relakkes) 11f2802624 docs: update README.md 2025-11-28 18:16:04 +08:00
程序员阿江-Relakkes ab19494883 Merge pull request #785 from hsparks-codes/feat/update_readme 
docs: Move data storage section to separate guide
 2025-11-28 18:07:56 +08:00
程序员阿江-Relakkes ba64c8ff9c Merge pull request #784 from NanmiCoder/feature/excel-export-and-tests 
feat: excel store with other platform
 2025-11-28 15:15:31 +08:00
程序员阿江-Relakkes ebbf86d67b Merge pull request #783 from hsparks-codes/feature/excel-export-and-tests 
feat: Add Excel export functionality and unit tests
 2025-11-28 15:14:25 +08:00
程序员阿江(Relakkes) 6e858c1a00 feat: excel store with other platform 2025-11-28 15:12:36 +08:00
程序员阿江-Relakkes 31a092c653 Merge pull request #782 from NanmiCoder/fix/xhs-sign-20251127 
feat: xhs sign playwright version
 2025-11-27 11:05:24 +08:00
程序员阿江(Relakkes) f989ce0788 feat: xhs sign playwright version 2025-11-27 10:53:08 +08:00
程序员阿江-Relakkes 15b98fa511 ip proxy expired logic switch 
Fix/proxy 20251125
 2025-11-26 16:05:01 +08:00
程序员阿江(Relakkes) f1e7124654 fix: proxy extract error 2025-11-26 16:01:54 +08:00
程序员阿江(Relakkes) 6eef02d08c feat: ip proxy expired check 2025-11-25 12:39:10 +08:00
程序员阿江(Relakkes) 1da347cbf8 docs: update index.md 2025-11-22 09:12:25 +08:00
程序员阿江(Relakkes) 422cc92dd1 docs: update README 2025-11-22 08:20:09 +08:00
程序员阿江(Relakkes) 13d2302c9c docs: update README 2025-11-18 17:56:55 +08:00
程序员阿江(Relakkes) ff8c92daad chore: add copyright to every file 2025-11-18 12:24:02 +08:00
程序员阿江(Relakkes) 5288bddb42 refactor: weibo search #771 2025-11-17 17:24:47 +08:00
程序员阿江(Relakkes) 6dcfd7e0a5 refactor: weibo login 2025-11-17 17:11:35 +08:00
程序员阿江(Relakkes) e89a6d5781 feat: cdp browser cleanup after crawler done 2025-11-17 12:21:53 +08:00
程序员阿江(Relakkes) a1c5e07df8 fix: xhs sub comment bugfix #769 2025-11-17 11:47:33 +08:00
程序员阿江(Relakkes) b6caa7a85e refactor: add xhs creator params 2025-11-10 21:10:03 +08:00
程序员阿江(Relakkes) 1e3637f238 refactor: update xhs note detail 2025-11-10 18:13:51 +08:00
程序员阿江(Relakkes)andClaude b5dab6d1e8 refactor: 使用 xhshow 替代 playwright 签名方案 
感谢 @Cloxl/xhshow 开源项目

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-10 18:12:45 +08:00
程序员阿江-Relakkes 54f23b8d1c Merge pull request #768 from yangtao210/main 
优化mongodb配置获取逻辑，移动存储基类位置。集成测试
 2025-11-07 05:44:07 -05:00
程序员阿江(Relakkes) 60cbb3e37d fix: weibo container error #568 2025-11-06 19:43:09 +08:00
程序员阿江-Relakkes 05a1782746 Merge pull request #764 from yangtao210/main 
新增存储到mongoDB
 2025-11-06 06:10:49 -05:00
程序员阿江(Relakkes) 45ec4b433a docs: update 2025-11-06 00:08:03 +08:00
程序员阿江(Relakkes) 0074e975dd fix: dy search 2025-11-04 00:14:16 +08:00
程序员阿江(Relakkes) 889fa01466 fix: bili词云图修复 2025-11-02 13:25:31 +08:00
程序员阿江(Relakkes) 3f5925e326 feat: update xhs sign 2025-10-27 19:06:07 +08:00
程序员阿江(Relakkes) ed6e0bfb5f refactor: tieba 改为浏览器获取数据 2025-10-19 17:09:55 +08:00
程序员阿江(Relakkes) 26a261bc09 Merge branch 'feature/config-refactor-20251018' 2025-10-19 15:32:42 +08:00
程序员阿江(Relakkes) 03e384bbe2 refactor: cdp模式下移除stealth注入 2025-10-19 15:32:03 +08:00
程序员阿江-Relakkes 56bf5d226f The configuration file supports URL crawling 
Feature/config refactor 20251018
 2025-10-18 07:42:14 +08:00
程序员阿江(Relakkes) ae7955787c feat: kuaishou support url link 2025-10-18 07:40:10 +08:00
程序员阿江(Relakkes) a9dd08680f feat: xhs support creator url link 2025-10-18 07:20:09 +08:00
程序员阿江(Relakkes) cae707cb2a feat: douyin support url link 2025-10-18 07:00:21 +08:00
程序员阿江(Relakkes) 906c259cc7 feat: bilibili support url link 2025-10-18 06:30:20 +08:00
程序员阿江(Relakkes) 3b6fae8a62 docs: update README.md 2025-10-17 15:30:44 +08:00
程序员阿江-Relakkes a72504a33d Merge pull request #739 from callmeiks/add-tikhub-sponsor 
docs: resize TikHub banner to smaller size
 2025-10-16 16:54:18 +08:00
程序员阿江-Relakkes 1a5dcb6db7 Merge pull request #738 from callmeiks/add-tikhub-sponsor 
docs: add TikHub as sponsor
 2025-10-16 16:41:19 +08:00
程序员阿江(Relakkes) d1f73e811c docs: update README.md 2025-10-12 21:19:11 +08:00
程序员阿江(Relakkes) 2d3e7555c6 docs: update README.md 2025-10-11 16:16:11 +08:00