fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-03 12:48:11 +08:00
Code Issues Packages Projects Releases Wiki Activity
717 Commits 11 Branches 0 Tags
51a7d94de85f69112ba722ab52679612f5b61e6b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江-Relakkes 51a7d94de8 Merge pull request #821 from wanzirong/feature/max-concurrency-param 
feat: 添加并发爬虫数量控制参数 --max_concurrency_num
 2026-01-31 00:31:15 +08:00
程序员阿江-Relakkes 94553fd818 Merge pull request #817 from wanzirong/dev 
feat: 添加命令行参数控制评论爬取数量
 2026-01-21 16:49:13 +08:00
程序员阿江(Relakkes) be5b786a74 docs: update docs 2026-01-19 12:23:04 +08:00
程序员阿江-Relakkes 04fb716a44 Merge pull request #815 from 2470370075g-ux/fix-typo 
修复拼写错误
 2026-01-18 22:24:57 +08:00
程序员阿江-Relakkes 00a9e19139 Merge pull request #809 from orbisai0security/fix-cve-2023-50447-requirements.txt 
[Security] Fix CRITICAL vulnerability: CVE-2023-50447
 2026-01-13 14:40:23 +08:00
程序员阿江(Relakkes) 4de2a325a9 feat: ks comment api upgrade to v2 2026-01-09 21:09:39 +08:00
程序员阿江-Relakkes 2517e51ed4 Merge pull request #805 from MissMyDearBear/feature-bear 
fix the login status error after scan the QR code
 2026-01-09 14:18:16 +08:00
程序员阿江-Relakkes 7c240747b6 Merge pull request #807 from DoiiarX/main 
feat(database): add PostgreSQL support and fix Windows subprocess encoding
 2026-01-09 10:53:57 +08:00
程序员阿江(Relakkes) 57b688fea4 feat: webui support light theme 2026-01-06 11:16:48 +08:00
程序员阿江(Relakkes) ee4539c8fa chore: stop tracking .DS_Store 2026-01-06 11:11:49 +08:00
程序员阿江(Relakkes) c895f53e22 fix: #803 2026-01-05 22:29:34 +08:00
程序员阿江(Relakkes) 99db95c499 fix: 'utf-8' codec can't decode error 2026-01-04 10:48:15 +08:00
程序员阿江-Relakkes 483c5ec8c6 Merge pull request #802 from Cae1anSou/fix/douyin-concurrent-comments 
fix: fetch Douyin comments concurrently after each page instead of waiting for all pages
 2026-01-03 22:38:26 +08:00
程序员阿江(Relakkes) a47c119303 docs: update 2025-12-30 17:10:13 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 157ddfb21b i18n: translate all Chinese comments, docstrings, and logger messages to English 
Comprehensive translation of Chinese text to English across the entire codebase:

- api/: FastAPI server documentation and logger messages
- cache/: Cache abstraction layer comments and docstrings
- database/: Database models and MongoDB store documentation
- media_platform/: All platform crawlers (Bilibili, Douyin, Kuaishou, Tieba, Weibo, Xiaohongshu, Zhihu)
- model/: Data model documentation
- proxy/: Proxy pool and provider documentation
- store/: Data storage layer comments
- tools/: Utility functions and browser automation
- test/: Test file documentation

Preserved: Chinese disclaimer header (lines 10-18) for legal compliance

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-26 23:27:19 +08:00
程序员阿江(Relakkes) 1544d13dd5 docs: update README.md 2025-12-26 22:41:32 +08:00
程序员阿江(Relakkes) 55d8c7783f feat: webo full context support 2025-12-26 19:22:24 +08:00
程序员阿江(Relakkes) ff1b681311 fix: weibo get note image fixed 2025-12-26 00:47:20 +08:00
程序员阿江(Relakkes) 11500ef57a fix: #799 2025-12-24 11:45:07 +08:00
程序员阿江(Relakkes) b9663c6a6d fix: #798 2025-12-22 17:44:35 +08:00
程序员阿江(Relakkes) 1a38ae12bd docs: update README.md 2025-12-19 00:23:55 +08:00
程序员阿江(Relakkes) 4ceb94f9c8 docs: webui 支持文档 2025-12-19 00:15:53 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 508675a251 feat(api): add WebUI API server with built frontend 
- Add FastAPI server with WebSocket support for real-time logs
- Add crawler management API endpoints (start/stop/status)
- Add data browsing API endpoints (list files, preview, download)
- Include pre-built WebUI assets for serving frontend

API endpoints:
- POST /api/crawler/start - Start crawler task
- POST /api/crawler/stop - Stop crawler task
- GET /api/crawler/status - Get crawler status
- WS /api/ws/logs - Real-time log streaming
- GET /api/data/files - List data files
- GET /api/data/stats - Get data statistics

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-19 00:02:08 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 eb66e57f60 feat(cmd): add --headless, --specified_id, --creator_id CLI options 
- Add --headless option to control headless mode for Playwright and CDP
- Add --specified_id option for detail mode video/post IDs (comma-separated)
- Add --creator_id option for creator mode IDs (comma-separated)
- Auto-configure platform-specific ID lists (XHS, Bilibili, Douyin, Weibo, Kuaishou)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 23:59:14 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 a8930555ac style: increase aside width and ad image size 
- 增加右侧 aside 宽度从 256px 到 300px
- 增加广告图片宽度从 200px 到 280px

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:29:36 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 fb66ef016d docs: add vitepress-plugin-mermaid for Mermaid diagram rendering 
- 添加 vitepress-plugin-mermaid 和 mermaid 依赖
- 更新 VitePress 配置以支持 Mermaid 图表渲染
- 在 sidebar 中添加项目架构文档链接

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:25:21 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 26c511e35f docs: add project architecture documentation with Mermaid diagrams 
添加项目架构文档，包含：
- 系统架构总览图
- 数据流向图
- 爬虫基类体系和生命周期图
- 存储层架构图
- 代理、登录、缓存系统图
- 模块依赖关系图

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:16:32 +08:00
程序员阿江(Relakkes) 08fcf68b98 docs: update README.md 2025-12-17 12:12:29 +08:00
程序员阿江(Relakkes) 2426095123 docs: update README.md 2025-12-17 11:04:26 +08:00
程序员阿江(Relakkes) 3c75d4f1d0 docs: update docs style 2025-12-16 14:49:14 +08:00
程序员阿江(Relakkes) 332a07ce62 docs: update docs 2025-12-16 14:41:28 +08:00
程序员阿江(Relakkes) 8a0fd49b96 refactor: 抽离应用 runner 并优化退出清理 
- 新增 tools/app_runner.py 统一信号/取消/清理超时逻辑
- main.py 精简为业务入口与资源清理实现
- CDPBrowserManager 不再覆盖已有 SIGINT/SIGTERM 处理器
 2025-12-15 18:06:57 +08:00
程序员阿江(Relakkes) 9ade3b3eef chore: use playwright sign xhs and update dependency 2025-12-09 14:47:48 +08:00
程序员阿江-Relakkes 2600c48359 fix: xhs sub comment sign error 
fix: params参数以及路径问题
 2025-12-03 11:02:52 +08:00
程序员阿江-Relakkes 630d4c1614 Merge pull request #789 from NanmiCoder/feature/test_new_rule_01 
docs: update data store
 2025-11-28 22:35:45 +08:00
程序员阿江(Relakkes) f14242c239 docs: update data store 2025-11-28 22:21:20 +08:00
程序员阿江(Relakkes) 29832ded91 chore: add coderowner rules 2025-11-28 22:17:40 +08:00
程序员阿江(Relakkes) 11f2802624 docs: update README.md 2025-11-28 18:16:04 +08:00
程序员阿江-Relakkes ab19494883 Merge pull request #785 from hsparks-codes/feat/update_readme 
docs: Move data storage section to separate guide
 2025-11-28 18:07:56 +08:00
程序员阿江-Relakkes ba64c8ff9c Merge pull request #784 from NanmiCoder/feature/excel-export-and-tests 
feat: excel store with other platform
 2025-11-28 15:15:31 +08:00
程序员阿江-Relakkes ebbf86d67b Merge pull request #783 from hsparks-codes/feature/excel-export-and-tests 
feat: Add Excel export functionality and unit tests
 2025-11-28 15:14:25 +08:00
程序员阿江(Relakkes) 6e858c1a00 feat: excel store with other platform 2025-11-28 15:12:36 +08:00
程序员阿江-Relakkes 31a092c653 Merge pull request #782 from NanmiCoder/fix/xhs-sign-20251127 
feat: xhs sign playwright version
 2025-11-27 11:05:24 +08:00
程序员阿江(Relakkes) f989ce0788 feat: xhs sign playwright version 2025-11-27 10:53:08 +08:00
程序员阿江-Relakkes 15b98fa511 ip proxy expired logic switch 
Fix/proxy 20251125
 2025-11-26 16:05:01 +08:00
程序员阿江(Relakkes) f1e7124654 fix: proxy extract error 2025-11-26 16:01:54 +08:00
程序员阿江(Relakkes) 6eef02d08c feat: ip proxy expired check 2025-11-25 12:39:10 +08:00
程序员阿江(Relakkes) 1da347cbf8 docs: update index.md 2025-11-22 09:12:25 +08:00
程序员阿江(Relakkes) 422cc92dd1 docs: update README 2025-11-22 08:20:09 +08:00
程序员阿江(Relakkes) 13d2302c9c docs: update README 2025-11-18 17:56:55 +08:00
程序员阿江(Relakkes) ff8c92daad chore: add copyright to every file 2025-11-18 12:24:02 +08:00
程序员阿江(Relakkes) 5288bddb42 refactor: weibo search #771 2025-11-17 17:24:47 +08:00
程序员阿江(Relakkes) 6dcfd7e0a5 refactor: weibo login 2025-11-17 17:11:35 +08:00
程序员阿江(Relakkes) e89a6d5781 feat: cdp browser cleanup after crawler done 2025-11-17 12:21:53 +08:00
程序员阿江(Relakkes) a1c5e07df8 fix: xhs sub comment bugfix #769 2025-11-17 11:47:33 +08:00
程序员阿江(Relakkes) b6caa7a85e refactor: add xhs creator params 2025-11-10 21:10:03 +08:00
程序员阿江(Relakkes) 1e3637f238 refactor: update xhs note detail 2025-11-10 18:13:51 +08:00
程序员阿江(Relakkes)andClaude b5dab6d1e8 refactor: 使用 xhshow 替代 playwright 签名方案 
感谢 @Cloxl/xhshow 开源项目

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-10 18:12:45 +08:00
程序员阿江-Relakkes 54f23b8d1c Merge pull request #768 from yangtao210/main 
优化mongodb配置获取逻辑，移动存储基类位置。集成测试
 2025-11-07 05:44:07 -05:00
程序员阿江(Relakkes) 60cbb3e37d fix: weibo container error #568 2025-11-06 19:43:09 +08:00
程序员阿江-Relakkes 05a1782746 Merge pull request #764 from yangtao210/main 
新增存储到mongoDB
 2025-11-06 06:10:49 -05:00
程序员阿江(Relakkes) 45ec4b433a docs: update 2025-11-06 00:08:03 +08:00
程序员阿江(Relakkes) 0074e975dd fix: dy search 2025-11-04 00:14:16 +08:00
程序员阿江(Relakkes) 889fa01466 fix: bili词云图修复 2025-11-02 13:25:31 +08:00
程序员阿江(Relakkes) 3f5925e326 feat: update xhs sign 2025-10-27 19:06:07 +08:00
程序员阿江(Relakkes) ed6e0bfb5f refactor: tieba 改为浏览器获取数据 2025-10-19 17:09:55 +08:00
程序员阿江(Relakkes) 26a261bc09 Merge branch 'feature/config-refactor-20251018' 2025-10-19 15:32:42 +08:00
程序员阿江(Relakkes) 03e384bbe2 refactor: cdp模式下移除stealth注入 2025-10-19 15:32:03 +08:00
程序员阿江-Relakkes 56bf5d226f The configuration file supports URL crawling 
Feature/config refactor 20251018
 2025-10-18 07:42:14 +08:00
程序员阿江(Relakkes) ae7955787c feat: kuaishou support url link 2025-10-18 07:40:10 +08:00
程序员阿江(Relakkes) a9dd08680f feat: xhs support creator url link 2025-10-18 07:20:09 +08:00
程序员阿江(Relakkes) cae707cb2a feat: douyin support url link 2025-10-18 07:00:21 +08:00
程序员阿江(Relakkes) 906c259cc7 feat: bilibili support url link 2025-10-18 06:30:20 +08:00
程序员阿江(Relakkes) 3b6fae8a62 docs: update README.md 2025-10-17 15:30:44 +08:00
程序员阿江-Relakkes a72504a33d Merge pull request #739 from callmeiks/add-tikhub-sponsor 
docs: resize TikHub banner to smaller size
 2025-10-16 16:54:18 +08:00
程序员阿江-Relakkes 1a5dcb6db7 Merge pull request #738 from callmeiks/add-tikhub-sponsor 
docs: add TikHub as sponsor
 2025-10-16 16:41:19 +08:00
程序员阿江(Relakkes) d1f73e811c docs: update README.md 2025-10-12 21:19:11 +08:00
程序员阿江(Relakkes) 2d3e7555c6 docs: update README.md 2025-10-11 16:16:11 +08:00
程序员阿江(Relakkes) 3c5b9e8035 docs: update wechat qrcode 2025-10-02 14:27:10 +08:00
程序员阿江(Relakkes) e6f3182ed7 Merge branch 'codex/replace-argparse-with-typer-for-cli' 2025-09-26 18:11:02 +08:00
程序员阿江(Relakkes) 2cf143cc7c fix: #730 2025-09-26 18:10:30 +08:00
程序员阿江-Relakkes eb625b0b48 Merge pull request #729 from NanmiCoder/codex/replace-argparse-with-typer-for-cli 
feat(cli): migrate CLI argument parsing to Typer
 2025-09-26 18:08:21 +08:00
程序员阿江(Relakkes) 84f6f650f8 fix: typer args bugfix 2025-09-26 18:07:57 +08:00
程序员阿江-Relakkes 9d6cf065e9 fix(cli): support runtime without peps604 2025-09-26 17:38:50 +08:00
程序员阿江-Relakkes 95c740dee2 refine: harden typer cli defaults 2025-09-26 17:38:44 +08:00
程序员阿江-Relakkes f97e0c18cd feat(cli): migrate argument parsing to typer 2025-09-26 17:21:47 +08:00
程序员阿江-Relakkes 879a72ea30 fix: 修复cdp启动的浏览器无法关闭的bug 
Improve BrowserLauncher shutdown reliability
 2025-09-26 16:57:48 +08:00
程序员阿江-Relakkes 3237073a0e Improve BrowserLauncher cleanup handling 2025-09-26 16:52:38 +08:00
程序员阿江-Relakkes 7b9db2f748 Merge pull request #726 from LePao1/main 
feat(bilibili)：增加视频清晰度参数，可以通过`BILI_QN`更改下载的视频清晰度；
 2025-09-25 01:58:24 +08:00
程序员阿江(Relakkes) e2554288e0 docs: update README 2025-09-23 09:46:44 +08:00
程序员阿江-Relakkes 1342797486 Merge pull request #718 from persist-1/refactor 
fix(store): 修复'crawler_type_var'的不当使用导致csv/json保存文件名异常的bug
 2025-09-11 06:45:26 +08:00
程序员阿江-Relakkes 4b346cfb61 Merge pull request #716 from persist-1/refactor 
Refactor: 使用 SQLAlchemy ORM 全面重构数据库层
 2025-09-10 14:38:39 +08:00
程序员阿江-Relakkes bf7a0098bd Merge pull request #717 from wisty/patch-1 
log client modify
 2025-09-09 17:14:37 +08:00
程序员阿江(Relakkes) fa5f07e9ee docs: update README.md 2025-09-05 17:51:36 +08:00
程序员阿江(Relakkes) 6b6fedd031 fix: #711 2025-09-02 18:57:18 +08:00
程序员阿江(Relakkes) 2bce3593f7 feat: support time deplay for all platform 2025-09-02 16:43:09 +08:00
程序员阿江(Relakkes) eb799e1fa7 refactor: xhs extractor 2025-09-02 14:50:32 +08:00
程序员阿江-Relakkes ce52c58b98 Merge pull request #707 from CzsGit/fix-douyin-json-format 
fix: 为抖音JSON存储添加格式化输出
 2025-08-18 19:15:50 +08:00
程序员阿江(Relakkes) 9e8c979164 fix: note_download_url field length error 2025-08-14 14:57:24 +08:00
程序员阿江(Relakkes) 4a68e79ed0 docs: update README.md 2025-08-12 22:25:21 +08:00