fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-08 14:58:11 +08:00
Code Issues Packages Projects Releases Wiki Activity
682 Commits 11 Branches 0 Tags
fb66ef016dd16692dcef8ea71c0dd61ebcd27549
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江(Relakkes)andClaude Opus 4.5 fb66ef016d docs: add vitepress-plugin-mermaid for Mermaid diagram rendering 
- 添加 vitepress-plugin-mermaid 和 mermaid 依赖
- 更新 VitePress 配置以支持 Mermaid 图表渲染
- 在 sidebar 中添加项目架构文档链接

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:25:21 +08:00
程序员阿江(Relakkes)andClaude Opus 4.5 26c511e35f docs: add project architecture documentation with Mermaid diagrams 
添加项目架构文档，包含：
- 系统架构总览图
- 数据流向图
- 爬虫基类体系和生命周期图
- 存储层架构图
- 代理、登录、缓存系统图
- 模块依赖关系图

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2025-12-18 13:16:32 +08:00
程序员阿江(Relakkes) 08fcf68b98 docs: update README.md 2025-12-17 12:12:29 +08:00
程序员阿江(Relakkes) 2426095123 docs: update README.md 2025-12-17 11:04:26 +08:00
程序员阿江(Relakkes) 3c75d4f1d0 docs: update docs style 2025-12-16 14:49:14 +08:00
程序员阿江(Relakkes) 332a07ce62 docs: update docs 2025-12-16 14:41:28 +08:00
程序员阿江(Relakkes) 8a0fd49b96 refactor: 抽离应用 runner 并优化退出清理 
- 新增 tools/app_runner.py 统一信号/取消/清理超时逻辑
- main.py 精简为业务入口与资源清理实现
- CDPBrowserManager 不再覆盖已有 SIGINT/SIGTERM 处理器
 2025-12-15 18:06:57 +08:00
程序员阿江(Relakkes) 9ade3b3eef chore: use playwright sign xhs and update dependency 2025-12-09 14:47:48 +08:00
程序员阿江-Relakkes 2600c48359 fix: xhs sub comment sign error 
fix: params参数以及路径问题
 2025-12-03 11:02:52 +08:00
程序员阿江-Relakkes 630d4c1614 Merge pull request #789 from NanmiCoder/feature/test_new_rule_01 
docs: update data store
 2025-11-28 22:35:45 +08:00
程序员阿江(Relakkes) f14242c239 docs: update data store 2025-11-28 22:21:20 +08:00
程序员阿江(Relakkes) 29832ded91 chore: add coderowner rules 2025-11-28 22:17:40 +08:00
程序员阿江(Relakkes) 11f2802624 docs: update README.md 2025-11-28 18:16:04 +08:00
程序员阿江-Relakkes ab19494883 Merge pull request #785 from hsparks-codes/feat/update_readme 
docs: Move data storage section to separate guide
 2025-11-28 18:07:56 +08:00
程序员阿江-Relakkes ba64c8ff9c Merge pull request #784 from NanmiCoder/feature/excel-export-and-tests 
feat: excel store with other platform
 2025-11-28 15:15:31 +08:00
程序员阿江-Relakkes ebbf86d67b Merge pull request #783 from hsparks-codes/feature/excel-export-and-tests 
feat: Add Excel export functionality and unit tests
 2025-11-28 15:14:25 +08:00
程序员阿江(Relakkes) 6e858c1a00 feat: excel store with other platform 2025-11-28 15:12:36 +08:00
程序员阿江-Relakkes 31a092c653 Merge pull request #782 from NanmiCoder/fix/xhs-sign-20251127 
feat: xhs sign playwright version
 2025-11-27 11:05:24 +08:00
程序员阿江(Relakkes) f989ce0788 feat: xhs sign playwright version 2025-11-27 10:53:08 +08:00
程序员阿江-Relakkes 15b98fa511 ip proxy expired logic switch 
Fix/proxy 20251125
 2025-11-26 16:05:01 +08:00
程序员阿江(Relakkes) f1e7124654 fix: proxy extract error 2025-11-26 16:01:54 +08:00
程序员阿江(Relakkes) 6eef02d08c feat: ip proxy expired check 2025-11-25 12:39:10 +08:00
程序员阿江(Relakkes) 1da347cbf8 docs: update index.md 2025-11-22 09:12:25 +08:00
程序员阿江(Relakkes) 422cc92dd1 docs: update README 2025-11-22 08:20:09 +08:00
程序员阿江(Relakkes) 13d2302c9c docs: update README 2025-11-18 17:56:55 +08:00
程序员阿江(Relakkes) ff8c92daad chore: add copyright to every file 2025-11-18 12:24:02 +08:00
程序员阿江(Relakkes) 5288bddb42 refactor: weibo search #771 2025-11-17 17:24:47 +08:00
程序员阿江(Relakkes) 6dcfd7e0a5 refactor: weibo login 2025-11-17 17:11:35 +08:00
程序员阿江(Relakkes) e89a6d5781 feat: cdp browser cleanup after crawler done 2025-11-17 12:21:53 +08:00
程序员阿江(Relakkes) a1c5e07df8 fix: xhs sub comment bugfix #769 2025-11-17 11:47:33 +08:00
程序员阿江(Relakkes) b6caa7a85e refactor: add xhs creator params 2025-11-10 21:10:03 +08:00
程序员阿江(Relakkes) 1e3637f238 refactor: update xhs note detail 2025-11-10 18:13:51 +08:00
程序员阿江(Relakkes)andClaude b5dab6d1e8 refactor: 使用 xhshow 替代 playwright 签名方案 
感谢 @Cloxl/xhshow 开源项目

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-10 18:12:45 +08:00
程序员阿江-Relakkes 54f23b8d1c Merge pull request #768 from yangtao210/main 
优化mongodb配置获取逻辑，移动存储基类位置。集成测试
 2025-11-07 05:44:07 -05:00
程序员阿江(Relakkes) 60cbb3e37d fix: weibo container error #568 2025-11-06 19:43:09 +08:00
程序员阿江-Relakkes 05a1782746 Merge pull request #764 from yangtao210/main 
新增存储到mongoDB
 2025-11-06 06:10:49 -05:00
程序员阿江(Relakkes) 45ec4b433a docs: update 2025-11-06 00:08:03 +08:00
程序员阿江(Relakkes) 0074e975dd fix: dy search 2025-11-04 00:14:16 +08:00
程序员阿江(Relakkes) 889fa01466 fix: bili词云图修复 2025-11-02 13:25:31 +08:00
程序员阿江(Relakkes) 3f5925e326 feat: update xhs sign 2025-10-27 19:06:07 +08:00
程序员阿江(Relakkes) ed6e0bfb5f refactor: tieba 改为浏览器获取数据 2025-10-19 17:09:55 +08:00
程序员阿江(Relakkes) 26a261bc09 Merge branch 'feature/config-refactor-20251018' 2025-10-19 15:32:42 +08:00
程序员阿江(Relakkes) 03e384bbe2 refactor: cdp模式下移除stealth注入 2025-10-19 15:32:03 +08:00
程序员阿江-Relakkes 56bf5d226f The configuration file supports URL crawling 
Feature/config refactor 20251018
 2025-10-18 07:42:14 +08:00
程序员阿江(Relakkes) ae7955787c feat: kuaishou support url link 2025-10-18 07:40:10 +08:00
程序员阿江(Relakkes) a9dd08680f feat: xhs support creator url link 2025-10-18 07:20:09 +08:00
程序员阿江(Relakkes) cae707cb2a feat: douyin support url link 2025-10-18 07:00:21 +08:00
程序员阿江(Relakkes) 906c259cc7 feat: bilibili support url link 2025-10-18 06:30:20 +08:00
程序员阿江(Relakkes) 3b6fae8a62 docs: update README.md 2025-10-17 15:30:44 +08:00
程序员阿江-Relakkes a72504a33d Merge pull request #739 from callmeiks/add-tikhub-sponsor 
docs: resize TikHub banner to smaller size
 2025-10-16 16:54:18 +08:00
程序员阿江-Relakkes 1a5dcb6db7 Merge pull request #738 from callmeiks/add-tikhub-sponsor 
docs: add TikHub as sponsor
 2025-10-16 16:41:19 +08:00
程序员阿江(Relakkes) d1f73e811c docs: update README.md 2025-10-12 21:19:11 +08:00
程序员阿江(Relakkes) 2d3e7555c6 docs: update README.md 2025-10-11 16:16:11 +08:00
程序员阿江(Relakkes) 3c5b9e8035 docs: update wechat qrcode 2025-10-02 14:27:10 +08:00
程序员阿江(Relakkes) e6f3182ed7 Merge branch 'codex/replace-argparse-with-typer-for-cli' 2025-09-26 18:11:02 +08:00
程序员阿江(Relakkes) 2cf143cc7c fix: #730 2025-09-26 18:10:30 +08:00
程序员阿江-Relakkes eb625b0b48 Merge pull request #729 from NanmiCoder/codex/replace-argparse-with-typer-for-cli 
feat(cli): migrate CLI argument parsing to Typer
 2025-09-26 18:08:21 +08:00
程序员阿江(Relakkes) 84f6f650f8 fix: typer args bugfix 2025-09-26 18:07:57 +08:00
程序员阿江-Relakkes 9d6cf065e9 fix(cli): support runtime without peps604 2025-09-26 17:38:50 +08:00
程序员阿江-Relakkes 95c740dee2 refine: harden typer cli defaults 2025-09-26 17:38:44 +08:00
程序员阿江-Relakkes f97e0c18cd feat(cli): migrate argument parsing to typer 2025-09-26 17:21:47 +08:00
程序员阿江-Relakkes 879a72ea30 fix: 修复cdp启动的浏览器无法关闭的bug 
Improve BrowserLauncher shutdown reliability
 2025-09-26 16:57:48 +08:00
程序员阿江-Relakkes 3237073a0e Improve BrowserLauncher cleanup handling 2025-09-26 16:52:38 +08:00
程序员阿江-Relakkes 7b9db2f748 Merge pull request #726 from LePao1/main 
feat(bilibili)：增加视频清晰度参数，可以通过`BILI_QN`更改下载的视频清晰度；
 2025-09-25 01:58:24 +08:00
程序员阿江(Relakkes) e2554288e0 docs: update README 2025-09-23 09:46:44 +08:00
程序员阿江-Relakkes 1342797486 Merge pull request #718 from persist-1/refactor 
fix(store): 修复'crawler_type_var'的不当使用导致csv/json保存文件名异常的bug
 2025-09-11 06:45:26 +08:00
程序员阿江-Relakkes 4b346cfb61 Merge pull request #716 from persist-1/refactor 
Refactor: 使用 SQLAlchemy ORM 全面重构数据库层
 2025-09-10 14:38:39 +08:00
程序员阿江-Relakkes bf7a0098bd Merge pull request #717 from wisty/patch-1 
log client modify
 2025-09-09 17:14:37 +08:00
程序员阿江(Relakkes) fa5f07e9ee docs: update README.md 2025-09-05 17:51:36 +08:00
程序员阿江(Relakkes) 6b6fedd031 fix: #711 2025-09-02 18:57:18 +08:00
程序员阿江(Relakkes) 2bce3593f7 feat: support time deplay for all platform 2025-09-02 16:43:09 +08:00
程序员阿江(Relakkes) eb799e1fa7 refactor: xhs extractor 2025-09-02 14:50:32 +08:00
程序员阿江-Relakkes ce52c58b98 Merge pull request #707 from CzsGit/fix-douyin-json-format 
fix: 为抖音JSON存储添加格式化输出
 2025-08-18 19:15:50 +08:00
程序员阿江(Relakkes) 9e8c979164 fix: note_download_url field length error 2025-08-14 14:57:24 +08:00
程序员阿江(Relakkes) 4a68e79ed0 docs: update README.md 2025-08-12 22:25:21 +08:00
程序员阿江-Relakkes 526c37822b Merge pull request #700 from 2513502304/main 
将捕获异常的类型由HTTPStatusError换成基类HTTPError，以便正确处理爬取媒体资源出现任何错误时，都不会导致爬取评论的中断，详情参见提交记录
 2025-08-06 17:26:29 +08:00
程序员阿江-Relakkes e4e0f659e0 Merge pull request #699 from 2513502304/main 
添加了对媒体资源服务器的异常处理，参见 issue #691
 2025-08-05 16:11:03 +08:00
程序员阿江(Relakkes) b9d30bbabb fix: #693 2025-08-01 15:55:21 +08:00
程序员阿江(Relakkes) 12450759d8 fix: httpx proxy format error 
feat: add a ip proxy provider
 2025-08-01 01:05:11 +08:00
程序员阿江-Relakkes 0024ce6ab4 feat: upgrade httpx version to 0.28.1 
fix issue #689
 2025-07-31 23:19:08 +08:00
程序员阿江-Relakkes a1535289c1 Merge pull request #687 from 2513502304/main 
添加抖音平台爬取短视频以及帖子图片的逻辑，并规范目前实现爬取媒体的抖音、b站、小红书、微博四个平台存储逻辑
 2025-07-30 23:06:35 +08:00
程序员阿江-Relakkes c61ed57a20 fix: 二维码在部分系统无法显示 #685 
fix: #685
 2025-07-30 21:26:41 +08:00
程序员阿江(Relakkes) 417c39de69 docs: add a sponsor 2025-07-30 16:44:10 +08:00
程序员阿江-Relakkes b2d52918ae Merge pull request #684 from 2513502304/main 
添加抖音笔记图片的存储逻辑
 2025-07-30 14:51:46 +08:00
程序员阿江(Relakkes) 8ab1b7ee4c fix: fixed circular import issue 2025-07-30 14:47:11 +08:00
程序员阿江-Relakkes 72a4d194c0 Merge pull request #680 from korruz/main 
refactor: move format_proxy_info to utils and update crawler classes
 2025-07-29 14:21:48 +08:00
程序员阿江-Relakkes 2935841872 Merge pull request #676 from andy9468/main 
add aiosqlite
 2025-07-28 11:33:42 +08:00
程序员阿江-Relakkes dbbc33a0df Merge pull request #674 from persist-1/chore 
chore: 增加--help参数中文显示支持；增加"douyin_aweme"表"music_download_url"字段与功能实现
 2025-07-25 17:23:03 +08:00
程序员阿江(Relakkes) fc06c783f5 fix: fixed xhs req headers 2025-07-23 13:28:58 +08:00
程序员阿江(Relakkes) b41896f4f3 docs: add a sponsor 2025-07-23 13:17:19 +08:00
程序员阿江(Relakkes) a4d9aaa34a refactor: xhs update 2025-07-21 21:26:16 +08:00
程序员阿江(Relakkes) 26a43358cb chore: update config 2025-07-20 14:34:56 +08:00
程序员阿江(Relakkes) 13b00f7a36 refactor: config update 2025-07-18 23:26:52 +08:00
程序员阿江(Relakkes) 122978b35c Merge pull request #652 from gaoxiaobei/dev 
feat(bilibili): Add flexible search modes and fix limit logic
 2025-07-18 21:41:20 +08:00
程序员阿江(Relakkes) 2753e7631e Merge pull request #664 from cfl-chenfangliang/feat-dyCommentBug 
feat: 修复抖音二级评论地理位置缺失问题
 2025-07-18 15:53:33 +08:00
程序员阿江(Relakkes) 08c28e6f7b Merge pull request #658 from persist-1/feature/sqlite-support 
增加对本地Sqlite数据库的支持（在不便于使用Mysql服务时也能使用数据库进行相关操作）
 2025-07-16 20:35:58 +08:00
程序员阿江(Relakkes) c5509ab91f fix: update ip proxy valid url for #662 2025-07-16 11:06:58 +08:00
程序员阿江(Relakkes) c795b1316a fix: import error for #663 2025-07-16 10:58:11 +08:00
程序员阿江(Relakkes) 3184e0a8d9 fix: sign data too long error #659 2025-07-14 18:43:17 +08:00
程序员阿江(Relakkes) dd8a3f5db8 docs: add a Sponsor 2025-07-12 23:26:30 +08:00