fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-07 23:18:11 +08:00
Code Issues Packages Projects Releases Wiki Activity
610 Commits 11 Branches 0 Tags
codex/fix-issue-with-mediacrawler-functionality
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江-Relakkes 3237073a0e Improve BrowserLauncher cleanup handling 2025-09-26 16:52:38 +08:00
程序员阿江-Relakkes 7b9db2f748 Merge pull request #726 from LePao1/main 
feat(bilibili)：增加视频清晰度参数，可以通过`BILI_QN`更改下载的视频清晰度；
 2025-09-25 01:58:24 +08:00
程序员阿江(Relakkes) e2554288e0 docs: update README 2025-09-23 09:46:44 +08:00
程序员阿江-Relakkes 1342797486 Merge pull request #718 from persist-1/refactor 
fix(store): 修复'crawler_type_var'的不当使用导致csv/json保存文件名异常的bug
 2025-09-11 06:45:26 +08:00
程序员阿江-Relakkes 4b346cfb61 Merge pull request #716 from persist-1/refactor 
Refactor: 使用 SQLAlchemy ORM 全面重构数据库层
 2025-09-10 14:38:39 +08:00
程序员阿江-Relakkes bf7a0098bd Merge pull request #717 from wisty/patch-1 
log client modify
 2025-09-09 17:14:37 +08:00
程序员阿江(Relakkes) fa5f07e9ee docs: update README.md 2025-09-05 17:51:36 +08:00
程序员阿江(Relakkes) 6b6fedd031 fix: #711 2025-09-02 18:57:18 +08:00
程序员阿江(Relakkes) 2bce3593f7 feat: support time deplay for all platform 2025-09-02 16:43:09 +08:00
程序员阿江(Relakkes) eb799e1fa7 refactor: xhs extractor 2025-09-02 14:50:32 +08:00
程序员阿江-Relakkes ce52c58b98 Merge pull request #707 from CzsGit/fix-douyin-json-format 
fix: 为抖音JSON存储添加格式化输出
 2025-08-18 19:15:50 +08:00
程序员阿江(Relakkes) 9e8c979164 fix: note_download_url field length error 2025-08-14 14:57:24 +08:00
程序员阿江(Relakkes) 4a68e79ed0 docs: update README.md 2025-08-12 22:25:21 +08:00
程序员阿江-Relakkes 526c37822b Merge pull request #700 from 2513502304/main 
将捕获异常的类型由HTTPStatusError换成基类HTTPError，以便正确处理爬取媒体资源出现任何错误时，都不会导致爬取评论的中断，详情参见提交记录
 2025-08-06 17:26:29 +08:00
程序员阿江-Relakkes e4e0f659e0 Merge pull request #699 from 2513502304/main 
添加了对媒体资源服务器的异常处理，参见 issue #691
 2025-08-05 16:11:03 +08:00
程序员阿江(Relakkes) b9d30bbabb fix: #693 2025-08-01 15:55:21 +08:00
程序员阿江(Relakkes) 12450759d8 fix: httpx proxy format error 
feat: add a ip proxy provider
 2025-08-01 01:05:11 +08:00
程序员阿江-Relakkes 0024ce6ab4 feat: upgrade httpx version to 0.28.1 
fix issue #689
 2025-07-31 23:19:08 +08:00
程序员阿江-Relakkes a1535289c1 Merge pull request #687 from 2513502304/main 
添加抖音平台爬取短视频以及帖子图片的逻辑，并规范目前实现爬取媒体的抖音、b站、小红书、微博四个平台存储逻辑
 2025-07-30 23:06:35 +08:00
程序员阿江-Relakkes c61ed57a20 fix: 二维码在部分系统无法显示 #685 
fix: #685
 2025-07-30 21:26:41 +08:00
程序员阿江(Relakkes) 417c39de69 docs: add a sponsor 2025-07-30 16:44:10 +08:00
程序员阿江-Relakkes b2d52918ae Merge pull request #684 from 2513502304/main 
添加抖音笔记图片的存储逻辑
 2025-07-30 14:51:46 +08:00
程序员阿江(Relakkes) 8ab1b7ee4c fix: fixed circular import issue 2025-07-30 14:47:11 +08:00
程序员阿江-Relakkes 72a4d194c0 Merge pull request #680 from korruz/main 
refactor: move format_proxy_info to utils and update crawler classes
 2025-07-29 14:21:48 +08:00
程序员阿江-Relakkes 2935841872 Merge pull request #676 from andy9468/main 
add aiosqlite
 2025-07-28 11:33:42 +08:00
程序员阿江-Relakkes dbbc33a0df Merge pull request #674 from persist-1/chore 
chore: 增加--help参数中文显示支持；增加"douyin_aweme"表"music_download_url"字段与功能实现
 2025-07-25 17:23:03 +08:00
程序员阿江(Relakkes) fc06c783f5 fix: fixed xhs req headers 2025-07-23 13:28:58 +08:00
程序员阿江(Relakkes) b41896f4f3 docs: add a sponsor 2025-07-23 13:17:19 +08:00
程序员阿江(Relakkes) a4d9aaa34a refactor: xhs update 2025-07-21 21:26:16 +08:00
程序员阿江(Relakkes) 26a43358cb chore: update config 2025-07-20 14:34:56 +08:00
程序员阿江(Relakkes) 13b00f7a36 refactor: config update 2025-07-18 23:26:52 +08:00
程序员阿江(Relakkes) 122978b35c Merge pull request #652 from gaoxiaobei/dev 
feat(bilibili): Add flexible search modes and fix limit logic
 2025-07-18 21:41:20 +08:00
程序员阿江(Relakkes) 2753e7631e Merge pull request #664 from cfl-chenfangliang/feat-dyCommentBug 
feat: 修复抖音二级评论地理位置缺失问题
 2025-07-18 15:53:33 +08:00
程序员阿江(Relakkes) 08c28e6f7b Merge pull request #658 from persist-1/feature/sqlite-support 
增加对本地Sqlite数据库的支持（在不便于使用Mysql服务时也能使用数据库进行相关操作）
 2025-07-16 20:35:58 +08:00
程序员阿江(Relakkes) c5509ab91f fix: update ip proxy valid url for #662 2025-07-16 11:06:58 +08:00
程序员阿江(Relakkes) c795b1316a fix: import error for #663 2025-07-16 10:58:11 +08:00
程序员阿江(Relakkes) 3184e0a8d9 fix: sign data too long error #659 2025-07-14 18:43:17 +08:00
程序员阿江(Relakkes) dd8a3f5db8 docs: add a Sponsor 2025-07-12 23:26:30 +08:00
程序员阿江(Relakkes) ec0d29cf0f Merge pull request #642 from cllei12/weibo-search-type 
增加选择微博搜索类型的配置
 2025-07-07 15:05:29 +08:00
程序员阿江(Relakkes) 0d21a27b6e Merge pull request #641 from cllei12/main 
Update playwright version to support Ubuntu 24.04
 2025-07-07 15:03:42 +08:00
程序员阿江(Relakkes) 3cb0e2f91f Merge pull request #637 from Mirza-Samad-Ahmed-Baig/fix/bilibili-creator-videos 
refactor(bilibili): process creator videos in batches
 2025-07-05 00:13:49 +08:00
程序员阿江(Relakkes) 66a68fbb13 docs: update multi language badges size 2025-07-04 14:15:03 +08:00
程序员阿江(Relakkes) 8dcc540797 Merge pull request #635 from Root-FTW/main 
🌐 Add multilingual documentation support (English & Spanish)
 2025-07-04 13:54:31 +08:00
程序员阿江(Relakkes) 848df2b491 feat: other platfrom support the cdp mode 2025-07-03 17:13:32 +08:00
程序员阿江(Relakkes) c892c3324c chore: update uv lock file use aliyun pypi 2025-07-03 16:07:07 +08:00
程序员阿江(Relakkes) 452aafeec8 chore: update uv depends source 2025-07-03 15:27:39 +08:00
程序员阿江(Relakkes) 89359aa259 docs: update README.md 2025-07-02 01:20:44 +08:00
程序员阿江(Relakkes) 0514758cff docs: 更新uv安装文档 2025-06-29 00:07:13 +08:00
程序员阿江(Relakkes) e83b2422d9 feat: 支持playwright通过cdp协议连接本地chrome浏览器
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
docs: 增加uv来管理python依赖的文档
 2025-06-25 23:22:39 +08:00
程序员阿江(Relakkes) fbc9788d54 docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-24 17:44:43 +08:00
Relakkes 71210e3f38 chore: delete unnecessary file
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-20 16:26:02 +08:00
Relakkes fc45b22963 fix: vitepress path problem 2025-06-20 16:10:44 +08:00
Relakkes 22ec34dca3 docs: add list of sponsors 2025-06-20 16:04:04 +08:00
Relakkes fd33813f8f feat: add like_count field to bilibi for issue #623 2025-06-20 15:50:38 +08:00
Relakkes 31bcdb191f docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-16 13:58:09 +08:00
Relakkes d55d8b1efa feat: Douyin supports obtaining video links and cover images. for issue #620
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-14 23:59:08 +08:00
Relakkes ed1dc7916a docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-08 15:56:02 +08:00
程序员阿江(Relakkes) 6323e2d45b Merge pull request #616 from chimeElm/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
修复CRAWLER_MAX_NOTES_COUNT在爬取小红书作者帖子时失效的问题
 2025-06-07 14:43:37 +08:00
Relakkes 23c8f8f87b docs: add english license
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-01 23:20:11 +08:00
Relakkes 1e7b950d3e Revert "chore: remove sponor"
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
This reverts commit 242c06c345.
 2025-05-26 22:35:18 +08:00
Relakkes 242c06c345 chore: remove sponor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-05-25 11:54:38 +08:00
程序员阿江(Relakkes) ff41faeb00 Merge pull request #608 from Bowenwin/bili_expand
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
Bili_function_expand
 2025-05-22 23:14:58 +08:00
程序员阿江(Relakkes) 7ed6621933 Merge pull request #603 from Bowenwin/fix_words
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
Fix words
 2025-05-19 23:16:12 +08:00
Relakkes 654260cbce docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-05-13 18:42:58 +08:00
Relakkes 79a9824f6a fix: modify dy schema
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-04-30 16:47:13 +08:00
Relakkes 67d31bf42a fix: dy update fp params 2025-04-30 13:26:22 +08:00
程序员阿江(Relakkes) 2a41b684ad Merge pull request #590 from 2513502304/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
关于 issue #589 的增强方法
 2025-04-20 14:14:55 +08:00
Relakkes cb2ae6cbab docs: add sponsor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-04-16 16:49:49 +08:00
Relakkes 0d715a9f32 fix: bili qrcode login fix
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-04-08 21:11:40 +08:00
Relakkes 660fd18a95 fix: dy login fix 2025-04-08 20:58:04 +08:00
程序员阿江(Relakkes) afbd4ec1bf Merge pull request #572 from crpa33/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
避免预料之外的数据为None的情况终止进程
 2025-04-02 13:34:41 +08:00
Relakkes dfddfa7fdc docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-23 20:35:11 +08:00
Relakkes daaea7155b feat: add uv tool to manage project 2025-03-23 18:13:13 +08:00
Relakkes 8030d2a02f docs: removed sponsor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-13 15:07:54 +08:00
Relakkes 061d1c15e2 feat: kuaishou search params update
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-11 23:42:34 +08:00
Relakkes f2cf864c27 fix: zhihu article url error #564
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-03 18:18:41 +08:00
Relakkes b43d6b7b91 chore: update config
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-02-12 10:58:48 +08:00
Relakkes 66a7ab1db8 refactor: bibi default to get without time data 2025-02-12 10:58:15 +08:00
Relakkes 678ce1bfac fix: bilibili bugfix
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-02-10 17:13:37 +08:00
Relakkes 457205efd8 docs: add sponsor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-02-08 15:28:18 +08:00
程序员阿江(Relakkes) 38f2b36bf5 Merge pull request #542 from 2513502304/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
Update core.py，为爬取类型为`detail`和`creator`的任务，添加了和`search`任务一样的，用于转存up主信息的`bilibili_store.update_up_info`的函数调用
 2025-01-20 19:30:10 +08:00
程序员阿江(Relakkes) 4b63ea68ec Merge pull request #538 from 2513502304/main 
feat: bilibli support date range filter
 2025-01-17 19:43:57 +08:00
Relakkes 30d0e733d5 feat: douyin adds comment images
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-15 14:50:05 +08:00
程序员阿江(Relakkes) a8e796e690 Merge pull request #536 from 2513502304/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
将微博评论爬取函数`get_note_all_comments`的`max_id_type`便为可变请求参数
 2025-01-13 13:58:15 +08:00
Relakkes 5d3e12169e fix: move dirctory
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-10 16:53:20 +08:00
Relakkes b89a8e5dda docs: add github issue template 2025-01-10 16:51:30 +08:00
Relakkes 792d3bbc34 docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-05 19:02:10 +08:00
Relakkes 5f7c8aea5b docs: 添加11-12月份捐赠者名单
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-02 20:37:28 +08:00
Relakkes fbbead814a fix: 贴吧创作者bug修复 2025-01-02 20:29:05 +08:00
程序员阿江(Relakkes) 7ce1273386 Merge pull request #528 from HuiGitH/hui/fix-xhs-gender
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
fix xhs gender
 2024-12-28 21:01:58 +08:00
Relakkes 79bf9fc05d chore: add xhs field comment for issue #526
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2024-12-26 18:28:23 +08:00
Relakkes 29ab6feded fix: 修复数据库密码中有特殊符号导致db连接不上的bug 2024-12-26 18:13:13 +08:00
Relakkes ea5223c708 feat: 知乎支持详情模式
Deploy VitePress site to Pages / build (push) Waiting to run
Deploy VitePress site to Pages / Deploy (push) Blocked by required conditions
2024-12-26 17:36:33 +08:00
程序员阿江(Relakkes) dc9116e098 Merge pull request #513 from liudongkai/feature_xhs 
feat: xhs 非代理模式下增加随机等待间隔, db存储模式下增加存储xsec_token字段
 2024-12-06 13:01:16 +08:00
程序员阿江(Relakkes) 9c7e1d499b Merge pull request #509 from leantli/feat/xhs_comments_upgrade 
feat: xhs comments add xsec_token
 2024-12-03 18:34:56 +08:00
程序员阿江(Relakkes) 6001fc8a52 Merge pull request #506 from JianxunRao/main 
fix：微博根据creator爬取note时，爬取评论失败。原因是解析的参数key有误
 2024-11-29 11:07:58 +08:00
Relakkes 453ea642fb docs: update README.md 2024-11-29 10:40:52 +08:00
Relakkes ca9b47ef63 fix: xhs 帖子详情优化 2024-11-27 09:41:24 +08:00
Relakkes 43dffeb2d1 feat: xhs帖子详情获取优化 2024-11-26 13:37:53 +08:00
Relakkes de32d06815 docs: update README.md 2024-11-19 12:58:28 +08:00