fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-09 05:38:12 +08:00
Code Issues Packages Projects Releases Wiki Activity
560 Commits 11 Branches 0 Tags
b2d52918ae61835a8d7828e805266d0cd97bcd74
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江-Relakkes b2d52918ae Merge pull request #684 from 2513502304/main 
添加抖音笔记图片的存储逻辑
 2025-07-30 14:51:46 +08:00
程序员阿江(Relakkes) 8ab1b7ee4c fix: fixed circular import issue 2025-07-30 14:47:11 +08:00
程序员阿江-Relakkes 72a4d194c0 Merge pull request #680 from korruz/main 
refactor: move format_proxy_info to utils and update crawler classes
 2025-07-29 14:21:48 +08:00
程序员阿江-Relakkes 2935841872 Merge pull request #676 from andy9468/main 
add aiosqlite
 2025-07-28 11:33:42 +08:00
程序员阿江-Relakkes dbbc33a0df Merge pull request #674 from persist-1/chore 
chore: 增加--help参数中文显示支持；增加"douyin_aweme"表"music_download_url"字段与功能实现
 2025-07-25 17:23:03 +08:00
程序员阿江(Relakkes) fc06c783f5 fix: fixed xhs req headers 2025-07-23 13:28:58 +08:00
程序员阿江(Relakkes) b41896f4f3 docs: add a sponsor 2025-07-23 13:17:19 +08:00
程序员阿江(Relakkes) a4d9aaa34a refactor: xhs update 2025-07-21 21:26:16 +08:00
程序员阿江(Relakkes) 26a43358cb chore: update config 2025-07-20 14:34:56 +08:00
程序员阿江(Relakkes) 13b00f7a36 refactor: config update 2025-07-18 23:26:52 +08:00
程序员阿江(Relakkes) 122978b35c Merge pull request #652 from gaoxiaobei/dev 
feat(bilibili): Add flexible search modes and fix limit logic
 2025-07-18 21:41:20 +08:00
程序员阿江(Relakkes) 2753e7631e Merge pull request #664 from cfl-chenfangliang/feat-dyCommentBug 
feat: 修复抖音二级评论地理位置缺失问题
 2025-07-18 15:53:33 +08:00
程序员阿江(Relakkes) 08c28e6f7b Merge pull request #658 from persist-1/feature/sqlite-support 
增加对本地Sqlite数据库的支持（在不便于使用Mysql服务时也能使用数据库进行相关操作）
 2025-07-16 20:35:58 +08:00
程序员阿江(Relakkes) c5509ab91f fix: update ip proxy valid url for #662 2025-07-16 11:06:58 +08:00
程序员阿江(Relakkes) c795b1316a fix: import error for #663 2025-07-16 10:58:11 +08:00
程序员阿江(Relakkes) 3184e0a8d9 fix: sign data too long error #659 2025-07-14 18:43:17 +08:00
程序员阿江(Relakkes) dd8a3f5db8 docs: add a Sponsor 2025-07-12 23:26:30 +08:00
程序员阿江(Relakkes) ec0d29cf0f Merge pull request #642 from cllei12/weibo-search-type 
增加选择微博搜索类型的配置
 2025-07-07 15:05:29 +08:00
程序员阿江(Relakkes) 0d21a27b6e Merge pull request #641 from cllei12/main 
Update playwright version to support Ubuntu 24.04
 2025-07-07 15:03:42 +08:00
程序员阿江(Relakkes) 3cb0e2f91f Merge pull request #637 from Mirza-Samad-Ahmed-Baig/fix/bilibili-creator-videos 
refactor(bilibili): process creator videos in batches
 2025-07-05 00:13:49 +08:00
程序员阿江(Relakkes) 66a68fbb13 docs: update multi language badges size 2025-07-04 14:15:03 +08:00
程序员阿江(Relakkes) 8dcc540797 Merge pull request #635 from Root-FTW/main 
🌐 Add multilingual documentation support (English & Spanish)
 2025-07-04 13:54:31 +08:00
程序员阿江(Relakkes) 848df2b491 feat: other platfrom support the cdp mode 2025-07-03 17:13:32 +08:00
程序员阿江(Relakkes) c892c3324c chore: update uv lock file use aliyun pypi 2025-07-03 16:07:07 +08:00
程序员阿江(Relakkes) 452aafeec8 chore: update uv depends source 2025-07-03 15:27:39 +08:00
程序员阿江(Relakkes) 89359aa259 docs: update README.md 2025-07-02 01:20:44 +08:00
程序员阿江(Relakkes) 0514758cff docs: 更新uv安装文档 2025-06-29 00:07:13 +08:00
程序员阿江(Relakkes) e83b2422d9 feat: 支持playwright通过cdp协议连接本地chrome浏览器
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
docs: 增加uv来管理python依赖的文档
 2025-06-25 23:22:39 +08:00
程序员阿江(Relakkes) fbc9788d54 docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-24 17:44:43 +08:00
Relakkes 71210e3f38 chore: delete unnecessary file
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-20 16:26:02 +08:00
Relakkes fc45b22963 fix: vitepress path problem 2025-06-20 16:10:44 +08:00
Relakkes 22ec34dca3 docs: add list of sponsors 2025-06-20 16:04:04 +08:00
Relakkes fd33813f8f feat: add like_count field to bilibi for issue #623 2025-06-20 15:50:38 +08:00
Relakkes 31bcdb191f docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-16 13:58:09 +08:00
Relakkes d55d8b1efa feat: Douyin supports obtaining video links and cover images. for issue #620
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-14 23:59:08 +08:00
Relakkes ed1dc7916a docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-08 15:56:02 +08:00
程序员阿江(Relakkes) 6323e2d45b Merge pull request #616 from chimeElm/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
修复CRAWLER_MAX_NOTES_COUNT在爬取小红书作者帖子时失效的问题
 2025-06-07 14:43:37 +08:00
Relakkes 23c8f8f87b docs: add english license
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-06-01 23:20:11 +08:00
Relakkes 1e7b950d3e Revert "chore: remove sponor"
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
This reverts commit 242c06c345.
 2025-05-26 22:35:18 +08:00
Relakkes 242c06c345 chore: remove sponor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-05-25 11:54:38 +08:00
程序员阿江(Relakkes) ff41faeb00 Merge pull request #608 from Bowenwin/bili_expand
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
Bili_function_expand
 2025-05-22 23:14:58 +08:00
程序员阿江(Relakkes) 7ed6621933 Merge pull request #603 from Bowenwin/fix_words
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
Fix words
 2025-05-19 23:16:12 +08:00
Relakkes 654260cbce docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-05-13 18:42:58 +08:00
Relakkes 79a9824f6a fix: modify dy schema
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-04-30 16:47:13 +08:00
Relakkes 67d31bf42a fix: dy update fp params 2025-04-30 13:26:22 +08:00
程序员阿江(Relakkes) 2a41b684ad Merge pull request #590 from 2513502304/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
关于 issue #589 的增强方法
 2025-04-20 14:14:55 +08:00
Relakkes cb2ae6cbab docs: add sponsor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-04-16 16:49:49 +08:00
Relakkes 0d715a9f32 fix: bili qrcode login fix
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-04-08 21:11:40 +08:00
Relakkes 660fd18a95 fix: dy login fix 2025-04-08 20:58:04 +08:00
程序员阿江(Relakkes) afbd4ec1bf Merge pull request #572 from crpa33/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
避免预料之外的数据为None的情况终止进程
 2025-04-02 13:34:41 +08:00
Relakkes dfddfa7fdc docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-23 20:35:11 +08:00
Relakkes daaea7155b feat: add uv tool to manage project 2025-03-23 18:13:13 +08:00
Relakkes 8030d2a02f docs: removed sponsor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-13 15:07:54 +08:00
Relakkes 061d1c15e2 feat: kuaishou search params update
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-11 23:42:34 +08:00
Relakkes f2cf864c27 fix: zhihu article url error #564
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-03-03 18:18:41 +08:00
Relakkes b43d6b7b91 chore: update config
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-02-12 10:58:48 +08:00
Relakkes 66a7ab1db8 refactor: bibi default to get without time data 2025-02-12 10:58:15 +08:00
Relakkes 678ce1bfac fix: bilibili bugfix
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-02-10 17:13:37 +08:00
Relakkes 457205efd8 docs: add sponsor
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-02-08 15:28:18 +08:00
程序员阿江(Relakkes) 38f2b36bf5 Merge pull request #542 from 2513502304/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
Update core.py，为爬取类型为`detail`和`creator`的任务，添加了和`search`任务一样的，用于转存up主信息的`bilibili_store.update_up_info`的函数调用
 2025-01-20 19:30:10 +08:00
程序员阿江(Relakkes) 4b63ea68ec Merge pull request #538 from 2513502304/main 
feat: bilibli support date range filter
 2025-01-17 19:43:57 +08:00
Relakkes 30d0e733d5 feat: douyin adds comment images
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-15 14:50:05 +08:00
程序员阿江(Relakkes) a8e796e690 Merge pull request #536 from 2513502304/main
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
将微博评论爬取函数`get_note_all_comments`的`max_id_type`便为可变请求参数
 2025-01-13 13:58:15 +08:00
Relakkes 5d3e12169e fix: move dirctory
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-10 16:53:20 +08:00
Relakkes b89a8e5dda docs: add github issue template 2025-01-10 16:51:30 +08:00
Relakkes 792d3bbc34 docs: update README.md
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-05 19:02:10 +08:00
Relakkes 5f7c8aea5b docs: 添加11-12月份捐赠者名单
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2025-01-02 20:37:28 +08:00
Relakkes fbbead814a fix: 贴吧创作者bug修复 2025-01-02 20:29:05 +08:00
程序员阿江(Relakkes) 7ce1273386 Merge pull request #528 from HuiGitH/hui/fix-xhs-gender
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled 
fix xhs gender
 2024-12-28 21:01:58 +08:00
Relakkes 79bf9fc05d chore: add xhs field comment for issue #526
Deploy VitePress site to Pages / build (push) Has been cancelled
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
2024-12-26 18:28:23 +08:00
Relakkes 29ab6feded fix: 修复数据库密码中有特殊符号导致db连接不上的bug 2024-12-26 18:13:13 +08:00
Relakkes ea5223c708 feat: 知乎支持详情模式
Deploy VitePress site to Pages / build (push) Waiting to run
Deploy VitePress site to Pages / Deploy (push) Blocked by required conditions
2024-12-26 17:36:33 +08:00
程序员阿江(Relakkes) dc9116e098 Merge pull request #513 from liudongkai/feature_xhs 
feat: xhs 非代理模式下增加随机等待间隔, db存储模式下增加存储xsec_token字段
 2024-12-06 13:01:16 +08:00
程序员阿江(Relakkes) 9c7e1d499b Merge pull request #509 from leantli/feat/xhs_comments_upgrade 
feat: xhs comments add xsec_token
 2024-12-03 18:34:56 +08:00
程序员阿江(Relakkes) 6001fc8a52 Merge pull request #506 from JianxunRao/main 
fix：微博根据creator爬取note时，爬取评论失败。原因是解析的参数key有误
 2024-11-29 11:07:58 +08:00
Relakkes 453ea642fb docs: update README.md 2024-11-29 10:40:52 +08:00
Relakkes ca9b47ef63 fix: xhs 帖子详情优化 2024-11-27 09:41:24 +08:00
Relakkes 43dffeb2d1 feat: xhs帖子详情获取优化 2024-11-26 13:37:53 +08:00
Relakkes de32d06815 docs: update README.md 2024-11-19 12:58:28 +08:00
Relakkes 935a928f90 docs: update README.md 2024-11-17 07:00:55 +08:00
Relakkes 8ab4c67443 feat: 抖音支持评论点赞数量 #495 2024-11-16 00:37:48 +08:00
Relakkes c12a63acb2 fix: weibo creator duplicate bug 2024-11-13 22:28:54 +08:00
Relakkes f9e7d56d4f chore: remove sponsor 2024-11-08 22:27:03 +08:00
Relakkes 3db2edd479 fix: weibo创作者bugfix 2024-11-06 16:01:07 +08:00
程序员阿江(Relakkes) 3567193a98 Merge pull request #479 from lyx0727/main 
fix: xhs出现验证码时报错，增加用户手动验证
 2024-11-01 23:09:01 +08:00
Relakkes ef4eba121c fix: 兼容windows编码问题 2024-10-28 15:26:49 +08:00
Relakkes 9bf6ea3650 docs: update README.md 2024-10-24 18:18:53 +08:00
Relakkes 6a96c00b4b chore: config update & xinqqiu img update 2024-10-24 15:35:34 +08:00
程序员阿江(Relakkes) 5a27ad089c Merge pull request #462 from FloRainRJY/xiaohongshu_comment_number_restrict 
All_platform_comments_restrict
 2024-10-24 15:31:13 +08:00
程序员阿江(Relakkes) 0cd01fbf8c Merge pull request #463 from liugangdao/main 
fix:翻页时search id不变
 2024-10-24 15:13:05 +08:00
程序员阿江(Relakkes) fa2bcc4181 Merge pull request #461 from FloRainRJY/xiaohongshu_comment_number_restrict 
feat: xhs增加最大评论数量限制
 2024-10-23 13:35:10 +08:00
Relakkes 0bb9298d26 docs: 添加捐赠名单 2024-10-21 21:17:28 +08:00
Relakkes dc11d39185 docs: update README.md 2024-10-20 01:05:57 +08:00
Relakkes 03e393949a fix: xhs帖子详情问题更新 2024-10-20 00:59:08 +08:00
Relakkes 9fe3e47b0f chore: 增加代码学习声明，严格禁止非法、禁止商业、不当用途 2024-10-20 00:43:25 +08:00
程序员阿江-Relakkes ac8537c296 Update LICENSE 2024-10-18 03:27:00 +08:00
Relakkes aa0f920369 feat: B站搜索接口增加发布日期筛选 2024-10-17 15:11:25 +08:00
Relakkes fa1932e105 docs: 添加14群交流二维码 2024-10-17 14:54:06 +08:00
Relakkes 96508fb050 docs: 知乎支持创作者主页数据 2024-10-16 21:03:06 +08:00
Relakkes da8f1c62b8 feat: 知乎支持创作者主页数据爬取(回答、文章、视频) 2024-10-16 21:02:27 +08:00