fork/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-10-07 06:58:11 +08:00
Code Issues Packages Projects Releases Wiki Activity
241 Commits 11 Branches 0 Tags
3725128a35daa57afc4459052d0b5f874ddb8b0d
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
程序员阿江-Relakkes 3725128a35 Merge pull request #273 from nelzomal/bili-level-two-comment 
feat: support bilibili level two comment
 2024-05-26 15:34:28 +08:00
Relakkes 764bafc626 feat: 抖音登录态检测逻辑更新支持 2024-05-23 22:15:14 +08:00
Relakkes 3c4695eed7 docs: update README.md 2024-05-18 17:09:51 +08:00
Relakkes e64df93edd feat: 由于xhs和dy现在检测playwright二维码登录了，大概率会出现滑块或者手机验证，增加登录态检测时间为5min，预留足够的时间手动过验证码。 2024-05-15 23:23:30 +08:00
Relakkes 46f8cc064a feat: 增加爬虫程序运行前的说明 2024-05-13 21:01:54 +08:00
Relakkes 704ee8cf9d docs: update README.md 2024-05-12 23:28:50 +08:00
Relakkes 60182213fa docs: update README.md 2024-05-10 22:42:41 +08:00
程序员阿江-Relakkes 2df6ebe0d4 Merge pull request #265 from leantli/fix/value_error&typo 
fix: 捕捉处理 ValueError 报错以及修改错字
 2024-05-10 22:28:37 +08:00
Relakkes 57ee8dd63b docs: update README.md 2024-05-03 18:31:18 +08:00
Relakkes 292cceb4a6 docs: update README.md 2024-05-01 16:56:45 +08:00
程序员阿江-Relakkes 411ed2ead9 Merge pull request #253 from kexinoh/main 
新增id，解决同一天内的不同查询写入同一个文件的问题
 2024-04-30 23:10:16 +08:00
程序员阿江-Relakkes 237d5be894 Merge pull request #255 from henryhyn/feat/log_config 
feat: 配置日志打印文件及行号
 2024-04-27 13:18:13 +08:00
程序员阿江-Relakkes 2d051cc670 Merge pull request #252 from henryhyn/fix/help-f-string 
fix: 修复 f-string 双引号问题
 2024-04-26 23:58:25 +08:00
程序员阿江-Relakkes abb82076eb Merge pull request #250 from bigfa/fixed-trace-id 
fix:兼容网页端上传的图片原图获取
 2024-04-25 22:18:40 +08:00
程序员阿江-Relakkes 829974c7df Merge pull request #251 from leantli/chore/add_frequency_qa 
chore: 增加常见问题[小红书扫码登录成功后如何手动验证]
 2024-04-25 22:18:16 +08:00
Relakkes Yang e41fc05eac docs: update README.md 2024-04-22 21:25:10 +08:00
Relakkes Yang 48943a25c5 fix: add uvicorn to requirements.txt 2024-04-21 21:41:27 +08:00
Relakkes Yang 59c683a057 refactor: remove tornado server add fastapi 2024-04-21 21:11:46 +08:00
Relakkes 5d97ffac98 feat: 支持初始化mysql表结构 2024-04-18 22:34:51 +08:00
Relakkes 5681dd6925 fix: #237 2024-04-17 23:32:17 +08:00
Relakkes 41fb7bb583 fix: remove kdl secret key 2024-04-17 23:19:44 +08:00
Relakkes 487afc8e0c refactor: 修改导报顺心 2024-04-17 23:13:40 +08:00
Relakkes d0efd4e4f0 feat: 快代理实现 2024-04-17 23:12:41 +08:00
程序员阿江-Relakkes 30ae3514c3 Merge pull request #238 from keeper-jie/patch-1 
Update 常见问题.md
 2024-04-17 21:40:06 +08:00
Relakkes 9443e9d28d docs: Update README.md 2024-04-14 20:45:49 +08:00
Relakkes 87eb8aa6a7 fix: #230 2024-04-13 20:18:04 +08:00
Relakkes c26270146f fix: 修复ip代理池设计bug 2024-04-13 18:04:33 +08:00
Relakkes e367129b5f fix: 修复pr#229的小bug 2024-04-13 13:48:29 +08:00
程序员阿江-Relakkes a341dc2aff Merge pull request #229 from Tianci-King/main 
feat(core): 新增控制爬虫参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数…
 2024-04-13 13:37:35 +08:00
程序员阿江-Relakkes da949f0c7a Merge pull request #226 from leantli/feat/sub_comments 
feat: 支持爬取小红书二级评论
 2024-04-13 01:17:55 +08:00
程序员阿江-Relakkes bba9841c26 Merge pull request #223 from Ermeng98/main 
新增对微博博客内照片获取的支持 文件存放路径data/weibo/images
 2024-04-11 00:11:04 +08:00
程序员阿江-Relakkes cab91474ad Merge pull request #224 from Klu5ure/main 
docs: 恢复文档中误删的字
 2024-04-09 23:14:14 +08:00
Relakkes 488acd04df fix: #222 2024-04-09 23:10:51 +08:00
Relakkes 5c409c6f0c docs: 群二维码失效，更新 2024-04-08 23:41:53 +08:00
Relakkes 8f02da73ad fix: #219 
docs: update README.md
 2024-04-08 00:19:50 +08:00
程序员阿江-Relakkes b54f168c6b Merge pull request #217 from Klu5ure/main 
docs: 修改错别字
 2024-04-07 23:11:19 +08:00
Relakkes d392747fe7 fix: 移除orm的所有内容 2024-04-06 23:51:03 +08:00
程序员阿江-Relakkes 1627ae4344 Merge pull request #216 from NanmiCoder/feature/refactor_db 
数据存储DB重构，移除tortoise-orm依赖
 2024-04-06 22:21:30 +08:00
Relakkes f9cbeb2b14 chore: 移除tortise-orm依赖 2024-04-06 22:14:03 +08:00
Relakkes 0c8484c334 feat: db数据存储重构完成 2024-04-06 22:11:10 +08:00
Relakkes de4a437dd7 style: rename image name 2024-04-06 12:23:55 +08:00
Relakkes 27f3461e31 docs: update README.md 2024-04-06 10:58:35 +08:00
Relakkes dde3c0429e refactor: IP代理池重构 2024-04-06 10:58:35 +08:00
程序员阿江-Relakkes d0c578c2bf Merge pull request #213 from Styunlen/patch-1 
chore: fix wrong log output when weibo crawler finished
 2024-04-06 00:51:36 +08:00
程序员阿江-Relakkes 5495d76fdb Merge pull request #212 from chunpat/fixbug_qrcode_show 
Remove duplication Qrcode Show
 2024-04-05 22:02:06 +08:00
程序员阿江-Relakkes 87e3d0848a Merge pull request #209 from xbsheng/chore/text 
chore: 修改错别字
 2024-04-04 18:11:42 +08:00
程序员阿江-Relakkes 6160428909 Merge pull request #202 from zhihuiio/patch-1 
Update playwright version for compiling
 2024-04-04 13:12:50 +08:00
程序员阿江-Relakkes 208978b88f Merge pull request #206 from leantli/fix/max_notes_count 
fix: 修复爬取视频/帖子最大数设置值较低导致不爬取的问题
 2024-04-04 00:46:42 +08:00
Relakkes 3de57459c1 docs: 增加交流2群二维码 2024-04-03 13:40:38 +08:00
Relakkes 85382ac7ed docs:更新许可证协议，不允许商用。 2024-04-03 12:34:34 +08:00
Relakkes e4836847cd docs: update README.md 2024-04-01 22:08:26 +08:00
程序员阿江-Relakkes 285ab7abe6 Update README.md 2024-04-01 12:54:53 +08:00
Relakkes bb732ed2ff docs: 增加MediaCrawler视频课程链接 2024-03-31 23:51:37 +08:00
Relakkes 6d8c8fb22e feat: add mysql table 2024-03-31 15:05:34 +08:00
程序员阿江-Relakkes 807af1abed Merge pull request #201 from Schofi/main 
Update README.md
 2024-03-31 14:40:11 +08:00
Relakkes e950e0d6e3 feat: add abstract api client to all platform 2024-03-30 21:27:25 +08:00
Relakkes 67ec49498a refactor: rename xhs to xiaohongshu 2024-03-30 21:17:33 +08:00
Relakkes aa257aab51 docs: update README.md 2024-03-30 14:14:10 +08:00
relakkes aca1924bd7 Merge pull request #183 from BaoZhuhan/fix-markdown 
fix ： 修复”常见问题.md“
 2024-03-19 23:13:34 +08:00
relakkes 96caede849 Merge pull request #182 from BaoZhuhan/fix-markdown 
fix: 修复了“常见问题.md"的语法问题。
 2024-03-19 22:51:45 +08:00
Relakkes 4ea9700db0 docs: 增加赞助名单 2024-03-19 21:50:25 +08:00
relakkes 3a318faee1 Update README.md 2024-03-19 13:27:58 +08:00
Relakkes 5f5147a236 docs: 添加捐赠者名单 2024-03-18 22:55:40 +08:00
Relakkes 8e8f5d41e4 docs: 添加捐赠者名单 2024-03-18 22:03:45 +08:00
relakkes ea66839d72 Update README.md 2024-03-18 13:40:36 +08:00
Relakkes 67443737f5 docs: 添加捐赠者名单 2024-03-17 22:16:58 +08:00
Relakkes 96309dcfee fix: 小红书创作者功能数据获取优化 2024-03-17 14:50:10 +08:00
Relakkes 78a9bf9f38 docs: 添加捐赠者名单 2024-03-17 01:05:10 +08:00
Relakkes 569202af78 feat: update user-agent list 2024-03-17 01:03:56 +08:00
Relakkes 59cd9f67a0 feat: 支持评论模式是否开启爬取选项 2024-03-16 11:52:42 +08:00
Relakkes 2d12ecb930 docs: 增加捐赠名单 2024-03-14 22:06:18 +08:00
Relakkes b48e4d297d docs: 将python3命令修改为python 2024-03-12 23:09:45 +08:00
Relakkes 9e2d1396b8 docs: 增加视频入门教程链接 2024-03-10 03:49:58 +08:00
Relakkes 9ca0a2b717 docs: update README.md 2024-03-09 15:37:51 +08:00
Relakkes 41fee4ff4f feat：小红书支持获取评论中的图片链接 #145 2024-03-07 22:30:44 +08:00
Relakkes 861019022a docs: 增加捐赠者名单 2024-03-05 22:45:32 +08:00
Relakkes 7606c1d993 docs: 增加捐赠者名单 2024-03-05 00:03:51 +08:00
Relakkes 149b6bcdc8 fix: 修复抖音关键词搜索为中文的情况下，有bug 2024-03-03 19:36:36 +08:00
relakkes 7002061d70 Merge pull request #142 from jayeeliu/main 
小红书支持通过博主ID采集笔记和评论
 2024-03-02 23:30:31 +08:00
Relakkes 384c8f9f7e fix: issue #140 2024-02-26 23:47:02 +08:00
relakkes 67d2b7cff8 Merge pull request #136 from jayeeliu/main 
增加sqlite配置示例，解决用sqlite保存数据时，抓取结束不退出的问题
 2024-02-22 21:32:19 +08:00
Relakkes 03ca445221 docs: update README.md 2024-02-18 13:06:12 +08:00
Relakkes 192e72588d fix: #123 2024-01-26 23:42:26 +08:00
relakkes f1d3bc6e28 Merge pull request #124 from aa65535/patch-1 
修复小红书评论重复插入
 2024-01-25 14:33:09 +08:00
relakkes eb89a6ada8 docs: 添加赞赏者信息 2024-01-21 16:42:37 +08:00
Relakkes e940a41033 refactor: 移除评论中指定数量和过滤特定关键词的逻辑 2024-01-17 23:02:05 +08:00
Relakkes e0f9a487e4 refactor: 代码优化 2024-01-16 00:40:07 +08:00
relakkes e490123fcd Merge pull request #112 from NanmiCoder/feature/stored_in_json 
refactor: 重构数据存储代码，新增jSON
 2024-01-14 22:45:00 +08:00
Relakkes 4dfa0d3fbf feat: 数据保存支持JSON格式 2024-01-14 22:40:01 +08:00
Relakkes 894dabcf63 refactor: 数据存储重构，分离不同类型的存储实现 2024-01-14 22:06:31 +08:00
Relakkes e31aebbdfb fix: 修复代理Bug 2024-01-13 15:50:02 +08:00
Relakkes 0240de7324 docs: 添加捐赠信息 2024-01-10 23:59:14 +08:00
Relakkes f85bd0c2b9 docs: 添加捐赠信息 2024-01-07 18:33:08 +08:00
Relakkes 4de14ad6a8 fix: 修复微博PC端登录后COOKIE在手机端无法使用的bug 2024-01-06 19:18:07 +08:00
Relakkes fe073801f8 feat: 小红书增加获取图片地址的方法 2024-01-04 00:11:05 +08:00
Relakkes 80ecc53cd3 fix: #107 2024-01-03 23:08:30 +08:00
Relakkes 38d6f10bf0 feat: 微博二维码登录done 2023-12-30 18:54:21 +08:00
Relakkes 27a2041929 doc: update README.md 2023-12-30 15:16:33 +08:00
Relakkes eee81622ac feat: 微博支持评论 & 指定帖子 2023-12-25 00:02:11 +08:00
Relakkes b1441ab4ae feat: 微博帖子支持保存到数据库中 2023-12-24 18:19:26 +08:00