程序员阿江(Relakkes)
071c8c0aca
fix(kuaishou): 降低服务端限流概率并支持限流退避重试
...
- 请求延时在固定基础上加 1-3 秒随机抖动，降低被限流概率
- 签名 REST 请求遇到限流(result:2)时指数退避重试 3 次，不再直接中断
- 重试时重新生成签名（签名绑定请求内容和时间窗口）
2026-08-05 17:39:33 +08:00
程序员阿江(Relakkes)
02294b9ea4
style(kuaishou): 优化终端日志输出为条目摘要
...
- 视频详情不再打印完整 JSON，改为单行摘要（id/作者/点赞/播放/标题）
- 搜索每页输出搜索到的视频条数
- 创作者输出获取到的视频总数
2026-08-04 01:29:21 +08:00
程序员阿江(Relakkes)
2e558f1352
fix(kuaishou): 支持网页端 REST 接口签名请求
...
- 新增页面签名环境捕获脚本，通过 caver 属性赋值轨迹获取签名调用入口
- 作品列表/关键词搜索接口迁移到带签名的 REST v2 请求，并显式校验 result 状态码
- 修复分页停止条件，空列表时结束翻页
- 更新默认创作者主页 ID
2026-08-04 01:14:39 +08:00
Keinoki
97e4142733
fix: 修复快手翻页逻辑
2026-06-15 15:37:18 +08:00
程序员阿江(Relakkes)
0c5f281212
fix: 避免复用浏览器时跨域 Cookie 过长导致请求失败
...
连接已有 Chrome 会把整个浏览器上下文的 cookie 带入平台 client。
除 xhs 外，多数平台仍直接读取全量 cookies，导致请求头过长并放大跨域污染。
本次将各平台的 cookie 读取统一收口到平台域名，并补上基础回归测试。
Constraint: 必须继续复用用户真实浏览器里的平台登录态
Rejected: 仅修复 xhs | 其他平台在连接已有浏览器时仍会携带超长 Cookie
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: 后续新增平台或调整 update_cookies 和 create client 流程时，只按平台域名读取 cookies
Tested: uv run pytest test/test_utils.py; python3 -m compileall tools/crawler_util.py media_platform/douyin/core.py media_platform/douyin/client.py media_platform/kuaishou/core.py media_platform/kuaishou/client.py media_platform/bilibili/core.py media_platform/bilibili/client.py media_platform/zhihu/core.py media_platform/zhihu/client.py media_platform/tieba/core.py media_platform/tieba/client.py media_platform/xhs/core.py media_platform/xhs/client.py media_platform/weibo/core.py media_platform/weibo/client.py test/test_utils.py
Not-tested: 各平台在真实 CDP 浏览器连接下的端到端抓取流程
2026-04-21 13:49:37 +08:00
程序员阿江(Relakkes) and Claude Opus 4.5
157ddfb21b
i18n: translate all Chinese comments, docstrings, and logger messages to English
...
Comprehensive translation of Chinese text to English across the entire codebase:
- api/: FastAPI server documentation and logger messages
- cache/: Cache abstraction layer comments and docstrings
- database/: Database models and MongoDB store documentation
- media_platform/: All platform crawlers (Bilibili, Douyin, Kuaishou, Tieba, Weibo, Xiaohongshu, Zhihu)
- model/: Data model documentation
- proxy/: Proxy pool and provider documentation
- store/: Data storage layer comments
- tools/: Utility functions and browser automation
- test/: Test file documentation
Preserved: Chinese disclaimer header (lines 10-18) for legal compliance
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2025-12-26 23:27:19 +08:00
程序员阿江(Relakkes)
6eef02d08c
feat: ip proxy expired check
2025-11-25 12:39:10 +08:00
程序员阿江(Relakkes)
ff8c92daad
chore: add copyright to every file
2025-11-18 12:24:02 +08:00
yt210
ef6948b305
新增存储到mongoDB
2025-11-06 10:40:30 +08:00
程序员阿江(Relakkes)
03e384bbe2
refactor: cdp模式下移除stealth注入
2025-10-19 15:32:03 +08:00
程序员阿江(Relakkes)
ae7955787c
feat: kuaishou support url link
2025-10-18 07:40:10 +08:00
程序员阿江(Relakkes)
2cf143cc7c
fix : #730
2025-09-26 18:10:30 +08:00
程序员阿江(Relakkes)
2bce3593f7
feat: support time deplay for all platform
2025-09-02 16:43:09 +08:00
未来可欺
0b81240aed
升级 httpx 版本至 0.28.1，并修改关键字参数 proxies 至 proxy
2025-07-31 22:48:02 +08:00
korruz
07a6e387ea
refactor: move format_proxy_info to utils and update crawler classes to use it
2025-07-29 14:16:24 +08:00
程序员阿江(Relakkes)
13b00f7a36
refactor: config update
2025-07-18 23:26:52 +08:00
gaoxiaobei
b913db64bb
refactor(config): move platform-specific configs to separate files
...
- Remove platform-specific configurations from base_config.py
- Create separate config files for each platform in their respective directories
- Update import statements in core files to use new platform-specific config modules
- Clean up unused and deprecated configuration options
2025-07-18 17:27:37 +08:00
程序员阿江(Relakkes)
848df2b491
feat: other platfrom support the cdp mode
2025-07-03 17:13:32 +08:00
Relakkes
061d1c15e2
feat: kuaishou search params update
Deploy VitePress site to Pages / Deploy (push) Has been cancelled
Deploy VitePress site to Pages / build (push) Has been cancelled
2025-03-11 23:42:34 +08:00
unknown
7e53c4acfc
All_platform_comments_restrict
2024-10-23 16:32:02 +08:00
Relakkes
9fe3e47b0f
chore: 增加代码学习声明，严格禁止非法、禁止商业、不当用途
2024-10-20 00:43:25 +08:00
Relakkes
c70bd9e071
feat: 增加搜索词来源渠道
2024-08-23 08:29:24 +08:00
Relakkes
d3eeccbaac
feat: logger record current search page
2024-06-24 22:24:51 +08:00
HIRO
a001556ba7
快手指定创作者主页和二级评论
2024-06-13 14:49:07 +08:00
nelzomal
eace7d1750
improve base config reading command line arg logic
2024-06-09 18:51:36 +08:00
Tianci-King
1115b0d90c
feat(core): 新增控制爬虫 参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数起始页面页数和关键字
2024-04-12 00:52:47 +08:00
leantli
68a60faa7f
chore: 简化判断方式
2024-04-04 00:11:22 +08:00
leantli
133f978477
fix: 修复爬取视频/帖子最大数设置值较低导致不爬取的问题
2024-04-03 12:18:23 +08:00
Relakkes
59cd9f67a0
feat: 支持评论模式是否开启爬取选项
2024-03-16 11:52:42 +08:00
Relakkes
894dabcf63
refactor: 数据存储重构，分离不同类型的存储实现
2024-01-14 22:06:31 +08:00
Relakkes
e31aebbdfb
fix: 修复代理Bug
2024-01-13 15:50:02 +08:00
Relakkes
aba9f14f50
refactor: 规范日志打印
...
feat: B站指定视频ID爬取（bvid）
2023-12-23 01:04:08 +08:00
Relakkes
273c9a316b
fix: 修复日志打印时参数格式错误
2023-12-22 23:10:44 +08:00
Relakkes
97d7a0c38b
feat: Bilibili comment done
2023-12-09 21:10:01 +08:00
Relakkes
1cec23f73d
feat: 代理IP功能 Done
2023-12-08 00:10:04 +08:00
Relakkes
986179b9c9
feat: 增加 IP 代理的最新实现
2023-12-02 16:14:36 +08:00
Relakkes
33721e5fbd
feat: 快手支持指定视频列表爬取
2023-11-27 23:07:04 +08:00
Relakkes
dfb1788141
feat: 快手视频评论爬取done；数据保存到DB、CSV done
2023-11-26 21:43:39 +08:00
Relakkes
bdf36ccb09
feat: 快手关键词搜索存储CSV完成
2023-11-26 01:05:52 +08:00
Relakkes
512192a93e
feat: 搜索接口调试完成
2023-11-25 00:02:33 +08:00
Relakkes
f08b2ceb76
feat: 1、命令行支持快手 2、快手playwright 代码 done
2023-11-24 00:04:33 +08:00
Relakkes
95ca606938
feat: 快手文件目录建立
2023-11-23 23:13:54 +08:00
程序员阿江(Relakkes)