-
+## 查看教程
+在线链接：https://nanmicoder.github.io/CrawlerTutorial/
+对应的视频链接近期也会同步更新出来，[查看B站合集地址](https://space.bilibili.com/434377496/channel/collectiondetail?sid=4035213&ctype=0)
## 爬虫入门
-### 爬虫入门教程目录大纲
-- [x] [01_为什么要写这个爬虫教程](爬虫入门/01_为什么要写这个爬虫教程.md)
-- [x] [02_个人学会爬虫能赚钱吗](爬虫入门/02_个人学会爬虫能赚钱吗.md)
-- [x] [03_网络爬虫到底是什么](爬虫入门/03_网络爬虫到底是什么.md)
-- [x] [04_爬虫的基本工作原理](爬虫入门/04_爬虫的基本工作原理.md)
-- [x] [05_常用的抓包工具有那些](爬虫入门/05_常用的抓包工具有那些.md)
-- [x] [06_为什么说用Python写爬虫有天生优势](爬虫入门/06_为什么说用Python写爬虫有天生优势.md)
-- [x] [07_Python常见的网络请求库](爬虫入门/07_Python常见的网络请求库.md)
-- [x] [08_爬虫入门实战1_静态网页数据提取](爬虫入门/08_爬虫入门实战1_静态网页数据提取.md)
-- [x] [09_爬虫入门实战2_动态数据提取](爬虫入门/09_爬虫入门实战2_动态数据提取.md)
-- [ ] [10_爬虫入门实战3_数据存储实现](爬虫入门/10_爬虫入门实战3_数据存储实现.md)
-- [ ] [11_爬虫入门实战4_高效率的爬虫实现](爬虫入门/11_爬虫入门实战4_高效率的爬虫实现.md)
-- [ ] [12_爬虫入门实战5_编写易于维护的爬虫代码](爬虫入门/12_爬虫入门实战5_编写易于维护的爬虫代码.md)
+- [✔] [01_为什么要写这个爬虫教程](docs/爬虫入门/01_为什么要写这个爬虫教程.md)
+- [✔] [02_个人学会爬虫能赚钱吗](docs/爬虫入门/02_个人学会爬虫能赚钱吗.md)
+- [✔] [03_网络爬虫到底是什么](docs/爬虫入门/03_网络爬虫到底是什么.md)
+- [✔] [04_爬虫的基本工作原理](docs/爬虫入门/04_爬虫的基本工作原理.md)
+- [✔] [05_常用的抓包工具有那些](docs/爬虫入门/05_常用的抓包工具有那些.md)
+- [✔] [06_为什么说用Python写爬虫有天生优势](docs/爬虫入门/06_为什么说用Python写爬虫有天生优势.md)
+- [✔] [07_Python常见的网络请求库](docs/爬虫入门/07_Python常见的网络请求库.md)
+- [✔] [08_爬虫入门实战1_静态网页数据提取](docs/爬虫入门/08_爬虫入门实战1_静态网页数据提取.md)
+- [✔] [09_爬虫入门实战2_动态数据提取](docs/爬虫入门/09_爬虫入门实战2_动态数据提取.md)
+- [✔] [10_爬虫入门实战3_数据存储实现](docs/爬虫入门/10_爬虫入门实战3_数据存储实现.md)
+- [✔] [11_爬虫入门实战4_高效率的爬虫实现](docs/爬虫入门/11_爬虫入门实战4_高效率的爬虫实现.md)
-## 打赏
-免费开源不易，如果项目帮到你了，可以给我打赏哦，您的支持就是我最大的动力！
-
## 免责声明
>本仓库的所有内容仅供学习和参考之用，禁止用于商业用途。任何人或组织不得将本仓库的内容用于非法用途或侵犯他人合法权益。本仓库所涉及的爬虫技术仅用于学习和研究，不得用于对其他平台进行大规模爬虫或其他非法行为。对于因使用本仓库内容而引起的任何法律责任，本仓库不承担任何责任。使用本仓库的内容即表示您同意本免责声明的所有条款和条件。
+
## Star History
-[](https://star-history.com/#NanmiCoder/CrawlerTutorial&Date)
\ No newline at end of file
+[](https://star-history.com/#NanmiCoder/CrawlerTutorial&Date)
diff --git a/docs/.vitepress/config.mjs b/docs/.vitepress/config.mjs
new file mode 100644
index 0000000..b9ad440
--- /dev/null
+++ b/docs/.vitepress/config.mjs
@@ -0,0 +1,92 @@
+import {defineConfig} from 'vitepress'
+import {withMermaid} from 'vitepress-plugin-mermaid'
+
+// https://vitepress.dev/reference/site-config
+export default withMermaid(defineConfig({
+ base: '/CrawlerTutorial/',
+ title: "程序员阿江-Relakkes的爬虫教程",
+ description: "程序员阿江-Relakkes的爬虫教程",
+ lastUpdated: true,
+ head: [
+ [
+ 'script',
+ {async: '', src: 'https://www.googletagmanager.com/gtag/js?id=G-B5H6D2HDGK'}
+ ],
+ [
+ 'script',
+ {},
+ `window.dataLayer = window.dataLayer || [];
+ function gtag(){dataLayer.push(arguments);}
+ gtag('js', new Date());
+ gtag('config', 'G-B5H6D2HDGK');`
+ ],
+ [
+ 'script',
+ {async: '', src: 'https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-5210914487984731', crossorigin: 'anonymous'},
+ ]
+ ],
+ themeConfig: {
+ editLink: {
+ pattern: 'https://github.com/NanmiCoder/CrawlerTutorial/tree/main/docs/:path'
+ },
+ search: {
+ provider: 'local'
+ },
+ // https://vitepress.dev/reference/default-theme-config
+ nav: [
+ {text: '首页', link: '/'},
+ {text: 'B站视频课程', link: 'https://space.bilibili.com/434377496/channel/collectiondetail?sid=4035213&ctype=0'},
+ {text: '联系作者', link: 'https://nanmicoder.github.io/MediaCrawler/%E4%BD%9C%E8%80%85%E4%BB%8B%E7%BB%8D.html'},
+ {text: '支持作者', link: 'https://nanmicoder.github.io/MediaCrawler/%E7%9F%A5%E8%AF%86%E4%BB%98%E8%B4%B9%E4%BB%8B%E7%BB%8D.html'},
+ ],
+
+ sidebar: [
+ {
+ text: 'Python爬虫入门',
+ collapsed: false,
+ items: [
+ {text: '01_为什么要写这个爬虫教程', link: '/爬虫入门/01_为什么要写这个爬虫教程'},
+ {text: '02_个人学会爬虫能赚钱吗', link: '/爬虫入门/02_个人学会爬虫能赚钱吗'},
+ {text: '03_网络爬虫到底是什么', link: '/爬虫入门/03_网络爬虫到底是什么'},
+ {text: '04_爬虫的基本工作原理', link: '/爬虫入门/04_爬虫的基本工作原理'},
+ {text: '05_常用的抓包工具有那些', link: '/爬虫入门/05_常用的抓包工具有那些'},
+ {
+ text: '06_Python写爬虫的优势',
+ link: '/爬虫入门/06_为什么说用Python写爬虫有天生优势'
+ },
+ {text: '07_Python常见的网络请求库', link: '/爬虫入门/07_Python常见的网络请求库'},
+ {text: '08_入门实战1_静态网页数据提取', link: '/爬虫入门/08_爬虫入门实战1_静态网页数据提取'},
+ {text: '09_入门实战2_动态数据提取', link: '/爬虫入门/09_爬虫入门实战2_动态数据提取'},
+ {text: '10_入门实战3_数据存储实现', link: '/爬虫入门/10_爬虫入门实战3_数据存储实现'},
+ {text: '11_入门实战4_高效率的爬虫实现', link: '/爬虫入门/11_爬虫入门实战4_高效率的爬虫实现'},
+ {
+ text: '12_入门实战5_编写易于维护的代码',
+ link: '/爬虫入门/12_爬虫入门实战5_编写易于维护的爬虫代码'
+ }
+ ]
+ },
+ {
+ text: 'Python爬虫进阶',
+ collapsed: false,
+ items: [
+ {text: '01_工程化爬虫开发规范', link: '/爬虫进价/01_工程化爬虫开发规范'},
+ {text: '02_反爬虫对抗基础_请求伪装', link: '/爬虫进价/02_反爬虫对抗基础_请求伪装'},
+ {text: '03_代理IP的使用与管理', link: '/爬虫进价/03_代理IP的使用与管理'},
+ {text: '04_Playwright浏览器自动化入门', link: '/爬虫进价/04_Playwright浏览器自动化入门'},
+ {text: '05_Playwright进阶_反检测与性能优化', link: '/爬虫进价/05_Playwright进阶_反检测与性能优化'},
+ {text: '06_登录认证_Cookie与Session管理', link: '/爬虫进价/06_登录认证_Cookie与Session管理'},
+ {text: '07_登录认证_扫码与短信登录实现', link: '/爬虫进价/07_登录认证_扫码与短信登录实现'},
+ {text: '08_验证码识别与处理', link: '/爬虫进价/08_验证码识别与处理'},
+ {text: '09_数据清洗与预处理', link: '/爬虫进价/09_数据清洗与预处理'},
+ {text: '10_数据分析与可视化', link: '/爬虫进价/10_数据分析与可视化'},
+ {text: '11_进阶综合实战项目', link: '/爬虫进价/11_进阶综合实战项目'}
+ ]
+ }
+ ],
+
+ socialLinks: [
+ {icon: 'github', link: 'https://github.com/NanmiCoder/CrawlerTutorial'}
+ ]
+ },
+
+}))
diff --git a/docs/.vitepress/theme/DynamicAds.vue b/docs/.vitepress/theme/DynamicAds.vue
new file mode 100644
index 0000000..3a3cbeb
--- /dev/null
+++ b/docs/.vitepress/theme/DynamicAds.vue
@@ -0,0 +1,86 @@
+
+
+
+
+
+ Loading ads...
+
+
+
+## 免责声明
+>本仓库的所有内容仅供学习和参考之用，禁止用于商业用途。任何人或组织不得将本仓库的内容用于非法用途或侵犯他人合法权益。本仓库所涉及的爬虫技术仅用于学习和研究，不得用于对其他平台进行大规模爬虫或其他非法行为。对于因使用本仓库内容而引起的任何法律责任，本仓库不承担任何责任。使用本仓库的内容即表示您同意本免责声明的所有条款和条件。
+
+
+## Star History
+
+[](https://star-history.com/#NanmiCoder/CrawlerTutorial&Date)
diff --git a/static/images/100000001.png b/docs/static/images/100000001.png
similarity index 100%
rename from static/images/100000001.png
rename to docs/static/images/100000001.png
diff --git a/static/images/1000000010.png b/docs/static/images/1000000010.png
similarity index 100%
rename from static/images/1000000010.png
rename to docs/static/images/1000000010.png
diff --git a/static/images/1000000011.png b/docs/static/images/1000000011.png
similarity index 100%
rename from static/images/1000000011.png
rename to docs/static/images/1000000011.png
diff --git a/static/images/1000000012.png b/docs/static/images/1000000012.png
similarity index 100%
rename from static/images/1000000012.png
rename to docs/static/images/1000000012.png
diff --git a/docs/static/images/1000000013.png b/docs/static/images/1000000013.png
new file mode 100644
index 0000000..b3e82e1
Binary files /dev/null and b/docs/static/images/1000000013.png differ
diff --git a/docs/static/images/1000000014.png b/docs/static/images/1000000014.png
new file mode 100644
index 0000000..8d9b293
Binary files /dev/null and b/docs/static/images/1000000014.png differ
diff --git a/static/images/100000002.png b/docs/static/images/100000002.png
similarity index 100%
rename from static/images/100000002.png
rename to docs/static/images/100000002.png
diff --git a/static/images/100000003.png b/docs/static/images/100000003.png
similarity index 100%
rename from static/images/100000003.png
rename to docs/static/images/100000003.png
diff --git a/static/images/100000004.png b/docs/static/images/100000004.png
similarity index 100%
rename from static/images/100000004.png
rename to docs/static/images/100000004.png
diff --git a/static/images/100000005.png b/docs/static/images/100000005.png
similarity index 100%
rename from static/images/100000005.png
rename to docs/static/images/100000005.png
diff --git a/static/images/100000006.png b/docs/static/images/100000006.png
similarity index 100%
rename from static/images/100000006.png
rename to docs/static/images/100000006.png
diff --git a/static/images/100000007.png b/docs/static/images/100000007.png
similarity index 100%
rename from static/images/100000007.png
rename to docs/static/images/100000007.png
diff --git a/static/images/100000008.png b/docs/static/images/100000008.png
similarity index 100%
rename from static/images/100000008.png
rename to docs/static/images/100000008.png
diff --git a/static/images/100000009.png b/docs/static/images/100000009.png
similarity index 100%
rename from static/images/100000009.png
rename to docs/static/images/100000009.png
diff --git a/docs/static/images/QIWEI.png b/docs/static/images/QIWEI.png
new file mode 100644
index 0000000..4c33fba
Binary files /dev/null and b/docs/static/images/QIWEI.png differ
diff --git a/docs/static/images/qrcode/1.JPG b/docs/static/images/qrcode/1.JPG
new file mode 100644
index 0000000..6f88a2a
Binary files /dev/null and b/docs/static/images/qrcode/1.JPG differ
diff --git a/static/images/rumen_08/img.png b/docs/static/images/rumen_08/img.png
similarity index 100%
rename from static/images/rumen_08/img.png
rename to docs/static/images/rumen_08/img.png
diff --git a/static/images/rumen_08/img_1.png b/docs/static/images/rumen_08/img_1.png
similarity index 100%
rename from static/images/rumen_08/img_1.png
rename to docs/static/images/rumen_08/img_1.png
diff --git a/static/images/rumen_08/img_10.png b/docs/static/images/rumen_08/img_10.png
similarity index 100%
rename from static/images/rumen_08/img_10.png
rename to docs/static/images/rumen_08/img_10.png
diff --git a/static/images/rumen_08/img_11.png b/docs/static/images/rumen_08/img_11.png
similarity index 100%
rename from static/images/rumen_08/img_11.png
rename to docs/static/images/rumen_08/img_11.png
diff --git a/static/images/rumen_08/img_12.png b/docs/static/images/rumen_08/img_12.png
similarity index 100%
rename from static/images/rumen_08/img_12.png
rename to docs/static/images/rumen_08/img_12.png
diff --git a/static/images/rumen_08/img_2.png b/docs/static/images/rumen_08/img_2.png
similarity index 100%
rename from static/images/rumen_08/img_2.png
rename to docs/static/images/rumen_08/img_2.png
diff --git a/static/images/rumen_08/img_3.png b/docs/static/images/rumen_08/img_3.png
similarity index 100%
rename from static/images/rumen_08/img_3.png
rename to docs/static/images/rumen_08/img_3.png
diff --git a/static/images/rumen_08/img_4.png b/docs/static/images/rumen_08/img_4.png
similarity index 100%
rename from static/images/rumen_08/img_4.png
rename to docs/static/images/rumen_08/img_4.png
diff --git a/static/images/rumen_08/img_5.png b/docs/static/images/rumen_08/img_5.png
similarity index 100%
rename from static/images/rumen_08/img_5.png
rename to docs/static/images/rumen_08/img_5.png
diff --git a/static/images/rumen_08/img_6.png b/docs/static/images/rumen_08/img_6.png
similarity index 100%
rename from static/images/rumen_08/img_6.png
rename to docs/static/images/rumen_08/img_6.png
diff --git a/static/images/rumen_08/img_7.png b/docs/static/images/rumen_08/img_7.png
similarity index 100%
rename from static/images/rumen_08/img_7.png
rename to docs/static/images/rumen_08/img_7.png
diff --git a/static/images/rumen_08/img_8.png b/docs/static/images/rumen_08/img_8.png
similarity index 100%
rename from static/images/rumen_08/img_8.png
rename to docs/static/images/rumen_08/img_8.png
diff --git a/static/images/rumen_08/img_9.png b/docs/static/images/rumen_08/img_9.png
similarity index 100%
rename from static/images/rumen_08/img_9.png
rename to docs/static/images/rumen_08/img_9.png
diff --git a/static/images/wechat_pay.jpeg b/docs/static/images/wechat_pay.jpeg
similarity index 100%
rename from static/images/wechat_pay.jpeg
rename to docs/static/images/wechat_pay.jpeg
diff --git a/static/images/zfb_pay.png b/docs/static/images/zfb_pay.png
similarity index 100%
rename from static/images/zfb_pay.png
rename to docs/static/images/zfb_pay.png
diff --git "a/docs/static/images/\347\237\245\350\257\206\346\230\237\347\220\203.png" "b/docs/static/images/\347\237\245\350\257\206\346\230\237\347\220\203.png"
new file mode 100644
index 0000000..05a73b1
Binary files /dev/null and "b/docs/static/images/\347\237\245\350\257\206\346\230\237\347\220\203.png" differ
diff --git "a/\347\210\254\350\231\253\345\205\245\351\227\250/01_\344\270\272\344\273\200\344\271\210\350\246\201\345\206\231\350\277\231\344\270\252\347\210\254\350\231\253\346\225\231\347\250\213.md" "b/docs/\347\210\254\350\231\253\345\205\245\351\227\250/01_\344\270\272\344\273\200\344\271\210\350\246\201\345\206\231\350\277\231\344\270\252\347\210\254\350\231\253\346\225\231\347\250\213.md"
similarity index 98%
rename from "\347\210\254\350\231\253\345\205\245\351\227\250/01_\344\270\272\344\273\200\344\271\210\350\246\201\345\206\231\350\277\231\344\270\252\347\210\254\350\231\253\346\225\231\347\250\213.md"
rename to "docs/\347\210\254\350\231\253\345\205\245\351\227\250/01_\344\270\272\344\273\200\344\271\210\350\246\201\345\206\231\350\277\231\344\270\252\347\210\254\350\231\253\346\225\231\347\250\213.md"
index ec197fb..4c15483 100644
--- "a/\347\210\254\350\231\253\345\205\245\351\227\250/01_\344\270\272\344\273\200\344\271\210\350\246\201\345\206\231\350\277\231\344\270\252\347\210\254\350\231\253\346\225\231\347\250\213.md"
+++ "b/docs/\347\210\254\350\231\253\345\205\245\351\227\250/01_\344\270\272\344\273\200\344\271\210\350\246\201\345\206\231\350\277\231\344\270\252\347\210\254\350\231\253\346\225\231\347\250\213.md"
@@ -1,4 +1,4 @@
-## 为什么写这个爬虫教程？
+# 为什么写这个爬虫教程？
- 满足需求：我的[自媒体平台爬虫](https://github.com/NanmiCoder/MediaCrawler)爆火了之后，越来越多人私信我都是问想要学习爬虫时，我意识到了有很多人对这个爬虫领域感兴趣，但却不知道从何入手。因此决定撰写这个爬虫教程。
- 分享经验：首先我这个人虽然技术一般，但是真的很喜欢分享，平时工作中我学到了什么新知识都迫不及待的想要跟身边同事去分享，所以既然大家有需要，那么我会分享自己在爬虫领域的一些经验知识和见解。
diff --git "a/\347\210\254\350\231\253\345\205\245\351\227\250/02_\344\270\252\344\272\272\345\255\246\344\274\232\347\210\254\350\231\253\350\203\275\350\265\232\351\222\261\345\220\227.md" "b/docs/\347\210\254\350\231\253\345\205\245\351\227\250/02_\344\270\252\344\272\272\345\255\246\344\274\232\347\210\254\350\231\253\350\203\275\350\265\232\351\222\261\345\220\227.md"
similarity index 95%
rename from "\347\210\254\350\231\253\345\205\245\351\227\250/02_\344\270\252\344\272\272\345\255\246\344\274\232\347\210\254\350\231\253\350\203\275\350\265\232\351\222\261\345\220\227.md"
rename to "docs/\347\210\254\350\231\253\345\205\245\351\227\250/02_\344\270\252\344\272\272\345\255\246\344\274\232\347\210\254\350\231\253\350\203\275\350\265\232\351\222\261\345\220\227.md"
index 18112da..f897710 100644
--- "a/\347\210\254\350\231\253\345\205\245\351\227\250/02_\344\270\252\344\272\272\345\255\246\344\274\232\347\210\254\350\231\253\350\203\275\350\265\232\351\222\261\345\220\227.md"
+++ "b/docs/\347\210\254\350\231\253\345\205\245\351\227\250/02_\344\270\252\344\272\272\345\255\246\344\274\232\347\210\254\350\231\253\350\203\275\350\265\232\351\222\261\345\220\227.md"
@@ -1,4 +1,4 @@
-## 个人学会爬虫能赚钱吗？
+# 个人学会爬虫能赚钱吗？
**先说结论**
>肯定可以赚钱，关键在于挣得多还是少。掌握任何一门技术，都有赚钱的机会，因为技术需求一直存在。
@@ -13,7 +13,7 @@
爬虫挣钱的分类我基于我个人的经验总结了一下，大致分为三类，`兼职小活`、`长期雇佣关系`、`专职爬虫`，下面对这三类都做一个简单介绍
-### 1、兼职小活
+## 1、兼职小活
我们经常能在各种卖爬虫课直播间看到一些主播晒关于接单成交截图之类，其实这些截图我认为有真有假，可能大部分都是真的，因为市场是一个供需关系来的，有需求自然就有提供方，爬虫这个也不例外。
在现如今这个数据为王的时代。做事儿都需要有强有力的数据做支撑然后去做决策，而不是盲目的去做。我打个比方：
@@ -24,7 +24,7 @@
> 这类活我们可以把它统称为：兼职小活,特点：`客单价低`，`大多是一次性需求`，`花的时间相对少`。
-### 2、长期雇佣关系
+## 2、长期雇佣关系
长期雇佣关系这个从字面意思就比较好理解了，基于某一类固定需求，长期给需求方提供服务，这里我举一个我之前兼职遇到的客户吧
她是一个女生，她们公式是xxx相关的业务，她接到了领导的一个需求，需要定期出一份她们公司所在省份的xxx业务数据。
@@ -39,10 +39,10 @@
> 这类活我们可以把它统称为：长期雇佣关系, 特点：`客单价有高有低`，`大多为数据导出清洗需求`，`回购 or 复购`。
-### 3、专职爬虫
-这一类就比较直观了，就是找一个`爬虫工程师`的工作，工作内容一般为：数据采集、数据清晰、爬虫系统设计与架构。
+## 3、专职爬虫
+这一类就比较直观了，就是找一个`爬虫工程师`的工作，工作内容一般为：数据采集、数据清洗、爬虫系统设计与架构。
薪水的话拿一线城市举个例子吧，初级的话大概在8-14k 中高级 15 - 30k，当然这是一个参考区间，有的可能更高，有的可能稍微少点。
> 这一类专职爬虫，通常对技术要求比较高，需要对技术知识积累覆盖面光，前端、后端、数据库、安全、逆向、网络都要去涉及。
-> 当你覆盖了不同的技术知识积累，在面对不对的case时都能很快的给出解决方案，那么恭喜你你的薪水可能就会涨的很快。
\ No newline at end of file
+> 当你覆盖了不同的技术知识积累，在面对不对的case时都能很快的给出解决方案，那么恭喜你你的薪水可能就会涨的很快。
diff --git "a/\347\210\254\350\231\253\345\205\245\351\227\250/03_\347\275\221\347\273\234\347\210\254\350\231\253\345\210\260\345\272\225\346\230\257\344\273\200\344\271\210.md" "b/docs/\347\210\254\350\231\253\345\205\245\351\227\250/03_\347\275\221\347\273\234\347\210\254\350\231\253\345\210\260\345\272\225\346\230\257\344\273\200\344\271\210.md"
similarity index 91%
rename from "\347\210\254\350\231\253\345\205\245\351\227\250/03_\347\275\221\347\273\234\347\210\254\350\231\253\345\210\260\345\272\225\346\230\257\344\273\200\344\271\210.md"
rename to "docs/\347\210\254\350\231\253\345\205\245\351\227\250/03_\347\275\221\347\273\234\347\210\254\350\231\253\345\210\260\345\272\225\346\230\257\344\273\200\344\271\210.md"
index a447b68..d69762b 100644
--- "a/\347\210\254\350\231\253\345\205\245\351\227\250/03_\347\275\221\347\273\234\347\210\254\350\231\253\345\210\260\345\272\225\346\230\257\344\273\200\344\271\210.md"
+++ "b/docs/\347\210\254\350\231\253\345\205\245\351\227\250/03_\347\275\221\347\273\234\347\210\254\350\231\253\345\210\260\345\272\225\346\230\257\344\273\200\344\271\210.md"
@@ -1,7 +1,7 @@
-## 网络爬虫到底是什么？
+# 网络爬虫到底是什么？
-### 1、看看维基百科的定义
+## 1、看看维基百科的定义
> 网络爬虫（英语：web crawler），也叫网络蜘蛛（spider），是一种用来自动浏览万维网的网络机器人。其目的一般为编纂网络索引。文本
` | 数据不纯净 | +| 空白字符 | 多余空格、换行 | 格式混乱 | +| 编码问题 | 乱码、特殊字符 | 无法正常显示 | +| 重复数据 | 相同内容多次出现 | 数据冗余 | +| 格式不统一 | 日期格式各异 | 难以分析 | +| 缺失值 | 空字段 | 数据不完整 | + +### 1.2 数据清洗流程 + +```mermaid +flowchart LR + raw[原始数据] --> format[格式清洗] + format --> content[内容清洗] + content --> dedup[去重处理] + dedup --> normalize[标准化] + normalize --> clean[干净数据] + + subgraph 格式清洗 + html[HTML标签移除] + encode[编码修复] + end + + subgraph 内容清洗 + whitespace[空白处理] + special[特殊字符] + end +``` + +### 1.3 数据质量评估 + +在清洗前后，评估数据质量： + +```python +from dataclasses import dataclass +from typing import List, Dict, Any + +@dataclass +class DataQualityReport: + """数据质量报告""" + total_records: int + valid_records: int + duplicate_count: int + missing_fields: Dict[str, int] + format_issues: Dict[str, int] + + @property + def validity_rate(self) -> float: + """有效率""" + return self.valid_records / self.total_records if self.total_records > 0 else 0 + + @property + def duplicate_rate(self) -> float: + """重复率""" + return self.duplicate_count / self.total_records if self.total_records > 0 else 0 + + def __str__(self) -> str: + return f""" +数据质量报告: + 总记录数: {self.total_records} + 有效记录: {self.valid_records} ({self.validity_rate:.1%}) + 重复记录: {self.duplicate_count} ({self.duplicate_rate:.1%}) + 缺失字段: {self.missing_fields} + 格式问题: {self.format_issues} + """.strip() +``` + +## 二、文本清洗 + +### 2.1 HTML 标签移除 + +```python +import re +from typing import Optional + +class HTMLCleaner: + """HTML 清洗器""" + + # 需要完全移除的标签（包括内容） + REMOVE_TAGS = ['script', 'style', 'head', 'meta', 'link'] + + @staticmethod + def remove_tags(html: str) -> str: + """移除所有 HTML 标签""" + # 先移除特定标签及其内容 + for tag in HTMLCleaner.REMOVE_TAGS: + pattern = f'<{tag}[^>]*>.*?' + html = re.sub(pattern, '', html, flags=re.DOTALL | re.IGNORECASE) + + # 移除所有标签 + html = re.sub(r'<[^>]+>', '', html) + + return html + + @staticmethod + def remove_tags_keep_text(html: str) -> str: + """移除标签但保留文本内容""" + # 处理常见的块级元素，添加换行 + html = re.sub(r'', '\n', html, flags=re.IGNORECASE) + # 移除其他标签 + html = re.sub(r'<[^>]+>', '', html) + return html + + @staticmethod + def decode_entities(text: str) -> str: + """解码 HTML 实体""" + import html + return html.unescape(text) + + +# 使用 BeautifulSoup（更可靠） +def clean_html_with_bs4(html: str) -> str: + """使用 BeautifulSoup 清洗 HTML""" + try: + from bs4 import BeautifulSoup + soup = BeautifulSoup(html, 'html.parser') + + # 移除脚本和样式 + for script in soup(['script', 'style']): + script.decompose() + + # 获取文本 + text = soup.get_text(separator='\n') + return text + except ImportError: + return HTMLCleaner.remove_tags(html) +``` + +### 2.2 空白字符处理 + +```python +class WhitespaceCleaner: + """空白字符清洗器""" + + @staticmethod + def normalize_whitespace(text: str) -> str: + """标准化空白字符""" + # 将所有空白字符转为普通空格 + text = re.sub(r'[\t\r\f\v]', ' ', text) + # 合并多个空格 + text = re.sub(r' +', ' ', text) + # 合并多个换行 + text = re.sub(r'\n+', '\n', text) + # 去除首尾空白 + return text.strip() + + @staticmethod + def remove_all_whitespace(text: str) -> str: + """移除所有空白字符""" + return re.sub(r'\s+', '', text) + + @staticmethod + def trim_lines(text: str) -> str: + """去除每行首尾空白""" + lines = text.split('\n') + return '\n'.join(line.strip() for line in lines) + + @staticmethod + def remove_empty_lines(text: str) -> str: + """移除空行""" + lines = text.split('\n') + return '\n'.join(line for line in lines if line.strip()) +``` + +### 2.3 特殊字符清理 + +```python +import unicodedata + +class SpecialCharCleaner: + """特殊字符清洗器""" + + @staticmethod + def remove_control_chars(text: str) -> str: + """移除控制字符""" + return ''.join( + char for char in text + if unicodedata.category(char) != 'Cc' + ) + + @staticmethod + def normalize_unicode(text: str, form: str = 'NFKC') -> str: + """ + Unicode 标准化 + + Args: + text: 输入文本 + form: 标准化形式 (NFC/NFD/NFKC/NFKD) + + Returns: + 标准化后的文本 + """ + return unicodedata.normalize(form, text) + + @staticmethod + def remove_emojis(text: str) -> str: + """移除 emoji""" + emoji_pattern = re.compile( + "[" + "\U0001F600-\U0001F64F" # 表情 + "\U0001F300-\U0001F5FF" # 符号和象形文字 + "\U0001F680-\U0001F6FF" # 交通和地图 + "\U0001F1E0-\U0001F1FF" # 旗帜 + "\U00002702-\U000027B0" # 装饰符号 + "\U000024C2-\U0001F251" # 封闭字符 + "]+", + flags=re.UNICODE + ) + return emoji_pattern.sub('', text) + + @staticmethod + def to_halfwidth(text: str) -> str: + """全角转半角""" + result = [] + for char in text: + code = ord(char) + # 全角空格 + if code == 0x3000: + result.append(' ') + # 其他全角字符 + elif 0xFF01 <= code <= 0xFF5E: + result.append(chr(code - 0xFEE0)) + else: + result.append(char) + return ''.join(result) +``` + +### 2.4 编码问题处理 + +```python +import chardet + +class EncodingFixer: + """编码问题修复器""" + + @staticmethod + def detect_encoding(data: bytes) -> str: + """检测编码""" + result = chardet.detect(data) + return result['encoding'] or 'utf-8' + + @staticmethod + def fix_encoding(text: str, source_encoding: str = None) -> str: + """修复编码问题""" + try: + # 如果是乱码，尝试重新编码 + if source_encoding: + return text.encode('latin1').decode(source_encoding) + else: + # 尝试常见编码 + for encoding in ['utf-8', 'gbk', 'gb2312', 'big5']: + try: + return text.encode('latin1').decode(encoding) + except (UnicodeDecodeError, UnicodeEncodeError): + continue + except Exception: + pass + return text + + @staticmethod + def safe_decode(data: bytes, fallback: str = 'utf-8') -> str: + """安全解码""" + # 检测编码 + detected = EncodingFixer.detect_encoding(data) + try: + return data.decode(detected) + except (UnicodeDecodeError, TypeError): + return data.decode(fallback, errors='ignore') +``` + +## 三、正则表达式高级应用 + +### 3.1 常用提取模式 + +```python +class RegexPatterns: + """常用正则表达式模式""" + + # 基础模式 + CHINESE = r'[\u4e00-\u9fa5]+' # 中文 + EMAIL = r'[\w.+-]+@[\w-]+\.[\w.-]+' # 邮箱 + PHONE = r'1[3-9]\d{9}' # 中国手机号 + URL = r'https?://[^\s<>"{}|\\^`\[\]]+' # URL + IP = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}' # IP 地址 + + # 日期时间模式 + DATE_YMD = r'\d{4}[-/年]\d{1,2}[-/月]\d{1,2}日?' # 年月日 + TIME_HMS = r'\d{1,2}:\d{2}(:\d{2})?' # 时分秒 + DATETIME = r'\d{4}[-/]\d{1,2}[-/]\d{1,2}\s+\d{1,2}:\d{2}(:\d{2})?' + + # 数值模式 + INTEGER = r'-?\d+' + FLOAT = r'-?\d+\.?\d*' + PRICE = r'[¥$￥]\s*\d+\.?\d*' # 价格 + PERCENTAGE = r'\d+\.?\d*%' # 百分比 + + +class RegexExtractor: + """正则表达式提取器""" + + @staticmethod + def extract_all(text: str, pattern: str) -> list: + """提取所有匹配""" + return re.findall(pattern, text) + + @staticmethod + def extract_first(text: str, pattern: str) -> str: + """提取第一个匹配""" + match = re.search(pattern, text) + return match.group() if match else '' + + @staticmethod + def extract_groups(text: str, pattern: str) -> dict: + """提取命名分组""" + match = re.search(pattern, text) + return match.groupdict() if match else {} + + @staticmethod + def extract_between(text: str, start: str, end: str) -> list: + """提取两个标记之间的内容""" + pattern = f'{re.escape(start)}(.*?){re.escape(end)}' + return re.findall(pattern, text, re.DOTALL) +``` + +### 3.2 高级替换 + +```python +class RegexReplacer: + """正则表达式替换器""" + + @staticmethod + def replace_with_callback( + text: str, + pattern: str, + callback + ) -> str: + """使用回调函数替换""" + return re.sub(pattern, callback, text) + + @staticmethod + def mask_sensitive(text: str) -> str: + """脱敏处理""" + # 手机号脱敏 + text = re.sub( + r'(1[3-9]\d)\d{4}(\d{4})', + r'\1****\2', + text + ) + # 邮箱脱敏 + text = re.sub( + r'([\w.+-]{1,3})[\w.+-]*(@[\w-]+\.[\w.-]+)', + r'\1***\2', + text + ) + # 身份证脱敏 + text = re.sub( + r'(\d{6})\d{8}(\d{4})', + r'\1********\2', + text + ) + return text + + @staticmethod + def clean_url_params(url: str, keep_params: list = None) -> str: + """清理 URL 参数""" + from urllib.parse import urlparse, parse_qs, urlencode, urlunparse + + parsed = urlparse(url) + params = parse_qs(parsed.query) + + if keep_params: + params = {k: v for k, v in params.items() if k in keep_params} + else: + params = {} + + new_query = urlencode(params, doseq=True) + return urlunparse(parsed._replace(query=new_query)) +``` + +## 四、数据去重 + +### 4.1 精确去重 + +```python +from typing import List, Dict, Any, Callable +import hashlib + +class ExactDeduplicator: + """精确去重器""" + + @staticmethod + def dedupe_list(items: List[str]) -> List[str]: + """列表去重（保持顺序）""" + seen = set() + result = [] + for item in items: + if item not in seen: + seen.add(item) + result.append(item) + return result + + @staticmethod + def dedupe_dicts( + items: List[Dict], + key_field: str + ) -> List[Dict]: + """字典列表去重""" + seen = set() + result = [] + for item in items: + key = item.get(key_field) + if key not in seen: + seen.add(key) + result.append(item) + return result + + @staticmethod + def dedupe_by_hash( + items: List[Dict], + fields: List[str] + ) -> List[Dict]: + """ + 根据多个字段计算哈希去重 + + Args: + items: 数据列表 + fields: 用于计算哈希的字段 + + Returns: + 去重后的列表 + """ + seen = set() + result = [] + + for item in items: + # 计算哈希 + key_str = '|'.join(str(item.get(f, '')) for f in fields) + key_hash = hashlib.md5(key_str.encode()).hexdigest() + + if key_hash not in seen: + seen.add(key_hash) + result.append(item) + + return result +``` + +### 4.2 模糊去重 + +```python +class FuzzyDeduplicator: + """模糊去重器""" + + @staticmethod + def levenshtein_distance(s1: str, s2: str) -> int: + """计算编辑距离""" + if len(s1) < len(s2): + return FuzzyDeduplicator.levenshtein_distance(s2, s1) + + if len(s2) == 0: + return len(s1) + + previous_row = range(len(s2) + 1) + for i, c1 in enumerate(s1): + current_row = [i + 1] + for j, c2 in enumerate(s2): + insertions = previous_row[j + 1] + 1 + deletions = current_row[j] + 1 + substitutions = previous_row[j] + (c1 != c2) + current_row.append(min(insertions, deletions, substitutions)) + previous_row = current_row + + return previous_row[-1] + + @staticmethod + def similarity(s1: str, s2: str) -> float: + """计算相似度 (0-1)""" + if not s1 or not s2: + return 0.0 + distance = FuzzyDeduplicator.levenshtein_distance(s1, s2) + max_len = max(len(s1), len(s2)) + return 1 - distance / max_len + + @staticmethod + def dedupe_fuzzy( + items: List[str], + threshold: float = 0.8 + ) -> List[str]: + """ + 模糊去重 + + Args: + items: 字符串列表 + threshold: 相似度阈值 + + Returns: + 去重后的列表 + """ + if not items: + return [] + + result = [items[0]] + + for item in items[1:]: + is_duplicate = False + for existing in result: + if FuzzyDeduplicator.similarity(item, existing) >= threshold: + is_duplicate = True + break + if not is_duplicate: + result.append(item) + + return result +``` + +## 五、数据标准化 + +### 5.1 日期时间标准化 + +```python +from datetime import datetime +from typing import Optional + +class DateTimeNormalizer: + """日期时间标准化器""" + + # 常见日期格式 + DATE_FORMATS = [ + '%Y-%m-%d', + '%Y/%m/%d', + '%Y年%m月%d日', + '%Y.%m.%d', + '%d-%m-%Y', + '%m/%d/%Y', + ] + + # 常见日期时间格式 + DATETIME_FORMATS = [ + '%Y-%m-%d %H:%M:%S', + '%Y-%m-%d %H:%M', + '%Y/%m/%d %H:%M:%S', + '%Y年%m月%d日 %H:%M:%S', + '%Y年%m月%d日 %H时%M分', + ] + + @classmethod + def parse_date(cls, text: str) -> Optional[datetime]: + """解析日期""" + text = text.strip() + + for fmt in cls.DATE_FORMATS + cls.DATETIME_FORMATS: + try: + return datetime.strptime(text, fmt) + except ValueError: + continue + + return None + + @classmethod + def normalize_date( + cls, + text: str, + output_format: str = '%Y-%m-%d' + ) -> str: + """标准化日期格式""" + dt = cls.parse_date(text) + if dt: + return dt.strftime(output_format) + return text + + @staticmethod + def parse_relative_time(text: str) -> Optional[datetime]: + """解析相对时间（如"3小时前"）""" + import re + from datetime import timedelta + + now = datetime.now() + patterns = [ + (r'(\d+)\s*秒前', lambda m: now - timedelta(seconds=int(m.group(1)))), + (r'(\d+)\s*分钟前', lambda m: now - timedelta(minutes=int(m.group(1)))), + (r'(\d+)\s*小时前', lambda m: now - timedelta(hours=int(m.group(1)))), + (r'(\d+)\s*天前', lambda m: now - timedelta(days=int(m.group(1)))), + (r'刚刚', lambda m: now), + (r'昨天', lambda m: now - timedelta(days=1)), + (r'前天', lambda m: now - timedelta(days=2)), + ] + + for pattern, handler in patterns: + match = re.search(pattern, text) + if match: + return handler(match) + + return None +``` + +### 5.2 数值标准化 + +```python +class NumberNormalizer: + """数值标准化器""" + + @staticmethod + def parse_number(text: str) -> float: + """ + 解析数字（支持中文单位） + + Examples: + "1.5万" -> 15000 + "3.2亿" -> 320000000 + "1,234.56" -> 1234.56 + """ + text = text.strip() + + # 中文单位映射 + units = { + '万': 10000, + '亿': 100000000, + 'k': 1000, + 'K': 1000, + 'm': 1000000, + 'M': 1000000, + 'b': 1000000000, + 'B': 1000000000, + } + + multiplier = 1 + for unit, value in units.items(): + if unit in text: + multiplier = value + text = text.replace(unit, '') + break + + # 移除逗号 + text = text.replace(',', '') + + # 提取数字 + match = re.search(r'-?\d+\.?\d*', text) + if match: + return float(match.group()) * multiplier + + return 0.0 + + @staticmethod + def format_number( + value: float, + precision: int = 2, + use_units: bool = True + ) -> str: + """格式化数字""" + if not use_units: + return f'{value:.{precision}f}' + + if value >= 100000000: + return f'{value/100000000:.{precision}f}亿' + elif value >= 10000: + return f'{value/10000:.{precision}f}万' + else: + return f'{value:.{precision}f}' +``` + +### 5.3 文本标准化 + +```python +class TextNormalizer: + """文本标准化器""" + + @staticmethod + def normalize(text: str) -> str: + """完整的文本标准化流程""" + # 1. Unicode 标准化 + text = unicodedata.normalize('NFKC', text) + # 2. 全角转半角 + text = SpecialCharCleaner.to_halfwidth(text) + # 3. 移除控制字符 + text = SpecialCharCleaner.remove_control_chars(text) + # 4. 标准化空白 + text = WhitespaceCleaner.normalize_whitespace(text) + return text + + @staticmethod + def normalize_punctuation(text: str) -> str: + """标准化标点符号""" + # 中文标点转英文 + mapping = { + '，': ', ', + '。': '. ', + '！': '! ', + '？': '? ', + '；': '; ', + '：': ': ', + '"': '"', + '"': '"', + ''': "'", + ''': "'", + } + for cn, en in mapping.items(): + text = text.replace(cn, en) + return text +``` + +## 六、综合数据清洗器 + +```python +from dataclasses import dataclass, field +from typing import List, Dict, Any, Callable + +@dataclass +class CleaningConfig: + """清洗配置""" + remove_html: bool = True + normalize_whitespace: bool = True + normalize_unicode: bool = True + remove_emojis: bool = False + to_halfwidth: bool = True + +class DataCleaner: + """综合数据清洗器""" + + def __init__(self, config: CleaningConfig = None): + self.config = config or CleaningConfig() + + def clean_text(self, text: str) -> str: + """清洗文本""" + if not text: + return '' + + # HTML 清洗 + if self.config.remove_html: + text = HTMLCleaner.remove_tags(text) + text = HTMLCleaner.decode_entities(text) + + # Unicode 标准化 + if self.config.normalize_unicode: + text = SpecialCharCleaner.normalize_unicode(text) + + # 全角转半角 + if self.config.to_halfwidth: + text = SpecialCharCleaner.to_halfwidth(text) + + # 移除 emoji + if self.config.remove_emojis: + text = SpecialCharCleaner.remove_emojis(text) + + # 空白处理 + if self.config.normalize_whitespace: + text = WhitespaceCleaner.normalize_whitespace(text) + + return text + + def clean_dict(self, data: Dict, text_fields: List[str] = None) -> Dict: + """清洗字典中的文本字段""" + result = data.copy() + text_fields = text_fields or list(data.keys()) + + for field in text_fields: + if field in result and isinstance(result[field], str): + result[field] = self.clean_text(result[field]) + + return result + + def clean_list( + self, + items: List[Dict], + text_fields: List[str] = None + ) -> List[Dict]: + """清洗字典列表""" + return [self.clean_dict(item, text_fields) for item in items] +``` + +## 七、实战示例 + +```python +import asyncio + +async def demo_data_cleaning(): + """数据清洗演示""" + + # 模拟爬取的原始数据 + raw_data = [ + { + "title": "这是一篇&关于爬虫的教程。
", + "date": "2024年1月15日", + "price": "￥99.00", + "views": "1.5万" + }, + { + "title": "这是另一篇关于爬虫的教程。
", + "date": "2024-01-15", + "price": "99元", + "views": "15000" + } + ] + + print("原始数据:") + for item in raw_data: + print(f" {item}") + + # 1. 文本清洗 + cleaner = DataCleaner(CleaningConfig( + remove_html=True, + normalize_whitespace=True, + to_halfwidth=True + )) + + cleaned_data = cleaner.clean_list(raw_data, ["title", "content"]) + + print("\n清洗后数据:") + for item in cleaned_data: + print(f" {item}") + + # 2. 日期标准化 + for item in cleaned_data: + item["date"] = DateTimeNormalizer.normalize_date(item["date"]) + + print("\n日期标准化后:") + for item in cleaned_data: + print(f" date: {item['date']}") + + # 3. 数值标准化 + for item in cleaned_data: + item["views_num"] = NumberNormalizer.parse_number(item["views"]) + + print("\n数值标准化后:") + for item in cleaned_data: + print(f" views: {item['views']} -> {item['views_num']}") + + # 4. 去重 + deduped = ExactDeduplicator.dedupe_by_hash(cleaned_data, ["title"]) + print(f"\n去重后记录数: {len(deduped)}") + + +if __name__ == "__main__": + asyncio.run(demo_data_cleaning()) +``` + +## 八、数据清洗实战 + +本节以网页数据为例，演示完整的数据清洗流程。 + +### 8.1 数据特点分析 + +```mermaid +flowchart TD + subgraph 原始数据问题 + title_em["标题含HTML标签这是一段 HTML 文本。
+包含&特殊<字符>。
+ + + """ + + print("\n1. HTML 清洗:") + print(f" 原始: {test_html[:50]}...") + cleaned = HTMLCleaner.clean_with_bs4(test_html) + print(f" 清洗后: {cleaned[:50]}...") + + print("\n2. 空白处理:") + test_whitespace = " 多个 空格 \n\n\n多个换行 " + print(f" 原始: '{test_whitespace}'") + normalized = WhitespaceCleaner.normalize(test_whitespace) + print(f" 标准化: '{normalized}'") + + print("\n3. 全角转半角:") + test_fullwidth = "ＡＢＣ１２３ 全角字符" + print(f" 原始: '{test_fullwidth}'") + halfwidth = SpecialCharCleaner.to_halfwidth(test_fullwidth) + print(f" 半角: '{halfwidth}'") + + print("\n4. 综合清洗:") + cleaner = TextCleaner() + test_complex = "复杂的 HTML&文本
" + print(f" 原始: '{test_complex}'") + result = cleaner.clean(test_complex) + print(f" 清洗: '{result}'") + + print("\n" + "=" * 50) + print("演示完成") + print("=" * 50) + + +if __name__ == "__main__": + demo() diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/README.md" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/README.md" new file mode 100644 index 0000000..af8b599 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/README.md" @@ -0,0 +1,27 @@ +# 第10章：数据分析与可视化 + +展示Pandas数据分析、词云生成、图表制作等数据处理功能。 + +## 快速开始 + +```bash +cd 10_数据分析与可视化 + +# 安装基础依赖 +uv sync + +# 安装交互式图表功能（可选） +uv sync --extra interactive + +# 运行示例 +uv run python pandas_analysis.py +uv run python wordcloud_generator.py +uv run python chart_demo.py +``` + +### 核心依赖 +- `pandas` - 数据分析 +- `jieba` - 中文分词 +- `wordcloud` - 词云生成 +- `matplotlib` - 图表绘制 +- `pyecharts`（可选）- 交互式图表 diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/chart_demo.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/chart_demo.py" new file mode 100644 index 0000000..36617c1 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/chart_demo.py" @@ -0,0 +1,582 @@ +# -*- coding: utf-8 -*- +# @Desc: 数据可视化工具 + +import os +from typing import List, Dict, Any, Optional, Tuple +from loguru import logger + +# 可选依赖 +try: + import matplotlib.pyplot as plt + import matplotlib + matplotlib.rcParams['font.sans-serif'] = ['SimHei', 'Arial Unicode MS', 'DejaVu Sans'] + matplotlib.rcParams['axes.unicode_minus'] = False + HAS_MATPLOTLIB = True +except ImportError: + HAS_MATPLOTLIB = False + logger.warning("matplotlib 未安装") + +try: + from pyecharts.charts import Bar, Line, Pie, Scatter, WordCloud as PyWordCloud + from pyecharts import options as opts + from pyecharts.globals import ThemeType + HAS_PYECHARTS = True +except ImportError: + HAS_PYECHARTS = False + logger.warning("pyecharts 未安装") + + +class MatplotlibCharts: + """Matplotlib 静态图表生成器""" + + def __init__(self, figsize: Tuple[int, int] = (10, 6), dpi: int = 150): + """ + 初始化 + + Args: + figsize: 图表大小 (宽, 高) + dpi: 分辨率 + """ + if not HAS_MATPLOTLIB: + raise ImportError("请安装 matplotlib: pip install matplotlib") + + self.figsize = figsize + self.dpi = dpi + + def line_chart( + self, + x_data: List, + y_data: List, + title: str, + xlabel: str, + ylabel: str, + output_path: str, + color: str = 'steelblue', + marker: str = 'o' + ) -> str: + """绘制折线图""" + plt.figure(figsize=self.figsize) + plt.plot(x_data, y_data, marker=marker, linewidth=2, markersize=6, color=color) + plt.title(title, fontsize=14) + plt.xlabel(xlabel, fontsize=12) + plt.ylabel(ylabel, fontsize=12) + plt.grid(True, alpha=0.3) + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"折线图已保存: {output_path}") + return output_path + + def multi_line_chart( + self, + x_data: List, + y_data_dict: Dict[str, List], + title: str, + xlabel: str, + ylabel: str, + output_path: str + ) -> str: + """绘制多条折线图""" + plt.figure(figsize=self.figsize) + + for label, y_data in y_data_dict.items(): + plt.plot(x_data, y_data, marker='o', label=label, linewidth=2) + + plt.title(title, fontsize=14) + plt.xlabel(xlabel, fontsize=12) + plt.ylabel(ylabel, fontsize=12) + plt.legend(loc='best') + plt.grid(True, alpha=0.3) + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"多折线图已保存: {output_path}") + return output_path + + def bar_chart( + self, + categories: List[str], + values: List, + title: str, + xlabel: str, + ylabel: str, + output_path: str, + horizontal: bool = False, + color: str = 'steelblue' + ) -> str: + """绘制柱状图""" + plt.figure(figsize=self.figsize) + + if horizontal: + plt.barh(categories, values, color=color) + plt.xlabel(ylabel) + plt.ylabel(xlabel) + else: + plt.bar(categories, values, color=color) + plt.xlabel(xlabel) + plt.ylabel(ylabel) + + plt.title(title, fontsize=14) + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"柱状图已保存: {output_path}") + return output_path + + def grouped_bar_chart( + self, + categories: List[str], + data_dict: Dict[str, List], + title: str, + xlabel: str, + ylabel: str, + output_path: str + ) -> str: + """绘制分组柱状图""" + import numpy as np + + plt.figure(figsize=self.figsize) + + x = np.arange(len(categories)) + width = 0.8 / len(data_dict) + + for i, (label, values) in enumerate(data_dict.items()): + offset = (i - len(data_dict) / 2 + 0.5) * width + plt.bar(x + offset, values, width, label=label) + + plt.xlabel(xlabel, fontsize=12) + plt.ylabel(ylabel, fontsize=12) + plt.title(title, fontsize=14) + plt.xticks(x, categories) + plt.legend() + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"分组柱状图已保存: {output_path}") + return output_path + + def pie_chart( + self, + labels: List[str], + sizes: List, + title: str, + output_path: str, + explode_max: bool = True + ) -> str: + """绘制饼图""" + plt.figure(figsize=(10, 8)) + + explode = None + if explode_max: + max_idx = sizes.index(max(sizes)) + explode = [0.05 if i == max_idx else 0 for i in range(len(sizes))] + + plt.pie( + sizes, + labels=labels, + explode=explode, + autopct='%1.1f%%', + startangle=90, + colors=plt.cm.Set3.colors[:len(labels)] + ) + plt.title(title, fontsize=14) + plt.axis('equal') + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"饼图已保存: {output_path}") + return output_path + + def scatter_chart( + self, + x_data: List, + y_data: List, + title: str, + xlabel: str, + ylabel: str, + output_path: str, + color: str = 'steelblue', + alpha: float = 0.6 + ) -> str: + """绘制散点图""" + plt.figure(figsize=self.figsize) + plt.scatter(x_data, y_data, c=color, alpha=alpha, s=50) + plt.title(title, fontsize=14) + plt.xlabel(xlabel, fontsize=12) + plt.ylabel(ylabel, fontsize=12) + plt.grid(True, alpha=0.3) + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"散点图已保存: {output_path}") + return output_path + + def histogram( + self, + data: List, + title: str, + xlabel: str, + ylabel: str, + output_path: str, + bins: int = 20, + color: str = 'steelblue' + ) -> str: + """绘制直方图""" + plt.figure(figsize=self.figsize) + plt.hist(data, bins=bins, color=color, edgecolor='white', alpha=0.7) + plt.title(title, fontsize=14) + plt.xlabel(xlabel, fontsize=12) + plt.ylabel(ylabel, fontsize=12) + plt.grid(axis='y', alpha=0.3) + plt.tight_layout() + plt.savefig(output_path, dpi=self.dpi) + plt.close() + logger.info(f"直方图已保存: {output_path}") + return output_path + + +class PyechartsCharts: + """Pyecharts 交互式图表生成器""" + + def __init__(self, theme: str = 'light'): + """ + 初始化 + + Args: + theme: 主题名称 + """ + if not HAS_PYECHARTS: + raise ImportError("请安装 pyecharts: pip install pyecharts") + + self.theme = getattr(ThemeType, theme.upper(), ThemeType.LIGHT) + + def bar_chart( + self, + categories: List[str], + values: List, + title: str, + output_path: str, + series_name: str = "数量" + ) -> str: + """创建交互式柱状图""" + bar = ( + Bar(init_opts=opts.InitOpts(theme=self.theme)) + .add_xaxis(categories) + .add_yaxis(series_name, values) + .set_global_opts( + title_opts=opts.TitleOpts(title=title), + toolbox_opts=opts.ToolboxOpts(is_show=True), + datazoom_opts=opts.DataZoomOpts(is_show=True) + ) + ) + bar.render(output_path) + logger.info(f"交互式柱状图已保存: {output_path}") + return output_path + + def stacked_bar_chart( + self, + categories: List[str], + data_dict: Dict[str, List], + title: str, + output_path: str + ) -> str: + """创建堆叠柱状图""" + bar = Bar(init_opts=opts.InitOpts(theme=self.theme)) + bar.add_xaxis(categories) + + for name, values in data_dict.items(): + bar.add_yaxis(name, values, stack="stack1") + + bar.set_global_opts( + title_opts=opts.TitleOpts(title=title), + toolbox_opts=opts.ToolboxOpts(is_show=True), + legend_opts=opts.LegendOpts(is_show=True) + ) + bar.render(output_path) + logger.info(f"堆叠柱状图已保存: {output_path}") + return output_path + + def line_chart( + self, + x_data: List, + y_data_dict: Dict[str, List], + title: str, + output_path: str + ) -> str: + """创建交互式折线图""" + line = Line(init_opts=opts.InitOpts(theme=self.theme)) + line.add_xaxis(x_data) + + for name, y_data in y_data_dict.items(): + line.add_yaxis( + name, + y_data, + is_smooth=True, + markpoint_opts=opts.MarkPointOpts( + data=[ + opts.MarkPointItem(type_="max", name="最大值"), + opts.MarkPointItem(type_="min", name="最小值"), + ] + ), + markline_opts=opts.MarkLineOpts( + data=[opts.MarkLineItem(type_="average", name="平均值")] + ) + ) + + line.set_global_opts( + title_opts=opts.TitleOpts(title=title), + tooltip_opts=opts.TooltipOpts(trigger="axis"), + toolbox_opts=opts.ToolboxOpts(is_show=True), + legend_opts=opts.LegendOpts(is_show=True) + ) + line.render(output_path) + logger.info(f"交互式折线图已保存: {output_path}") + return output_path + + def pie_chart( + self, + data: List[Tuple[str, int]], + title: str, + output_path: str, + rose_type: str = None + ) -> str: + """ + 创建交互式饼图 + + Args: + data: [(名称, 数值), ...] 列表 + title: 标题 + output_path: 输出路径 + rose_type: 玫瑰图类型 ('radius' 或 'area') + """ + pie = Pie(init_opts=opts.InitOpts(theme=self.theme)) + + if rose_type: + pie.add( + "", + data, + radius=["30%", "70%"], + rosetype=rose_type + ) + else: + pie.add("", data, radius=["30%", "70%"]) + + pie.set_global_opts( + title_opts=opts.TitleOpts(title=title), + legend_opts=opts.LegendOpts( + orient="vertical", + pos_top="15%", + pos_left="2%" + ) + ) + pie.set_series_opts( + label_opts=opts.LabelOpts(formatter="{b}: {d}%") + ) + pie.render(output_path) + logger.info(f"交互式饼图已保存: {output_path}") + return output_path + + def scatter_chart( + self, + data: List[Tuple[float, float]], + title: str, + output_path: str, + series_name: str = "数据" + ) -> str: + """创建交互式散点图""" + scatter = ( + Scatter(init_opts=opts.InitOpts(theme=self.theme)) + .add_xaxis([d[0] for d in data]) + .add_yaxis(series_name, [d[1] for d in data]) + .set_global_opts( + title_opts=opts.TitleOpts(title=title), + toolbox_opts=opts.ToolboxOpts(is_show=True), + xaxis_opts=opts.AxisOpts(type_="value"), + ) + ) + scatter.render(output_path) + logger.info(f"交互式散点图已保存: {output_path}") + return output_path + + def wordcloud_chart( + self, + words: List[Tuple[str, int]], + title: str, + output_path: str, + shape: str = "circle" + ) -> str: + """ + 创建交互式词云 + + Args: + words: [(词语, 频次), ...] 列表 + title: 标题 + output_path: 输出路径 + shape: 形状 ('circle', 'cardioid', 'diamond', 'triangle-forward', 'triangle', 'pentagon', 'star') + """ + wc = ( + PyWordCloud(init_opts=opts.InitOpts(theme=self.theme)) + .add( + "", + words, + word_size_range=[20, 100], + shape=shape + ) + .set_global_opts( + title_opts=opts.TitleOpts(title=title), + toolbox_opts=opts.ToolboxOpts(is_show=True) + ) + ) + wc.render(output_path) + logger.info(f"交互式词云已保存: {output_path}") + return output_path + + +class ChartFactory: + """图表工厂类""" + + def __init__(self, output_dir: str = "./charts"): + """ + 初始化 + + Args: + output_dir: 输出目录 + """ + self.output_dir = output_dir + os.makedirs(output_dir, exist_ok=True) + + self.matplotlib = MatplotlibCharts() if HAS_MATPLOTLIB else None + self.pyecharts = PyechartsCharts() if HAS_PYECHARTS else None + + def get_output_path(self, filename: str) -> str: + """获取输出路径""" + return os.path.join(self.output_dir, filename) + + def create_bar( + self, + categories: List[str], + values: List, + title: str, + filename: str, + chart_type: str = 'matplotlib' + ) -> str: + """创建柱状图""" + output_path = self.get_output_path(filename) + + if chart_type == 'matplotlib' and self.matplotlib: + return self.matplotlib.bar_chart( + categories, values, title, "分类", "数值", output_path + ) + elif chart_type == 'pyecharts' and self.pyecharts: + return self.pyecharts.bar_chart(categories, values, title, output_path) + else: + raise ValueError(f"Chart type '{chart_type}' not available") + + def create_line( + self, + x_data: List, + y_data_dict: Dict[str, List], + title: str, + filename: str, + chart_type: str = 'matplotlib' + ) -> str: + """创建折线图""" + output_path = self.get_output_path(filename) + + if chart_type == 'matplotlib' and self.matplotlib: + return self.matplotlib.multi_line_chart( + x_data, y_data_dict, title, "X轴", "Y轴", output_path + ) + elif chart_type == 'pyecharts' and self.pyecharts: + return self.pyecharts.line_chart(x_data, y_data_dict, title, output_path) + else: + raise ValueError(f"Chart type '{chart_type}' not available") + + def create_pie( + self, + data: List[Tuple[str, int]], + title: str, + filename: str, + chart_type: str = 'matplotlib' + ) -> str: + """创建饼图""" + output_path = self.get_output_path(filename) + + if chart_type == 'matplotlib' and self.matplotlib: + labels = [d[0] for d in data] + sizes = [d[1] for d in data] + return self.matplotlib.pie_chart(labels, sizes, title, output_path) + elif chart_type == 'pyecharts' and self.pyecharts: + return self.pyecharts.pie_chart(data, title, output_path) + else: + raise ValueError(f"Chart type '{chart_type}' not available") + + +def demo(): + """演示图表生成功能""" + print("=" * 50) + print("数据可视化工具演示") + print("=" * 50) + + # 测试数据 + categories = ["Python", "Java", "JavaScript", "Go", "Rust"] + values = [85, 70, 75, 45, 30] + + x_data = ["1月", "2月", "3月", "4月", "5月"] + y_data = { + "浏览量": [1000, 1200, 1500, 1300, 1800], + "点赞数": [100, 150, 200, 180, 250] + } + + pie_data = [ + ("技术", 45), + ("生活", 30), + ("娱乐", 15), + ("其他", 10) + ] + + print("\n可用的图表类型:") + + if HAS_MATPLOTLIB: + print(" - Matplotlib (静态图表): 已安装") + print(" 示例: charts.matplotlib.bar_chart(...)") + else: + print(" - Matplotlib: 未安装 (pip install matplotlib)") + + if HAS_PYECHARTS: + print(" - Pyecharts (交互式图表): 已安装") + print(" 示例: charts.pyecharts.bar_chart(...)") + else: + print(" - Pyecharts: 未安装 (pip install pyecharts)") + + print("\n示例数据:") + print(f" 柱状图数据: {dict(zip(categories, values))}") + print(f" 折线图数据: {y_data}") + print(f" 饼图数据: {pie_data}") + + print("\n使用示例:") + print(""" + from chart_demo import ChartFactory + + factory = ChartFactory(output_dir="./output") + + # 创建柱状图 + factory.create_bar(categories, values, "编程语言流行度", "bar.png") + + # 创建折线图 + factory.create_line(x_data, y_data, "月度数据趋势", "line.png") + + # 创建饼图 + factory.create_pie(pie_data, "内容分类分布", "pie.png") + + # 创建交互式图表 (HTML) + factory.create_bar(categories, values, "编程语言流行度", "bar.html", chart_type='pyecharts') + """) + + print("\n" + "=" * 50) + print("演示完成") + print("=" * 50) + + +if __name__ == "__main__": + demo() diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/pandas_analysis.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/pandas_analysis.py" new file mode 100644 index 0000000..b4f90e9 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/pandas_analysis.py" @@ -0,0 +1,353 @@ +# -*- coding: utf-8 -*- +# @Desc: pandas 数据分析工具 + +import pandas as pd +from typing import List, Dict, Any, Optional +from datetime import datetime +from loguru import logger + + +class DataFrameAnalyzer: + """DataFrame 数据分析器""" + + def __init__(self, data: List[Dict[str, Any]]): + """ + 初始化分析器 + + Args: + data: 字典列表形式的数据 + """ + self.df = pd.DataFrame(data) + logger.info(f"加载数据: {len(self.df)} 行, {len(self.df.columns)} 列") + + @classmethod + def from_csv(cls, file_path: str, **kwargs) -> 'DataFrameAnalyzer': + """从 CSV 文件创建分析器""" + df = pd.read_csv(file_path, **kwargs) + return cls(df.to_dict('records')) + + @classmethod + def from_json(cls, file_path: str, **kwargs) -> 'DataFrameAnalyzer': + """从 JSON 文件创建分析器""" + df = pd.read_json(file_path, **kwargs) + return cls(df.to_dict('records')) + + def info(self) -> Dict[str, Any]: + """获取数据集基本信息""" + return { + "rows": len(self.df), + "columns": len(self.df.columns), + "column_names": self.df.columns.tolist(), + "dtypes": self.df.dtypes.to_dict(), + "memory_usage": self.df.memory_usage(deep=True).sum(), + "null_counts": self.df.isnull().sum().to_dict(), + } + + def describe(self, include: str = 'all') -> pd.DataFrame: + """获取描述性统计""" + return self.df.describe(include=include) + + def value_counts(self, column: str, top_n: int = 10) -> pd.Series: + """统计某列的值分布""" + return self.df[column].value_counts().head(top_n) + + def group_by_count(self, group_col: str) -> pd.Series: + """按列分组计数""" + return self.df.groupby(group_col).size().sort_values(ascending=False) + + def group_by_sum(self, group_col: str, value_col: str) -> pd.Series: + """按列分组求和""" + return self.df.groupby(group_col)[value_col].sum().sort_values(ascending=False) + + def group_by_mean(self, group_col: str, value_col: str) -> pd.Series: + """按列分组求均值""" + return self.df.groupby(group_col)[value_col].mean().sort_values(ascending=False) + + def group_by_agg( + self, + group_col: str, + agg_dict: Dict[str, List[str]] + ) -> pd.DataFrame: + """ + 按列分组并进行多种聚合 + + Args: + group_col: 分组列 + agg_dict: 聚合字典，如 {'views': ['sum', 'mean'], 'likes': ['max', 'min']} + """ + return self.df.groupby(group_col).agg(agg_dict) + + def pivot_table( + self, + values: str, + index: str, + columns: str, + aggfunc: str = 'sum', + fill_value: Any = 0 + ) -> pd.DataFrame: + """创建数据透视表""" + return pd.pivot_table( + self.df, + values=values, + index=index, + columns=columns, + aggfunc=aggfunc, + fill_value=fill_value + ) + + def correlation(self, columns: List[str] = None) -> pd.DataFrame: + """计算相关系数矩阵""" + if columns: + return self.df[columns].corr() + return self.df.select_dtypes(include=['number']).corr() + + def filter(self, condition: str) -> 'DataFrameAnalyzer': + """ + 根据条件筛选数据 + + Args: + condition: 查询条件，如 "views > 1000 and category == '技术'" + """ + filtered_df = self.df.query(condition) + return DataFrameAnalyzer(filtered_df.to_dict('records')) + + def sort(self, by: str, ascending: bool = False) -> 'DataFrameAnalyzer': + """排序""" + sorted_df = self.df.sort_values(by=by, ascending=ascending) + return DataFrameAnalyzer(sorted_df.to_dict('records')) + + def top_n(self, n: int, by: str) -> pd.DataFrame: + """获取 Top N""" + return self.df.nlargest(n, by) + + def bottom_n(self, n: int, by: str) -> pd.DataFrame: + """获取 Bottom N""" + return self.df.nsmallest(n, by) + + +class TimeSeriesAnalyzer: + """时间序列分析器""" + + def __init__(self, df: pd.DataFrame, date_col: str): + """ + 初始化时间序列分析器 + + Args: + df: 数据框 + date_col: 日期列名 + """ + self.df = df.copy() + self.date_col = date_col + + # 转换日期列 + self.df[date_col] = pd.to_datetime(self.df[date_col], errors='coerce') + self.df = self.df.dropna(subset=[date_col]) + self.df = self.df.set_index(date_col) + + logger.info(f"时间序列分析器初始化: {len(self.df)} 条记录") + + def resample( + self, + freq: str, + value_col: str, + agg_func: str = 'sum' + ) -> pd.DataFrame: + """ + 重采样 + + Args: + freq: 频率，如 'D'(天), 'W'(周), 'M'(月) + value_col: 值列 + agg_func: 聚合函数 + """ + return self.df.resample(freq)[value_col].agg(agg_func) + + def rolling( + self, + window: int, + value_col: str, + agg_func: str = 'mean' + ) -> pd.Series: + """ + 滚动窗口统计 + + Args: + window: 窗口大小 + value_col: 值列 + agg_func: 聚合函数 + """ + return getattr(self.df[value_col].rolling(window=window), agg_func)() + + def growth_rate(self, value_col: str, periods: int = 1) -> pd.Series: + """ + 计算增长率 + + Args: + value_col: 值列 + periods: 周期数 + """ + return self.df[value_col].pct_change(periods=periods) * 100 + + def trend_analysis(self, value_col: str) -> Dict[str, Any]: + """ + 趋势分析 + + Returns: + 包含趋势统计信息的字典 + """ + series = self.df[value_col] + growth = series.pct_change().dropna() + + return { + "start_value": series.iloc[0], + "end_value": series.iloc[-1], + "total_change": series.iloc[-1] - series.iloc[0], + "total_change_pct": (series.iloc[-1] - series.iloc[0]) / series.iloc[0] * 100, + "avg_growth_rate": growth.mean() * 100, + "max_growth_rate": growth.max() * 100, + "min_growth_rate": growth.min() * 100, + "positive_periods": (growth > 0).sum(), + "negative_periods": (growth < 0).sum(), + } + + def seasonal_decompose( + self, + value_col: str, + period: int = 7 + ) -> Dict[str, pd.Series]: + """ + 季节性分解（简化版） + + Args: + value_col: 值列 + period: 周期 + """ + series = self.df[value_col] + + # 移动平均作为趋势 + trend = series.rolling(window=period, center=True).mean() + + # 去趋势 + detrended = series - trend + + # 季节性（按周期位置平均） + seasonal = detrended.groupby( + detrended.index.dayofweek + ).transform('mean') + + # 残差 + residual = series - trend - seasonal + + return { + "original": series, + "trend": trend, + "seasonal": seasonal, + "residual": residual + } + + +class StatisticsCalculator: + """统计计算器""" + + @staticmethod + def percentile(series: pd.Series, q: float) -> float: + """计算分位数""" + return series.quantile(q) + + @staticmethod + def iqr(series: pd.Series) -> float: + """计算四分位距""" + return series.quantile(0.75) - series.quantile(0.25) + + @staticmethod + def outliers(series: pd.Series, method: str = 'iqr') -> pd.Series: + """ + 检测异常值 + + Args: + series: 数据序列 + method: 检测方法 ('iqr' 或 'zscore') + + Returns: + 布尔序列，True 表示异常值 + """ + if method == 'iqr': + q1 = series.quantile(0.25) + q3 = series.quantile(0.75) + iqr = q3 - q1 + lower = q1 - 1.5 * iqr + upper = q3 + 1.5 * iqr + return (series < lower) | (series > upper) + elif method == 'zscore': + z_scores = (series - series.mean()) / series.std() + return abs(z_scores) > 3 + else: + raise ValueError(f"Unknown method: {method}") + + @staticmethod + def distribution_stats(series: pd.Series) -> Dict[str, float]: + """计算分布统计量""" + return { + "count": series.count(), + "mean": series.mean(), + "std": series.std(), + "min": series.min(), + "q25": series.quantile(0.25), + "median": series.median(), + "q75": series.quantile(0.75), + "max": series.max(), + "skewness": series.skew(), + "kurtosis": series.kurtosis(), + } + + +def demo(): + """演示 pandas 数据分析功能""" + print("=" * 50) + print("pandas 数据分析演示") + print("=" * 50) + + # 模拟数据 + data = [ + {"title": "Python入门", "category": "技术", "views": 15000, "likes": 320, "date": "2024-01-15"}, + {"title": "爬虫教程", "category": "技术", "views": 12000, "likes": 280, "date": "2024-01-16"}, + {"title": "美食分享", "category": "生活", "views": 8000, "likes": 450, "date": "2024-01-17"}, + {"title": "数据分析", "category": "技术", "views": 18000, "likes": 520, "date": "2024-01-18"}, + {"title": "旅行日记", "category": "生活", "views": 10000, "likes": 380, "date": "2024-01-19"}, + {"title": "机器学习", "category": "技术", "views": 20000, "likes": 600, "date": "2024-01-20"}, + ] + + # 创建分析器 + analyzer = DataFrameAnalyzer(data) + + # 基本信息 + print("\n1. 数据集信息:") + info = analyzer.info() + print(f" 行数: {info['rows']}, 列数: {info['columns']}") + print(f" 列名: {info['column_names']}") + + # 描述性统计 + print("\n2. 描述性统计:") + print(analyzer.describe()) + + # 分组统计 + print("\n3. 按分类分组统计:") + print(analyzer.group_by_agg('category', {'views': ['sum', 'mean'], 'likes': ['sum', 'mean']})) + + # Top N + print("\n4. 浏览量 Top 3:") + print(analyzer.top_n(3, 'views')) + + # 分布统计 + print("\n5. 浏览量分布统计:") + stats = StatisticsCalculator.distribution_stats(analyzer.df['views']) + for key, value in stats.items(): + print(f" {key}: {value:.2f}") + + print("\n" + "=" * 50) + print("演示完成") + print("=" * 50) + + +if __name__ == "__main__": + demo() diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/pyproject.toml" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/pyproject.toml" new file mode 100644 index 0000000..4ed32e4 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/pyproject.toml" @@ -0,0 +1,24 @@ +[project] +name = "chapter10-data-analysis" +version = "0.1.0" +description = "第10章：数据分析与可视化 - Pandas分析、词云生成、图表制作" +readme = "README.md" +requires-python = ">=3.11" +dependencies = [ + "pandas>=2.2.0", + "jieba>=0.42.0", + "wordcloud>=1.9.0", + "matplotlib>=3.8.0", +] + +[project.optional-dependencies] +interactive = [ + "pyecharts>=2.0.0", # 交互式图表 +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.uv] +dev-dependencies = [] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/wordcloud_generator.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/wordcloud_generator.py" new file mode 100644 index 0000000..10c1d3c --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/10_\346\225\260\346\215\256\345\210\206\346\236\220\344\270\216\345\217\257\350\247\206\345\214\226/wordcloud_generator.py" @@ -0,0 +1,398 @@ +# -*- coding: utf-8 -*- +# @Desc: 词云生成工具 +from __future__ import annotations + +import os +from typing import List, Dict, Tuple, Optional +from collections import Counter +from loguru import logger + +# 可选依赖 +try: + import jieba + import jieba.analyse + HAS_JIEBA = True +except ImportError: + HAS_JIEBA = False + logger.warning("jieba 未安装，中文分词功能不可用") + +try: + from wordcloud import WordCloud, ImageColorGenerator + HAS_WORDCLOUD = True +except ImportError: + HAS_WORDCLOUD = False + logger.warning("wordcloud 未安装，词云生成功能不可用") + +try: + import numpy as np + from PIL import Image + HAS_PIL = True +except ImportError: + HAS_PIL = False + + +class ChineseTokenizer: + """中文分词器""" + + # 默认停用词 + DEFAULT_STOPWORDS = { + '的', '是', '在', '了', '和', '与', '或', '有', '个', '人', + '这', '那', '就', '都', '也', '为', '对', '到', '从', '把', + '被', '让', '给', '向', '往', '于', '及', '以', '等', '不', + '很', '会', '能', '可', '要', '我', '你', '他', '她', '它', + '啊', '吧', '呢', '呀', '哦', '嗯', '哈', '嘿', '么', '吗', + '什么', '怎么', '这样', '那样', '如何', '为什么', '怎样', + '没有', '已经', '可以', '一个', '一些', '有些', '还是', + '但是', '然后', '如果', '因为', '所以', '虽然', '而且', + '或者', '并且', '只是', '只有', '就是', '还有', '这个', + '那个', '自己', '什么', '这里', '那里', '这些', '那些', + } + + def __init__(self, stopwords: set = None, user_dict: str = None): + """ + 初始化分词器 + + Args: + stopwords: 自定义停用词集合 + user_dict: 用户词典路径 + """ + if not HAS_JIEBA: + raise ImportError("请安装 jieba: pip install jieba") + + self.stopwords = stopwords or self.DEFAULT_STOPWORDS + + if user_dict and os.path.exists(user_dict): + jieba.load_userdict(user_dict) + logger.info(f"加载用户词典: {user_dict}") + + def add_stopwords(self, words: List[str]): + """添加停用词""" + self.stopwords.update(words) + + def tokenize(self, text: str, min_length: int = 2) -> List[str]: + """ + 分词 + + Args: + text: 输入文本 + min_length: 最小词长度 + + Returns: + 词语列表 + """ + words = jieba.lcut(text) + return [ + w for w in words + if w not in self.stopwords + and len(w) >= min_length + and not w.isspace() + ] + + def extract_keywords( + self, + text: str, + top_k: int = 20, + method: str = 'tfidf' + ) -> List[Tuple[str, float]]: + """ + 提取关键词 + + Args: + text: 输入文本 + top_k: 返回关键词数量 + method: 提取方法 ('tfidf' 或 'textrank') + + Returns: + (关键词, 权重) 列表 + """ + if method == 'tfidf': + return jieba.analyse.extract_tags(text, topK=top_k, withWeight=True) + elif method == 'textrank': + return jieba.analyse.textrank(text, topK=top_k, withWeight=True) + else: + raise ValueError(f"Unknown method: {method}") + + def word_frequency( + self, + texts: List[str], + top_n: int = 100 + ) -> List[Tuple[str, int]]: + """ + 统计词频 + + Args: + texts: 文本列表 + top_n: 返回 Top N + + Returns: + (词语, 频次) 列表 + """ + all_words = [] + for text in texts: + words = self.tokenize(text) + all_words.extend(words) + + return Counter(all_words).most_common(top_n) + + +class WordCloudGenerator: + """词云生成器""" + + # 默认配色方案 + COLOR_SCHEMES = { + 'default': None, + 'viridis': 'viridis', + 'plasma': 'plasma', + 'inferno': 'inferno', + 'magma': 'magma', + 'cool': 'cool', + 'hot': 'hot', + } + + def __init__( + self, + font_path: str = None, + width: int = 800, + height: int = 600, + background_color: str = 'white', + max_words: int = 200, + max_font_size: int = 100, + min_font_size: int = 10, + colormap: str = None + ): + """ + 初始化词云生成器 + + Args: + font_path: 字体路径（中文需要指定） + width: 图片宽度 + height: 图片高度 + background_color: 背景颜色 + max_words: 最大词数 + max_font_size: 最大字体大小 + min_font_size: 最小字体大小 + colormap: matplotlib 色彩映射名称 + """ + if not HAS_WORDCLOUD: + raise ImportError("请安装 wordcloud: pip install wordcloud") + + self.font_path = font_path + self.width = width + self.height = height + self.background_color = background_color + self.max_words = max_words + self.max_font_size = max_font_size + self.min_font_size = min_font_size + self.colormap = colormap + + def _create_wordcloud(self, **kwargs) -> WordCloud: + """创建 WordCloud 对象""" + params = { + 'font_path': self.font_path, + 'width': self.width, + 'height': self.height, + 'background_color': self.background_color, + 'max_words': self.max_words, + 'max_font_size': self.max_font_size, + 'min_font_size': self.min_font_size, + 'colormap': self.colormap, + 'random_state': 42, + } + params.update(kwargs) + return WordCloud(**params) + + def generate_from_text(self, text: str, output_path: str) -> str: + """ + 从文本生成词云 + + Args: + text: 空格分隔的词语文本 + output_path: 输出路径 + + Returns: + 输出文件路径 + """ + wc = self._create_wordcloud() + wc.generate(text) + wc.to_file(output_path) + logger.info(f"词云已保存: {output_path}") + return output_path + + def generate_from_frequencies( + self, + frequencies: Dict[str, int], + output_path: str + ) -> str: + """ + 从词频字典生成词云 + + Args: + frequencies: {词语: 频次} 字典 + output_path: 输出路径 + + Returns: + 输出文件路径 + """ + wc = self._create_wordcloud() + wc.generate_from_frequencies(frequencies) + wc.to_file(output_path) + logger.info(f"词云已保存: {output_path}") + return output_path + + def generate_shaped( + self, + text: str, + mask_image_path: str, + output_path: str, + use_mask_colors: bool = True + ) -> str: + """ + 生成自定义形状的词云 + + Args: + text: 词语文本 + mask_image_path: 形状蒙版图片路径 + output_path: 输出路径 + use_mask_colors: 是否使用蒙版图片的颜色 + + Returns: + 输出文件路径 + """ + if not HAS_PIL: + raise ImportError("请安装 pillow 和 numpy") + + # 读取蒙版 + mask = np.array(Image.open(mask_image_path)) + + wc = self._create_wordcloud( + mask=mask, + contour_width=1, + contour_color='steelblue' + ) + + wc.generate(text) + + # 使用蒙版颜色 + if use_mask_colors: + image_colors = ImageColorGenerator(mask) + wc.recolor(color_func=image_colors) + + wc.to_file(output_path) + logger.info(f"形状词云已保存: {output_path}") + return output_path + + +class TextToWordCloud: + """从原始文本到词云的完整流程""" + + def __init__( + self, + font_path: str = None, + stopwords: set = None, + user_dict: str = None + ): + """ + 初始化 + + Args: + font_path: 字体路径 + stopwords: 停用词 + user_dict: 用户词典 + """ + self.tokenizer = ChineseTokenizer(stopwords, user_dict) + self.generator = WordCloudGenerator(font_path) + + def process( + self, + texts: List[str], + output_path: str, + top_words: int = 200 + ) -> str: + """ + 处理文本并生成词云 + + Args: + texts: 文本列表 + output_path: 输出路径 + top_words: 使用的词数量 + + Returns: + 输出文件路径 + """ + # 1. 统计词频 + word_freq = self.tokenizer.word_frequency(texts, top_words) + logger.info(f"统计完成: {len(word_freq)} 个词") + + # 2. 转换为字典 + freq_dict = dict(word_freq) + + # 3. 生成词云 + return self.generator.generate_from_frequencies(freq_dict, output_path) + + def get_word_stats(self, texts: List[str], top_n: int = 20) -> List[Tuple[str, int]]: + """ + 获取词频统计 + + Args: + texts: 文本列表 + top_n: 返回数量 + + Returns: + (词语, 频次) 列表 + """ + return self.tokenizer.word_frequency(texts, top_n) + + +def demo(): + """演示词云生成功能""" + print("=" * 50) + print("词云生成工具演示") + print("=" * 50) + + if not HAS_JIEBA: + print("请安装 jieba: pip install jieba") + return + + if not HAS_WORDCLOUD: + print("请安装 wordcloud: pip install wordcloud") + return + + # 测试文本 + texts = [ + "Python是一门优雅的编程语言，适合数据分析和机器学习", + "爬虫技术可以帮助我们获取互联网上的数据", + "数据分析是数据科学的重要组成部分", + "机器学习和深度学习是人工智能的核心技术", + "Python在数据科学领域有广泛的应用", + "爬虫需要注意遵守网站的规则和法律法规", + "数据可视化可以帮助我们更好地理解数据", + "编程是一项需要不断学习和实践的技能", + ] + + # 分词 + tokenizer = ChineseTokenizer() + print("\n1. 分词示例:") + sample_words = tokenizer.tokenize(texts[0]) + print(f" 原文: {texts[0]}") + print(f" 分词: {sample_words}") + + # 词频统计 + print("\n2. 词频统计 Top 10:") + word_freq = tokenizer.word_frequency(texts, 10) + for word, freq in word_freq: + print(f" {word}: {freq}") + + # 生成词云（仅打印说明，不实际生成文件） + print("\n3. 词云生成:") + print(" 词云生成需要指定中文字体路径") + print(" 示例代码:") + print(" generator = WordCloudGenerator(font_path='/path/to/font.ttf')") + print(" generator.generate_from_frequencies(dict(word_freq), 'wordcloud.png')") + + print("\n" + "=" * 50) + print("演示完成") + print("=" * 50) + + +if __name__ == "__main__": + demo() diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/README.md" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/README.md" new file mode 100644 index 0000000..1e450cf --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/README.md" @@ -0,0 +1,67 @@ +# 第11章：进阶综合实战项目 + +完整的书籍电商数据采集工具，爬取 **books.toscrape.com**（专门的爬虫练习网站）。 + +## 快速开始 + +### 使用 uv 安装依赖 + +```bash +cd 11_进阶综合实战项目 + +# 安装依赖 +uv sync + +# 安装可选功能（配置管理增强） +uv sync --extra all + +# 安装Playwright浏览器驱动 +uv run playwright install chromium + +# 运行项目 +uv run python main.py +``` + +### 目标网站 + +- **网站**：http://books.toscrape.com +- **类型**：专门用于爬虫练习的合法网站 +- **特点**：电商结构完整，50页书籍数据，无需登录 + +### 核心依赖 + +- `playwright` - 浏览器自动化 +- `httpx` - HTTP客户端 +- `pydantic` - 数据验证 +- `loguru` - 日志系统 +- `pandas` - 数据分析 +- `jieba` + `wordcloud` - 词云生成 + +## 项目结构 + +``` +11_进阶综合实战项目/ +├── config/ # 配置模块 +├── core/ # 核心模块（浏览器管理） +├── login/ # 登录模块 +├── crawler/ # 爬虫模块 +├── store/ # 存储模块 +├── proxy/ # 代理池模块 +├── analysis/ # 分析模块 +└── main.py # 主程序入口 +``` + +## 功能特性 + +- ✅ 浏览器自动化采集 +- ✅ 反检测技术（stealth.js） +- ✅ 多格式数据存储（JSON/CSV） +- ✅ 词云和统计报告生成 +- ✅ 代理池支持（可选） + +## 运行结果 + +运行成功后会在 `output/` 目录下生成： +- `data_*.json` 或 `data_*.csv` - 采集的数据 +- `report.md` - 数据分析报告 +- `wordcloud.png` - 词云图片 diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/analysis/__init__.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/analysis/__init__.py" new file mode 100644 index 0000000..047a1e0 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/analysis/__init__.py" @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +"""数据分析模块""" +from .report import DataAnalyzer, BilibiliAnalyzer, ReportGenerator, generate_report + +__all__ = ['DataAnalyzer', 'BilibiliAnalyzer', 'ReportGenerator', 'generate_report'] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/analysis/report.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/analysis/report.py" new file mode 100644 index 0000000..7d31de5 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/analysis/report.py" @@ -0,0 +1,584 @@ +# -*- coding: utf-8 -*- +""" +数据分析与报告生成模块 + +本模块实现了 B站视频数据的分析功能，包括： +- DataAnalyzer: 数据分析器（基础统计、词频分析、分布统计） +- BilibiliAnalyzer: B站视频专用分析器 +- ReportGenerator: 报告生成器（Markdown 格式） + +参考 MediaCrawler 项目的数据分析实践。 +""" + +import os +from typing import List, Dict, Any, Optional, Union +from datetime import datetime +from collections import Counter +from pathlib import Path +from loguru import logger + +# 尝试导入模型 +try: + from models.bilibili import BilibiliVideo + HAS_MODEL = True +except ImportError: + HAS_MODEL = False + +# 可选依赖 +try: + import jieba + HAS_JIEBA = True +except ImportError: + HAS_JIEBA = False + +try: + from wordcloud import WordCloud + HAS_WORDCLOUD = True +except ImportError: + HAS_WORDCLOUD = False + +try: + import pandas as pd + HAS_PANDAS = True +except ImportError: + HAS_PANDAS = False + + +class DataAnalyzer: + """数据分析器""" + + # 中文停用词 + STOPWORDS = { + '的', '是', '在', '了', '和', '与', '或', '有', '个', '人', + '这', '那', '就', '都', '也', '为', '对', '到', '从', '把', + '被', '让', '给', '向', '往', '于', '及', '以', '等', '不', + '很', '会', '能', '可', '要', '我', '你', '他', '她', '它', + '啊', '吧', '呢', '呀', '哦', '嗯', '哈', '嘿', '么', '吗', + '什么', '怎么', '这样', '那样', '如何', '为什么', '怎样', + '一个', '一些', '一种', '一下', '没有', '还是', '已经', + } + + def __init__(self, data: List[Dict], output_dir: str = "./output"): + """ + 初始化数据分析器 + + Args: + data: 数据列表 + output_dir: 输出目录 + """ + self.data = data + self.output_dir = Path(output_dir) + self.output_dir.mkdir(parents=True, exist_ok=True) + + if HAS_PANDAS and data: + self.df = pd.DataFrame(data) + else: + self.df = None + + def basic_stats(self) -> Dict[str, Any]: + """基础统计""" + stats = { + 'total_records': len(self.data), + 'fields': list(self.data[0].keys()) if self.data else [], + } + + if self.df is not None: + # 数值列统计 + numeric_cols = self.df.select_dtypes(include=['number']).columns.tolist() + if numeric_cols: + stats['numeric_fields'] = numeric_cols + for col in numeric_cols: + stats[f'{col}_sum'] = float(self.df[col].sum()) + stats[f'{col}_mean'] = float(self.df[col].mean()) + stats[f'{col}_max'] = float(self.df[col].max()) + stats[f'{col}_min'] = float(self.df[col].min()) + + return stats + + def word_frequency( + self, + text_field: str, + top_n: int = 50, + min_length: int = 2 + ) -> List[tuple]: + """ + 词频统计 + + Args: + text_field: 文本字段名 + top_n: 返回 Top N + min_length: 最小词长度 + + Returns: + (词语, 频次) 列表 + """ + if not HAS_JIEBA: + logger.warning("jieba 未安装，跳过词频分析") + return [] + + all_words = [] + for item in self.data: + text = item.get(text_field, '') + if text: + words = jieba.lcut(str(text)) + words = [ + w for w in words + if w not in self.STOPWORDS + and len(w) >= min_length + and not w.isspace() + ] + all_words.extend(words) + + return Counter(all_words).most_common(top_n) + + def generate_wordcloud( + self, + text_field: str, + output_file: str = "wordcloud.png", + font_path: str = None, + width: int = 1200, + height: int = 800 + ) -> Optional[str]: + """ + 生成词云 + + Args: + text_field: 文本字段名 + output_file: 输出文件名 + font_path: 字体路径（中文需要指定） + width: 图片宽度 + height: 图片高度 + + Returns: + 输出文件路径，失败返回 None + """ + if not HAS_WORDCLOUD: + logger.warning("wordcloud 未安装，跳过词云生成") + return None + + word_freq = self.word_frequency(text_field, 200) + if not word_freq: + logger.warning("没有词频数据，跳过词云生成") + return None + + freq_dict = dict(word_freq) + + try: + wc = WordCloud( + width=width, + height=height, + background_color='white', + font_path=font_path, + max_words=200, + max_font_size=150, + random_state=42 + ) + + wc.generate_from_frequencies(freq_dict) + + output_path = self.output_dir / output_file + wc.to_file(str(output_path)) + logger.info(f"词云已保存: {output_path}") + return str(output_path) + except Exception as e: + logger.error(f"生成词云失败: {e}") + return None + + def value_distribution(self, field: str) -> Dict[Any, int]: + """ + 值分布统计 + + Args: + field: 字段名 + + Returns: + {值: 计数} 字典 + """ + counter = Counter() + for item in self.data: + value = item.get(field) + if value is not None: + counter[value] += 1 + return dict(counter) + + +class BilibiliAnalyzer(DataAnalyzer): + """ + B站视频数据分析器 + + 提供针对 B站视频数据的专用分析方法。 + """ + + def __init__( + self, + videos: List[Union[Dict, "BilibiliVideo"]], + output_dir: str = "./output" + ): + """ + 初始化 B站分析器 + + Args: + videos: 视频数据列表（BilibiliVideo 或字典） + output_dir: 输出目录 + """ + # 转换为字典列表 + data = [] + for video in videos: + if HAS_MODEL and isinstance(video, BilibiliVideo): + data.append(video.to_dict()) + elif hasattr(video, 'model_dump'): + data.append(video.model_dump()) + elif isinstance(video, dict): + data.append(video) + else: + data.append(dict(video)) + + super().__init__(data, output_dir) + + def video_metrics_stats(self) -> Dict[str, Any]: + """ + 视频指标统计 + + 统计播放量、点赞、收藏、投币等指标。 + + Returns: + 统计结果字典 + """ + metrics = { + 'play_count': [], + 'liked_count': [], + 'coin_count': [], + 'favorite_count': [], + 'share_count': [], + 'danmaku_count': [], + 'comment_count': [], + } + + for item in self.data: + for key in metrics.keys(): + value = item.get(key, 0) + if value is not None: + try: + metrics[key].append(int(value)) + except (ValueError, TypeError): + metrics[key].append(0) + + stats = {} + for key, values in metrics.items(): + if values: + stats[key] = { + 'total': sum(values), + 'avg': sum(values) / len(values), + 'max': max(values), + 'min': min(values), + } + + return stats + + def up_distribution(self, top_n: int = 10) -> List[tuple]: + """ + UP主分布统计 + + Args: + top_n: 返回 Top N + + Returns: + [(UP主昵称, 视频数量)] 列表 + """ + counter = Counter() + for item in self.data: + nickname = item.get('nickname', '未知UP主') + if nickname: + counter[nickname] += 1 + return counter.most_common(top_n) + + def keyword_distribution(self) -> Dict[str, int]: + """ + 搜索关键词分布统计 + + Returns: + {关键词: 视频数量} 字典 + """ + counter = Counter() + for item in self.data: + keyword = item.get('source_keyword', '') + if keyword: + counter[keyword] += 1 + return dict(counter) + + def top_videos( + self, + metric: str = 'play_count', + top_n: int = 10 + ) -> List[Dict]: + """ + 获取指标排名前 N 的视频 + + Args: + metric: 排序指标（play_count, liked_count 等） + top_n: 返回数量 + + Returns: + 视频列表 + """ + sorted_data = sorted( + self.data, + key=lambda x: x.get(metric, 0) or 0, + reverse=True + ) + return sorted_data[:top_n] + + def generate_title_wordcloud( + self, + output_file: str = "title_wordcloud.png", + font_path: str = None + ) -> Optional[str]: + """ + 生成视频标题词云 + + Args: + output_file: 输出文件名 + font_path: 字体路径 + + Returns: + 输出文件路径 + """ + return self.generate_wordcloud( + text_field='title', + output_file=output_file, + font_path=font_path + ) + + +class ReportGenerator: + """ + 报告生成器 + + 生成 B站视频数据的 Markdown 格式分析报告。 + """ + + def __init__( + self, + videos: List[Union[Dict, "BilibiliVideo"]], + output_dir: str = "./output" + ): + """ + 初始化报告生成器 + + Args: + videos: 视频数据列表 + output_dir: 输出目录 + """ + self.videos = videos + self.output_dir = Path(output_dir) + self.output_dir.mkdir(parents=True, exist_ok=True) + + # 创建分析器 + self.analyzer = BilibiliAnalyzer(videos, output_dir) + + def generate( + self, + font_path: str = None, + title: str = "B站视频数据分析报告" + ) -> str: + """ + 生成完整分析报告 + + Args: + font_path: 字体路径（词云使用） + title: 报告标题 + + Returns: + 报告文件路径 + """ + lines = [] + + # 标题 + lines.append(f"# {title}") + lines.append("") + lines.append(f"> 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}") + lines.append(f"> 数据来源: B站视频采集") + lines.append(f"> 数据量: {len(self.analyzer.data)} 条") + lines.append("") + lines.append("---") + lines.append("") + + # 1. 视频指标统计 + lines.append("## 1. 视频指标统计") + lines.append("") + metrics_stats = self.analyzer.video_metrics_stats() + if metrics_stats: + lines.append("| 指标 | 总计 | 平均 | 最高 | 最低 |") + lines.append("| --- | ---: | ---: | ---: | ---: |") + + metric_names = { + 'play_count': '播放量', + 'liked_count': '点赞数', + 'coin_count': '投币数', + 'favorite_count': '收藏数', + 'share_count': '分享数', + 'danmaku_count': '弹幕数', + 'comment_count': '评论数', + } + + for key, name in metric_names.items(): + if key in metrics_stats: + stat = metrics_stats[key] + lines.append( + f"| {name} | " + f"{stat['total']:,} | " + f"{stat['avg']:,.0f} | " + f"{stat['max']:,} | " + f"{stat['min']:,} |" + ) + lines.append("") + + # 2. 热门视频 TOP 10 + lines.append("## 2. 热门视频 TOP 10（按播放量）") + lines.append("") + top_videos = self.analyzer.top_videos('play_count', 10) + if top_videos: + lines.append("| 排名 | 标题 | UP主 | 播放量 | 点赞 |") + lines.append("| --- | --- | --- | ---: | ---: |") + for i, video in enumerate(top_videos, 1): + title_short = video.get('title', '')[:30] + if len(video.get('title', '')) > 30: + title_short += '...' + lines.append( + f"| {i} | {title_short} | " + f"{video.get('nickname', '未知')} | " + f"{video.get('play_count', 0):,} | " + f"{video.get('liked_count', 0):,} |" + ) + lines.append("") + + # 3. UP主分布 + lines.append("## 3. UP主分布 TOP 10") + lines.append("") + up_dist = self.analyzer.up_distribution(10) + if up_dist: + lines.append("| 排名 | UP主 | 视频数 |") + lines.append("| --- | --- | ---: |") + for i, (name, count) in enumerate(up_dist, 1): + lines.append(f"| {i} | {name} | {count} |") + lines.append("") + + # 4. 关键词分布 + keyword_dist = self.analyzer.keyword_distribution() + if keyword_dist and len(keyword_dist) > 1: + lines.append("## 4. 搜索关键词分布") + lines.append("") + lines.append("| 关键词 | 视频数 |") + lines.append("| --- | ---: |") + for keyword, count in sorted( + keyword_dist.items(), + key=lambda x: x[1], + reverse=True + ): + lines.append(f"| {keyword} | {count} |") + lines.append("") + + # 5. 标题热词 TOP 20 + if HAS_JIEBA: + lines.append("## 5. 标题热词 TOP 20") + lines.append("") + word_freq = self.analyzer.word_frequency('title', 20) + if word_freq: + lines.append("| 排名 | 词汇 | 频次 |") + lines.append("| --- | --- | ---: |") + for i, (word, count) in enumerate(word_freq, 1): + lines.append(f"| {i} | {word} | {count} |") + lines.append("") + + # 生成词云 + if HAS_WORDCLOUD: + wordcloud_path = self.analyzer.generate_title_wordcloud( + font_path=font_path + ) + if wordcloud_path: + lines.append("### 标题词云") + lines.append("") + lines.append("") + lines.append("") + + # 6. 数据样本 + lines.append("## 附录: 数据样本 (前5条)") + lines.append("") + sample_fields = ['bvid', 'title', 'nickname', 'play_count', 'liked_count'] + sample_data = self.analyzer.data[:5] + if sample_data: + lines.append("| " + " | ".join(sample_fields) + " |") + lines.append("| " + " | ".join(["---"] * len(sample_fields)) + " |") + for item in sample_data: + row = [] + for f in sample_fields: + val = str(item.get(f, ''))[:40] + val = val.replace('|', '\\|').replace('\n', ' ') + row.append(val) + lines.append("| " + " | ".join(row) + " |") + lines.append("") + + # 保存报告 + report_content = '\n'.join(lines) + report_path = self.output_dir / "report.md" + with open(report_path, 'w', encoding='utf-8') as f: + f.write(report_content) + + logger.info(f"报告已保存: {report_path}") + return str(report_path) + + +# 便捷函数 +def generate_report( + videos: List[Union[Dict, "BilibiliVideo"]], + output_dir: str = "./output", + font_path: str = None +) -> str: + """ + 生成分析报告（便捷函数） + + Args: + videos: 视频数据列表 + output_dir: 输出目录 + font_path: 字体路径 + + Returns: + 报告文件路径 + """ + generator = ReportGenerator(videos, output_dir) + return generator.generate(font_path=font_path) + + +if __name__ == '__main__': + # 测试代码 + test_data = [ + { + 'bvid': 'BV1234567890', + 'title': 'Python 爬虫入门教程', + 'nickname': '技术UP主', + 'play_count': 10000, + 'liked_count': 500, + 'coin_count': 200, + 'favorite_count': 300, + 'share_count': 50, + 'danmaku_count': 100, + 'comment_count': 80, + 'source_keyword': 'Python教程', + }, + { + 'bvid': 'BV0987654321', + 'title': 'Python 数据分析实战', + 'nickname': '数据分析师', + 'play_count': 8000, + 'liked_count': 400, + 'coin_count': 150, + 'favorite_count': 250, + 'share_count': 30, + 'danmaku_count': 80, + 'comment_count': 60, + 'source_keyword': 'Python教程', + }, + ] + + report_path = generate_report(test_data, './test_output') + print(f"报告已生成: {report_path}") diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/client/__init__.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/client/__init__.py" new file mode 100644 index 0000000..79a75e7 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/client/__init__.py" @@ -0,0 +1,7 @@ +# -*- coding: utf-8 -*- +""" +B站 API 客户端模块 +""" +from .bilibili_client import BilibiliClient + +__all__ = ["BilibiliClient"] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/client/bilibili_client.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/client/bilibili_client.py" new file mode 100644 index 0000000..09839f7 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/client/bilibili_client.py" @@ -0,0 +1,369 @@ +# -*- coding: utf-8 -*- +""" +B站 API 客户端 + +本模块封装了 B站的 API 请求，包括： +- 视频搜索 API +- 视频详情 API +- WBI 签名处理 + +使用 httpx 作为 HTTP 客户端，支持异步请求。 + +参考 MediaCrawler 项目的实现： +- https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/client.py +""" + +import json +from typing import Dict, Optional, List, Callable, Any +from loguru import logger + +# 可选依赖 +try: + import httpx + HAS_HTTPX = True +except ImportError: + HAS_HTTPX = False + +try: + from playwright.async_api import BrowserContext, Page + HAS_PLAYWRIGHT = True +except ImportError: + HAS_PLAYWRIGHT = False + +from tools.sign import BilibiliSign, extract_wbi_keys_from_urls +from models.bilibili import BilibiliVideo, BilibiliSearchResponse +from config import bilibili_config + + +class BilibiliClient: + """ + B站 API 客户端 + + 封装 B站的 API 请求，支持 WBI 签名。 + + 使用示例： + ```python + client = BilibiliClient() + await client.update_cookies(browser_context) + + # 搜索视频 + videos = await client.search_video_by_keyword("Python教程", page=1) + + # 获取视频详情 + video = await client.get_video_info(bvid="BV1xx411c7mD") + ``` + """ + + def __init__(self): + """初始化客户端""" + self.headers = bilibili_config.DEFAULT_HEADERS.copy() + self.cookie_dict: Dict[str, str] = {} + self._signer: Optional[BilibiliSign] = None + self._timeout = bilibili_config.REQUEST_TIMEOUT + + async def update_cookies(self, browser_context: "BrowserContext"): + """ + 从浏览器上下文更新 Cookie + + 登录成功后调用此方法，将浏览器的 Cookie 同步到客户端。 + + Args: + browser_context: Playwright 浏览器上下文 + """ + cookies = await browser_context.cookies() + cookie_str = "; ".join([f"{c['name']}={c['value']}" for c in cookies]) + self.headers["Cookie"] = cookie_str + self.cookie_dict = {c['name']: c['value'] for c in cookies} + logger.info(f"[BilibiliClient] 更新了 {len(cookies)} 个 Cookie") + + async def init_wbi_sign(self, page: "Page"): + """ + 初始化 WBI 签名器 + + 从浏览器的 localStorage 中获取 WBI 密钥。 + + Args: + page: Playwright 页面对象 + """ + try: + # 从 localStorage 获取 wbi_img_urls + wbi_img_urls = await page.evaluate(""" + () => { + return localStorage.getItem('wbi_img_urls'); + } + """) + + if not wbi_img_urls: + logger.warning("[BilibiliClient] 未找到 wbi_img_urls，尝试从 API 获取") + await self._fetch_wbi_keys() + return + + # 解析 JSON + wbi_data = json.loads(wbi_img_urls) + img_url = wbi_data.get("imgUrl", "") + sub_url = wbi_data.get("subUrl", "") + + if img_url and sub_url: + img_key, sub_key = extract_wbi_keys_from_urls(img_url, sub_url) + self._signer = BilibiliSign(img_key, sub_key) + logger.info("[BilibiliClient] WBI 签名器初始化成功") + else: + logger.warning("[BilibiliClient] wbi_img_urls 数据不完整") + await self._fetch_wbi_keys() + + except Exception as e: + logger.error(f"[BilibiliClient] 初始化 WBI 签名器失败: {e}") + await self._fetch_wbi_keys() + + async def _fetch_wbi_keys(self): + """ + 从 API 获取 WBI 密钥（备用方案） + """ + try: + async with httpx.AsyncClient(timeout=self._timeout) as client: + response = await client.get( + "https://api.bilibili.com/x/web-interface/nav", + headers=self.headers + ) + data = response.json() + + if data.get("code") == 0: + wbi_img = data.get("data", {}).get("wbi_img", {}) + img_url = wbi_img.get("img_url", "") + sub_url = wbi_img.get("sub_url", "") + + if img_url and sub_url: + img_key, sub_key = extract_wbi_keys_from_urls(img_url, sub_url) + self._signer = BilibiliSign(img_key, sub_key) + logger.info("[BilibiliClient] 从 API 获取 WBI 密钥成功") + return + + logger.error("[BilibiliClient] 无法获取 WBI 密钥") + + except Exception as e: + logger.error(f"[BilibiliClient] 获取 WBI 密钥失败: {e}") + + async def _request( + self, + method: str, + url: str, + params: Optional[Dict] = None, + data: Optional[Dict] = None, + enable_sign: bool = False + ) -> Optional[Dict]: + """ + 发送 HTTP 请求 + + Args: + method: 请求方法（GET/POST） + url: 请求 URL + params: URL 参数 + data: POST 数据 + enable_sign: 是否启用 WBI 签名 + + Returns: + Dict: 响应数据 + """ + if not HAS_HTTPX: + logger.error("[BilibiliClient] httpx 未安装") + return None + + try: + # 如果需要签名 + if enable_sign and self._signer and params: + params = self._signer.sign(params) + + async with httpx.AsyncClient(timeout=self._timeout) as client: + if method.upper() == "GET": + response = await client.get(url, params=params, headers=self.headers) + else: + response = await client.post(url, params=params, data=data, headers=self.headers) + + if response.status_code == 200: + return response.json() + else: + logger.error(f"[BilibiliClient] 请求失败: {response.status_code}") + return None + + except Exception as e: + logger.error(f"[BilibiliClient] 请求出错: {e}") + return None + + async def search_video_by_keyword( + self, + keyword: str, + page: int = 1, + page_size: int = 20, + order: str = "", + ) -> List[BilibiliVideo]: + """ + 按关键词搜索视频 + + Args: + keyword: 搜索关键词 + page: 页码（从1开始） + page_size: 每页数量（B站固定为20） + order: 排序方式（空=综合，click=最多点击，pubdate=最新发布） + + Returns: + List[BilibiliVideo]: 视频列表 + """ + logger.info(f"[BilibiliClient] 搜索视频: {keyword}, 第 {page} 页") + + params = { + "keyword": keyword, + "search_type": "video", + "page": page, + "page_size": page_size, + "order": order, + } + + data = await self._request( + "GET", + bilibili_config.SEARCH_URL, + params=params, + enable_sign=True + ) + + if not data: + return [] + + if data.get("code") != 0: + logger.error(f"[BilibiliClient] 搜索失败: {data.get('message')}") + return [] + + # 解析搜索结果 + result = data.get("data", {}) + video_list = result.get("result", []) + + videos = [] + for item in video_list: + try: + video = BilibiliVideo.from_search_result(item, keyword) + videos.append(video) + except Exception as e: + logger.debug(f"[BilibiliClient] 解析视频失败: {e}") + continue + + logger.info(f"[BilibiliClient] 搜索到 {len(videos)} 个视频") + return videos + + async def get_video_info( + self, + aid: Optional[str] = None, + bvid: Optional[str] = None + ) -> Optional[BilibiliVideo]: + """ + 获取视频详情 + + aid 和 bvid 至少提供一个。 + + Args: + aid: 视频 aid + bvid: 视频 BV 号 + + Returns: + BilibiliVideo: 视频信息 + """ + if not aid and not bvid: + logger.error("[BilibiliClient] aid 和 bvid 至少提供一个") + return None + + params = {} + if bvid: + params["bvid"] = bvid + elif aid: + params["aid"] = aid + + logger.info(f"[BilibiliClient] 获取视频详情: {bvid or aid}") + + data = await self._request( + "GET", + bilibili_config.VIDEO_INFO_URL, + params=params, + enable_sign=False # 视频详情 API 不需要签名 + ) + + if not data: + return None + + if data.get("code") != 0: + logger.error(f"[BilibiliClient] 获取视频详情失败: {data.get('message')}") + return None + + video_data = data.get("data", {}) + return BilibiliVideo.from_api_response(video_data) + + async def get_video_info_batch( + self, + bvid_list: List[str], + callback: Optional[Callable[[BilibiliVideo], Any]] = None + ) -> List[BilibiliVideo]: + """ + 批量获取视频详情 + + Args: + bvid_list: BV 号列表 + callback: 每获取一个视频后的回调函数 + + Returns: + List[BilibiliVideo]: 视频列表 + """ + videos = [] + + for bvid in bvid_list: + video = await self.get_video_info(bvid=bvid) + if video: + videos.append(video) + if callback: + await callback(video) if asyncio.iscoroutinefunction(callback) else callback(video) + + return videos + + async def pong(self) -> bool: + """ + 检查登录状态 + + 通过调用 nav API 检查是否已登录。 + + Returns: + bool: 是否已登录 + """ + try: + data = await self._request( + "GET", + "https://api.bilibili.com/x/web-interface/nav", + enable_sign=False + ) + + if data and data.get("code") == 0: + user_data = data.get("data", {}) + if user_data.get("isLogin"): + username = user_data.get("uname", "未知用户") + logger.info(f"[BilibiliClient] 已登录: {username}") + return True + + return False + + except Exception as e: + logger.debug(f"[BilibiliClient] 检查登录状态失败: {e}") + return False + + +# 为了兼容性，添加 asyncio 导入 +import asyncio + + +if __name__ == '__main__': + # 测试代码 + async def test(): + client = BilibiliClient() + + # 测试搜索（无需登录，但可能被限制） + videos = await client.search_video_by_keyword("Python教程", page=1) + print(f"搜索到 {len(videos)} 个视频") + + for video in videos[:3]: + print(f" - {video.title} (播放: {video.play_count})") + + asyncio.run(test()) diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/__init__.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/__init__.py" new file mode 100644 index 0000000..668c896 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/__init__.py" @@ -0,0 +1,15 @@ +# -*- coding: utf-8 -*- +""" +配置模块 +""" +from .settings import settings, Settings, StorageType, LoginType, CrawlerType +from . import bilibili_config + +__all__ = [ + 'settings', + 'Settings', + 'StorageType', + 'LoginType', + 'CrawlerType', + 'bilibili_config', +] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/bilibili_config.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/bilibili_config.py" new file mode 100644 index 0000000..d039bc6 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/bilibili_config.py" @@ -0,0 +1,100 @@ +# -*- coding: utf-8 -*- +""" +B站特定配置 + +本模块包含 B站 API 相关的常量和配置，包括： +- API 端点地址 +- 请求头配置 +- WBI 签名相关常量 +- 搜索排序类型 +""" + +from enum import Enum + + +class SearchOrderType(str, Enum): + """搜索排序类型""" + DEFAULT = "" # 综合排序 + MOST_CLICK = "click" # 最多点击 + LAST_PUBLISH = "pubdate" # 最新发布 + MOST_DANMU = "dm" # 最多弹幕 + MOST_MARK = "stow" # 最多收藏 + + +# ==================== API 端点 ==================== + +# B站主站 +BILIBILI_URL = "https://www.bilibili.com" + +# 搜索 API +SEARCH_URL = "https://api.bilibili.com/x/web-interface/wbi/search/type" + +# 视频详情 API +VIDEO_INFO_URL = "https://api.bilibili.com/x/web-interface/view" + +# 视频播放地址 API +VIDEO_PLAY_URL = "https://api.bilibili.com/x/player/playurl" + +# 用户信息 API +USER_INFO_URL = "https://api.bilibili.com/x/space/wbi/acc/info" + + +# ==================== 请求头配置 ==================== + +DEFAULT_HEADERS = { + "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", + "Referer": "https://www.bilibili.com", + "Accept": "application/json, text/plain, */*", + "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8", + "Origin": "https://www.bilibili.com", +} + + +# ==================== WBI 签名相关 ==================== + +# WBI 签名混淆映射表（固定值） +WBI_MIXIN_KEY_ENC_TAB = [ + 46, 47, 18, 2, 53, 8, 23, 32, 15, 50, 10, 31, 58, 3, 45, 35, + 27, 43, 5, 49, 33, 9, 42, 19, 29, 28, 14, 39, 12, 38, 41, 13, + 37, 48, 7, 16, 24, 55, 40, 61, 26, 17, 0, 1, 60, 51, 30, 4, + 22, 25, 54, 21, 56, 59, 6, 63, 57, 62, 11, 36, 20, 34, 44, 52, +] + + +# ==================== 登录相关 ==================== + +# 登录页面 URL +LOGIN_URL = "https://www.bilibili.com" + +# 登录按钮选择器 +LOGIN_BUTTON_SELECTOR = "xpath=//div[@class='right-entry__outside go-login-btn']//div" + +# 二维码选择器 +QRCODE_SELECTOR = "//div[@class='login-scan-box']//img" + +# 登录成功后的 Cookie 关键字段 +LOGIN_COOKIE_KEYS = ["SESSDATA", "DedeUserID", "bili_jct"] + + +# ==================== 页面选择器 ==================== + +# 搜索结果项选择器 +SEARCH_ITEM_SELECTOR = ".video-list-item" + +# 视频标题选择器 +VIDEO_TITLE_SELECTOR = ".title" + +# 视频播放量选择器 +VIDEO_PLAY_COUNT_SELECTOR = ".play-count" + + +# ==================== 其他配置 ==================== + +# 每页搜索结果数量（B站固定值） +SEARCH_PAGE_SIZE = 20 + +# 最大重试次数 +MAX_RETRY_COUNT = 3 + +# 请求超时时间（秒） +REQUEST_TIMEOUT = 10 diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/settings.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/settings.py" new file mode 100644 index 0000000..24f7d61 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/config/settings.py" @@ -0,0 +1,121 @@ +# -*- coding: utf-8 -*- +""" +项目配置模块 + +本模块定义了 B站爬虫项目的所有配置项，包括： +- 基础配置（应用名称、调试模式） +- 浏览器配置（无头模式、超时时间） +- 登录配置（登录方式、Cookie文件） +- 代理配置（是否启用、API地址） +- 爬虫配置（最大页数、延迟时间） +- 存储配置（存储类型、输出目录） +""" + +from typing import Optional, List +from enum import Enum + +# 尝试导入 pydantic-settings，如果不存在则使用简单的配置类 +try: + from pydantic_settings import BaseSettings + from pydantic import Field + HAS_PYDANTIC_SETTINGS = True +except ImportError: + HAS_PYDANTIC_SETTINGS = False + + +class StorageType(str, Enum): + """存储类型枚举""" + JSON = "json" + CSV = "csv" + + +class LoginType(str, Enum): + """登录类型枚举""" + COOKIE = "cookie" + QRCODE = "qrcode" + + +class CrawlerType(str, Enum): + """爬虫类型枚举""" + SEARCH = "search" # 关键词搜索 + DETAIL = "detail" # 指定视频详情 + + +if HAS_PYDANTIC_SETTINGS: + class Settings(BaseSettings): + """项目配置（使用 pydantic-settings）""" + + # 基础配置 + app_name: str = "BilibiliCrawler" + debug: bool = False + + # 浏览器配置 + browser_headless: bool = False # B站扫码登录需要显示浏览器 + browser_timeout: int = 30000 + browser_user_data_dir: Optional[str] = "browser_data/bili_user_data" + + # 登录配置 + login_type: LoginType = LoginType.QRCODE + cookie_str: str = "" # Cookie 字符串，当 login_type=cookie 时使用 + save_login_state: bool = True # 是否保存登录状态 + + # 代理配置 + proxy_enabled: bool = False + proxy_api_url: Optional[str] = None + + # B站爬虫配置 + crawler_type: CrawlerType = CrawlerType.SEARCH + keywords: str = "Python教程" # 搜索关键词，多个用逗号分隔 + specified_id_list: List[str] = [] # 指定视频列表（BV号） + max_video_count: int = 20 # 最大爬取视频数量 + crawl_delay_min: float = 1.0 + crawl_delay_max: float = 3.0 + max_concurrency: int = 3 # 最大并发数 + + # 存储配置 + storage_type: StorageType = StorageType.JSON + storage_output_dir: str = "./output" + + class Config: + env_file = ".env" + env_prefix = "BILI_" + +else: + class Settings: + """项目配置（简单实现，无 pydantic-settings 依赖）""" + + def __init__(self): + # 基础配置 + self.app_name = "BilibiliCrawler" + self.debug = False + + # 浏览器配置 + self.browser_headless = False # B站扫码登录需要显示浏览器 + self.browser_timeout = 30000 + self.browser_user_data_dir = "browser_data/bili_user_data" + + # 登录配置 + self.login_type = LoginType.QRCODE + self.cookie_str = "" # Cookie 字符串 + self.save_login_state = True + + # 代理配置 + self.proxy_enabled = False + self.proxy_api_url = None + + # B站爬虫配置 + self.crawler_type = CrawlerType.SEARCH + self.keywords = "Python教程" + self.specified_id_list = [] + self.max_video_count = 20 + self.crawl_delay_min = 1.0 + self.crawl_delay_max = 3.0 + self.max_concurrency = 3 + + # 存储配置 + self.storage_type = StorageType.JSON + self.storage_output_dir = "./output" + + +# 全局配置实例 +settings = Settings() diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/core/__init__.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/core/__init__.py" new file mode 100644 index 0000000..db73a96 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/core/__init__.py" @@ -0,0 +1,4 @@ +# -*- coding: utf-8 -*- +from .browser import BrowserManager + +__all__ = ['BrowserManager'] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/core/browser.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/core/browser.py" new file mode 100644 index 0000000..00c76e0 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/core/browser.py" @@ -0,0 +1,155 @@ +# -*- coding: utf-8 -*- +# @Desc: 浏览器管理模块 +from __future__ import annotations + +from typing import Optional +from loguru import logger + +# 可选依赖 +try: + from playwright.async_api import async_playwright, Browser, BrowserContext, Page + HAS_PLAYWRIGHT = True +except ImportError: + HAS_PLAYWRIGHT = False + logger.warning("playwright 未安装") + +# stealth.min.js 内容（简化版） +STEALTH_JS = """ +// 隐藏 webdriver 属性 +Object.defineProperty(navigator, 'webdriver', { + get: () => undefined +}); + +// 修改 plugins +Object.defineProperty(navigator, 'plugins', { + get: () => [1, 2, 3, 4, 5] +}); + +// 修改 languages +Object.defineProperty(navigator, 'languages', { + get: () => ['zh-CN', 'zh', 'en'] +}); + +// 修改 platform +Object.defineProperty(navigator, 'platform', { + get: () => 'MacIntel' +}); + +// 隐藏 automation 特征 +window.chrome = { + runtime: {} +}; + +// 修改 permissions +const originalQuery = window.navigator.permissions.query; +window.navigator.permissions.query = (parameters) => ( + parameters.name === 'notifications' ? + Promise.resolve({ state: Notification.permission }) : + originalQuery(parameters) +); +""" + + +class BrowserManager: + """浏览器管理器""" + + def __init__( + self, + headless: bool = True, + timeout: int = 30000, + user_data_dir: str = None, + proxy: str = None + ): + """ + 初始化浏览器管理器 + + Args: + headless: 是否无头模式 + timeout: 默认超时时间（毫秒） + user_data_dir: 用户数据目录 + proxy: 代理服务器地址 + """ + if not HAS_PLAYWRIGHT: + raise ImportError("请安装 playwright: pip install playwright && playwright install") + + self.headless = headless + self.timeout = timeout + self.user_data_dir = user_data_dir + self.proxy = proxy + + self._playwright = None + self._browser: Optional[Browser] = None + self._context: Optional[BrowserContext] = None + + async def start(self) -> BrowserContext: + """启动浏览器""" + self._playwright = await async_playwright().start() + + # 浏览器启动参数 + launch_args = [ + '--disable-blink-features=AutomationControlled', + '--no-sandbox', + '--disable-dev-shm-usage', + ] + + # 启动浏览器 + self._browser = await self._playwright.chromium.launch( + headless=self.headless, + args=launch_args + ) + + # 创建上下文 + context_options = { + 'viewport': {'width': 1920, 'height': 1080}, + 'user_agent': ( + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + 'Chrome/131.0.0.0 Safari/537.36' + ), + 'locale': 'zh-CN', + 'timezone_id': 'Asia/Shanghai', + } + + if self.proxy: + context_options['proxy'] = {'server': self.proxy} + + self._context = await self._browser.new_context(**context_options) + + # 注入反检测脚本 + await self._context.add_init_script(STEALTH_JS) + + logger.info("浏览器启动成功") + return self._context + + async def new_page(self) -> 'Page': + """创建新页面""" + if not self._context: + await self.start() + page = await self._context.new_page() + page.set_default_timeout(self.timeout) + return page + + async def close(self): + """关闭浏览器""" + if self._context: + await self._context.close() + self._context = None + if self._browser: + await self._browser.close() + self._browser = None + if self._playwright: + await self._playwright.stop() + self._playwright = None + logger.info("浏览器已关闭") + + @property + def context(self) -> Optional[BrowserContext]: + """获取浏览器上下文""" + return self._context + + async def __aenter__(self): + await self.start() + return self + + async def __aexit__(self, exc_type, exc_val, exc_tb): + await self.close() diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/crawler/__init__.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/crawler/__init__.py" new file mode 100644 index 0000000..524d609 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/crawler/__init__.py" @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +"""爬虫模块""" +from .spider import BilibiliCrawler, run_crawler + +__all__ = ['BilibiliCrawler', 'run_crawler'] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/crawler/spider.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/crawler/spider.py" new file mode 100644 index 0000000..bab1756 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/crawler/spider.py" @@ -0,0 +1,328 @@ +# -*- coding: utf-8 -*- +""" +B站爬虫模块 + +本模块实现了 B站视频数据的爬取功能，包括： +- 关键词搜索视频 +- 获取指定视频详情 +- 并发爬取控制 +- 数据存储回调 + +参考 MediaCrawler 项目的实现： +- https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/core.py +""" + +import asyncio +import random +from typing import List, Dict, Optional, Callable, Any +from loguru import logger + +# 可选依赖 +try: + from playwright.async_api import async_playwright, Browser, BrowserContext, Page + HAS_PLAYWRIGHT = True +except ImportError: + HAS_PLAYWRIGHT = False + +from config import settings, CrawlerType +from config import bilibili_config +from core.browser import BrowserManager +from login.auth import BilibiliLogin +from client.bilibili_client import BilibiliClient +from models.bilibili import BilibiliVideo +from tools.sign import parse_video_info_from_url + + +class BilibiliCrawler: + """ + B站爬虫类 + + 整合浏览器管理、登录认证、API客户端，实现完整的爬取流程。 + + 使用示例： + ```python + crawler = BilibiliCrawler() + videos = await crawler.start() + ``` + """ + + def __init__(self): + """初始化爬虫""" + self.browser_manager: Optional[BrowserManager] = None + self.browser_context: Optional[BrowserContext] = None + self.context_page: Optional[Page] = None + self.bili_client: Optional[BilibiliClient] = None + + # 爬取结果 + self._results: List[BilibiliVideo] = [] + + # 配置 + self.max_video_count = settings.max_video_count + self.max_concurrency = settings.max_concurrency + self.delay_min = settings.crawl_delay_min + self.delay_max = settings.crawl_delay_max + + async def start(self) -> List[BilibiliVideo]: + """ + 启动爬虫 + + 完整流程： + 1. 启动浏览器 + 2. 执行登录 + 3. 初始化 API 客户端 + 4. 根据配置执行爬取 + 5. 关闭浏览器 + + Returns: + List[BilibiliVideo]: 爬取的视频列表 + """ + logger.info(f"[BilibiliCrawler] 启动爬虫，类型: {settings.crawler_type}") + + try: + # 1. 启动浏览器 + await self._init_browser() + + # 2. 执行登录 + login_success = await self._do_login() + if not login_success: + logger.error("[BilibiliCrawler] 登录失败，退出") + return [] + + # 3. 初始化 API 客户端 + await self._init_client() + + # 4. 根据配置执行爬取 + if settings.crawler_type == CrawlerType.SEARCH: + await self.search_by_keywords() + elif settings.crawler_type == CrawlerType.DETAIL: + await self.get_specified_videos() + else: + logger.error(f"[BilibiliCrawler] 不支持的爬取类型: {settings.crawler_type}") + + logger.info(f"[BilibiliCrawler] 爬取完成，共 {len(self._results)} 个视频") + return self._results + + except Exception as e: + logger.exception(f"[BilibiliCrawler] 爬取出错: {e}") + return self._results + + finally: + # 5. 关闭浏览器 + await self.close() + + async def _init_browser(self): + """初始化浏览器""" + logger.info("[BilibiliCrawler] 初始化浏览器...") + + self.browser_manager = BrowserManager( + headless=settings.browser_headless, + timeout=settings.browser_timeout, + user_data_dir=settings.browser_user_data_dir if settings.save_login_state else None + ) + + self.browser_context = await self.browser_manager.start() + self.context_page = await self.browser_manager.new_page() + + logger.info("[BilibiliCrawler] 浏览器初始化完成") + + async def _do_login(self) -> bool: + """ + 执行登录 + + Returns: + bool: 是否登录成功 + """ + # 先检查是否已登录（通过保存的状态） + self.bili_client = BilibiliClient() + await self.bili_client.update_cookies(self.browser_context) + + if await self.bili_client.pong(): + logger.info("[BilibiliCrawler] 已有登录状态，跳过登录") + return True + + # 执行登录 + login = BilibiliLogin( + login_type=settings.login_type.value, + browser_context=self.browser_context, + context_page=self.context_page, + cookie_str=settings.cookie_str + ) + + success = await login.begin() + + if success: + # 更新客户端 Cookie + await self.bili_client.update_cookies(self.browser_context) + + return success + + async def _init_client(self): + """初始化 API 客户端""" + # 初始化 WBI 签名器 + await self.bili_client.init_wbi_sign(self.context_page) + + async def search_by_keywords(self) -> List[BilibiliVideo]: + """ + 按关键词搜索视频 + + 支持多个关键词（逗号分隔）。 + + Returns: + List[BilibiliVideo]: 视频列表 + """ + keywords = [kw.strip() for kw in settings.keywords.split(",") if kw.strip()] + + if not keywords: + logger.warning("[BilibiliCrawler] 未配置搜索关键词") + return [] + + logger.info(f"[BilibiliCrawler] 开始搜索，关键词: {keywords}") + + for keyword in keywords: + await self._search_single_keyword(keyword) + + # 达到最大数量后停止 + if len(self._results) >= self.max_video_count: + break + + return self._results + + async def _search_single_keyword(self, keyword: str): + """ + 搜索单个关键词 + + Args: + keyword: 搜索关键词 + """ + page = 1 + page_size = bilibili_config.SEARCH_PAGE_SIZE + + while len(self._results) < self.max_video_count: + logger.info(f"[BilibiliCrawler] 搜索 '{keyword}'，第 {page} 页") + + # 搜索视频 + videos = await self.bili_client.search_video_by_keyword( + keyword=keyword, + page=page, + page_size=page_size + ) + + if not videos: + logger.info(f"[BilibiliCrawler] '{keyword}' 第 {page} 页无结果，停止搜索") + break + + # 获取视频详情 + for video in videos: + if len(self._results) >= self.max_video_count: + break + + # 获取完整视频详情 + video_detail = await self.bili_client.get_video_info(bvid=video.bvid) + if video_detail: + video_detail.source_keyword = keyword + self._results.append(video_detail) + logger.info(f"[BilibiliCrawler] 获取视频: {video_detail.title[:30]}...") + else: + # 如果获取详情失败，使用搜索结果 + self._results.append(video) + + # 随机延迟 + await self._random_delay() + + page += 1 + + # 防止无限循环 + if page > 50: + break + + async def get_specified_videos(self) -> List[BilibiliVideo]: + """ + 获取指定视频列表的详情 + + 从配置中读取视频列表（BV号或URL）。 + + Returns: + List[BilibiliVideo]: 视频列表 + """ + video_list = settings.specified_id_list + + if not video_list: + logger.warning("[BilibiliCrawler] 未配置指定视频列表") + return [] + + logger.info(f"[BilibiliCrawler] 获取 {len(video_list)} 个指定视频") + + for video_id in video_list: + if len(self._results) >= self.max_video_count: + break + + # 解析 BV 号 + try: + video_info = parse_video_info_from_url(video_id) + bvid = video_info.video_id + except ValueError: + logger.warning(f"[BilibiliCrawler] 无法解析视频 ID: {video_id}") + continue + + # 获取视频详情 + video = await self.bili_client.get_video_info(bvid=bvid) + if video: + self._results.append(video) + logger.info(f"[BilibiliCrawler] 获取视频: {video.title[:30]}...") + + # 随机延迟 + await self._random_delay() + + return self._results + + async def _random_delay(self): + """随机延迟，避免请求过快""" + delay = random.uniform(self.delay_min, self.delay_max) + await asyncio.sleep(delay) + + def get_results(self) -> List[BilibiliVideo]: + """ + 获取爬取结果 + + Returns: + List[BilibiliVideo]: 视频列表 + """ + return self._results + + async def close(self): + """关闭浏览器""" + if self.browser_manager: + await self.browser_manager.close() + logger.info("[BilibiliCrawler] 浏览器已关闭") + + +async def run_crawler( + on_video: Optional[Callable[[BilibiliVideo], Any]] = None +) -> List[BilibiliVideo]: + """ + 运行爬虫（便捷函数） + + Args: + on_video: 每获取一个视频后的回调函数 + + Returns: + List[BilibiliVideo]: 视频列表 + """ + crawler = BilibiliCrawler() + return await crawler.start() + + +if __name__ == '__main__': + # 测试代码 + async def test(): + crawler = BilibiliCrawler() + videos = await crawler.start() + + print(f"\n爬取完成，共 {len(videos)} 个视频：") + for i, video in enumerate(videos[:5], 1): + print(f"{i}. {video.title}") + print(f" UP主: {video.nickname}") + print(f" 播放: {video.play_count}") + print() + + asyncio.run(test()) diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/login/__init__.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/login/__init__.py" new file mode 100644 index 0000000..5923317 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/login/__init__.py" @@ -0,0 +1,19 @@ +# -*- coding: utf-8 -*- +"""登录认证模块""" +from .auth import ( + BilibiliLogin, + AbstractLogin, + convert_cookies_to_str, + convert_cookies_to_dict, + save_cookies_to_file, + load_cookies_from_file, +) + +__all__ = [ + 'BilibiliLogin', + 'AbstractLogin', + 'convert_cookies_to_str', + 'convert_cookies_to_dict', + 'save_cookies_to_file', + 'load_cookies_from_file', +] diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/login/auth.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/login/auth.py" new file mode 100644 index 0000000..41cd240 --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/login/auth.py" @@ -0,0 +1,487 @@ +# -*- coding: utf-8 -*- +""" +B站登录认证模块 + +本模块实现了 B站的登录认证功能，支持两种登录方式： +1. 扫码登录：显示二维码，用户使用 B站 APP 扫码登录 +2. Cookie 登录：使用已有的 Cookie 字符串直接登录 + +登录成功后会将 Cookie 保存到浏览器上下文中，后续请求自动携带。 + +参考 MediaCrawler 项目的实现： +- https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/login.py +""" + +import asyncio +import base64 +import json +from abc import ABC, abstractmethod +from pathlib import Path +from typing import Optional, List, Dict +from loguru import logger + +# 可选依赖 +try: + from playwright.async_api import BrowserContext, Page + HAS_PLAYWRIGHT = True +except ImportError: + HAS_PLAYWRIGHT = False + +# 尝试导入图片显示库 +try: + from PIL import Image + from io import BytesIO + HAS_PIL = True +except ImportError: + HAS_PIL = False + +# 尝试导入 httpx +try: + import httpx + HAS_HTTPX = True +except ImportError: + HAS_HTTPX = False + + +# ==================== B站登录相关常量 ==================== + +# B站主页 +BILIBILI_URL = "https://www.bilibili.com" + +# 登录按钮选择器 +LOGIN_BUTTON_SELECTOR = "xpath=//div[@class='right-entry__outside go-login-btn']//div" + +# 二维码选择器 +QRCODE_SELECTOR = "//div[@class='login-scan-box']//img" + +# 登录成功后的 Cookie 关键字段 +LOGIN_COOKIE_KEYS = ["SESSDATA", "DedeUserID", "bili_jct"] + + +class AbstractLogin(ABC): + """登录抽象基类""" + + @abstractmethod + async def begin(self) -> bool: + """开始登录流程""" + pass + + @abstractmethod + async def login_by_qrcode(self) -> bool: + """扫码登录""" + pass + + @abstractmethod + async def login_by_cookies(self) -> bool: + """Cookie 登录""" + pass + + @abstractmethod + async def check_login_state(self) -> bool: + """检查登录状态""" + pass + + +class BilibiliLogin(AbstractLogin): + """ + B站登录类 + + 支持扫码登录和 Cookie 登录两种方式。 + + 使用示例： + ```python + login = BilibiliLogin( + login_type="qrcode", + browser_context=context, + context_page=page + ) + success = await login.begin() + ``` + """ + + def __init__( + self, + login_type: str, + browser_context: "BrowserContext", + context_page: "Page", + cookie_str: str = "", + ): + """ + 初始化 B站登录 + + Args: + login_type: 登录类型，"qrcode" 或 "cookie" + browser_context: Playwright 浏览器上下文 + context_page: Playwright 页面对象 + cookie_str: Cookie 字符串（当 login_type="cookie" 时使用） + """ + self.login_type = login_type + self.browser_context = browser_context + self.context_page = context_page + self.cookie_str = cookie_str + + async def begin(self) -> bool: + """ + 开始登录流程 + + 根据 login_type 自动选择登录方式。 + + Returns: + bool: 登录是否成功 + """ + logger.info(f"[BilibiliLogin] 开始登录，方式: {self.login_type}") + + if self.login_type == "qrcode": + return await self.login_by_qrcode() + elif self.login_type == "cookie": + return await self.login_by_cookies() + else: + logger.error(f"[BilibiliLogin] 不支持的登录类型: {self.login_type}") + return False + + async def login_by_qrcode(self) -> bool: + """ + 扫码登录 + + 流程： + 1. 访问 B站首页 + 2. 点击登录按钮 + 3. 获取二维码图片并显示 + 4. 等待用户扫码 + 5. 检查登录状态 + + Returns: + bool: 登录是否成功 + """ + logger.info("[BilibiliLogin] 开始扫码登录...") + + try: + # 1. 访问 B站首页 + await self.context_page.goto(BILIBILI_URL) + await asyncio.sleep(2) + + # 2. 点击登录按钮 + try: + login_button = await self.context_page.wait_for_selector( + LOGIN_BUTTON_SELECTOR, + timeout=10000 + ) + if login_button: + await login_button.click() + await asyncio.sleep(1) + except Exception as e: + logger.warning(f"[BilibiliLogin] 点击登录按钮失败: {e}") + # 可能已经有登录弹窗，继续尝试 + + # 3. 获取二维码 + qrcode_img = await self._find_login_qrcode() + if not qrcode_img: + logger.error("[BilibiliLogin] 未找到二维码") + return False + + # 4. 显示二维码 + await self._show_qrcode(qrcode_img) + + # 5. 等待登录成功 + logger.info("[BilibiliLogin] 请使用 B站 APP 扫描二维码登录...") + logger.info("[BilibiliLogin] 等待登录成功（最长等待 120 秒）...") + + # 轮询检查登录状态 + for _ in range(120): # 最多等待 120 秒 + if await self.check_login_state(): + logger.info("[BilibiliLogin] 扫码登录成功！") + await asyncio.sleep(2) # 等待页面跳转 + return True + await asyncio.sleep(1) + + logger.error("[BilibiliLogin] 扫码登录超时") + return False + + except Exception as e: + logger.error(f"[BilibiliLogin] 扫码登录失败: {e}") + return False + + async def login_by_cookies(self) -> bool: + """ + Cookie 登录 + + 将 Cookie 字符串解析后注入到浏览器上下文中。 + + Returns: + bool: 登录是否成功 + """ + logger.info("[BilibiliLogin] 开始 Cookie 登录...") + + if not self.cookie_str: + logger.error("[BilibiliLogin] Cookie 字符串为空") + return False + + try: + # 解析 Cookie 字符串 + cookies = self._parse_cookie_str(self.cookie_str) + if not cookies: + logger.error("[BilibiliLogin] Cookie 解析失败") + return False + + # 注入 Cookie + await self.browser_context.add_cookies(cookies) + logger.info(f"[BilibiliLogin] 成功注入 {len(cookies)} 个 Cookie") + + # 刷新页面验证 + await self.context_page.goto(BILIBILI_URL) + await asyncio.sleep(2) + + # 检查登录状态 + if await self.check_login_state(): + logger.info("[BilibiliLogin] Cookie 登录成功！") + return True + else: + logger.error("[BilibiliLogin] Cookie 登录失败，Cookie 可能已过期") + return False + + except Exception as e: + logger.error(f"[BilibiliLogin] Cookie 登录失败: {e}") + return False + + async def check_login_state(self) -> bool: + """ + 检查登录状态 + + 通过检查 Cookie 中是否包含关键字段来判断是否已登录。 + 关键字段：SESSDATA、DedeUserID + + Returns: + bool: 是否已登录 + """ + try: + # 获取当前 Cookie + cookies = await self.browser_context.cookies() + cookie_dict = {c['name']: c['value'] for c in cookies} + + # 检查关键 Cookie 是否存在 + for key in ["SESSDATA", "DedeUserID"]: + if key in cookie_dict and cookie_dict[key]: + return True + + return False + + except Exception as e: + logger.debug(f"[BilibiliLogin] 检查登录状态出错: {e}") + return False + + async def _find_login_qrcode(self) -> Optional[str]: + """ + 查找登录二维码 + + Returns: + str: Base64 编码的二维码图片，如果未找到返回 None + """ + try: + # 等待二维码出现 + qrcode_element = await self.context_page.wait_for_selector( + QRCODE_SELECTOR, + timeout=10000 + ) + + if not qrcode_element: + return None + + # 获取二维码图片 src + qrcode_src = await qrcode_element.get_attribute("src") + + if not qrcode_src: + return None + + # 如果是 URL，下载图片 + if qrcode_src.startswith("http"): + return await self._download_qrcode(qrcode_src) + + # 如果是 base64，直接返回 + if qrcode_src.startswith("data:image"): + return qrcode_src.split(",")[1] if "," in qrcode_src else qrcode_src + + return qrcode_src + + except Exception as e: + logger.error(f"[BilibiliLogin] 获取二维码失败: {e}") + return None + + async def _download_qrcode(self, url: str) -> Optional[str]: + """ + 下载二维码图片 + + Args: + url: 二维码图片 URL + + Returns: + str: Base64 编码的图片数据 + """ + if not HAS_HTTPX: + logger.warning("[BilibiliLogin] httpx 未安装，无法下载二维码") + return None + + try: + async with httpx.AsyncClient() as client: + response = await client.get(url, timeout=10) + if response.status_code == 200: + return base64.b64encode(response.content).decode('utf-8') + except Exception as e: + logger.error(f"[BilibiliLogin] 下载二维码失败: {e}") + + return None + + async def _show_qrcode(self, qrcode_base64: str): + """ + 显示二维码 + + 优先使用 PIL 显示，如果不可用则保存到文件。 + + Args: + qrcode_base64: Base64 编码的二维码图片 + """ + # 解码 base64 + try: + qrcode_bytes = base64.b64decode(qrcode_base64) + except Exception: + logger.error("[BilibiliLogin] 二维码 Base64 解码失败") + return + + # 保存到文件 + qrcode_path = Path("qrcode.png") + with open(qrcode_path, 'wb') as f: + f.write(qrcode_bytes) + logger.info(f"[BilibiliLogin] 二维码已保存到: {qrcode_path.absolute()}") + + # 尝试显示图片 + if HAS_PIL: + try: + image = Image.open(BytesIO(qrcode_bytes)) + image.show() + logger.info("[BilibiliLogin] 二维码已显示，请扫码登录") + except Exception as e: + logger.warning(f"[BilibiliLogin] 无法显示二维码: {e}") + logger.info(f"[BilibiliLogin] 请手动打开文件: {qrcode_path.absolute()}") + else: + logger.info(f"[BilibiliLogin] 请手动打开文件扫码: {qrcode_path.absolute()}") + + # 打印提示 + print("\n" + "=" * 60) + print(" 请使用 B站 APP 扫描二维码登录") + print(f" 二维码文件: {qrcode_path.absolute()}") + print(" 等待登录中...") + print("=" * 60 + "\n") + + def _parse_cookie_str(self, cookie_str: str) -> List[Dict]: + """ + 解析 Cookie 字符串 + + Args: + cookie_str: Cookie 字符串，格式如 "name1=value1; name2=value2" + + Returns: + List[Dict]: Playwright 格式的 Cookie 列表 + """ + cookies = [] + + for item in cookie_str.split(";"): + item = item.strip() + if not item or "=" not in item: + continue + + parts = item.split("=", 1) + name = parts[0].strip() + value = parts[1].strip() if len(parts) > 1 else "" + + if name: + cookies.append({ + "name": name, + "value": value, + "domain": ".bilibili.com", + "path": "/" + }) + + return cookies + + +# ==================== 工具函数 ==================== + +def convert_cookies_to_str(cookies: List[Dict]) -> str: + """ + 将 Cookie 列表转换为字符串 + + Args: + cookies: Playwright 格式的 Cookie 列表 + + Returns: + str: Cookie 字符串 + """ + return "; ".join([f"{c['name']}={c['value']}" for c in cookies]) + + +def convert_cookies_to_dict(cookies: List[Dict]) -> Dict[str, str]: + """ + 将 Cookie 列表转换为字典 + + Args: + cookies: Playwright 格式的 Cookie 列表 + + Returns: + Dict[str, str]: Cookie 字典 + """ + return {c['name']: c['value'] for c in cookies} + + +async def save_cookies_to_file(context: "BrowserContext", filepath: str): + """ + 保存 Cookie 到文件 + + Args: + context: Playwright 浏览器上下文 + filepath: 保存路径 + """ + cookies = await context.cookies() + with open(filepath, 'w', encoding='utf-8') as f: + json.dump(cookies, f, indent=2, ensure_ascii=False) + logger.info(f"[BilibiliLogin] Cookie 已保存到: {filepath}") + + +async def load_cookies_from_file(context: "BrowserContext", filepath: str) -> bool: + """ + 从文件加载 Cookie + + Args: + context: Playwright 浏览器上下文 + filepath: Cookie 文件路径 + + Returns: + bool: 是否加载成功 + """ + path = Path(filepath) + if not path.exists(): + logger.warning(f"[BilibiliLogin] Cookie 文件不存在: {filepath}") + return False + + try: + with open(filepath, 'r', encoding='utf-8') as f: + cookies = json.load(f) + await context.add_cookies(cookies) + logger.info(f"[BilibiliLogin] 从文件加载了 {len(cookies)} 个 Cookie") + return True + except Exception as e: + logger.error(f"[BilibiliLogin] 加载 Cookie 失败: {e}") + return False + + +if __name__ == '__main__': + # 测试 Cookie 解析 + test_cookie_str = "SESSDATA=abc123; DedeUserID=12345; bili_jct=xyz789" + login = BilibiliLogin( + login_type="cookie", + browser_context=None, + context_page=None, + cookie_str=test_cookie_str + ) + cookies = login._parse_cookie_str(test_cookie_str) + print("解析后的 Cookie:") + for c in cookies: + print(f" {c['name']}: {c['value']}") diff --git "a/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/main.py" "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/main.py" new file mode 100644 index 0000000..3491b7c --- /dev/null +++ "b/\346\272\220\344\273\243\347\240\201/\347\210\254\350\231\253\350\277\233\351\230\266/11_\350\277\233\351\230\266\347\273\274\345\220\210\345\256\236\346\210\230\351\241\271\347\233\256/main.py" @@ -0,0 +1,232 @@ +# -*- coding: utf-8 -*- +""" +B站视频数据采集工具 - 主程序入口 + +综合实战项目：B站视频数据采集与分析工具 + +功能特点： +- 多种登录方式（扫码登录 / Cookie 登录） +- 反检测浏览器自动化（Playwright + stealth.js） +- WBI 签名算法支持（B站 API 签名） +- 视频搜索和详情获取 +- 多格式数据存储（JSON / CSV） +- 词云和统计报告自动生成 + +参考 MediaCrawler 项目的实现： +- https://github.com/NanmiCoder/MediaCrawler + +使用方法： + python main.py + +配置说明： + 修改 config/settings.py 中的配置项： + - crawler_type: 爬取类型（search=搜索, detail=指定视频） + - keywords: 搜索关键词（逗号分隔） + - max_video_count: 最大爬取数量 + - login_type: 登录方式（qrcode=扫码, cookie=Cookie） +""" + +import asyncio +import sys +from pathlib import Path +from typing import List +from loguru import logger + +# 添加项目根目录到路径 +sys.path.insert(0, str(Path(__file__).parent)) + +# 导入各模块 +from config import settings, CrawlerType +from crawler.spider import BilibiliCrawler +from store.backend import StorageManager +from analysis.report import ReportGenerator, generate_report +from models.bilibili import BilibiliVideo + + +# 配置日志 +def setup_logger(): + """配置日志""" + logger.remove() + logger.add( + sys.stderr, + format="