[{"data":1,"prerenderedAt":349},["ShallowReactive",2],{"blog-post-/zh/blogs/cloud-based-scraping-solutions":3},{"id":4,"title":5,"body":6,"description":331,"extension":332,"meta":333,"navigation":344,"ogImage":335,"path":345,"seo":346,"stem":347,"__hash__":348},"zh/blogs/zh/11.cloud-based-scraping-solutions.md","网页抓取迁移到云端",{"type":7,"value":8,"toc":316},"minimal",[9,13,17,24,27,30,33,38,49,58,61,65,74,77,80,84,92,95,137,140,143,146,191,193,197,200,204,207,214,218,225,228,230,233,236,278,281,283,286,289,306,309,312],[10,11,12],"h2",{"id":12},"我尝试解决的问题",[14,15,16],"p",{},"上个月，我在做一个需要从多个JavaScript密集型网站抓取数据的项目。我一开始使用了常见的工具——本地运行Puppeteer和Playwright——但很快就遇到了瓶颈。我的笔记本风扇不停地转，内存使用率飙升，而且扩展到几个并发请求以上似乎是不可能的。",[14,18,19,20],{},"我心想：",[21,22,23],"em",{},"\"一定有更好的方法可以做到这一点，而不会把我的MacBook变成一个暖气片。\"",[14,25,26],{},"于是我开始研究基于云的替代方案，这些方案可以处理浏览器自动化的重任，同时保持我的本地环境轻量化。以下是我花了一周时间测试不同解决方案后的发现。",[28,29],"hr",{},[10,31,32],{"id":32},"我测试的基于云的无头浏览器服务",[34,35,37],"h3",{"id":36},"apify一站式平台","Apify：一站式平台",[14,39,40,41,48],{},"我首先尝试了",[42,43,47],"a",{"href":44,"rel":45},"https://apify.com",[46],"nofollow","Apify","，它感觉就像是\"网页抓取的AWS\"。他们的云平台提供预构建的\"Actors\"（可以理解为抓取用的无服务器函数），内置了代理、调度和存储等所有功能。",[14,50,51,52,57],{},"我最喜欢的是，我可以拿我现有的Node.js爬虫代码，做最小的修改，然后将其部署为无服务器Actor。他们提供的",[42,53,56],{"href":54,"rel":55},"https://crawlee.dev/",[46],"Crawlee","库使这个转换过程出奇地顺利。",[14,59,60],{},"仪表板让你可以查看运行情况、内存使用和日志——这比我在本地使用console.log进行调试的方式要先进得多。",[34,62,64],{"id":63},"browserlessio当你只需要浏览器时","Browserless.io：当你只需要浏览器时",[14,66,67,68,73],{},"接下来，我尝试了",[42,69,72],{"href":70,"rel":71},"https://browserless.io",[46],"Browserless.io","，它采用了更加专注的方法。它本质上是一个托管的无头Chrome和Playwright环境，你可以通过REST API访问。",[14,75,76],{},"最棒的部分？我可以保留大部分现有的Puppeteer代码，只需将其指向他们的服务，而不是启动本地浏览器。他们的API在后台处理所有的代理轮换和CAPTCHA解决方案。",[14,78,79],{},"对于已经投入使用Puppeteer/Playwright工作流的团队来说，这感觉是阻力最小的路径。",[34,81,83],{"id":82},"scrapingbee一行代码解决方案","ScrapingBee：一行代码解决方案",[14,85,86,91],{},[42,87,90],{"href":88,"rel":89},"https://scrapingbee.com",[46],"ScrapingBee","采用了一种不同的方法，在我只想要结果而不想费心处理浏览器自动化代码的日子里，我真的很欣赏这种方法。",[14,93,94],{},"他们的API非常简单——发送一个带有目标URL的HTTP请求，然后获取完全渲染的HTML。不需要管理浏览器实例，处理JavaScript执行，或担心被封禁。",[96,97,102],"pre",{"className":98,"code":99,"language":100,"meta":101,"style":101},"language-javascript shiki shiki-themes dracula","// 这真的就是全部所需的代码\nconst response = await fetch(\n  `https://app.scrapingbee.com/api/v1/?api_key=${apiKey}&url=${targetUrl}&render_js=true`,\n)\nconst html = await response.text()\n","javascript","",[103,104,105,113,119,125,131],"code",{"__ignoreMap":101},[106,107,110],"span",{"class":108,"line":109},"line",1,[106,111,112],{},"// 这真的就是全部所需的代码\n",[106,114,116],{"class":108,"line":115},2,[106,117,118],{},"const response = await fetch(\n",[106,120,122],{"class":108,"line":121},3,[106,123,124],{},"  `https://app.scrapingbee.com/api/v1/?api_key=${apiKey}&url=${targetUrl}&render_js=true`,\n",[106,126,128],{"class":108,"line":127},4,[106,129,130],{},")\n",[106,132,134],{"class":108,"line":133},5,[106,135,136],{},"const html = await response.text()\n",[14,138,139],{},"当我需要快速结果而不想承担浏览器自动化的认知负担时，我发现自己会选择这种方式。",[34,141,142],{"id":142},"其他值得一提的服务",[14,144,145],{},"我还简单测试了：",[147,148,149,161,171,181],"ul",{},[150,151,152,160],"li",{},[153,154,155],"strong",{},[42,156,159],{"href":157,"rel":158},"https://zyte.com",[46],"Zyte API","（前身是Splash）：凭借其反封禁技术，非常适合JavaScript密集型页面",[150,162,163,170],{},[153,164,165],{},[42,166,169],{"href":167,"rel":168},"https://phantombuster.com",[46],"PhantomBuster","：如果你需要自动化抓取之外的工作流程，这是一个很好的选择",[150,172,173,180],{},[153,174,175],{},[42,176,179],{"href":177,"rel":178},"https://scraping-bot.io",[46],"ScrapingBot","：使用真实浏览器规避检测的可靠选项",[150,182,183,190],{},[153,184,185],{},[42,186,189],{"href":187,"rel":188},"https://zyte.com/scrapy-cloud/",[46],"Scrapy Cloud","：如果你已经在使用Scrapy，这是完美的选择",[28,192],{},[10,194,196],{"id":195},"无服务器路线diy云抓取","无服务器路线：DIY云抓取",[14,198,199],{},"对于某些项目，我想要更多地控制执行环境，同时仍然避免本地资源限制。这时我开始探索在无服务器函数中部署无头浏览器：",[34,201,203],{"id":202},"aws-lambda实验","AWS Lambda实验",[14,205,206],{},"我在Lambda Layers中打包了带有无头Chrome的Puppeteer，令人惊讶的是它工作得很好。设置比使用专用服务更复杂，但对于我偶尔的抓取需求来说，按执行时间计费的模式很合理。",[14,208,209,210,213],{},"关键的发现是使用",[103,211,212],{},"chrome-aws-lambda","包，它提供了一个适当编译的Chromium版本，保持在Lambda的大小限制之内。",[34,215,217],{"id":216},"google-cloud-functions体验","Google Cloud Functions体验",[14,219,220,221,224],{},"Google Cloud Functions（第二代）设置起来更容易，因为它已经包含了无头Chrome所需的许多系统包。我只需要使用",[103,222,223],{},"puppeteer-core","来保持部署大小可控。",[14,226,227],{},"这种方法为我提供了最佳的控制和可扩展性平衡，特别是在那些需要围绕抓取过程进行自定义逻辑的项目中。",[28,229],{},[10,231,232],{"id":232},"我学到的经验和建议",[14,234,235],{},"经过几周的测试，如果你面临类似的挑战，以下是我的实用建议：",[147,237,238,247,256,264],{},[150,239,240,243,244,246],{},[153,241,242],{},"如果你刚刚开始","：先尝试",[153,245,90],{},"。单个API调用的简单性将为你节省数小时的设置时间。",[150,248,249,252,253,255],{},[153,250,251],{},"如果你有现有的Puppeteer/Playwright代码","：",[153,254,72],{},"提供了最平滑的过渡，只需最少的代码更改。",[150,257,258,252,261,263],{},[153,259,260],{},"对于大规模、复杂的项目",[153,262,47],{},"提供了最完整的生态系统，内置了调度、存储和代理管理。",[150,265,266,269,270,273,274,277],{},[153,267,268],{},"如果你熟悉云服务","：部署到",[153,271,272],{},"AWS Lambda","或",[153,275,276],{},"Google Cloud Functions","可以给你最大的控制权，潜在成本更低，特别是对于间歇性工作负载。",[14,279,280],{},"我学到的最大教训？**不要让基础设施问题限制你的数据收集项目。**这些云解决方案已经成熟到了这样一个程度：几乎没有什么好理由在本地机器上运行资源密集型浏览器。",[28,282],{},[10,284,285],{"id":285},"我的抓取项目下一步计划",[14,287,288],{},"目前，我采用了一种混合方法：",[147,290,291,296,301],{},[150,292,293,295],{},[153,294,90],{},"用于快速、一次性的抓取任务",[150,297,298,300],{},[153,299,47],{},"用于定期运行的计划数据收集作业",[150,302,303,305],{},[153,304,272],{},"用于需要自定义逻辑的专门抓取器",[14,307,308],{},"在我的下一篇文章中，我将分享一些实际的代码示例，展示我如何将这些服务集成到一个统一的数据管道中，为我的分析系统提供数据。",[14,310,311],{},"你尝试过这些服务吗？或者你有其他推荐的云抓取解决方案吗？请在评论中告诉我！",[313,314,315],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":101,"searchDepth":115,"depth":115,"links":317},[318,319,325,329,330],{"id":12,"depth":115,"text":12},{"id":32,"depth":115,"text":32,"children":320},[321,322,323,324],{"id":36,"depth":121,"text":37},{"id":63,"depth":121,"text":64},{"id":82,"depth":121,"text":83},{"id":142,"depth":121,"text":142},{"id":195,"depth":115,"text":196,"children":326},[327,328],{"id":202,"depth":121,"text":203},{"id":216,"depth":121,"text":217},{"id":232,"depth":115,"text":232},{"id":285,"depth":115,"text":285},"我探索了不需要在本地运行Puppeteer或Playwright等重型服务的基于云的无头浏览器抓取替代方案","md",{"date":334,"image":335,"alt":336,"tags":337,"category":340,"topics":341,"published":344},"5th May 2025","/blogs-img/blog4.jpg","基于云的网页抓取解决方案",[338,339],"Engineering","Productivity","product-execution",[342,343],"execution","ai-native",true,"/blogs/zh/cloud-based-scraping-solutions",{"title":5,"description":331},"blogs/zh/11.cloud-based-scraping-solutions","uM9QzpKp_NhPukz5GZY1gr77cNsjvKMMxPT4VW0tsZQ",1781532114662]