簡單的Google爬蟲
網站的傳輸結構
假設我們要爬取的關鍵字為【proxy list filetype:txt】
會看到輸入搜索後，網址為
https://www.google.com/search?ei=edWMXeLuLozW0gTJ557IDw&q=proxy+list+filetype%3Atxt&oq=proxy+list+filetype%3Atxt&gs_l=psy-ab.3...1044.1044..1239...0.0..0.33.33.1......0....1..gws-wiz.LRfqMT7q3m8&ved=0ahUKEwiij4Wj4-7kAhUMq5QKHcmzB_kQ4dUDCAs&uact=5一般來說你看不懂的那堆亂碼，通常是網站的追踨碼，實際上有意義的參數應該只有
q=proxy+list+filetype%3Atxt
嘗試將網址改成
https://www.google.com/search?q=proxy+list+filetype%3Atxt
會發現一樣會提供搜索結果進行程式開發
初步開發
#一個非常簡單的Google爬蟲
import requests
def Google_Run(keyword):#函式只要輸入關鍵字就行
url = "https://www.google.com/search?q={}".format(keyword)#組合網址
rs = requests.get(url)#請求連結
print(rs.text)
fp = open("google_rs.txt",'w',encoding="utf-8")#保存資料
fp.write(rs.text)
fp.close()
Google_Run("proxy list filetype:txt")#觸發函式
得到的輸出為：
<!doctype html><html lang="zh-TW"><head><meta charset="UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>proxy list filetype:txt - Google 搜尋</title><script nonce="plzeSSnxl3tB4sbNbCkAQA==">(function(){var a=window.performance;window.start=(new Date).getTime();a:{var b=window;if(a){var c=a.timing;if(c){var d=c.navigationStart,e=c.responseStart;if(e>d&&e<=window.start){window.start=e;b.wsrt=e-d;break a}}a.now&&(b.wsrt=Math.floor(a.now()))}}window.google=window.google||{};google.aft=function(f){f.setAttribute("data-iml",+new Date)};}).call(this);(function(){window.jsarwt=function(){return!1};}).call(this);(function(){var c=[],e=0;window.ping=function(b){-1==b.indexOf("&zx")&&(b+="&zx="+(new Date).getTime());var a=new Image,d=e++;c[d]=a;a.onerror=a.onload=a.onabort=function(){delete c[d]};a.src=b};}).call(this);</script><style>body{margin:0 auto;max-width:736px;padding:0 8px}a{color:#1967D2;text-decoration:none;tap-highlight-color:rgba(0,0,0,.1)}a:visited{color:#4B11A8}a:hover{text-decoration:underline}img{border:0}html{font-family:Roboto,HelveticaNeue,Arial,sans-serif;font-size:14px;line-height:20px;text-size-adjust:100%;color:#3C4043;word-wrap:break-word;background-color:#fff}.bRsWnc{background-color:#fff;border-top:1px solid #e0e0e0;height:39px;overflow:hidden}.N6RWV{height:51px;overflow-scrolling:touch;overflow-x:auto;overflow-y:hidden}.Uv67qb{box-pack:justify;font-size:12px;line-height:37px;justify-content:space-between;justify-content:space-between}.Uv67qb a,.Uv67qb span{color:#757575;display:block;flex:none;padding:0 16px;text-align:center;text-transform:uppercase;}span.OXXup{border-bottom:2px solid #4285f4;color:#4285f4;font-weight:bold}a.eZt8xd:visited{color:#757575}.FElbsf{border-left:1px solid rgba(0,0,0,.12)}header article{overflow:visible}.Pg70bf{height:39px;display:box;display:flex;display:flex;width:100%}.H0PQec{position:relative;flex:1}.sbc{display:flex;width:100%}.Pg70bf input{margin:2px 4px 2px 8px;}.x{width:26px;color:#757575;font:27px/38px arial, sans-serif;line-height:40px;}#qdClwb{flex:0 0 auto;width:39px;height:39px;border-bottom:0;padding:0;border-top-right-radius:8px;background-color:#3b78e7;border:1px solid #3367d6;background-image:url(data:image/gif;base64,R0lGODdhJAAjAPIHAODr/nCk+MPZ/FmV96zK+/7+/5K5+kqL9iwAAAAAJAAjAEADani63P4wykmbKcQRXDscQAEMXmmeaLQVLCukzyC09AjfeK7v/MAajACLhPMVAgwjsUcEiZa8xgAYrVqv2Kx2iwsIAAABknfBBAKrTE4IcMyot8ur8datqIbQfJdnAfo2WE6BV05wXIiJigkAOw==);}.sc{font-size:;position:absolute;top:39px;left:0;right:0;box-shadow:0px 2px 5px rgba(0,0,0,0.2);z-index:2;background-color:#fff}.sc>div{padding:10px 10px;padding-left:16px;padding-left:14px;border-top:1px solid #DFE1E5}.scs{background-color:#f5f5f5;}.noHIxc{display:block;font-size:16px;padding:0 0 0 8px;flex:1;height:35px;outline:none;border:none;width:100%;-webkit-tap-highlight-color:rgba(0,0,0,0);overflow:hidden;}.sbc input[type=text]{background:none}.sml .cOl4Id{display:none}.l{display:none}.sml header{background:none}.sml .l{display:block;padding:0 8px}.sml .l{letter-spacing:-1px;text-align:center;border-radius:2px 0 0 0;font:22px/36px Futura, Arial, sans-serif;font-smoothing:antialiased}.bz1lBb{background:#fff;border-radius:8px 8px 0 0;box-shadow:0 1px 6px rgba(32, 33, 36, 0.18);margin-top:10px}.KP7LCb{border-radius:0 0 8px 8px;box-shadow:0 2px 3px rgba(32, 33, 36, 0.18);margin-bottom:10px;overflow:hidden}.cOl4Id{letter-spacing:-1px;text-align:center;font:22pt Futura, Arial, sans-serif;padding:10px 0 5px 0;height:37px;font-smoothing:antialiased}.cOl4Id span{display:inline-block}.S591j{height:100%}.V6gwVd{color:#4285F4}.iWkuvd{color:#EA4335}.cDrQ7{color:#fbcc05}.ntlR9{color:#34A853}.tJ3Myc{-webkit-transform:rotate(-20deg);position:relative;left:-1px;display:inline-block}footer{text-align:center;margin-top:18px}footer a,footer a:visited,.smiUbb{color:#5f6368}.ksTU4c{margin:0 13px}#mCljob{margin-top:36px}#mCljob>div{margin:20px}</style></head><body jsmodel=" "><header id="hdr"><script nonce="plzeSSnxl3tB4sbNbCkAQA==">(function(){var c=500;(function(){window.screen&&window.screen.width<=c&&window.screen.height<=c&&document.getElementById("hdr").classList.add("sml");}).call(this);})();</script><div class="cOl4Id"><a href="/?sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQOwgC"><span class="V6gwVd">G</span><span class="iWkuvd">o</span><span class="cDrQ7">o</span><span class="V6gwVd">g</span><span class="ntlR9">l</span><span class="iWkuvd tJ3Myc">e</span></a></div><div class="bz1lBb"><form class="Pg70bf" id="sf"><a class="l" href="/?output=search&ie=UTF-8&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQPAgE"><span class="V6gwVd">G</span><span class="iWkuvd">o</span><span class="cDrQ7">o</span><span class="V6gwVd">g</span><span class="ntlR9">l</span><span class="iWkuvd tJ3Myc">e</span></a><input name="ie" value="ISO-8859-1" type="hidden"><div class="H0PQec"><div class="sbc esbc"><input class="noHIxc" value="proxy list filetype:txt" autocapitalize="none" autocomplete="off" name="q" spellcheck="false" type="text"><input name="oq" type="hidden"><input name="aqs" type="hidden"><div class="x">×</div><div class="sc"></div></div></div><button id="qdClwb" type="submit"></button></form></div><noscript><meta content="0;url=/search?q=proxy+list+filetype:txt&ie=UTF-8&gbv=1&sei=i9eMXbHyDoKkmAWBjYOwCg" http-equiv="refresh"><style>table,div,span,p{display:none}</style><div style="display:block">如果系統沒有在數秒鐘後將您重新導向，請按一下<a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&gbv=1&sei=i9eMXbHyDoKkmAWBjYOwCg">這裡</a>。</div></noscript></header><div id="main"><div><div class="KP7LCb"> <div class="bRsWnc"> <div class="N6RWV"> <div class="Pg70bf Uv67qb"> <span class="OXXup">全部</span><a class="eZt8xd" href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ_AUICCgB">圖片</a><a class="eZt8xd" href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnms&tbm=nws&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ_AUICSgC">新聞</a><a class="eZt8xd" href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnms&tbm=vid&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ_AUICigD">影片</a> <a href="https://maps.google.com/maps?q=proxy+list+filetype:txt&um=1&ie=UTF-8&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ_AUICygE">地圖</a> <a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnms&tbm=bks&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ_AUIDCgF">書籍</a> <div class="FElbsf"><a href="/advanced_search" style="white-space:nowrap" id="st-toggle" role="button">搜尋工具</a></div> </div> </div> </div> </div><div class="Pg70bf wEsjbd ZINbbc xpd O9g5cc uUPGi" style="display:none" id="st-card"><style>.wEsjbd{background-color:#fff;height:44px;white-space:nowrap}.coPU8c{height:60px;overflow-scrolling:touch;overflow-x:auto;overflow-y:hidden}.Xj2aue{height:44px;overflow:hidden}.RnNGze{margin:11px 16px}.wEsjbd div,.wEsjbd a,.wEsjbd li{outline-width:0;outline:none}</style><div class="Xj2aue"><div class="coPU8c"><div class="RnNGze"><style>.PA9J5{display:inline-block}.RXaOfd{display:inline-block;height:22px;position:relative;padding-top:0;padding-bottom:0;padding-right:16px;padding-left:0;line-height:22px;cursor:pointer;text-transform:uppercase;font-size:12px;color:#757575}.sa1toc{display:none;position:absolute;background:#fff;border:1px solid #d6d6d6;box-shadow:0 2px 4px rgba(0,0,0,0.3);margin:0;white-space:nowrap;z-index:103;line-height:17px;padding-top:5px;padding-bottom:5px;padding-left:0px}.PA9J5:hover .sa1toc{display:block}.mGSy8d a:active,.RXaOfd:active{color:#4285f4}</style><div class="PA9J5"><div class="RXaOfd" role="button" tabindex="0"><style>.TWMOUc{display:inline-block;padding-right:14px;white-space:nowrap}.vQYuGf{font-weight:bold}.OmTIzf{border-color:#909090 transparent;border-style:solid;border-width:4px 4px 0 4px;width:0;height:0;margin-left:-10px;top:50%;margin-top:-2px;position:absolute}.RXaOfd:active .OmTIzf{border-color:#4285f4 transparent}</style><div class="TWMOUc">不限語言</div><span class="OmTIzf"></span></div><ul class="sa1toc ozatM"><style>.ozatM{font-size:12px;text-transform:uppercase}.ozatM .yNFsl,.ozatM li{list-style-type:none;list-style-position:outside;list-style-image:none}.yNFsl.SkUj4c,.yNFsl a{color:rgba(0,0,0,0.54);text-decoration:none;padding:6px 44px 6px 14px;line-height:17px;display:block}.SkUj4c{background-image:url(//ssl.gstatic.com/ui/v1/menu/checkmark2.png);background-position:right center;background-repeat:no-repeat}.SkUj4c:active{background-color:#f5f5f5}</style><li class="yNFsl SkUj4c">不限語言</li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=lr:lang_1zh-CN%7Clang_1zh-TW&lr=lang_zh-CN%7Clang_zh-TW&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIDg">搜尋所有中文網頁</a></li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=lr:lang_1zh-TW&lr=lang_zh-TW&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIDw">搜尋繁體中文網頁</a></li></ul></div><div class="PA9J5"><div class="RXaOfd" role="button" tabindex="0"><div class="TWMOUc">不限時間</div><span class="OmTIzf"></span></div><ul class="sa1toc ozatM"><li class="yNFsl SkUj4c">不限時間</li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=qdr:h&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIEQ"> 過去 1 小時</a></li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=qdr:d&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIEg"> 過去 24 小時</a></li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=qdr:w&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIEw"> 過去 1 週</a></li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=qdr:m&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIFA"> 過去 1 個月</a></li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=qdr:y&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIFQ"> 過去 1 年</a></li></ul></div><div class="PA9J5"><div class="RXaOfd" role="button" tabindex="0"><div class="TWMOUc">所有結果</div><span class="OmTIzf"></span></div><ul class="sa1toc ozatM"><li class="yNFsl SkUj4c">所有結果</li><li class="yNFsl"><a href="/search?q=proxy+list+filetype:txt&ie=UTF-8&source=lnt&tbs=li:1&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQpwUIFw">一字不差</a></li></ul></div></div></div></div><script nonce="plzeSSnxl3tB4sbNbCkAQA==">(function(){var a=document.getElementById("st-toggle"),b=document.getElementById("st-card");a&&b&&a.addEventListener("click",function(c){b.style.display=b.style.display?"":"none";c.preventDefault()},!1);}).call(this);</script></div></div><style>.ZINbbc{background-color:#fff;margin-bottom:10px;box-shadow:0 1px 6px rgba(32, 33, 36, 0.28);border-radius:8px}.uUPGi{font-size:14px;line-height:20px;}.O9g5cc>*:first-child{border-top-left-radius:8px;border-top-right-radius:8px}.O9g5cc>*:last-child{border-bottom-left-radius:8px;border-bottom-right-radius:8px}.rl7ilb{display:block;clear:both}.kCrYT{padding:12px 16px 12px}a.fdYsqf{color:#4B11A8}.BNeawe{white-space:pre-line;word-wrap:break-word}.vvjwJb{color:#1967D2;font-size:16px;line-height:20px}.vvjwJb a:visited{color:#4B11A8}.vvjwJb.HrGdeb{color:rgba(255,255,255,1)}.vvjwJb.HrGdeb a:visited{color:rgba(255,255,255,.7)}.UPmit{font-size:14px;line-height:20px}.UPmit.HrGdeb{color:rgba(255,255,255,.7)}.UPmit.AP7Wnd{color:rgba(0,102,33,1)}.x54gtf{height:1px;background-color:#dfe1e5;margin:0 16px}.Ap5OSd{padding-bottom:12px}.s3v9rd{font-size:14px;line-height:20px}.s3v9rd.HrGdeb{color:rgba(255,255,255,1)}.s3v9rd.AP7Wnd{color:#202124}.mSx1Ee{padding-left:48px;margin:0}.v9i61e{padding-bottom:8px}.XLloXe{color:#1967D2;font-size:14px;line-height:20px}.XLloXe a:visited{color:#4B11A8}.XLloXe.HrGdeb{color:rgba(255,255,255,1)}.XLloXe.HrGdeb a:visited{color:rgba(255,255,255,.7)}.r0bn4c.tr0dw{color:rgba(255,255,255,.7)}.r0bn4c.rQMQod{color:#70757A}.ZTv9Bb{display:block}.deIvCb{font-size:16px;line-height:20px;font-weight:400}.deIvCb.HrGdeb{color:rgba(255,255,255,1)}.deIvCb.AP7Wnd{color:#202124}.FCUp0c{font-weight:bold}.X7NTVe{display:table;width:100%;padding-right:16px;box-sizing:border-box}.tHmfQe{display:table-cell;padding:12px 0 12px 16px}.UHtrk{width:72px}.HBTM6d{width:30px}.XS7yGd{display:table-cell;text-align:center;vertical-align:middle;padding:12px 0 12px 8px}.am3QBf{display:table;vertical-align:top}.Icx6Cd{margin:0 auto 8px}.mAdjQc{text-align:right}.uEec3{font-size:12px;line-height:16px}.uEec3.HrGdeb{color:rgba(255,255,255,.7)}.uEec3.AP7Wnd{color:#70757A}.rLshyf,.BmP5tf{padding-top:12px;padding-bottom:12px}.w1C3Le,.BmP5tf,.G5NbBd{padding-left:16px;padding-right:16px;}.G5NbBd{padding-bottom:12px}.nMymef{display:flex}.G5eFlf{flex:1;display:block}.nMymef span{text-align:center}</style><div><!--SW_C_X--></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=https://github.com/clarketm/proxy-list/blob/master/proxy-list.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAAegQIAhAB&usg=AOvVaw3UEuJtdXtrbi70XdDQGIjK"><div class="BNeawe vvjwJb AP7Wnd">proxy-list/proxy-list.txt at master · clarketm/proxy-list · GitHub</div><div class="BNeawe UPmit AP7Wnd">https://github.com › clarketm › proxy-list › blob › master › proxy-list</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">A list of free, public, forward proxy servers. UPDATED DAILY! - clarketm/proxy-list .</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=https://github.com/opsxcq/proxy-list/blob/master/list.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjABegQICRAB&usg=AOvVaw3DjfdK6yVR6K2jGPur2j2N"><div class="BNeawe vvjwJb AP7Wnd">proxy-list/list.txt at master · opsxcq/proxy-list · GitHub</div><div class="BNeawe UPmit AP7Wnd">https://github.com › opsxcq › proxy-list › blob › master › list</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">A curated list of free public proxy servers. Contribute to opsxcq/proxy-list development by creating an account on GitHub.</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=http://multiproxy.org/txt_all/proxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjACegQIChAB&usg=AOvVaw0Em2rK99uxr1ceeCNPHTVP"><div class="BNeawe vvjwJb AP7Wnd">anonymous and non-anonymous proxy list - MultiProxy</div><div class="BNeawe UPmit AP7Wnd">multiproxy.org › txt_all › proxy</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">沒有這個頁面的資訊。 · <span class="BNeawe"><a href="/url?q=https://support.google.com/webmasters/answer/7489871%3Fhl%3Dzh&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQvxowAnoECAoQAg&usg=AOvVaw1tpJ6a6KOO4nwT-f8gMeyV"><span class="XLloXe AP7Wnd"><span class="r0bn4c rQMQod">瞭解原因</span></span></a></span></div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=http://spys.me/proxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjADegQICBAB&usg=AOvVaw2cn3iMY-M8UBaalXhFryKX"><div class="BNeawe vvjwJb AP7Wnd">Proxy list updated at Thu, 26 Sep 19 08:55:09 +0300 Mirrors=http ...</div><div class="BNeawe UPmit AP7Wnd">spys.me › proxy</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">Proxy list updated at Thu, 26 Sep 19 08:55:09 +0300 Mirrors=http://spys.me/proxy .txt https://twitter.com/spys_one https://t.me/spys_one Support by donations: ...</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=http://www.freeproxy.ru/download/lists/goodproxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAEegQIBhAB&usg=AOvVaw1G_prbMjf_s-vng9wKi_8w"><div class="BNeawe vvjwJb AP7Wnd">Proxy List - Freeproxy.ru</div><div class="BNeawe UPmit AP7Wnd">www.freeproxy.ru › download › lists › goodproxy</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">http://www.FreeProxy.ru/ - this is site about proxies. Proxy FAQ, many free proxy lists, CGI proxy (anonymizers), programs to work with proxy and many more.</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=https://www.forumproxyleecher.com/forumlist.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAFegQIBRAB&usg=AOvVaw15mAWmvyijYnq6tnx_S507"><div class="BNeawe vvjwJb AP7Wnd">FPL Forum List (2019-09-23) # Format: forum link|topic link ...</div><div class="BNeawe UPmit AP7Wnd">https://www.forumproxyleecher.com › forumlist</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">... Leech List Statistics Report: http://fpl.my-proxy.com/liststat.php ## Rank 1, Non- forum, Fresh Proxies http://www.rmccurdy.com/scripts/proxy/good.txt ## Rank 2, ...</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=https://www.ietf.org/proceedings/52/I-D/draft-ietf-sip-dhcp-05.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAGegQIBBAB&usg=AOvVaw3F4ypLB3v0bxA9PW2z0aaQ"><div class="BNeawe vvjwJb AP7Wnd">DHCP Option for SIP Servers - IETF</div><div class="BNeawe UPmit AP7Wnd">https://www.ietf.org › proceedings › draft-ietf-sip-dhcp-05</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">H.Schulzrinne [Page 1] Internet Draft November 21, 2001 SIP server: As defined in RFC 2543 [2]. This server MUST be an outbound proxy server, as defined in ...</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=https://www.ietf.org/proceedings/50/I-D/sip-state-01.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAHegQIBxAB&usg=AOvVaw08wtkrqHhdGwCYlBgkQHZ_"><div class="BNeawe vvjwJb AP7Wnd">SIP Extensions for supporting Distributed Call State - IETF</div><div class="BNeawe UPmit AP7Wnd">https://www.ietf.org › proceedings › sip-state-01</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">This mechanism allows a proxy server to provide services that depend on call state, while still being stateless. 2. Conventions used in this document The key ...</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=http://docs.trendmicro.com/all/ent/ddan/v6.0/en-us/ddan_6.0_readme.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAIegQIARAB&usg=AOvVaw3zvvOEeihux2sqYu802xho"><div class="BNeawe vvjwJb AP7Wnd">Readme - Trend Micro</div><div class="BNeawe UPmit AP7Wnd">docs.trendmicro.com › all › ent › ddan › en-us › ddan_6.0_readme.txt</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">Using a proxy server configured with multiple accounts where each account uses a different authentication method may prevent some Deep Discovery Analyzer ...</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><a href="/url?q=http://www.binary-zone.com/files/MyProxyList.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAJegQIAxAB&usg=AOvVaw2eawRiQKrZEgxMYsc1281t"><div class="BNeawe vvjwJb AP7Wnd">IP address Anonymity level Country 217.174.98.198: 8080 ...</div><div class="BNeawe UPmit AP7Wnd">www.binary-zone.com › files › MyProxyList</div></a></div><div class="x54gtf"></div><div class="kCrYT"><div><div class="BNeawe s3v9rd AP7Wnd"><div><div><div class="BNeawe s3v9rd AP7Wnd">... proxy server Japan 208.66.171.216: 8080 anonymous proxy United States 213.132.44.92: 80 anonymous proxy server United Arab Emirates 164.100. 42.162: ...</div></div></div></div></div></div></div></div><div><div class="ZINbbc xpd O9g5cc uUPGi"><div class="kCrYT"><span><div class="BNeawe deIvCb AP7Wnd"><span class="FCUp0c rQMQod">相關搜尋</span></div></span></div><div class="x54gtf"></div><div class="X7NTVe"><a class="tHmfQe" href="/search?ie=UTF-8&q=proxy+list+github&sa=X&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ1QJ6BAgAEAE"><div class="am3QBf"><div><span><div class="BNeawe deIvCb AP7Wnd">proxy list github</div></span></div></div></a><div class="HBTM6d XS7yGd"><a href="/search?ie=UTF-8&q=proxy+list+github"><div class="BNeawe mAdjQc uEec3 AP7Wnd">></div></a></div></div><div class="x54gtf"></div><div class="X7NTVe"><a class="tHmfQe" href="/search?ie=UTF-8&q=jdownloader+proxy+list&sa=X&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ1QJ6BAgAEAI"><div class="am3QBf"><div><span><div class="BNeawe deIvCb AP7Wnd">jdownloader proxy list</div></span></div></div></a><div class="HBTM6d XS7yGd"><a href="/search?ie=UTF-8&q=jdownloader+proxy+list"><div class="BNeawe mAdjQc uEec3 AP7Wnd">></div></a></div></div><div class="x54gtf"></div><div class="X7NTVe"><a class="tHmfQe" href="/search?ie=UTF-8&q=proxy+list+api+free&sa=X&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ1QJ6BAgAEAM"><div class="am3QBf"><div><span><div class="BNeawe deIvCb AP7Wnd">proxy list api free</div></span></div></div></a><div class="HBTM6d XS7yGd"><a href="/search?ie=UTF-8&q=proxy+list+api+free"><div class="BNeawe mAdjQc uEec3 AP7Wnd">></div></a></div></div></div></div><footer> <div> <div class="ZINbbc xpd O9g5cc uUPGi BmP5tf"><div class="nMymef MUxGbd lyLwlc"><a class="nBDE1b G5eFlf" href="/search?q=proxy+list+filetype:txt&ie=UTF-8&ei=i9eMXbHyDoKkmAWBjYOwCg&start=10&sa=N" aria-label="下一頁">下一頁 ></a></div></div> </div> <div id="mCljob"><div><a href="/url?q=https://accounts.google.com/ServiceLogin%3Fcontinue%3Dhttps://www.google.com/search%253Fq%253Dproxy%252Blist%252Bfiletype:txt%26hl%3Dzh-TW&sa=U&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQxs8CCDQ&usg=AOvVaw2FgQT9QDqjPgiSap-96fB7">登入</a></div><div><a class="ksTU4c" href="https://www.google.com/preferences?hl=zh-TW&fg=1&sa=X&ved=0ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQ5fUCCDU">設定</a><a class="ksTU4c" href="//policies.google.com/privacy?fg=1">隱私權</a><a class="ksTU4c" href="//policies.google.com/terms?fg=1">服務條款</a></div></div> </footer><script nonce="plzeSSnxl3tB4sbNbCkAQA==">(function(){var hl='zh-TW';(function(){var b=this||self,d=/^[\w+/_-]+[=]{0,2}$/,e=null;var f=document.querySelector(".l"),g=document.querySelector("#sf"),k=g.querySelector(".sbc"),l=g.querySelector("[type=text]"),m=g.querySelector("[type=submit]"),n=g.querySelector(".sc"),p=g.querySelector(".x"),q=l.value,r=[],t=-1,u=q,w,x,y;q||(p&&(p.style.display="none"),z(!1));function z(a){if(k.classList.contains("esbc")){var c=k.classList.contains("chsbc"),h=k.classList.contains("rtlsbc");a&&(n.style.display="block",c?(g.style.borderRadius="20px 20px 0 0",n.style.borderBottom="1px solid #DFE1E5",m.style.borderRadius=h?"20px 0 0 0":"0 20px 0 0"):k.style.borderRadius=h?"0 8px 0 0":"8px 0 0 0");a||(n.style.display="none",c?(g.style.borderRadius="20px",n.style.borderBottom="none",m.style.borderRadius=h?"20px 0 0 20px":"0 20px 20px 0"):k.style.borderRadius=h?"0 8px 8px 0":"8px 0 0 8px")}}function A(){g.querySelector("[name=oq]").value=u;g.querySelector("[name=aqs]").value="heirloom-srp."+(0<=t?t:"")+"."+(0<r.length?"0l"+r.length:"")}
function C(){w=null;if(x){var a="/complete/search?client=heirloom-srp&hl="+hl+"&json=t&callback=hS&q="+encodeURIComponent(x);"undefined"!==typeof ds&&ds&&(a+="&ds="+ds);var c=document.createElement("script");c.src=a;if(null===e)a:{a=b.document;if((a=a.querySelector&&a.querySelector("script[nonce]"))&&(a=a.nonce||a.getAttribute("nonce"))&&d.test(a)){e=a;break a}e=""}(a=e)&&c.setAttribute("nonce",a);document.body.appendChild(c);x=null;w=setTimeout(C,500)}}
function D(){for(;n.firstChild;)n.removeChild(n.firstChild);r=[];t=-1;z(!1)}function E(){var a=n.querySelector(".scs");a&&(a.className="");0<=t?(a=n.childNodes[t],a.className="scs",q=a.textContent):q=u;l.value=q}l.addEventListener("focus",function(){f&&(f.style.display="none")},!1);l.addEventListener("blur",function(){D();f&&(f.style.display="")},!1);l.addEventListener("keyup",function(a){q=l.value;y=!1;13==a.which?A():27==a.which?(D(),f&&(f.style.display=""),q=u,l.value=q):40==a.which?(t++,t>=r.length&&(t=-1),E()):38==a.which?(t--,-1>t&&(t=r.length-1),E()):(a=q)?(p&&(p.style.display=""),x=a,w||C(),u=a):(p&&(p.style.display="none"),z(!1),D(),u="",y=!0)},!1);m.addEventListener("click",A,!1);p.addEventListener("click",function(){l.value="";p.style.display="none";z(!1)},!1);k.addEventListener("click",function(){l.focus()},!1);window.hS=function(a){if(!y){D();0==a[1].length&&z(!1);for(var c=0;c<a[1].length;c++){var h=a[1][c][0],v=document.createElement("div");v.innerHTML=h;v.addEventListener("mousedown",function(B){B.preventDefault();return!1},!1);h=h.replace(/<\/?b>/g,"");v.addEventListener("click",function(B){return function(){t=B;A();E();D();g.submit()}}(c),!1);n.appendChild(v);z(!0);r.push(h)}}};}).call(this);})();(function(){function b(a){for(a=a.target||a.srcElement;a&&"A"!=a.nodeName;)a=a.parentElement;a&&(a.href||"").match(/\/search.*[?&]tbm=isch/)&&(a.href+="&biw="+document.documentElement.clientWidth,a.href+="&bih="+document.documentElement.clientHeight)}document.addEventListener("click",b,!1);document.addEventListener("touchStart",b,!1);}).call(this);</script></div><!-- cctlcm 5 cctlcm --><textarea class="csi" name="csi" style="display:none"></textarea><script nonce="plzeSSnxl3tB4sbNbCkAQA==">(function(){var e='i9eMXbHyDoKkmAWBjYOwCg';var sn='web';(function(){var q=function(a){var c=window,d=document;if(!a||"none"==a.style.display)return 0;if(d.defaultView&&d.defaultView.getComputedStyle){var b=d.defaultView.getComputedStyle(a);if(b&&("hidden"==b.visibility||"0px"==b.height&&"0px"==b.width))return 0}if(!a.getBoundingClientRect)return 1;var f=a.getBoundingClientRect();a=f.left+c.pageXOffset;b=f.top+c.pageYOffset;var k=f.width;f=f.height;return 0>=f&&0>=k?0:0>b+f?2:b>=(c.innerHeight||d.documentElement.clientHeight)?3:0>a+k||a>=(c.innerWidth||d.documentElement.clientWidth)?4:1};var r=e,v=sn;function w(a,c,d){a="/gen_204?atyp=csi&s="+(v||"web")+"&t="+a+("&lite=1&ei="+r+"&conn="+(window.navigator&&window.navigator.connection?window.navigator.connection.type:-1)+c);c="&rt=";for(var b in d)a+=c+b+"."+d[b],c=",";return a}function x(a){a={prt:a};window.wsrt&&(a.wsrt=window.wsrt);return a}function y(a){window.ping?window.ping(a):(new Image).src=a}
(function(){for(var a=+new Date-window.start,c=x(a),d=0,b=0,f=0,k=document.getElementsByTagName("img"),m="&imn="+k.length+"&biw="+window.innerWidth+"&bih="+window.innerHeight,A=function(h,z){h.onload=function(){b=+new Date-window.start;z&&++n==f&&(d=b,t());h.onload=null}},t=function(){m+="&ima="+f;c.aft=d;y(w("aft",m,c))},n=0,B=0,g=void 0;g=k[B++];){var l=1==q(g);l&&++f;var u=g.complete&&!g.getAttribute("data-deferred"),p=u&&Number(g.getAttribute("data-iml"))||0;u&&p?(l&&++n,p&&(g=p-window.start,l&&(d=Math.max(d,g)),b=Math.max(b,g))):A(g,l)}d||(d=a);b||(b=d);n==f&&t();window.addEventListener("load",function(){window.setTimeout(function(){c.ol=+new Date-window.start;c.iml=b;var h=window.performance&&window.performance.timing;h&&(c.rqst=h.responseEnd-h.requestStart,c.rspt=h.responseEnd-h.responseStart);y(w("all",m,c))},0)},!1)})();}).call(this);})();</script><script nonce="plzeSSnxl3tB4sbNbCkAQA==">(function(){var e='i9eMXbHyDoKkmAWBjYOwCg';(function(){var a=e,b=window.performance&&window.performance.navigation;b&&2==b.type&&window.ping("/gen_204?ct=backbutton&ei="+a);}).call(this);})();(function(){(function(){google.csct={};google.csct.ps='AOvVaw2j5Ky71nvaeB6K5vsrBdLN\x26ust\x3d1569597707286945';})();})();(function(){(function(){google.csct.rd=true;})();})();google.drty&&google.drty();</script></body></html>
隨意搜了一下 手動測試會出現的結果關鍵字【aster · opsxcq/proxy-list · GitHub】
內容存在，所以這個功能ok!內容解析
透過Chrome F12的開發工具，會發現 幾乎每一個搜索的結果 都會帶一個
class="ellip" 的標記，我們簡單的搜索一下 會發現 class="ellip" 這個標記是專門為搜索結果設計的。
在經過程式測試後，發現ellip是找不到的，所以我們將保存下來的文件，簡單斷行後會發現如下圖的現象
一段搜索結果，通常是長這樣
<div class="ZINbbc xpd O9g5cc uUPGi">
<div class="kCrYT">
#這裡是連結
<a href="/url?q=http://www.freeproxy.ru/download/lists/goodproxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAEegQIBhAB&usg=AOvVaw1G_prbMjf_s-vng9wKi_8w">
#這裡是標題
<div class="BNeawe vvjwJb AP7Wnd">Proxy List - Freeproxy.ru</div>
----
<div class="ZINbbc xpd O9g5cc uUPGi">
<div class="kCrYT">
#這裡是連結
<a href="/url?q=http://spys.me/proxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjADegQICBAB&usg=AOvVaw2cn3iMY-M8UBaalXhFryKX"><div class="BNeawe vvjwJb AP7Wnd">Proxy list updated at Thu, 26 Sep 19 08:55:09 +0300 Mirrors=http ...</div>
#這裡是標題
<div class="BNeawe UPmit AP7Wnd">spys.me › proxy</div>
可以看到連結與標題 都是被包覆在 【kCrYT】這個div裡
連結是 <a XXXXX 在html裡 代表為 a tag
標題的class 一律會出現 【BNeawe】
所以程式的部份改為如下
#一個非常簡單的Google爬蟲
import requests
def Google_Run(keyword):#函式只要輸入關鍵字就行
url = "https://www.google.com/search?q={}".format(keyword)#組合網址
rs = requests.get(url)#請求連結
print(rs.text)
fp = open("google_rs.txt",'w',encoding="utf-8")#保存資料
fp.write(rs.text)
fp.close()
# Google_Run("proxy list filetype:txt")#觸發函式
import json #引用json
from pyquery import PyQuery as pq #引用pyquery 並且採用css 選擇器來進行資料抓取 ，安裝請下 pip install pyquery
def Decode_Result(google_data):
pq_data = pq(google_data) #初始化py_query
result_list = pq_data.find(".kCrYT") #透過這個標記 會發現 一共抓到21個結果
#實際上 google 一頁只會提供10個結果
end = [] #保存結果的陣列
for item in result_list:
link = pq(item).find("a").attr('href')
#簡單抓取每個內容裡面的連結，會發現不是確的 kCrYT div，會沒有link，所以做以下的判斷
if link is None: #link == 空值 就跳過
continue
print("link:{}".format(link))
title = pq(item).find("a").text()
if "瞭解原因" in title:#過濾掉 Google 的雜訊
continue
pack = {"title": title, "link": link} #將結果包裝成dict
end.append(pack)
return end
fp = open("google_rs.txt",'r',encoding="utf-8")#為了避免不斷查詢在Google被擋ip 所以用讀取的方式來進行測試
google_data = fp.read()
fp.close()
Google_rs = Decode_Result(google_data)
print(json.dumps(Google_rs))輸出結果為JSON格式
[{"title": "proxy-list/proxy-list.txt at master \u00b7 clarketm/proxy-list \u00b7 GitHub\nhttps://github.com \u203a clarketm \u203a proxy-list \u203a blob \u203a master \u203a proxy-list", "link": "/url?q=https://github.com/clarketm/proxy-list/blob/master/proxy-list.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAAegQIAhAB&usg=AOvVaw3UEuJtdXtrbi70XdDQGIjK"}, {"title": "proxy-list/list.txt at master \u00b7 opsxcq/proxy-list \u00b7 GitHub\nhttps://github.com \u203a opsxcq \u203a proxy-list \u203a blob \u203a master \u203a list", "link": "/url?q=https://github.com/opsxcq/proxy-list/blob/master/list.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjABegQICRAB&usg=AOvVaw3DjfdK6yVR6K2jGPur2j2N"}, {"title": "anonymous and non-anonymous proxy list - MultiProxy\nmultiproxy.org \u203a txt_all \u203a proxy", "link": "/url?q=http://multiproxy.org/txt_all/proxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjACegQIChAB&usg=AOvVaw0Em2rK99uxr1ceeCNPHTVP"}, {"title": "Proxy list updated at Thu, 26 Sep 19 08:55:09 +0300 Mirrors=http ...\nspys.me \u203a proxy", "link": "/url?q=http://spys.me/proxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjADegQICBAB&usg=AOvVaw2cn3iMY-M8UBaalXhFryKX"}, {"title": "Proxy List - Freeproxy.ru\nwww.freeproxy.ru \u203a download \u203a lists \u203a goodproxy", "link": "/url?q=http://www.freeproxy.ru/download/lists/goodproxy.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAEegQIBhAB&usg=AOvVaw1G_prbMjf_s-vng9wKi_8w"}, {"title": "FPL Forum List (2019-09-23) # Format: forum link|topic link ...\nhttps://www.forumproxyleecher.com \u203a forumlist", "link": "/url?q=https://www.forumproxyleecher.com/forumlist.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAFegQIBRAB&usg=AOvVaw15mAWmvyijYnq6tnx_S507"}, {"title": "DHCP Option for SIP Servers - IETF\nhttps://www.ietf.org \u203a proceedings \u203a draft-ietf-sip-dhcp-05", "link": "/url?q=https://www.ietf.org/proceedings/52/I-D/draft-ietf-sip-dhcp-05.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAGegQIBBAB&usg=AOvVaw3F4ypLB3v0bxA9PW2z0aaQ"}, {"title": "SIP Extensions for supporting Distributed Call State - IETF\nhttps://www.ietf.org \u203a proceedings \u203a sip-state-01", "link": "/url?q=https://www.ietf.org/proceedings/50/I-D/sip-state-01.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAHegQIBxAB&usg=AOvVaw08wtkrqHhdGwCYlBgkQHZ_"}, {"title": "Readme - Trend Micro\ndocs.trendmicro.com \u203a all \u203a ent \u203a ddan \u203a en-us \u203a ddan_6.0_readme.txt", "link": "/url?q=http://docs.trendmicro.com/all/ent/ddan/v6.0/en-us/ddan_6.0_readme.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAIegQIARAB&usg=AOvVaw3zvvOEeihux2sqYu802xho"}, {"title": "IP address Anonymity level Country 217.174.98.198: 8080 ...\nwww.binary-zone.com \u203a files \u203a MyProxyList", "link": "/url?q=http://www.binary-zone.com/files/MyProxyList.txt&sa=U&ved=2ahUKEwjx48Gf5e7kAhUCEqYKHYHGAKYQFjAJegQIAxAB&usg=AOvVaw2eawRiQKrZEgxMYsc1281t"}]以上就是簡單的Google爬蟲
Last updated
