python - How to connect to https site with Scrapy via Polipo over TOR? -

-

not entirely sure problem here.

running python 2.7.3, , scrapy 0.16.5

i've created simple scrapy spider test connecting local polipo proxy can send requests out via tor. basic code of spider follows:

from scrapy.spider import basespider  class torspider(basespider):     name = "tor"     allowed_domains = ["check.torproject.org"]     start_urls = [         "https://check.torproject.org"     ]      def parse(self, response):         print response.body

for proxy middleware, i've defined:

class proxymiddleware(object):     def process_request(self, request, spider):         request.meta['proxy'] = settings.get('http_proxy')

my http_proxy in settings file defined http_proxy = 'http://localhost:8123'.

now, if change start url http://check.torproject.org, works fine, no problems.

if attempt run against https://check.torproject.org, 400 bad request error every time (i've tried different https:// sites, , of them have same problem):

2013-07-23 21:36:18+0100 [scrapy] info: scrapy 0.16.5 started (bot: arachnid) 2013-07-23 21:36:18+0100 [scrapy] debug: enabled extensions: logstats, telnetconsole, closespider, webservice, corestats, spiderstate 2013-07-23 21:36:18+0100 [scrapy] debug: enabled downloader middlewares: httpauthmiddleware, downloadtimeoutmiddleware, randomuseragentmiddleware, proxymiddleware, retrymiddleware, defaultheadersmiddleware, redirectmiddleware, cookiesmiddleware, httpcompressionmiddleware, chunkedtransfermiddleware, downloaderstats 2013-07-23 21:36:18+0100 [scrapy] debug: enabled spider middlewares: httperrormiddleware, offsitemiddleware, referermiddleware, urllengthmiddleware, depthmiddleware 2013-07-23 21:36:18+0100 [scrapy] debug: enabled item pipelines:  2013-07-23 21:36:18+0100 [tor] info: spider opened 2013-07-23 21:36:18+0100 [tor] info: crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2013-07-23 21:36:18+0100 [scrapy] debug: telnet console listening on 0.0.0.0:6023 2013-07-23 21:36:18+0100 [scrapy] debug: web service listening on 0.0.0.0:6080 2013-07-23 21:36:18+0100 [tor] debug: retrying <get https://check.torproject.org> (failed 1 times): 400 bad request 2013-07-23 21:36:18+0100 [tor] debug: retrying <get https://check.torproject.org> (failed 2 times): 400 bad request 2013-07-23 21:36:18+0100 [tor] debug: gave retrying <get https://check.torproject.org> (failed 3 times): 400 bad request 2013-07-23 21:36:18+0100 [tor] debug: crawled (400) <get https://check.torproject.org> (referer: none) 2013-07-23 21:36:18+0100 [tor] info: closing spider (finished)

and double check isn't wrong tor/polipo set up, i'm able run following curl command in terminal, , connect fine: curl --proxy localhost:8123 https://check.torproject.org/

any suggestions what's wrong here?

not sure if these may you:


Comments

Post a Comment

Popular posts from this blog

javascript - DIV "hiding" when changing dropdown value -

-
this related previous question asked: hide / show multiple divs i have code in place previous question , seems work ok apart when change value in dropdown "after" ticket selection made. i have number of javascrpts in place wondering if there clash somewhere? first bit of code in head of document. <head> <script type="text/javascript"> $(function() { $('.cat_dropdown').change(function() { $('#paymethod').toggle($(this).val() >= 2); }); }); </script> <script type="text/javascript"> $(document).ready(function () { $(".paymentmethod").click(function () { $(".paymentinfo").hide(); switch ($(this).val()) { case "credit card authorisation": $("#pay0").show("slow"); break; ...
Read more

Does Firefox offer AppleScript support to get URL of windows? -

-
i trying write applescript that, among other things, gets url of every open webpage in firefox. in safari (and chrome), done simply: tell application "safari" return url of every tab in every window however, seems me firefox offers no real applescript support, such getting url of tab or window. when google terms "firefox" , "applescript" together, firefox bug requests asking applescript support restored, last updated in 2010 or 2011 (like this , this ). am right in thinking, then, firefox no longer offers proper applescript support? realize there semi-workarounds, such simulating key-commands in applescript, aren't practical purposes. to see of applescript commands firefox responds to, launch applescript editor, select menu file > open dictionary... , , choose firefox application. you'll find you're expecting: firefox doesn't offer useful applescript commands.
Read more

android - How to install packaged app on Firefox for mobile? -

-
how test-install packaged (zip) app on fennec? device: physical android phone or android emulator, don't care. install mozilla-apk-cli using npm: npm install -g mozilla-apk-cli use generate "debuggable" apk app either source directory or url mini-manifest: mozilla-apk-cli /path/to/source/dir/ arbitrary-name.apk mozilla-apk-cli http://example.com/path/to/mini/manifest.webapp arbitrary-name.apk (context-click > inspect element on "free" button in marketplace discover mini-manifest url app in marketplace.) install apk on android device: adb install -r arbitrary-name.apk launch app on device. notification area notification port remote debugger server listening on. forward port on desktop, f.e. if port 12345: adb forward tcp:12345 tcp:12345 go web developer > connect… in firefox on desktop , connect localhost @ forwarded port. commence debugging! notes: use nightly builds of fennec best experience. bug 929382 tracks ...
Read more