在使用同样的技术之前,我一直在抓网站.但是这个网站似乎不起作用.
import urllib2
from BeautifulSoup import BeautifulSoup
url = "http://www.weatheronline.co.uk/weather/maps/current?LANG=en&DATE=1354104000&CONT=euro&LAND=UK&KEY=UK&SORT=1&INT=06&TYP=sonne&ART=tabelle&RUBRIK=akt&R=310&CEL=C"
page=urllib2.urlopen(url).read()
soup = BeautifulSoup(page)
print soup
GIF89a(它也跟着我在这里不能复制的一些符号)
关于问题是什么以及我应该如何进行的任何想法.
最佳答案
but I want to know why I am getting a gif accesing the url like that
and when I access it via my browser I get the website perfectly.
因为这些人很聪明,不希望他们的网站在网络浏览器外访问.您需要做的是通过向标头添加User-agent来伪造已知的浏览器.这是一个可行的修改示例
>>> import urllib2
>>> opener = urllib2.build_opener()
>>> opener.addheaders = [('User-agent','Mozilla/5.0')]
>>> url = "http://www.weatheronline.co.uk/weather/maps/current?LANG=en&DATE=1354104000&CONT=euro&LAND=UK&KEY=UK&SORT=1&INT=06&TYP=sonne&ART=tabelle&RUBRIK=akt&R=310&CEL=C"
>>> response = opener.open(url)
>>> page = response.read()
>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup(page)