python网络爬虫与信息采取之下载存储数据(一)-----

发布时间:2017-09-09 10:46:44
python网络爬虫与信息采取之下载存储数据(一)-----下载储存媒体文件模板

存储媒体文件有两种方式:一是只获取URL链接;二是直接把源文件下载下来

下面这个就是直接把源文件下载下来的实例:

其中,

urlretrieve()函数用于下载文件

代码如下:

<span style="font-size:18px;">import os from urllib.request import urlretrieve from urllib.request import urlopen from bs4 import BeautifulSoup downloadDirectory = "D:\downloaded" baseUrl = "" def getAbsoluteURL(baseUrl, source): if source.startswith(""): url = "" + source[11:] elif source.startswith(""): url = source elif source.startswith(""): url = source[4:] url = "" + source else: url = baseUrl + "/" + source if baseUrl not in url: return None return url def getDownloadPath(baseUrl, absoluteUrl, downloadDirectory): path = absoluteUrl.replace("", "") path = path.replace(baseUrl, "") path = downloadDirectory + path directory = os.path.dirname(path) if not os.path.exists(directory): os.makedirs(directory) return path html = urlopen("") bsObj = BeautifulSoup(html) downloadList = bsObj.findAll(src=True) for download in downloadList: fileUrl = getAbsoluteURL(baseUrl, download["src"]) if fileUrl is not None: print(fileUrl) urlretrieve(fileUrl, getDownloadPath(baseUrl, fileUrl, downloadDirectory)) </span>

企业建站2800元起,携手武汉肥猫科技,做一个有见地的颜值派!更多优惠请戳:潜江网站建设 http://qianjiang.45qun.com

上一篇:php学习随记4
下一篇:最后一页