Download Files From a URL Using Python
Автор: SurvivingMSc
Загружено: 2022-07-19
Просмотров: 21179
In this lesson we are going to do a simple web scraping- downloading files from a given URL using python.
The code I used in this session is this-
-*- coding: utf-8 -*-
"""
Created on Mon Jul 18 11:48:18 2022
@author: karin
"""
import requests
from requests.auth import HTTPBasicAuth
from bs4 import BeautifulSoup
def get_url_paths(url, ext=''):
response = requests.get(url)
response = requests.get(url, auth=HTTPBasicAuth('username', 'password'))
if response.ok:
response_text = response.text
else:
return response.raise_for_status()
soup = BeautifulSoup(response_text, 'html.parser')
parent = [url + node.get('href') for node in soup.find_all('a') if node.get('href').endswith(ext)]
return parent
def main():
url = 'https://www.ncei.noaa.gov/data/total-...
ext = '.nc'
url = 'http://wwlln.net/hostdata//'
ext = '.loc'
result = get_url_paths(url, ext)
for file in result:
f_name = file[-19:-13]
f_name = file[-12:-4]
r = requests.get(file)
r = requests.get(file, auth=HTTPBasicAuth('username', 'password'))
with open(f'C:/Users/karin/Desktop/Work/PikFix/WebScraping/Folder1/{f_name}.nc', 'wb') as f:
f.write(r.content)
if _name_ == '__main__':
main()
Доступные форматы для скачивания:
Скачать видео mp4
-
Информация по загрузке:
