Troubleshooting Python requests GET with Proxies: Why HTTP URLs Return Headers Instead of Page Cont

Learn why Python requests using HTTP URLs with proxies may return headers instead of page content and how to fix this by selecting proper proxies.
---
This video is based on the question https://stackoverflow.com/q/79431674/ asked by the user 'Simonhawk' ( https://stackoverflow.com/u/8921861/ ) and on the answer https://stackoverflow.com/a/79431937/ provided by the user 'S Ahmed' ( https://stackoverflow.com/u/7013205/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Python Requests GET with Proxy - HTTPS scheme returns expected result but HTTP returns header

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to drop me a comment under this video.
---
Introduction

When scraping or accessing web pages with Python's requests library using proxies, you might encounter a puzzling issue: HTTPS URLs return the expected page source, but HTTP URLs return only headers or unusual responses without redirects, even though the browser handles redirects automatically. This behavior can disrupt workflows that process both HTTP and HTTPS URLs dynamically.

Understanding the Problem

The root cause often lies in the proxy server itself rather than the Python code or requests library. Specifically:

Public proxies may respond differently to HTTP and HTTPS requests.

Using an HTTPS proxy address for an HTTP request can lead to receiving the proxy's own headers or status page, not the target website's content.

Requests normally handle redirects automatically (allow_redirects=True), but if the proxy doesn't forward requests properly, redirects won't occur.

For example, requesting an HTTP URL through an HTTPS proxy that is not configured to handle HTTP may return proxy-related headers instead of the page source.

Diagnosing the Issue

You can verify the proxy behavior by sending a direct request to the proxy address:

[[See Video to Reveal this Text or Code Snippet]]

If you receive header info or a proxy landing page, it indicates that the proxy is not forwarding your HTTP request as expected.

Solution: Choose a Proper Proxy

Switching to a proxy that supports both HTTP and HTTPS forwarding correctly resolves this. Here's how you can update your proxy configuration:

[[See Video to Reveal this Text or Code Snippet]]

Key notes:

Use proxies matching the URL schema (HTTP URL with HTTP proxy).

Verify that proxies support transparent forwarding and redirection.

Public proxies can be flaky; consider private proxies for reliability.

Summary

If HTTP URLs return only headers or incorrect responses when proxied but HTTPS URLs work fine:

The proxy server may not handle HTTP schemes correctly.

Verify proxy behavior by requesting the proxy address directly.

Switch to a properly functioning proxy supporting HTTP and HTTPS.

Ensure your requests call's proxy matches the URL scheme.

By selecting the right proxies, Python requests will correctly follow redirects and fetch page sources for both HTTP and HTTPS URLs.