why would curl and wget result in a 403 forbidden?
I try to download a file with wget and curl and it is rejected with a 403 error (forbidden).
I can view the file using the web browser on the same machine.
I try again with my browser’s user agent, obtained by http://www.whatsmyuseragent.com. I do this:
but it is still forbidden. What other reasons might there be for the 403, and what ways can I alter the wget and curl commands to overcome them?
(this is not about being able to get the file — I know I can just save it from my browser; it’s about understanding why the command-line tools work differently)
update
Thanks to all the excellent answers given to this question. The specific problem I had encountered was that the server was checking the referrer. By adding this to the command-line I could get the file using curl and wget .
The server that checked the referrer bounced through a 302 to another location that performed no checks at all, so a curl or wget of that site worked cleanly.
If anyone is interested, this came about because I was reading this page to learn about embedded CSS and was trying to look at the site’s css for an example. The actual URL I was getting trouble with was this and the curl I ended up with is
and the wget is
5 Answers 5
A HTTP request may contain more headers that are not set by curl or wget. For example:
- Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val , you can set it with the -b key=val (or —cookie key=val ) option for curl .
- Referer (sic): when clicking a link on a web page, most browsers tend to send the current page as referrer. It should not be relied on, but even eBay failed to reset a password when this header was absent. So yes, it may happen. The curl option for this is -e URL and —referer URL .
- Authorization: this is becoming less popular now due to the uncontrollable UI of the username/password dialog, but it is still possible. It can be set in curl with the -u user:password (or —user user:password ) option.
- User-Agent: some requests will yield different responses depending on the User Agent. This can be used in a good way (providing the real download rather than a list of mirrors) or in a bad way (reject user agents which do not start with Mozilla , or contain Wget or curl ).
You can normally use the Developer tools of your browser (Firefox and Chrome support this) to read the headers sent by your browser. If the connection is not encrypted (that is, not using HTTPS), then you can also use a packet sniffer such as Wireshark for this purpose.
Besides these headers, websites may also trigger some actions behind the scenes that change state. For example, when opening a page, it is possible that a request is performed on the background to prepare the download link. Or a redirect happens on the page. These actions typically make use of Javascript, but there may also be a hidden frame to facilitate these actions.
If you are looking for a method to easily fetch files from a download site, have a look at plowdown, included with plowshare.
wget failed connection refused- Causes and fixes!
Ever wondered why we get a wget failed connection refused error?
This is a common error while downloading files from servers. It mainly occurs when no process listens to the port correctly, or port blocks in the firewall and so on.
At Bobcares, we often receive requests to solve the wget connection refused error as part of our Server Management Services.
Today, let’s discuss the various reasons behind this error and see how our Support Engineers fix it for customers.
What causes wget failed connection refused error?
Wget comes handy to fetch files from remote servers. For instance, when installing a new program, we often need to fetch the installer package first using wget.
Many times, this can result in the connection refused error.
Here, wget fails on port 7183 of the server. Finding the ultimate cause of this error can be quite tricky.
So now, let’s discuss some major reasons that cause the wget connection refused error and their respective fixes.
1. No process is listening
Usually, this is the main reason for the connection refused message. For wget to work, the webserver must be listening correctly on port 80.
Recently, one of our customers approached us with a connection refused error in wget. Our Support Engineers checked and found that he was connecting to the correct server.
However, Apache was failing on the server. Thus no service was listening on port 80 of the server.
Initially, we started checking whether the httpd process was listening on the port correctly. We use the command,
No output means nothing listens to the given port.
So, we restarted the Apache server to make the service listen to the port correctly.
2. Listen to address in Webserver
Misconfigured Listen address in webserver can also create problems with wget.
For instance, when running Apache, we always make sure that the configuration file holds the correct “BindAddress” or “Listen” directive. If we specific IP(s) listed in the conf, then Apache will only listen to those IP addresses. Thus wget works only on those IPs.
3. Firewall errors
Sometimes, the firewall blocks certain ports. Blocking the port in the firewall can also cause wget failed connection refused error.
For such cases, the firewall responds with icmp-port-unreachable.
Usually, we use utilities like tcpdump to get the details of the connection refused message.
This gives us the exact point at which the block appears. Then, we remove the port block from the respective server firewall. In other words, we open ports 22 and 80. Or if SSH runs on any custom port, we open it in the server firewall.
Therefore, this resolves the error and make wget working again.
[Need more assistance to solve wget failed connection refused?- We’ll help you.]
Conclusion
In short, the wget failed connection refused error occurs when no process listens to the port, firewall errors and so on. In today’s article, we discussed the main causes of this error and saw how our Support Engineers fix this error for our customers.
PREVENT YOUR SERVER FROM CRASHING!
Never again lose customers to poor server speed! Let us help you.
Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.
Error wget failed to download
Сообщения: 221
Благодарности: 8
| Конфигурация компьютера |
| Прочее: Apple iMac |
Это сообщение посчитали полезным следующие участники:
| Конфигурация компьютера |
| Прочее: Apple iMac |
