nginx: Enable resumable downloads for release files

[silverwind]

Disabling gzip for the release downloads enables support for the HTTP ranges mechanism, as seen by the advertized header on matching requests:

Accept-Ranges: bytes

For further explaination on why nginx disables ranges on gzip, see here.

[rvagg]

@silverwind this is now active on the server, can you confirm for me that it's working before I merge to master?

thanks so much for this

[jbergstroem]

@rvagg Whatever you did in the server configuration, I recommend reverting. All tarballs now 404, both for 1.0.1 and 1.0.0.

Edit: you forgot to add root inside the location {} scope.

[rvagg]

reverted

[silverwind]

Sorry, haven't been around to test. It worked on my local nginx, but that one has a slightly different gzip setup. Will test further.

[rvagg]

one guess is that I actually manually put this at the top of the location list but the order probably matters

[silverwind]

Yeah, it shouldn't matter. What nginx version are you running, by the way?

[rvagg]

nginx/1.4.6 (Ubuntu)

[rvagg]

tried putting it last and get 404s still

[silverwind]

Anything in /var/log/nginx/iojs.org-error.log ?

[rvagg]

hard to tell with all the noise but this could have been during a turned on period:

2015/01/15 22:23:35 [error] 22024#0: *772283 open() "/usr/share/nginx/html/dist/v1.0.1/iojs-v1.0.1.tar.gz" failed (2: No such file or directory), client: www.xxx.yyy.zzz, server: iojs.org, request: "GET /dist/v1.0.1/iojs-v1.0.1.tar.gz HTTP/1.1", host: "iojs.org:443"

so it's looking in /usr/share/nginx/html/ for dist, so it's overriding the proper location for /dist/ (although why is it showing an index?)

[silverwind]

You're right, I see the problem. The / location is pointing to the correct path, but archives don't have that root. One possible fix could be something like this, but I'll check the docs if there's a more elegant way, so root is only defined once.

location ~* (\.xz|\.gz|\.pkg|\.msi)$ {
    root /home/iojs/www;
    index index.html;
    default_type text/plain;
    gzip off;
}

[jbergstroem]

As mentioned on IRC and slightly above, either add root within the location scope or move root outside of the other location scope to inherit it.

I started doing a new branch with some other improvements (namely removing the www-check since it's executed on every request) - but that's for another day.

[silverwind]

Another possibilty would be to nest the regex location, but moving root outside and into the server scope would get my vote for being cleanest.

[rvagg]

I'm open to changes here, my background is deeply with Apache and it's only the last 6 months or so that I've been really digging more deeply into nginx but I still consider myself a n00b with it

[silverwind]

Updated the PR. I moved the root into server scope and removed the / location because the remaining directives were set to their default values, so not needed any more.

I'm available for some tests in the next hour, but almost certain that it'll work as expected now.

[jbergstroem]

LGTM. Lets add other improvements in other PR's -- but it's really heading into premature optimisation-ville :-)

[silverwind]

@ljharb could you please retest download resumption? While this change isn't live, I think this may actually have never been an issue in first place.

[Fishrock123]

@silverwind

location ~* (\.xz|\.gz|\.pkg|\.msi)$ {
    root /home/iojs/www;
    index index.html;
    default_type text/plain;
    gzip off;
}

If you do that default type I don't think you'll be able to download the files normally.

[silverwind]

@jbergstroem No, I meant the original issue of non-working resumption.

Here's the correct Accept-Ranges, which means the download is resumable:

curl -I --compressed https://iojs.org/download/release/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 16 Jan 2015 05:09:51 GMT
Content-Type: application/x-gzip
Content-Length: 6463647
Last-Modified: Wed, 14 Jan 2015 04:38:07 GMT
Connection: keep-alive
ETag: "54b5f2af-62a09f"
Strict-Transport-Security: max-age=63072000
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Accept-Ranges: bytes

And here's a gzip response, supressing Accept-Ranges:

curl -I --compressed https://iojs.org/download/release/v1.0.1/SHASUMS256.txt
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 16 Jan 2015 05:10:00 GMT
Content-Type: text/plain
Last-Modified: Wed, 14 Jan 2015 05:04:19 GMT
Connection: keep-alive
Strict-Transport-Security: max-age=63072000
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Content-Encoding: gzip

I believe that nodejs/node#424 actually never was an issue.

[silverwind]

@Fishrock123 Already refactored that. It's just an fallback mime, and it's at its default value, so shouldn't have been an issue.

[ljharb]

@silverwind I still get curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume. from curl --progress-bar -L -C - https://iojs.org/dist/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz -o /tmp/iojs.tar.gz - I'm not sure what you mean by "may actually have never been an issue in the first place". It's clearly an issue; it doesn't affect nodejs.org (curl --progress-bar -L -C - http://nodejs.org/dist/v0.10.35/node-v0.10.35-darwin-x64.tar.gz -o /tmp/node.tar.gz multiple times in a row works just fine); and it does affect iojs.org.

[silverwind]

I don't see an issue, the second curl finishes instantly. What's your curl -V?

$ curl --progress-bar -L -C - https://iojs.org/dist/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz -o /tmp/iojs.tar.gz
######################################################################## 100.0%
$ curl --progress-bar -L -C - https://iojs.org/dist/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz -o /tmp/iojs.tar.gz
######################################################################## 100.0%
$ curl -V
curl 7.39.0 (armv6l-unknown-linux-gnueabihf) libcurl/7.39.0 OpenSSL/1.0.1k zlib/1.2.8 libidn/1.29 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API SPNEGO NTLM NTLM_WB SSL libz TLS-SRP

Edit: sorry, ran against nodejs first, but same result on iojs.

[ljharb]

curl 7.37.1 (x86_64-apple-darwin14.0) libcurl/7.37.1 SecureTransport zlib/1.2.5 I presume whatever comes with OS X Yosemite.

[silverwind]

And what do you get on this one?

$ curl -I https://iojs.org/dist/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 16 Jan 2015 05:37:40 GMT
Content-Type: application/x-gzip
Content-Length: 6463647
Last-Modified: Wed, 14 Jan 2015 04:38:07 GMT
Connection: keep-alive
ETag: "54b5f2af-62a09f"
Strict-Transport-Security: max-age=63072000
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Accept-Ranges: bytes

[ljharb]

Looks like exactly the same:

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 16 Jan 2015 05:42:04 GMT
Content-Type: application/x-gzip
Content-Length: 6463647
Last-Modified: Wed, 14 Jan 2015 04:38:07 GMT
Connection: keep-alive
ETag: "54b5f2af-62a09f"
Strict-Transport-Security: max-age=63072000
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Accept-Ranges: bytes

[silverwind]

I'm using the exact same curl version here on OS X and don't get your error, so I'm really baffled. Could there be an proxy interfering on your side? Maybe your curl is aliased?

Still wouldn't explain why nodejs.org works.

[silverwind]

Added some -vvv and I finally saw your 'error':

On OS X (curl 7.37.1):

> GET /dist/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz HTTP/1.1
> Range: bytes=6463868-
> User-Agent: curl/7.37.1
> Host: iojs.org
> Accept: */*
>
< HTTP/1.1 416 Requested Range Not Satisfiable
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Fri, 16 Jan 2015 05:50:35 GMT
< Content-Type: text/html
< Content-Length: 221
< Connection: keep-alive
< Strict-Transport-Security: max-age=63072000
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Range: bytes */6463647
<
* HTTP server doesn't seem to support byte ranges. Cannot resume.
* Closing connection 0

On Linux (curl 7.39.0):

> GET /dist/v1.0.1/iojs-v1.0.1-darwin-x64.tar.gz HTTP/1.1
> Range: bytes=6463647-
> User-Agent: curl/7.39.0
> Host: iojs.org
> Accept: */*
>
< HTTP/1.1 416 Requested Range Not Satisfiable
< Server: nginx/1.4.6 (Ubuntu)
< Date: Fri, 16 Jan 2015 05:51:18 GMT
< Content-Type: text/html
< Content-Length: 221
< Connection: keep-alive
< Strict-Transport-Security: max-age=63072000
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Range: bytes */6463647
* Connection #0 to host iojs.org left intact

As you see, it requests bytes beyond the Content-Length to which the server responds correctly. OS X's version displays an 'error', which really isn't one.

Finally, I CTRL-C mid download, restarted, and it resumed fine on both Linux and OS X, so if there's an issue here, then it's with OS X's outdated curl.

[silverwind]

@ljharb Here's the relevant bug which was fixed in curl 7.39.0.

[ljharb]

Thanks - so you're saying the nodejs.org server is responding incorrectly, which is why the error doesn't appear when installing node?

[silverwind]

@ljharb both servers respond correctly, it's just curl getting confused by the missing Accept-Ranges. By the way, I still see it on nodejs.org (second run there):

curl -vvv --progress-bar -L -C - http://nodejs.org/dist/v0.10.35/node-v0.10.35-darwin-x64.tar.gz -o /tmp/node.tar.gz
* Hostname was NOT found in DNS cache
*   Trying 165.225.133.150...
* Connected to nodejs.org (165.225.133.150) port 80 (#0)
> GET /dist/v0.10.35/node-v0.10.35-darwin-x64.tar.gz HTTP/1.1
> Range: bytes=5119341-
> User-Agent: curl/7.37.1
> Host: nodejs.org
> Accept: */*
>
< HTTP/1.1 416 Requested Range Not Satisfiable
* Server nginx is not blacklisted
< Server: nginx
< Date: Fri, 16 Jan 2015 18:46:06 GMT
< Content-Type: text/html
< Content-Length: 206
< Connection: keep-alive
< Expires: Thu, 31 Dec 2037 23:55:55 GMT
< Cache-Control: max-age=315360000
< Content-Range: bytes */5119135
<
* HTTP server doesn't seem to support byte ranges. Cannot resume.
* Closing connection 0

[ljharb]

Interesting - I'm not using -vvv or any verbose flag on either one, so it's strange that you do need to do that.

Could iojs.org be made to provide Accept-Ranges? (or did i misunderstand, and it's nodejs.org that would need to)

[silverwind]

Both support Accept-Ranges on 2xx responses, which (along with correct Range response) is all that's needed for resumption. There's no reason to support byte ranges when there's no bytes left to transfer, so that's why the servers leave out the header on that 416 response.

Go ahead and cancel a download at 50%, and you will see it work. Really nothing broken here.

[ljharb]

Cool, thanks for the exhaustive explanation. I'll just ignore the "error" output in nvm.sh

[silverwind]

Regarding the -vvv, I actually just discovered something had aliased my curl to curl -s 😩