Why isn't url-http-end-of-headers set correctly when using url-automatic-caching?

Question

This is a follow-up question to How can I make url.el work with the If-None-Match header?

Evaluate the following form:

(let ((url-request-method "GET")
      (url-automatic-caching t))
  (with-current-buffer
      (url-retrieve-synchronously
       "https://api.github.com/users/octocat")
    (message "'%c' (%S)"
             (char-after url-http-end-of-headers)
             url-http-end-of-headers)))

It thinks url-http-end-of-headers is position 1 in the following buffer:

HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 19 May 2017 12:18:32 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1495197406
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: W/"d093767a41c43cbc64dcc3bda1b5880f"
Last-Modified: Wed, 17 May 2017 21:18:07 GMT
X-GitHub-Media-Type: github.v3; format=json
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'none'
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Vary: Accept-Encoding
X-Served-By: 075bb2f6b7031ca3c0e69edb17939fae
Content-Encoding: gzip
X-GitHub-Request-Id: E86D:15875:2064CAA:268A8E5:591EE291

{"login":"octocat","id":583231,...}

Answer 1

This seems to be a bug in emacs at url-cache-extract with its use of erase-buffer and wiping out that marker. More details when I'm not mobile. I'll be submitting a bug report :-)

---

This seems to be a bug in either url-http-parse-headers or url-cache-extract. Before url-http-parse-headers calls url-cache-extract, url-http-end-of-headers is set correctly. However, when url-cache-extract calls erase-buffer to replace the response buffer (which has header HTTP-304), url-http-end-of-headers is blown away (rightly so), but not reset.

Redefining url-cache-extract to find the end of the headers and set the marker appropriately should fix the immediate issue, but I'll be reporting this bug to Emacs' bug tracker since I'm pretty sure this isn't all that needs to happen in this case.