Go to body after url-retrieve-synchronously

Question

I use url-retrieve-synchronously to download a JSON file from a remote server. This function returns a buffer which contains the entire content of the response, including HTTP headers. However, I'm just interested in the body.

Now, short of parsing a HTTP response manually, how can I jump to the beginning of the body? I have found the variable url-http-end-of-headers, which works, but it's not documented at all, and using it feels “dirty”. Is there a proper API for this purpose?

Answer 1

With respect to a "proper API", I think the short answer, at present, is "no" -- unless you are willing to rely on an external library. url-http-end-of-headers seems to be as good as it gets, although it isn't mentioned in the URL Programmer's Manual nor does it appear to be documented in emacs source.

Answer 2

A higher-level alternative to using url-http-end-of-headers, but equally undocumented, is url-insert-file-contents:

(with-temp-buffer
  (url-insert-file-contents
   "https://api.stackexchange.com/2.2/questions/12464?site=emacs")
  (json-parse-buffer :object-type 'alist))

or equivalently, prior to Emacs 27 compiled with native JSON support:

(require 'json)
(with-temp-buffer
  (url-insert-file-contents
   "https://api.stackexchange.com/2.2/questions/12464?site=emacs")
  (let ((json-false :false))
    (json-read)))

These result in:

((items .
  [((tags .
     ["url" "http"])
    (owner
     (reputation . 10741)
     (user_id . 227)
     (user_type . "registered")
     (accept_rate . 89)
     (profile_image . "https://i.stack.imgur.com/ebO5J.jpg?s=128&g=1")
     (display_name . "lunaryorn")
     (link . "https://emacs.stackexchange.com/users/227/lunaryorn"))
    (is_answered . t)
    (view_count . 867)
    (accepted_answer_id . 29798)
    (answer_count . 3)
    (score . 4)
    (last_activity_date . 1517363132)
    (creation_date . 1431861037)
    (question_id . 12464)
    (link . "https://emacs.stackexchange.com/questions/12464/go-to-body-after-url-retrieve-synchronously")
    (title . "Go to body after url-retrieve-synchronously"))])
 (has_more . :false)
 (quota_max . 300)
 (quota_remaining . 276))

The function url-insert-file-contents wraps the better-documented functions url-insert-buffer-contents and url-insert and thus comes with the following goodies:

1. Autoloaded by default.
2. Exists since at least as far back as Emacs 21.
3. Simple HTTP response error handling via url-http-response-status.
4. Decodes data.

FWIW, it also comes with the guarantee of being used by lisp/emacs-lisp/package.el.

Answer 3

Since there is a newline after the HTTP headers, the only alternative I'm aware of is using a re-search-forward:

(with-current-buffer
    (url-retrieve-synchronously my-url)
  (goto-char (point-min))
  (re-search-forward "^$")
  (delete-region (point) (point-min))
  (buffer-string))