3

I am trying to implement a web scraper that does the following:

  1. Make a request to https://m.weather.naver.com/,
  2. Get interesting pieces of information (regarding today's weather), then
  3. Organize them nicely into a new org-buffer.

So far, I have this piece of code (credit: http://teaching.sociology.ul.ie/bhalpin/wordpress/?p=580):

;;; scrape_weather.el ---  -*- lexical-binding: t; -*-

(defun get-weather-page ()
  "get the title text of Naver Weather"
  (set-buffer (url-retrieve-synchronously "https://m.weather.naver.com")) ; set this buffer to current buffer
  (re-search-forward "<title>\\(.+\\)</title>")                           ; re-search for pattern
  (match-string 1))                                                       ; return the first parens in regexp


(defvar *str* (get-weather-page))

;; *str* evals to "\263\327\300\314\271\366 \263\257\276\276".. is there any way I can encode it back to Korean?

;;; scrape_weather.el ends here

For debugging purposes, I evaluated the variable *str* and got weird sequences of hex codes. That must be 네이버 날씨, but I'm sure that the decoding of Korean characters got broken after the execution of re-search-forward.

Is there any way I can correct the decoding?

1 Answer 1

5

In order to handle non-ASCII characters, you need turn on multibyte via (set-buffer-multibyte t) (see also (elisp) Selecting a Representation), e.g.,

(with-current-buffer (url-retrieve-synchronously
                      "https://m.weather.naver.com")
  (set-buffer-multibyte t)
  (goto-char (point-min))
  (re-search-forward "<title>\\(.+\\)</title>")
  (match-string 1))
;; => "네이버 날씨"

Emacs uses multibyte mode by default, e.g.,

(with-temp-buffer
  enable-multibyte-characters)
;; => t

however url-retrieve-synchronously turns it off, hence you need to turn it on according to you needs.

For debugging purposes

here is another way to debug which is more intuitive, eval this code

(display-buffer
 (url-retrieve-synchronously
  "https://m.weather.naver.com"))

you will see some characters in octal format, and after M-x toggle-enable-multibyte-characters, you'll see the actual characters.

1
  • I ended up selecting string-as-multibyte for my use case. Thanks for your guidance. Dec 8, 2020 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.