Org mode - Parsing rich HTML directly when pasting?

Question

Currently, for notetaking tools like Evernote and Quiver, I can directly copy HTML content from my favorite browser and then paste them into the app, with all the formatting + link preserved. However in orgmode it seems that all the formatting info is lost.

I've seen somebody suggest using eww to browse the web and copy the content via eww-org. However that is really tedious(I don't think there would be a lot of people browsing the web using eww instead of modern browsers nowadays. I'll have to open that link again in eww and do the copying, not to mention sometimes eww doesn't render the contents nicely).

Is it possible to let Emacs directly parse the copied HTML when pasting? Even if there's no existing tool for that yet, is it feasible to make one?

This is almost the only thing that stops me from switching to orgmode from other notetaking tools.

Answer 1

is it feasible to make one?

Since this is emacs, yes.

My approach is to use a 3rd party tools that can take HTML and convert to plain text or even directly to Org format. I think this is an ugly hack, and there may be better ways to do this, but it looks like it works for my test cases.

(defun kdm/html2org-clipboard ()
  "Convert clipboard contents from HTML to Org and then paste (yank)."
  (interactive)
  (kill-new (shell-command-to-string "osascript -e 'the clipboard as \"HTML\"' | perl -ne 'print chr foreach unpack(\"C*\",pack(\"H*\",substr($_,11,-3)))' | pandoc -f html -t json | pandoc -f json -t org | sed 's/ / /g'"))
  (yank))

Unfortunately, HTML is incredibly complex now - no longer some simple hand-written tags. This complex HTML tagging requires the complicated shell command above. It does the following:

1. osascript gets the HTML text from the clipboard. It is hex encoded, so
2. perl converts the hex to a string
3. We could convert that HTML to Org directly with pandoc, but the HTML is full of complicated tags and therefore produces a ton of Org code. In order to simply the HTML to the minimal set of tags needed to capture the formatting, I
4. Convert the HTML to json, and then
5. Convert the json to Org (these two steps simplify the HTML).
6. Replace non-standard spaces with standard ones.

Note that osascript is for MacOS. To modify steps 1-2 for Linux, replace the argument of shell-command-to-string with

"xclip -o -t text/html | pandoc -f html -t json | pandoc -f json -t org"

In any case, the output of the pandoc command is returned to emacs, and inserted into the buffer.

Bind the new Emacs command to a key similar to "paste" but that means "paste-and-convert-from-html" to you, and it should work.

Alternatively, if you don't want to think about which paste command to use, here is a Linux version that will convert HTML when that is available on the clipboard and will otherwise fall back to plain text:

"xclip -o -t TARGETS | grep -q text/html && (xclip -o -t text/html | pandoc -f html -t json | pandoc -f json -t org) || xclip -o"

Answer 2

I wrote an add-on Copy as Org-Mode for Firefox which can do this in browser directly (copying the rich HTML directly, instead of copying the raw HTML code), it even can handle HTML tables into Org-mode format.

Answer 3

Short answer:
consider using org-web-tools-read-url-as-org.
Long answer:
If you just want to parse html, you can use dom module for example:

(let* ((dom (with-temp-buffer
                (insert html)
                (libxml-parse-html-region (point-min) (point-max))))
         (title (cl-caddr (car (dom-by-tag dom 'title)))))

If you want to automatically render html into something more or less readable, you can use eww, w3m and other text browsers. Personally I prefer to convert web documents into org mode. I know several tools of converting html to org mode: already mentioned pandoc, org-web-tools and html2org.
My favorate tool is org-web-tools, because pandoc and html2org are converting the hole document. org-web-tools conversly detect potential poorly-rendered parts and removes it. Below is short description of how it is done:
org-web-tools-read-url-as-org function loads html from specified url, converts it into org-mode document and then loads it in selected window. The main advantage of this approach is that its code org-web-tools--url-as-readable-org -> org-web-tools--eww-readable -> eww-score-readability calculates score for document parts and removes parts that are probably not worth viewing (adds, poorly rendered menu).
By the way html2org is extremely simple tool, which probably can be easily hacked to render specific parts of html page.

Answer 4

Assuming macOS, to get the clipboard as HTML, install:

git clone https://github.com/chbrown/macos-pasteboard
cd macos-pasteboard
make install

Then you can use the function

function pbpaste-html() {
    command pbv public.html public.utf8-plain-text
}

Together with pandoc:

        input="$(gmktemp)"
        pbpaste-plus > "$input"

        pandoc --wrap=none --from "html" --to "org" "$input" -o "-" | pbcopy

Then paste the result in emacs.