2

This is a followup to this question, as I originally was uncertain as to how to make this question into a suitable format for this site. So, I'm a user of pcmpl-args.el, which is an Emacs package that allows for more sophisticated completion of shell commands in environments like M-x shell, M-x eshell, M-!, etc. It achieves this by creating "argspecs" by parsing commands' manpages or the output of <command> --help invocations. However, after having upgraded to Emacs 25.1.1, I've noticed that pcmpl-args is no longer completing commands that are completed via manpage parsing. The really weird thing about it is that it does complete commands which are completed via <command> --help parsing, and both of these types of parsing are handled by the same function. I can only conclude that the upgrade must have introduced some sort of compatibility issue since I've re-compiled my whole elpa directory and this package hasn't been developed since 2012, so it's not far-fetched for it to be incompatible with 25.1.1.

So, the specific error that I get when I try to complete a command, such as rm (which uses manpage parsing to create its argspecs), is No completions of - in the minibuffer (since I tried to complete rm options via tab completion of $ rm -). When I take a look at pcomplete/rm, I see that it's an alias to pcmpl-args-pcomplete-on-man. pcmpl-args-pcomplete-on-man is mainly just a wrapper to the pcmpl-args-extract-argspecs-from-manpage function, which (as the name suggests) extracts argspecs from manpages:

(defun pcmpl-args-extract-argspecs-from-manpage (name &rest args)
  "Return a list of argspecs by parsing the manpage identified by NAME.
ARGS are passed to `pcmpl-args-parse-help-buffer'."
  (ignore-errors (kill-buffer " *pcmpl-args-output*"))
  (with-current-buffer (get-buffer-create " *pcmpl-args-output*")
    (erase-buffer)
    (funcall pcmpl-args-man-function name)
    (goto-char (point-min))
    (apply 'pcmpl-args-extract-argspecs-from-buffer args)))

As you can see, it uses two principal functions to do this. The first is pcmpl-args-default-man-function, which invokes man to output manpages in a particular way:

(defun pcmpl-args-default-man-function (name)
  (let ((process-environment process-environment))
    ;; Setting MANWIDTH to a high number makes most paragraphs fit on a single
    ;; line, reducing the number of false positives that result from lines
    ;; starting with `-' that aren't really options.
    (push "MANWIDTH=10000" process-environment)
    (pcmpl-args-process-file "man" "--nj" "--nh" "--" name)))

Note for OS X users: pcmpl-args is expecting to deal with man-db, which is a man library that is commonly used on Linux. So, you'll need to compile man-db by hand in order to get it working on OS X since I haven't found it in package managers such as brew.

$ ./configure CC="gcc -arch x86_64" && sudo make CFLAGS="-Wl,-flat_namespace,-undefined,suppress" && sudo make install

I had to use the above command to get man-db as well as one of its dependencies, libpipeline, to compile. Also note that the usage of sudo there can cause you to not be able to run man without using sudo because it writes all of man-db's files as root, so you'll need to fix the permissions like so:

$ find /usr/local -user root -exec sudo chown <your_username> {} \;

Anyway, the other function that pcmpl-args-extract-argspecs-from-manpage uses, pcmpl-args-extract-argspecs-from-buffer, is a wrapper to the real culprit here: pcmpl-args-parse-help-buffer. This function performs the parsing of manpages and <command> --help output in a special buffer called *pcmpl-args-output*. Unfortunately, whenever this function is called with *pcmpl-args-output* as the current buffer, it returns nil for manpages. This is incorrect because it's supposed to return a list of cons cells / argspecs of the form (OPTION . DESCRIPTION).

Here is the source for this rather monolithic and somewhat convoluted function:

(defun pcmpl-args-parse-help-buffer (&rest keyword-args)
  "Return a list of options found in the current buffer.
The current buffer should contain text describing option usage,
such as the output of command called with the `--help' option.

Due to the variations in formatting, this function tries to
recognize and handle many different styles.  The best handled
style is the GNU long-option style.

The following keyword arguments are recognized:

:filters
    Function or a list of functions to call before parsing.
    May be used to modify the buffer contents.

:start-regexp
    Regular expression indicating where to start parsing.

:end-regexp
    Regular expression indicating where to stop parsing.

If the variable `pcmpl-args-debug-parse-help' is non-nil, the
matched options will be highlighted.

Returns a list of cons cells of the form:

    \(OPTION . DESCRIPTION)
"
  (when pcmpl-args-debug-parse-help
    (dolist (ov (append (car (overlay-lists))
                        (cdr (overlay-lists))))
      (delete-overlay ov)))
  (goto-char (point-min))
  (when (plist-get keyword-args :filters)
    (dolist (f (or (and (functionp (plist-get keyword-args :filters))
                        (list (plist-get keyword-args :filters)))
                   (plist-get keyword-args :filters)))
      (goto-char (point-min))
      (funcall f)))
  (goto-char (point-min))
  (save-restriction
    (when (or (plist-get keyword-args :start-regexp)
              (plist-get keyword-args :end-regexp))
      (when (plist-get keyword-args :start-regexp)
        (re-search-forward (plist-get keyword-args :start-regexp) nil t))
      (narrow-to-region
       (point)
       (or (save-excursion
             (and (plist-get keyword-args :end-regexp)
                  (re-search-forward (plist-get keyword-args :end-regexp) nil t)
                  (match-beginning 0)))
           (point-max))))
    (let* ((start-time (float-time))
           (rgx (concat "^[ ]\\{1,60\\}"
                        "\\("
                        (concat "\\(?:\\(?:\\(?:"
                                "-+[^- \n][^ \n]*"         ;option
                                "\\(?: [^- \n][^ \n]*\\)?" ;optarg
                                "\\)\\)"                   ;
                                "\\(?:[ ]\\{,60\\},\\|[ ]\\{1,60\\}or \\)[ ]\\{0,60\\}" ;delimiter
                                "\\)*")
                        "\\(?:"
                        "\\(?:[ ]\\{,60\\}-+[^- \n][^ \n]*\\)+" ;option
                        ;; "\\>"
                        (concat "\\("
                                ;; optarg followed by description or newline
                                "\\(?: [^- \n][^ \n]*\\($\\|[ ][ ]+\\)\\)"
                                "\\|"
                                "\\(?: <[^ \n]*>\\)+" ;<optarg>...
                                "\\|"
                                "\\(?: \\[[^ \n]*\\]\\)+" ;[optarg]...
                                "\\|"
                                "\\(?: [A-Z][-:_@A-Z0-9]+\\)" ;OPTARG
                                "\\|"
                                "\\(?: [a-zA-Z]=[^ \n]*\\)" ;optarg=value
                                "\\)")
                        "?"
                        "\\)+\\)"))
           opts
           opt doc
           opt-beg-pos opt-end-pos
           doc-beg-pos doc-end-pos
           doc-column)
      (while (let (case-fold-search)
               (re-search-forward rgx nil t))
        (skip-chars-forward " ")
        (setq opt-beg-pos (match-beginning 1)
              opt-end-pos (match-end 0)
              doc-beg-pos (point)
              doc-end-pos (line-end-position)
              doc-column (save-excursion
                           (goto-char doc-beg-pos)
                           (current-column))
              opt (replace-regexp-in-string
                   " or " ", "
                   (replace-regexp-in-string
                    "[ \t\f\v\r\n]+" " "
                    (buffer-substring-no-properties
                     opt-beg-pos opt-end-pos)))
              doc nil)
        ;; Look for a description which may span multiple lines.
        (if (eolp)
            ;; Description may start on the next line.
            (save-excursion
              (forward-line)
              (skip-chars-forward " ")
              (when (or (eolp) (bolp))
                (forward-line)
                (skip-chars-forward " "))
              (if (or (eolp) (bolp)
                      (<= (current-column)
                          (save-excursion
                            (goto-char opt-beg-pos)
                            (current-column))))
                  (setq doc-column nil)
                (setq doc-beg-pos (point)
                      doc-column (current-column))))
          ;; The description starts on the same line so be more discerning
          ;; about parsing the next line if it looks like an option.
          (save-excursion
            (forward-line)
            (skip-chars-forward " ")
            (if (or (eolp)
                    (bolp)
                    (and (looking-at "-")
                         (< (current-column) doc-column)))
                (setq doc-column nil)
              (setq doc-column (min doc-column (current-column))))))
        (goto-char doc-beg-pos)
        ;; Parse indented text.
        (while (and doc-column
                    (not (eobp)))
          (setq doc-end-pos (line-end-position))
          (forward-line)
          (skip-chars-forward " ")
          (when (or (eolp)
                    (< (current-column) doc-column))
            (setq doc-column nil)))
        (goto-char doc-end-pos)
        ;; (save-excursion
        ;;   (goto-char doc-beg-pos)
        ;;   (setq doc-end-pos
        ;;         (min (+ (point) 300)
        ;;              (or (and (re-search-forward
        ;;                        "\\=\\(.\\|\n\\)+?\\.\\([ ][ ]\\|[ ]*$\\)"
        ;;                        doc-end-pos t)
        ;;                       (match-beginning 2))
        ;;                  doc-end-pos))))
        (setq doc (replace-regexp-in-string
                   " *\n *" " "
                   (pcmpl-args-strip
                    (buffer-substring-no-properties doc-beg-pos doc-end-pos))))
        (when pcmpl-args-debug-parse-help
          (pcmpl-args-debug "Found option: %S  %S" opt doc)
          (let ((ov (make-overlay opt-beg-pos opt-end-pos)))
            (overlay-put ov 'face '(:background "light green")))
          (let ((ov (make-overlay doc-beg-pos doc-end-pos)))
            (overlay-put ov 'face '(:background "pink"))))
        (push (cons opt doc) opts))
      (setq opts (nreverse opts))
      ;; We assume that options without descriptions are probably aliases
      ;; so we assign them the subsequent option's description.
      (let ((lst opts) el)
        (while lst
          (setq el (pop lst))
          (when (and lst (string= "" (cdr el)))
            (setcdr el (dolist (p lst)
                         (when (not (string= "" (cdr p)))
                           (return (cdr p))))))))
      (when pcmpl-args-debug
        (pcmpl-args-debug "Found %s options in %f seconds"
                          (length opts) (- (float-time) start-time)))
      opts)))

Does anyone see what could be causing the issue here with compatibility? For reference, here's the (commentated) changelog for Emacs 25.1.1. You can also, of course, view the changelog via C-h n in Emacs 25.1.1.

EDIT 1:

I forgot to mention that pcmpl-args malfunctions differently depending on whether it's in M-x eshell or M-x shell. In Eshell, it outputs No completions of - for something like rm; in M-x shell, it outputs a list of filenames containing -'s or whatever you tried to complete. In either case, this is obviously incorrect as it's supposed to output a list of options to that command.

EDIT 2:

According to a discussion on reddit that I had with someone, this issue appears to be platform-dependent. The person in question is unable to reproduce my issue, however, they are running on Ubuntu 16.04 while I'm running on OS X 10.11.6. Check out the discussion thread for more details.

EDIT 3:

Upon further inspection, I've found that the output of man is different in different Emacs versions on my platform (OS X 10.11.6). I took the output of *pcmpl-args-output* in both versions and diff'd them. I found that the newer version of Emacs is using some special bytes for hyphens and other things to display the output of manpages, which makes a lot of sense now in terms of why that's breaking pcmpl-args. The regexps in pcmpl-args-parse-help-buffer are failing in the new version because there are now some weird bytes in the output of man that are causing the function to not recognize any options (because it's not seeing any normal hyphens). I'm going to look into this further since it's obviously specific to OS X and I'm guessing that there must be some way to disable this special output that is breaking pcmpl-args-parse-help-buffer.

1 Answer 1

2

Well, the solution to this problem was a very simple one. The issue was that the encoding used for putting manpages (and maybe other things) into buffers in 25.1.1 was unsuitable for pcmpl-args because it used special characters for quotation marks, hyphens, etc. These special characters are used by default in man-db, but older versions of Emacs forced buffers which contained manpages to not use said special characters and hence everything was in the standard (ASCII) format. However, 25.1.1 has (for whatever reason) allowed said special characters back into buffers which contain manpages, which broke the regexps that the pcmpl-args parser was using to detect options for argspecs. This, obviously, caused completion to break anywhere that was using man for extracting argspecs, while completion functions using <command> --help invocations did not suffer the same issue because I guess most commands make sure that they don't put any weird characters in their --help output.

With man-db, one can force the encoding via the --encoding=<encoding> option. In my case, I need it to be ASCII so that all characters are standard and can be consumed by the pcmpl-args parser with no trouble. So, all that I had to do to fix this issue was put the following in one of my initialization files:

;; ASCII encoding needs to be enforced for pcmpl-args to work properly
;; with newer versions of Emacs.
(setq pcmpl-args-man-function (lambda (name)
                                (let ((process-environment process-environment))
                                  ;; Setting MANWIDTH to a high number makes most paragraphs fit on a single
                                  ;; line, reducing the number of false positives that result from lines
                                  ;; starting with `-' that aren't really options.
                                  (push "MANWIDTH=10000" process-environment)
                                  (pcmpl-args-process-file "man" "--nj" "--nh" "--encoding=ascii" "--" name))))))

All that this does is set the pcmpl-args-man-function variable to a function which handles the invocation of man. This function is almost identical to pcmpl-args-default-man-function, but with the slight modification of forcing the encoding of the man output to ASCII with "--encoding=ascii" in the argument list. Now, commands such as rm et al. work again through tab completion and I don't have to revert to an older version.

EDIT:

git sub-command option completion is still broken if one uses only the above code. I didn't realize this at the time of writing that, but completion for git uses a special way to get argspecs from manpages by invoking git help --man -- <command>. Since I've not found a way to pass arguments to this special type of man invocation, I've found it easy enough to redefine pcmpl-args-git-extract-argspecs-from-help like so:

;; `pcmpl-args-git-extract-argspecs-from-help' normally uses `git help --man -- <command>'
;; to get manpages for git, but you can't pass arguments to `man' in this way,
;; which means that ASCII encoding can't be enforced (see above). So, we're going
;; to get argspecs for git via `pcmpl-args-extract-argspecs-from-manpage', which
;; will use `man <args> git-<command>' to get the manpage.
(defun pcmpl-args-git-extract-argspecs-from-help (cmd)
  (pcmpl-args-extract-argspecs-from-manpage (if (or (null cmd) (string= "" cmd))
                                                ""
                                              (concat "git-" cmd))))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.