This is a followup to this question, as I originally was uncertain as to how to make this question into a suitable format for this site. So, I'm a user of pcmpl-args.el, which is an Emacs package that allows for more sophisticated completion of shell commands in environments like M-x shell, M-x eshell, M-!, etc. It achieves this by creating "argspecs" by parsing commands' manpages or the output of <command> --help invocations. However, after having upgraded to Emacs 25.1.1, I've noticed that pcmpl-args is no longer completing commands that are completed via manpage parsing. The really weird thing about it is that it does complete commands which are completed via <command> --help parsing, and both of these types of parsing are handled by the same function. I can only conclude that the upgrade must have introduced some sort of compatibility issue since I've re-compiled my whole elpa directory and this package hasn't been developed since 2012, so it's not far-fetched for it to be incompatible with 25.1.1.
So, the specific error that I get when I try to complete a command, such as rm (which uses manpage parsing to create its argspecs), is No completions of - in the minibuffer (since I tried to complete rm options via tab completion of $ rm -). When I take a look at pcomplete/rm, I see that it's an alias to pcmpl-args-pcomplete-on-man. pcmpl-args-pcomplete-on-man is mainly just a wrapper to the pcmpl-args-extract-argspecs-from-manpage function, which (as the name suggests) extracts argspecs from manpages:
(defun pcmpl-args-extract-argspecs-from-manpage (name &rest args)
"Return a list of argspecs by parsing the manpage identified by NAME.
ARGS are passed to `pcmpl-args-parse-help-buffer'."
(ignore-errors (kill-buffer " *pcmpl-args-output*"))
(with-current-buffer (get-buffer-create " *pcmpl-args-output*")
(erase-buffer)
(funcall pcmpl-args-man-function name)
(goto-char (point-min))
(apply 'pcmpl-args-extract-argspecs-from-buffer args)))
As you can see, it uses two principal functions to do this. The first is pcmpl-args-default-man-function, which invokes man to output manpages in a particular way:
(defun pcmpl-args-default-man-function (name)
(let ((process-environment process-environment))
;; Setting MANWIDTH to a high number makes most paragraphs fit on a single
;; line, reducing the number of false positives that result from lines
;; starting with `-' that aren't really options.
(push "MANWIDTH=10000" process-environment)
(pcmpl-args-process-file "man" "--nj" "--nh" "--" name)))
Note for OS X users: pcmpl-args is expecting to deal with man-db, which is a man library that is commonly used on Linux. So, you'll need to compile man-db by hand in order to get it working on OS X since I haven't found it in package managers such as brew.
$ ./configure CC="gcc -arch x86_64" && sudo make CFLAGS="-Wl,-flat_namespace,-undefined,suppress" && sudo make install
I had to use the above command to get man-db as well as one of its dependencies, libpipeline, to compile. Also note that the usage of sudo there can cause you to not be able to run man without using sudo because it writes all of man-db's files as root, so you'll need to fix the permissions like so:
$ find /usr/local -user root -exec sudo chown <your_username> {} \;
Anyway, the other function that pcmpl-args-extract-argspecs-from-manpage uses, pcmpl-args-extract-argspecs-from-buffer, is a wrapper to the real culprit here: pcmpl-args-parse-help-buffer. This function performs the parsing of manpages and <command> --help output in a special buffer called *pcmpl-args-output*. Unfortunately, whenever this function is called with *pcmpl-args-output* as the current buffer, it returns nil for manpages. This is incorrect because it's supposed to return a list of cons cells / argspecs of the form (OPTION . DESCRIPTION).
Here is the source for this rather monolithic and somewhat convoluted function:
(defun pcmpl-args-parse-help-buffer (&rest keyword-args)
"Return a list of options found in the current buffer.
The current buffer should contain text describing option usage,
such as the output of command called with the `--help' option.
Due to the variations in formatting, this function tries to
recognize and handle many different styles. The best handled
style is the GNU long-option style.
The following keyword arguments are recognized:
:filters
Function or a list of functions to call before parsing.
May be used to modify the buffer contents.
:start-regexp
Regular expression indicating where to start parsing.
:end-regexp
Regular expression indicating where to stop parsing.
If the variable `pcmpl-args-debug-parse-help' is non-nil, the
matched options will be highlighted.
Returns a list of cons cells of the form:
\(OPTION . DESCRIPTION)
"
(when pcmpl-args-debug-parse-help
(dolist (ov (append (car (overlay-lists))
(cdr (overlay-lists))))
(delete-overlay ov)))
(goto-char (point-min))
(when (plist-get keyword-args :filters)
(dolist (f (or (and (functionp (plist-get keyword-args :filters))
(list (plist-get keyword-args :filters)))
(plist-get keyword-args :filters)))
(goto-char (point-min))
(funcall f)))
(goto-char (point-min))
(save-restriction
(when (or (plist-get keyword-args :start-regexp)
(plist-get keyword-args :end-regexp))
(when (plist-get keyword-args :start-regexp)
(re-search-forward (plist-get keyword-args :start-regexp) nil t))
(narrow-to-region
(point)
(or (save-excursion
(and (plist-get keyword-args :end-regexp)
(re-search-forward (plist-get keyword-args :end-regexp) nil t)
(match-beginning 0)))
(point-max))))
(let* ((start-time (float-time))
(rgx (concat "^[ ]\\{1,60\\}"
"\\("
(concat "\\(?:\\(?:\\(?:"
"-+[^- \n][^ \n]*" ;option
"\\(?: [^- \n][^ \n]*\\)?" ;optarg
"\\)\\)" ;
"\\(?:[ ]\\{,60\\},\\|[ ]\\{1,60\\}or \\)[ ]\\{0,60\\}" ;delimiter
"\\)*")
"\\(?:"
"\\(?:[ ]\\{,60\\}-+[^- \n][^ \n]*\\)+" ;option
;; "\\>"
(concat "\\("
;; optarg followed by description or newline
"\\(?: [^- \n][^ \n]*\\($\\|[ ][ ]+\\)\\)"
"\\|"
"\\(?: <[^ \n]*>\\)+" ;<optarg>...
"\\|"
"\\(?: \\[[^ \n]*\\]\\)+" ;[optarg]...
"\\|"
"\\(?: [A-Z][-:_@A-Z0-9]+\\)" ;OPTARG
"\\|"
"\\(?: [a-zA-Z]=[^ \n]*\\)" ;optarg=value
"\\)")
"?"
"\\)+\\)"))
opts
opt doc
opt-beg-pos opt-end-pos
doc-beg-pos doc-end-pos
doc-column)
(while (let (case-fold-search)
(re-search-forward rgx nil t))
(skip-chars-forward " ")
(setq opt-beg-pos (match-beginning 1)
opt-end-pos (match-end 0)
doc-beg-pos (point)
doc-end-pos (line-end-position)
doc-column (save-excursion
(goto-char doc-beg-pos)
(current-column))
opt (replace-regexp-in-string
" or " ", "
(replace-regexp-in-string
"[ \t\f\v\r\n]+" " "
(buffer-substring-no-properties
opt-beg-pos opt-end-pos)))
doc nil)
;; Look for a description which may span multiple lines.
(if (eolp)
;; Description may start on the next line.
(save-excursion
(forward-line)
(skip-chars-forward " ")
(when (or (eolp) (bolp))
(forward-line)
(skip-chars-forward " "))
(if (or (eolp) (bolp)
(<= (current-column)
(save-excursion
(goto-char opt-beg-pos)
(current-column))))
(setq doc-column nil)
(setq doc-beg-pos (point)
doc-column (current-column))))
;; The description starts on the same line so be more discerning
;; about parsing the next line if it looks like an option.
(save-excursion
(forward-line)
(skip-chars-forward " ")
(if (or (eolp)
(bolp)
(and (looking-at "-")
(< (current-column) doc-column)))
(setq doc-column nil)
(setq doc-column (min doc-column (current-column))))))
(goto-char doc-beg-pos)
;; Parse indented text.
(while (and doc-column
(not (eobp)))
(setq doc-end-pos (line-end-position))
(forward-line)
(skip-chars-forward " ")
(when (or (eolp)
(< (current-column) doc-column))
(setq doc-column nil)))
(goto-char doc-end-pos)
;; (save-excursion
;; (goto-char doc-beg-pos)
;; (setq doc-end-pos
;; (min (+ (point) 300)
;; (or (and (re-search-forward
;; "\\=\\(.\\|\n\\)+?\\.\\([ ][ ]\\|[ ]*$\\)"
;; doc-end-pos t)
;; (match-beginning 2))
;; doc-end-pos))))
(setq doc (replace-regexp-in-string
" *\n *" " "
(pcmpl-args-strip
(buffer-substring-no-properties doc-beg-pos doc-end-pos))))
(when pcmpl-args-debug-parse-help
(pcmpl-args-debug "Found option: %S %S" opt doc)
(let ((ov (make-overlay opt-beg-pos opt-end-pos)))
(overlay-put ov 'face '(:background "light green")))
(let ((ov (make-overlay doc-beg-pos doc-end-pos)))
(overlay-put ov 'face '(:background "pink"))))
(push (cons opt doc) opts))
(setq opts (nreverse opts))
;; We assume that options without descriptions are probably aliases
;; so we assign them the subsequent option's description.
(let ((lst opts) el)
(while lst
(setq el (pop lst))
(when (and lst (string= "" (cdr el)))
(setcdr el (dolist (p lst)
(when (not (string= "" (cdr p)))
(return (cdr p))))))))
(when pcmpl-args-debug
(pcmpl-args-debug "Found %s options in %f seconds"
(length opts) (- (float-time) start-time)))
opts)))
Does anyone see what could be causing the issue here with compatibility? For reference, here's the (commentated) changelog for Emacs 25.1.1. You can also, of course, view the changelog via C-h n in Emacs 25.1.1.
EDIT 1:
I forgot to mention that pcmpl-args malfunctions differently depending on whether it's in M-x eshell or M-x shell. In Eshell, it outputs No completions of - for something like rm; in M-x shell, it outputs a list of filenames containing -'s or whatever you tried to complete. In either case, this is obviously incorrect as it's supposed to output a list of options to that command.
EDIT 2:
According to a discussion on reddit that I had with someone, this issue appears to be platform-dependent. The person in question is unable to reproduce my issue, however, they are running on Ubuntu 16.04 while I'm running on OS X 10.11.6. Check out the discussion thread for more details.
EDIT 3:
Upon further inspection, I've found that the output of man is different in different Emacs versions on my platform (OS X 10.11.6). I took the output of *pcmpl-args-output* in both versions and diff'd them. I found that the newer version of Emacs is using some special bytes for hyphens and other things to display the output of manpages, which makes a lot of sense now in terms of why that's breaking pcmpl-args. The regexps in pcmpl-args-parse-help-buffer are failing in the new version because there are now some weird bytes in the output of man that are causing the function to not recognize any options (because it's not seeing any normal hyphens). I'm going to look into this further since it's obviously specific to OS X and I'm guessing that there must be some way to disable this special output that is breaking pcmpl-args-parse-help-buffer.
