This is done via a process filter.
By default comint-output-filter-functions includes comint-watch-for-password-prompt, which is the filter function that handles this.
If it sees text matching comint-password-prompt-regexp then it calls send-invisible to prompt the user for a password.
For more information about how process filters work, refer to C-hig (...
Yes, if you want to make shell buffers limited in length, add the function comint-truncate-buffer to comint-output-filter-functions. You should be able to add this to your .emacs:
(add-hook 'comint-output-filter-functions #'comint-truncate-buffer)