What short comings can I expect from this mode, and what are some use cases for which I should really prefer my operating system's terminal emulator?
ansi-term almost exclusively, so I would say "almost".
Speed is your main concern -- anything which rapidly generates large amounts of output is going to cause some pain, as Emacs won't keep up with it. If I know I'm about to issue such a shell command, I pull up a dedicated terminal emulator.
If you're using Emacs 24 then https://www.emacswiki.org/emacs/AnsiTermHints#bidi can cause awful slow-downs in certain situations.
Emacs commands can potentially put the buffer in an inconsistent state, which is confusing at best. You might find https://emacs.stackexchange.com/a/17086 helpful.