') returns its argument sexp - any Lisp object (thing).
#') returns its argument sexp as a function object. This also means that in code you byte-compile it tells the byte compiler that that thing is a function.
'foobar just evaluates to the symbol
#'foobar evaluates to a function.
'(lambda) () (something)) evaluates to the list whose car is the symbol
lambda, whose cadr is
nil, and whose caddr is the list
(something). There is no way for the byte-compiler or any code that uses this return value to know that that list is to be interpreted/used as a function (except code that in fact uses it that way).
#'(lambda () (something)) evaluates to a function that accepts no arguments and invokes function
something (passing it no args). (
(lambda () (something) evaluates to the same function - unquoted
lambda forms are self-evaluating.)
In Lisp, data and code (programs) have the same syntax.
(lambda () (something) is both a list (when not evaluated) and a function (when evaluated). When you quote such a list,
'(lambda () (something)), you get just that list, not a function.
If you then use that list as a function, it works, because when evaluated it is interpreted as a lambda form, which returns an anonymous function (which can have the same form as the lambda form itself).
When Emacs - in particular the byte compiler - can know that some bit of code is a function it can act accordingly, error-check, optimize, etc. If all Emacs knows is that there is a list whose car is the symbol
lambda etc., there's no way it can or should make any assumption about it representing a function. The code in which it occurs could use it in some other way.
See the Elisp manual, nodes Anonymous Functions and Quoting.