(customize-set-variable case-fold-search  nil)

Or bind that variable (option) in your own command that is otherwise just a wrapper around `query-replace`.  This has the advantage that it doesn't change the variable value for general use, outside of query-replacing.

See also variable `case-replace`, which controls case for the replacement text.

See also `C-h f query-replace`, where it says, for example:

> Matching is independent of case if `case-fold-search` is non-nil and
FROM-STRING has no uppercase letters.  Replacement transfers the case
pattern of the old text to the new text, if `case-replace` and
`case-fold-search` are non-nil and FROM-STRING has no uppercase