As the README will tell you, the
:mode keyword allows you to define a deferred binding (i.e. on-demand), which will load the package when visiting a buffer which filename matches the expression you provided.
:config keywords are what you are looking for:
They allow you to respectively execute code before and after the package is loaded.
Keeping this in mind, here's what you could do to only load yasnippet in prog-mode buffers:
:init (add-hook 'prog-mode-hook #'yas-minor-mode)
Using this snippet, yasnippet will actually be loaded on startup, but will only be enabled in the buffers which major mode derive from
prog-mode. The appropriate snippets (if any) will then be loaded just-in-time.