Is there a way to manage packages in emacs where I can point to a git repository directly? (like vim-plug for eg).
There are several:
Quelpa allows for fetching packages from arbitrary sources and building them into a format that can be installed by package.el
Install Emacs packages listed on MELPA, GNU ELPA, or Emacsmirror, or provide your own recipes. Packages are cloned as Git (or other) repositories.
Assimilate Emacs packages as Git submodules.
El-Get allows you to install and manage elisp code for Emacs. It supports lots of different types of sources and is able to install them, update them and remove them, but more importantly it will init them for you.
There is a detailed comparison on the straight.el github page.