Emacs for some reason did not vacate equal spaces for each line number, So the lines were starting at different places which made it harder to keep track of alignment. It wasn't just the relative mode which had this problem, even the absolute mode does. Linum mode is quite heavy when compared to display-line-numbers.el.
This is what fixed my issue. All this is doing is, when the buffer is being setup for reading it would read the last line number and add the width of the number as the display-line-numbers-width.
(defun display-line-numbers-equalize () "Equalize The width" (setq display-line-numbers-width (length (number-to-string (line-number-at-pos (point-max)))))) (add-hook 'find-file-hook 'display-line-numbers-equalize)