As you can see the borders are unbalanced and the buffer is "pushed" from the top left.
This is likely to be window manager related (bspwm in my case). Because when I for example full screen emacs after setting the internal border width everything looks as expected. Same when I open another application or anything like that which "repositions" and that causes a "redraw":
Can this be solved from within my .emacs.d or do I need to look in my window manager or somewhere else?