Ip domain still can access with https event not activate

all ok, but i realize google index my https ip adress and hurt my seo…
i dont activate in first place, and without https ip domain still ok and redir to nginx page…how to fix it, anyone can help?
im using https domain with letsencrpt