People are afraid of those who envy or hate them but there is nothing to fear. If you have done nothing bad, you have nothing to fear because Allāh ﷻ has said He ﷻ will defend you and what they wish for you will come on them. So keep yourself clean and do not be afraid.