At the beginning of my Cuba trip I was tired and not used to the heat so I did not take many pictures. I was also afraid that people will get angry if I take photos of them without asking. But once I got to Trinidad I forgot about all that and I started taking tons of pictures and could not stop anymore. There was so much to see, so many interesting people and things behind every corner.
While I was walking around I saw this barbershop with open doors, almost on the street. I took a picture, the barber recognized it and he did not get angry at all, instead we started talking to each other.
Actually no one ever got really angry at me (so far) by taking pictures of them. Some people start posing and enjoy the attention, some smile and some just look at me a little confused. No one ever shouted at me or forced me to delete the picture, which is a constant fear of mine while doing street photography.
That is the same when you are traveling. You might be scared to get robbed or hurt by someone. Of course that can happen. But most of the time you will see how kind people are even if you are foreigner and sometimes just because you are a foreigner.
So never let yourself be held back by your fear. If you just start traveling (or taking pictures) you will experience help and kindness by strangers much more often that you could have imagined.