We went to Belize in May. It was by far my favorite port! We booked the Lamanai Mayan ruins excursion through a private vendor. We took a bus through the city with our awesome guide telling us about their history and their current goings on. We then took a boat through the New river, had a Belize lunch that they fixed for us, then walked the Lamanai ruins. My boys and I climbed 125 to the top of the ruins while the hubby and mother in law watched. It was so beautiful looking down on all the trees with the Howler monkeys making their odd noises. I have the most wonderful picture of my boys and myself at the top of the ruins with the New river behind us.
Before and after the excursion we walked very little in the city itself, but the little that we did, I didn't feel "unsafe". We had people right at the port asking us where we were going and giving us guidance in getting there. There was a lady that insisted on braiding my hair and I might have done it if we weren't in a hurry to get to our excursion. I didn't see her after.
As for gang related issues. We have that here in the states. I live about 45 minutes from Seattle and still see it here in my neck of the woods. Sadly, gangs are everywhere.
I have every intention of going to Belize again but next time we're doing the cave tubing! Hopefully we can go again next year. This year we're doing Cayman Island, Cozumel and Montego Bay. I hope you decide to do something at this port. It would truly be a shame to miss such a beautiful country!