The guy asked you to golly off, so he basically closed off any future communication between you. You were completely ok in not communicating with him after that. Had you seen him drop the money, then you should have given it to him. Otherwise, finders keepers.
I generally don't give beggars money, but prefer to give to organized charities. Like homeless shelters that give meals, offer clothing, toiletries and provide bathing and laundry facilities. I know some might disagree with this, but I have no way of knowing what any random person's background is or what they might do with the money. At least I know where my funds go in the prior situation.
I found $50 one night, next to a curb, with no cars in sight. It seemed as if someone had dropped it out of their pocket getting into or out of a car, but how to figure out who that was, or when, was basically impossible. Two 20's and a 10. I just put it in the little "Found money" tin that we have. We put mostly coins that we find on the ground there. I figured that if I handed it over to the cops, that would have opened another can of worms and eaten a bunch of time with little likelihood the owner would be located, or that I'd get it back if they weren't. What more can you do?