Poverty is not the issue, though it is the cause. Restated, I have no problem with someone begging for change _because they are poor_. The problem comes from being hassled, repeatedly, daily, by anywhere from 1 to 8 (not kidding, kept count) people who are trying to sell me something, trying to get me to give them change, or trying to have me sign a petition of some sort. Where does the line for my choosing not to interact with anyone blur into being forcibly interacted with?
