How secure people feel in a particular region is obviously linked to the actual crime suffered in that region but the exact relationship between crime and its fear is quite subtle. Two regions may have the same crime rate but their local perception of security may differ. Equally, two places may have the same perception of security even though one may have a significantly lower crime rate. Furthermore, a negative perception might persist for many years, even when crime rates drop. Here, we develop a model for the dynamics of the perception of security of a region based on the distribution of crime suffered by the population using concepts similar to those used for opinion dynamics. Simulations under a variety of conditions illustrate different scenarios and help us determine the impact of suffering more, or less, crime. The inhomogeneous concentration of crime together with a memory loss process is incorporated into the model for the perception of security, and results explain why people are often more fearful than actually victimized; why a region is perceived as being insecure despite a low crime rate; and why a decrease in the crime rate might not significantly improve the perception of security.