Admitedly, the 520i will be slower as it is heavier than the 320i. Handling probably is better in the 320i but that could be changed if you choose to mod the suspension.

One thing I have noticed here in America are rather high residuals on E36 cars. You could probably get a 5 series with the same or less miles than the 3 for less money. At least that is how it is here.

What it really boils down to is what type of car you prefer. If you want a large car get the 5 if you want something smaller and a bit sportier get the three. My friend has a 328 and i have a 525 and we both love our cars but their characters are very different. It all depends on what you prefer.