I have an identity for my sweet red berry plant! It's a Chilean Guava, also known (presumably only in NZ) as a New Zealand Cranberry. Thanks to Tom from Sydney on Flickr for providing the first diagnosis of the Chilean Guava, after which I googled and found that my plant matched the images and descriptions. Then my workmate brought in a sprig from her cranberry bush, and I was puzzled to see that it matched that too. She'd also brought in the plant information card from when she bought the plant, and eventually I realised that the botanical name was the same for both - myrtus ugni (I'm sure it means pretty)
So there we have it - it's a cranberry and a guava. Soon it will be dessert - if I can stop snacking on it when I get the mail, that is!