I just copied this whole blog (wow, it was >500 MB already, but that is mostly log files and some downloads) to a better server, which can handle all those requests much better. The last few days the old server was completely overwhelmed (it was just a server with a crappy 1mbit dsl line staying in the office).
This is actually the first time this blog is being hosted on a real server, in the past it was always hosted by my own server staying at home or at the office with some crappy 1-2mbit dsl and a dynamic IP, which changed quite often and confused Google a lot. Hopefully with the new static IP things will be better now :)