I see your point, and I think your scheme is a good one and
would solve many problems. However, 99% of the spam I am getting
now is coming from virus-infested PCs (i.e. spambots on PCs the
owners of which don't even know they are being used for spamming).
Since I run my own mail server, they send stuff directly from
their address to my port 25. Since there is no middle man, then
I have to deal with these connections myself. I can't throttle them;
I can only reject them or accept them. (More precisely, I could
run a tarpit and slow them down, but also slow down real mail I want
to get quickly.)

How could I avoid this with your scheme? I see how your scheme
would work if the email was relayed somewhere on the way, but
what about point-to-point connections?

Of course, one could combine the two schemes: I could accept from
trusted servers, and these servers, if relaying, would throttle
stuff from addresses not (yet) known to be non-spammers.